Troubleshooting
Symptoms, causes and fixes for the local system and for installations, plus the endpoints that help to diagnose them.
Start with the log of myriarchd and the page of the object in the UI (agent, rollout, GitOps). The tables below are
ordered by area. Issue codes of the GitOps repository, such as blueprint_not_found, are explained in the
GitOps schema reference.
Local development
| Symptom | Cause and fix |
|---|---|
too many open files, connections break | Raise the limit with ulimit -n 65536 in every terminal that runs myriarchd or the simulator. The macOS default is 256. |
myriarchd dev does not start: address already in use | Port 8080 (myriarchd) or 54329 (PostgreSQL) is taken. If a dev PostgreSQL with the dev credentials already runs there, it is reused, even for another --data-dir. For parallel environments set dev.postgresPort in a -c file. |
myriarchd dev refuses to start: “refusing to start dev mode” | The settings look like production (non-loopback --listen, another database host, OIDC, TLS, an https publicUrl). Dev mode must stay on the local machine; see Local development. --allow-nonlocal overrides the check on purpose. |
| The first start takes long and downloads files | A one-time download of the PostgreSQL binaries (about 30 MB) into myriarch/embedded-postgres below the user cache directory (~/Library/Caches on macOS, ~/.cache on Linux). |
A postgres process is still running after kill -9 of myriarchd | The embedded PostgreSQL is stopped only on a regular exit: kill $(head -1 .dev/postgres/data/postmaster.pid). |
The UI shows nothing, or / answers 404 | myriarchd was built without the embedded UI. Run make web-build && make build (build tag embedui). |
A real collector stays in applying for more than 30 seconds on its first start | macOS Gatekeeper checks otelcol-contrib at the first start; later starts take seconds. |
Server and GitOps
| Symptom | Cause and fix |
|---|---|
/readyz answers 503 not ready | The body names no reason. The cause (the database or the event bus; the schema is checked at start) is in the log line readiness check failed. /healthz answers ok as long as the process runs. |
myriarchd stops at start with the database schema is incompatible with this myriarchd (from v0.2.0) | A newer version migrated the database in a way that this version cannot work with. The error names the schema version of the database, its compatibility level and the schema level of this binary. Nothing was changed. Start a version whose schema level is at least the compatibility level in the error, or restore a backup of the database from before the upgrade. |
myriarchd stops at start with the database schema is at version N but this myriarchd needs version M and database.migrateOnStart is false (from v0.2.0) | The database lacks migrations and the server is not allowed to apply them. Run myriarchd migrate (it needs only the database settings; --check shows the state without changing anything), or set database.migrateOnStart to true. |
The log warns the database schema is newer than this myriarchd but compatible with it (from v0.2.0) | An older version runs on a database that a newer version migrated: after a rollback, or while a rolling update is under way. It works because the compatibility level allows it. Run the version that matches the schema as soon as you can. |
GitOps status error | GET /api/v1/gitops/status lists the issues with file and line. Reproduce them locally with myrctl validate <repo> (see Validate a GitOps repository). An invalid state never replaces the applied one. |
| Changes in Git do not arrive | The server polls every gitops.pollInterval (60 s by default, 10 s in dev mode). Trigger a sync with POST /api/v1/gitops/actions/sync or the webhook POST /api/v1/gitops/webhook, and check the status for a validation error. |
Two leaders in system/info | Only briefly after a crash: a dead leader keeps its last heartbeat row. system/info names only the replica with the newest heartbeat as leader. |
x509: certificate signed by unknown authority for OIDC, Git, GitHub, the mirror, TUF or Teams | myriarchd does not know the CA of the target. Set outbound.caFile to a PEM file with the CA (it applies to all outbound connections; in the Helm chart mount the CA bundle with extraVolumes and extraVolumeMounts, for example from a ConfigMap, and set config.outbound.caFile to the mounted path). For Git there is also gitops.auth.caFile, for an AWX executor its caFile; both add to the trust store. Check with myriarchd config validate --config ..., which reads the file. |
myriarchd config validate prints ERROR lines | Each line names the setting (<path>: <message>). File and certificate errors disappear when the secrets are mounted. Use --skip-files in CI. |
helm test or myriarchd healthcheck fails with readyz failed: cannot reach ... (from v0.2.0) | The Service has no ready replica, or the NetworkPolicy keeps the test Pod out. With config.server.trustedProxies or config.opamp.clientCertHeader the chart adds no rule for the test Pod; list it in networkPolicy.ingress.from (Helm test). connection refused and no answer within ... both mean that there was no answer; exit code 3. |
myriarchd healthcheck fails with readyz failed: ... answered 503 or opamp failed: ... (from v0.2.0) | Exit code 1: the server answered, but not as expected. For 503 see the row about /readyz above. answered with 404 at the OpAMP path means that --opamp-path differs from opamp.path. refused with 401 although a token was given means that the token of --token-file is revoked, expired or unknown, or that the server also requires a client certificate. Exit code 2 means a wrong flag or a token file or CA file that cannot be used; no request was made. See Server behaviour. |
myriarch_opamp_connect_rejected_total{reason="unauthorized"} rises regularly although no collector is misconfigured (from v0.2.0) | myriarchd healthcheck without --token-file counts as one unauthorized connect per run, for example the health check of a container every ten seconds. That is intended and far below the alert MyriarchUnauthorizedConnects (one per second); a check with a token counts as an accepted connect instead. |
Installing a host from the server
Available from v0.2.0. The installer of a host that was installed with the command of the server (Linux: install-linux.sh --server, Windows: install-windows.ps1 -Server) ends with one line [myriarch-install] ERROR [<code>] <message>. The code is the stable interface; the message names the cause and the next step. The options in the table are those of Linux; on Windows they are the parameters of the same name (-Server, -Fleet, -PlanWait, -Verifier, -CollectorVersion). The table lists the codes that this way of installing adds or uses in a special way; the other codes are in When the installer stops. Nothing is installed when one of these codes ends the run, except where a row says otherwise, and the installer removes nothing that it did not create.
| Code | Cause | Next step |
|---|---|---|
plan_failed | The request for the plan got no answer: the server cannot be reached, its certificate is not trusted, or it answered with an error of its own (HTTP 500, 502, 504). The installer asks again for a few seconds first. | Check the address of the command, the network and the firewall. For a server with a certificate of a private CA put the CA file on the host and pass it (--ca-file; the command of the server does it when you give the path). On Windows the installer trusts the certificate store of the machine only: import the CA there (Import-Certificate -FilePath ca.cer -CertStoreLocation Cert:\LocalMachine\Root) and run the command again. Run the command again. If the server answers with 5xx, read the log of myriarchd. |
plan_invalid | The answer of the server breaks the rules of a plan, and the installer reads a plan as data only: a key that it does not know or a repeated key, a value that does not match the pattern of its key, a character that a plan may not have, more than 64 lines or 8 KiB, another format than 1, an artifact or endpoint URL on another host than --server, a file name that is not the one expected, a collector version other than the one that you asked for, or a redirect to another origin. The message names the rule. | A server of the same release as the installer never answers like this. Check that the host runs the installer bundle of the server that you name in --server (the command of the server does), and that no proxy or load balancer between them rewrites the answer or redirects to another host. A redirect from the server to itself is followed; one to another host is not, because the request carries the token. Use the manual path if you cannot fix the path. |
plan_rejected | The server refused the plan. The message carries the reason that the server gave, for example fleet_mismatch, no_fleet, fleet_ambiguous, rollout_blocking, no_target_version, version_not_allowed, unknown_version, invalid_value, unauthorized or not_found. | unauthorized: the token is unknown, revoked or expired, create a new one. not_found: the server has install.enabled off, or the address is not that of a myriarch server. no_fleet, fleet_ambiguous, fleet_mismatch: compare --site-type, --site-id and the other site options and the labels of the token with the selectors and priorities of the fleets in Git, and leave --fleet out if you are not sure. rollout_blocking: a version rollout of the fleet is paused, failed or rolling back, resolve it first (Rollouts). no_target_version: the fleet has no target version or is on hold, set one in Git or in the UI. version_not_allowed, unknown_version: leave --collector-version out, or choose a version inside the frame of the fleet that the release tracker knows. |
plan_unavailable | The server cannot give a plan now: it answered 503 (the database, or the artifacts are not ready) or 429 (too many requests from this address), or it was still preparing the artifacts after --plan-wait seconds (900 by default). The first host of a platform makes the server download about 112 MB, and a target that was just changed needs seconds for the rollout planner. | Run the command again after the time the message names. A server that stays at preparing has an artifact source that fails: read the log of myriarchd and GET /api/v1/artifacts, and check artifacts.sourceBaseUrl and the outbound policy. Give --plan-wait more seconds on a slow link. |
https_required | The address of the server, a URL of the plan or a redirect is not https. The installer never sends the token over plain http. | Give the https address of the server. A development server over plain http needs the lab variant of the command, which adds --insecure; it is for labs only. A plan or a redirect that names http means that the server or a proxy in front of it is set up wrongly: fix server.publicUrl. |
conflicting_install | A deb or rpm package of the upstream project is installed and its unit otelcol-contrib.service or opampsupervisor.service is active or enabled. The installer stops before it contacts the server, so no token is sent and nothing is created, and it removes nothing. | Linux: systemctl disable --now <the units of the message>, remove the package with your package manager, run the command again. Windows: the message names the service opampsupervisor or otelcol-contrib of an MSI of the upstream project; uninstall the MSI (Settings, Apps, or msiexec /x with the MSI file), or sc.exe delete for a service that you created yourself, and run the command again. The Windows installer checks this before it creates a working directory or asks the server. |
supervisor_missing | When the installation step of the supervisor is over, /opt/myriarch/supervisor/opampsupervisor (Windows: C:\Program Files\Myriarch\supervisor\opampsupervisor.exe, the path of the service) does not exist, is no regular file, is not the file whose SHA-256 was verified, or on Linux is not executable. The message names the path. | A file that is gone or not executable: look for a security or cleanup tool and for a noexec mount (findmnt -T /opt/myriarch/supervisor), run the command again. On Windows look at the protection history of Microsoft Defender or of your endpoint protection for a quarantined opampsupervisor.exe (Windows Security, Protection history, or Get-MpThreatDetection), allow the folder, and run the command again. A directory or a link at the path: remove it yourself, the installer does not replace it. |
sigstore_verifier_missing | The installer did not find the myriarch-updater (Windows: myriarch-updater.exe) of its installer bundle next to deploy/, or it is not a regular file. | Run the installer from the unpacked installer bundle (the command of the server does), or pass --verifier (-Verifier) with cosign 3.0.4 or newer. |
language_mode_unsupported (Windows) | PowerShell runs the installer in Constrained Language Mode, which App Control for Business with script enforcement does to a script that no rule allows. It is the first check of the installer; nothing was read, asked or changed. | Allow the installer by a hash rule for this release and run the command again (Connect a host, App Control and AllSigned). |
execution_policy_blocked or privilege_required (Windows, raised by the command itself) | Group Policy sets the execution policy to AllSigned or Restricted and wins over the policy that the command sets for its process, or the session is not elevated. Nothing was downloaded. | Ask the owner of the policy for an exception, or start an elevated Windows PowerShell (Run as administrator) or SYSTEM. |
sigstore_verification_failed or checksum_mismatch after the plan | An artifact is not what the upstream project signed or what its checksum file says, or (sigstore_verification_failed only) the host cannot reach the Sigstore TUF repository (tuf-repo-cdn.sigstore.dev), for example behind a proxy or a firewall: without --trusted-root the installer verifies through TUF and the message carries no detail. Nothing of it was installed or run. | Run the command again to get a new plan. To tell the causes apart, run sudo curl -fsS -o /dev/null https://tuf-repo-cdn.sigstore.dev/timestamp.json on the host: the command runs under sudo, which does not pass the proxy variables of your shell, so test in the same environment. If it fails, the host cannot reach the repository (a host that reaches TUF only through a proxy uses --trusted-root): create the file on a machine that can, transfer it by a trusted channel to a place only root can write and append '--trusted-root' '<absolute path>' to the options of the command (see One command). If the fetch succeeds and the error repeats, the artifact cache of the server holds a bad file or a proxy changes the bytes: do not work around it, report it as described in SECURITY.md if you cannot explain it. |
download_failed with HTTP 401 | The install grant of the plan expired (install.grantTtl) or the enrollment token was revoked in the meantime. | Run the command again. |
option_conflict | An option does not combine with --server (-Server): --with-updater and the --updater-* options (the plan decides whether the updater is installed, and it comes from the installer bundle), --release-base-url, --supervisor-version, --verify-signatures skip, --collector-version latest (Linux; on Windows latest is option_invalid). --fleet and --plan-wait without --server are option_conflict as well. | Leave the option out. |
Agents and assignment
| Symptom | Cause and fix |
|---|---|
An agent does not appear, or the handshake answers 401 | Check the enrollment token (header Authorization: Bearer myre_...); in dev mode it is optional. Use a ws:// endpoint for a plain local server. |
The installer of a host stops with ERROR [<code>] ..., or a Windows host has no service otelcol-supervisor after the installer ran | The code names the cause, for example option_missing (the collector version is required), sigstore_verifier_missing or template_missing. A supervisor from an upstream MSI is another service (opampsupervisor) that myriarch does not manage; from v0.2.0 the installer stops with conflicting_install when it finds one, and it names a missing supervisor file (supervisor_missing) and a script that PowerShell runs in Constrained Language Mode (language_mode_unsupported). The table “When the installer stops” in Connect a host lists the codes of a first installation and the fixes. The codes of the installation from the server (plan_failed, plan_invalid, plan_rejected, plan_unavailable and others) are in Installing a host from the server. |
The agent is unassigned | No fleet selector matches. Compare the attributes on the agent page (site.type, site.country, …) with the selector of the fleets, or use Fleets, “Which fleet?”. Two fleets with the same priority that match the same agent also leave it unassigned (fleet_ambiguous): give overlapping fleets different priorities. |
| The agent reports “the instance uid is bound to another credential” | The instance UID was registered with another enrollment token or certificate (a cloned machine, a re-enrolled host). The administrator action POST /api/v1/agents/{id}/actions/reenroll releases the binding. It is also needed after the old token was revoked, unless opamp.autoRebindOnTokenRotation is on. |
429 at login or on API calls | Rate limiting: the dev login and the bootstrap admin allow 5 attempts per minute per address, repeated invalid bearer tokens delay the client address (header Retry-After). Only distinct invalid tokens count; recently valid tokens of other clients still work. Behind an ingress set server.trustedProxies, otherwise all clients share the address of the ingress. |
The supervisor reports FAILED and the collector stays on the old configuration | Intended (automatic_config_rollback). supervisor.log and lastError on the agent name the validation error. |
| Dashboard shows 100 % “outdated” after a new release | By definition: outdated means below the newest stable release. The target version of a fleet is decided by Git (spec.collector.version). |
Rollouts
| Symptom | Cause and fix |
|---|---|
A configuration target stays in_progress and the agent reports applying after it restarted | A supervisor that restarts while it applies a configuration reports the stored applying again, and nothing ends it. After opamp.staleApplyingAfter (90 seconds by default) on the connection, the server offers the configuration once more (agent event config_offered with the reason stale_applying, counter myriarch_opamp_config_reoffers_total), and the supervisor answers applied. If it does not, the target waits for the applyTimeout of its policy. With opamp.staleApplyingAfter: 0 the server never offers it again. |
A rollout waits forever in waiting_approval | A wave with requireApproval needs an approval (UI, or .../actions/approve with the wave). Waves with no targets are skipped. |
A rollout is paused with errors | The failure threshold was reached. Read lastError of the failed targets, then retry-failed, resume, rollback, or fix Git (a fix replaces the paused rollout). |
A rollout failed with “fleet … was removed from the GitOps repository” | The fleet was removed from Git. The rollout needs your confirmation: .../actions/abort. |
| A configuration target fails with “agent became unhealthy during the health wait” | The collector restarted during healthWait, for example through a simultaneous version update of the same host (a configuration and a version rollout of one fleet run independently). Use retry-failed, or finish the version rollout first. |
A version rollout stays pending or “retry later” | The artifact is still being downloaded or verified. GET /api/v1/artifacts shows pending, downloading or failed with the error. Without internet access use a mirror (Roll out a collector version). |
Targets stay in waiting_window | The rollout policy has maintenance windows and the current time is outside them (site-local time, or UTC where a window sets it). Wait, or use a policy without windows in a test setup. |
An update of a Linux host fails in the step verifying, and the journal of myriarch-updater says preflight: ... fork/exec ... operation not permitted | A defect of v0.1.0: its unit myriarch-updater.service sets NoNewPrivileges=true, which stops the preflight from starting the new collector under the account of the supervisor. The old version keeps running. Fixed from v0.2.0 (run its installer again). On v0.1.0 run systemctl edit myriarch-updater, add the two lines [Service] and NoNewPrivileges=false, save, and run systemctl restart myriarch-updater. Then retry the failed target (retry-failed). |
INSTALL_FAILED ... not retrying for another ... at the updater | The retry brake: the same content is not tried again for 10 minutes after a failure (update.failedRetryBackoff). |
| A webhook or AWX call does not arrive | Is the target in executors.allowedWebhookHosts (an empty list refuses everything; http://host allows plain HTTP) and is its address released by outbound.allowedCidrs (“not allowed by the outbound policy” for private and loopback ranges)? Does the secret environment variable (secretEnv) start with MYRIARCH_EXEC_ and exist in the process (or is the name in executors.secrets with matching allowedHosts)? In production the secret must also be bound to the target host (executors.secretEnvHosts or secretFileHosts; message “is not bound to a destination”). Redirects are not followed. |
An artifact is failed with HTTP 401 or HTTP 403 from the mirror | The mirror rejects the credentials or wants some: set artifacts.sourceAuth or check the token or password and the permissions of the account (read access to the remote or proxy repository). The message is in GET /api/v1/artifacts. The credentials go only to the origin of artifacts.sourceBaseUrl; if the mirror redirects to another host, that host must allow anonymous reads and be listed in artifacts.allowedRedirectHosts. |
An artifact is failed with fetch sigstore trusted root via TUF | myriarchd cannot reach the TUF repository. Check the proxy (HTTPS_PROXY, NO_PROXY) and outbound.caFile, point artifacts.verify.tufRepositoryUrl at a mirror, or set a fixed artifacts.verify.trustedRootFile (trusted_root.json). |
Sign-in
| Symptom | Cause and fix |
|---|---|
OIDC sign-in fails with oidc: issuer did not match, or myriarchd does not start with an issuer error | The issuer must equal the issuer of the discovery document exactly. The formats of common providers and the fixes are in Authentication (sections 4 and 5). Look at the document by hand: curl <issuerUrl>/.well-known/openid-configuration (through the proxy if there is one). |
OIDC sign-in works but ends with access_denied | No role matched (the audit log shows no_role). Check auth.oidc.groupsClaim, roleMapping and the claims of the ID token; see Authentication (section 7). |
Simulator
| Symptom | Cause and fix |
|---|---|
Many connectErrors after myriarchd restarted | Expected: while shutting down the server answers 503 with Retry-After, and the agents reconnect with backoff (1,900 agents take one to two minutes). |
Useful endpoints
| Endpoint | Content |
|---|---|
GET /healthz | Liveness: ok. |
GET /readyz | Readiness: ready, or 503 not ready. |
GET /metrics | Prometheus metrics, for example myriarch_opamp_connections, myriarch_agents, myriarch_config_sync, myriarch_leader, myriarch_rollouts_active, myriarch_gitops_sync_total and myriarch_http_requests_total. In production they are on the separate port server.metricsListenAddr (default :9090). |
GET /api/v1/system/info | Version, replicas, leader, database state, features. |
GET /api/v1/events/stream | Live events (server-sent events). |
GET /api/v1/gitops/status, GET /api/v1/gitops/syncs | State and history of the GitOps sync. |
GET /api/v1/audit | The audit log. |
Still stuck? The production checklist covers the settings that myriarchd refuses, and the
myriarchd config validate reference explains how to check a configuration.