Manage many OpenTelemetry Collectors with OpAMP and Git

To manage many OpenTelemetry Collectors with OpAMP, you need three pieces: the upstream OpAMP Supervisor with a collector on every host, an OpAMP server (myriarchd), and a Git repository that declares what the fleet should run. Operators change the fleet by merging a change in Git; the server renders one configuration per collector, offers it over OpAMP and rolls it out in waves, with approvals and rollback, and records every action in the audit log. This guide walks through the setup from an empty repository to the first rollout. It is designed for around 1,000 collectors, with headroom up to 5,000 (design targets, see the architecture).

Myriarch is alpha; the README lists the support level of each component. Read OpAMP, the Supervisor and the Collector and the concepts first if OpAMP or the terms blueprint, fleet and rollout are new to you. The comparison shows when another option fits better.

1. Run the server

The server is one binary that needs PostgreSQL. To try it on your machine, follow Local development or Docker Compose; both run the server in development mode with well-known logins, which is not a production setup. For a real installation use Install with Helm (the chart is beta from v0.2.0) and go through the production checklist.

2. Start a GitOps repository

Copy the starter in gitops-template/ into an empty repository on your Git server, protect the default branch so that changes go through merge requests, and give the server a read-only token. The repository holds blueprints/, fleets/, policies/, executors/ and inventory/ below the path that you configure as gitops.path; the server reads every YAML file there as an object, except samples/ directly below that path and directories whose name starts with a dot. The examples in steps 3 and 7 come from the complete example repository in examples/quickstart; the starter template uses other names and values for sites and policies, so take the structure from the examples and the names from your own repository. See Git as the source of truth for the layout and the rules.

3. Describe your fleets

A fleet selects agents by their attributes and says what they run. This is an excerpt of examples/quickstart/fleets/linux-hosts.yaml, shortened (it leaves out description, displayName, parameters and the selection frame for the UI):

apiVersion: myriarch.dev/v1alpha1
kind: Fleet
metadata:
  name: linux-hosts
spec:
  priority: 100                    # the higher priority wins when selectors overlap
  selector:
    matchAttributes: {os.type: linux}
    matchExpressions:
      - {key: site.type, operator: In, values: [edge, datacenter]}
  blueprint: linux-host            # required
  collector:
    distribution: otelcol-contrib
    version: 0.162.0               # the Git target, without "v"
  update:
    executor: updater
  rollout:
    configPolicy: config-canary
    versionPolicy: canary-then-all

The configuration itself comes from a blueprint: layered, parameterized, rendered on the server. The rollout policies define the waves, approvals and failure handling (Rollouts). Give fleets whose selectors can overlap different priorities; two fleets with the same priority leave an agent unassigned. Every field is listed in the GitOps schema reference.

4. Check every change in CI

Run myrctl validate on merge requests. It runs the same validation as the server, so a change that passes is accepted by it:

myrctl validate --strict --components ci/components-otelcol-contrib-0.162.0.json config

Render the configuration of a sample agent to review what a host would receive, not only the blueprint source. See Validate a GitOps repository for the options and a CI pipeline.

5. Connect the server to the repository

Set the repository in the configuration of myriarchd (gitops.*; see the configuration reference). The server polls every 60 seconds by default, and a webhook at POST /api/v1/gitops/webhook syncs at once. It applies a revision only if the whole revision is valid. GET /api/v1/gitops/status shows state: ok when it has read your repository, and the fleets appear in the UI.

6. Enroll the hosts

Create an enrollment token in the UI (Settings, Tokens). The dialog Add host renders the install command for each platform; copy it to the host and run it (the installer runs with --server on Linux and -Server on Windows). The manual path, with the installation script and its own inputs, is the alternative. The script sets up the upstream supervisor, the collector and optionally myriarch-updater. The site.* options become attributes that your fleet selectors match on. Connect a host has the inputs and the exact commands; Linux hosts with systemd are beta and Windows hosts are experimental (levels and tested platforms). Collectors in Kubernetes run as pods of the Helm chart myriarch-collector with unchanged upstream images; see Install the collector with Helm.

7. Change the fleet

Edit a blueprint or a fleet in Git and merge. The server renders the fleets again and creates a configuration rollout for each fleet whose rendered configuration changed. With the quickstart policy config-canary, the canary hosts get the change first, the rollout waits for their health check and stops for approval before the rest of the fleet follows. If the failure threshold is reached, the policy decides between rolling back and pausing. Roll out a configuration change shows the happy path and the failure path, and the actions you have in the UI and the API.

Do not change the configuration and the collector version in the same merge request; the two kinds of rollout run independently. To change versions, continue with Upgrade the OpenTelemetry Collector on many hosts.

8. Audit

The UI and the REST API only operate inside the frame that Git sets, and every action is audited: approvals with the user and the reason, and a version chosen in the UI (fleet.version.select, with who, when and why). The audit log is part of the API reference (REST API).