hyperhive/docs/4-implementation/swarm/telemetry.md

Swarm telemetry: metrics, logs and the collector

How the swarm's metrics store, Grafana, log store and OTEL collector connect, and who may reach each. The operator's overview, with the options to touch and how to open each service, is on the services page.

Metrics (VictoriaMetrics + Grafana)

The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it, in two containers at metrics.<swarm-domain> and grafana.<swarm-domain>. Two containers rather than one so you can restart or break Grafana without taking the time-series database with it.

The Grafana container sits on the hive bridge in a network namespace of its own and dials authelia, the secret store, the collector and both datasources by name through it. It binds no TCP port: it listens on the unix socket /run/swarm-grafana/grafana.sock, a directory only the gateway's nginx and the container can open.

They derive together: a store with no UI is unreadable and a UI with no store is empty. To run one without the other, set it directly:

services.hyperhive.deploy = {
  victoriametrics.enable = true;
  grafana.enable = false;
};

⚠️ This starts a database that grows for as long as the swarm runs. See retentionPeriod below before leaving it at its default.

Option	When you'd touch it
deploy.victoriametrics.retentionPeriod	Default 5y. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back.
swarm.grafana.oidc.role	Default Admin for everyone who logs in. Lower to Viewer/Editor if the swarm grows operators who shouldn't be able to reconfigure Grafana.
deploy.grafana.datasourceUrl	Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (https://<swarm.victoriametrics.domain>), so Grafana and the store can run on different hosts.
deploy.grafana.logsDatasourceUrl	Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (https://<swarm.victorialogs.domain>), so Grafana and the store can run on different hosts.

Logging in. Grafana is behind swarm SSO, so the accounts are the authelia ones, and the local login form is always off. Grafana's API still accepts HTTP basic auth for its built-in admin account, so that account's password is a random one minted into the swarm secret store (swarm/services/<grafana client id>/admin/password) and applied on every Grafana start — see credentials.md.

Dashboards. Grafana provisions these from nix/host-modules/swarm-grafana/dashboards/; edits made in the UI aren't kept.

• agents: per-agent resource usage and Claude Code activity. The activity tab ends with a Busiest agents table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The turns tab holds what each agent's own /stats page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the /stats page shows, except the per-bucket token bars the cost tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in hive-agent; before it, they read a flat 10 seconds.
• attic: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the swarm-builder and hive-c0re logs, and the hyperhive.attic.cache_bytes and hyperhive.attic.cache_nars gauges, which a timer in the swarm-attic container (swarm-attic-cache-size) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
• authelia, bao, queue: the identity provider, the secret store and the NATS queue.
• claude-usage: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
• egress: what agents send through the egress proxy, in three tabs. overview has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, the busiest agents and destinations, and data transferred by domain over time (the top 10 domains plus an other line, summed from the proxy's access log in VictoriaLogs; the proxy logs a CONNECT tunnel when it closes, so a long-lived connection shows as one spike at close). refused lists the hosts the proxy refused for each agent. would-deny lists what agents in log mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent.
• builds: the container builds hive-c0re ran on every hive, in two tabs. builds has the count, the failures and the longest, builds over time by outcome, and a table of each build with its id, agent, kind, outcome and runtime. output shows every line of the build picked in the Build filter, with its start and finish lines. It reads the lines hive-c0re logs for each build, so a hive-c0re that doesn't log them leaves it empty. Filter by agent.
• controller: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
• review-classifier: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
• forge: the forge, in three tabs. content has what the forge holds (repositories, users, issues, releases, discussion volume and the issues by label). runtime has the forge process's CPU, memory, goroutines and descriptors. CI runs has Actions run time and queue wait per repo and workflow, with a per-PR table; the Repo and Workflow filters apply to that tab.
• gateway: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
• host: the machine under each hive, from the collector's host metrics: CPU by state, load, memory, swap and paging, disk and network throughput, filesystem usage, the Nix store's share and the busiest agent containers. Filter by hive and host.
• collector: what the hive and swarm OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
• logstore, metricstore: health of the log store and the metrics store.
• grafana: Grafana itself. Dashboards and data sources, alert instances firing, requests by status and their latency, alert rule evaluations and failures, sign-ins, database connections, memory and goroutines.
• builder: the swarm builder. Builds finished, by kind and outcome, their mean and 95th percentile time, the queue depth and the wait before a build starts.
• ci-runner: the forge's Actions runner. Its uptime and job slots, jobs finished by status, how many slots are in use, job time, and how often it polls the forge for work.
• matrix: the Matrix homeserver. Whether its client API answers, how fast, the share of the range it answered, and its database size.
• notify-router: the notification router. Forge events received by type and outcome, todos delivered, review requests that failed, time per event, and the backlog of the agents' queue.

A new panel can read "No data" until its source exists: the push and commit counters on controller appear after the first push webhook, the refused and failure series on collector appear once something fails, and the review-classifier panels fill as review requests arrive.

The collector dashboard. It shows the self-metrics of the hive's collector and of the swarm's collector, which both carry the collector binary's default service.name, otelcol-contrib. The timeseries panels split by server.port: 8888 is the hive collector, 8889 the swarm collector. Every per-container collector (the secret store, the other service containers and the agents' forwarders) also scrapes its own endpoint on loopback and pushes the result up its normal hop, so each has a line of its own, on its own port: agents all report on 8904, service containers on 8890 to 8903. The series carry container (services) or agent (agents) to tell them apart. scripts/check-dashboard-service-names.sh runs in CI and in scripts/pre-push. It fails when a dashboard query selects a service.name that nothing emits.

Where the data comes from. The swarm's OTEL collector, below.

The controller panels come from the controller itself. It exports OTLP through the gateway to the swarm's collector from whatever host it runs on, whether or not that host runs the collector.

Neither container is reachable except through the gateway: each listens on a unix socket only the gateway's nginx can open, and VictoriaMetrics runs in a network namespace of its own with no interface but loopback. VictoriaMetrics authenticates nothing itself; the metrics.<swarm-domain> vhost does it:

• The read API (the Prometheus query, series, label and metadata routes): Grafana's datasource, an agent, or an operator. Grafana sends a bearer that swarm-bao-grafana-metrics-bearer creates in the swarm secret store, and swarm-metrics-auth on the store's host checks it against the same store path. An agent's swarm-metrics sends its swarm-agent.<agent>.<secret> bearer, which swarm-metrics-agent-auth on the store's host checks against swarm/agents/<agent>/queue. It also needs the metrics permission, or the answer is a 403. A request without a bearer needs an operator's authelia session (group admins); without one it gets a 401, not a login redirect.
• vmui: an operator's authelia session, with a redirect to the login. The bare domain redirects to vmui.
• The collector's OTLP push, and its scrape of the store's own /metrics: only the swarm collector's authelia client. Authelia refuses any other session or client there, operators included.
• Every other path, the store's write and admin API included: 403.

swarm-metrics-auth and swarm-metrics-agent-auth read the store with one leaf, deploy.victoriametrics.authBaoClientCertFile / authBaoClientKeyFile. Where the secret store runs on the same host, these default to the leaf it mints. On any other host you place a leaf whose CN is deploy.bao.metricsAuthCommonName, and evaluation refuses the config until you set both. swarm-metrics-auth reads the stored bearer at most once a minute, so a replaced bearer starts working up to a minute later. When the store doesn't answer within five seconds, it checks requests against the last value it read instead of denying, and still refuses a wrong bearer; it tries the store again after three seconds. A revoked or rotated bearer stays accepted while the store is down, for an hour after the last read at most; after that the responder denies again. The store's next answer replaces the value, and a process that holds none (it started while the store was down) denies. swarm-logs-grafana-auth is the same binary and does the same.

swarm-metrics-agent-auth is the log store's swarm-logs-auth binary, with the same grant TTL and rate limit, under deploy.victoriametrics.agentAuth* (table above).

The store's container has no network, so its journal forwarder leaves through a relay. Inside the container, swarm-victoriametrics-egress-relay listens on loopback and passes each connection to a socket the host's nginx listens on, /run/swarm-victoriametrics-egress/egress.sock. nginx sends the forwarder's route to the swarm collector and its token request to authelia, both by name and verified, and refuses every other path. On the host, swarm-victoriametrics-forwarder-oidc.service fetches the forwarder's OIDC secret under the same leaf as swarm-metrics-auth, and the container sees it read-only through a bind mount.

The collector scrapes Grafana's own metrics at https://<swarm.grafana.domain>/metrics, so a collector on a host without Grafana still gets them. The gateway admits only the collector's client to that path; any other session gets deny.

The collector scrapes the store's own metrics the same way, at https://<swarm.victoriametrics.domain>/metrics, so a collector on a host that doesn't run the store still gets them. On this name the gateway admits only the collector's client to /metrics and the push route; any other session gets deny.

Logs (VictoriaLogs)

Each hive ships its journals to one VictoriaLogs at logs.<swarm-domain>, behind the same SSO as everything else: the swarm's own service containers, the hive's daemons and infra containers, and the harness units inside every agent container. The collector below is what writes to it.

Reading them. Open Grafana, pick Explore, and choose the VictoriaLogs datasource — it's provisioned for you. Grafana's Logs Drilldown app is deliberately not installed: it only supports Loki, and no setting here changes that, so Explore is the log browser for this swarm.

Option	When you'd touch it
deploy.victorialogs.retentionPeriod	Default 30d, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter.
deploy.victorialogs.authRate	Default 50r/s per client address, for agent queries checked by swarm-logs-auth; over it and authBurst, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together.
deploy.victorialogs.authBurst	Default 500 requests above authRate before the 403s start.
deploy.victorialogs.authVerdictTtl	Default 60 seconds that swarm-logs-auth keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. A revoked logs permission does too.
deploy.victorialogs.grafanaAuthPackage	The swarm-metrics-auth build that runs as swarm-logs-grafana-auth. Set on its own: deploy.victoriametrics.authPackage doesn't reach it.
swarm.victorialogs.domain	Only to rename it.
deploy.victorialogs.port	Only if something else on the services host already claims 9428.

It binds loopback and takes no credential of its own: the gateway vhost is the only way in, and the collector is the only intended writer.

Who may query. /select/logsql/ takes one of three credentials. An agent presents its swarm-agent.<agent>.<secret> bearer, checked by swarm-logs-auth. The agent must also hold the logs permission, which is off by default; without it the answer is a 403 (docs/5-project/process/conventions.md, Permissions). Grafana's datasource presents the same bearer as its metrics datasource, which swarm-bao-grafana-metrics-bearer creates, and a second swarm-metrics-auth instance (swarm-logs-grafana-auth) on the log store's host checks it against the same store path; the gateway tells it from the others by its shape, 64 hex digits. Anything else needs an operator's authelia session. Because one bearer opens both read APIs, a rotation (see credentials.md) restarts Grafana once, and the log store accepts the new value within a minute.

The collector also scrapes this store's own metrics, at https://<swarm.victorialogs.domain>/metrics, so a collector on a host without the store still gets them. On this name the gateway admits only the collector's client to /metrics and the ingest route /insert/opentelemetry/v1/logs; any other session gets deny on both. The read side stays open to any authenticated caller.

Telemetry collector (OTEL)

The swarm's collector receives from every hive's own collector and is the only process that decides where telemetry goes: it always writes the stores above, and exports to deploy.swarm-otel.upstream.endpoint as well when that's set. Which signals go there is deploy.swarm-otel.upstream.signals (default [ "metrics" "logs" ]); the stores always get both. It holds the upstream credential, which is why no hive and no agent needs one.

It runs in a swarm-otel container on the hive bridge, in a network namespace of its own. It binds no OTLP port: every receiver (one per hive, the swarm tier's and one per service forwarder) listens on a unix socket in /run/swarm-otel, a directory only the gateway's nginx and the container can open, and the gateway forwards to them. The gateway's routes and the collector's receivers come from one list, so they name the same sockets. The swarm-otel-sockets unit connects to each socket as the gateway's user after the container starts, then requests every route through the gateway vhost. It fails if a socket refuses for 30 seconds, or if a route answers 000, 502 or 504, or lacks its own X-Swarm-Otel-Route response header, for 30 seconds. The gateway sets that header, named after the route, on every response from a route, so a collector that's up but not listening, or an nginx that doesn't have the route, shows as a failed unit. Any status other than those passes, the collector's 4xx included. A switch leaves a window of about 20 seconds in which nginx and the container are out of step: the container stops before nginx restarts and starts about 18 seconds after it. Senders retry through that window.

The collector fetches authelia's discovery document through the gateway when it starts and exits if that fails. systemd restarts it with no start limit and a delay that grows from 2 to 60 seconds, so it comes back by itself once the gateway and authelia answer. If ingest stops, systemctl status swarm-otel-sockets on the host shows whether the receiver sockets are there; machinectl shell swarm-otel /run/current-system/sw/bin/journalctl -u opentelemetry-collector shows why the collector didn't start. A collector that keeps retrying never reaches failed, so systemctl --failed doesn't list it. The signals are swarm-otel-sockets failing and the Grafana alert no log records from <hive>, which fires after about 15 minutes without a log record from a hive.

Every hive's own collector reaches this one by its gateway name, swarm.otel.domain (default otel.<swarm domain>) — the same by-domain-through-the-gateway shape every other swarm service uses, not a loopback URL an operator has to redirect. Nothing needs setting on a hive that doesn't run the swarm's services; the name resolves through the gateway either way.

Option	When you'd touch it
swarm.otel.domain	Only to rename it — the default already resolves correctly for every hive in the swarm.

The swarm-bao, swarm-nats, swarm-builder, swarm-claude-login, swarm-notify-router, swarm-review-classifier, swarm-ci-runner, swarm-grafana, swarm-victorialogs, swarm-victoriametrics, swarm-authelia, swarm-attic, hive-forge and hive-matrix containers each run a collector of their own that forwards the container's journal here, on a route and receiver per service (https://<swarm.otel.domain>/<container>/), authenticated as an OIDC client of that service's. The queue, builder, login minter, notification router and classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store, authelia, cache, forge and homeserver have a forwarder client of their own (swarm.<service>.otel.clientId). Their records carry service.name (bao, nats, builder, claude-login, notify-router, review-classifier, ci-runner, grafana, victorialogs, victoriametrics, authelia, attic, forgejo, tuwunel) and no hive label. The forwarder of all but the store's is nix/container-modules/swarm-journal-forwarder.nix.

The homeserver's metrics. tuwunel 1.9.4 exports no metrics of its own, so request rates, sync load and federation traffic aren't available. Two series stand in for them, and neither adds a service:

• The hive collector on the homeserver's host requests https://<gatewayHost>/_matrix/client/versions every 60 s with its httpcheck receiver (services.hyperhive.otel.httpChecks). It reports httpcheck.status (one series per status class), httpcheck.duration and httpcheck.error, labelled with the URL. The container's forwarder can't make this request, because httpcheck dials HTTP and HTTPS only and tuwunel listens on a unix socket. The request passes through the gateway, so it covers the gateway's route as well as the homeserver.
• The hive-matrix-database-size timer in the container runs every five minutes. It measures tuwunel's database directory and pushes the gauge hyperhive.matrix.database_bytes through the forwarder's OTLP port, stamped service.name hive-matrix. It exists only where the container has a forwarder, which needs matrix-ctl's store identity set. A size under 4096 bytes is no database, so the timer logs a warning and pushes nothing.

The swarm collector's own container, swarm-otel, has no forwarder: its collector reads the container's journal itself (receiver journald/self) and exports it to the log store like the host's journal, stamped service.name swarm-otel and with no hive label. Unlike the host's journal, it goes to the log store only, never the upstream: the collector writes an upstream export failure to this journal, and shipping it upstream would fail again. A filter processor also drops the collector's own failure lines that name the store exporter (otlphttp/victorialogs), which can't reach the store anyway; failures of every other exporter still arrive. The container mounts the host's journal read-only at /run/host-journal, not over its own /var/log/journal.

Both store exporters are unconditional, and deploy.victoriametrics.enable doesn't gate them: that option says this host runs the store, while the swarm has one either way, reached by its swarm name through the gateway.

Agent-side configuration, and what a hive's own collector does, are in scheduler/observability.md.