mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: the swarm and services pages become overviews

Browse source 
The controller's status, queue, AGENTS.md and forge-webhook details move to
4-implementation/swarm/controller.md. The metrics, logs and OTEL detail moves to
4-implementation/swarm/telemetry.md. Each leaves a short operator blurb with the
options to touch, and links into the moved text are updated.
This commit is contained in:
iris 2026-11-10 20:59:45 +01:00 • committed by mara
commit f7cdd194f7
1 changed files with 793 additions and 707 deletions
Download patch file Download diff file Expand all files Collapse all files

296
docs/4-implementation/swarm/telemetry.md Normal file
View file

@ -0,0 +1,296 @@
# Swarm telemetry: metrics, logs and the collector
How the swarm's metrics store, Grafana, log store and OTEL collector connect,
and who may reach each. The operator's overview, with the options to touch and
how to open each service, is on
[the services page](../../2-handbook/swarm/services.md#metrics-victoriametrics--grafana).
## Metrics (VictoriaMetrics + Grafana)
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
reads it, in two containers at `metrics.<swarm-domain>` and
`grafana.<swarm-domain>`. Two containers rather than one so you can
restart or break Grafana without taking the time-series database with it.
The Grafana container sits on the hive bridge in a network namespace of its
own and dials authelia, the secret store, the collector and both datasources
by name through it. It binds no TCP port: it listens on the unix socket
`/run/swarm-grafana/grafana.sock`, a directory only the gateway's nginx and
the container can open.
They derive together: a store with no UI is unreadable and a UI with no
store is empty. To run one without the other, set it directly:
```nix
services.hyperhive.deploy = {
victoriametrics.enable = true;
grafana.enable = false;
};
```
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
<!-- vale write-good.Passive = NO -->
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`credentials.md`](../../2-handbook/swarm/credentials.md).
<!-- vale write-good.Passive = YES -->
**Dashboards.** Grafana provisions these from
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
- `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, the busiest agents and destinations, and data transferred by domain over time (the top 10 domains plus an other line, summed from the proxy's access log in VictoriaLogs; the proxy logs a CONNECT tunnel when it closes, so a long-lived connection shows as one spike at close). **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent.
- `builds`: the container builds hive-c0re ran on every hive, in two tabs. **builds** has the count, the failures and the longest, builds over time by outcome, and a table of each build with its id, agent, kind, outcome and runtime. **output** shows every line of the build picked in the **Build** filter, with its start and finish lines. It reads the lines hive-c0re logs for each build, so a hive-c0re that doesn't log them leaves it empty. Filter by agent.
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
- `review-classifier`: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
- `forge`: the forge, in three tabs. **content** has what the forge holds (repositories, users, issues, releases, discussion volume and the issues by label). **runtime** has the forge process's CPU, memory, goroutines and descriptors. **CI runs** has Actions run time and queue wait per repo and workflow, with a per-PR table; the Repo and Workflow filters apply to that tab.
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
- `host`: the machine under each hive, from the collector's host metrics: CPU by state, load, memory, swap and paging, disk and network throughput, filesystem usage, the Nix store's share and the busiest agent containers. Filter by hive and host.
- `collector`: what the hive and swarm OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
- `logstore`, `metricstore`: health of the log store and the metrics store.
- `grafana`: Grafana itself. Dashboards and data sources, alert instances firing, requests by status and their latency, alert rule evaluations and failures, sign-ins, database connections, memory and goroutines.
- `builder`: the swarm builder. Builds finished, by kind and outcome, their mean and 95th percentile time, the queue depth and the wait before a build starts.
- `ci-runner`: the forge's Actions runner. Its uptime and job slots, jobs finished by status, how many slots are in use, job time, and how often it polls the forge for work.
- `matrix`: the Matrix homeserver. Whether its client API answers, how fast, the share of the range it answered, and its database size.
- `notify-router`: the notification router. Forge events received by type and outcome, todos delivered, review requests that failed, time per event, and the backlog of the agents' queue.
A new panel can read "No data" until its source exists: the push and commit
counters on `controller` appear after the first push webhook, the refused
and failure series on `collector` appear once something fails, and the
`review-classifier` panels fill as review requests arrive.
**The `collector` dashboard.** It shows the self-metrics of the hive's
collector and of the swarm's collector, which both carry the collector
binary's default `service.name`, `otelcol-contrib`. The timeseries panels
split by `server.port`: 8888 is the hive collector, 8889 the swarm
collector. Every per-container collector (the secret store, the other service
containers and the agents' forwarders) also scrapes its own endpoint on
loopback and pushes the result up its normal hop, so each has a line of its
own, on its own port: agents all report on 8904, service containers on
8890 to 8903. The series carry `container` (services) or `agent` (agents)
to tell them apart. `scripts/check-dashboard-service-names.sh` runs in CI
and in `scripts/pre-push`. It fails when a dashboard query selects a
`service.name` that nothing emits.
**Where the data comes from.** The swarm's OTEL collector, below.
The `controller` panels come from the controller itself. It exports OTLP
through the gateway to the swarm's collector from whatever host it runs on,
whether or not that host runs the collector.
Neither container is reachable except through the gateway: each listens
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
runs in a network namespace of its own with no interface but loopback.
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it:
- **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource, an agent, or an operator. Grafana sends a
bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
store, and `swarm-metrics-auth` on the store's host checks it against the
same store path. An agent's `swarm-metrics` sends its
`swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. It also needs the `metrics`
permission, or the answer is a 403. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later. When the
store doesn't answer within five seconds, it checks requests against the
last value it read instead of denying, and still refuses a wrong bearer;
it tries the store again after three seconds. A revoked or rotated bearer
stays accepted while the store is down, for an hour after the last read at
most; after that the responder denies again. The store's next answer
replaces the value, and a process that holds none (it started while the
store was down) denies. `swarm-logs-grafana-auth`
is the same binary and does the same.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to
that path; any other session gets `deny`.
The collector scrapes the store's own metrics the same way, at
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
that doesn't run the store still gets them. On this name the gateway admits
only the collector's client to `/metrics` and the push route; any other
session gets `deny`.
## Logs (VictoriaLogs)
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
behind the same SSO as everything else: the swarm's own service containers,
the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
| Option | When you'd touch it |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. A revoked `logs` permission does too. |
| `deploy.victorialogs.grafanaAuthPackage` | The `swarm-metrics-auth` build that runs as `swarm-logs-grafana-auth`. Set on its own: `deploy.victoriametrics.authPackage` doesn't reach it. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `deploy.victorialogs.port` | Only if something else on the services host already claims `9428`. |
It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. The agent must also hold the `logs` permission,
which is off by default; without it the answer is a 403 (`docs/5-project/process/conventions.md`, Permissions). Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`credentials.md`](../../2-handbook/swarm/credentials.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. On this name the gateway admits only the
collector's client to `/metrics` and the ingest route
`/insert/opentelemetry/v1/logs`; any other session gets `deny` on both. The
read side stays open to any authenticated caller.
## Telemetry collector (OTEL)
The swarm's collector receives from every hive's own collector and is the
only process that decides where telemetry goes: it always writes the stores
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
always get both. It holds
the upstream credential, which is why no hive and no agent needs one.
It runs in a `swarm-otel` container on the hive bridge, in a network
namespace of its own. It binds no OTLP port: every receiver (one per hive,
the swarm tier's and one per service forwarder) listens on a unix socket in
`/run/swarm-otel`, a directory only the gateway's nginx and the container can
open, and the gateway forwards to them. The gateway's routes and the collector's
receivers come from one list, so they name the same sockets. The
`swarm-otel-sockets` unit connects to each socket as the gateway's user after
the container starts, then requests every route through the gateway vhost. It
fails if a socket refuses for 30 seconds, or if a route answers `000`, `502` or
`504`, or lacks its own `X-Swarm-Otel-Route` response header, for 30 seconds.
The gateway sets that header, named after the route, on every response from a
route, so a collector that's up but not listening, or an nginx that doesn't
have the route, shows as a failed unit. Any status other than those passes,
the collector's 4xx included. A switch leaves a window of about 20 seconds in which
nginx and the container are out of step: the container stops before nginx
restarts and starts about 18 seconds after it. Senders retry through that window.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a
loopback URL an operator has to redirect. Nothing needs setting on a hive
that doesn't run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `swarm-attic`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter, notification router and
classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store,
authelia, cache, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `attic`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**The homeserver's metrics.** tuwunel 1.9.4 exports no metrics of its own, so
request rates, sync load and federation traffic aren't available. Two series
stand in for them, and neither adds a service:
- The hive collector on the homeserver's host requests
`https://<gatewayHost>/_matrix/client/versions` every 60 s with its
`httpcheck` receiver (`services.hyperhive.otel.httpChecks`). It reports
`httpcheck.status` (one series per status class), `httpcheck.duration` and
`httpcheck.error`, labelled with the URL. The container's forwarder can't
make this request, because `httpcheck` dials HTTP and HTTPS only and tuwunel
listens on a unix socket. The request passes through the gateway, so it
covers the gateway's route as well as the homeserver.
- The `hive-matrix-database-size` timer in the container runs every five
minutes. It measures tuwunel's database directory and pushes the gauge
`hyperhive.matrix.database_bytes` through the forwarder's OTLP port,
stamped `service.name` `hive-matrix`. It exists only where the container
has a forwarder, which needs matrix-ctl's store identity set. A size under
4096 bytes is no database, so the timer logs a warning and pushes nothing.
The swarm collector's own container, `swarm-otel`, has no forwarder: its
collector reads the container's journal itself (receiver `journald/self`) and
exports it to the log store like the host's journal, stamped `service.name`
`swarm-otel` and with no `hive` label. Unlike the host's journal, it goes to the
log store only, never the upstream: the collector writes an upstream export failure to this
journal, and shipping it upstream would fail again. A `filter` processor also
drops the collector's own failure lines that name the store exporter
(`otlphttp/victorialogs`), which can't reach the store anyway; failures of
every other exporter still arrive. The container mounts the host's journal
read-only at `/run/host-journal`, not over its own
`/var/log/journal`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in
[`scheduler/observability.md`](../scheduler/observability.md).