mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: the swarm and services pages become overviews

Browse source 
The controller's status, queue, AGENTS.md and forge-webhook details move to
4-implementation/swarm/controller.md. The metrics, logs and OTEL detail moves to
4-implementation/swarm/telemetry.md. Each leaves a short operator blurb with the
options to touch, and links into the moved text are updated.
This commit is contained in:
iris 2026-11-10 20:59:45 +01:00 • committed by mara
commit f7cdd194f7
8 changed files with 793 additions and 707 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/2-handbook/agents/persistence.md
View file

@ -287,7 +287,7 @@ Under `/var/lib/hyperhive/agents/<name>/`:
to the swarm, which reads that copy, and applies the git diffs the
swarm sends to edit it. A new agent's first start writes the text given at
create, only when no file exists. This file stays the only copy that counts:
[`docs/2-handbook/swarm/README.md`](../swarm/README.md#an-agents-agentsmd).
[`docs/4-implementation/swarm/controller.md`](../../4-implementation/swarm/controller.md#an-agents-agentsmd).
- `harness/` — harness-internal ephemeral state; not intended for
agent consumption. Bind-mounted to `/agents/<name>/harness`
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:

442
docs/2-handbook/swarm/README.md
View file

@ -83,432 +83,22 @@ What it serves, why it's a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../../swarm-controller/README.md).
### Per-hive status (`GET /api/hives/status`)
One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.
A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.
| freshness | what to do about it |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive removed from the roster but still running |
The snapshot is the hive's `/health/ready` document plus `capabilities`, a list
of names for what the hive's release can do. Two exist: `tolerant-permissions`, and `no-config-checkout`, which
[turning an archetype agent into a custom one](archetypes.md#turn-into-custom)
needs. A hive on an older release lists none.
The snapshot also carries `egress_deny_applied`, the revision of the egress deny
list the hive last rendered into its proxy (`0` when the swarm has none). A hive
on an older release, or one that hasn't read the list yet, leaves it out; see
[the egress proxy](../../4-implementation/networking/network.md#egress-proxy-squid).
Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload. A roster hive's row
carries `dashboard`, the host its dashboard answers on; the swarm UI's
hive list links there.
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.
### Making a hive report
Two options on the **hive**, plus its store identity. They sit in two
namespaces, because the queue's URL is a fact about _this machine_ and the
token endpoint is the swarm's single address:
| option | what to set it to |
| ---------------------------------------------- | --------------------------------------------- |
| `deploy.hive-controller.statusPublish.natsUrl` | `tls://<swarm.nats.domain>:<swarm.nats.port>` |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). Authelia's mint, in its
container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config`
warning and tries again, waiting 5 seconds after the first failure and doubling
up to 5 minutes. It starts publishing as soon as a read succeeds, with no
restart.
The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.
The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue
host without the store gets no default, even when it holds the hive's own
reader `deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.natsCommonName`, place it, and name it in both options. Until you
do, `swarm-bao-nats-tls` fails naming both whenever the queue's certificate is
missing or due for renewal.
The queue host's gateway also serves the queue's metrics, at
`https://<swarm.nats.domain>/metrics`, so a swarm collector on any host
scrapes them. Only the collector's authelia client gets through; any other
session gets `deny`, and every other path on that name is 403. With the
gateway's default self-signed TLS, that vhost serves the queue's own
certificate, so a host whose only swarm service is the queue needs no
swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.
The endpoint answers **503** when this host has no swarm queue
configured, or has one and can't read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
can't see. The body says which. Status survives a controller restart:
it's stored in the queue, not in the daemon.
### Giving the agents the queue too
An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:
| option | what to set it to |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
It reaches each container at start, as `HIVE_AGENT_NATS_URL` in
`/etc/hyperhive-hive-facts`, so no agent build carries it. On every hive,
`agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. The option takes one
`<scheme>://<host>:<port>` (a hostname or IPv4 address), not an IPv6 literal or a
list: evaluation refuses anything else. On the queue's
host the name resolves inside a container to the bridge address, where the
firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.
**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:
- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits. The error reads
`no swarm queue connection; exiting, since this agent cannot run without one`,
and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.
Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading
the credential from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for a `swarm-logs` built before it read the agent's
own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.
<details><summary>What an agent publishes over the queue</summary>
What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. The queue keeps the rows in
the stream `term-main-<agent>` for 24 hours or 64 MiB, the same limits and the
same creator as the subagent stream below, so a swarm terminal that attaches
later replays the newest of them. The two streams hold disjoint subjects.
An agent's **subagents** publish their terminals too, from the agent's
subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`,
classified the same way. The queue grants that family to the agent's own queue
credential alone, so the daemon reads it from the store under the agent's store
identity, exactly as the harness does, and publishes nothing without it. The
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
whichever comes first, dropping the oldest rows once either limit kicks in so
a publish never fails because of it, and the swarm lists an agent's subagents
from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for
every agent a hive's wanted state names, within a minute of the agent
appearing there. The agent's grant is publish on its own
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
subjects and limits are the controller's and never the agent's. Subagents
publish output only and read nothing.
The queue would refuse a row too large for its `max_payload` outright and
take the connection down with it, so the harness drops such a row's body before
sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.
The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model and effort (`model`, `effort`, where each comes from as
`model_source` and `effort_source`, and the resolved id the last turn
actually ran on), the
context budget and the last turn's context and cost token blocks,
`agent_state`, and `needs_login`: `true` while the agent's claude has no
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.
A header also carries `paused_by_hive` while the harness's own pause is in force:
the reason it wrote into the pause marker when it paused its turn loop. A pause
by an operator (`hivectl agent <name> pause`, the dashboard) leaves the marker
empty and reports nothing here, and so does a harness that predates the field.
The controller reacts by declaring the agent `paused`, but only when the swarm
declares it `up` at that moment. Without that, the hive resumes any paused agent the
swarm declares `up` on its next convergence. It acts once per pause: the entry
stays until a header without the field arrives. Setting the agent `up` again
resumes it, and the hive clears the marker. If the swarm is on a controller that
predates the field, the hive's pause lasts until its next convergence, as before.
`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
spellings. ⚠️ From inside the container only two of those four are sayable: the
harness reports `up`, or `paused` when the pause marker is present. `offline` and
`destroyed` are hive-c0re's observations — a stopped agent publishes nothing and
a destroyed one doesn't exist — so a view that needs the full four-state picture
takes them from the `agent-status` bucket and uses this subject to sharpen the
rest.
Headers go out **on transition**: the harness rebuilds the header whenever its
event bus moves and publishes when the result differs from what it last sent.
That's the whole point of the subject — the `agent-status` bucket already
republishes once a minute, which is far too slow for "is this agent thinking
right now." A core subject keeps nothing for a late subscriber, so the harness
also re-sends the current header once a minute: a subscriber attaching
mid-idle sees it within that minute. The swarm's stream sends a new client the
latest header it holds first, when that's under 150 seconds old. A renderer opens with the bucket's
snapshot and lets this stream refine it.
Swarm-side, `GET /api/agents/<name>/state/stream` relays that subject as SSE,
resolving the agent's hive at request time exactly as the terminal stream does.
The payload passes through opaquely. Separately, the controller keeps each
agent's latest `model`, `effort` and their sources for
`GET /api/agents/<name>/inference` ([`inference.md`](../../4-implementation/swarm/inference.md)), and each
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it. The row's `paused_by_hive` comes from the same
headers and is `null` in the same cases.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key of that bucket. The harness writes
once per start, because a config change reaches an agent by restarting its
container. An agent with no icon deletes its key. Swarm-side,
`GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.
The fourth is its **`AGENTS.md`**, which goes into the `agents-md` bucket under
the key `<agent>`, writable by the agent's own credential alone
(`$KV.agents-md.<agent>`). The swarm reads that copy and edits the file through
the agent; [An agent's AGENTS.md](#an-agents-agentsmd) has the details.
The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>`, its operator-input subject `$SWARM.input.<agent>` and
its command subject `$SWARM.control.<agent>`, and nothing else. Besides the
subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies.
Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
sets as their inbox prefix. A KV watch or key listing delivers there as well. A
hive may also subscribe to `$SWARM.knowledge` and its own `$SWARM.deploy.<hive>`,
so it can't read another hive's deploys or any agent's terminal. The controller
may also subscribe to `$SWARM.term.<agent>`, `$SWARM.term.<agent>.sub.<subagent>`
and `$SWARM.agent-state.<agent>`, the streams it relays to swarm-ui. A hive or
controller that connects with the library's default `_INBOX.<nuid>` prefix gets
no replies, so every request it makes times out.
</details>
### An agent's AGENTS.md
The file in the agent's state dir, `state/AGENTS.md`, is the only copy that
counts, and only the agent writes it: itself, or its harness applying an edit
from the swarm. No hive is in either path.
**Reading.** The harness publishes the file's text and its revision, the git
blob id that `git hash-object` prints for it, at start, within ten seconds of
any change, and straight after an edit it applied. An agent with no `AGENTS.md`
publishes `""`. `GET /api/agents/<name>/agents-md` answers `{content, revision}`
from that copy, so it works while the agent isn't running. A 404 means the agent
hasn't published one: its harness hasn't run since it began to.
**Initial text.** `POST /api/agents` and `swarmctl agent create --agents-md <file>`
take an optional `agents_md`, the text the file starts with, up to 64 KiB; a
longer text is a 400. The controller stores it under the key `<agent>` of the
`agents-md-initial` bucket, which only it writes, before it queues the creation.
At start the harness reads its own key, with one direct get on
`$JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>`, and
writes the text to `AGENTS.md` only when no file exists there. It does this
before the first mirror publish and the first turn. An existing file is never
replaced, so a restart or a re-create keeps the agent's edits; a later change
to the file never reaches the key. Destroying the agent deletes the key. When
the harness can't read the key it starts without the text and logs a warning.
The text holds no secret, so nothing goes through the secret store.
**Editing.** `PUT /api/agents/<name>/agents-md` takes `{base_revision, diff}`.
The diff is a unified diff that names `a/AGENTS.md` and `b/AGENTS.md`, made
against the text at `base_revision`. swarm-controller sends it as a request on
the agent's command subject `$SWARM.control.<name>`, as it sends a cancel. The
harness refuses it when the file isn't at `base_revision`. Otherwise it runs
`git apply --check` and then `git apply` on a copy of the file, and writes the
result over the file only when both pass:
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| 200 | applied; the body is `{revision}`, the file's new revision |
| 409 | the file isn't at `base_revision` any more; nothing applied, and `revision` in the body is the one it's at |
| 422 | `git apply` refused the diff, or it touches a file other than `AGENTS.md`; `detail` carries git's message, and the file stays as it was |
| 503 | the agent isn't running or isn't connected to the queue |
| 504 | the harness didn't answer within ten seconds, so whether the edit applied is unknown |
| 500 | the harness couldn't read or write the file or run git, or the queue failed the request |
The harness refuses a diff that sets any file mode other than `100644`, so an
edit can't turn `AGENTS.md` into a symlink. `PUT` is operator-only, behind
`Remote-User` like cancel: whoever can call it can rewrite the agent's
instructions, so don't widen who holds it without revisiting this.
An edit needs the agent running, and a read of a stopped agent shows the text
it last published. A harness older than this route doesn't know the command and
drops it unanswered, which reads as a 504. `swarmctl agent agents-md get` and
`apply` call the two routes.
### Swarm-wide forge objects
The controller also keeps the forge objects that are one per swarm, not
one per hive. It ensures them at start and every five minutes after
(`swarm-controller/src/forge/objects.rs`):
- the orgs `agent-configs`, `internal` and `agents`, plus each mirror's
owner org;
- the empty `operators` merge-gate team in `agents` and `agent-configs`;
- the `main` merge gate on every `agent-configs` repo: merge and approval
whitelists = the `operators` team, and no user. The controller leaves a
repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
host (with the `actions/checkout` and `actions/upload-artifact` ones
`deploy.forgejo.actionsFromSelf` adds, and `hyperhive/hyperhive` from
`deploy.forgejo.hyperhiveUpstream`);
- `internal/docs` (private) and `internal/knowledge` (public, with a
seed `README.md` while empty);
- the `agent-configs` org avatar
(`deploy.swarm-controller.configOrgAvatarPng`).
A pass that can't finish logs a
`warn` line per object plus `swarm forge objects: pass incomplete` in
`journalctl -u swarm-controller`, and retries on the next tick. While the
controller is down the objects stay as they are.
The controller gives each agent **read** on `internal/docs`. A `GrantDocsRead`
node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the
store's `hive-agent-*` roster (`forge/docs_grant.rs`, at start and every five
minutes) adds any agent that isn't one, so an agent whose create-time grant failed
gets it within five minutes. The pass only adds: a roster it can't read, a
forge read that fails, or an `internal/docs` that doesn't exist (checked once per
pass, logged at warn) changes nothing. The grant doesn't depend on the hive
the agent runs on, or on a forge container there.
### Swarm-wide forge webhooks
At startup the controller registers its Forgejo hooks pointing at
itself, all under `https://<swarm.domain>/webhook/forge/`:
- a `push` hook on `internal/knowledge`
- a `pull_request` hook on the `agent-configs` org
- an instance-wide `push` system hook, for commit metrics
- an instance-wide `pull_request_review_request` system hook, which starts
the [review classifier](../../4-implementation/swarm/review-classifier.md#trigger-and-lifecycle)
- instance-wide `issues` + `issue_assign` and `pull_request` +
`pull_request_assign` + `pull_request_sync` + `pull_request_review_approved` +
`pull_request_review_rejected` + `pull_request_review_comment` system hooks, and
an instance-wide `workflow_run_failure` + `workflow_run_cancelled` system hook,
whose deliveries it keeps in the `SWARM_EVENTS` stream for the
[notification router](../../4-implementation/swarm/notify-router.md#the-path)
The controller **interprets** a delivery and sends hives a specific
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.
The `issues` and `pull_request` deliveries are the exception: no hive gets
them, and the notification router, not the controller, decides what they mean.
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
of its `merge_commit_sha` on the one hive whose wanted state places the
agent; with no such hive, or more than one, the controller deploys nothing. See
[config-changes.md § Config changes](../agents/config-changes.md#config-changes).
A merge whose `closed` delivery never arrives deploys nothing either; the
operator recovers by redelivering that delivery from the forge hook page,
which re-enters the same handler and re-queues the deploy.
**`internal/knowledge` is on that path.** The controller's is the only
hook on it ([`knowledge.md`](knowledge.md) covers clearing a
leftover). A webhook has exactly one target URL, so a second registration
would take delivery away from the first rather than add a recipient.
<!-- vale write-good.Passive = NO -->
**The `agent-configs` org is on it too.** The controller's hook is the
only one that acts on config PRs. No hive serves `/webhook/config-pr`
any more and none deletes it: a hook of that name left on the org by an
older release stays registered and collects failed deliveries until an
operator deletes it by hand. Removing
the controller's hook stops forge-UI merges from deploying until its
next start recreates it.
<!-- vale write-good.Passive = YES -->
Nothing to configure. The controller registers the hooks only when this
host also serves the swarm UI vhost — that's what publishes the
endpoint, and a hook the forge can't reach would collect failed
deliveries while looking healthy. The controller generates the HMAC
secret on first start and keeps it
(see [`docs/2-handbook/agents/persistence.md`](../agents/persistence.md)).
To check it's working, push to `internal/knowledge` and look for
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
refused delivery logs `webhook: refused delivery` with the reason.
The controller registers the hooks within five minutes of having both the forge
token and the secret. It keeps trying, at most five minutes apart, for as long
as the forge or the secret store is down. A restart isn't needed to recover.
### What the controller keeps
- **Hive status.** Each hive publishes its state upward through the swarm queue.
The swarm UI's hive list shows when each last reported: `fresh`, `stale`,
`never_reported` or `unknown`.
`services.hyperhive.swarm.controller.staleAfterSeconds` (default `120`) sets
how long a hive may stay silent before it reads `stale`.
- **Agent queue access.** Each agent connects to the queue as itself, with a
credential the controller mints when it creates the agent.
- **An agent's `AGENTS.md`.** The swarm holds the revisions and the harness
applies them.
- **Forge objects and webhooks.** Swarm-wide labels, repos and webhooks the
controller registers on the forge.
The options, wire formats and failure modes of each:
[`controller.md`](../../4-implementation/swarm/controller.md).
## Hives: the substrate

2
docs/2-handbook/swarm/schedules.md
View file

@ -80,7 +80,7 @@ or within a minute if the controller's first attempt to pass it on failed.
its own swarm-queue credential. A harness that can't connect retries for
about five minutes, then exits and comes back; it reads no prompts until a
connect succeeds. See
[Giving the agents the queue too](README.md#giving-the-agents-the-queue-too).
[Giving the agents the queue too](../../4-implementation/swarm/controller.md#giving-the-agents-the-queue-too).
- **The agent hasn't read its prompts since it started.** It fires from the
last list it read; until the first read succeeds it fires nothing. A read
that fails later keeps the previous list.

305
docs/2-handbook/swarm/services.md
View file

@ -188,293 +188,54 @@ generates and reads each of its keys.
### Metrics (VictoriaMetrics + Grafana)
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
reads it, in two containers at `metrics.<swarm-domain>` and
`grafana.<swarm-domain>`. Two containers rather than one so you can
restart or break Grafana without taking the time-series database with it.
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it,
at `metrics.<swarm-domain>` and `grafana.<swarm-domain>`. They sit in two
containers so you can restart or break Grafana without losing the time-series
database. Enabling either enables both; to run one alone, set it directly with
`deploy.victoriametrics.enable` or `deploy.grafana.enable`.
The Grafana container sits on the hive bridge in a network namespace of its
own and dials authelia, the secret store, the collector and both datasources
by name through it. It binds no TCP port: it listens on the unix socket
`/run/swarm-grafana/grafana.sock`, a directory only the gateway's nginx and
the container can open.
Log in to Grafana with your swarm SSO account. Grafana provisions its
dashboards from the repo, so edits made in the UI aren't kept. Each service has
its own board; open the folder to see them.
They derive together: a store with no UI is unreadable and a UI with no
store is empty. To run one without the other, set it directly:
⚠️ **The metrics store grows for as long as the swarm runs.** Options to touch:
```nix
services.hyperhive.deploy = {
victoriametrics.enable = true;
grafana.enable = false;
};
```
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you know how fast the swarm fills a disk. A too-short value discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower it to `Viewer` or `Editor` if some operators shouldn't reconfigure Grafana. |
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
<!-- vale write-good.Passive = NO -->
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`credentials.md`](credentials.md).
<!-- vale write-good.Passive = YES -->
**Dashboards.** Grafana provisions these from
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
- `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, the busiest agents and destinations, and data transferred by domain over time (the top 10 domains plus an other line, summed from the proxy's access log in VictoriaLogs; the proxy logs a CONNECT tunnel when it closes, so a long-lived connection shows as one spike at close). **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent.
- `builds`: the container builds hive-c0re ran on every hive, in two tabs. **builds** has the count, the failures and the longest, builds over time by outcome, and a table of each build with its id, agent, kind, outcome and runtime. **output** shows every line of the build picked in the **Build** filter, with its start and finish lines. It reads the lines hive-c0re logs for each build, so a hive-c0re that doesn't log them leaves it empty. Filter by agent.
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
- `review-classifier`: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
- `forge`: the forge, in three tabs. **content** has what the forge holds (repositories, users, issues, releases, discussion volume and the issues by label). **runtime** has the forge process's CPU, memory, goroutines and descriptors. **CI runs** has Actions run time and queue wait per repo and workflow, with a per-PR table; the Repo and Workflow filters apply to that tab.
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
- `host`: the machine under each hive, from the collector's host metrics: CPU by state, load, memory, swap and paging, disk and network throughput, filesystem usage, the Nix store's share and the busiest agent containers. Filter by hive and host.
- `collector`: what the hive and swarm OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
- `logstore`, `metricstore`: health of the log store and the metrics store.
- `grafana`: Grafana itself. Dashboards and data sources, alert instances firing, requests by status and their latency, alert rule evaluations and failures, sign-ins, database connections, memory and goroutines.
- `builder`: the swarm builder. Builds finished, by kind and outcome, their mean and 95th percentile time, the queue depth and the wait before a build starts.
- `ci-runner`: the forge's Actions runner. Its uptime and job slots, jobs finished by status, how many slots are in use, job time, and how often it polls the forge for work.
- `matrix`: the Matrix homeserver. Whether its client API answers, how fast, the share of the range it answered, and its database size.
- `notify-router`: the notification router. Forge events received by type and outcome, todos delivered, review requests that failed, time per event, and the backlog of the agents' queue.
A new panel can read "No data" until its source exists: the push and commit
counters on `controller` appear after the first push webhook, the refused
and failure series on `collector` appear once something fails, and the
`review-classifier` panels fill as review requests arrive.
**The `collector` dashboard.** It shows the self-metrics of the hive's
collector and of the swarm's collector, which both carry the collector
binary's default `service.name`, `otelcol-contrib`. The timeseries panels
split by `server.port`: 8888 is the hive collector, 8889 the swarm
collector. Every per-container collector (the secret store, the other service
containers and the agents' forwarders) also scrapes its own endpoint on
loopback and pushes the result up its normal hop, so each has a line of its
own, on its own port: agents all report on 8904, service containers on
8890 to 8903. The series carry `container` (services) or `agent` (agents)
to tell them apart. `scripts/check-dashboard-service-names.sh` runs in CI
and in `scripts/pre-push`. It fails when a dashboard query selects a
`service.name` that nothing emits.
**Where the data comes from.** The swarm's OTEL collector, below.
The `controller` panels come from the controller itself. It exports OTLP
through the gateway to the swarm's collector from whatever host it runs on,
whether or not that host runs the collector.
Neither container is reachable except through the gateway: each listens
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
runs in a network namespace of its own with no interface but loopback.
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it:
- **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource, an agent, or an operator. Grafana sends a
bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
store, and `swarm-metrics-auth` on the store's host checks it against the
same store path. An agent's `swarm-metrics` sends its
`swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. It also needs the `metrics`
permission, or the answer is a 403. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later. When the
store doesn't answer within five seconds, it checks requests against the
last value it read instead of denying, and still refuses a wrong bearer;
it tries the store again after three seconds. A revoked or rotated bearer
stays accepted while the store is down, for an hour after the last read at
most; after that the responder denies again. The store's next answer
replaces the value, and a process that holds none (it started while the
store was down) denies. `swarm-logs-grafana-auth`
is the same binary and does the same.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to
that path; any other session gets `deny`.
The collector scrapes the store's own metrics the same way, at
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
that doesn't run the store still gets them. On this name the gateway admits
only the collector's client to `/metrics` and the push route; any other
session gets `deny`.
The dashboard list, the "No data" cases, and how the gateway guards the store:
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#metrics-victoriametrics--grafana).
### Logs (VictoriaLogs)
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
behind the same SSO as everything else: the swarm's own service containers,
the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
behind the same SSO as everything else: the swarm's service containers, the
hive's daemons and the harness units in every agent container.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
To read them, open Grafana, pick **Explore** and choose the `VictoriaLogs`
datasource. Agents query it with `swarm-logs`, and need the `logs` permission.
| Option | When you'd touch it |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. A revoked `logs` permission does too. |
| `deploy.victorialogs.grafanaAuthPackage` | The `swarm-metrics-auth` build that runs as `swarm-logs-grafana-auth`. Set on its own: `deploy.victoriametrics.authPackage` doesn't reach it. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `deploy.victorialogs.port` | Only if something else on the services host already claims `9428`. |
| Option | When you'd touch it |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's. Raise it to answer questions about last quarter. |
It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. The agent must also hold the `logs` permission,
which is off by default; without it the answer is a 403 (`docs/5-project/process/conventions.md`, Permissions). Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`credentials.md`](credentials.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. On this name the gateway admits only the
collector's client to `/metrics` and the ingest route
`/insert/opentelemetry/v1/logs`; any other session gets `deny` on both. The
read side stays open to any authenticated caller.
The other options, who may query, and the gateway rules:
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#logs-victorialogs).
### Telemetry collector (OTEL)
The swarm's collector receives from every hive's own collector and is the
only process that decides where telemetry goes: it always writes the stores
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
always get both. It holds
the upstream credential, which is why no hive and no agent needs one.
The swarm's collector receives from every hive's own collector and decides where
telemetry goes. It always writes the metrics and log stores, and exports to
`deploy.swarm-otel.upstream.endpoint` as well when you set one. It holds the
upstream credential, so no hive and no agent needs one. Every hive reaches it
by its gateway name, `otel.<swarm-domain>`, so a hive that doesn't run the
swarm's services needs nothing set.
It runs in a `swarm-otel` container on the hive bridge, in a network
namespace of its own. It binds no OTLP port: every receiver (one per hive,
the swarm tier's and one per service forwarder) listens on a unix socket in
`/run/swarm-otel`, a directory only the gateway's nginx and the container can
open, and the gateway forwards to them. The gateway's routes and the collector's
receivers come from one list, so they name the same sockets. The
`swarm-otel-sockets` unit connects to each socket as the gateway's user after
the container starts, then requests every route through the gateway vhost. It
fails if a socket refuses for 30 seconds, or if a route answers `000`, `502` or
`504`, or lacks its own `X-Swarm-Otel-Route` response header, for 30 seconds.
The gateway sets that header, named after the route, on every response from a
route, so a collector that's up but not listening, or an nginx that doesn't
have the route, shows as a failed unit. Any status other than those passes,
the collector's 4xx included. A switch leaves a window of about 20 seconds in which
nginx and the container are out of step: the container stops before nginx
restarts and starts about 18 seconds after it. Senders retry through that window.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a
loopback URL an operator has to redirect. Nothing needs setting on a hive
that doesn't run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `swarm-attic`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter, notification router and
classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store,
authelia, cache, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `attic`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**The homeserver's metrics.** tuwunel 1.9.4 exports no metrics of its own, so
request rates, sync load and federation traffic aren't available. Two series
stand in for them, and neither adds a service:
- The hive collector on the homeserver's host requests
`https://<gatewayHost>/_matrix/client/versions` every 60 s with its
`httpcheck` receiver (`services.hyperhive.otel.httpChecks`). It reports
`httpcheck.status` (one series per status class), `httpcheck.duration` and
`httpcheck.error`, labelled with the URL. The container's forwarder can't
make this request, because `httpcheck` dials HTTP and HTTPS only and tuwunel
listens on a unix socket. The request passes through the gateway, so it
covers the gateway's route as well as the homeserver.
- The `hive-matrix-database-size` timer in the container runs every five
minutes. It measures tuwunel's database directory and pushes the gauge
`hyperhive.matrix.database_bytes` through the forwarder's OTLP port,
stamped `service.name` `hive-matrix`. It exists only where the container
has a forwarder, which needs matrix-ctl's store identity set. A size under
4096 bytes is no database, so the timer logs a warning and pushes nothing.
The swarm collector's own container, `swarm-otel`, has no forwarder: its
collector reads the container's journal itself (receiver `journald/self`) and
exports it to the log store like the host's journal, stamped `service.name`
`swarm-otel` and with no `hive` label. Unlike the host's journal, it goes to the
log store only, never the upstream: the collector writes an upstream export failure to this
journal, and shipping it upstream would fail again. A `filter` processor also
drops the collector's own failure lines that name the store exporter
(`otlphttp/victorialogs`), which can't reach the store anyway; failures of
every other exporter still arrive. The container mounts the host's journal
read-only at `/run/host-journal`, not over its own
`/var/log/journal`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in
[`4-implementation/scheduler/observability.md`](../../4-implementation/scheduler/observability.md).
How the collector connects, its per-service routes and the homeserver's
stand-in metrics:
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#telemetry-collector-otel).
### CI runner

2
docs/2-handbook/swarm/ui.md
View file

@ -877,7 +877,7 @@ The controller refuses a text over 64 KiB. Its reason shows under the
box, and the swarm creates no agent. After a successful
create the box empties. The
[instructions editor](#editing-an-agents-instructions) changes the file later.
The [swarm README](README.md#an-agents-agentsmd) has the wire details.
The [swarm controller page](../../4-implementation/swarm/controller.md#an-agents-agentsmd) has the wire details.
### Linking external accounts

433
docs/4-implementation/swarm/controller.md Normal file
View file

@ -0,0 +1,433 @@
# Swarm controller: status, queue and forge objects
How the swarm controller takes in hive status, what a hive and an agent need to
reach the queue, how it keeps an agent's `AGENTS.md`, and the forge objects and
webhooks it owns. The operator's overview is in
[the swarm page](../../2-handbook/swarm/README.md#swarm-controller).
## Per-hive status (`GET /api/hives/status`)
One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.
A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.
| freshness | what to do about it |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive removed from the roster but still running |
The snapshot is the hive's `/health/ready` document plus `capabilities`, a list
of names for what the hive's release can do. Two exist: `tolerant-permissions`, and `no-config-checkout`, which
[turning an archetype agent into a custom one](../../2-handbook/swarm/archetypes.md#turn-into-custom)
needs. A hive on an older release lists none.
The snapshot also carries `egress_deny_applied`, the revision of the egress deny
list the hive last rendered into its proxy (`0` when the swarm has none). A hive
on an older release, or one that hasn't read the list yet, leaves it out; see
[the egress proxy](../networking/network.md#egress-proxy-squid).
Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload. A roster hive's row
carries `dashboard`, the host its dashboard answers on; the swarm UI's
hive list links there.
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.
## Making a hive report
Two options on the **hive**, plus its store identity. They sit in two
namespaces, because the queue's URL is a fact about _this machine_ and the
token endpoint is the swarm's single address:
| option | what to set it to |
| ---------------------------------------------- | --------------------------------------------- |
| `deploy.hive-controller.statusPublish.natsUrl` | `tls://<swarm.nats.domain>:<swarm.nats.port>` |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). Authelia's mint, in its
container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config`
warning and tries again, waiting 5 seconds after the first failure and doubling
up to 5 minutes. It starts publishing as soon as a read succeeds, with no
restart.
The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.
The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue
host without the store gets no default, even when it holds the hive's own
reader `deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.natsCommonName`, place it, and name it in both options. Until you
do, `swarm-bao-nats-tls` fails naming both whenever the queue's certificate is
missing or due for renewal.
The queue host's gateway also serves the queue's metrics, at
`https://<swarm.nats.domain>/metrics`, so a swarm collector on any host
scrapes them. Only the collector's authelia client gets through; any other
session gets `deny`, and every other path on that name is 403. With the
gateway's default self-signed TLS, that vhost serves the queue's own
certificate, so a host whose only swarm service is the queue needs no
swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.
The endpoint answers **503** when this host has no swarm queue
configured, or has one and can't read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
can't see. The body says which. Status survives a controller restart:
it's stored in the queue, not in the daemon.
## Giving the agents the queue too
An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](../../2-handbook/swarm/credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:
| option | what to set it to |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
It reaches each container at start, as `HIVE_AGENT_NATS_URL` in
`/etc/hyperhive-hive-facts`, so no agent build carries it. On every hive,
`agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. The option takes one
`<scheme>://<host>:<port>` (a hostname or IPv4 address), not an IPv6 literal or a
list: evaluation refuses anything else. On the queue's
host the name resolves inside a container to the bridge address, where the
firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.
**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:
- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits. The error reads
`no swarm queue connection; exiting, since this agent cannot run without one`,
and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.
Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading
the credential from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for a `swarm-logs` built before it read the agent's
own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.
<details><summary>What an agent publishes over the queue</summary>
What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. The queue keeps the rows in
the stream `term-main-<agent>` for 24 hours or 64 MiB, the same limits and the
same creator as the subagent stream below, so a swarm terminal that attaches
later replays the newest of them. The two streams hold disjoint subjects.
An agent's **subagents** publish their terminals too, from the agent's
subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`,
classified the same way. The queue grants that family to the agent's own queue
credential alone, so the daemon reads it from the store under the agent's store
identity, exactly as the harness does, and publishes nothing without it. The
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
whichever comes first, dropping the oldest rows once either limit kicks in so
a publish never fails because of it, and the swarm lists an agent's subagents
from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for
every agent a hive's wanted state names, within a minute of the agent
appearing there. The agent's grant is publish on its own
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
subjects and limits are the controller's and never the agent's. Subagents
publish output only and read nothing.
The queue would refuse a row too large for its `max_payload` outright and
take the connection down with it, so the harness drops such a row's body before
sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.
The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model and effort (`model`, `effort`, where each comes from as
`model_source` and `effort_source`, and the resolved id the last turn
actually ran on), the
context budget and the last turn's context and cost token blocks,
`agent_state`, and `needs_login`: `true` while the agent's claude has no
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.
A header also carries `paused_by_hive` while the harness's own pause is in force:
the reason it wrote into the pause marker when it paused its turn loop. A pause
by an operator (`hivectl agent <name> pause`, the dashboard) leaves the marker
empty and reports nothing here, and so does a harness that predates the field.
The controller reacts by declaring the agent `paused`, but only when the swarm
declares it `up` at that moment. Without that, the hive resumes any paused agent the
swarm declares `up` on its next convergence. It acts once per pause: the entry
stays until a header without the field arrives. Setting the agent `up` again
resumes it, and the hive clears the marker. If the swarm is on a controller that
predates the field, the hive's pause lasts until its next convergence, as before.
`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
spellings. ⚠️ From inside the container only two of those four are sayable: the
harness reports `up`, or `paused` when the pause marker is present. `offline` and
`destroyed` are hive-c0re's observations — a stopped agent publishes nothing and
a destroyed one doesn't exist — so a view that needs the full four-state picture
takes them from the `agent-status` bucket and uses this subject to sharpen the
rest.
Headers go out **on transition**: the harness rebuilds the header whenever its
event bus moves and publishes when the result differs from what it last sent.
That's the whole point of the subject — the `agent-status` bucket already
republishes once a minute, which is far too slow for "is this agent thinking
right now." A core subject keeps nothing for a late subscriber, so the harness
also re-sends the current header once a minute: a subscriber attaching
mid-idle sees it within that minute. The swarm's stream sends a new client the
latest header it holds first, when that's under 150 seconds old. A renderer opens with the bucket's
snapshot and lets this stream refine it.
Swarm-side, `GET /api/agents/<name>/state/stream` relays that subject as SSE,
resolving the agent's hive at request time exactly as the terminal stream does.
The payload passes through opaquely. Separately, the controller keeps each
agent's latest `model`, `effort` and their sources for
`GET /api/agents/<name>/inference` ([`inference.md`](inference.md)), and each
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it. The row's `paused_by_hive` comes from the same
headers and is `null` in the same cases.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key of that bucket. The harness writes
once per start, because a config change reaches an agent by restarting its
container. An agent with no icon deletes its key. Swarm-side,
`GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.
The fourth is its **`AGENTS.md`**, which goes into the `agents-md` bucket under
the key `<agent>`, writable by the agent's own credential alone
(`$KV.agents-md.<agent>`). The swarm reads that copy and edits the file through
the agent; [An agent's AGENTS.md](#an-agents-agentsmd) has the details.
The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>`, its operator-input subject `$SWARM.input.<agent>` and
its command subject `$SWARM.control.<agent>`, and nothing else. Besides the
subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies.
Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
sets as their inbox prefix. A KV watch or key listing delivers there as well. A
hive may also subscribe to `$SWARM.knowledge` and its own `$SWARM.deploy.<hive>`,
so it can't read another hive's deploys or any agent's terminal. The controller
may also subscribe to `$SWARM.term.<agent>`, `$SWARM.term.<agent>.sub.<subagent>`
and `$SWARM.agent-state.<agent>`, the streams it relays to swarm-ui. A hive or
controller that connects with the library's default `_INBOX.<nuid>` prefix gets
no replies, so every request it makes times out.
</details>
## An agent's AGENTS.md
The file in the agent's state dir, `state/AGENTS.md`, is the only copy that
counts, and only the agent writes it: itself, or its harness applying an edit
from the swarm. No hive is in either path.
**Reading.** The harness publishes the file's text and its revision, the git
blob id that `git hash-object` prints for it, at start, within ten seconds of
any change, and straight after an edit it applied. An agent with no `AGENTS.md`
publishes `""`. `GET /api/agents/<name>/agents-md` answers `{content, revision}`
from that copy, so it works while the agent isn't running. A 404 means the agent
hasn't published one: its harness hasn't run since it began to.
**Initial text.** `POST /api/agents` and `swarmctl agent create --agents-md <file>`
take an optional `agents_md`, the text the file starts with, up to 64 KiB; a
longer text is a 400. The controller stores it under the key `<agent>` of the
`agents-md-initial` bucket, which only it writes, before it queues the creation.
At start the harness reads its own key, with one direct get on
`$JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>`, and
writes the text to `AGENTS.md` only when no file exists there. It does this
before the first mirror publish and the first turn. An existing file is never
replaced, so a restart or a re-create keeps the agent's edits; a later change
to the file never reaches the key. Destroying the agent deletes the key. When
the harness can't read the key it starts without the text and logs a warning.
The text holds no secret, so nothing goes through the secret store.
**Editing.** `PUT /api/agents/<name>/agents-md` takes `{base_revision, diff}`.
The diff is a unified diff that names `a/AGENTS.md` and `b/AGENTS.md`, made
against the text at `base_revision`. swarm-controller sends it as a request on
the agent's command subject `$SWARM.control.<name>`, as it sends a cancel. The
harness refuses it when the file isn't at `base_revision`. Otherwise it runs
`git apply --check` and then `git apply` on a copy of the file, and writes the
result over the file only when both pass:
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| 200 | applied; the body is `{revision}`, the file's new revision |
| 409 | the file isn't at `base_revision` any more; nothing applied, and `revision` in the body is the one it's at |
| 422 | `git apply` refused the diff, or it touches a file other than `AGENTS.md`; `detail` carries git's message, and the file stays as it was |
| 503 | the agent isn't running or isn't connected to the queue |
| 504 | the harness didn't answer within ten seconds, so whether the edit applied is unknown |
| 500 | the harness couldn't read or write the file or run git, or the queue failed the request |
The harness refuses a diff that sets any file mode other than `100644`, so an
edit can't turn `AGENTS.md` into a symlink. `PUT` is operator-only, behind
`Remote-User` like cancel: whoever can call it can rewrite the agent's
instructions, so don't widen who holds it without revisiting this.
An edit needs the agent running, and a read of a stopped agent shows the text
it last published. A harness older than this route doesn't know the command and
drops it unanswered, which reads as a 504. `swarmctl agent agents-md get` and
`apply` call the two routes.
## Swarm-wide forge objects
The controller also keeps the forge objects that are one per swarm, not
one per hive. It ensures them at start and every five minutes after
(`swarm-controller/src/forge/objects.rs`):
- the orgs `agent-configs`, `internal` and `agents`, plus each mirror's
owner org;
- the empty `operators` merge-gate team in `agents` and `agent-configs`;
- the `main` merge gate on every `agent-configs` repo: merge and approval
whitelists = the `operators` team, and no user. The controller leaves a
repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
host (with the `actions/checkout` and `actions/upload-artifact` ones
`deploy.forgejo.actionsFromSelf` adds, and `hyperhive/hyperhive` from
`deploy.forgejo.hyperhiveUpstream`);
- `internal/docs` (private) and `internal/knowledge` (public, with a
seed `README.md` while empty);
- the `agent-configs` org avatar
(`deploy.swarm-controller.configOrgAvatarPng`).
A pass that can't finish logs a
`warn` line per object plus `swarm forge objects: pass incomplete` in
`journalctl -u swarm-controller`, and retries on the next tick. While the
controller is down the objects stay as they are.
The controller gives each agent **read** on `internal/docs`. A `GrantDocsRead`
node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the
store's `hive-agent-*` roster (`forge/docs_grant.rs`, at start and every five
minutes) adds any agent that isn't one, so an agent whose create-time grant failed
gets it within five minutes. The pass only adds: a roster it can't read, a
forge read that fails, or an `internal/docs` that doesn't exist (checked once per
pass, logged at warn) changes nothing. The grant doesn't depend on the hive
the agent runs on, or on a forge container there.
## Swarm-wide forge webhooks
At startup the controller registers its Forgejo hooks pointing at
itself, all under `https://<swarm.domain>/webhook/forge/`:
- a `push` hook on `internal/knowledge`
- a `pull_request` hook on the `agent-configs` org
- an instance-wide `push` system hook, for commit metrics
- an instance-wide `pull_request_review_request` system hook, which starts
the [review classifier](review-classifier.md#trigger-and-lifecycle)
- instance-wide `issues` + `issue_assign` and `pull_request` +
`pull_request_assign` + `pull_request_sync` + `pull_request_review_approved` +
`pull_request_review_rejected` + `pull_request_review_comment` system hooks, and
an instance-wide `workflow_run_failure` + `workflow_run_cancelled` system hook,
whose deliveries it keeps in the `SWARM_EVENTS` stream for the
[notification router](notify-router.md#the-path)
The controller **interprets** a delivery and sends hives a specific
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.
The `issues` and `pull_request` deliveries are the exception: no hive gets
them, and the notification router, not the controller, decides what they mean.
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
of its `merge_commit_sha` on the one hive whose wanted state places the
agent; with no such hive, or more than one, the controller deploys nothing. See
[config-changes.md § Config changes](../../2-handbook/agents/config-changes.md#config-changes).
A merge whose `closed` delivery never arrives deploys nothing either; the
operator recovers by redelivering that delivery from the forge hook page,
which re-enters the same handler and re-queues the deploy.
**`internal/knowledge` is on that path.** The controller's is the only
hook on it ([`knowledge.md`](../../2-handbook/swarm/knowledge.md) covers clearing a
leftover). A webhook has exactly one target URL, so a second registration
would take delivery away from the first rather than add a recipient.
<!-- vale write-good.Passive = NO -->
**The `agent-configs` org is on it too.** The controller's hook is the
only one that acts on config PRs. No hive serves `/webhook/config-pr`
any more and none deletes it: a hook of that name left on the org by an
older release stays registered and collects failed deliveries until an
operator deletes it by hand. Removing
the controller's hook stops forge-UI merges from deploying until its
next start recreates it.
<!-- vale write-good.Passive = YES -->
Nothing to configure. The controller registers the hooks only when this
host also serves the swarm UI vhost — that's what publishes the
endpoint, and a hook the forge can't reach would collect failed
deliveries while looking healthy. The controller generates the HMAC
secret on first start and keeps it
(see [`docs/2-handbook/agents/persistence.md`](../../2-handbook/agents/persistence.md)).
To check it's working, push to `internal/knowledge` and look for
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
refused delivery logs `webhook: refused delivery` with the reason.
The controller registers the hooks within five minutes of having both the forge
token and the secret. It keeps trying, at most five minutes apart, for as long
as the forge or the secret store is down. A restart isn't needed to recover.

296
docs/4-implementation/swarm/telemetry.md Normal file
View file

@ -0,0 +1,296 @@
# Swarm telemetry: metrics, logs and the collector
How the swarm's metrics store, Grafana, log store and OTEL collector connect,
and who may reach each. The operator's overview, with the options to touch and
how to open each service, is on
[the services page](../../2-handbook/swarm/services.md#metrics-victoriametrics--grafana).
## Metrics (VictoriaMetrics + Grafana)
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
reads it, in two containers at `metrics.<swarm-domain>` and
`grafana.<swarm-domain>`. Two containers rather than one so you can
restart or break Grafana without taking the time-series database with it.
The Grafana container sits on the hive bridge in a network namespace of its
own and dials authelia, the secret store, the collector and both datasources
by name through it. It binds no TCP port: it listens on the unix socket
`/run/swarm-grafana/grafana.sock`, a directory only the gateway's nginx and
the container can open.
They derive together: a store with no UI is unreadable and a UI with no
store is empty. To run one without the other, set it directly:
```nix
services.hyperhive.deploy = {
victoriametrics.enable = true;
grafana.enable = false;
};
```
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
<!-- vale write-good.Passive = NO -->
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`credentials.md`](../../2-handbook/swarm/credentials.md).
<!-- vale write-good.Passive = YES -->
**Dashboards.** Grafana provisions these from
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
- `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, the busiest agents and destinations, and data transferred by domain over time (the top 10 domains plus an other line, summed from the proxy's access log in VictoriaLogs; the proxy logs a CONNECT tunnel when it closes, so a long-lived connection shows as one spike at close). **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent.
- `builds`: the container builds hive-c0re ran on every hive, in two tabs. **builds** has the count, the failures and the longest, builds over time by outcome, and a table of each build with its id, agent, kind, outcome and runtime. **output** shows every line of the build picked in the **Build** filter, with its start and finish lines. It reads the lines hive-c0re logs for each build, so a hive-c0re that doesn't log them leaves it empty. Filter by agent.
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
- `review-classifier`: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
- `forge`: the forge, in three tabs. **content** has what the forge holds (repositories, users, issues, releases, discussion volume and the issues by label). **runtime** has the forge process's CPU, memory, goroutines and descriptors. **CI runs** has Actions run time and queue wait per repo and workflow, with a per-PR table; the Repo and Workflow filters apply to that tab.
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
- `host`: the machine under each hive, from the collector's host metrics: CPU by state, load, memory, swap and paging, disk and network throughput, filesystem usage, the Nix store's share and the busiest agent containers. Filter by hive and host.
- `collector`: what the hive and swarm OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
- `logstore`, `metricstore`: health of the log store and the metrics store.
- `grafana`: Grafana itself. Dashboards and data sources, alert instances firing, requests by status and their latency, alert rule evaluations and failures, sign-ins, database connections, memory and goroutines.
- `builder`: the swarm builder. Builds finished, by kind and outcome, their mean and 95th percentile time, the queue depth and the wait before a build starts.
- `ci-runner`: the forge's Actions runner. Its uptime and job slots, jobs finished by status, how many slots are in use, job time, and how often it polls the forge for work.
- `matrix`: the Matrix homeserver. Whether its client API answers, how fast, the share of the range it answered, and its database size.
- `notify-router`: the notification router. Forge events received by type and outcome, todos delivered, review requests that failed, time per event, and the backlog of the agents' queue.
A new panel can read "No data" until its source exists: the push and commit
counters on `controller` appear after the first push webhook, the refused
and failure series on `collector` appear once something fails, and the
`review-classifier` panels fill as review requests arrive.
**The `collector` dashboard.** It shows the self-metrics of the hive's
collector and of the swarm's collector, which both carry the collector
binary's default `service.name`, `otelcol-contrib`. The timeseries panels
split by `server.port`: 8888 is the hive collector, 8889 the swarm
collector. Every per-container collector (the secret store, the other service
containers and the agents' forwarders) also scrapes its own endpoint on
loopback and pushes the result up its normal hop, so each has a line of its
own, on its own port: agents all report on 8904, service containers on
8890 to 8903. The series carry `container` (services) or `agent` (agents)
to tell them apart. `scripts/check-dashboard-service-names.sh` runs in CI
and in `scripts/pre-push`. It fails when a dashboard query selects a
`service.name` that nothing emits.
**Where the data comes from.** The swarm's OTEL collector, below.
The `controller` panels come from the controller itself. It exports OTLP
through the gateway to the swarm's collector from whatever host it runs on,
whether or not that host runs the collector.
Neither container is reachable except through the gateway: each listens
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
runs in a network namespace of its own with no interface but loopback.
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it:
- **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource, an agent, or an operator. Grafana sends a
bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
store, and `swarm-metrics-auth` on the store's host checks it against the
same store path. An agent's `swarm-metrics` sends its
`swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. It also needs the `metrics`
permission, or the answer is a 403. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later. When the
store doesn't answer within five seconds, it checks requests against the
last value it read instead of denying, and still refuses a wrong bearer;
it tries the store again after three seconds. A revoked or rotated bearer
stays accepted while the store is down, for an hour after the last read at
most; after that the responder denies again. The store's next answer
replaces the value, and a process that holds none (it started while the
store was down) denies. `swarm-logs-grafana-auth`
is the same binary and does the same.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to
that path; any other session gets `deny`.
The collector scrapes the store's own metrics the same way, at
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
that doesn't run the store still gets them. On this name the gateway admits
only the collector's client to `/metrics` and the push route; any other
session gets `deny`.
## Logs (VictoriaLogs)
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
behind the same SSO as everything else: the swarm's own service containers,
the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
| Option | When you'd touch it |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. A revoked `logs` permission does too. |
| `deploy.victorialogs.grafanaAuthPackage` | The `swarm-metrics-auth` build that runs as `swarm-logs-grafana-auth`. Set on its own: `deploy.victoriametrics.authPackage` doesn't reach it. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `deploy.victorialogs.port` | Only if something else on the services host already claims `9428`. |
It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. The agent must also hold the `logs` permission,
which is off by default; without it the answer is a 403 (`docs/5-project/process/conventions.md`, Permissions). Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`credentials.md`](../../2-handbook/swarm/credentials.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. On this name the gateway admits only the
collector's client to `/metrics` and the ingest route
`/insert/opentelemetry/v1/logs`; any other session gets `deny` on both. The
read side stays open to any authenticated caller.
## Telemetry collector (OTEL)
The swarm's collector receives from every hive's own collector and is the
only process that decides where telemetry goes: it always writes the stores
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
always get both. It holds
the upstream credential, which is why no hive and no agent needs one.
It runs in a `swarm-otel` container on the hive bridge, in a network
namespace of its own. It binds no OTLP port: every receiver (one per hive,
the swarm tier's and one per service forwarder) listens on a unix socket in
`/run/swarm-otel`, a directory only the gateway's nginx and the container can
open, and the gateway forwards to them. The gateway's routes and the collector's
receivers come from one list, so they name the same sockets. The
`swarm-otel-sockets` unit connects to each socket as the gateway's user after
the container starts, then requests every route through the gateway vhost. It
fails if a socket refuses for 30 seconds, or if a route answers `000`, `502` or
`504`, or lacks its own `X-Swarm-Otel-Route` response header, for 30 seconds.
The gateway sets that header, named after the route, on every response from a
route, so a collector that's up but not listening, or an nginx that doesn't
have the route, shows as a failed unit. Any status other than those passes,
the collector's 4xx included. A switch leaves a window of about 20 seconds in which
nginx and the container are out of step: the container stops before nginx
restarts and starts about 18 seconds after it. Senders retry through that window.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a
loopback URL an operator has to redirect. Nothing needs setting on a hive
that doesn't run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `swarm-attic`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter, notification router and
classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store,
authelia, cache, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `attic`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**The homeserver's metrics.** tuwunel 1.9.4 exports no metrics of its own, so
request rates, sync load and federation traffic aren't available. Two series
stand in for them, and neither adds a service:
- The hive collector on the homeserver's host requests
`https://<gatewayHost>/_matrix/client/versions` every 60 s with its
`httpcheck` receiver (`services.hyperhive.otel.httpChecks`). It reports
`httpcheck.status` (one series per status class), `httpcheck.duration` and
`httpcheck.error`, labelled with the URL. The container's forwarder can't
make this request, because `httpcheck` dials HTTP and HTTPS only and tuwunel
listens on a unix socket. The request passes through the gateway, so it
covers the gateway's route as well as the homeserver.
- The `hive-matrix-database-size` timer in the container runs every five
minutes. It measures tuwunel's database directory and pushes the gauge
`hyperhive.matrix.database_bytes` through the forwarder's OTLP port,
stamped `service.name` `hive-matrix`. It exists only where the container
has a forwarder, which needs matrix-ctl's store identity set. A size under
4096 bytes is no database, so the timer logs a warning and pushes nothing.
The swarm collector's own container, `swarm-otel`, has no forwarder: its
collector reads the container's journal itself (receiver `journald/self`) and
exports it to the log store like the host's journal, stamped `service.name`
`swarm-otel` and with no `hive` label. Unlike the host's journal, it goes to the
log store only, never the upstream: the collector writes an upstream export failure to this
journal, and shipping it upstream would fail again. A `filter` processor also
drops the collector's own failure lines that name the store exporter
(`otlphttp/victorialogs`), which can't reach the store anyway; failures of
every other exporter still arrive. The container mounts the host's journal
read-only at `/run/host-journal`, not over its own
`/var/log/journal`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in
[`scheduler/observability.md`](../scheduler/observability.md).

6
docs/README.md
View file

@ -169,6 +169,12 @@ Nothing of it sits in this tree.
it?** → [`4-implementation/swarm/ci-metrics.md`](4-implementation/swarm/ci-metrics.md).
- **How does an agent get a todo when someone assigns it an issue, and lose it
when the issue closes?** → [`4-implementation/swarm/notify-router.md`](4-implementation/swarm/notify-router.md).
- **How does the controller take in hive status, give agents the queue, keep an
agent's `AGENTS.md` and register forge webhooks?** →
[`4-implementation/swarm/controller.md`](4-implementation/swarm/controller.md).
- **How are the metrics store, Grafana, log store and OTEL collector wired, and
who may reach each?** →
[`4-implementation/swarm/telemetry.md`](4-implementation/swarm/telemetry.md).
### Trust boundary mechanics