mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
16 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

682
docs/swarm/README.md
View file

@ -1,682 +0,0 @@
# The swarm
The **swarm** is where things live: agent identities and accounts, secrets,
the job graph that creates and places agents, telemetry, and the UI you
drive it all from. **Hives are the substrate** — NixOS hosts that run agent
containers on the swarm's behalf. Every hive belongs to a swarm; a single
host is a swarm of one.
This page is for the operator. It covers the control plane, how a hive
joins the directory, and how hives and agents report upward. The steps for a
fresh swarm are in [`setup.md`](../getting-started/setup.md); the all-local
config is the [README quick start](../../README.md#quick-start-an-all-local-swarm).
Option reference: `services.hyperhive.swarm.*` (swarm-wide facts, identical
on every host) and `services.hyperhive.deploy.*` (whether _this_ host runs
grafana, the controller, authelia, …) →
[options reference](https://hyperhive.darkest.space/options/), or
`nix build .#docs-swarm` / `.#docs-deploy`.
## Where each piece lives
- **control plane** — `swarm-controller`: hive directory, agent roster, job
graph, agent creation. → [below](#swarm-controller)
- **scheduled prompts** — named prompts for one or more agents at set times,
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
over its own. → [`inference.md`](inference.md)
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
over its hive's. →
[`coordinator.md`](../scheduler/coordinator.md#container-resource-limits)
- **review classifier** — checks a PR against its repo's own yes/no rules
and posts the result as a review. →
[`review-classifier.md`](review-classifier.md)
- **notification router** — forge events turned into todos on the agents
they concern: assigned, unassigned, cleared on close; and the classifier's
review requested on a push. →
[`notify-router.md`](notify-router.md)
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
`admins` only. → [`ui.md`](ui.md)
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
stack, each on whichever host you put it. → [`services.md`](services.md)
- **SSO** — the secrets authelia generates and how each reaches its reader.
→ [`sso.md`](sso.md)
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
every credential the swarm holds, who mints it, who reads it, how it renews
and where it lives → [`credentials-reference.md`](credentials-reference.md) ·
how the store comes up, who writes its grants and how it unseals →
[`bao.md`](bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer. → [`ca.md`](ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint,
`swarm.snapshotStore.{address,port}`. →
[`snapshot-store.md`](../networking/snapshot-store.md)
## Swarm controller
`swarm-controller` is the swarm's control plane: it holds the hive directory,
the agent roster and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent,
the step it stopped at and the error, which the Grafana alert rule
`hh-agent-identity-mint-failed` fires on. The agent is still created, without a
store identity, and the controller's five-minute credential pass mints one for
it (also when only the certificate exists and its cert-auth role is missing) as
long as the agent isn't `Destroyed`. A mint that runs after a hive declares the
agent `Destroyed` logs `identity mint skipped` and writes nothing.
`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
runs the controller.** A swarm has one controller, so enabling it states a
fact about swarm topology, not about whether this host runs a hive.
What it serves, why it's a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../swarm-controller/README.md).
### Per-hive status (`GET /api/hives/status`)
One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.
A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.
| freshness | what to do about it |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive removed from the roster but still running |
The snapshot is the hive's `/health/ready` document plus `capabilities`, a list
of names for what the hive's release can do. Two exist: `tolerant-permissions`, and `no-config-checkout`, which
[turning an archetype agent into a custom one](archetypes.md#turn-into-custom)
needs. A hive on an older release lists none.
The snapshot also carries `egress_deny_applied`, the revision of the egress deny
list the hive last rendered into its proxy (`0` when the swarm has none). A hive
on an older release, or one that hasn't read the list yet, leaves it out; see
[the egress proxy](../networking/network.md#egress-proxy-squid).
Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload. A roster hive's row
carries `dashboard`, the host its dashboard answers on; the swarm UI's
hive list links there.
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.
### Making a hive report
Two options on the **hive**, plus its store identity. They sit in two
namespaces, because the queue's URL is a fact about _this machine_ and the
token endpoint is the swarm's single address:
| option | what to set it to |
| ---------------------------------------------- | --------------------------------------------- |
| `deploy.hive-controller.statusPublish.natsUrl` | `tls://<swarm.nats.domain>:<swarm.nats.port>` |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). Authelia's mint, in its
container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config`
warning and tries again, waiting 5 seconds after the first failure and doubling
up to 5 minutes. It starts publishing as soon as a read succeeds, with no
restart.
The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.
The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue
host without the store gets no default, even when it holds the hive's own
reader `deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.natsCommonName`, place it, and name it in both options. Until you
do, `swarm-bao-nats-tls` fails naming both whenever the queue's certificate is
missing or due for renewal.
The queue host's gateway also serves the queue's metrics, at
`https://<swarm.nats.domain>/metrics`, so a swarm collector on any host
scrapes them. Only the collector's authelia client gets through; any other
session gets `deny`, and every other path on that name is 403. With the
gateway's default self-signed TLS, that vhost serves the queue's own
certificate, so a host whose only swarm service is the queue needs no
swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.
The endpoint answers **503** when this host has no swarm queue
configured, or has one and can't read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
can't see. The body says which. Status survives a controller restart:
it's stored in the queue, not in the daemon.
### Giving the agents the queue too
An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:
| option | what to set it to |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
It reaches each container at start, as `HIVE_AGENT_NATS_URL` in
`/etc/hyperhive-hive-facts`, so no agent build carries it. On every hive,
`agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. The option takes one
`<scheme>://<host>:<port>` (a hostname or IPv4 address), not an IPv6 literal or a
list: evaluation refuses anything else. On the queue's
host the name resolves inside a container to the bridge address, where the
firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.
**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:
- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits. The error reads
`no swarm queue connection; exiting, since this agent cannot run without one`,
and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.
Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading
the credential from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for a `swarm-logs` built before it read the agent's
own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.
<details><summary>What an agent publishes over the queue</summary>
What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. The queue keeps the rows in
the stream `term-main-<agent>` for 24 hours or 64 MiB, the same limits and the
same creator as the subagent stream below, so a swarm terminal that attaches
later replays the newest of them. The two streams hold disjoint subjects.
An agent's **subagents** publish their terminals too, from the agent's
subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`,
classified the same way. The queue grants that family to the agent's own queue
credential alone, so the daemon reads it from the store under the agent's store
identity, exactly as the harness does, and publishes nothing without it. The
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
whichever comes first, dropping the oldest rows once either limit kicks in so
a publish never fails because of it, and the swarm lists an agent's subagents
from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for
every agent a hive's wanted state names, within a minute of the agent
appearing there. The agent's grant is publish on its own
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
subjects and limits are the controller's and never the agent's. Subagents
publish output only and read nothing.
The queue would refuse a row too large for its `max_payload` outright and
take the connection down with it, so the harness drops such a row's body before
sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.
The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model and effort (`model`, `effort`, where each comes from as
`model_source` and `effort_source`, and the resolved id the last turn
actually ran on), the
context budget and the last turn's context and cost token blocks,
`agent_state`, and `needs_login`: `true` while the agent's claude has no
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.
`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
spellings. ⚠️ From inside the container only two of those four are sayable: the
harness reports `up`, or `paused` when the pause marker is present. `offline` and
`destroyed` are hive-c0re's observations — a stopped agent publishes nothing and
a destroyed one doesn't exist — so a view that needs the full four-state picture
takes them from the `agent-status` bucket and uses this subject to sharpen the
rest.
Headers go out **on transition**: the harness rebuilds the header whenever its
event bus moves and publishes when the result differs from what it last sent.
That's the whole point of the subject — the `agent-status` bucket already
republishes once a minute, which is far too slow for "is this agent thinking
right now." A core subject keeps nothing for a late subscriber, so the harness
also re-sends the current header once a minute: a subscriber attaching
mid-idle sees it within that minute. The swarm's stream sends a new client the
latest header it holds first, when that's under 150 seconds old. A renderer opens with the bucket's
snapshot and lets this stream refine it.
Swarm-side, `GET /api/agents/<name>/state/stream` relays that subject as SSE,
resolving the agent's hive at request time exactly as the terminal stream does.
The payload passes through opaquely. Separately, the controller keeps each
agent's latest `model`, `effort` and their sources for
`GET /api/agents/<name>/inference` ([`inference.md`](inference.md)), and each
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key of that bucket. The harness writes
once per start, because a config change reaches an agent by restarting its
container. An agent with no icon deletes its key. Swarm-side,
`GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.
The fourth is its **`AGENTS.md`**, which goes into the `agents-md` bucket under
the key `<agent>`, writable by the agent's own credential alone
(`$KV.agents-md.<agent>`). The swarm reads that copy and edits the file through
the agent; [An agent's AGENTS.md](#an-agents-agentsmd) has the details.
The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>`, its operator-input subject `$SWARM.input.<agent>` and
its command subject `$SWARM.control.<agent>`, and nothing else. Besides the
subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies.
Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
sets as their inbox prefix. A KV watch or key listing delivers there as well. A
hive may also subscribe to `$SWARM.knowledge` and its own `$SWARM.deploy.<hive>`,
so it can't read another hive's deploys or any agent's terminal. The controller
may also subscribe to `$SWARM.term.<agent>`, `$SWARM.term.<agent>.sub.<subagent>`
and `$SWARM.agent-state.<agent>`, the streams it relays to swarm-ui. A hive or
controller that connects with the library's default `_INBOX.<nuid>` prefix gets
no replies, so every request it makes times out.
</details>
### An agent's AGENTS.md
The file in the agent's state dir, `state/AGENTS.md`, is the only copy that
counts, and only the agent writes it: itself, or its harness applying an edit
from the swarm. No hive is in either path.
**Reading.** The harness publishes the file's text and its revision, the git
blob id that `git hash-object` prints for it, at start, within ten seconds of
any change, and straight after an edit it applied. An agent with no `AGENTS.md`
publishes `""`. `GET /api/agents/<name>/agents-md` answers `{content, revision}`
from that copy, so it works while the agent isn't running. A 404 means the agent
hasn't published one: its harness hasn't run since it began to.
**Initial text.** `POST /api/agents` and `swarmctl agent create --agents-md <file>`
take an optional `agents_md`, the text the file starts with, up to 64 KiB; a
longer text is a 400. The controller stores it under the key `<agent>` of the
`agents-md-initial` bucket, which only it writes, before it queues the creation.
At start the harness reads its own key, with one direct get on
`$JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>`, and
writes the text to `AGENTS.md` only when no file exists there. It does this
before the first mirror publish and the first turn. An existing file is never
replaced, so a restart or a re-create keeps the agent's edits; a later change
to the file never reaches the key. Destroying the agent deletes the key. When
the harness can't read the key it starts without the text and logs a warning.
The text holds no secret, so nothing goes through the secret store.
**Editing.** `PUT /api/agents/<name>/agents-md` takes `{base_revision, diff}`.
The diff is a unified diff that names `a/AGENTS.md` and `b/AGENTS.md`, made
against the text at `base_revision`. swarm-controller sends it as a request on
the agent's command subject `$SWARM.control.<name>`, as it sends a cancel. The
harness refuses it when the file isn't at `base_revision`. Otherwise it runs
`git apply --check` and then `git apply` on a copy of the file, and writes the
result over the file only when both pass:
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| 200 | applied; the body is `{revision}`, the file's new revision |
| 409 | the file isn't at `base_revision` any more; nothing applied, and `revision` in the body is the one it's at |
| 422 | `git apply` refused the diff, or it touches a file other than `AGENTS.md`; `detail` carries git's message, and the file stays as it was |
| 503 | the agent isn't running or isn't connected to the queue |
| 504 | the harness didn't answer within ten seconds, so whether the edit applied is unknown |
| 500 | the harness couldn't read or write the file or run git, or the queue failed the request |
The harness refuses a diff that sets any file mode other than `100644`, so an
edit can't turn `AGENTS.md` into a symlink. `PUT` is operator-only, behind
`Remote-User` like cancel: whoever can call it can rewrite the agent's
instructions, so don't widen who holds it without revisiting this.
An edit needs the agent running, and a read of a stopped agent shows the text
it last published. A harness older than this route doesn't know the command and
drops it unanswered, which reads as a 504. `swarmctl agent agents-md get` and
`apply` call the two routes.
### Swarm-wide forge objects
The controller also keeps the forge objects that are one per swarm, not
one per hive. It ensures them at start and every five minutes after
(`swarm-controller/src/forge/objects.rs`):
- the orgs `agent-configs`, `internal` and `agents`, plus each mirror's
owner org;
- the empty `operators` merge-gate team in `agents` and `agent-configs`;
- the `main` merge gate on every `agent-configs` repo: merge and approval
whitelists = the `operators` team, and no user. The controller leaves a
repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
host (with the `actions/checkout` and `actions/upload-artifact` ones
`deploy.forgejo.actionsFromSelf` adds);
- `internal/docs` (private) and `internal/knowledge` (public, with a
seed `README.md` while empty);
- the `agent-configs` org avatar
(`deploy.swarm-controller.configOrgAvatarPng`).
A pass that can't finish logs a
`warn` line per object plus `swarm forge objects: pass incomplete` in
`journalctl -u swarm-controller`, and retries on the next tick. While the
controller is down the objects stay as they are.
The controller gives each agent **read** on `internal/docs`. A `GrantDocsRead`
node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the
store's `hive-agent-*` roster (`forge/docs_grant.rs`, at start and every five
minutes) adds any agent that isn't one, so an agent whose create-time grant failed
gets it within five minutes. The pass only adds: a roster it can't read, a
forge read that fails, or an `internal/docs` that doesn't exist (checked once per
pass, logged at warn) changes nothing. The grant doesn't depend on the hive
the agent runs on, or on a forge container there.
### Swarm-wide forge webhooks
At startup the controller registers its Forgejo hooks pointing at
itself, all under `https://<swarm.domain>/webhook/forge/`:
- a `push` hook on `internal/knowledge`
- a `pull_request` hook on the `agent-configs` org
- an instance-wide `push` system hook, for commit metrics
- an instance-wide `pull_request_review_request` system hook, which starts
the [review classifier](review-classifier.md#trigger-and-lifecycle)
- instance-wide `issues` + `issue_assign` and `pull_request` +
`pull_request_assign` + `pull_request_sync` + `pull_request_review_approved` +
`pull_request_review_rejected` + `pull_request_review_comment` system hooks, whose deliveries it
keeps in the `SWARM_EVENTS` stream for the
[notification router](notify-router.md#the-path)
The controller **interprets** a delivery and sends hives a specific
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.
The `issues` and `pull_request` deliveries are the exception: no hive gets
them, and the notification router, not the controller, decides what they mean.
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
of its `merge_commit_sha` on the one hive whose wanted state places the
agent; with no such hive, or more than one, the controller deploys nothing. See
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes).
A merge whose `closed` delivery never arrives deploys nothing either; the
operator recovers by redelivering that delivery from the forge hook page,
which re-enters the same handler and re-queues the deploy.
**`internal/knowledge` is on that path.** The controller's is the only
hook on it ([`knowledge.md`](../integrations/knowledge.md) covers clearing a
leftover). A webhook has exactly one target URL, so a second registration
would take delivery away from the first rather than add a recipient.
<!-- vale write-good.Passive = NO -->
**The `agent-configs` org is on it too.** The controller's hook is the
only one that acts on config PRs. Each hive deletes its own
`/webhook/config-pr` org hook on startup, since a route no hive serves
would otherwise sit on the org collecting failed deliveries. Removing
the controller's hook stops forge-UI merges from deploying until its
next start recreates it.
<!-- vale write-good.Passive = YES -->
Nothing to configure. The controller registers the hooks only when this
host also serves the swarm UI vhost — that's what publishes the
endpoint, and a hook the forge can't reach would collect failed
deliveries while looking healthy. The controller generates the HMAC
secret on first start and keeps it
(see [`docs/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)).
To check it's working, push to `internal/knowledge` and look for
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
refused delivery logs `webhook: refused delivery` with the reason.
The controller registers the hooks within five minutes of having both the forge
token and the secret. It keeps trying, at most five minutes apart, for as long
as the forge or the secret store is down. A restart isn't needed to recover.
## Hives: the substrate
- **hive** — one host running `hive-c0re` and its agent containers
(`deploy.hive-controller.enable`). Addressed as `<hiveName>.<swarm.domain>`.
- **swarm** — every hive in `services.hyperhive.swarm.hives`, plus the
shared services and controller. You can qualify an agent as
`agent@hive-domain`.
- **peer hive** — any hive in the directory other than this one. Peers are
_derived_, not declared: the directory lists every hive including
yourself, and `hiveName` says which one you are.
### Hive identity config
```nix
services.hyperhive = {
swarm.domain = "example.com"; # required — the swarm's DNS domain
deploy.hive-controller.hiveName = "pr1ma"; # required on a hive host — this hive's label in it
swarm.name = "constellat1on"; # shared swarm display name (optional)
# required — the directory, identical on every host in the swarm.
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
swarm.hives = {
pr1ma = { }; # this host, per hiveName
lab = { }; # a second hive
edge = { domain = "edge.elsewhere.example"; }; # addressed off-convention
};
};
```
<!-- vale write-good.Passive = NO -->
Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.
<!-- vale write-good.Passive = YES -->
**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.
```
# hive A # hive B
deploy.hive-controller.hiveName = "pr1ma"; deploy.hive-controller.hiveName = "edge";
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
```
One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.
`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`iris@pr1ma.example.com`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.
`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
The env-var chain and `qualify()` semantics:
[`conventions.md`](../process/conventions.md) § Hive identity.
Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.
### What the directory feeds
1. **Swarm-wide hive roster** — swarm-controller reads this same
directory and serves it at `GET /api/hives`; the swarm UI's front page
renders it ([`ui.md`](ui.md)).
2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
federates with the peer's matrix server (discovered via the peer's
`.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
Federation validates the peer's TLS certificate against the matrix
**container's** trust bundle, independent of this directory.
⚠️ On a hive with self-signed gateway certificates, the container
also trusts this hive's `trust-bundle.pem`, bound in at runtime and
added to the public CAs; it holds the store's `pki` root, so a peer whose
certificate chains to the same root validates. A peer outside this
swarm's root needs a CA-issued certificate (ACME). Federation firewall
and TLS requirements: [`integrations/matrix.md`](../integrations/matrix.md).
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).
### WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.
#### Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
#### Config example (two hives)
```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
pr1ma = {
domain = "pr1ma.example.com";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
edge = {
domain = "edge.corp";
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = { /* … identical to hive A's … */ };
};
```
#### What the mesh does
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
host (not inside agent containers; containers reach peers via the
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
from `services.hyperhive.swarm.hives`, minus this host's own hive, to
build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
#### NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.
## Cross-references
- [`ui.md`](ui.md) — the swarm UI, the operator surface for "what hives exist"
- [`../networking/snapshot-store.md`](../networking/snapshot-store.md) — the
swarm's `btrfs receive` endpoint and the `swarm.snapshotStore` option
- [`../process/conventions.md`](../process/conventions.md) § Hive identity —
env vars, qualified labels
- [`../integrations/matrix.md`](../integrations/matrix.md) — matrix
federation, TLS cert autogeneration, firewall posture
- [`../networking/gateway.md`](../networking/gateway.md) — nginx vhosts and
the `.well-known/matrix/` autodiscovery scheme

195
docs/swarm/archetypes.md
View file

@ -1,195 +0,0 @@
# Agent archetypes
An archetype is a named agent setup: settings to create the agent with, and
a NixOS module its container runs. The swarm UI's create form lists the
archetypes the swarm knows. Picking one fills the form's permissions,
resource caps and inference fields from it, and you can still change each
field before you create the agent.
An agent created from an archetype has no `agent-configs/<name>` repo of its
own. The swarm's `internal/meta` flake builds it from the archetype's module,
so every agent of that archetype runs the same configuration. To change what it runs, change the
archetype. Picking "none" creates a custom agent, with its own config repo. The
swarm can also turn an archetype agent into a custom one later
([below](#turn-into-custom)).
## The built-in archetypes
The hyperhive flake ships these:
| archetype | what it sets |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reviewer` | permissions `messaging`, `inbox`, `execution`: the default set without `meta`. With `execution` it can run `hive-forge`, `git` and other commands. |
| `android-dev` | installs `android-tools` (`adb`, `fastboot`), Gradle and a JDK; default permissions. |
| `reverse-engineer` | sets `services.hyperhive.agent.rea.enable`: installs REA, Ghidra 12.1 and procps, and allows all REA MCP tools; default permissions and the agent's normal memory cap. |
## Where an archetype comes from
A flake declares archetypes in its `hyperhiveArchetypes` output:
```nix
hyperhiveArchetypes.reviewer = {
module = { };
meta = {
description = "Reviews what it is sent.";
permissions = [ "messaging" "inbox" "execution" ];
limits = { memory_max = "4G"; };
inference = { effort = "high"; };
};
};
```
- `meta` is plain data: no derivation, no path, no function. The swarm
evaluates `meta` alone.
- Every archetype needs a `description`. `permissions`, `limits` and
`inference` are optional and take the shapes `POST /api/agents` takes. The swarm ignores
other keys in `meta`.
- `module` is a NixOS module for the agent's container. The meta flake
imports it into each agent created from the archetype.
Today the swarm reads one flake: the hyperhive source the
[builder](builder.md)'s host runs, so only the built-ins appear. The meta
flake has one archetype source too, its own `hyperhive` input, and refuses
an archetype from any other source name.
## How the list stays current
1. swarm-controller asks the builder for the list on the queue subject
`$SWARM.builder.archetypes` once the queue connects, and every 10 minutes
after that.
2. The builder evaluates the `meta` of each `hyperhiveArchetypes` entry on
its own hyperhive and nixpkgs sources, and answers with the JSON or the
last lines `nix` printed:
```sh
nix eval --json <hyperhive>#hyperhiveArchetypes \
--apply 'builtins.mapAttrs (_: a: a.meta)'
```
3. The controller checks each archetype the way `POST /api/agents` checks
its settings and stores the result in the `agent-archetypes` KV bucket,
under the key `hyperhive`, when it differs from the stored one. The list
leaves out an archetype that fails the check, and its name and reason go
into the record's `refused` map and the controller's journal.
4. `GET /api/archetypes` answers from that record, and the create form lists
from it.
A builder host redeploy can change the list. The next pass, within 10
minutes, picks it up. When no builder answers, the controller keeps the
stored list and logs a warning on each pass. A swarm with no builder enabled
shows no archetype select.
The stored value:
```json
{
"source": "/nix/store/…-hyperhive-flake-source",
"archetypes": {
"reviewer": {
"description": "…",
"permissions": ["messaging", "inbox", "execution"]
}
},
"refused": { "broken": "limits: …" }
}
```
## Creating an agent from an archetype
The create form sends the picked archetype as `archetype` on
`POST /api/agents`, `{ "source": "hyperhive", "name": "reviewer" }`. The
controller refuses with 400 an archetype the stored list doesn't hold, then:
1. stores the choice in the `agent-archetype` KV bucket, keyed by agent name,
next to the agent's permissions, caps and inference;
2. queues the creation graph without `CreateRepo`, `AddRepoMember` and
`InitAgentConfigRepo`. Identity, forge user, docs read, the mints and the
wanted state stay, and the deploy waits on those alone.
Destroying the agent deletes its key.
Every hive may read any key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-archetype.>`) and write none; only the
controller writes it.
Deploy `swarm-nats-auth` before the hives: every provision reads the
agent's key, and a hive whose queue connection predates the grant gets a
refused read, so its `Provision` nodes fail until it reconnects.
swarm-controller renders `internal/meta`'s `flake.nix` with the
`hive-meta-flake` crate, and reads each agent's archetype from the
`agent-archetype` bucket to do so. For an archetype agent the render:
- adds no `agent-<name>` input and reads no config repo, and lists the agent
as `mkAgent { …; archetype = { source = "hyperhive"; name = "<name>"; }; }`;
- makes `mkAgent` import `hyperhive.hyperhiveArchetypes.<name>.module` in
place of the agent's own flake, and set `HIVE_ARCHETYPE` in the harness
environment.
The swarm builds from `internal/meta` alone and the hive deploys that closure. The hive reads the agent's key when it
provisions the agent and on every rebuild, and keeps the last value read in
`/var/lib/hyperhive/swarm-agent-archetypes.json`. When the read fails, a rebuild keeps the stored value. For an
archetype agent the hive also:
- deploys the closure its record declares, as every agent does.
The agent's system prompt says it has no config repo and names its
archetype.
The module comes from the `hyperhive` input that `internal/meta` pins, not
from the hive's own. When that pin predates the archetype, the agent's build
fails with `this hive's hyperhive input has no archetype <name>; update that
input`. Update the `hyperhive` pin in `internal/meta` to fix it.
## Turn into custom
An archetype agent can become a custom agent with its own
`agent-configs/<name>` repo, either from the agent's overview in the swarm UI
(the "turn into custom" button asks first) or with `POST /api/agents/{name}/custom`.
The agent keeps its state, permissions, caps, inference settings and accounts.
Its configuration stays the same, except that it no longer has `HIVE_ARCHETYPE`
in its environment. Keeping the config current is then up to the repo's owner.
Nothing here undoes it.
The controller refuses with 409 an agent that no hive or more than one hive
places, and one whose hive doesn't list the
`no-config-checkout` capability (see below). Otherwise it queues a job graph:
1. `CreateRepo`, `CreateForgeUser` and `AddRepoMember`: the repo and the
agent's access to the repo, as for any custom agent. Each is a no-op for what
exists.
2. `SeedArchetypeConfig` commits two files to the repo. It copies nothing from
the archetype:
- `flake.nix` declares the archetype's source as an input, at the URL
`internal/meta` holds for that source today (a pinned rev, which is the input's only pin: the repo has no lock, and a source meta doesn't pin to a commit fails the node), and imports
`inputs.<source>.hyperhiveArchetypes.<name>.module` next to `./agent.nix`;
- `agent.nix` is an empty module for the owner's edits.
The node leaves a repo whose `flake.nix` already is this file alone, whatever
URL it pins. Any other `flake.nix` fails the node.
3. `DropArchetype` deletes the agent's `agent-archetype` key, only after the
seed and the member grant succeeded. A failure before it leaves the agent
archetype-built and working, and you can send the same request again.
4. `UpdateMeta`, then the build and the deploy, as for a merged config PR.
If the build fails, the agent is custom and has no key. Sending the request
again then queues only this step, as the earlier ones have run.
When the source is `hyperhive`, the meta flake sets
`agent-<name>.inputs.hyperhive.follows = "hyperhive"` for an agent whose
`flake.nix` or `flake.lock` declares a `hyperhive` input, so the lock holds one
hyperhive, and the module comes from the pin `internal/meta` holds. Any other
source keeps the pinned URL from its seeded `flake.nix`, and the meta lock
records that rev. The `follows` applies to every custom agent whose `flake.nix`
declares a `hyperhive` input, not only converted ones. A `hyperhive` node left
in a lock doesn't count.
### Which hives may convert
A hive that keeps a checkout of a custom agent's config would create one for
the converted agent, under the container's running mount. A hive that lists
`no-config-checkout` under `capabilities` in the status it publishes (the
document `GET /api/hives/status` serves as `snapshot`) keeps none: it deploys a
custom agent's closure from the swarm alone, and the agent clones its own repo from
the forge. The controller reads the agent's hive's latest status and refuses
unless it's `fresh` and lists the capability; an older hive lists none. Deploy
the release to every hive that places archetype agents before converting theirs.

152
docs/swarm/bao.md
View file

@ -1,152 +0,0 @@
# The swarm secret store (OpenBao)
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
credential an agent or service needs comes from it under that
principal's own certificate identity. What lives in it and who reads what:
[`credentials-reference.md`](credentials-reference.md). This page covers the store itself — how it
comes up, who may write its grants, and how it's sealed and reached.
The operator steps (init, the one-time granter bootstrap) are in
[`../getting-started/setup.md`](../getting-started/setup.md#1--secret-store);
this page is the why behind them.
<!-- vale write-good.Passive = NO -->
## The granter
Cert auth answers a _role_, so nothing can authenticate until some role
exists. A short-lived bootstrap token breaks that cycle once, and it's the
only step that needs the root token.
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
covers the auth mounts and the granter's own policy and role, and nothing
else. CI fails when the unit using the token needs a path it lacks. The token
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
names for you; on a store host that isn't all-local, set it and rebuild first.
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
auth methods (and disables `approle` if an older store still has it mounted),
writes the `bao-granter` policy, and creates the `bao-granter` role, which
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
mounts the KV and pki engines and writes the `swarm-controller` role, and each
sibling unit writes its own principal's policy and role. Every one runs on the
host, because every API listener but the loopback UI one demands a client
certificate and the host is the side that has one.
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
the token file is absent, which is the steady state afterwards; the token's
TTL means a forgotten file expires rather than lingering.
A sealed or unreachable store, or one whose listing fails with something other
than a refusal (a node still becoming active), makes the unit retry every
30 s with the store's error in the log. A store that refuses the token (expired,
revoked, or missing the `bao-bootstrap` policy)
makes it exit 4 and stay `failed` with the one-time step in its log; mint a
fresh token or remove the file, then restart it. If the granter's own
certificate login already works, the unit logs that the bootstrap token isn't
needed, deletes the dead token file and exits 0, so a stale file is harmless
and the unit runs once. A successful run deletes the token file too.
After that, a new or changed `swarm-*` grant needs no operator step: the unit
that writes it changes, and the deploy restarts it. A root step comes back only
when the granter itself needs a path it lacks, such as a new mount.
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
them, not on every deploy. When a grant drifts in the store and its unit stays
the same, the next boot re-asserts it, not the next switch.
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
wrapper carries an address, a CA and a client certificate but deliberately
**no token**, so that read answers `403` whether or not the role exists. Read
the unit's journal instead.
### Residual risk
The granter is root-equivalent. It may write any `swarm-*` policy with any
content, and attach it to a role that accepts any certificate; no bao ACL can
constrain what a policy says. What bounds it:
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
module-eval pins each principal's grants. **Merging a change to that policy
text is granting it**: it takes effect on the next deploy with no bao step,
so code review is the only gate.
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
root in a `0700` directory, readable only by root units on the store's host.
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
and `controller-key.pem`, whose policy is already root-equivalent. Root on
that host gains nothing new.
- **Never copy `granter-key.pem` off the host** the way operators copy the
other leaves in that directory. That hands out root-equivalence.
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
subject. Rotate the store's CA, or have root point the `bao-granter` role at
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
one.
## Sealing
`services.hyperhive.deploy.bao.seal`:
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
unseals itself on every restart. `init` is the only manual step.
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
restart, with the keys `init` printed.
⚠️ **A sealed store still answers.** The container is up and the port
responds while every read times out — the failure looks like a stall, not like
a store that was never initialised.
## TLS
The store serves TLS, and on a host that deploys it you need do nothing:
`swarm-bao-pki` mints a CA of the store's own plus the leaves it signs — the
store's server certificate, this host's client certificate, and one per service
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
It runs at every boot and every deploy, and before each start of a container
that loads one of its leaves, and signs only the files that are missing, so a
leaf a deploy adds, or a service a deploy enables, has its file on that deploy.
The store's readers and granting units on this host order after it and want it,
but none requires it: when a run fails, each of them still starts, and one
whose leaf is missing fails its login into its own retry. It has no start
limit, since a mass container start pulls it in once for each unit it starts.
Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about
which. A host that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see
[per-principal identities](credentials-reference.md#store-identities-one-per-principal) for the list.
They're the credentials the store can't hand you, being what opens it, so the
store's host signs them and the operator copies them to the host that needs
them and names them in config. The service principals' leaves are at the top of
`/var/lib/swarm-bao-pki/`, named after the principal.
For every hive in `swarm.hives` other than its own, the store's host signs
that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
| file | subject | name it in |
| ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on
the store host's next deploy; a leaf already there is never re-signed.
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
between services and the store**, a separate trust domain, because a store
that took its identity from an authority it itself distributes could never
come up before that authority.
Inside the store's container (`nixos-container root-login swarm-bao`) the
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
and its DNS name resolves to the bridge from in there: export
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
domain>`. The host's `bao` wrapper already carries the address, CA and client
certificate, so the host is the shorter path.
<!-- vale write-good.Passive = YES -->

607
docs/swarm/builder.md
View file

@ -1,607 +0,0 @@
# Swarm builder
The swarm builder checks that an agent config PR builds before an operator
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
push, the builder builds the PR's head on the agent base, and the result shows as a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a failed build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](services.md#builder).
## What happens on a config PR
1. The forge delivers the `pull_request` event on the `config-pr` webhook
swarm-controller already registers on `agent-configs`. An `opened`,
`reopened` or `synchronized` action on an open PR queues one `build_agent`
node carrying the agent, the head commit and the PR number. An edit, a
label or a close builds nothing. Every `build_agent` node needs the one
`builder` slot, so builds run one at a time and the rest wait pending. A
newer head cancels its PR's pending build; a build already running
finishes and writes its status on its own, older commit.
2. The node sets the status to `pending`, then sends the build as a request on
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
reply. The builder runs one build at a time and refuses a request that has
waited 30 seconds for its turn, such as one behind a build a controller
restart orphaned: a build started later could still run when the 61
minutes end. Reading the forge token before a build has 20 seconds, and
the rest of the minute covers the reply's trip back.
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
with every input overridden:
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
deployed with (`deploy.hyperhiveFlake`, `deploy.builder.nixpkgsFlake`). A hive's meta
flake pins `hyperhive` by forge URL and rev instead. The builder reads the
hyperhive path from `/etc/hyperhive/swarm-builder.json` in its container at
the start of each build, so a deploy that changes only that source doesn't
restart it or end a build in progress.
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
A build has one hour.
4. The reply is the system closure's store path, or the last lines `nix`
printed. The node writes the status on the head commit:
| Outcome | Status | Description | Node |
| ----------------------------------------- | --------- | --------------------------- | ------ |
| built | `success` | the store path | done |
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
| the builder busy for 30 seconds | `failure` | why | failed |
| no builder listening, or no reply in time | `error` | why | failed |
The controller records every build that gets a final status, and the
status links to that record: see [Build records](#build-records). The
full tail is also in the failed node's error on the Jobs page and in the
builder's journal line for the build, as the field `F_ERROR_TAIL`:
```console
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
```
The builder holds the last 64 KiB of each of `nix`'s output streams while the
build runs, and the reply is the last 30 lines of that (at most 4000
bytes), so a long build log never grows its memory.
## Build records
`swarm-controller` records each config PR build when it writes the final status:
built, failed, or no build ran or came back. It also records each [deploy-gate
build](#deploy-gate) the builder answers, in the same list. The newest 100 are one JSON
array under the key `recent` in the `builder-runs` KV bucket on the swarm
queue, which only the controller reads or writes. It creates the bucket on
first use and adds each build with a compare-and-set write, so the records
survive a controller restart. Recording the 101st drops the oldest. A record
the controller can't decode stays in the bucket, out of the list, and counts
toward the 100.
Each record has:
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>` for a
config PR's build, `meta-<agent>-<first 12 characters of the commit>-<started_unix>`
for a deploy's
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`; a deploy's build has
`repo` `internal/meta`, no `pr`, the `internal/meta` commit as `head_sha`,
and `agent`, the agent it built
- `started_unix` and `finished_unix`
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
or `error`
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
`failed`, and `detail` (why) for `error`
`GET /api/builder/runs` lists them, newest first, and
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
when the controller has no swarm queue or can't open the bucket, 500 when it
can't read it.
The controller records a deploy-gate build when the builder answers it, or when
asking it fails (no builder listening, no reply). A build that never got as far
as a request (no forge access, no `internal/meta` head) leaves no record, so a
controller that can't reach the forge doesn't fill the list with retries. Every
ten-minute retry of a failing agent is a record, and a version bump records one
build per agent, so the 100 cover fewer days when builds fail.
The final status links to `https://<swarm domain>/jobs?run=<id>`, the swarm
UI's jobs tab, where the builds are a card under the job graph. Selecting
**Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record
stays at `GET /api/builder/runs/<id>`. A status has no link when the record wasn't
stored (the controller logs the failed write) or when the controller doesn't
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
status has no link.
## What the check is, and isn't
- It composes the config the way a hive's meta flake does
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
what a hive adds: its CA bundle, `claude-code` path, ports and
OTEL settings, and the `follows` that point the config's own inputs at the
hive's. A green check can therefore still fail on a hive, and the reverse.
- The agent base is the builder's host's, not the hive's. Hives on another
hyperhive or nixpkgs pin build something else.
- The check pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
can push to [a binary cache](#binary-cache).
- The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery of a PR push isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
## Archetype evaluation
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
`meta` of every agent archetype its own hyperhive source declares, with no
forge token, and answers with the JSON. An evaluation doesn't wait for a
build in progress and has five minutes; the controller waits six. The builder
logs each evaluation's time as `elapsed_ms`, and a timeout's error carries the last
lines nix printed. What the controller does with the answer: [`archetypes.md`](archetypes.md).
## Meta-flake lock
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
rendered `flake.nix`. The builder writes it into a scratch directory, runs
`nix flake lock` there with the same forge credential helper its builds use,
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
lock doesn't wait for a build in progress and has ten minutes. The builder
commits nothing.
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
with the `hive-meta-flake` renderer, all over `https`.
Each custom agent that a hive places pins to its `agent-configs/<agent>`
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
running rev (`services.hyperhive.deploy.hive-controller.hyperhiveRev`, from `self.rev`):
- the first commit pins hyperhive's forge `main`;
- later, the pin moves to the controller's rev only when the forge's compare
API shows that rev is a strict descendant of the committed pin: commits
ahead of it, and none the pin has that it lacks. The compare endpoint counts
histories with no common ancestor as zero commits ahead, so the check needs
both directions;
- an older controller, a rev on another line, a controller built from a dirty
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
`main` moves nothing.
The controller reads its rev from `/etc/hyperhive/swarm-controller.json` at each
render, not from its unit, so a deploy that changes no Rust doesn't restart it.
A missing file, or one with no rev, counts as a controller without one.
The controller commits `flake.nix` and the returned `flake.lock` to
`internal/meta` in one commit, and gives each hive's forge account read on the
repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run only after the
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
builds only from `internal/meta`, so after a failed update the scheduler skips
the deploy node, and the graph shows the update as failed and the deploy as
skipped. Each ten-minute retry of a failed update declares a fresh build and
deploy behind it for every agent whose latest deploy the failed update skipped,
so the run that succeeds sends them (see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.
The controller sends a deploy to every agent a hive places whenever the pin in
`internal/meta` differs from the pin it last sent, so they rebuild on the new
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
which only the controller reads or writes, and a `RecordFanout` job writes the
key once every deploy of the fan-out has finished or the deploy gate has skipped
it. The check runs after each update and once when the controller starts.
- The first commit sends a deploy, because the key holds no pin yet.
- A controller that starts with no key, as after the upgrade that added it,
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because its build failed.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
writes no key.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake the swarm builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. A green build here means the
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
this build produced and never builds one itself.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
| What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs |
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
On every ten-minute round the timer declares a build and deploy again for each
agent whose latest build failed, without a meta update when the latest update
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
still records the pin when the gate skips a deploy, so one failing agent doesn't
resend the healthy ones. A controller that updates before its builder host does
is the same case: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each round adds a build and deploy node per failing agent.
A merged config PR whose webhook delivery never arrives would leave the agent
on its old config. Every five minutes, after the config-PR scan, the controller
reads `main` of each `agent-configs/<agent>` repo that has a commit and that
exactly one hive places. It compares that head with the config commit pinned for
the agent in the `flake.nix` of the `internal/meta` commit its `agent-closures`
key names, since the pin in `internal/meta`'s own head also moves for other
agents' merges. When they differ, it declares the same update, build and deploy
a merge declares, and logs at info `config drift: …`. It leaves alone an agent
with no key, a key whose flake pins no config commit for the agent, and a head
the graph already holds a deploy for, in any state: the ten-minute rounds above
build a head whose build failed again, not this pass. The graph is
in memory, so a controller restart tries each such head once more.
When the build belongs to a merged config PR and `nix` exited with an error or hit
its deadline, the controller also comments once on that PR, with its forge
account: the agent, the `internal/meta` commit of the first red build, `nix`'s last lines (at most 4000
bytes), and that the deploy waits and the swarm builds it again every round. A comment
carries a marker line with the agent, and the controller posts only if the PR
holds no comment of its own with that marker; a copy of the marker in someone else's comment is neither counted nor edited. The retry rounds stay quiet whatever commit
they build, and the latest error stays on the Jobs page. A green build of the
agent for that PR rewrites the marker and appends the build's commit to the comment, so a later red comments again.
A `no verdict` failure, and a build that isn't a
merged PR's (a new agent, the pin fan-out), leave no comment. When the forge
refuses the comment, the controller logs a warning and the node ends as before.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
not updating.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key.
The record's `cached` field says whether the builder pushed the closure to
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
The built reply and the record also carry two options of the agent's evaluated
config, `private_users` (`services.hyperhive.agent.user.privateUsers`) and
`nix_builds` (`services.hyperhive.agent.nixBuilds`). The builder reads each with
a `nix eval` in its sandbox, and an eval has three outcomes:
- **A value:** the field holds it.
- **The module doesn't declare the option** (`nix` says the flake doesn't
provide the attribute): the field holds the option's default, off for
`private_users` and `host` for `nix_builds`.
- **Any other failure, or a timeout:** the value is unknown, and the reply and
the record leave the field out. hive-c0re then sends nothing for that option,
and hive-priv keeps what the container's last write chose, as it does for an
agent with no record.
Mixed versions:
| builder | controller and hive-c0re | what happens |
| ------- | ------------------------ | -------------------------------------------------------------------------------------------------------------- |
| new | new | an unread option leaves the field out, and the last write stands |
| old | new | the old builder always sends a value, which the record carries |
| new | old | the old side reads a missing field as the default, so an unread option behaves as it did before this behaviour |
A record or reply from before the fields reads as unknown.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
grant the forge refuses fails the build with no verdict.
## Binary cache
`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
each green meta build's closure to it, or to the cache you override them with,
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
not `attic watch`, and the builder pushes again when the controller runs a build again. The
push has ten minutes, or less when the reply is due sooner, because it must end
inside the 61 minutes the controller waits. `attic` checks the server's certificate
against the system roots and pins nothing, so a Let's Encrypt certificate
works.
Set both options to a name or both to null: evaluation refuses one without the
other. Null in both is how a swarm opts out of pushing.
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
starting with a letter or digit, and evaluation refuses any other name.
The push token is the swarm's own cache's to create, or the operator's to write for an
external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to
a file and an `attic` config naming that file (`token-file`) in the unit's private
runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push.
The token is in no argument or variable, so no process listing shows it.
The reply's `cached` says whether the push finished. A push that fails, times out
or can't read the token leaves the build green with `cached` false and logs a
warning with the reason; the deploy goes ahead and the next build of the agent
tries again. With no attic behind the defaults, every push fails this way. With
both options null, the builder pushes nothing and `cached` is always false.
Mixed versions:
| controller | builder | what happens |
| ---------- | ------- | ------------------------------------------------------------- |
| new | new | `cached` in the reply and in the `agent-closures` record |
| new | old | the reply has no `cached`, which reads as `false` |
| old | new | the controller ignores `cached`; its record has no such field |
A record written before the field existed also reads as `false`.
### Rebuild on a cache miss
The cache drops a closure nobody fetched for its retention period, but the
record in `agent-closures` stays. Every ten minutes `swarm-controller` checks
each agent a hive places: when the record is from `internal/meta`'s head and
says the builder pushed it, the controller fetches the narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token it reads from
`secret/swarm/services/attic/` (the same four leaves it
[copies to the hives](#what-hives-do-with-the-cache); with any missing, or a URL
that isn't `https`, the pass does nothing). A 404 is a miss, and the controller
sends the builder `build-meta` for that agent and the record's own commit. The
builder builds it and pushes as for any meta build. Any other answer or a failed
request isn't a miss, and rebuilds nothing. The next ordinary build handles a record
from an older commit, or one with `cached` false.
A pass sends at most three builds, one after the other. After an agent's rebuild
failed, got no answer or built without pushing, the pass skips it for 30 minutes, then
for twice as long after each further failure, up to 12 hours. The wait is in memory,
so a controller restart tries every agent again. The first pass runs ten
minutes after the controller starts.
The record keeps naming what hives deploy. A rebuild that lands on the record's
path writes nothing. A rebuild that lands on another path means the build isn't
reproducible: the controller logs a warning, and replaces the record only if it
still holds the same commit and no write came since the controller read it. The
record otherwise stays as it was.
The controller counts each rebuild in `hyperhive.builder.cache_miss_rebuilds` with
the label `outcome`: `rebuilt`, `path_changed` (the record now names the new path),
`path_changed_kept` (the record stayed), `not_pushed`, `failed` (`nix` failed) or
`unavailable` (the builder was busy, absent or gave no answer).
Mixed versions: a builder needs no change, since a meta build already names
its commit. An old controller does nothing about a miss, as before. A hive
sees nothing new.
### The swarm's own cache
`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic`
container behind the gateway, at `attic.<swarm domain>`. It defaults on where
you set `deploy.allSwarmServices` and the host holds the cache's store identity
(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write
no secret: on first start, two host units log in to the store as the cache's own
principal and create what's missing.
- `swarm-bao-attic-secret` creates the key atticd signs tokens with at
`secret/swarm/services/attic/jwt-secret` and writes it where the container
reads it.
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
the builder reads, at `secret/swarm/services/attic/push-token`, and a
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
It also stores the facts a hive pulls with: the cache's address at
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
`secret/swarm/services/attic/cache`, and the public key it signs with at
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
A unit creates each secret only while the store holds none, so a second start
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
write the new token to its path in the store, in its UI under the OIDC login:
nothing mints over a value that exists. A new signing key invalidates both tokens.
The unit creates the address, name and key once and never updates them. After
you change `swarm.attic.domain`, or after the server's database goes and the
cache gets a new key, delete the stale leaf and restart
`swarm-bao-attic-tokens`.
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any
host pushes to it.
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
path nobody added or fetched for 14 days goes at the next garbage collection,
which runs every 12 hours (`deploy.attic.retentionPeriod`). The controller
[rebuilds a closure the cache lost](#rebuild-on-a-cache-miss).
The container sits on the hive bridge in a network namespace of its own. atticd
listens on that namespace's loopback (`deploy.attic.port`), and a socket-activated
relay forwards `/run/swarm-attic/atticd.sock` to it, a directory only the
gateway's nginx and the container can open. The gateway terminates TLS with a
swarm services certificate and forwards to that socket with no login in front:
attic checks its own tokens, and a nix client can't follow a login redirect.
The two host units reach the cache through the gateway, at
`https://<swarm.attic.domain>`, like every other client; the host resolves the
name through the hive resolver, and where the gateway serves a self-signed
certificate `swarm-bao-attic-tokens` trusts the swarm's CA. The container logs
to its journal only; its journal doesn't reach the swarm log store.
### External cache
The cache can be an attic tenant on an instance outside the swarm. To point
the swarm at an existing tenant:
1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
instance's host name (no scheme, `host:port` if it isn't on 443) and
`services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
both options on every host that reads them.
2. Seed the push token in the swarm secret store at
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
command line. The token needs push rights on that cache. Until it
exists, every push fails and builds stay green with `cached` false.
Seed the hives' pull token the same way at
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
3. Seed the three facts hives pull with, each as field `value`:
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
cache's page or `attic cache info`). Until all four exist, hives report
`no-credentials`.
The builder checks the instance's certificate against the system roots and
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
reads the token again on every push, so a new token takes effect on the next
build.
To use an external cache, override the two defaults and don't set
`deploy.attic.enable` anywhere. Where you host the shared services with
`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to.
A host that has `deploy.attic.enable` on and `swarm.attic.domain` set to another
name gets an evaluation warning that names this fix.
## What hives do with the cache
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
leaves (`pull-token`, `url`, `cache`, `public-key`) to
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
own prefix. It writes a copy only when the value differs, skips a leaf the
store doesn't hold, and leaves a copy whose source disappeared.
`hive-c0re` reads its four leaves every five minutes under the hive's own store
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
token goes out in a header and in the netrc; a hive that finds another scheme
writes nothing.
`hive-priv` doesn't read those files. The key that must have signed a download and
the pull token come from `hive-priv-cache-facts`, a root unit that reads the
hive's own `attic/public-key` and `attic/pull-token` from the store under the
hive's own store identity every ten minutes and writes them to
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`,
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes
decides nothing `hive-priv` trusts.
The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
It checks that a signature by the public key covers the narinfo, in Rust, with
no help from the host's nix daemon. It downloads nothing. The result is a log
line (at info when it changes) and the gauge
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
latest closure, else `0` with a `reason` label. The first check that fails
decides the reason:
| `reason` | Meaning |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
| `no-record` | `agent-closures` holds no build for the agent. |
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
A failed read of the store or the queue leaves the last result in
place and raises a warning after three failed passes. A hive reports nothing
before its first pass.
Mixed versions:
| controller | hive | what happens |
| ---------- | ---- | ----------------------------------------------------------------------------------- |
| new | new | the copies exist and the hive reports per agent |
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
| new | old | the controller writes copies nobody reads |
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
The signature check follows Nix's fingerprint format but hasn't run against a
live attic. A `bad-signature` on every agent in the first pass after a deploy
means check attic's fingerprint format; it says nothing about an attacker. A
`fetch-failed` on every agent points at the URL, the token or the network.
### Rebuilds download the closure
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
`Swap` then does the same lookup and download, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. A stopped agent skips
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
same and creates the container from that path.
A rebuild or create has two outcomes, by what the lookup and the download find. The hive deploys the record's closure whatever `internal/meta` commit the builder built it from, and reads no forge head to choose one:
| Found | What happens |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| a record that the builder pushed | `MetaSync` reads the record once and pins its store path, so a record that changes during the rebuild doesn't change what `Swap` applies. The hive downloads the pinned closure and deploys it, then writes the store path to the agent's `applied/.<name>.closure` marker. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
The hive never falls back to a local build. The second row has one owner in the
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
an unreachable cache costs is one function.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
| old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
## Credentials
| What | Where | Who reads it |
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo, and on `internal/meta` through the meta update) and mints a
`read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
status with its own forge token. Details per path:
[`credentials-reference.md`](credentials-reference.md).
The builder exports build counts, durations and queue wait; see
[Swarm builder metrics](../scheduler/observability.md#swarm-builder-metrics).
## Isolation
The container has its own network namespace on the bridge, as the CI runner
does, so a config's evaluation can't reach a host-loopback service. Evaluation
is pure and runs in the container. Builds go to the host's nix daemon as an
untrusted client: a config can set no daemon option, and its derivations build
in the host's sandbox. A fixed-output derivation still has the network, as it
does for any build on that host.

289
docs/swarm/ca.md
View file

@ -1,289 +0,0 @@
# Swarm CA
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
through a role that belongs to that hive. A peer that trusts the mount's
root once validates every hive's gateway and every swarm service name,
present and future, instead of pinning to each one by hand.
That's the point of one issuer — it turns per-peer trust from O(n²)
hand-pinning into one anchor per swarm.
## The gateway leaf comes from the secret store
`hive-tls-resign.service` logs in to the store with this hive's own
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
the files the gateway copies from. The leaf's subject is the hive domain,
and its SANs are the hive domain and `*.<hive domain>`.
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
hive's domain and `*.<hive domain>`, read from
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and `read` on its own `attic/public-key`, `attic/pull-token` and
`squid-selfcheck`
(see [the cache facts for hive-priv](credentials-reference.md) and the
squid self-check password), and nothing
else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every
hive in `swarm.hives`.
**Every hive needs its own store identity.** It's a client certificate
from the store's own client CA, with the subject
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
`baoHiveIdentityKeyFile`:
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
- On every other host, copy `hive-leaf.pem` and `hive-leaf-key.pem` from
`/var/lib/swarm-bao-pki/hives/<hiveName>/` on the store's host, which
`swarm-bao-pki.service` mints for every other hive in `swarm.hives`, and
set both options, before the host's next deploy.
Without it, `hive-tls-resign.service` fails with a message naming both
options, and so does `swarm-services-cert.service` on a hive fronting no
swarm name. The unit checks the identity before the leaf, so it fails even while a
valid leaf keeps the gateway up. The identity is the hive's alone:
it issues this hive's leaf and fetches the store's root, and it's never the
identity a swarm service uses.
**When it issues.** The unit runs at boot, on every deploy, and from
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
the current one is missing, past half the role's 720-hour lifetime, missing
a configured name, doesn't match the key beside it, or doesn't chain to the
store's root (`swarm-services-root.pem`). The first run that reaches the
store replaces a leaf that doesn't chain to that root, whoever signed it.
**The gateway doesn't wait for the store.** Nothing orders the gateway after
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
current leaf in place, says so in the journal, and retries every 30 seconds
for a day. Only a hive with no leaf yet needs the store reachable and
unsealed: its gateway's certificate import fails until the first leaf
lands, and then the unit starts nginx.
nginx serves the leaf alone, without its issuer: the issuer is the
self-signed root a consumer already holds in its trust bundle.
**A gateway host that runs no hive** (`deploy.hive-controller.enable` off,
such as a dedicated swarm-services or secret-store host) has no hive leaf, no
`hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its
gateway serves the swarm-services leaf as its default certificate too. Where it
fronts other swarm names it waits for `swarm-services-cert.service` at start.
Where the store's UI is its only swarm name it starts on a throwaway
self-signed certificate and switches to the services leaf once the store
issues it.
## The trust bundle
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
the file every consumer anchors on. It holds one certificate, a copy of
`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
the swarm-services leaf both chain to. It's the only root the swarm has.
**Without the store's root, the bundle is empty and its readers go red until
it lands.**
While `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem` is
missing or empty, as on a first boot before the store answers,
`hive-tls-ca.service` writes an empty bundle and logs that file's path. Every
`<name>-ca-bundle` unit then fails with `<bundle> contributed no certificate
to the bundle`, and the consumers that `Requires=` it don't start.
`swarm-services-cert.service` reloads `hive-tls-ca.service` when it writes a
new root. That rewrites the bundle in place and restarts no container, the
store's included.
**Each `<name>-ca-bundle` unit converges on its own.** Beside it, a
`<name>-ca-bundle-refresh.path` unit watches the file the bundle derives
from: inside a container the bound `/run/hive-ca/trust-bundle.pem`, on the host
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`. A write to it starts
`<name>-ca-bundle-refresh.service`. That service does nothing while the file
holds no certificate, or while it matches the copy the last assembly took
(`/run/<name>-ca/anchor.pem`). Otherwise it restarts the bundle unit and starts
its consumers without waiting. The restart reaches the running consumers
through their `Requires=`; the start picks up those a first boot left failed.
The same service runs once at boot, after the bundle unit, which covers a root
that landed before the watch began. A first boot needs no second apply, and a
root rotation restarts the consumers once and the container never. The watch
fires for the host's in-place rewrite because inotify keys on the inode and a
bind mount shares it, so `hive-tls-ca` must keep rewriting the file in place
and never rename over it.
A deploy that changes `hive-tls-ca.service` reloads it too, with the same
effect.
⚠️ **A consumer outside `trustBundle` reads its trust when it starts.** A
container whose nix predates the refresh units keeps the bundle it started
with until it restarts, and fails TLS to a gateway serving a store-issued leaf
in the meantime.
## The swarm's service names come from the secret store
The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
**single-label** wildcard over it, and its role allows nothing outside the hive's
domain, so it can't cover them.
The secret store's `pki` mount issues them a leaf of their own, through its
`swarm-services` role. `swarm-services-cert.service` logs in with
the services-issuer identity, calls
`pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
copies from. The store's role, not a key on the host, decides which names a
gateway can serve.
**Every self-signed gateway that fronts a swarm name needs the
services-issuer identity**, whether or not a hive runs on its host. A hive
fronting none needs no copy of it: it fetches the store's root with its own
store identity, as below. It's a client certificate
from the store's own client CA, with the subject
`deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default),
and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile`
and `servicesIssuerClientKeyFile`. Its policy may issue from
`pki/issue/swarm-services` and nothing else, and it's the same certificate on
every gateway host:
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to
it.
- On every other gateway host fronting a swarm name, copy
`services-issuer.pem` and `services-issuer-key.pem` from the store's host
and set both options.
A host fronting a swarm name without both options set still evaluates and deploys, and `swarm-services-cert.service` fails, naming
them. One that names files it can't read fails the same unit, naming the
paths. The old spellings,
`deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are
renames of these two: a config that still sets them evaluates with a warning
naming both paths.
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
same list on every host, populates the role's
`allowed_domains`, so asking for a name nobody configured is a refusal
from the store naming that name — not a certificate quietly issued for
it. A host's own SANs are narrower still: it asks only for the names it
fronts a vhost for (`deploy.hive-controller.tls.localServiceDomains`), so the key on one
gateway can't serve a swarm service that runs behind another.
**The root's public certificate is a file, on every self-signed gateway host:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
swarm name asks for no services leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its own store identity
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
`baoHiveIdentityKeyFile`) for the handshake without logging in with it.
Every hive must hold one: without it, or with either file absent,
`swarm-services-cert.service` fails with a message naming both options.
On the host running
the store it's also at
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
hand a browser, and reading it needs no store login — which matters,
because every store listener but the loopback UI one demands a client
certificate, and authelia gates that one to the `admins` group, not
open to an anonymous browser fetching a trust anchor.
**The granting unit generates the root once, and never again.** It asks
the mount whether it already has an issuer (`bao list pki/issuers`)
before calling `pki/root/generate/internal`, so a rebuild or a reboot
re-asserts the role and the grant without touching the anchor. A root
that changed per boot would invalidate every certificate issued under it
and every browser taught to trust it.
**A daily timer renews the leaf.** The store's role issues it for 720
hours. `swarm-services-cert-renew.timer` runs the same script as
`swarm-services-cert.service` once a day, and that script asks the store
for a new leaf once the current one is past half that window, is
missing a configured name, or doesn't chain to the store's root
(`swarm-services-root.pem`). A rotation re-imports the leaf into the
gateway and reloads nginx. A store that can't answer — still sealed, or
out of reach — fails the unit, which leaves the current leaf in place,
and the next day's run tries again: about fifteen tries before the leaf
lapses. `systemctl
list-timers swarm-services-cert-renew` shows the next run.
## Constraints on the material
No CA private key exists on any hive. The `pki` root's key stays inside the
store's mount (`root/generate/internal`), and each hive's narrowing is its
role's `allowed_domains`, enforced by the store at issue time: a hive's
identity that leaks can get certificates for that hive's names and no
others.
No host holds a CA or its key, except the store's mTLS CA
(`glue-bao-tls.nix`), which signs only the store's server certificate and the
certificates that log in to it. Every host deletes `/var/lib/swarm-ca` and
the `ca.pem`, `ca-previous.pem`, `ca-key.pem`, `ca.srl` and
`.swarm-ca-adopted` in
`<deploy.hive-controller.tls.stateDir>`: `systemd-tmpfiles` removes them at
boot, and on a deploy that changes the host's `tmpfiles.d` rules. The swarm
distributes only certificates.
⚠️ **The deletion is irreversible.** Rolling back to a generation that still
has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a
new one, unrelated to the deleted root. That's harmless: nothing issues
under it.
A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
evaluation, however it's spelled with `//`, `.`, `..` or a trailing `/`. A
symlink to one isn't resolved, so it gets past the check. Those options name
the store's mTLS CA, `/var/lib/swarm-bao-pki/ca.pem` on the host that runs
the store.
## What to hand a peer
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
block a peer operator pastes into their own config. When this hive's
gateway serves a self-signed leaf (detected by the presence of
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints the
secret store's `pki` root itself, read from
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
gateway leaf and the swarm's service names under that root, so it's the one
anchor a peer host needs.
The operator installs the root **once per swarm host**, not once per peer,
so a hive joining later needs no edit on the hives already running. A hive
whose cert already chains to a public CA has nothing to install —
`peer-config` says so instead.
## Distributing the root
The root **certificate** is a runtime file: each hive fetches it from the
store at boot, so nothing can name it at build time. That has a cost worth
naming, because it's subtle and it bites at a distance:
> **Nothing whose trust store nix assembles at build time can reference
> the store's root.** The derivation reads `security.pki.certificateFiles`
> inside itself; the root doesn't exist there.
Two consumers, both reading the root at runtime:
- **Agents fetch it themselves.** `hive-agent-trust-bundle.service` (`nix/agent-modules/hive-ca.nix`) reads `<mount>/ca/pem` from the store with the agent's own store certificate, then writes `/run/hive-agent-trust-bundle/bundle.pem` (system CAs plus the root) and `root.pem` (the root alone). The agent's units point `SSL_CERT_FILE` at the first (Rust, Go and OpenSSL clients) and `NODE_EXTRA_CA_CERTS` at the second (Node and Claude Code), and order after the unit with `Requires=`. The fetch retries inside its start job for up to 14 minutes, with a wait that doubles from 2 to 60 seconds, so a store name that doesn't resolve yet delays those units instead of failing them; after that the job fails and they fail with it. The meta flake names no CA file, and a timer re-reads the root hourly. A consumer reads it at start, so a rotated root reaches an agent unit at its next start.
- **The Matrix container gets a bind mount**, via the shared
`lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
use: the bundle is bind-mounted read-only into the container, and a
oneshot concatenates it with the system CAs before tuwunel starts.
The consumption differs per runtime and is the part worth knowing.
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
— it's not: its rustls-based TLS stack still resolves trust through the
platform certificate store on Linux, and that store honors
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
an OpenSSL-linked binary.
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
> `openssl-probe` uses it _instead of_ the default store, so naming
> just the bundle would drop every public CA and break federation with
> the wider matrix network — a much bigger outage than the one being
> fixed. The same caveat applies to `hive-forge` (Go) for the same
> reason.
A hive in the swarm fetches the root itself. A host outside it gets the
certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)).
One anchor per host, not one per peer: a hive joining later needs no edit
on the hives already running.

103
docs/swarm/ci-metrics.md
View file

@ -1,103 +0,0 @@
# CI metrics
`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
can show run time and queue wait per repo, workflow and PR, over any range back
to a repo's first run.
## Where it runs
A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
has the poller's store identity, which the host that runs the store has. Run it
on one host per swarm: each poller keeps its own cursor, so two of them push
every run twice.
It reaches the forge, authelia, the store and the collector by name, so it needs
none of them on its host. On a host without the store, its leaf
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
file to copy, as for the CI runner.
With a self-signed gateway, those names serve leaves the swarm's own CA issued.
The poller reads them through `swarm-ci-metrics-ca-bundle.service`, which joins
the system CAs and the host's `trust-bundle.pem` into
`/run/swarm-ci-metrics-ca/trust-bundle.pem` and sets that file as the poller's
`SSL_CERT_FILE` ([CA](ca.md#the-trust-bundle)). Without it, every pass fails
with `invalid peer certificate: UnknownIssuer`.
## What it reads, and as whom
| what | where | identity |
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| its forge token | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass | the same leaf |
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account `ci-metrics`, with that token |
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
swarm-controller creates the `ci-metrics` account and mints its token with the
scope `read:repository`, the way it does for the builder. It also keeps a
`ci-metrics` team in every organisation, with read on the Actions unit of every
repo in it, and the account as its one member. The poller therefore covers:
- every public repo, which needs no grant;
- every private repo an organisation owns, through that team;
- **no** private repo a user owns. The forge grants a user's repo only to a
collaborator, and a collaborator reads every unit, code included.
## What it pushes
Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
forwards them to the metrics store. Each sample carries the run's `stopped`
time as its timestamp, which the store keeps, so a run from last month lands
last month.
| metric | value, in seconds |
| ----------------------------- | ------------------- |
| `hyperhive.ci.run.duration` | `stopped − started` |
| `hyperhive.ci.run.queue_wait` | `started − created` |
Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
them, every run is a series of its own, and the board reads each with
`last_over_time`.
A run becomes no sample when its status is `cancelled` or `skipped`, it has no
start (or the epoch as its start, which the forge writes for a run cancelled
before it started), or it stops no later than it starts. A run still going a day
after its `created` time counts as abandoned. The journal line for each repo counts
these by reason.
## The cursor and the backfill
The poller keeps one file, `cursor.json`, in its state directory
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
poller has pushed or skipped every run, and the run numbers above it already
handled. A run still going holds that number, so a run that finishes after a
newer one still gets pushed.
With no file, the cursor of each repo is at run 0, so the first pass is the
backfill: it walks every run back to the first, a page of 50 a second. For a
repo with 5000 runs that takes a few minutes.
The poller writes the file only after the collector accepted a push, never
before. A crash between the two pushes those runs again on the next pass. The
store then holds the same point twice in the same series, which changes no
panel. Deleting the file re-runs the backfill the same way.
A failed repo leaves its cursor where it was, the pass goes on with the next
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
timer runs it again 5 minutes later.
## The board
The **CI runs** tab of the `forge` board (`nix/host-modules/swarm-grafana/dashboards/forge.json`), with a repo and a workflow
variable:
- mean and 90th percentile run time of successful runs, over time;
- queue wait over time, mean and longest;
- finished runs per day, by outcome;
- a table per PR over the time range: mean and longest successful run time,
runs, failed runs.

104
docs/swarm/claude-login.md
View file

@ -1,104 +0,0 @@
# Claude login from the swarm UI
An operator logs an agent in to claude from the agent's card in the swarm UI.
No hive is in the path: the login runs in the swarm's `swarm-claude-login`
service, the credentials go to the agent's own leaf in the secret store, and the
agent's harness installs them from there.
Where it runs and how it's enabled: [`services.md`](services.md#claude-login-minter).
The service runs the `claude-code` that its host's
`services.hyperhive.deploy.hive-controller.claudeCodePackage`
pins for agents, and its container's own nixpkgs' where the host pins none.
## What happens on a login
1. The agent publishes `needs_login` in its turn-state header, and its card
shows **login needed** with a **log in** button.
2. **log in** opens a dialog, which calls
`POST /api/agents/<agent>/claude-login`. swarm-controller relays the start as
a request on the queue subject `$SWARM.claude-login.request`.
3. The minter runs `claude auth login` with `HOME` and `CLAUDE_CONFIG_DIR` in a
fresh scratch directory, over plain stdio pipes, and answers with the first
`https://` URL it prints. The dialog shows that URL.
4. The operator opens the URL, signs in to the agent's own claude account, and
pastes the code the page shows into the dialog, which posts it to
`POST /api/agents/<agent>/claude-login/code`.
5. The minter writes the code to claude's stdin and waits for it to exit. It
stores the `.credentials.json` claude wrote at
`swarm/agents/<agent>/claude-credentials` (field `value`), as a new version
over any stored before, and removes the scratch directory.
6. The agent's harness, while it waits for a login, reads that leaf every 30
seconds under the agent's own certificate. When the stored version is newer
than the one it installed last, it writes the file to
`~/.claude/.credentials.json`, records the version, and resumes its turn
loop as it does after a login from the agent's web UI.
A started login waits 15 minutes for its code, then the minter stops it.
Starting another login for the same agent replaces the running one. Each step
has a minute to finish; a refused code fails the login, and the dialog shows
what claude printed.
## Why the harness takes only a newer version
`claude` rotates the refresh token in `.credentials.json` as it uses the
session, and the agent can't write the store. After the first refresh the
stored copy is stale. A harness that installed it whenever the bytes differed
would replace the agent's live session with a stale copy and log the agent
out. The harness therefore compares the store's KV version with the last
version it installed (`claude-credentials-version` under the harness dir):
- it installs a newer version, that is, a new login;
- it leaves an equal or older version alone, even when the agent needs a
login again, so it never puts back a session that has gone bad;
- it reads nothing while the agent has a session.
## Credentials
| Principal | Grant |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `swarm-claude-login`, in its container | `create`, `update` on `swarm/agents/+/claude-credentials`; `read` on `swarm/services/<minter client id>/oidc/client` |
| the agent, under its own certificate | `read` on its own `swarm/agents/<agent>/*`, which covers the leaf |
The minter has no read on the leaf and nothing else under an agent. Every
hive's policy reads `swarm/agents/*`, so every hive can read every agent's
claude session, as it can read the GitHub token.
The `operatorsOnly` authelia rule on the swarm domain, not the controller,
keeps these routes to operators: the controller checks only that the gateway
named a user. Without that rule any account authelia admits could store its
own claude session as any agent's, so no agent account may exist before the
rule reaches the swarm.
The minter refuses a step whose agent name isn't an identifier before it
makes a directory or runs `claude`.
## Caveat
The installed-version record lives under the harness dir and the credentials
file under `~/.claude`. If only the record survives a rebuild, the agent
ignores the stored version until the operator logs it in again, which writes a
newer one.
## What stays
The agent's own web UI login (`/login`) still works, unchanged. It goes away
once the operator has confirmed the swarm login works.
## Observability
The minter pushes OTLP metrics through its container's forwarder to the swarm
collector, labelled `service.name=hyperhive-claude-login`. With no
`OTEL_EXPORTER_OTLP_ENDPOINT` it exports nothing. The client-certificate
gauges (`bao.client_cert.*`) come with them.
| Metric | Unit | Kind | Attributes |
| ----------------------------------------- | ---- | --------- | --------------------------------------------------------------------------------------------------- |
| `hyperhive.claude_login.steps` | — | counter | `step`: `start`, `code`, `cancel`, `invalid`; `outcome`: `started`, `stored`, `cancelled`, `failed` |
| `hyperhive.claude_login.step.duration` | `s` | histogram | `step` |
| `hyperhive.claude_login.sessions.expired` | — | counter | — |
| `hyperhive.claude_login.sessions.active` | — | gauge | logins waiting for a code, as of the last step |
`invalid` is a request that didn't decode or named no agent. A `start` spawns
`claude auth login` and a `code` stores the credentials, so those two are the
slow steps. `expired` counts logins stopped because their code never came.

189
docs/swarm/credentials-reference.md
View file

@ -1,189 +0,0 @@
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
# Credentials: the per-secret reference
Every credential in the swarm's secret store, in one table: who mints it, who
reads it, and whether it renews on its own. The overview and the rules a
credential follows are in [`credentials.md`](credentials.md); this page is
the detail, so it changes whenever someone adds a credential or moves its minting.
The second half, [where each secret lives on disk](#where-each-secret-lives-on-disk), maps the
files that exist today.
In the table, **automatic re-mint** is something replacing the stored value
without an operator. **Automatic re-pull** is the reader picking up a replaced
value without a restart. ✅ means it happens today, ❌ means it doesn't, and
the rest of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice render` in `hive-matrix`, when the store answers and holds none; before that, the tokens an earlier version kept in the container, stored unchanged | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` render the new pair and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read once; a store down at start is re-read in the background until it answers |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation, and by the renewal pass when none is stored or its cert-auth role is missing | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity; minted when none is stored or its cert-auth role is missing, for agents not `Destroyed` | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/observability` | `swarm-controller`, when an operator sets, clears or creates an agent's permissions with `logs` or `metrics` in the old or new list, and when it destroys the agent; the leaf stays absent for an agent that never held either | the agent (read-only, its own prefix); `swarm-logs-auth` and `swarm-metrics-agent-auth`, which deny when it's absent or unreadable | ✅ the controller rewrites it with the permission, and every five minutes makes it equal the record's | ✅ the stores honour a changed grant for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/gitlab/<label>` | `swarm-controller`, when you link a GitLab account through the API | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes; a deleted account leaves the agent's `glab` configuration on the next run |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/backend-env` | `hive-agent-backend-env`, in the agent's container, from the agent's `backendEnvironmentFile` when the leaf holds nothing (field `env`, the file's exact text); or `swarm-controller`, when you set the API-key environment in the swarm UI | the agent | ❌ nothing replaces it without you; a save in the swarm UI overwrites it, including text an agent wrote into an empty leaf; the agent's file stays as it was | ❌ the file in `/run` refreshes every 2 minutes, and the harness and the subagent daemon load it at their next start |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/agents/<agent>/proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds |
| `swarm/hives/<hive>/squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute, and while they can't reach the store they keep accepting the value they last read, for an hour at most; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, to copy it and to check narinfos | ❌ write a new token | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ❌ as the URL. A cache recreated with a new key leaves the stored one stale: every hive then reports `bad-signature` | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->
## Operating notes
The name `main` belongs to the swarm's own homeserver account, which the
controller mints, so the controller rejects any account you link under it.
`swarm-controller` checks every forge token itself: it looks at every agent
that has a store identity at start and every five minutes. For any whose
stored token is missing or stale it creates the forge user if there isn't one,
then mints the token. To check one agent now:
```sh
swarmctl agent mint-forge-token <agent>
```
An agent without a store identity gets no swarm token and keeps using the
`forge-token` file in its state dir, if it has one; re-create it with
`swarmctl agent create` to give it one. The swarm leaves an agent it already
declares as it is. For an agent with no declaration, such as one only its hive
knows about, `agent create` declares it `paused`; set it `up` from its card in
the swarm UI.
⚠️ **Re-create existing agents before deploying a hive-side change that makes a
container require a credential it may not have.** A container whose store
identity is absent doesn't start, so create the identities first, setting any
newly paused agent `up` as above.
**Revoking an agent's queue credential.** Declaring an agent `destroyed` —
`PUT /api/hives/{hive}/agents/{agent}/state` — deletes
`swarm/agents/<agent>/queue` and every version it held. An open queue
connection closes within an hour. An agent recreated under the same name gets a
fresh secret. It leaves the agent's mTLS leaf alone. If the store refuses the
delete, `swarm-controller` logs an `error` naming the agent and the teardown
continues; declare the agent destroyed again to re-run the revocation.
A failed store login retries every 30 seconds for a day. When a later attempt
lands a changed value, the service that loaded it restarts. If the fetch unit
can't compare file times it logs `cannot compare the mtime of <path> … —
leaving <unit> as it is` at `err`, and the service keeps the old value until
the next timer tick.
## Where each secret lives on disk
One row per secret, by the place that owns it: who generates it, where the file
or store path is, and what an operator of a hive elsewhere places. How the
pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.md).
### Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store is down, refuses the login, or fails the read of a leaf (a refusal, a timeout, an error), the unit runs on the files, logs the store's error, and writes nothing to the store; the unit seeds or mints only a leaf the store reports as absent. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting |
| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services/<grafana client id>/secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk, unused once the store holds the key. A store that's down or doesn't grant the leaf fails the unit, and Grafana doesn't start | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it |
| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 until the store answers: the controller re-reads it in the background, at most five minutes apart, and registers the forge hooks once it has the secret |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once. An attempt that fails or runs past 30 seconds retries after 5 seconds, then at growing intervals up to 5 minutes | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the fetch fails fast while the store is down and retries with a delay that grows to 5 minutes; the attempt that succeeds starts the collector, which isn't part of the container's boot, and it resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these leaves has
a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the cache's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, `swarm-bao-grafana-secret-key`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` and `swarm/services/<grafana client id>/secret-key` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services/<runner forwarder client id>/oidc/client` |
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The option names are `deploy.bao.<pair>`; the common names, what a missing leaf
costs and how to place a leaf by hand are in
[per-principal identities](credentials.md#per-principal-identities).

639
docs/swarm/credentials.md
View file

@ -1,639 +0,0 @@
# Credentials: the target shape
<!-- vale write-good.Passive = NO -->
The swarm's credential store is bao. This page is the overview of how the swarm
treats credentials; [`credentials-reference.md`](credentials-reference.md) lists
each one: who mints it, who reads it, and how it renews, with the notes for
operating them. It describes the target, not what's on disk today; the second half of the
reference maps [the files that exist right now](credentials-reference.md#where-each-secret-lives-on-disk).
<!-- vale write-good.Passive = YES -->
**Store secrets stay off disk.** A process pulls a value from the store at
runtime and holds it in memory only.
**The mTLS client certificate is the only credential on disk.** Each
identity (an agent, a hive, a swarm-level service) has one. It's what logs a
principal in to the store, so you place a hive's or service's certificate by
hand. The nix option for it takes a path to the certificate, never its bytes,
because a literal in a nix expression lands in the world-readable nix store.
A hive hands each agent its certificate; the agent pulls everything else
itself.
**Each credential has a minter, a reader and a renewal story.** Who mints it, who
reads it, and whether it renews without an operator are what the
[reference table](credentials-reference.md) gives for every credential, with the
operating notes that go with them.
## Where a secret lives
A swarm generates its credentials in three different places and reads them
in a fourth, so "where does this file go" has a different answer per deployment.
This page is that answer, one row per secret.
Two rules run through all of it.
**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.
The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
the queue callout's user key, reaches every client that connects,
so the store is a perfectly good place for it.
**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.
### Topologies, by who places secrets
Read every row of the [reference tables](credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level secrets
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
test worth applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
each with the secret publisher's identity and writes it to the file authelia loads. A store
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
database, and a new issuer key invalidates every token issued. A container without the
publisher's identity generates the files as before.
The same unit mints the client secrets, straight into the store. It reads
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from
`swarm/services/swarm-authelia-bridge/oidc/client` into
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
answer keeps the bridge down while it retries. A container without the publisher's
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.
The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
### The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Hive-level secrets
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
mint in authelia's container has stored the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the mint in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.
All agents share one policy, `hive-agent`, which the agent's cert-auth role
attaches and no other. It states the rule with `{{identity.entity.name}}`
where an agent's name would go, so the rule is one document and a change to it
is one write. `swarm-controller` writes it at agent creation and at its own
start.
For that the controller names each agent's identity entity after the agent: at
each certificate mint or renewal, before it writes the agent's cert-auth role,
and for every agent at its own start. It reads the cert mount's accessor and
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
entity a first login created under a generated name. At a mint or renewal this is
a precondition: when the store refuses a call, or the alias sits on any other
entity, the mint fails and the role stays as it was, so a live agent keeps
working until an operator resolves it. The start pass only logs the same
failures, because it writes no role. The controller needs the `identity/` and
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.
A cert-auth role that predates the shared policy may still name a policy
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
objects stay in the store and keep granting what they did, to a token that
already carries them and to a role not yet rewritten. A live agent's role is
rewritten at its next certificate renewal.
A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and rewrites the roles to attach it. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. The leaves, their option names
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token.
The other roles issue service tokens that live an hour (`token_ttl=1h` and
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
agent and hive roles the controller writes through
`SecretStore::write_cert_role`, which takes the same value from
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
expires, and the mount default of 32 days leaves one lease per login for a
month; at the 17 thousand logins a day this swarm made, that came to half a
million leases. A reader logs in per operation and holds the token for seconds.
The longest hold is the controller's forge work, which stops after two minutes.
A client that keeps a `SecretStore` handle longer than an hour must log in
again, because nothing renews the token. A role the controller writes takes the
TTL at its next write: a hive's at the controller's start, an agent's at its
next certificate mint or renewal. At its start the controller also reads every
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
differs from the TTL, writes those two fields and no others. The store keeps the
fields a write leaves out, so the role's certificate, allowed names and
policies stay. A role already at the TTL isn't written. The start logs one line
with `rewritten`, `already_current` and `failed` counts, and a role that fails
gets a warning that names it, then the pass moves on. Tokens issued before
expire on their own; the controller revokes nothing. At that rate an hour keeps
about 700 tokens live.
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
it, and a deployed store never rewrites it, so the longer TTL would not reach
it; the granter logs in and finishes within seconds.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
### OIDC login
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
`url`, `cache` and `public-key`). That shows the key tree and each key's
versions and timestamps, and no value: the store refuses every `secret/data/`
read. The writes are the leaves the operator fills in by hand: the review
classifier's keys, and the cache's tokens and facts, which configure an external
cache or replace the in-swarm cache's own. The cache's host units create those
leaves only while the store holds none, so a saved value survives their next run
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
Anything more needs a token.
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
`services` > `attic`, pick the leaf and choose **Create new version**. The
form starts empty because the login can't read the current value: type the
key `value` and the new value, and save. A new version replaces the whole
secret, so `value` is the only key to enter. `bao kv put` with the same login
works too.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.
## Adding a secret
State three things, in the row you add to the [reference](credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.

133
docs/swarm/inference.md
View file

@ -1,133 +0,0 @@
# Per-agent inference settings
The swarm can set the model and reasoning effort each agent's turns run on,
plus the models it may start subagents on and how many subagents it may run
at once. A field you set wins over the
agent's own value; a field you leave unset keeps the agent on its own value,
so an agent you set nothing for runs as it does today.
## Setting them
```sh
swarmctl agent inference set iris --model opus --effort high
swarmctl agent inference get iris
swarmctl agent inference clear iris
```
- `set` changes only the flags you pass; the other fields keep what's
stored. A hive must declare the agent.
- `--effort` is one of `low`, `medium`, `high`, `xhigh`, `max`.
- `--subagent-models` takes a comma-separated list. It replaces the
agent's `availableModels` as the models its subagent `start` and
`continue` calls may name, read on each call
([`subagent.md`](../tools/subagent.md#which-models-a-subagent-may-run-on)).
- You set the subagent cap, `max_subagents`, in the swarm UI's inference
dialog; `set` has no flag for it and keeps the stored one. It's a whole
number from 0 to 32, where 0 allows no subagents; unset means no cap
([`subagent.md`](../tools/subagent.md#how-many-subagents-may-run-at-once)).
- `get` shows each field, naming the unset ones (or says the swarm sets
nothing for the agent), and then the model and effort the agent reports
running on, each with where it comes from.
- `clear` removes every field; the agent goes back to its own values.
The full flag reference is [`swarmctl-cli.md`](../tools/swarmctl-cli.md). The
swarm UI's agents page edits the same settings; see [UI](ui.md). The
swarm is the only place to change an agent's model or effort at runtime:
its web UI shows both but has no picker and no `/model` or `/effort`
command.
## What an agent runs on
`get` shows what the agent runs on next to what the swarm sets:
```text
model: opus
effort: (not set; the agent's own)
subagent_models: (not set; the agent's own)
max_subagents: (not set; no cap)
running model: opus (swarm)
running effort: medium (local_file)
```
Each running value names where it comes from:
| source | meaning |
| ------------ | -------------------------------------------------------------------------- |
| `swarm` | the value set here |
| `agent_nix` | `services.hyperhive.agent.model` or `effortLevel` in the agent's agent.nix |
| `local_file` | a `hyperhive-model` or `hyperhive-effort` file in the agent's harness dir |
| `default` | the harness's built-in default (`haiku`, `medium`) |
When the agent starts without a swarm value, its agent.nix model wins over
a `hyperhive-model` file, and a `hyperhive-effort` file wins over its
agent.nix effort. These files come from runtime picks the agent's web UI no
longer offers; the agent still reads them but never writes them, so a
`local_file` value stays until the swarm sets that field or someone deletes
the file.
The agent reports these whenever they change, and again once a minute.
swarm-controller keeps only what it has received since it started, so for up
to a minute after a controller restart `get` shows `running: not reported…`.
## When an agent picks a change up
Each agent reads its settings every 10 seconds and switches on the next
read: its next turn runs on the new model or effort, and its web UI badge
shows it. A turn already running keeps what it started with.
- **Its own value is the one it runs without the swarm**: its agent.nix or a
`hyperhive-model` / `hyperhive-effort` file. Setting a swarm value doesn't
overwrite it.
- **Clearing a field** (or the whole key) puts the agent back on its own
value on the next read.
- **After a restart**, the agent runs its own values until its first read
of its settings succeeds.
<details><summary>Implementation</summary>
The settings live in the `agent-inference` KV bucket, one key per agent,
which swarm-controller alone writes:
```json
{
"model": "opus",
"effort": "high",
"subagent_models": ["haiku", "sonnet"],
"max_subagents": 3
}
```
Every field is optional. The agent runs the same with `{}` as with no key;
`GET` answers `"configured": null` for no key (or a cleared one) and `{}` for
a stored value with no fields. A stored value the controller can't decode,
such as one a newer build wrote, also answers `"configured": null`, with a
warning in its log, until the next `PUT` replaces it.
| method | path | body |
| -------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GET` | `/api/agents/<agent>/inference` | —; answers `{"configured": <the stored value, or null>, "effective": <what the agent reported, or null>}` |
| `PUT` | `/api/agents/<agent>/inference` | the whole value, replacing the stored one; the controller refuses unknown fields, an unknown effort, or a model name with whitespace or a leading `-`, or a `max_subagents` above 32 |
| `DELETE` | `/api/agents/<agent>/inference` | —; answers 204 |
An agent's own queue credential grants it
`$JS.API.DIRECT.GET.KV_agent-inference.$KV.agent-inference.<agent>`, the
direct read of its own key, and nothing else in this bucket — the same shape
as its schedule key ([`schedules.md`](schedules.md)).
The harness (`hive-agent`'s `swarm_inference`) applies a set field to the
in-memory model or effort without writing `hyperhive-model` or
`hyperhive-effort`, which keep the agent's own value.
The subagent daemon (`hive-subagent-mcp`'s `swarm_models`) reads
`subagent_models` and `max_subagents` with the same direct get, under the
same credential, on each `start` or `continue`.
`effective` comes from the agent's turn-state header on
`$SWARM.agent-state.<agent>` ([`README.md`](README.md)), which carries
`model`, `effort`, `model_source` and `effort_source`. swarm-controller
subscribes to every agent's header with the reader grant it already holds for
the state relay and keeps the latest per agent in memory. A header without
those fields, from an agent that doesn't report them, leaves `effective`
`null`.
</details>

317
docs/swarm/notify-router.md
View file

@ -1,317 +0,0 @@
# Notification router
The swarm's `swarm-notify-router` service turns forge events into todos on the
agents they concern. No hive is in the path, and the agent can be on any hive:
swarm-controller keeps the events, the router decides, and each agent's harness
takes its own todos off the queue.
Where it runs and how it's enabled: [`services.md`](services.md#notification-router).
It runs beside `hive-forge-notify`, which still delivers forge notifications
the way it did. An agent can get the same news from both.
## What an agent gets
| Flow | Event | The agent's todo |
| ------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `assigned` | someone assigns an agent to an issue or PR | `you are assigned to issue <repo>#<n>: <title> (<url>)`, key `forge:<repo>#<n>:assignment` |
| `unassigned` | someone takes an agent off an issue or PR | the same key rewritten: `you are not assigned to issue <repo>#<n> anymore: <title> (<url>)` |
| `closed` | someone closes or merges the issue or PR | the router clears every router todo it queued for it, then queues `closed` on each agent that held one: `<kind> <repo>#<n> was closed: <title> (<url>)`, key `forge:<repo>#<n>:closed` |
The todos are subsystem `router` in the agent's todo list. A todo under the
same key replaces the one before, so an issue gives an agent one assignment
todo however often it changes hands. A new event reopens a todo the agent
marked done.
The `closed` todo has its own key, apart from the assignment todo. A reopened
issue leaves it in place until the next close replaces it; a new assignment
writes the `assignment` key and leaves it alone.
Only agents get todos. An assignee with no agent of that name gets nothing.
An agent assigned before it first reports its status gets its todo on the
issue's next assign event.
Forgejo's assign events carry the assignees after the change, not which one
changed. The router keeps the assignees it saw on the last assign event per
issue and takes the difference. With no earlier assign event for an issue,
every current assignee counts as newly assigned and nobody as removed: an
router notices an unassign only on an issue it has seen an assign event for.
## The path
1. swarm-controller registers two instance-wide hooks, one for the `issues`
and `issue_assign` events at `/webhook/forge/issues` and one for
`pull_request`, `pull_request_assign`, `pull_request_sync` (a push to an
open PR's branch) and the submitted-review events `pull_request_review_approved`,
`pull_request_review_rejected` and `pull_request_review_comment` at `/webhook/forge/pull-request`. At each start it adds
any of these events a registered hook lacks.
2. On each delivery the controller checks the HMAC and publishes the body,
unchanged, to the `SWARM_EVENTS` stream on
`$SWARM.events.forge.<X-Forgejo-Event-Type>`, such as
`$SWARM.events.forge.issue_assign`. The stream keeps a delivery 7 days and
drops a redelivery by its `X-Forgejo-Delivery` id. A delivery the stream did
not keep answers 503, so the forge's delivery log shows it.
3. The router drains its durable consumer, `notify-router`, in order. For each
event it runs every flow, queues the todo ops the flows emit, writes the
issue's snapshot to its `notify-router-state` bucket, then acks the event. A
failed step stops the batch: that event and the rest come back in stream
order after 30 seconds, and the stream drops the ops queued twice by their
message id.
4. Each op goes to the `ROUTER_TODOS` stream on the agent's own subject,
`$SWARM.router.todo.<agent>`. Work-queue retention: an op stays until its
agent acks it, up to 30 days and 1000 per agent.
5. The agent's harness creates its own durable consumer, `todo-<agent>`,
applies each op to its todo list and acks it. An agent that's down, or
whose harness predates this, leaves its ops waiting.
The router holds no forge token and no webhook secret, and registers no hook.
## Flows
A flow is a graph of nodes joined by edges: one `source` node naming the forge
event types it starts from, `match` nodes that pass when every condition
holds, `todo` nodes that emit one todo op per recipient, and `request_reviewer`
nodes that ask for a review. A match node can have more than one child, and
each one runs, so one event can reach more than one leaf. A flow must be a tree:
one source, one parent for every other node, every node reachable, no children
under a `todo` or a `request_reviewer`. The most is 64 nodes, 16
levels deep, and 100 custom flows. A `todo` node sends to the agents an event
field holds (`assignees_added`, `todo_holders`, …) or to one named agent.
The router ships the three flows above, each a source → match on `action` →
todo, and `head-classifier`: source `pull_request` and `pull_request_sync` →
match on `action` (`opened`, `reopened`, `synchronized`) and `draft` = `false`, or on
`undrafted` → `request_reviewer` for `review-classifier` ([below](#requesting-a-review)). The code defines them
(`swarm_wire::notify_flows::builtins`), so a build that changes one reaches
every swarm. Each switches off on the notification router card on `/swarm`. A custom flow has the same type and
the same evaluator, so it can do anything a built-in does.
**The stored document.** swarm-controller keeps the rest in one document, the
`all` key of the `notify-flows` bucket: the ids of the built-in flows you
switched off, and the custom flows. Only the controller writes it, through
`GET` and `PUT /api/notify/flows`. A `PUT` replaces the whole document in one
compare-and-set. The controller answers 400 unless every flow passes the tree
check above and reads only what exists: a source event from `issues`,
`issue_assign`, `pull_request`, `pull_request_assign`, `pull_request_sync` or one of the
three `pull_request_review_*` events, and condition and recipient fields from the event's field list. The same goes for the `{field}`
placeholders in a todo's key and text. A `PUT` must carry the `revision` it read, `0` for no
document, and gets 409 when the stored revision is another. Swarm UI's
[notification router card on `/swarm`](ui.md#notification-flows) shows the document and
switches built-ins.
**The schema.** `GET /api/notify/flows` also returns a `schema` object, built
from the constants the check reads, so an editor offers only what a `PUT`
accepts:
- `source_events`: `[{name, description}]`, the source events.
- `fields`: `[{name, description, values, values_from?, closed}]`, the condition
and recipient fields, and the `{field}` placeholders. `values` lists what the
field holds, as far as the controller knows, as `{name, description}` pairs, such as the `issue` and
`PR` of `kind`.
- `node_kinds`: `[{kind, description, fields}]` for `source`, `match`, `todo`
and `request_reviewer`; each field is
`{name, description, required, values, values_from?, closed}`, and `values`
lists the accepted `{name, description}` pairs (the `upsert` and `clear` of a
todo's `op`), empty for other fields.
- `values_from` on a field says where an editor asks for values that are only
known live: `agents` is the names `GET /api/agents` lists, `repos` the
repositories with an open issue, which `GET /api/repos` lists, so a repository
with none is missing. `closed` says `values` and that source are every value
the field takes, which an editor can show as a dropdown. When false, they
are suggestions: the field still takes any other text, such as an `action` the
forge adds later or the login of a reviewer who is no agent.
- `limits`: `{max_nodes, max_depth, max_flows}`.
A controller that predates `schema` leaves it out of the answer; a client reads
it as optional.
**Requesting a review.** A `request_reviewer` node has one field, `reviewer`,
a forge login. For the PR the event concerns (`repo` and `number`; an event
about an issue sends nothing), the router sends a NATS request on
`$SWARM.router.sink.request-reviewer` with `{repo, number, reviewer}` and
swarm-controller answers with one line:
| Reply | Meaning |
| --------------- | ---------------------------------------------------------------------------------- |
| `requested` | the forge accepted the request |
| `skipped:<why>` | the reviewer's review of the PR's head already stands; the controller sent nothing |
| `error:<msg>` | the forge refused or didn't answer, or the request didn't decode |
The controller asks the forge under its own token, which the router never
holds. It skips a reviewer whose latest review is neither stale (the PR's head
moved past the commit it reviewed) nor dismissed, because requesting a review
from someone who already submitted one makes the forge dismiss it, approval
included. The controller asks again once the review is older than the head.
`hive-forge pr assign-reviewer` applies the same rule (`hive_types::review`).
The router sends the request once, after it queued the event's todo ops and
wrote the snapshot, and logs the answer. A controller that's down or answers
`error:` doesn't hold up the event or any agent's todos, and the request is
not retried: the next push asks again. A request repeated for the same head is
harmless: a reviewer with a pending request is a no-op for the forge, and one
who has reviewed gets `skipped`.
`head-classifier` requests the review classifier's review when someone opens or
reopens a PR and on every push to an open PR that isn't a draft. A draft is a
PR the forge reports with `draft` true, which it does for a title starting
`WIP:` or `[WIP]`. A title edit that drops that prefix (`undrafted` is true:
the event is `edited`, the PR is no draft, and the old title in the payload
started with the prefix) requests the review once, so a PR whose last push
came as a draft gets a classification. The classifier then runs as it does
for any requested review
([`review-classifier.md`](review-classifier.md#trigger-and-lifecycle)), through
its own request queue, which replaces a waiting request for the same PR. A
repo the classifier's account can't read, or a PR that account opened, answers
`error:`.
**Reviews as a source.** A submitted review arrives as a pull request
payload: `pull_request` and `repository` as for any PR event, `action` is
`reviewed`, and `sender` is the reviewer. Approving, requesting changes and
commenting are three event types, so a flow picks the verdict with its source
and reads the reviewer with a condition on `sender`. A flow that asks argus to
review once the classifier has approved:
| node | setting |
| ---------------- | ----------------------------------- |
| source | `pull_request_review_approved` |
| match | `sender` equals `review-classifier` |
| request_reviewer | `reviewer` = `argus` |
The skip rule above applies: the router asks for argus's review once per head.
**What the router runs.** It reads the key at most every 10 seconds, when an
event arrives, and runs the built-ins you left on, then the custom flows that
are on and pass the tree check. It logs a custom flow that fails the check,
leaves it out, and runs the rest.
| the key | the router runs |
| -------------------------------------- | ----------------------- |
| absent, deleted, or its bucket missing | the built-in flows |
| a document it reads | the above |
| a document it can't read | the flows it ran before |
| a read that fails | the flows it ran before |
"Can't read" means not JSON, not a flows document, or a `version` other than 1.
Before the router has read any document, "before" means the built-ins, so a bad
document never leaves it with no flows. The controller refuses to overwrite a
document of a version it doesn't know.
**Deploying in any order.** The controller, the router and the UI can be on
different hosts. A router that predates `request_reviewer` reads a document
holding such a node (a custom flow with one) as one it can't read and keeps the
flows it had; it also has no `head-classifier`. A router that has `head-classifier`
but predates the `draft` and `undrafted` fields requests the review of a draft PR
as before. A router that has `request_reviewer`, with a
controller that predates it, gets no answer to the request and logs that. A controller that predates this leaves the key absent, and the
router runs the built-ins as before. A router that predates this ignores the
document, so the page and the API show a state the router isn't yet running.
## Deliveries and dry runs
**The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG`
stream for each delivery it makes, and the stream keeps the newest 500. A
delivery is a todo op queued for an agent or a review requested from the
controller, whether it went through or not. A row has:
| Field | Holds |
| ------------------------- | ---------------------------------------------------------------------------------- |
| `at` | Unix seconds |
| `flow`, `node` | the flow and the node that delivered |
| `event`, `repo`, `number` | the forge event type and the issue or PR it concerns |
| `kind`, `target` | `todo` and the agent, or `request_reviewer` and the reviewer |
| `ok`, `detail` | whether it went through, and the op and key, the controller's answer, or the error |
A review the controller skipped (`skipped: <why>`) is `ok`: nothing was
wrong. An event the router retries after a failed step writes its rows again,
so a retry shows twice, and the page says so under the table. A row that
takes longer than 2 seconds to write counts as failed: the router logs a
warning and the delivery goes on.
**The journal.** The router also logs each delivery it handles, so
`swarm-logs` answers whether a flow ran for a given PR. The lines carry the
event and sender, never the payload body:
| Message | Level | Fields |
| -------------------------------------------- | ----- | ----------------------------------------------------------------------------------------- |
| `notify router: forge event routed` | info | `event_type`, `action`, `repo`, `number`, `sender`, `delivery_id`, `flows` (at least one) |
| `notify router: forge event matched no flow` | debug | the same fields, with `flows` of 0 |
| `notify router: sink ran` | info | `flow`, `repo`, `number`, `kind`, `target`, `outcome` |
| `notify router: sink failed` | warn | `flow`, `repo`, `number`, `kind`, `target`, `error` |
`flows` counts the flows that produced a todo op or a review request.
`delivery_id` is the forge's delivery id, or `none` when the event carried no
id. A review the controller skipped is a `sink ran` line, as in the log rows.
**The dry run.** The router runs a flows document over the newest 50 events of
the `SWARM_EVENTS` stream and returns, for each flow the document turns on,
how many of those events reached each node. A `match` node counts the events
that passed it. It uses the walk the router delivers with and delivers nothing:
no todo op, no review request, no log row and no snapshot written. It keeps
its own per-issue snapshots, starting empty, so the first assign event it
replays for an issue counts every assignee as newly added. Every recipient
counts as an agent.
**The API.** The router answers two NATS requests from swarm-controller, which
serves them next to `GET /api/notify/flows`:
| Route | Router subject | Answers |
| ---------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------- |
| `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default |
| `GET /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the saved flows document, run over the newest 50 events |
| `POST /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the body's draft flows document, checked as a `PUT` checks it, run over the newest 50 events |
The dry-run request carries a whole document. The `GET` sends the saved one;
the `POST` takes a `PUT` body (it ignores a `revision`), answers 400
with the `PUT` error for a document that fails the check, and forwards a valid one.
[notification router card on `/swarm`](ui.md#notification-flows) shows both: the recent deliveries,
and the counts beside each node.
**Deploying in any order.** A router that predates these subjects doesn't
subscribe to them, and the queue answers the controller's request at once with
no responders. Both routes then answer 503 saying the router doesn't answer
and may be an older build, and the page shows that in place of the table and
the counts. A request the router hears and doesn't answer within 10 seconds is
the same 503. A router that answers with a refusal, or with something the
controller can't read, is 502. A controller that predates the routes has no
route: the page, if newer, shows the failure.
## Credentials and grants
| Principal | Grant |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds |
| swarm-controller | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two
`$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or
`$SWARM.events.forge.*`, so each has one writer, and the controller alone
hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the
controller asks the forge under the token it already holds. The agent's create subject
carries the consumer's filter, and the queue server checks it against the
request, so an agent can't create a consumer that delivers another agent's
todos.
## Observability
The router pushes OTLP metrics through its container's forwarder to the swarm
collector, labelled `service.name=hyperhive-notify-router`. With no
`OTEL_EXPORTER_OTLP_ENDPOINT` it exports nothing. The client-certificate
gauges (`bao.client_cert.*`) come with them.
| Metric | Unit | Kind | Attributes |
| ---------------------------------------------- | ---- | --------- | -------------------------------------------------------------------------------- |
| `hyperhive.notify_router.events` | — | counter | `event_type`; `outcome`: `routed`, `unmatched`, `ignored`, `unusable`, `retried` |
| `hyperhive.notify_router.event.duration` | `s` | histogram | `event_type` |
| `hyperhive.notify_router.todos` | — | counter | `op`: `upsert`, `clear`; `outcome`: `delivered`, `failed` |
| `hyperhive.notify_router.review_requests` | — | counter | `outcome`: `requested`, `skipped`, `failed` |
| `hyperhive.notify_router.pull.failures` | — | counter | — |
| `hyperhive.notify_router.consumer.pending` | — | gauge | events not yet delivered to the router's consumer |
| `hyperhive.notify_router.consumer.ack_pending` | — | gauge | events delivered and not yet acked |
| `hyperhive.notify_router.consumer.redelivered` | — | gauge | events delivered more than once |
`ignored` is an event that names no issue or pull request. `unmatched` is one
no flow matched. `retried` is one the queue or the store failed on: it comes back
and counts again. A growing `consumer.pending` means the router is falling
behind. The router reads the consumer counts at the start of each pull, at most
30 seconds apart.

711
docs/swarm/review-classifier.md
View file

@ -1,711 +0,0 @@
# Review classifier
The review classifier checks a pull request against short, named yes/no
rules that the repo itself defines, and posts the result as a forge review.
A small model answers one rule at a time, so mundane rule checks come out
the same way on every PR.
It doesn't replace a full review by a human or agent reviewer. A swarm can
request that review only once the classifier has passed the PR, and use a
larger model there. The classifier takes the cheap, repetitive checks off
that reviewer.
## Repo configuration
A repo defines its rules in a file at its root, in YAML as
`.hyperhive/review-classifier.yml` or in JSON as
`.hyperhive/review-classifier.json`. It holds one of the two; with both, the
rules don't load. `swarm-controller` always reads the file from the repo's
**default branch**, never from the PR, so a PR can't edit the rules it's
judged by. It reads it with its own forge token on every load, with no cache,
so a rule merged to the default branch counts from the next load on.
The file holds one key, `classifications`: a list of one to ten entries, each
with exactly these four keys:
- `name`: short and unique in the repo, shown in the result table
- `description`: the yes/no question the model answers about the PR
- `threshold`: the certainty, 0 to 100, a verdict needs to count
- `polarity`: `positive` means a match is the wanted outcome, and
`negative` means a match is a violation
Any other key, at the top or in an entry, is an error: a misspelt key fails
the load instead of vanishing. An empty list, more than ten entries, an empty
name or description, and two entries with one name are errors too. A file
that exists but fails these checks never falls back to a pass: the run posts
no verdict.
A repo whose default branch holds neither file has no rules. When someone
requests the classifier's review on its PR, the account approves at once. The
body reads `no review rules are configured for owner/name; approved without
checks`. The controller makes no model call. The instance has no per-repo
filter yet, so every repo without a rules file gets this approval.
An example:
```yaml
classifications:
- name: datetime
description: All datetimes carry time zone information.
polarity: positive
threshold: 80
- name: no-ads
description: >-
A commit message contains a co-authored-by line or other
advertising for an AI model.
polarity: negative
threshold: 90
- name: bao-first
description: Any added secret reaches its reader through the bao store.
polarity: positive
threshold: 70
- name: docs-updated
description: >-
A change to what an operator or user sees, such as a command, flag,
option, endpoint, service or default, updates the matching page under
docs/ or the crate's README in the same diff. A diff that changes
nothing an operator or user sees makes the question not applicable.
polarity: positive
threshold: 65
```
## Trigger and lifecycle
The classifier has a dedicated forge service account, `review-classifier`
(see [The service account](#the-service-account)). Requesting that account
as a reviewer on a PR starts a run:
1. The forge sends the `pull_request_review_request` delivery to
[`swarm-controller`](../../swarm-controller/README.md), the swarm's one
webhook owner, on its instance-wide hook `/webhook/forge/review-request`.
A delivery for any other reviewer, or a removed request, does nothing.
2. The controller reads the PR and the repo's rules from the default branch,
and hands the run to the classifier service as one queue request on
`$SWARM.review-classifier.try-out`, the subject a try-out uses.
3. The classifier runs every classification against the PR's commits and
diff.
4. The controller posts the result on the PR as the service account's
review, on the commit it read.
Opening or pushing to a PR starts no run by itself. The notification router's
`head-classifier` flow requests the account's review when someone opens or
reopens a PR and on every push to an open PR, unless its review of the new head
already stands ([`notify-router.md`](notify-router.md#requesting-a-review)). It
requests nothing for a draft PR, one whose title starts with `WIP:` or `[WIP]`
and which the forge reports as `draft`. The edit that takes a PR out of draft
requests the review once, for the head the PR has then. That edit recognises
only the default `WIP:` and `[WIP]` title prefixes: after taking a PR out of
draft that used another prefix, request the review by hand. A request that
reaches the controller is always reviewed, whether or not the PR is a draft.
When a post can't reach the secret store or the forge, because the
connection fails or a try gets no answer within two minutes, the controller
tries it up to six times over about five minutes, enough to outlast a
swarm-bao restart, with the same verdict. It doesn't retry any other failure,
such as the forge refusing the review. A post that still fails ends the run
as `post-failed`, and nothing appears on the PR. The account's review
request is still pending then, and the forge sends no event for a request
that's already standing: remove the service account's review request, then
request it again to start a fresh run.
A verdict belongs to the commit it ran on. A push starts no run. Once the
PR's head moves, requesting the service account as a reviewer again starts a
fresh run on the new head.
Requesting the account again on a head it already reviewed depends on that
head's newest recorded review, as the [run store](#run-store-and-retention)
holds it. When that review approved or requested changes, nothing runs: the
account posts the same verdict and result table again, opening with
"Already reviewed `<sha>`, verdict unchanged." When it had no clean
verdict, or its post failed, the request starts a fresh run. A head with no
recorded review, such as one whose runs have dropped out of the newest 100, or
a run store the controller can't read, gets a fresh run too.
The re-request also retires the account's earlier review. The forge clears
the `official` flag on a reviewer's earlier reviews when it's requested
again, and only official reviews count toward the required approvals or
block a merge. Removing the request before the new review lands restores
the earlier one. A repo whose branch protection also dismisses stale approvals (see
[PR review gate](../process/pr-review-gate.md)) stops counting an approval
on an older commit even without a re-request.
How many requested reviews run at once is a swarm-wide setting, two by
default (see [Swarm-level settings](#swarm-level-settings)). The controller
turns no request away. It puts each one in the `review-requests` work-queue
stream on the swarm queue, one subject per PR, and the stream keeps one
message per subject: a newer request for a PR replaces one still waiting.
The controller takes a request once a run place is free and acks it after
writing the run's record. While a run goes, the controller tells the queue
every 20 seconds that the request is still in hand. A request it took and
never acked, because the controller stopped first, comes back within a
minute, and the controller that's running then reviews it. A restart
therefore repeats a run that was going. When it can't queue a request, the
account posts a `**No verdict:**` comment saying so.
When the controller takes a request, it reads the PR first. A request whose
head is no longer the PR's head runs on the PR's current head, and never
re-posts the old head's recorded verdict.
A request for a PR whose review is still going waits in the queue until that
review ends, so a PR never has two runs at once. A request that re-posts an
earlier verdict waits for a run place too, and frees it before it posts.
## Components
### The classifier service
A swarm-level service in its own container, on whichever host the operator
puts it. It talks to `swarm-controller` through async messages only, and
gets its secrets from the swarm's secret store, bao
([`bao.md`](bao.md), [`credentials.md`](credentials.md)).
It looks like an agent from the inside, with three things missing: it has no
full agent identity, no state directory and no tools. Every model session is
ephemeral, and nothing carries over from one run to the next.
On the API key runtime, each classification is one `claude --print` the
service spawns through the `hive-claude` crate directly, with no session to
resume, no tools (`--tools ""`), no MCP server (`--strict-mcp-config` and no
config) and an empty directory of its own. The prompt reaches it on stdin as
one user message in `--input-format stream-json`, so that it can carry content
blocks (see [Shared context](#shared-context)).
The `claude` is the one its host's
`services.hyperhive.deploy.hive-controller.claudeCodePackage` pins for
agents, or its container's own nixpkgs' where the host pins none.
On the ACP runtime, each classification is one prompt on a new `opencode acp`
session. Its config and opencode's state live in a directory of its own,
removed after it. Its working directory is one empty directory that every
classification shares: opencode writes that path into the system message, so
one path keeps the system message the same from one classification to the
next, and the provider's prompt cache can hold. The service drives it through
[`hive-runtime`](../../hive-runtime/README.md)'s `AcpRuntime`, as the subagent
daemon does a one-run subagent, and stops the agent when the classification
ends. The agent has no tool. opencode runs some tools, `read` among them,
without asking the client, so the service sets opencode's own permission to
deny every tool (`OPENCODE_PERMISSION={"*":"deny"}`), which leaves the model
with no tool offered at all, and refuses any permission request on top. The
config it writes for opencode names one provider and model, from the settings,
and turns off opencode's title call, which would send the prompt a second
time. It also defines the agent the session runs as, whose prompt is the
classifier's system prompt, so opencode sends it as the system message, with
only its environment block (model, directory, platform, date) after it. The
first user message holds the PR alone. That agent has no `permission` of its
own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
there would offer the model tools again.
The service is `swarm-review-classifier`, and its container is the one
[`services.md`](services.md#review-classifier) describes. It holds no forge
credential: `swarm-controller` reads the PR and sends it along.
### Swarm-level settings
The runtime, the model, the effort and how many requested reviews run at once
are one document for the whole swarm. `swarm-controller` keeps it in its
`review-classifier-settings` KV bucket under the key `all`, and no other
service reads it. With no document stored, the classifier runs on the API key
runtime on `haiku` at claude's default effort, two reviews at a time. The
controller ignores a stored field it doesn't know, and a document it still
can't decode counts as none, with a warning in its log, until the next PUT
replaces it.
`GET /api/review-classifier/settings` answers the settings in force:
```json
{
"model": "haiku",
"effort": null,
"max_concurrent_reviews": 2,
"runtime": "api-key",
"acp": null
}
```
- `model`: the model every classification on the API key runtime runs on, as
claude's `--model` takes it: one word with no whitespace, not starting with
`-`
- `effort`: `low`, `medium`, `high`, `xhigh` or `max`; `null` runs at the
runtime's own default. On the ACP runtime it applies only where the agent
offers it
- `max_concurrent_reviews`: 1 to 8, how many requested reviews run at once.
Try-outs don't count toward it. The rest wait in the queue for a place
- `runtime`: `api-key` or `acp` (see [Runtimes](#runtimes))
- `acp`: the provider the ACP runtime runs on, or `null`. It stays stored
while the runtime is `api-key`, so switching back and forth needs no
re-entry. It holds:
- `base_url`: the provider's OpenAI-compatible API root, `https://` only
- `model`: the model as the provider names it, with no whitespace
- `prices`: what the provider charges in USD per million tokens,
`input` and `output`, and optionally `cache_read` and `cache_write`, which
otherwise count as `input`. A price is 0 or more
`PUT /api/review-classifier/settings` takes any of these fields, changes only
those, and answers the whole document it stored. A field left out keeps its
stored value; `null` sets `effort` or `acp` to none. A form that sends the
model, effort and limit alone therefore leaves the runtime as it was. No secret
goes in the document: an `api_key` field is an unknown field.
The controller reads the settings for every run and sends the runtime, model
and effort in the run request, so a change applies from the next run with no
restart. A try-out reads them when it's posted, a review when it gets its
place. Lowering the limit lets the runs already going finish. A run request
with no model runs on `haiku`, and one with no runtime on the API key.
The PUT needs the `Remote-User` header the gateway sets, as the try-out POST
does. Every error is problem+json with the reason in `detail`:
- **400** for a body that isn't JSON or isn't this shape: an unknown field, a
`null` other than for `effort` or `acp`, an effort outside the five levels,
a `max_concurrent_reviews` that isn't a whole number, a runtime other than
the two, or an `acp` without `base_url`, `model` or `prices`. Also **400**
when the document the PUT would store breaks a rule: an empty `model`, one
with whitespace or one starting with `-`, a `max_concurrent_reviews` outside
1 to 8, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
with whitespace, a negative price, or the `acp` runtime with no `acp`
- **401** for a PUT with no `Remote-User`
- **503** when this host has no swarm queue, or it isn't connected
- **500** when the controller can't read or write the settings
When the controller can't read the settings, a try-out answers the same 503
or 500, and a review posts a no-verdict comment. The operator writes each
runtime's key by hand in the secret store
([`services.md`](services.md#review-classifier)).
The per-agent equivalent is [`inference.md`](inference.md), where the
controller writes model and effort to a queue KV bucket
(`swarm-wire/src/agent_inference.rs`) and each agent reads its own
key.
### The service account
A dedicated forge user, separate from swarm-controller's site-admin account
and from every agent's account. It's the reviewer someone requests, it
authors every classifier review, and it pushes the branches the try-out section
creates.
`swarm-controller` creates it as `review-classifier`, with the email
`review-classifier@services.hyperhive.local` and no repo creation, and mints
its one token, `swarm-review-classifier` with scope `write:repository`, into
the store at `swarm/services/review-classifier/forge/token`. A pass at start
and every five minutes re-mints when the token is missing or no longer
matches the forge. The controller refuses an existing account by that name
with any other email. The controller posts the reviews and pushes the try-out
branches with that token, and the classifier service never sees it.
**Giving a repo access is a repo admin's step, by hand.** Nothing grants
it automatically. In the repo's settings, add `review-classifier` as a
collaborator:
- with **read**, so someone can request it as a reviewer, and its reviews show,
but its approval doesn't count
- with **write**, so its approval counts toward the required approvals and
its request-changes blocks the merge where the branch protection blocks on
rejected reviews. Where the branch protection restricts approvals to an
allowlist, the account counts only once it's on that list, and read access
is then enough for reviews.
The try-out's **open a PR** pushes a branch to the repo itself, so it
needs **write** whatever the branch protection says.
The repo also needs a rules file on its default branch (see
[Repo configuration](#repo-configuration)). Without one, a request posts a
comment saying so.
## The classification run
### Shared context
Every classification in a run sees the same context, in the same order:
1. a short system prompt
2. the PR's commits, each with its full message
3. the full diff
Only the last part differs per classification: that classification's
description and the task to classify the PR against it. The identical prefix
lets the runtime's prompt cache reuse it across questions.
The controller builds the diff itself, from each changed file at the PR's
merge base and at its head, with 25 lines of context around each change
rather than the 3 of the forge's `.diff`. An enclosing gate, such as a
`config = lib.mkIf …` or an option's `default`, is in view when it's within
25 lines of the change, so a move that restates it doesn't read as adding one.
A gate further away is still out of view. At review time the controller takes
the merge base from the PR's commits: the one parent that isn't a PR commit.
A rebased or force-pushed PR therefore gets the diff against the base it sits on
now, not the base it started from. The controller sends the forge's `.diff`
instead when the PR changes more than 100 files, when the commits don't name
a single merge base (the head merges the base branch in), when reading a
changed file fails, or when the wide diff is over 256 KiB. The review then says
so, with the reason.
On the API key runtime the commits and diff are one content block of the user
message and the question another, with a cache breakpoint at the end of the
first. The cache writes an entry only where a breakpoint is, and `claude` sets
one only after the last block, so that breakpoint is what lets the next
question read the commits and diff from the cache. The PR stays in the user
message rather than the system prompt, which tells the model to judge it as
data. A shared context shorter than the model's minimum cacheable length is
not cached at all.
No object hash is in that prefix: the header names the repo and the PR number,
not the head commit, each commit's heading gives its position instead of its
sha, and the diff loses each file's `index <hash>..<hash>` line, keeping its
`diff --git` paths and every hunk. A rebase rewrites all of those hashes, so
with them in, a rebased PR with the same messages and hunks would miss the
cache from the first line. The head commit still goes to the controller,
which posts the review against it.
### One question, one answer
The model answers each classification in JSON. The answer holds:
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
- a **certainty** from 0 to 100, how sure the model is of that verdict
- a **reason**; the prompt asks for under 350 characters, and the classifier
truncates one over 500 to 499 plus `…`, keeping the verdict and certainty
An answer:
```json
{
"verdict": "no-match",
"certainty": 92,
"reason": "Two new timestamps use naive datetimes without a zone."
}
```
The verdict field holds `match`, `no-match` or `not-applicable`, and the
certainty field a whole number. The classifier reads the object from the
reply, or from the first code fence in it. When an answer isn't that object,
the classifier asks that classification once more, with a reminder to reply
with the JSON object alone; the other classifications aren't asked again. The
second ask runs on what the first left of the classification's three minutes,
so a retry never lengthens the run, and it doesn't happen when the first ask
leaves less than 20 seconds. The classifier also doesn't ask again after a
runtime error or a timeout. A runtime error, a timeout, or an answer that
still isn't that JSON object after the second ask leaves the classification without a verdict: it's listed with its error, and
doesn't count. A run with no key for its runtime, or in which no
classification got an answer, fails as a whole, and shows its error instead of a
verdict.
### Aggregation into the verdict
The classifier collects the answers into one result, keyed by
classification name, and logs it. A classification counts toward the
overall verdict only if it's applicable and its certainty reaches the
threshold. The table still lists every classification below its threshold or
not applicable, marked as excluded.
A counted classification fails when its verdict lands on the wrong side of
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
failing classification makes the verdict **request changes**. Otherwise the
verdict is **approve**, including when the run counted no classification at
all.
### The review
The service account posts the verdict as a forge review, approve or request
changes, and the table is the review's body. An illustrative table:
| result | check | certainty | reason |
| ----------- | ------------ | --------- | -------------------------------------------- |
| ❌ no match | datetime | 92 / 80 | Two new timestamps use naive datetimes. |
| ✅ no match | no-ads | 99 / 90 | No commit message carries an AI attribution. |
| ❔ unsure | bao-first | 95 / 70 | The PR adds no secret. |
| ❔ match | docs-updated | 41 / 65 | Below threshold. |
**Overall: ❌ request changes**
The result cell is the symbol for what the row did to the review, then the
model's verdict. ✅ is a counted row that passed its polarity and ❌ a counted
row that failed it, so _match_ is ✅ on a positive rule and ❌ on a negative one.
❔ is a row that doesn't count: below its threshold, not applicable (shown as
_unsure_), or without an answer. The certainty cell reads _certainty / threshold_.
The approval is a real one: it counts toward a repo's required approvals
like any other ([`pr-review-gate.md`](../process/pr-review-gate.md)). The
Repos that rely on it typically raise their required approval count to
account for it.
A run with no clean verdict never approves. When the PR or the rules don't
load (a repo with no rules file gets an approval instead), the run fails, the classifier
answers a different number of
classifications than the controller sent it, or **any** classification has no answer,
the account posts a **comment** instead, starting with `**No verdict:**` and the reason,
followed by the table when there is one. A comment neither approves nor
blocks the merge: nine timeouts and one pass post a comment, not an
approval.
The controller flattens every cell to one line and backslash-escapes the
characters that could end the cell or start markup, math or raw HTML: `|`, `\`,
backticks, `[`, `]`, `<`, `>`, `&`, `*`, `~`, `$`, and `_` except between two
letters or digits (`ci_metrics.rs` stays as is). Ordinary punctuation such as
`(`, `)` and `.` passes through unescaped. Each `@`, `#`, `!`, `://` and `www.`
gets a zero-width space after it, with no backslash, so a model's `reason`
can't break the table, embed markup, mention anyone, reference an issue or PR,
or carry a link. The prompt asks the model for plain text in `reason`.
## Runtimes
The classifier runs on one of two runtimes, the settings' `runtime`:
- **API key** (`api-key`, the default): the `claude` CLI on the Anthropic API
key at `secret/swarm/services/review-classifier/anthropic-api-key`, at
`model` and `effort`
- **ACP** (`acp`): `opencode` on the provider `acp` names, with the provider
key at `secret/swarm/services/review-classifier/acp-provider`, at
`acp.model`
Model names don't carry across runtimes. A claude alias means nothing to an
ACP provider, so each runtime has a model field of its own.
To switch to ACP, an admin:
1. writes the provider key in the secret store's UI under the OIDC login, at
`secret/swarm/services/review-classifier/acp-provider`, field `api_key`
2. PUTs the runtime and the provider:
```json
{
"runtime": "acp",
"acp": {
"base_url": "https://llm.example/v1",
"model": "Qwen/Qwen3-35B",
"prices": { "input": 0.5, "output": 2 }
}
}
```
The next run goes to that provider. `{"runtime": "api-key"}` switches back and
keeps the provider for next time.
A split deploy runs on the API key: a classifier older than the ACP runtime
ignores `runtime` and runs every classification on `claude`, and a controller
older than it ignores the stored `runtime` and `acp` and sends neither.
A third runtime, **claude subscription** (the `claude` CLI on a Claude
subscription login), comes later.
> **Open point.** The subscription runtime needs a login, which is the one
> piece of durable state the service would keep. A swarm-level claude
> subscription login may have to exist before this runtime works.
## Swarm UI
Two pages in the [swarm UI](ui.md), `admins` only like the rest of it.
### Recent runs
One card per run: the repo, the PR, and how many classifications ran.
Selecting a card expands it to show each classification's result, the same
rows as the review table.
### Try-out
A page for writing classification prompts without touching the repo. Pick a
repo and a PR, write or edit a classification, and run it against that PR;
change the prompt and run it again until it answers the way you want.
Nothing reaches the forge while you iterate. A button the page disables
points at a line that says why, such as the 10-classification cap on **add a
classification**, and a screen reader announces the run's progress once, with
the elapsed seconds outside the announcement.
The page posts the run to `swarm-controller` (`POST
/api/review-classifier/try-out`), which reads the PR with its own forge token
and sends it to the classifier as one queue request on
`$SWARM.review-classifier.try-out`. The page then reads the result back (`GET
/api/review-classifier/try-out/{id}`): a run can take longer than a request
stays open. The controller keeps the result in memory for an hour, and the
finished run goes into the [run store](#run-store-and-retention).
`GET /api/review-classifier/rules?repo=owner/name` returns the repo's own
rules, for the page to load into its editor; a run posts them like typed
ones. The route answers:
- **200** with `path` (the file read), `branch` (the default branch),
`commit` (the commit the route read the file at) and `classifications`, in
the file's order and in the POST's shape
- **400** for a `repo` that's missing or not `owner/name`
- **404** for a repo the forge doesn't know; a default branch with no commit,
in an empty repo or when the default branch doesn't exist; or a default
branch with neither file
- **422** for both files on the default branch, or a file that doesn't parse
or breaks a rule from [Repo configuration](#repo-configuration); the
`detail` names the file and what's wrong
- **503** when this host has no forge token yet, or the forge failed or
didn't answer within two minutes
The page's repo and PR fields offer values to pick from, and stay free text,
since a run also works on a merged or closed PR. Two routes feed them:
- `GET /api/pulls/repos` lists the repos with at least one open PR, as
`owner/name`.
- `GET /api/pulls?repo=owner/name` lists the repo's open PRs, highest number
first, each with `number`, `title` and `head_sha`. It answers **400** for a
`repo` that's missing or not `owner/name`, **404** for a repo the forge
doesn't know, and **503** when this host has no forge token yet, or the
forge failed or didn't answer within two minutes.
Every error is problem+json with the reason in `detail`.
When it's ready, **open a PR** (`POST /api/review-classifier/open-pr`, with
the repo and the classifications) has the service account push a branch that
adds the classifications to the repo's rules, with you, the swarm UI user, as
commit author. The page then links you to the forge's compare page for that
branch, where you log in as usual and open the PR yourself.
- **The branch** is `review-classifier/<you>-<8 hex digits>`, made from the
default branch's head with one commit, through the forge's contents API.
- **The file** is the rules file on the default branch, merged by name: a
classification named like an entry replaces it, a new name goes at the end,
and every other entry stays. A repo with no rules file gets
`.hyperhive/review-classifier.yml`. In a YAML file, the entries the merge
leaves alone keep their text, comments included. The merged file has to
pass the same checks the `rules` route makes.
- **The author** is the forge account named like your swarm UI login. A forge
account made by an SSO login carries the SSO username, so it's yours once
you've logged in to the forge. The commit carries its full name, or the
login when it has none, and its noreply address,
`<login, lowercased>@noreply.<forge domain>`. It's the address the forge
itself uses for an account that keeps its email private, so the commit
links to your account and your email stays out of the repo's history. The
committer is the service account.
- **The token** is the service account's own, read from the store for each
push (see [The service account](#the-service-account)), so the account
needs **write** access to the repo.
The route answers **201** with `branch`, `path` (the file written) and
`compare_url`, or:
- **400** for a body the try-out POST would refuse in its repo or
classifications
- **401** without the `Remote-User` header the gateway sets
- **403** when the forge refuses the push: the service account isn't a
collaborator with write access
- **404** for a repo the forge doesn't know, or one with no commit on its
default branch
- **409** when the forge has no account named like you yet, or the rules
file changed on the default branch during the push
- **422** for a default branch holding both rules files or one the `rules`
route refuses; classifications the file already holds unchanged; or a
merge that breaks a rule, such as more than ten classifications
- **503** when this host has no forge token, the service account's token
isn't minted yet, or the forge failed or didn't answer within two minutes
## Run store and retention
`swarm-controller` records every run when it ends: a try-out when the
classifier's reply arrives, a requested review once the controller has posted
its review or the post has failed, and a request the controller couldn't queue
once its comment is on the forge or the post has failed. The controller records a
re-posted verdict the same way, with the outcome and answer of the run it
repeats and that run's id in its `detail`. A run still going isn't in the list; a
controller restart stops it, and the queue hands its request back.
The newest 100 runs are one JSON array under the key `recent` in the
`review-classifier-runs` KV bucket on the swarm queue, which only the
controller reads or writes. The controller creates the bucket on first use and
adds each run with a compare-and-set write, so the list survives a controller
restart or a move to another host. Recording the 101st run drops the oldest;
there's no time-based retention. The controller leaves a run it can't decode, such
as one a newer build recorded, out of the list with a warning in its log. It
keeps the run in the store as it was and counts it toward the 100. A document
that isn't a list reads as no runs. The controller logs a failed write, and it
changes nothing on the forge.
`GET /api/review-classifier/runs` returns the list, newest first. Like the
try-out GET, the gateway's `admins` rule gates it, and the controller checks no
header. Each run has:
- `id`: a try-out's is the id its POST answered with
- `trigger`: `try-out` or `review`
- `repo` (`owner/name`), `pr`, and `requester`: the swarm UI user who started
the try-out, or the forge user who requested the review
- `head_sha`: the commit the run read, absent when the PR wasn't read, as for
an unreadable PR or a request the controller couldn't queue
- `started_unix` and `finished_unix`: a try-out starts at its POST, a review
once it has a run place and has read the PR
- `outcome`: `approved`, `changes-requested`, `no-verdict` or `post-failed`
for a review, or `queue-full` on a record written before the request queue;
`done` or `failed` for a try-out
- `detail`: the reason, for every outcome but `approved`,
`changes-requested` and `done`. For `post-failed` it's the run's own reason,
when it has one, then why the post failed.
- `rows` and `overall`, as the try-out GET returns them, when the classifier
answered
- `review_url`: the posted review, when the forge names its URL
A run holds no diff, no commit text and no token count or cost. The route
answers 503 when the controller has no swarm queue or can't reach the bucket,
and 500 when it can't read the list, both as problem+json.
Each recorded run is also one `info` event from `swarm-controller` with the
message `review classifier run finished`. The event carries the same fields,
with `rows` as JSON. It reaches VictoriaLogs through the host journal, where
each field arrives as `F_` and the name in capitals (`F_OUTCOME`, `F_REPO`), so
a run whose write failed is still there:
```console
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-controller.service" _msg:="review classifier run finished" | fields _time, F_TRIGGER, F_REPO, F_PR, F_OUTCOME, F_DETAIL, F_REVIEW_URL' --format json
```
Add a filter such as `F_OUTCOME:="post-failed"` before the `|` to narrow it.
## Observability and cost
The swarm UI shows no token counts, cache counts or cost, on any page. On the
claude runtimes, every classifier run starts its `claude` process with
`OTEL_RESOURCE_ATTRIBUTES` naming the repo and the PR, so the cost reads from
the metrics store by repo and PR. The process exports to the OTLP receiver of the
container's own forwarder on loopback, which pushes to the swarm collector as the
classifier's OAuth2 client, on a host with a hive or without one. On a subscription, cost is notional.
On the ACP runtime, the service computes each classification's cost itself,
from the settings' `acp.prices` and the token counts the agent reports for the
prompt, and adds it to the counter `hyperhive.review_classifier.cost.usage`,
labelled `repo`, `pr`, `model` (the provider's) and `currency` (`USD`). It
exports to the same receiver `claude` does. A classification stopped at its
deadline, or whose agent reports no token counts, adds nothing. opencode
reports no cost of its own for a provider without prices in its config, which
is why the prices are a setting.
Each such classification also adds the tokens it used to the counter
`hyperhive.review_classifier.token.usage`, with the same labels bar `currency`
and a `type` of `input`, `output`, `cacheRead` or `cacheCreation`. Those are
the types of `claude_code.token.usage`, which `claude` exports on the claude
runtimes, so the cache hit rate reads the same way on every runtime: `cacheRead`
over the sum of `input`, `cacheRead` and `cacheCreation`.
The controller counts what requests and runs come to as the counters
`hyperhive.review_classifier.requests`, `.verdicts` and `.checks` and the
histogram `.run.duration`, and the service counts each ask it repeats because
an answer failed validation as `hyperhive.review_classifier.ask.retries`. The
[metrics table](../scheduler/observability.md#review-classifier-result-metrics-swarm-controller)
lists their labels, and the `review-classifier` Grafana dashboard charts them.
The classifier logs each run's result to its journal, which ships to
VictoriaLogs like every swarm service container's
([`services.md`](services.md#logs-victorialogs)).
## Still open
These get settled, and filled in here, as the work reaches them:
- **Cache strategy.** Fully independent sessions per classification, or one
warm session and then the rest in parallel. To measure through the cache
token counts before choosing.
- **Oversized diffs.** What happens when even the forge's 3-line diff doesn't fit
the model's context. Lockfiles already collapse to one line each, as in
`hive-forge diff`.
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
cap.
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
goes to a third-party provider. Which repos may use which runtime, and
whether anything gets redacted.
- **Prompt injection.** A PR's diff and commit messages are model input, so
a PR can try to talk the classifier into approving it.
- **Subscription credential.** Where in bao the subscription runtime's login
lives. The service account's forge token is at
`secret/swarm/services/review-classifier/forge/token`.
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
classifier passed a PR before it starts its own review.

153
docs/swarm/schedules.md
View file

@ -1,153 +0,0 @@
# Scheduled prompts
A scheduled prompt hands one or more agents the same text at set times: once,
or every interval from a start time. Each prompt has a name, its title. The
swarm keeps every prompt, and each agent it names fires it in its own
container, so a fire needs neither the hive nor the swarm controller to be up
at that moment, only the agent.
Operators manage them with `swarmctl schedule` on the swarm controller's
host. Agents can't create, change or remove one. For a one-off wake an agent
sets itself, it uses `remind`, which stays inside its container.
## Managing them
```console
# swarmctl schedule add --name standup --agent scribe --agent iris \
--body "post the standup summary" --at 2026-10-05T09:00:00+02:00 --every 1d
added schedule kqzpmwab
kqzpmwab "standup" every 1d from 2026-10-05T07:00:00Z next 2026-10-05T07:00:00Z agents: iris, scribe | post the standup summary
# swarmctl schedule list
# swarmctl schedule list --agent scribe
# swarmctl schedule edit kqzpmwab --body "post the standup summary to the team room"
# swarmctl schedule edit kqzpmwab --add-agent atlas --remove-agent iris
# swarmctl schedule pause kqzpmwab
# swarmctl schedule resume kqzpmwab
# swarmctl schedule rm kqzpmwab
```
- `--agent` names an agent to fire the prompt in; repeat it for more. Each
must be an agent some hive's wanted state declares.
- `--at` takes an RFC 3339 time with its offset, or unix seconds. Without it
the first fire is now. A time in the past fires its latest past time once,
as soon as each agent reads the prompt.
- `--every` takes a count and a unit, `s`, `m`, `h` or `d`. Without it the
prompt fires once.
- `schedule edit` changes only the flags you pass and keeps the id. `--once`
turns a recurring prompt into a one-shot. `--add-agent` and
`--remove-agent` change who gets it; removing the last agent fails, so
remove the prompt instead.
- `schedule list` shows one row per prompt: id, name, cadence, start, next
fire time, agents, and the first line of the body.
The full flag reference is [`swarmctl-cli.md`](../tools/swarmctl-cli.md).
## When a prompt fires
A prompt's fire times are its start time plus whole intervals. Each agent
works them out from the prompt itself, so:
- **Editing the text or the name doesn't move them.** Neither does changing
the agents. Only a new `--at` or `--every` does.
- **An agent that was down or disconnected fires once.** When it's back, it
fires the latest time it missed and then carries on at the next one. The
earlier missed times don't fire.
- **A restart refires nothing.** Each agent records the last time it fired
for each prompt in its state database, and fires a time only when it's
later than that. An agent removed from a prompt and added back keeps that
record, so it doesn't refire a time it already fired.
- **A one-shot fires once**, even though it stays in `schedule list` until
you remove it. Moving its `--at` to a later time fires it again then.
- **A paused prompt doesn't fire**, and the times that pass while it's paused
don't fire when it resumes. An agent marks those times as passed while it
watches the paused prompt; if the agent was down for the whole pause, it
fires the latest time it missed once, as above.
A fire reaches the agent as a `schedule` loose end with the prompt's text.
The agent's turn loop wakes for it like any other loose end. If the agent
hasn't dealt with the previous fire yet, the new fire replaces it rather
than adding a second one; once the agent dismissed the previous one, the
next fire opens a new one.
Each agent reads its prompts every 10 seconds, so a fire can land up to 10
seconds after its time. A change reaches each agent within the same window,
or within a minute if the controller's first attempt to pass it on failed.
## When a prompt doesn't fire
- **The agent isn't connected to the swarm queue.** It reads its prompts with
its own swarm-queue credential. A harness that can't connect retries for
about five minutes, then exits and comes back; it reads no prompts until a
connect succeeds. See
[Giving the agents the queue too](README.md#giving-the-agents-the-queue-too).
- **The agent hasn't read its prompts since it started.** It fires from the
last list it read; until the first read succeeds it fires nothing. A read
that fails later keeps the previous list.
<details><summary>Implementation</summary>
swarm-controller is the only client that writes either of the two KV buckets
involved.
`schedule-definitions` holds every prompt as one JSON document under the key
`all`. Agents hold no grant on this bucket. Each change re-reads the key and
writes against its revision, so a prompt exists for every one of its agents or for
none of them, and two changes at once don't lose one. The controller leaves a
prompt it can't decode, such as one a newer build wrote, out of the list, with
a warning in its log. An agent's key that already held the prompt keeps it as
it was, so it goes on firing, and no other key gets it. The controller keeps
the prompt in the document as it was, after the others, and won't reuse its
id. A document that isn't an object with a list of `schedules` is an error.
`agent-schedules` holds one key per agent: the prompts naming that agent,
without their names or other agents. A key the controller can't decode counts
as no prompts, with a warning in its log, and the next rewrite replaces it.
```json
{
"schedules": [
{
"id": "kqzpmwab",
"body": "post the standup summary",
"anchor_unix": 1791183600,
"interval_seconds": 86400,
"paused": false
}
]
}
```
`interval_seconds` is absent for a one-shot. After each change the controller
rewrites the key of every agent the change named before or after, and once a
minute it rewrites every agent's key, so a rewrite that failed converges. Ids
are 8 lowercase letters, the same in every agent's key, never reused, and so
never equal to a hive schedule's numeric id.
`swarmctl schedule` calls these routes over the controller's socket:
| method | path | body |
| -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `GET` | `/api/schedules` | —; answers every prompt, sorted by name then id |
| `POST` | `/api/schedules` | `name`, `agents`, `body`, and optional `anchor_unix`, `interval_seconds`, `paused`; answers 201 with the prompt |
| `PATCH` | `/api/schedules/<id>` | any of `name`, `body`, `anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, `add_agents`, `remove_agents` |
| `DELETE` | `/api/schedules/<id>` | —; answers 204 |
Each prompt they answer with has `id`, `name`, `agents` (sorted), `body`,
`anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, and
`next_fire_unix`: the next time on the prompt's grid, as of the moment the
controller answers, `null` when paused or a one-shot whose time has passed.
An agent's own queue credential grants exactly one subject in this bucket:
`$JS.API.DIRECT.GET.KV_agent-schedules.$KV.agent-schedules.<agent>`,
the direct read of its own key. It can't read another agent's key, open or
watch the bucket, or write to it. A KV watch would need a consumer, and a
consumer create subject doesn't name a key, so granting one would let the
agent read every key; the agent reads its key every 10 seconds instead.
The harness (`hive-agent`'s `schedule_timer`) keeps the last fired time per
prompt id in the `schedule_fired_slots` table of its state database, and
pushes each fire as the todo `schedule`/`schedule:<id>` with source
`scheduled`, reopening an acknowledged one.
</details>

609
docs/swarm/services.md
View file

@ -1,609 +0,0 @@
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. |
| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |
# Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge,
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
log stack. This page says which host runs them and what a hive that runs none
of them configures instead. The all-local quick start sets everything with one
line → [README](../../README.md#quick-start-an-all-local-swarm).
Two options say where they live, and everything else derives:
```nix
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.deploy.allSwarmServices = true;
```
**`deploy.allSwarmServices` is what "the swarm's shared services run
here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service
sections below don't repeat it, so a service that stops deriving is a
visible difference rather than one more paragraph saying the same thing.
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the
host's `/etc/hosts` entries for the names it serves
(`gateway.localHostsEntry`) and where the secret store's bootstrap
token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived
toggle on its own, and that setting wins.
**Both default to off**, and that's deliberate: a host can't tell
whether it's meant to be the swarm's service host, so this is an
operator saying so rather than something inferred. With them off, a hive
is a _client_ of those services — it configures how to reach them and
runs none of them.
That includes the forge (`deploy.forgejo.enable`). Every swarm needs
one, being the canonical store for the meta flake and every agent's
config repo, so **exactly one host must turn it on**: `singleHostSwarm`,
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A
host that enables its services one by one without any of those runs no
forge.
A hive can't start or stop any of these. `hivectl stop`, `start` and
`restart` act on the hive's agents and its gateway only, even on the
host that runs the swarm's services. The forge and matrix containers are
systemd units on that host (`container@hive-forge.service`,
`container@hive-matrix.service`), and the operator manages them there
with `systemctl`.
A hive that runs none of these reaches each one by name — the forge at
`forge.<swarm-domain>`, for example. Only the host running a service
answers its name from its own resolver, so on a swarm spread over
more than one host, the operator's DNS has to resolve those names to that
host.
<details><summary>Moving an existing hive's forge to the swarm's</summary>
A hive that stops running the forge keeps the old container's state at
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
forge: push anything worth keeping there by hand.
</details>
## Deployment shapes
Those two options are what makes the difference between deployments, so
the shapes worth naming are the ones they produce:
- **All-local.** Everything on one machine:
`singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for
a hive to run agents on. After the first switch, the steps in
[`setup.md`](../getting-started/setup.md) remain.
- **Services on the swarm controller host.**
Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there,
with hives elsewhere. The controller doesn't derive from
`allSwarmServices`.
- **Fully spread out.** One container / VM / machine per service,
somewhere.
**These are a set, not a ladder with a correct top, and hyperhive
supports in-between shapes.** You can set each derived toggle on its own (see
above), which is what makes "all local except X" a configuration rather
than an unsupported edge case. Nothing in hyperhive prescribes a
deployment model, so a doc that treats one shape as the real one and
the others as compromises is wrong about the system rather than
opinionated.
The practical consequence is that **co-location is normal**: in the
first two shapes services share a host by design. Where a specific
service has something to say about sharing a host --- extra
requirements, or a cost worth weighing --- that belongs in the doc for
that service, not here.
## Services
### SSO (authelia)
One authelia per swarm, in a `swarm-authelia` container, at
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
provider, differentiated by roles and claims rather than by mechanism —
there is one IdP and one auth path.
The container sits on the hive bridge in a network namespace of its own and
dials the store, the collector and its own public name through it. It binds
no TCP port: the login server, the metrics listener and the bridge each
listen on a unix socket in `/run/swarm-authelia`, a directory only the
gateway's nginx and the container's processes can open.
- **`deploy.authelia`** — run the container here.
- **`swarm.authelia.url`** — where clients go to authenticate. Present
on **every** hive and the same value on all of them:
`https://<swarm.authelia.domain>`, whether or not this host runs the
container. Set it explicitly when joining a swarm whose IdP is under
another name.
Agent subjects come from swarm-controller's agent-creation job, written
into the users database by `swarm-authelia-bridge`; human ones come from
`swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account).
The bridge listens on a unix socket on the authelia host, and the gateway
publishes it as
`https://<swarm.authelia.domain>/bridge/`, which is where
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
It answers only swarm-controller's OIDC client
(`swarm.controller.queueClientId`) and refuses every other token `403`.
On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one
invalidates data already written (sessions, the encrypted store), so
that's an operator action.
Storage is local sqlite and the notifier writes to a file. Both are
small-deployment choices, and the scope is the justification: redis
buys shared session state across replicas and there is one instance;
SMTP exists to mail humans, and provisioning here is programmatic.
The swarm collector scrapes authelia's own metrics at
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`.
The container's `authelia-swarm-secrets` unit creates each client secret directly in
the swarm secret store, under the secret publisher's own store leaf handed in as
a system credential, and writes only the digest to disk; see
[`credentials.md`](credentials.md). Authelia starts from its digests, so the IdP
comes up while the store is down, and a unit that needs the store retries every
30 seconds. The container's journal forwarder reads its client secret back from
the store under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`credentials.md`](credentials.md) for where authelia
generates and reads each of its keys.
### Metrics (VictoriaMetrics + Grafana)
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
reads it, in two containers at `metrics.<swarm-domain>` and
`grafana.<swarm-domain>`. Two containers rather than one so you can
restart or break Grafana without taking the time-series database with it.
The Grafana container sits on the hive bridge in a network namespace of its
own and dials authelia, the secret store, the collector and both datasources
by name through it. It binds no TCP port: it listens on the unix socket
`/run/swarm-grafana/grafana.sock`, a directory only the gateway's nginx and
the container can open.
They derive together: a store with no UI is unreadable and a UI with no
store is empty. To run one without the other, set it directly:
```nix
services.hyperhive.deploy.victoriametrics.enable = true;
services.hyperhive.deploy.grafana.enable = false;
```
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
<!-- vale write-good.Passive = NO -->
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`credentials.md`](credentials.md).
<!-- vale write-good.Passive = YES -->
**Dashboards.** Grafana provisions these from
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
- `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, and the busiest agents and destinations. **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent.
- `builds`: the container builds hive-c0re ran on every hive, in two tabs. **builds** has the count, the failures and the longest, builds over time by outcome, and a table of each build with its id, agent, kind, outcome and runtime. **output** shows every line of the build picked in the **Build** filter, with its start and finish lines. It reads the lines hive-c0re logs for each build, so a hive-c0re that doesn't log them leaves it empty. Filter by agent.
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
- `review-classifier`: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
- `forge`: the forge, in three tabs. **content** has what the forge holds (repositories, users, issues, releases, discussion volume and the issues by label). **runtime** has the forge process's CPU, memory, goroutines and descriptors. **CI runs** has Actions run time and queue wait per repo and workflow, with a per-PR table; the Repo and Workflow filters apply to that tab.
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
- `collector`: what the hive and swarm OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
- `logstore`, `metricstore`: health of the log store and the metrics store.
- `grafana`: Grafana itself. Dashboards and data sources, alert instances firing, requests by status and their latency, alert rule evaluations and failures, sign-ins, database connections, memory and goroutines.
- `builder`: the swarm builder. Builds finished, by kind and outcome, their mean and 95th percentile time, the queue depth and the wait before a build starts.
- `ci-runner`: the forge's Actions runner. Its uptime and job slots, jobs finished by status, how many slots are in use, job time, and how often it polls the forge for work.
- `matrix`: the Matrix homeserver. Whether its client API answers, how fast, the share of the range it answered, and its database size.
- `notify-router`: the notification router. Forge events received by type and outcome, todos delivered, review requests that failed, time per event, and the backlog of the agents' queue.
A new panel can read "No data" until its source exists: the push and commit
counters on `controller` appear after the first push webhook, the refused
and failure series on `collector` appear once something fails, and the
`review-classifier` panels fill as review requests arrive.
**The `collector` dashboard.** It shows the self-metrics of the hive's
collector and of the swarm's collector, which both carry the collector
binary's default `service.name`, `otelcol-contrib`. The timeseries panels
split by `server.port`: 8888 is the hive collector, 8889 the swarm
collector. Every per-container collector (the secret store, the other service
containers and the agents' forwarders) also scrapes its own endpoint on
loopback and pushes the result up its normal hop, so each has a line of its
own, on its own port: agents all report on 8904, service containers on
8890 to 8903. The series carry `container` (services) or `agent` (agents)
to tell them apart. `scripts/check-dashboard-service-names.sh` runs in CI
and in `scripts/pre-push`. It fails when a dashboard query selects a
`service.name` that nothing emits.
**Where the data comes from.** The swarm's OTEL collector, below.
The `controller` panels come from the controller itself. It exports OTLP
through the gateway to the swarm's collector from whatever host it runs on,
whether or not that host runs the collector.
Neither container is reachable except through the gateway: each listens
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
runs in a network namespace of its own with no interface but loopback.
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it:
- **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource, an agent, or an operator. Grafana sends a
bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
store, and `swarm-metrics-auth` on the store's host checks it against the
same store path. An agent's `swarm-metrics` sends its
`swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. It also needs the `metrics`
permission, or the answer is a 403. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later. When the
store doesn't answer within five seconds, it checks requests against the
last value it read instead of denying, and still refuses a wrong bearer;
it tries the store again after three seconds. A revoked or rotated bearer
stays accepted while the store is down, for an hour after the last read at
most; after that the responder denies again. The store's next answer
replaces the value, and a process that holds none (it started while the
store was down) denies. `swarm-logs-grafana-auth`
is the same binary and does the same.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to
that path; any other session gets `deny`.
The collector scrapes the store's own metrics the same way, at
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
that doesn't run the store still gets them. On this name the gateway admits
only the collector's client to `/metrics` and the push route; any other
session gets `deny`.
### Logs (VictoriaLogs)
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
behind the same SSO as everything else: the swarm's own service containers,
the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
| Option | When you'd touch it |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. A revoked `logs` permission does too. |
| `deploy.victorialogs.grafanaAuthPackage` | The `swarm-metrics-auth` build that runs as `swarm-logs-grafana-auth`. Set on its own: `deploy.victoriametrics.authPackage` doesn't reach it. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `deploy.victorialogs.port` | Only if something else on the services host already claims `9428`. |
It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. The agent must also hold the `logs` permission,
which is off by default; without it the answer is a 403 (`docs/process/conventions.md`, Permissions). Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`credentials.md`](credentials.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. On this name the gateway admits only the
collector's client to `/metrics` and the ingest route
`/insert/opentelemetry/v1/logs`; any other session gets `deny` on both. The
read side stays open to any authenticated caller.
### Telemetry collector (OTEL)
The swarm's collector receives from every hive's own collector and is the
only process that decides where telemetry goes: it always writes the stores
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
always get both. It holds
the upstream credential, which is why no hive and no agent needs one.
It runs in a `swarm-otel` container on the hive bridge, in a network
namespace of its own. It binds no OTLP port: every receiver (one per hive,
the swarm tier's and one per service forwarder) listens on a unix socket in
`/run/swarm-otel`, a directory only the gateway's nginx and the container can
open, and the gateway forwards to them. The gateway's routes and the collector's
receivers come from one list, so they name the same sockets. The
`swarm-otel-sockets` unit connects to each socket as the gateway's user after
the container starts, then requests every route through the gateway vhost. It
fails if a socket refuses for 30 seconds, or if a route answers `000`, `502` or
`504`, or lacks its own `X-Swarm-Otel-Route` response header, for 30 seconds.
The gateway sets that header, named after the route, on every response from a
route, so a collector that's up but not listening, or an nginx that doesn't
have the route, shows as a failed unit. Any status other than those passes,
the collector's 4xx included. A switch leaves a window of about 20 seconds in which
nginx and the container are out of step: the container stops before nginx
restarts and starts about 18 seconds after it. Senders retry through that window.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a
loopback URL an operator has to redirect. Nothing needs setting on a hive
that doesn't run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `swarm-attic`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter, notification router and
classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store,
authelia, cache, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `attic`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**The homeserver's metrics.** tuwunel 1.9.4 exports no metrics of its own, so
request rates, sync load and federation traffic aren't available. Two series
stand in for them, and neither adds a service:
- The hive collector on the homeserver's host requests
`https://<gatewayHost>/_matrix/client/versions` every 60 s with its
`httpcheck` receiver (`services.hyperhive.otel.httpChecks`). It reports
`httpcheck.status` (one series per status class), `httpcheck.duration` and
`httpcheck.error`, labelled with the URL. The container's forwarder can't
make this request, because `httpcheck` dials HTTP and HTTPS only and tuwunel
listens on a unix socket. The request passes through the gateway, so it
covers the gateway's route as well as the homeserver.
- The `hive-matrix-database-size` timer in the container runs every five
minutes. It measures tuwunel's database directory and pushes the gauge
`hyperhive.matrix.database_bytes` through the forwarder's OTLP port,
stamped `service.name` `hive-matrix`. It exists only where the container
has a forwarder, which needs matrix-ctl's store identity set. A size under
4096 bytes is no database, so the timer logs a warning and pushes nothing.
The swarm collector's own container, `swarm-otel`, has no forwarder: its
collector reads the container's journal itself (receiver `journald/self`) and
exports it to the log store like the host's journal, stamped `service.name`
`swarm-otel` and with no `hive` label. Unlike the host's journal, it goes to the
log store only, never the upstream: the collector writes an upstream export failure to this
journal, and shipping it upstream would fail again. A `filter` processor also
drops the collector's own failure lines that name the store exporter
(`otlphttp/victorialogs`), which can't reach the store anyway; failures of
every other exporter still arrive. The container mounts the host's journal
read-only at `/run/host-journal`, not over its own
`/var/log/journal`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in
[`../scheduler/observability.md`](../scheduler/observability.md).
### CI runner
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
runner executes pull-request code, so running one is a decision of its own.
It needs neither the forge nor a hive on its host. It reaches the forge at
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
publishes to the secret store, which it reads under its own `ci-runner` leaf.
On a host without the store, that leaf is the one file to copy. Registration,
the container and the threat model are in
[`../scheduler/ci.md`](../scheduler/ci.md).
The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in
the container's own network namespace. The container's journal forwarder
scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and
pushes the series to the swarm collector under the job `gitea-runner`, so
nothing listens on the bridge. Series, all prefixed `gitea_runner_`:
`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`,
`info` (labels `version` and `name`), `job_total{status}`,
`job_duration_seconds`, `poll_fetch_total{result}`,
`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`,
`report_state_*` and `client_errors_total`. A pull-request job runs in the same
namespace and can read the endpoint, which carries counts, durations, the
runner's name and its version, and no secrets.
### Builder
The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
has the builder's store identity, as for the review classifier below: the host
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It
evaluates unreviewed config PRs and builds them through its host's nix daemon,
from a container with its own network namespace.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
newer push replaces its PR's job while that job is still waiting. The builder
builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a failed build holds that deploy back (`build_meta` job;
[deploy gate](builder.md#deploy-gate)). Those jobs share the one-build slot with
the PR checks.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
### Binary cache
The swarm's attic server, in a `swarm-attic` container on the host that sets
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
the cache's store identity, as for the builder: the host that runs the store has
it. For an external cache, override `swarm.attic.{domain,cache}` and don't set
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
`allSwarmServices`.
The container sits on the hive bridge in a network namespace of its own, and atticd
listens on that namespace's loopback only; the gateway reaches it over a unix socket.
The gateway serves it at `swarm.attic.domain` (default
`attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
its own tokens. Storage is local disk in the container's state.
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
create its signing key, the cache and the push and pull tokens in the secret
store, each only while absent. On a host without the store, that leaf is the one
file to copy. The builder pushes green builds to it. How it works, and the
external alternative, are in [`builder.md`](builder.md#binary-cache).
The container forwards atticd's journal to the swarm collector like the other
service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host
units' lines reach the log store too: through the hive collector's journal
allow-list on a hive host, or through the swarm collector's whole-journal
receiver where the host runs that. A host with neither keeps them in its own
journal.
### CI metrics
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
`allSwarmServices` defaults it on wherever the host has the poller's store
identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to
opt out.
Each pass pushes the run time and queue wait of every newly finished forge
Actions run to the metrics store, through its own route on the swarm collector;
the first pass pushes every run since the first. It reads the forge with the
read-only token of its own `ci-metrics` forge account, under its own
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
copy. What it reads and pushes are in [`ci-metrics.md`](ci-metrics.md).
### Review classifier
The review classifier's try-out service, in a `swarm-review-classifier`
container on the host that sets `deploy.review-classifier.enable`.
`allSwarmServices` defaults it on wherever the host has the classifier's store
identity: the host that runs the store has it, and a services host with the
store elsewhere gets the classifier only once it holds a copy of the pair (see
below). Set `deploy.review-classifier.enable` directly to override either way.
The operator writes the key of the runtime the swarm settings choose by hand,
in the secret store's UI under the OIDC login, field `api_key`:
- the API key runtime's Anthropic key at
`secret/swarm/services/review-classifier/anthropic-api-key`
- the ACP runtime's provider key at
`secret/swarm/services/review-classifier/acp-provider`
Nothing writes them for you: until the chosen one exists the service runs and
every run fails with `no API key…` or `no ACP provider key…`. The service reads
the key on every run, so writing or removing it takes effect on the next run,
with no restart. Every run spends money on that key.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `review-classifier` leaf. On a host
without the store, that leaf is the one file to copy, as for the CI runner.
The swarm UI's try-out drives it through swarm-controller; what a run is
and what it costs are in [`review-classifier.md`](review-classifier.md).
### Claude login minter
The claude login minter, in a `swarm-claude-login` container on the host that
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever
the host has the minter's store identity, on the review classifier's terms.
Run one: a login in progress lives in the service's memory between its start
and its code.
An operator logs an agent in from the **log in** button on its card in the
swarm UI. swarm-controller relays the start and the pasted code over the queue,
the minter runs `claude auth login` and writes the credentials to the agent's
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
version newer than the one it installed last.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `claude-login` leaf, and reaches claude's
login endpoints over the network. On a host without the store, that leaf is
the one file to copy, as for the CI runner. The flow and the version rule are
in [`claude-login.md`](claude-login.md).
### Notification router
The notification router, in a `swarm-notify-router` container on the host
that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on
wherever the host has the router's store identity, on the review classifier's
terms. Run one: it handles each issue's events in order.
swarm-controller keeps every `issues` and `pull_request` delivery from its
instance-wide hooks in the `SWARM_EVENTS` stream. The router reads that
stream and queues todos for the agents an event concerns: an agent assigned to
an issue or pull request gets a todo, one taken off it gets that todo rewritten
to say so, and closing it clears them. A push to an open pull request makes
it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos
from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's
back.
It needs neither the forge nor a hive on its host, and holds no forge token.
It reaches the queue and the secret store by name, under its own
`notify-router` leaf. On a host without the store, that leaf is the one file to
copy, as for the CI runner. The flows and the delivery path are in
[`notify-router.md`](notify-router.md).

324
docs/swarm/sso.md
View file

@ -1,324 +0,0 @@
# Swarm SSO
The swarm runs one authelia, and it's two things at once: the **session
provider** every protected vhost checks (`auth_request`), and — once an
operator declares any client — an **OIDC provider** issuing tokens to
relying parties: the forge and the matrix homeserver.
The second role derives rather than switches:
`services.hyperhive.swarm.authelia.oidc.clients` being non-empty turns it
on. authelia refuses to start with a provider that has no clients, so a
separate `enable` would be a second fact free to disagree with the first.
## Getting in the first time
authelia binds no TCP port: its listeners are unix sockets only the gateway can reach. The **gateway** on the host running it
publishes it as `auth.<swarm.domain>` — vhost, dnsmasq record and TLS
name all follow `deploy.authelia`, so there is nothing to turn on
separately. (Details, including why a client hive must not declare that
vhost: [`../networking/gateway.md`](../networking/gateway.md).)
**Nobody can log in until the first user exists.** authelia exits at
startup on an empty user set:
```
error reading the authentication database: could not validate the schema:
users: non zero value required
```
While the set is empty this module writes one placeholder subject,
`swarm.placeholder`, that authelia accepts and that can't log in: it's
`disabled: true`, and its password digest has an all-zero key that no
known password hashes to. authelia starts and serves its login page, and
refuses every login. The placeholder stays in `users.yml` once real
users exist, shows up in `swarmctl user list`, and `swarmctl user add`
refuses its name.
⚠️ The step below is **required to finish the install**, not an
optional first-login convenience.
Add the first subject on the host running authelia:
```console
# swarmctl user add mara --display-name Mara --email mara@example.com --group admins
added mara to /var/lib/authelia-swarm/users.yml
password: <generated>
this password is stored nowhere — record it now
```
`swarmctl` generates, hashes, and prints the password once, keeping only
the hash. It reads and writes authelia's `users.yml` directly — it's
the one user store, shared with `swarm-authelia-bridge`, which creates
agent identities in the same file. No restart: authelia watches it. Full
reference: [`../tools/swarmctl-cli.md`](../tools/swarmctl-cli.md).
You can edit `users.yml` by hand, and `swarmctl` will read what you
wrote. ⚠️ It rewrites the whole file on every change, so **comments and
formatting don't survive**; values and unrecognised keys do.
This step stays manual on purpose. Bootstrapping an identity provider
non-interactively means a secret arriving from somewhere — a file, an
env var, a nix expression — and every one of those is worse than an
operator typing one command once.
### Changing a subject afterwards
`user add` only ever adds: on a name that already exists it refuses,
rather than resurfacing as a second account or a silent overwrite.
Editing an existing subject is `user update`, and the flags compose, so
one call can change multiple things:
```console
# swarmctl user update mara --add-group admins --email mara@example.com
added to group "admins"
email: unset -> "mara@example.com"
mara is now in groups: admins
```
Two behaviours worth knowing before you rely on them:
- **`--remove-group` fails if the user isn't in that group.** Every
other flag is idempotent — setting what's already set is fine, so a
"make these four things true" call doesn't break when one of them
already was. Revocation is the exception on purpose: a typo'd group
name that reported success would leave an account holding access you
believe you took away, and that's the one outcome nobody re-checks.
- **`swarmctl` prints the resulting group list** because group names have no
registry anywhere. A misspelled `--add-group` creates a real group that
no access-control rule mentions, so the user gains nothing and no error
is possible — reading the line back is the only check there is.
Passwords are deliberately out of scope here: regenerating a credential
is a different intent from editing an attribute, and folding them means
an attribute edit can invalidate a login by accident. Resetting one is a
separate verb, `user reset-password`, which generates and hashes a new
password the same way `user add` does and refuses on an unknown user:
```console
# swarmctl user reset-password mara
reset password for mara in /var/lib/authelia-swarm/users.yml
password: <generated>
this password is stored nowhere — record it now
```
Authelia's file store has no SMTP notifier, so this is the only way a
human account gets a new password once an operator forgets the old one.
## What secrets exist, and where each one lives
[`credentials-reference.md`](credentials-reference.md) tabulates every secret in the swarm, with its
generator and its path, in one place — including authelia's own keys
(session, JWT, storage-encryption, OIDC HMAC, OIDC issuer) and the two
halves of each client secret. That page's two rules — a secret is always
a path, never a value, and the generator and the reader typically live in
different containers — are why the client secret's plaintext half needs
the delivery step below and the rest of authelia's keys don't.
## Getting the plaintext to the relying party
Three cases, and they're genuinely different mechanisms rather than one
mechanism with flags.
### 1. All-local — one host runs both
Nothing to configure at all. authelia mints each client's secret at its
first boot, and the mint in its container creates it in the swarm
secret store at `swarm/services/<client id>/oidc/client`, keeping only the digest on disk. A host-side unit
then reads it from the store and writes it into the service's container:
`swarm-bao-forge-oidc` for the forge, under the forge publisher's store
leaf, into `/run/hive-forge-oidc/<client id>.secret` on the host, which
`forgejo-sso-source` loads as a credential; `swarm-bao-matrix-oidc` for matrix, under
matrix-ctl's. A unit that can't read the secret yet fails and retries
every 30 seconds for a day, so there is no start order to get right. The
service's own module contributes its client entry — callback URL
included — to authelia's client list.
Each relying service's own glue module builds its callback once, and
both authelia's client list and the service's own OIDC client config
read that same value, so the redirect URI authelia accepts and the one
the service actually sends can't drift apart. A mismatch there is a
rejected login with no error text worth reading.
<!-- vale write-good.Passive = NO -->
⚠️ nixos-container refuses to start a container whose bind source is
missing, and this secret doesn't exist until authelia's first boot has run
and the mint has stored it. The forge therefore binds the **directory**
`/run/hive-forge-oidc`, which a tmpfiles rule creates at boot, never the
file. The directory is tmpfs, so the secret is gone after a reboot until
the fetch runs again; forgejo keeps it in its login-source row, so logins
still work meanwhile. Matrix's secret is written into the container's state
directory instead.
<!-- vale write-good.Passive = YES -->
### 2. Swarm-managed services
The controller side owns provisioning: `swarmctl` writes both halves, the
same way it already owns authelia's user store (`users.yml`, read and
written in place).
### 3. A hive elsewhere
The forge's host needn't be the store's or authelia's host, and the
operator places no secret file: `swarm-bao-forge-oidc` runs on the
forge's host as in the all-local case and fetches the secret over the
store's address. What that host needs is the forge publisher's store
leaf, issued from the store's CA and copied over from the store's host:
```nix
services.hyperhive = {
swarm.authelia.url = "https://auth.example.com";
deploy.bao = {
forgePublisherClientCertFile = "/var/lib/swarm-bao-pki/forge-publisher.pem";
forgePublisherClientKeyFile = "/var/lib/swarm-bao-pki/forge-publisher-key.pem";
};
};
```
**A forge host without that leaf still evaluates.** `swarm-bao-forge-oidc`
renders and fails at start, naming both options, and the forge has no
login source until an operator places the leaf. Where the store runs on
the same host, `glue-bao-tls.nix` defaults both options to the leaf it
mints.
## Where each relying party differs
The registration half is identical; what each service does with the
result isn't.
| | forge | matrix |
| ------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| how it learns the config | a oneshot calls `forgejo admin auth`, writing a login-source row into its database | tuwunel reads a `[[global.identity_provider]]` entry from its config file |
| how it reads the secret | a path inside its container | the same path, handed on by `LoadCredential` |
| callback URL | `<root>/user/oauth2/<source>/callback` | `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`, a shape tuwunel fixes rather than accepts |
| cost of a malformed entry | the login source is missing | the homeserver can refuse to start |
Four consequences worth stating plainly:
- **A person's first forge login creates their forge account**, named
after authelia's `preferred_username` (`[oauth2_client]` in the forge
module), provided the subject has an email. It starts as an ordinary user; `swarmctl forge make-admin
<name>` makes it a site admin. A name that already has a local forge
account doesn't get it handed over: forgejo asks for that account's
own password first (`ACCOUNT_LINKING = login`). Agents, `core` and
`swarm-controller` all have local accounts, and `swarmctl user add`
refuses none of those names.
- **A member of `admins` becomes a matrix homeserver admin** once their
first matrix login has created the account: `swarm-controller` promotes
them within five minutes, and never demotes. See
[`../integrations/matrix.md`](../integrations/matrix.md#operators-are-homeserver-admins).
- **tuwunel re-reads its secret file on every OAuth exchange**, not only
at startup, and its own sandboxing hides most paths from it. It gets the
file through `LoadCredential` for the same reason the appservice
registration does — that keeps `DynamicUser` and `PrivateUsers` intact,
with no host-side ownership arrangement to maintain.
- **Matrix SSO lives inside the homeserver.** The client-server API is
spoken by non-browser clients holding matrix access tokens — every
agent's own daemon — as well as by federation, so the homeserver
serves `/_matrix/` directly and authenticates it itself. The
forward-auth vhosts protect browser surfaces; this isn't one of them.
## Machine clients
Everything above is a person in a browser. A swarm also has callers that
hold no session and follow no redirect: the telemetry collector, the
queue's auth-callout responder, and each hive.
**One client per hive, not one per service.** A hive's identity belongs to
the directory rather than to whichever service happens to consume it, so a
hive holds a single OIDC client — `<hiveClientPrefix><hive>` — and mints a
different token per service from it. The alternative, letting each
consuming subsystem declare its own list, collides on the same client id
the moment a second consumer appears.
**The audience is that client id.** A swarm service that has to tell hives
apart needs one name both sides already agree on, and the client id is
already that name. A parallel per-hive naming scheme would be a second
thing to keep in step, and it drifts silently — a mismatch presents as a
valid token refused at the target, which reads like a broken credential
rather than a broken name.
**Authelia signs tokens (`RS256`), not opaque ones**, because a resource server
that can't call the provider back is a real case here: the telemetry
collector verifies offline against `/jwks.json`, and an opaque token gives
it nothing to verify. The queue's responder introspects instead — a
different question asked of the same token, and the reason both
`/api/oidc/introspection` and `/jwks.json` have to stay reachable.
### Machine callers must fail closed
⚠️ **An error page that answers `200` is a security bug, not a cosmetic
one.** The browser surface intercepts upstream errors and serves a
friendly "SSO is unavailable" page; that page is a file, so it returns
`200`. Any machine caller routed through it receives a success carrying
HTML instead of the failure that actually happened:
- `/api/authz/auth-request` — nginx `auth_request` treats **any 2xx as
success**, so a down provider means _access granted_
- `/api/oidc/introspection` — a token check that answers `200`
- `/api/oidc/token`, `/.well-known/openid-configuration` — a client
parsing an error page as its JSON document
The gateway routes authelia's `/api/` and `/.well-known/` prefixes, and
the bridge's `/bridge/`, **without** error interception. The split is by
_audience_, not by an enumerated path list: a human gets the page, every
machine caller gets the status.
Enumerating endpoints individually would leave the next one added
silently intercepted.
The same shape bites any machine route behind a browser-shaped gate: the
caller follows a `302` to a login page, the login page answers `200`,
and the caller reports success while nothing happened. Log ingest hit exactly this
and lost eleven hours of delivery in silence.
**Checking it, if you change this routing.** Point the vhost at a dead
upstream and compare three requests, not one:
1. through `/` — must still serve the friendly page
2. through `/api/` — must deny
3. a direct dial to authelia — must match what (2) did
All three matter. A change that silently deleted the browser page would
pass a deny-only check, and one that quietly stopped denying would pass a
page-only check. The person who introduced the split verified it this way.
## Operator CLI login
`swarmctl login` logs an operator in as their own account, from any
machine, through the public `swarmctl` client authelia always registers: no
secret, authorization code with PKCE, a loopback redirect, and a refresh
token. Its access token carries `authelia.bearer.authz` with the audience
`https://<swarm>/api/`, so the gateway admits it at the controller's API and
nowhere else, under the same access rule as a browser session there.
Revocation is per account: once authelia stops accepting the account, the
gateway refuses its token too. See [`swarmctl/README.md`](../../swarmctl/README.md#remote-use-swarmctl-login).
## What this doesn't do
- **It doesn't disable local login — except on Grafana.** Forgejo and
tuwunel keep their password databases and gain a second door: an identity
provider that can take a service offline when it hiccups is worse than one
with two ways in, and making authelia the only path there is a separate,
reversible switch (tuwunel's `login_with_password`, forgejo's own setting).
⚠️ **Grafana is the exception, and not a reversible one.** Its module sets
`auth.disable_login_form` unconditionally: its vhost is on the public
gateway, so a password box there would be a way in whatever the
deployment. The form is all this hides. Grafana's API still takes HTTP
basic auth for the built-in `admin` account, whose password is a random
one minted into the swarm secret store, not the upstream `admin`. A
dead provider locks out the SSO accounts but not that one: an operator
reads the password from `swarm/services/<grafana client id>/admin/password`
in the store (or, as root on the Grafana host, from
`/run/swarm-grafana-secrets/admin-password`)
and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana)
and [`credentials.md`](credentials.md). SSO logins are the everyday way in, which
is why `swarm.grafana.oidc.role` defaults to `Admin`.
The three secrets Grafana reads live on the host's tmpfs, so after a reboot
Grafana stays down (no login, no dashboards) until the secret store
answers; the delivery units retry and start it with no manual step.
- **It doesn't provision users.** `swarmctl` creates and destroys agents
continuously today, so the subject set belongs to a program rather
than to a config file.

1092
docs/swarm/ui.md
View file

File diff suppressed because it is too large Load diff