mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
9 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

682
docs/2-handbook/swarm/README.md Normal file
View file

@ -0,0 +1,682 @@
# The swarm
The **swarm** is where things live: agent identities and accounts, secrets,
the job graph that creates and places agents, telemetry, and the UI you
drive it all from. **Hives are the substrate** — NixOS hosts that run agent
containers on the swarm's behalf. Every hive belongs to a swarm; a single
host is a swarm of one.
This page is for the operator. It covers the control plane, how a hive
joins the directory, and how hives and agents report upward. The steps for a
fresh swarm are in [`setup.md`](../../1-getting-started/setup.md); the all-local
config is the [README quick start](../../../README.md#quick-start-an-all-local-swarm).
Option reference: `services.hyperhive.swarm.*` (swarm-wide facts, identical
on every host) and `services.hyperhive.deploy.*` (whether _this_ host runs
grafana, the controller, authelia, …) →
[options reference](https://hyperhive.darkest.space/options/), or
`nix build .#docs-swarm` / `.#docs-deploy`.
## Where each piece lives
- **control plane** — `swarm-controller`: hive directory, agent roster, job
graph, agent creation. → [below](#swarm-controller)
- **scheduled prompts** — named prompts for one or more agents at set times,
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
over its own. → [`inference.md`](../../4-implementation/swarm/inference.md)
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
over its hive's. →
[`coordinator.md`](../../4-implementation/scheduler/coordinator.md#container-resource-limits)
- **review classifier** — checks a PR against its repo's own yes/no rules
and posts the result as a review. →
[`review-classifier.md`](../../4-implementation/swarm/review-classifier.md)
- **notification router** — forge events turned into todos on the agents
they concern: assigned, unassigned, cleared on close; and the classifier's
review requested on a push. →
[`notify-router.md`](../../4-implementation/swarm/notify-router.md)
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
`admins` only. → [`ui.md`](ui.md)
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
stack, each on whichever host you put it. → [`services.md`](services.md)
- **SSO** — the secrets authelia generates and how each reaches its reader.
→ [`sso.md`](sso.md)
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
every credential the swarm holds, who mints it, who reads it, how it renews
and where it lives → [`credentials-reference.md`](credentials-reference.md) ·
how the store comes up, who writes its grants and how it unseals →
[`bao.md`](../../4-implementation/swarm/bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer. → [`ca.md`](../../4-implementation/swarm/ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint,
`swarm.snapshotStore.{address,port}`. →
[`snapshot-store.md`](../../4-implementation/networking/snapshot-store.md)
## Swarm controller
`swarm-controller` is the swarm's control plane: it holds the hive directory,
the agent roster and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent,
the step it stopped at and the error, which the Grafana alert rule
`hh-agent-identity-mint-failed` fires on. The agent is still created, without a
store identity, and the controller's five-minute credential pass mints one for
it (also when only the certificate exists and its cert-auth role is missing) as
long as the agent isn't `Destroyed`. A mint that runs after a hive declares the
agent `Destroyed` logs `identity mint skipped` and writes nothing.
`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
runs the controller.** A swarm has one controller, so enabling it states a
fact about swarm topology, not about whether this host runs a hive.
What it serves, why it's a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../../swarm-controller/README.md).
### Per-hive status (`GET /api/hives/status`)
One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.
A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.
| freshness | what to do about it |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive removed from the roster but still running |
The snapshot is the hive's `/health/ready` document plus `capabilities`, a list
of names for what the hive's release can do. Two exist: `tolerant-permissions`, and `no-config-checkout`, which
[turning an archetype agent into a custom one](archetypes.md#turn-into-custom)
needs. A hive on an older release lists none.
The snapshot also carries `egress_deny_applied`, the revision of the egress deny
list the hive last rendered into its proxy (`0` when the swarm has none). A hive
on an older release, or one that hasn't read the list yet, leaves it out; see
[the egress proxy](../../4-implementation/networking/network.md#egress-proxy-squid).
Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload. A roster hive's row
carries `dashboard`, the host its dashboard answers on; the swarm UI's
hive list links there.
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.
### Making a hive report
Two options on the **hive**, plus its store identity. They sit in two
namespaces, because the queue's URL is a fact about _this machine_ and the
token endpoint is the swarm's single address:
| option | what to set it to |
| ---------------------------------------------- | --------------------------------------------- |
| `deploy.hive-controller.statusPublish.natsUrl` | `tls://<swarm.nats.domain>:<swarm.nats.port>` |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). Authelia's mint, in its
container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config`
warning and tries again, waiting 5 seconds after the first failure and doubling
up to 5 minutes. It starts publishing as soon as a read succeeds, with no
restart.
The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.
The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue
host without the store gets no default, even when it holds the hive's own
reader `deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.natsCommonName`, place it, and name it in both options. Until you
do, `swarm-bao-nats-tls` fails naming both whenever the queue's certificate is
missing or due for renewal.
The queue host's gateway also serves the queue's metrics, at
`https://<swarm.nats.domain>/metrics`, so a swarm collector on any host
scrapes them. Only the collector's authelia client gets through; any other
session gets `deny`, and every other path on that name is 403. With the
gateway's default self-signed TLS, that vhost serves the queue's own
certificate, so a host whose only swarm service is the queue needs no
swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.
The endpoint answers **503** when this host has no swarm queue
configured, or has one and can't read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
can't see. The body says which. Status survives a controller restart:
it's stored in the queue, not in the daemon.
### Giving the agents the queue too
An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:
| option | what to set it to |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
It reaches each container at start, as `HIVE_AGENT_NATS_URL` in
`/etc/hyperhive-hive-facts`, so no agent build carries it. On every hive,
`agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. The option takes one
`<scheme>://<host>:<port>` (a hostname or IPv4 address), not an IPv6 literal or a
list: evaluation refuses anything else. On the queue's
host the name resolves inside a container to the bridge address, where the
firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.
**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:
- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits. The error reads
`no swarm queue connection; exiting, since this agent cannot run without one`,
and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.
Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading
the credential from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for a `swarm-logs` built before it read the agent's
own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.
<details><summary>What an agent publishes over the queue</summary>
What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. The queue keeps the rows in
the stream `term-main-<agent>` for 24 hours or 64 MiB, the same limits and the
same creator as the subagent stream below, so a swarm terminal that attaches
later replays the newest of them. The two streams hold disjoint subjects.
An agent's **subagents** publish their terminals too, from the agent's
subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`,
classified the same way. The queue grants that family to the agent's own queue
credential alone, so the daemon reads it from the store under the agent's store
identity, exactly as the harness does, and publishes nothing without it. The
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
whichever comes first, dropping the oldest rows once either limit kicks in so
a publish never fails because of it, and the swarm lists an agent's subagents
from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for
every agent a hive's wanted state names, within a minute of the agent
appearing there. The agent's grant is publish on its own
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
subjects and limits are the controller's and never the agent's. Subagents
publish output only and read nothing.
The queue would refuse a row too large for its `max_payload` outright and
take the connection down with it, so the harness drops such a row's body before
sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.
The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model and effort (`model`, `effort`, where each comes from as
`model_source` and `effort_source`, and the resolved id the last turn
actually ran on), the
context budget and the last turn's context and cost token blocks,
`agent_state`, and `needs_login`: `true` while the agent's claude has no
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.
`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
spellings. ⚠️ From inside the container only two of those four are sayable: the
harness reports `up`, or `paused` when the pause marker is present. `offline` and
`destroyed` are hive-c0re's observations — a stopped agent publishes nothing and
a destroyed one doesn't exist — so a view that needs the full four-state picture
takes them from the `agent-status` bucket and uses this subject to sharpen the
rest.
Headers go out **on transition**: the harness rebuilds the header whenever its
event bus moves and publishes when the result differs from what it last sent.
That's the whole point of the subject — the `agent-status` bucket already
republishes once a minute, which is far too slow for "is this agent thinking
right now." A core subject keeps nothing for a late subscriber, so the harness
also re-sends the current header once a minute: a subscriber attaching
mid-idle sees it within that minute. The swarm's stream sends a new client the
latest header it holds first, when that's under 150 seconds old. A renderer opens with the bucket's
snapshot and lets this stream refine it.
Swarm-side, `GET /api/agents/<name>/state/stream` relays that subject as SSE,
resolving the agent's hive at request time exactly as the terminal stream does.
The payload passes through opaquely. Separately, the controller keeps each
agent's latest `model`, `effort` and their sources for
`GET /api/agents/<name>/inference` ([`inference.md`](../../4-implementation/swarm/inference.md)), and each
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key of that bucket. The harness writes
once per start, because a config change reaches an agent by restarting its
container. An agent with no icon deletes its key. Swarm-side,
`GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.
The fourth is its **`AGENTS.md`**, which goes into the `agents-md` bucket under
the key `<agent>`, writable by the agent's own credential alone
(`$KV.agents-md.<agent>`). The swarm reads that copy and edits the file through
the agent; [An agent's AGENTS.md](#an-agents-agentsmd) has the details.
The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>`, its operator-input subject `$SWARM.input.<agent>` and
its command subject `$SWARM.control.<agent>`, and nothing else. Besides the
subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies.
Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
sets as their inbox prefix. A KV watch or key listing delivers there as well. A
hive may also subscribe to `$SWARM.knowledge` and its own `$SWARM.deploy.<hive>`,
so it can't read another hive's deploys or any agent's terminal. The controller
may also subscribe to `$SWARM.term.<agent>`, `$SWARM.term.<agent>.sub.<subagent>`
and `$SWARM.agent-state.<agent>`, the streams it relays to swarm-ui. A hive or
controller that connects with the library's default `_INBOX.<nuid>` prefix gets
no replies, so every request it makes times out.
</details>
### An agent's AGENTS.md
The file in the agent's state dir, `state/AGENTS.md`, is the only copy that
counts, and only the agent writes it: itself, or its harness applying an edit
from the swarm. No hive is in either path.
**Reading.** The harness publishes the file's text and its revision, the git
blob id that `git hash-object` prints for it, at start, within ten seconds of
any change, and straight after an edit it applied. An agent with no `AGENTS.md`
publishes `""`. `GET /api/agents/<name>/agents-md` answers `{content, revision}`
from that copy, so it works while the agent isn't running. A 404 means the agent
hasn't published one: its harness hasn't run since it began to.
**Initial text.** `POST /api/agents` and `swarmctl agent create --agents-md <file>`
take an optional `agents_md`, the text the file starts with, up to 64 KiB; a
longer text is a 400. The controller stores it under the key `<agent>` of the
`agents-md-initial` bucket, which only it writes, before it queues the creation.
At start the harness reads its own key, with one direct get on
`$JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>`, and
writes the text to `AGENTS.md` only when no file exists there. It does this
before the first mirror publish and the first turn. An existing file is never
replaced, so a restart or a re-create keeps the agent's edits; a later change
to the file never reaches the key. Destroying the agent deletes the key. When
the harness can't read the key it starts without the text and logs a warning.
The text holds no secret, so nothing goes through the secret store.
**Editing.** `PUT /api/agents/<name>/agents-md` takes `{base_revision, diff}`.
The diff is a unified diff that names `a/AGENTS.md` and `b/AGENTS.md`, made
against the text at `base_revision`. swarm-controller sends it as a request on
the agent's command subject `$SWARM.control.<name>`, as it sends a cancel. The
harness refuses it when the file isn't at `base_revision`. Otherwise it runs
`git apply --check` and then `git apply` on a copy of the file, and writes the
result over the file only when both pass:
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| 200 | applied; the body is `{revision}`, the file's new revision |
| 409 | the file isn't at `base_revision` any more; nothing applied, and `revision` in the body is the one it's at |
| 422 | `git apply` refused the diff, or it touches a file other than `AGENTS.md`; `detail` carries git's message, and the file stays as it was |
| 503 | the agent isn't running or isn't connected to the queue |
| 504 | the harness didn't answer within ten seconds, so whether the edit applied is unknown |
| 500 | the harness couldn't read or write the file or run git, or the queue failed the request |
The harness refuses a diff that sets any file mode other than `100644`, so an
edit can't turn `AGENTS.md` into a symlink. `PUT` is operator-only, behind
`Remote-User` like cancel: whoever can call it can rewrite the agent's
instructions, so don't widen who holds it without revisiting this.
An edit needs the agent running, and a read of a stopped agent shows the text
it last published. A harness older than this route doesn't know the command and
drops it unanswered, which reads as a 504. `swarmctl agent agents-md get` and
`apply` call the two routes.
### Swarm-wide forge objects
The controller also keeps the forge objects that are one per swarm, not
one per hive. It ensures them at start and every five minutes after
(`swarm-controller/src/forge/objects.rs`):
- the orgs `agent-configs`, `internal` and `agents`, plus each mirror's
owner org;
- the empty `operators` merge-gate team in `agents` and `agent-configs`;
- the `main` merge gate on every `agent-configs` repo: merge and approval
whitelists = the `operators` team, and no user. The controller leaves a
repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
host (with the `actions/checkout` and `actions/upload-artifact` ones
`deploy.forgejo.actionsFromSelf` adds);
- `internal/docs` (private) and `internal/knowledge` (public, with a
seed `README.md` while empty);
- the `agent-configs` org avatar
(`deploy.swarm-controller.configOrgAvatarPng`).
A pass that can't finish logs a
`warn` line per object plus `swarm forge objects: pass incomplete` in
`journalctl -u swarm-controller`, and retries on the next tick. While the
controller is down the objects stay as they are.
The controller gives each agent **read** on `internal/docs`. A `GrantDocsRead`
node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the
store's `hive-agent-*` roster (`forge/docs_grant.rs`, at start and every five
minutes) adds any agent that isn't one, so an agent whose create-time grant failed
gets it within five minutes. The pass only adds: a roster it can't read, a
forge read that fails, or an `internal/docs` that doesn't exist (checked once per
pass, logged at warn) changes nothing. The grant doesn't depend on the hive
the agent runs on, or on a forge container there.
### Swarm-wide forge webhooks
At startup the controller registers its Forgejo hooks pointing at
itself, all under `https://<swarm.domain>/webhook/forge/`:
- a `push` hook on `internal/knowledge`
- a `pull_request` hook on the `agent-configs` org
- an instance-wide `push` system hook, for commit metrics
- an instance-wide `pull_request_review_request` system hook, which starts
the [review classifier](../../4-implementation/swarm/review-classifier.md#trigger-and-lifecycle)
- instance-wide `issues` + `issue_assign` and `pull_request` +
`pull_request_assign` + `pull_request_sync` + `pull_request_review_approved` +
`pull_request_review_rejected` + `pull_request_review_comment` system hooks, whose deliveries it
keeps in the `SWARM_EVENTS` stream for the
[notification router](../../4-implementation/swarm/notify-router.md#the-path)
The controller **interprets** a delivery and sends hives a specific
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.
The `issues` and `pull_request` deliveries are the exception: no hive gets
them, and the notification router, not the controller, decides what they mean.
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
of its `merge_commit_sha` on the one hive whose wanted state places the
agent; with no such hive, or more than one, the controller deploys nothing. See
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes).
A merge whose `closed` delivery never arrives deploys nothing either; the
operator recovers by redelivering that delivery from the forge hook page,
which re-enters the same handler and re-queues the deploy.
**`internal/knowledge` is on that path.** The controller's is the only
hook on it ([`knowledge.md`](../integrations/knowledge.md) covers clearing a
leftover). A webhook has exactly one target URL, so a second registration
would take delivery away from the first rather than add a recipient.
<!-- vale write-good.Passive = NO -->
**The `agent-configs` org is on it too.** The controller's hook is the
only one that acts on config PRs. Each hive deletes its own
`/webhook/config-pr` org hook on startup, since a route no hive serves
would otherwise sit on the org collecting failed deliveries. Removing
the controller's hook stops forge-UI merges from deploying until its
next start recreates it.
<!-- vale write-good.Passive = YES -->
Nothing to configure. The controller registers the hooks only when this
host also serves the swarm UI vhost — that's what publishes the
endpoint, and a hook the forge can't reach would collect failed
deliveries while looking healthy. The controller generates the HMAC
secret on first start and keeps it
(see [`docs/2-handbook/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)).
To check it's working, push to `internal/knowledge` and look for
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
refused delivery logs `webhook: refused delivery` with the reason.
The controller registers the hooks within five minutes of having both the forge
token and the secret. It keeps trying, at most five minutes apart, for as long
as the forge or the secret store is down. A restart isn't needed to recover.
## Hives: the substrate
- **hive** — one host running `hive-c0re` and its agent containers
(`deploy.hive-controller.enable`). Addressed as `<hiveName>.<swarm.domain>`.
- **swarm** — every hive in `services.hyperhive.swarm.hives`, plus the
shared services and controller. You can qualify an agent as
`agent@hive-domain`.
- **peer hive** — any hive in the directory other than this one. Peers are
_derived_, not declared: the directory lists every hive including
yourself, and `hiveName` says which one you are.
### Hive identity config
```nix
services.hyperhive = {
swarm.domain = "example.com"; # required — the swarm's DNS domain
deploy.hive-controller.hiveName = "pr1ma"; # required on a hive host — this hive's label in it
swarm.name = "constellat1on"; # shared swarm display name (optional)
# required — the directory, identical on every host in the swarm.
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
swarm.hives = {
pr1ma = { }; # this host, per hiveName
lab = { }; # a second hive
edge = { domain = "edge.elsewhere.example"; }; # addressed off-convention
};
};
```
<!-- vale write-good.Passive = NO -->
Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.
<!-- vale write-good.Passive = YES -->
**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.
```
# hive A # hive B
deploy.hive-controller.hiveName = "pr1ma"; deploy.hive-controller.hiveName = "edge";
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
```
One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.
`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`iris@pr1ma.example.com`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.
`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
The env-var chain and `qualify()` semantics:
[`conventions.md`](../../5-project/process/conventions.md) § Hive identity.
Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](../../4-implementation/swarm/ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.
### What the directory feeds
1. **Swarm-wide hive roster** — swarm-controller reads this same
directory and serves it at `GET /api/hives`; the swarm UI's front page
renders it ([`ui.md`](ui.md)).
2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
federates with the peer's matrix server (discovered via the peer's
`.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
Federation validates the peer's TLS certificate against the matrix
**container's** trust bundle, independent of this directory.
⚠️ On a hive with self-signed gateway certificates, the container
also trusts this hive's `trust-bundle.pem`, bound in at runtime and
added to the public CAs; it holds the store's `pki` root, so a peer whose
certificate chains to the same root validates. A peer outside this
swarm's root needs a CA-issued certificate (ACME). Federation firewall
and TLS requirements: [`integrations/matrix.md`](../integrations/matrix.md).
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).
### WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.
#### Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
#### Config example (two hives)
```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
pr1ma = {
domain = "pr1ma.example.com";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
edge = {
domain = "edge.corp";
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = { /* … identical to hive A's … */ };
};
```
#### What the mesh does
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
host (not inside agent containers; containers reach peers via the
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
from `services.hyperhive.swarm.hives`, minus this host's own hive, to
build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
#### NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.
## Cross-references
- [`ui.md`](ui.md) — the swarm UI, the operator surface for "what hives exist"
- [`../networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) — the
swarm's `btrfs receive` endpoint and the `swarm.snapshotStore` option
- [`../process/conventions.md`](../../5-project/process/conventions.md) § Hive identity —
env vars, qualified labels
- [`../integrations/matrix.md`](../integrations/matrix.md) — matrix
federation, TLS cert autogeneration, firewall posture
- [`../networking/gateway.md`](../../4-implementation/networking/gateway.md) — nginx vhosts and
the `.well-known/matrix/` autodiscovery scheme

195
docs/2-handbook/swarm/archetypes.md Normal file
View file

@ -0,0 +1,195 @@
# Agent archetypes
An archetype is a named agent setup: settings to create the agent with, and
a NixOS module its container runs. The swarm UI's create form lists the
archetypes the swarm knows. Picking one fills the form's permissions,
resource caps and inference fields from it, and you can still change each
field before you create the agent.
An agent created from an archetype has no `agent-configs/<name>` repo of its
own. The swarm's `internal/meta` flake builds it from the archetype's module,
so every agent of that archetype runs the same configuration. To change what it runs, change the
archetype. Picking "none" creates a custom agent, with its own config repo. The
swarm can also turn an archetype agent into a custom one later
([below](#turn-into-custom)).
## The built-in archetypes
The hyperhive flake ships these:
| archetype | what it sets |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reviewer` | permissions `messaging`, `inbox`, `execution`: the default set without `meta`. With `execution` it can run `hive-forge`, `git` and other commands. |
| `android-dev` | installs `android-tools` (`adb`, `fastboot`), Gradle and a JDK; default permissions. |
| `reverse-engineer` | sets `services.hyperhive.agent.rea.enable`: installs REA, Ghidra 12.1 and procps, and allows all REA MCP tools; default permissions and the agent's normal memory cap. |
## Where an archetype comes from
A flake declares archetypes in its `hyperhiveArchetypes` output:
```nix
hyperhiveArchetypes.reviewer = {
module = { };
meta = {
description = "Reviews what it is sent.";
permissions = [ "messaging" "inbox" "execution" ];
limits = { memory_max = "4G"; };
inference = { effort = "high"; };
};
};
```
- `meta` is plain data: no derivation, no path, no function. The swarm
evaluates `meta` alone.
- Every archetype needs a `description`. `permissions`, `limits` and
`inference` are optional and take the shapes `POST /api/agents` takes. The swarm ignores
other keys in `meta`.
- `module` is a NixOS module for the agent's container. The meta flake
imports it into each agent created from the archetype.
Today the swarm reads one flake: the hyperhive source the
[builder](../../4-implementation/swarm/builder.md)'s host runs, so only the built-ins appear. The meta
flake has one archetype source too, its own `hyperhive` input, and refuses
an archetype from any other source name.
## How the list stays current
1. swarm-controller asks the builder for the list on the queue subject
`$SWARM.builder.archetypes` once the queue connects, and every 10 minutes
after that.
2. The builder evaluates the `meta` of each `hyperhiveArchetypes` entry on
its own hyperhive and nixpkgs sources, and answers with the JSON or the
last lines `nix` printed:
```sh
nix eval --json <hyperhive>#hyperhiveArchetypes \
--apply 'builtins.mapAttrs (_: a: a.meta)'
```
3. The controller checks each archetype the way `POST /api/agents` checks
its settings and stores the result in the `agent-archetypes` KV bucket,
under the key `hyperhive`, when it differs from the stored one. The list
leaves out an archetype that fails the check, and its name and reason go
into the record's `refused` map and the controller's journal.
4. `GET /api/archetypes` answers from that record, and the create form lists
from it.
A builder host redeploy can change the list. The next pass, within 10
minutes, picks it up. When no builder answers, the controller keeps the
stored list and logs a warning on each pass. A swarm with no builder enabled
shows no archetype select.
The stored value:
```json
{
"source": "/nix/store/…-hyperhive-flake-source",
"archetypes": {
"reviewer": {
"description": "…",
"permissions": ["messaging", "inbox", "execution"]
}
},
"refused": { "broken": "limits: …" }
}
```
## Creating an agent from an archetype
The create form sends the picked archetype as `archetype` on
`POST /api/agents`, `{ "source": "hyperhive", "name": "reviewer" }`. The
controller refuses with 400 an archetype the stored list doesn't hold, then:
1. stores the choice in the `agent-archetype` KV bucket, keyed by agent name,
next to the agent's permissions, caps and inference;
2. queues the creation graph without `CreateRepo`, `AddRepoMember` and
`InitAgentConfigRepo`. Identity, forge user, docs read, the mints and the
wanted state stay, and the deploy waits on those alone.
Destroying the agent deletes its key.
Every hive may read any key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-archetype.>`) and write none; only the
controller writes it.
Deploy `swarm-nats-auth` before the hives: every provision reads the
agent's key, and a hive whose queue connection predates the grant gets a
refused read, so its `Provision` nodes fail until it reconnects.
swarm-controller renders `internal/meta`'s `flake.nix` with the
`hive-meta-flake` crate, and reads each agent's archetype from the
`agent-archetype` bucket to do so. For an archetype agent the render:
- adds no `agent-<name>` input and reads no config repo, and lists the agent
as `mkAgent { …; archetype = { source = "hyperhive"; name = "<name>"; }; }`;
- makes `mkAgent` import `hyperhive.hyperhiveArchetypes.<name>.module` in
place of the agent's own flake, and set `HIVE_ARCHETYPE` in the harness
environment.
The swarm builds from `internal/meta` alone and the hive deploys that closure. The hive reads the agent's key when it
provisions the agent and on every rebuild, and keeps the last value read in
`/var/lib/hyperhive/swarm-agent-archetypes.json`. When the read fails, a rebuild keeps the stored value. For an
archetype agent the hive also:
- deploys the closure its record declares, as every agent does.
The agent's system prompt says it has no config repo and names its
archetype.
The module comes from the `hyperhive` input that `internal/meta` pins, not
from the hive's own. When that pin predates the archetype, the agent's build
fails with `this hive's hyperhive input has no archetype <name>; update that
input`. Update the `hyperhive` pin in `internal/meta` to fix it.
## Turn into custom
An archetype agent can become a custom agent with its own
`agent-configs/<name>` repo, either from the agent's overview in the swarm UI
(the "turn into custom" button asks first) or with `POST /api/agents/{name}/custom`.
The agent keeps its state, permissions, caps, inference settings and accounts.
Its configuration stays the same, except that it no longer has `HIVE_ARCHETYPE`
in its environment. Keeping the config current is then up to the repo's owner.
Nothing here undoes it.
The controller refuses with 409 an agent that no hive or more than one hive
places, and one whose hive doesn't list the
`no-config-checkout` capability (see below). Otherwise it queues a job graph:
1. `CreateRepo`, `CreateForgeUser` and `AddRepoMember`: the repo and the
agent's access to the repo, as for any custom agent. Each is a no-op for what
exists.
2. `SeedArchetypeConfig` commits two files to the repo. It copies nothing from
the archetype:
- `flake.nix` declares the archetype's source as an input, at the URL
`internal/meta` holds for that source today (a pinned rev, which is the input's only pin: the repo has no lock, and a source meta doesn't pin to a commit fails the node), and imports
`inputs.<source>.hyperhiveArchetypes.<name>.module` next to `./agent.nix`;
- `agent.nix` is an empty module for the owner's edits.
The node leaves a repo whose `flake.nix` already is this file alone, whatever
URL it pins. Any other `flake.nix` fails the node.
3. `DropArchetype` deletes the agent's `agent-archetype` key, only after the
seed and the member grant succeeded. A failure before it leaves the agent
archetype-built and working, and you can send the same request again.
4. `UpdateMeta`, then the build and the deploy, as for a merged config PR.
If the build fails, the agent is custom and has no key. Sending the request
again then queues only this step, as the earlier ones have run.
When the source is `hyperhive`, the meta flake sets
`agent-<name>.inputs.hyperhive.follows = "hyperhive"` for an agent whose
`flake.nix` or `flake.lock` declares a `hyperhive` input, so the lock holds one
hyperhive, and the module comes from the pin `internal/meta` holds. Any other
source keeps the pinned URL from its seeded `flake.nix`, and the meta lock
records that rev. The `follows` applies to every custom agent whose `flake.nix`
declares a `hyperhive` input, not only converted ones. A `hyperhive` node left
in a lock doesn't count.
### Which hives may convert
A hive that keeps a checkout of a custom agent's config would create one for
the converted agent, under the container's running mount. A hive that lists
`no-config-checkout` under `capabilities` in the status it publishes (the
document `GET /api/hives/status` serves as `snapshot`) keeps none: it deploys a
custom agent's closure from the swarm alone, and the agent clones its own repo from
the forge. The controller reads the agent's hive's latest status and refuses
unless it's `fresh` and lists the capability; an older hive lists none. Deploy
the release to every hive that places archetype agents before converting theirs.

104
docs/2-handbook/swarm/claude-login.md Normal file
View file

@ -0,0 +1,104 @@
# Claude login from the swarm UI
An operator logs an agent in to claude from the agent's card in the swarm UI.
No hive is in the path: the login runs in the swarm's `swarm-claude-login`
service, the credentials go to the agent's own leaf in the secret store, and the
agent's harness installs them from there.
Where it runs and how it's enabled: [`services.md`](services.md#claude-login-minter).
The service runs the `claude-code` that its host's
`services.hyperhive.deploy.hive-controller.claudeCodePackage`
pins for agents, and its container's own nixpkgs' where the host pins none.
## What happens on a login
1. The agent publishes `needs_login` in its turn-state header, and its card
shows **login needed** with a **log in** button.
2. **log in** opens a dialog, which calls
`POST /api/agents/<agent>/claude-login`. swarm-controller relays the start as
a request on the queue subject `$SWARM.claude-login.request`.
3. The minter runs `claude auth login` with `HOME` and `CLAUDE_CONFIG_DIR` in a
fresh scratch directory, over plain stdio pipes, and answers with the first
`https://` URL it prints. The dialog shows that URL.
4. The operator opens the URL, signs in to the agent's own claude account, and
pastes the code the page shows into the dialog, which posts it to
`POST /api/agents/<agent>/claude-login/code`.
5. The minter writes the code to claude's stdin and waits for it to exit. It
stores the `.credentials.json` claude wrote at
`swarm/agents/<agent>/claude-credentials` (field `value`), as a new version
over any stored before, and removes the scratch directory.
6. The agent's harness, while it waits for a login, reads that leaf every 30
seconds under the agent's own certificate. When the stored version is newer
than the one it installed last, it writes the file to
`~/.claude/.credentials.json`, records the version, and resumes its turn
loop as it does after a login from the agent's web UI.
A started login waits 15 minutes for its code, then the minter stops it.
Starting another login for the same agent replaces the running one. Each step
has a minute to finish; a refused code fails the login, and the dialog shows
what claude printed.
## Why the harness takes only a newer version
`claude` rotates the refresh token in `.credentials.json` as it uses the
session, and the agent can't write the store. After the first refresh the
stored copy is stale. A harness that installed it whenever the bytes differed
would replace the agent's live session with a stale copy and log the agent
out. The harness therefore compares the store's KV version with the last
version it installed (`claude-credentials-version` under the harness dir):
- it installs a newer version, that is, a new login;
- it leaves an equal or older version alone, even when the agent needs a
login again, so it never puts back a session that has gone bad;
- it reads nothing while the agent has a session.
## Credentials
| Principal | Grant |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `swarm-claude-login`, in its container | `create`, `update` on `swarm/agents/+/claude-credentials`; `read` on `swarm/services/<minter client id>/oidc/client` |
| the agent, under its own certificate | `read` on its own `swarm/agents/<agent>/*`, which covers the leaf |
The minter has no read on the leaf and nothing else under an agent. Every
hive's policy reads `swarm/agents/*`, so every hive can read every agent's
claude session, as it can read the GitHub token.
The `operatorsOnly` authelia rule on the swarm domain, not the controller,
keeps these routes to operators: the controller checks only that the gateway
named a user. Without that rule any account authelia admits could store its
own claude session as any agent's, so no agent account may exist before the
rule reaches the swarm.
The minter refuses a step whose agent name isn't an identifier before it
makes a directory or runs `claude`.
## Caveat
The installed-version record lives under the harness dir and the credentials
file under `~/.claude`. If only the record survives a rebuild, the agent
ignores the stored version until the operator logs it in again, which writes a
newer one.
## What stays
The agent's own web UI login (`/login`) still works, unchanged. It goes away
once the operator has confirmed the swarm login works.
## Observability
The minter pushes OTLP metrics through its container's forwarder to the swarm
collector, labelled `service.name=hyperhive-claude-login`. With no
`OTEL_EXPORTER_OTLP_ENDPOINT` it exports nothing. The client-certificate
gauges (`bao.client_cert.*`) come with them.
| Metric | Unit | Kind | Attributes |
| ----------------------------------------- | ---- | --------- | --------------------------------------------------------------------------------------------------- |
| `hyperhive.claude_login.steps` | — | counter | `step`: `start`, `code`, `cancel`, `invalid`; `outcome`: `started`, `stored`, `cancelled`, `failed` |
| `hyperhive.claude_login.step.duration` | `s` | histogram | `step` |
| `hyperhive.claude_login.sessions.expired` | — | counter | — |
| `hyperhive.claude_login.sessions.active` | — | gauge | logins waiting for a code, as of the last step |
`invalid` is a request that didn't decode or named no agent. A `start` spawns
`claude auth login` and a `code` stores the credentials, so those two are the
slow steps. `expired` counts logins stopped because their code never came.

189
docs/2-handbook/swarm/credentials-reference.md Normal file
View file

@ -0,0 +1,189 @@
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
# Credentials: the per-secret reference
Every credential in the swarm's secret store, in one table: who mints it, who
reads it, and whether it renews on its own. The overview and the rules a
credential follows are in [`credentials.md`](credentials.md); this page is
the detail, so it changes whenever someone adds a credential or moves its minting.
The second half, [where each secret lives on disk](#where-each-secret-lives-on-disk), maps the
files that exist today.
In the table, **automatic re-mint** is something replacing the stored value
without an operator. **Automatic re-pull** is the reader picking up a replaced
value without a restart. ✅ means it happens today, ❌ means it doesn't, and
the rest of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice render` in `hive-matrix`, when the store answers and holds none; before that, the tokens an earlier version kept in the container, stored unchanged | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` render the new pair and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read once; a store down at start is re-read in the background until it answers |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation, and by the renewal pass when none is stored or its cert-auth role is missing | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity; minted when none is stored or its cert-auth role is missing, for agents not `Destroyed` | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/observability` | `swarm-controller`, when an operator sets, clears or creates an agent's permissions with `logs` or `metrics` in the old or new list, and when it destroys the agent; the leaf stays absent for an agent that never held either | the agent (read-only, its own prefix); `swarm-logs-auth` and `swarm-metrics-agent-auth`, which deny when it's absent or unreadable | ✅ the controller rewrites it with the permission, and every five minutes makes it equal the record's | ✅ the stores honour a changed grant for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/gitlab/<label>` | `swarm-controller`, when you link a GitLab account through the API | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes; a deleted account leaves the agent's `glab` configuration on the next run |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/backend-env` | `hive-agent-backend-env`, in the agent's container, from the agent's `backendEnvironmentFile` when the leaf holds nothing (field `env`, the file's exact text); or `swarm-controller`, when you set the API-key environment in the swarm UI | the agent | ❌ nothing replaces it without you; a save in the swarm UI overwrites it, including text an agent wrote into an empty leaf; the agent's file stays as it was | ❌ the file in `/run` refreshes every 2 minutes, and the harness and the subagent daemon load it at their next start |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/agents/<agent>/proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds |
| `swarm/hives/<hive>/squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute, and while they can't reach the store they keep accepting the value they last read, for an hour at most; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, to copy it and to check narinfos | ❌ write a new token | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ❌ as the URL. A cache recreated with a new key leaves the stored one stale: every hive then reports `bad-signature` | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->
## Operating notes
The name `main` belongs to the swarm's own homeserver account, which the
controller mints, so the controller rejects any account you link under it.
`swarm-controller` checks every forge token itself: it looks at every agent
that has a store identity at start and every five minutes. For any whose
stored token is missing or stale it creates the forge user if there isn't one,
then mints the token. To check one agent now:
```sh
swarmctl agent mint-forge-token <agent>
```
An agent without a store identity gets no swarm token and keeps using the
`forge-token` file in its state dir, if it has one; re-create it with
`swarmctl agent create` to give it one. The swarm leaves an agent it already
declares as it is. For an agent with no declaration, such as one only its hive
knows about, `agent create` declares it `paused`; set it `up` from its card in
the swarm UI.
⚠️ **Re-create existing agents before deploying a hive-side change that makes a
container require a credential it may not have.** A container whose store
identity is absent doesn't start, so create the identities first, setting any
newly paused agent `up` as above.
**Revoking an agent's queue credential.** Declaring an agent `destroyed` —
`PUT /api/hives/{hive}/agents/{agent}/state` — deletes
`swarm/agents/<agent>/queue` and every version it held. An open queue
connection closes within an hour. An agent recreated under the same name gets a
fresh secret. It leaves the agent's mTLS leaf alone. If the store refuses the
delete, `swarm-controller` logs an `error` naming the agent and the teardown
continues; declare the agent destroyed again to re-run the revocation.
A failed store login retries every 30 seconds for a day. When a later attempt
lands a changed value, the service that loaded it restarts. If the fetch unit
can't compare file times it logs `cannot compare the mtime of <path> … —
leaving <unit> as it is` at `err`, and the service keeps the old value until
the next timer tick.
## Where each secret lives on disk
One row per secret, by the place that owns it: who generates it, where the file
or store path is, and what an operator of a hive elsewhere places. How the
pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.md).
### Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store is down, refuses the login, or fails the read of a leaf (a refusal, a timeout, an error), the unit runs on the files, logs the store's error, and writes nothing to the store; the unit seeds or mints only a leaf the store reports as absent. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting |
| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services/<grafana client id>/secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk, unused once the store holds the key. A store that's down or doesn't grant the leaf fails the unit, and Grafana doesn't start | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it |
| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 until the store answers: the controller re-reads it in the background, at most five minutes apart, and registers the forge hooks once it has the secret |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once. An attempt that fails or runs past 30 seconds retries after 5 seconds, then at growing intervals up to 5 minutes | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the fetch fails fast while the store is down and retries with a delay that grows to 5 minutes; the attempt that succeeds starts the collector, which isn't part of the container's boot, and it resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these leaves has
a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the cache's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, `swarm-bao-grafana-secret-key`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` and `swarm/services/<grafana client id>/secret-key` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services/<runner forwarder client id>/oidc/client` |
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The option names are `deploy.bao.<pair>`; the common names, what a missing leaf
costs and how to place a leaf by hand are in
[per-principal identities](credentials.md#per-principal-identities).

639
docs/2-handbook/swarm/credentials.md Normal file
View file

@ -0,0 +1,639 @@
# Credentials: the target shape
<!-- vale write-good.Passive = NO -->
The swarm's credential store is bao. This page is the overview of how the swarm
treats credentials; [`credentials-reference.md`](credentials-reference.md) lists
each one: who mints it, who reads it, and how it renews, with the notes for
operating them. It describes the target, not what's on disk today; the second half of the
reference maps [the files that exist right now](credentials-reference.md#where-each-secret-lives-on-disk).
<!-- vale write-good.Passive = YES -->
**Store secrets stay off disk.** A process pulls a value from the store at
runtime and holds it in memory only.
**The mTLS client certificate is the only credential on disk.** Each
identity (an agent, a hive, a swarm-level service) has one. It's what logs a
principal in to the store, so you place a hive's or service's certificate by
hand. The nix option for it takes a path to the certificate, never its bytes,
because a literal in a nix expression lands in the world-readable nix store.
A hive hands each agent its certificate; the agent pulls everything else
itself.
**Each credential has a minter, a reader and a renewal story.** Who mints it, who
reads it, and whether it renews without an operator are what the
[reference table](credentials-reference.md) gives for every credential, with the
operating notes that go with them.
## Where a secret lives
A swarm generates its credentials in three different places and reads them
in a fourth, so "where does this file go" has a different answer per deployment.
This page is that answer, one row per secret.
Two rules run through all of it.
**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.
The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
the queue callout's user key, reaches every client that connects,
so the store is a perfectly good place for it.
**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.
### Topologies, by who places secrets
Read every row of the [reference tables](credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level secrets
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
test worth applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
each with the secret publisher's identity and writes it to the file authelia loads. A store
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
database, and a new issuer key invalidates every token issued. A container without the
publisher's identity generates the files as before.
The same unit mints the client secrets, straight into the store. It reads
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from
`swarm/services/swarm-authelia-bridge/oidc/client` into
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
answer keeps the bridge down while it retries. A container without the publisher's
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.
The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
### The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Hive-level secrets
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
mint in authelia's container has stored the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the mint in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.
All agents share one policy, `hive-agent`, which the agent's cert-auth role
attaches and no other. It states the rule with `{{identity.entity.name}}`
where an agent's name would go, so the rule is one document and a change to it
is one write. `swarm-controller` writes it at agent creation and at its own
start.
For that the controller names each agent's identity entity after the agent: at
each certificate mint or renewal, before it writes the agent's cert-auth role,
and for every agent at its own start. It reads the cert mount's accessor and
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
entity a first login created under a generated name. At a mint or renewal this is
a precondition: when the store refuses a call, or the alias sits on any other
entity, the mint fails and the role stays as it was, so a live agent keeps
working until an operator resolves it. The start pass only logs the same
failures, because it writes no role. The controller needs the `identity/` and
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.
A cert-auth role that predates the shared policy may still name a policy
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
objects stay in the store and keep granting what they did, to a token that
already carries them and to a role not yet rewritten. A live agent's role is
rewritten at its next certificate renewal.
A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and rewrites the roles to attach it. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. The leaves, their option names
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](../../4-implementation/swarm/bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token.
The other roles issue service tokens that live an hour (`token_ttl=1h` and
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
agent and hive roles the controller writes through
`SecretStore::write_cert_role`, which takes the same value from
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
expires, and the mount default of 32 days leaves one lease per login for a
month; at the 17 thousand logins a day this swarm made, that came to half a
million leases. A reader logs in per operation and holds the token for seconds.
The longest hold is the controller's forge work, which stops after two minutes.
A client that keeps a `SecretStore` handle longer than an hour must log in
again, because nothing renews the token. A role the controller writes takes the
TTL at its next write: a hive's at the controller's start, an agent's at its
next certificate mint or renewal. At its start the controller also reads every
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
differs from the TTL, writes those two fields and no others. The store keeps the
fields a write leaves out, so the role's certificate, allowed names and
policies stay. A role already at the TTL isn't written. The start logs one line
with `rewritten`, `already_current` and `failed` counts, and a role that fails
gets a warning that names it, then the pass moves on. Tokens issued before
expire on their own; the controller revokes nothing. At that rate an hour keeps
about 700 tokens live.
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
it, and a deployed store never rewrites it, so the longer TTL would not reach
it; the granter logs in and finishes within seconds.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
### OIDC login
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
`url`, `cache` and `public-key`). That shows the key tree and each key's
versions and timestamps, and no value: the store refuses every `secret/data/`
read. The writes are the leaves the operator fills in by hand: the review
classifier's keys, and the cache's tokens and facts, which configure an external
cache or replace the in-swarm cache's own. The cache's host units create those
leaves only while the store holds none, so a saved value survives their next run
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
Anything more needs a token.
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
`services` > `attic`, pick the leaf and choose **Create new version**. The
form starts empty because the login can't read the current value: type the
key `value` and the new value, and save. A new version replaces the whole
secret, so `value` is the only key to enter. `bao kv put` with the same login
works too.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../../1-getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.
## Adding a secret
State three things, in the row you add to the [reference](credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.

153
docs/2-handbook/swarm/schedules.md Normal file
View file

@ -0,0 +1,153 @@
# Scheduled prompts
A scheduled prompt hands one or more agents the same text at set times: once,
or every interval from a start time. Each prompt has a name, its title. The
swarm keeps every prompt, and each agent it names fires it in its own
container, so a fire needs neither the hive nor the swarm controller to be up
at that moment, only the agent.
Operators manage them with `swarmctl schedule` on the swarm controller's
host. Agents can't create, change or remove one. For a one-off wake an agent
sets itself, it uses `remind`, which stays inside its container.
## Managing them
```console
# swarmctl schedule add --name standup --agent scribe --agent iris \
--body "post the standup summary" --at 2026-10-05T09:00:00+02:00 --every 1d
added schedule kqzpmwab
kqzpmwab "standup" every 1d from 2026-10-05T07:00:00Z next 2026-10-05T07:00:00Z agents: iris, scribe | post the standup summary
# swarmctl schedule list
# swarmctl schedule list --agent scribe
# swarmctl schedule edit kqzpmwab --body "post the standup summary to the team room"
# swarmctl schedule edit kqzpmwab --add-agent atlas --remove-agent iris
# swarmctl schedule pause kqzpmwab
# swarmctl schedule resume kqzpmwab
# swarmctl schedule rm kqzpmwab
```
- `--agent` names an agent to fire the prompt in; repeat it for more. Each
must be an agent some hive's wanted state declares.
- `--at` takes an RFC 3339 time with its offset, or unix seconds. Without it
the first fire is now. A time in the past fires its latest past time once,
as soon as each agent reads the prompt.
- `--every` takes a count and a unit, `s`, `m`, `h` or `d`. Without it the
prompt fires once.
- `schedule edit` changes only the flags you pass and keeps the id. `--once`
turns a recurring prompt into a one-shot. `--add-agent` and
`--remove-agent` change who gets it; removing the last agent fails, so
remove the prompt instead.
- `schedule list` shows one row per prompt: id, name, cadence, start, next
fire time, agents, and the first line of the body.
The full flag reference is [`swarmctl-cli.md`](../reference/swarmctl-cli.md).
## When a prompt fires
A prompt's fire times are its start time plus whole intervals. Each agent
works them out from the prompt itself, so:
- **Editing the text or the name doesn't move them.** Neither does changing
the agents. Only a new `--at` or `--every` does.
- **An agent that was down or disconnected fires once.** When it's back, it
fires the latest time it missed and then carries on at the next one. The
earlier missed times don't fire.
- **A restart refires nothing.** Each agent records the last time it fired
for each prompt in its state database, and fires a time only when it's
later than that. An agent removed from a prompt and added back keeps that
record, so it doesn't refire a time it already fired.
- **A one-shot fires once**, even though it stays in `schedule list` until
you remove it. Moving its `--at` to a later time fires it again then.
- **A paused prompt doesn't fire**, and the times that pass while it's paused
don't fire when it resumes. An agent marks those times as passed while it
watches the paused prompt; if the agent was down for the whole pause, it
fires the latest time it missed once, as above.
A fire reaches the agent as a `schedule` loose end with the prompt's text.
The agent's turn loop wakes for it like any other loose end. If the agent
hasn't dealt with the previous fire yet, the new fire replaces it rather
than adding a second one; once the agent dismissed the previous one, the
next fire opens a new one.
Each agent reads its prompts every 10 seconds, so a fire can land up to 10
seconds after its time. A change reaches each agent within the same window,
or within a minute if the controller's first attempt to pass it on failed.
## When a prompt doesn't fire
- **The agent isn't connected to the swarm queue.** It reads its prompts with
its own swarm-queue credential. A harness that can't connect retries for
about five minutes, then exits and comes back; it reads no prompts until a
connect succeeds. See
[Giving the agents the queue too](README.md#giving-the-agents-the-queue-too).
- **The agent hasn't read its prompts since it started.** It fires from the
last list it read; until the first read succeeds it fires nothing. A read
that fails later keeps the previous list.
<details><summary>Implementation</summary>
swarm-controller is the only client that writes either of the two KV buckets
involved.
`schedule-definitions` holds every prompt as one JSON document under the key
`all`. Agents hold no grant on this bucket. Each change re-reads the key and
writes against its revision, so a prompt exists for every one of its agents or for
none of them, and two changes at once don't lose one. The controller leaves a
prompt it can't decode, such as one a newer build wrote, out of the list, with
a warning in its log. An agent's key that already held the prompt keeps it as
it was, so it goes on firing, and no other key gets it. The controller keeps
the prompt in the document as it was, after the others, and won't reuse its
id. A document that isn't an object with a list of `schedules` is an error.
`agent-schedules` holds one key per agent: the prompts naming that agent,
without their names or other agents. A key the controller can't decode counts
as no prompts, with a warning in its log, and the next rewrite replaces it.
```json
{
"schedules": [
{
"id": "kqzpmwab",
"body": "post the standup summary",
"anchor_unix": 1791183600,
"interval_seconds": 86400,
"paused": false
}
]
}
```
`interval_seconds` is absent for a one-shot. After each change the controller
rewrites the key of every agent the change named before or after, and once a
minute it rewrites every agent's key, so a rewrite that failed converges. Ids
are 8 lowercase letters, the same in every agent's key, never reused, and so
never equal to a hive schedule's numeric id.
`swarmctl schedule` calls these routes over the controller's socket:
| method | path | body |
| -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `GET` | `/api/schedules` | —; answers every prompt, sorted by name then id |
| `POST` | `/api/schedules` | `name`, `agents`, `body`, and optional `anchor_unix`, `interval_seconds`, `paused`; answers 201 with the prompt |
| `PATCH` | `/api/schedules/<id>` | any of `name`, `body`, `anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, `add_agents`, `remove_agents` |
| `DELETE` | `/api/schedules/<id>` | —; answers 204 |
Each prompt they answer with has `id`, `name`, `agents` (sorted), `body`,
`anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, and
`next_fire_unix`: the next time on the prompt's grid, as of the moment the
controller answers, `null` when paused or a one-shot whose time has passed.
An agent's own queue credential grants exactly one subject in this bucket:
`$JS.API.DIRECT.GET.KV_agent-schedules.$KV.agent-schedules.<agent>`,
the direct read of its own key. It can't read another agent's key, open or
watch the bucket, or write to it. A KV watch would need a consumer, and a
consumer create subject doesn't name a key, so granting one would let the
agent read every key; the agent reads its key every 10 seconds instead.
The harness (`hive-agent`'s `schedule_timer`) keeps the last fired time per
prompt id in the `schedule_fired_slots` table of its state database, and
pushes each fire as the todo `schedule`/`schedule:<id>` with source
`scheduled`, reopening an acknowledged one.
</details>

609
docs/2-handbook/swarm/services.md Normal file
View file

@ -0,0 +1,609 @@
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. |
| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |
# Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge,
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
log stack. This page says which host runs them and what a hive that runs none
of them configures instead. The all-local quick start sets everything with one
line → [README](../../../README.md#quick-start-an-all-local-swarm).
Two options say where they live, and everything else derives:
```nix
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.deploy.allSwarmServices = true;
```
**`deploy.allSwarmServices` is what "the swarm's shared services run
here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service
sections below don't repeat it, so a service that stops deriving is a
visible difference rather than one more paragraph saying the same thing.
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the
host's `/etc/hosts` entries for the names it serves
(`gateway.localHostsEntry`) and where the secret store's bootstrap
token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived
toggle on its own, and that setting wins.
**Both default to off**, and that's deliberate: a host can't tell
whether it's meant to be the swarm's service host, so this is an
operator saying so rather than something inferred. With them off, a hive
is a _client_ of those services — it configures how to reach them and
runs none of them.
That includes the forge (`deploy.forgejo.enable`). Every swarm needs
one, being the canonical store for the meta flake and every agent's
config repo, so **exactly one host must turn it on**: `singleHostSwarm`,
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A
host that enables its services one by one without any of those runs no
forge.
A hive can't start or stop any of these. `hivectl stop`, `start` and
`restart` act on the hive's agents and its gateway only, even on the
host that runs the swarm's services. The forge and matrix containers are
systemd units on that host (`container@hive-forge.service`,
`container@hive-matrix.service`), and the operator manages them there
with `systemctl`.
A hive that runs none of these reaches each one by name — the forge at
`forge.<swarm-domain>`, for example. Only the host running a service
answers its name from its own resolver, so on a swarm spread over
more than one host, the operator's DNS has to resolve those names to that
host.
<details><summary>Moving an existing hive's forge to the swarm's</summary>
A hive that stops running the forge keeps the old container's state at
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
forge: push anything worth keeping there by hand.
</details>
## Deployment shapes
Those two options are what makes the difference between deployments, so
the shapes worth naming are the ones they produce:
- **All-local.** Everything on one machine:
`singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for
a hive to run agents on. After the first switch, the steps in
[`setup.md`](../../1-getting-started/setup.md) remain.
- **Services on the swarm controller host.**
Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there,
with hives elsewhere. The controller doesn't derive from
`allSwarmServices`.
- **Fully spread out.** One container / VM / machine per service,
somewhere.
**These are a set, not a ladder with a correct top, and hyperhive
supports in-between shapes.** You can set each derived toggle on its own (see
above), which is what makes "all local except X" a configuration rather
than an unsupported edge case. Nothing in hyperhive prescribes a
deployment model, so a doc that treats one shape as the real one and
the others as compromises is wrong about the system rather than
opinionated.
The practical consequence is that **co-location is normal**: in the
first two shapes services share a host by design. Where a specific
service has something to say about sharing a host --- extra
requirements, or a cost worth weighing --- that belongs in the doc for
that service, not here.
## Services
### SSO (authelia)
One authelia per swarm, in a `swarm-authelia` container, at
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
provider, differentiated by roles and claims rather than by mechanism —
there is one IdP and one auth path.
The container sits on the hive bridge in a network namespace of its own and
dials the store, the collector and its own public name through it. It binds
no TCP port: the login server, the metrics listener and the bridge each
listen on a unix socket in `/run/swarm-authelia`, a directory only the
gateway's nginx and the container's processes can open.
- **`deploy.authelia`** — run the container here.
- **`swarm.authelia.url`** — where clients go to authenticate. Present
on **every** hive and the same value on all of them:
`https://<swarm.authelia.domain>`, whether or not this host runs the
container. Set it explicitly when joining a swarm whose IdP is under
another name.
Agent subjects come from swarm-controller's agent-creation job, written
into the users database by `swarm-authelia-bridge`; human ones come from
`swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account).
The bridge listens on a unix socket on the authelia host, and the gateway
publishes it as
`https://<swarm.authelia.domain>/bridge/`, which is where
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
It answers only swarm-controller's OIDC client
(`swarm.controller.queueClientId`) and refuses every other token `403`.
On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one
invalidates data already written (sessions, the encrypted store), so
that's an operator action.
Storage is local sqlite and the notifier writes to a file. Both are
small-deployment choices, and the scope is the justification: redis
buys shared session state across replicas and there is one instance;
SMTP exists to mail humans, and provisioning here is programmatic.
The swarm collector scrapes authelia's own metrics at
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`.
The container's `authelia-swarm-secrets` unit creates each client secret directly in
the swarm secret store, under the secret publisher's own store leaf handed in as
a system credential, and writes only the digest to disk; see
[`credentials.md`](credentials.md). Authelia starts from its digests, so the IdP
comes up while the store is down, and a unit that needs the store retries every
30 seconds. The container's journal forwarder reads its client secret back from
the store under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`credentials.md`](credentials.md) for where authelia
generates and reads each of its keys.
### Metrics (VictoriaMetrics + Grafana)
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
reads it, in two containers at `metrics.<swarm-domain>` and
`grafana.<swarm-domain>`. Two containers rather than one so you can
restart or break Grafana without taking the time-series database with it.
The Grafana container sits on the hive bridge in a network namespace of its
own and dials authelia, the secret store, the collector and both datasources
by name through it. It binds no TCP port: it listens on the unix socket
`/run/swarm-grafana/grafana.sock`, a directory only the gateway's nginx and
the container can open.
They derive together: a store with no UI is unreadable and a UI with no
store is empty. To run one without the other, set it directly:
```nix
services.hyperhive.deploy.victoriametrics.enable = true;
services.hyperhive.deploy.grafana.enable = false;
```
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
<!-- vale write-good.Passive = NO -->
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`credentials.md`](credentials.md).
<!-- vale write-good.Passive = YES -->
**Dashboards.** Grafana provisions these from
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
- `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, and the busiest agents and destinations. **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent.
- `builds`: the container builds hive-c0re ran on every hive, in two tabs. **builds** has the count, the failures and the longest, builds over time by outcome, and a table of each build with its id, agent, kind, outcome and runtime. **output** shows every line of the build picked in the **Build** filter, with its start and finish lines. It reads the lines hive-c0re logs for each build, so a hive-c0re that doesn't log them leaves it empty. Filter by agent.
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
- `review-classifier`: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
- `forge`: the forge, in three tabs. **content** has what the forge holds (repositories, users, issues, releases, discussion volume and the issues by label). **runtime** has the forge process's CPU, memory, goroutines and descriptors. **CI runs** has Actions run time and queue wait per repo and workflow, with a per-PR table; the Repo and Workflow filters apply to that tab.
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
- `collector`: what the hive and swarm OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
- `logstore`, `metricstore`: health of the log store and the metrics store.
- `grafana`: Grafana itself. Dashboards and data sources, alert instances firing, requests by status and their latency, alert rule evaluations and failures, sign-ins, database connections, memory and goroutines.
- `builder`: the swarm builder. Builds finished, by kind and outcome, their mean and 95th percentile time, the queue depth and the wait before a build starts.
- `ci-runner`: the forge's Actions runner. Its uptime and job slots, jobs finished by status, how many slots are in use, job time, and how often it polls the forge for work.
- `matrix`: the Matrix homeserver. Whether its client API answers, how fast, the share of the range it answered, and its database size.
- `notify-router`: the notification router. Forge events received by type and outcome, todos delivered, review requests that failed, time per event, and the backlog of the agents' queue.
A new panel can read "No data" until its source exists: the push and commit
counters on `controller` appear after the first push webhook, the refused
and failure series on `collector` appear once something fails, and the
`review-classifier` panels fill as review requests arrive.
**The `collector` dashboard.** It shows the self-metrics of the hive's
collector and of the swarm's collector, which both carry the collector
binary's default `service.name`, `otelcol-contrib`. The timeseries panels
split by `server.port`: 8888 is the hive collector, 8889 the swarm
collector. Every per-container collector (the secret store, the other service
containers and the agents' forwarders) also scrapes its own endpoint on
loopback and pushes the result up its normal hop, so each has a line of its
own, on its own port: agents all report on 8904, service containers on
8890 to 8903. The series carry `container` (services) or `agent` (agents)
to tell them apart. `scripts/check-dashboard-service-names.sh` runs in CI
and in `scripts/pre-push`. It fails when a dashboard query selects a
`service.name` that nothing emits.
**Where the data comes from.** The swarm's OTEL collector, below.
The `controller` panels come from the controller itself. It exports OTLP
through the gateway to the swarm's collector from whatever host it runs on,
whether or not that host runs the collector.
Neither container is reachable except through the gateway: each listens
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
runs in a network namespace of its own with no interface but loopback.
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it:
- **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource, an agent, or an operator. Grafana sends a
bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
store, and `swarm-metrics-auth` on the store's host checks it against the
same store path. An agent's `swarm-metrics` sends its
`swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. It also needs the `metrics`
permission, or the answer is a 403. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later. When the
store doesn't answer within five seconds, it checks requests against the
last value it read instead of denying, and still refuses a wrong bearer;
it tries the store again after three seconds. A revoked or rotated bearer
stays accepted while the store is down, for an hour after the last read at
most; after that the responder denies again. The store's next answer
replaces the value, and a process that holds none (it started while the
store was down) denies. `swarm-logs-grafana-auth`
is the same binary and does the same.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to
that path; any other session gets `deny`.
The collector scrapes the store's own metrics the same way, at
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
that doesn't run the store still gets them. On this name the gateway admits
only the collector's client to `/metrics` and the push route; any other
session gets `deny`.
### Logs (VictoriaLogs)
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
behind the same SSO as everything else: the swarm's own service containers,
the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
| Option | When you'd touch it |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. A revoked `logs` permission does too. |
| `deploy.victorialogs.grafanaAuthPackage` | The `swarm-metrics-auth` build that runs as `swarm-logs-grafana-auth`. Set on its own: `deploy.victoriametrics.authPackage` doesn't reach it. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `deploy.victorialogs.port` | Only if something else on the services host already claims `9428`. |
It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. The agent must also hold the `logs` permission,
which is off by default; without it the answer is a 403 (`docs/5-project/process/conventions.md`, Permissions). Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`credentials.md`](credentials.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. On this name the gateway admits only the
collector's client to `/metrics` and the ingest route
`/insert/opentelemetry/v1/logs`; any other session gets `deny` on both. The
read side stays open to any authenticated caller.
### Telemetry collector (OTEL)
The swarm's collector receives from every hive's own collector and is the
only process that decides where telemetry goes: it always writes the stores
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
always get both. It holds
the upstream credential, which is why no hive and no agent needs one.
It runs in a `swarm-otel` container on the hive bridge, in a network
namespace of its own. It binds no OTLP port: every receiver (one per hive,
the swarm tier's and one per service forwarder) listens on a unix socket in
`/run/swarm-otel`, a directory only the gateway's nginx and the container can
open, and the gateway forwards to them. The gateway's routes and the collector's
receivers come from one list, so they name the same sockets. The
`swarm-otel-sockets` unit connects to each socket as the gateway's user after
the container starts, then requests every route through the gateway vhost. It
fails if a socket refuses for 30 seconds, or if a route answers `000`, `502` or
`504`, or lacks its own `X-Swarm-Otel-Route` response header, for 30 seconds.
The gateway sets that header, named after the route, on every response from a
route, so a collector that's up but not listening, or an nginx that doesn't
have the route, shows as a failed unit. Any status other than those passes,
the collector's 4xx included. A switch leaves a window of about 20 seconds in which
nginx and the container are out of step: the container stops before nginx
restarts and starts about 18 seconds after it. Senders retry through that window.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a
loopback URL an operator has to redirect. Nothing needs setting on a hive
that doesn't run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `swarm-attic`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter, notification router and
classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store,
authelia, cache, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `attic`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**The homeserver's metrics.** tuwunel 1.9.4 exports no metrics of its own, so
request rates, sync load and federation traffic aren't available. Two series
stand in for them, and neither adds a service:
- The hive collector on the homeserver's host requests
`https://<gatewayHost>/_matrix/client/versions` every 60 s with its
`httpcheck` receiver (`services.hyperhive.otel.httpChecks`). It reports
`httpcheck.status` (one series per status class), `httpcheck.duration` and
`httpcheck.error`, labelled with the URL. The container's forwarder can't
make this request, because `httpcheck` dials HTTP and HTTPS only and tuwunel
listens on a unix socket. The request passes through the gateway, so it
covers the gateway's route as well as the homeserver.
- The `hive-matrix-database-size` timer in the container runs every five
minutes. It measures tuwunel's database directory and pushes the gauge
`hyperhive.matrix.database_bytes` through the forwarder's OTLP port,
stamped `service.name` `hive-matrix`. It exists only where the container
has a forwarder, which needs matrix-ctl's store identity set. A size under
4096 bytes is no database, so the timer logs a warning and pushes nothing.
The swarm collector's own container, `swarm-otel`, has no forwarder: its
collector reads the container's journal itself (receiver `journald/self`) and
exports it to the log store like the host's journal, stamped `service.name`
`swarm-otel` and with no `hive` label. Unlike the host's journal, it goes to the
log store only, never the upstream: the collector writes an upstream export failure to this
journal, and shipping it upstream would fail again. A `filter` processor also
drops the collector's own failure lines that name the store exporter
(`otlphttp/victorialogs`), which can't reach the store anyway; failures of
every other exporter still arrive. The container mounts the host's journal
read-only at `/run/host-journal`, not over its own
`/var/log/journal`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in
[`../scheduler/observability.md`](../../4-implementation/scheduler/observability.md).
### CI runner
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
runner executes pull-request code, so running one is a decision of its own.
It needs neither the forge nor a hive on its host. It reaches the forge at
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
publishes to the secret store, which it reads under its own `ci-runner` leaf.
On a host without the store, that leaf is the one file to copy. Registration,
the container and the threat model are in
[`../scheduler/ci.md`](../../4-implementation/scheduler/ci.md).
The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in
the container's own network namespace. The container's journal forwarder
scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and
pushes the series to the swarm collector under the job `gitea-runner`, so
nothing listens on the bridge. Series, all prefixed `gitea_runner_`:
`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`,
`info` (labels `version` and `name`), `job_total{status}`,
`job_duration_seconds`, `poll_fetch_total{result}`,
`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`,
`report_state_*` and `client_errors_total`. A pull-request job runs in the same
namespace and can read the endpoint, which carries counts, durations, the
runner's name and its version, and no secrets.
### Builder
The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
has the builder's store identity, as for the review classifier below: the host
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It
evaluates unreviewed config PRs and builds them through its host's nix daemon,
from a container with its own network namespace.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
newer push replaces its PR's job while that job is still waiting. The builder
builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a failed build holds that deploy back (`build_meta` job;
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). Those jobs share the one-build slot with
the PR checks.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](../../4-implementation/swarm/builder.md).
### Binary cache
The swarm's attic server, in a `swarm-attic` container on the host that sets
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
the cache's store identity, as for the builder: the host that runs the store has
it. For an external cache, override `swarm.attic.{domain,cache}` and don't set
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
`allSwarmServices`.
The container sits on the hive bridge in a network namespace of its own, and atticd
listens on that namespace's loopback only; the gateway reaches it over a unix socket.
The gateway serves it at `swarm.attic.domain` (default
`attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
its own tokens. Storage is local disk in the container's state.
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
create its signing key, the cache and the push and pull tokens in the secret
store, each only while absent. On a host without the store, that leaf is the one
file to copy. The builder pushes green builds to it. How it works, and the
external alternative, are in [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache).
The container forwards atticd's journal to the swarm collector like the other
service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host
units' lines reach the log store too: through the hive collector's journal
allow-list on a hive host, or through the swarm collector's whole-journal
receiver where the host runs that. A host with neither keeps them in its own
journal.
### CI metrics
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
`allSwarmServices` defaults it on wherever the host has the poller's store
identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to
opt out.
Each pass pushes the run time and queue wait of every newly finished forge
Actions run to the metrics store, through its own route on the swarm collector;
the first pass pushes every run since the first. It reads the forge with the
read-only token of its own `ci-metrics` forge account, under its own
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
copy. What it reads and pushes are in [`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md).
### Review classifier
The review classifier's try-out service, in a `swarm-review-classifier`
container on the host that sets `deploy.review-classifier.enable`.
`allSwarmServices` defaults it on wherever the host has the classifier's store
identity: the host that runs the store has it, and a services host with the
store elsewhere gets the classifier only once it holds a copy of the pair (see
below). Set `deploy.review-classifier.enable` directly to override either way.
The operator writes the key of the runtime the swarm settings choose by hand,
in the secret store's UI under the OIDC login, field `api_key`:
- the API key runtime's Anthropic key at
`secret/swarm/services/review-classifier/anthropic-api-key`
- the ACP runtime's provider key at
`secret/swarm/services/review-classifier/acp-provider`
Nothing writes them for you: until the chosen one exists the service runs and
every run fails with `no API key…` or `no ACP provider key…`. The service reads
the key on every run, so writing or removing it takes effect on the next run,
with no restart. Every run spends money on that key.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `review-classifier` leaf. On a host
without the store, that leaf is the one file to copy, as for the CI runner.
The swarm UI's try-out drives it through swarm-controller; what a run is
and what it costs are in [`review-classifier.md`](../../4-implementation/swarm/review-classifier.md).
### Claude login minter
The claude login minter, in a `swarm-claude-login` container on the host that
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever
the host has the minter's store identity, on the review classifier's terms.
Run one: a login in progress lives in the service's memory between its start
and its code.
An operator logs an agent in from the **log in** button on its card in the
swarm UI. swarm-controller relays the start and the pasted code over the queue,
the minter runs `claude auth login` and writes the credentials to the agent's
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
version newer than the one it installed last.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `claude-login` leaf, and reaches claude's
login endpoints over the network. On a host without the store, that leaf is
the one file to copy, as for the CI runner. The flow and the version rule are
in [`claude-login.md`](claude-login.md).
### Notification router
The notification router, in a `swarm-notify-router` container on the host
that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on
wherever the host has the router's store identity, on the review classifier's
terms. Run one: it handles each issue's events in order.
swarm-controller keeps every `issues` and `pull_request` delivery from its
instance-wide hooks in the `SWARM_EVENTS` stream. The router reads that
stream and queues todos for the agents an event concerns: an agent assigned to
an issue or pull request gets a todo, one taken off it gets that todo rewritten
to say so, and closing it clears them. A push to an open pull request makes
it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos
from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's
back.
It needs neither the forge nor a hive on its host, and holds no forge token.
It reaches the queue and the secret store by name, under its own
`notify-router` leaf. On a host without the store, that leaf is the one file to
copy, as for the CI runner. The flows and the delivery path are in
[`notify-router.md`](../../4-implementation/swarm/notify-router.md).

324
docs/2-handbook/swarm/sso.md Normal file
View file

@ -0,0 +1,324 @@
# Swarm SSO
The swarm runs one authelia, and it's two things at once: the **session
provider** every protected vhost checks (`auth_request`), and — once an
operator declares any client — an **OIDC provider** issuing tokens to
relying parties: the forge and the matrix homeserver.
The second role derives rather than switches:
`services.hyperhive.swarm.authelia.oidc.clients` being non-empty turns it
on. authelia refuses to start with a provider that has no clients, so a
separate `enable` would be a second fact free to disagree with the first.
## Getting in the first time
authelia binds no TCP port: its listeners are unix sockets only the gateway can reach. The **gateway** on the host running it
publishes it as `auth.<swarm.domain>` — vhost, dnsmasq record and TLS
name all follow `deploy.authelia`, so there is nothing to turn on
separately. (Details, including why a client hive must not declare that
vhost: [`../networking/gateway.md`](../../4-implementation/networking/gateway.md).)
**Nobody can log in until the first user exists.** authelia exits at
startup on an empty user set:
```
error reading the authentication database: could not validate the schema:
users: non zero value required
```
While the set is empty this module writes one placeholder subject,
`swarm.placeholder`, that authelia accepts and that can't log in: it's
`disabled: true`, and its password digest has an all-zero key that no
known password hashes to. authelia starts and serves its login page, and
refuses every login. The placeholder stays in `users.yml` once real
users exist, shows up in `swarmctl user list`, and `swarmctl user add`
refuses its name.
⚠️ The step below is **required to finish the install**, not an
optional first-login convenience.
Add the first subject on the host running authelia:
```console
# swarmctl user add mara --display-name Mara --email mara@example.com --group admins
added mara to /var/lib/authelia-swarm/users.yml
password: <generated>
this password is stored nowhere — record it now
```
`swarmctl` generates, hashes, and prints the password once, keeping only
the hash. It reads and writes authelia's `users.yml` directly — it's
the one user store, shared with `swarm-authelia-bridge`, which creates
agent identities in the same file. No restart: authelia watches it. Full
reference: [`../tools/swarmctl-cli.md`](../reference/swarmctl-cli.md).
You can edit `users.yml` by hand, and `swarmctl` will read what you
wrote. ⚠️ It rewrites the whole file on every change, so **comments and
formatting don't survive**; values and unrecognised keys do.
This step stays manual on purpose. Bootstrapping an identity provider
non-interactively means a secret arriving from somewhere — a file, an
env var, a nix expression — and every one of those is worse than an
operator typing one command once.
### Changing a subject afterwards
`user add` only ever adds: on a name that already exists it refuses,
rather than resurfacing as a second account or a silent overwrite.
Editing an existing subject is `user update`, and the flags compose, so
one call can change multiple things:
```console
# swarmctl user update mara --add-group admins --email mara@example.com
added to group "admins"
email: unset -> "mara@example.com"
mara is now in groups: admins
```
Two behaviours worth knowing before you rely on them:
- **`--remove-group` fails if the user isn't in that group.** Every
other flag is idempotent — setting what's already set is fine, so a
"make these four things true" call doesn't break when one of them
already was. Revocation is the exception on purpose: a typo'd group
name that reported success would leave an account holding access you
believe you took away, and that's the one outcome nobody re-checks.
- **`swarmctl` prints the resulting group list** because group names have no
registry anywhere. A misspelled `--add-group` creates a real group that
no access-control rule mentions, so the user gains nothing and no error
is possible — reading the line back is the only check there is.
Passwords are deliberately out of scope here: regenerating a credential
is a different intent from editing an attribute, and folding them means
an attribute edit can invalidate a login by accident. Resetting one is a
separate verb, `user reset-password`, which generates and hashes a new
password the same way `user add` does and refuses on an unknown user:
```console
# swarmctl user reset-password mara
reset password for mara in /var/lib/authelia-swarm/users.yml
password: <generated>
this password is stored nowhere — record it now
```
Authelia's file store has no SMTP notifier, so this is the only way a
human account gets a new password once an operator forgets the old one.
## What secrets exist, and where each one lives
[`credentials-reference.md`](credentials-reference.md) tabulates every secret in the swarm, with its
generator and its path, in one place — including authelia's own keys
(session, JWT, storage-encryption, OIDC HMAC, OIDC issuer) and the two
halves of each client secret. That page's two rules — a secret is always
a path, never a value, and the generator and the reader typically live in
different containers — are why the client secret's plaintext half needs
the delivery step below and the rest of authelia's keys don't.
## Getting the plaintext to the relying party
Three cases, and they're genuinely different mechanisms rather than one
mechanism with flags.
### 1. All-local — one host runs both
Nothing to configure at all. authelia mints each client's secret at its
first boot, and the mint in its container creates it in the swarm
secret store at `swarm/services/<client id>/oidc/client`, keeping only the digest on disk. A host-side unit
then reads it from the store and writes it into the service's container:
`swarm-bao-forge-oidc` for the forge, under the forge publisher's store
leaf, into `/run/hive-forge-oidc/<client id>.secret` on the host, which
`forgejo-sso-source` loads as a credential; `swarm-bao-matrix-oidc` for matrix, under
matrix-ctl's. A unit that can't read the secret yet fails and retries
every 30 seconds for a day, so there is no start order to get right. The
service's own module contributes its client entry — callback URL
included — to authelia's client list.
Each relying service's own glue module builds its callback once, and
both authelia's client list and the service's own OIDC client config
read that same value, so the redirect URI authelia accepts and the one
the service actually sends can't drift apart. A mismatch there is a
rejected login with no error text worth reading.
<!-- vale write-good.Passive = NO -->
⚠️ nixos-container refuses to start a container whose bind source is
missing, and this secret doesn't exist until authelia's first boot has run
and the mint has stored it. The forge therefore binds the **directory**
`/run/hive-forge-oidc`, which a tmpfiles rule creates at boot, never the
file. The directory is tmpfs, so the secret is gone after a reboot until
the fetch runs again; forgejo keeps it in its login-source row, so logins
still work meanwhile. Matrix's secret is written into the container's state
directory instead.
<!-- vale write-good.Passive = YES -->
### 2. Swarm-managed services
The controller side owns provisioning: `swarmctl` writes both halves, the
same way it already owns authelia's user store (`users.yml`, read and
written in place).
### 3. A hive elsewhere
The forge's host needn't be the store's or authelia's host, and the
operator places no secret file: `swarm-bao-forge-oidc` runs on the
forge's host as in the all-local case and fetches the secret over the
store's address. What that host needs is the forge publisher's store
leaf, issued from the store's CA and copied over from the store's host:
```nix
services.hyperhive = {
swarm.authelia.url = "https://auth.example.com";
deploy.bao = {
forgePublisherClientCertFile = "/var/lib/swarm-bao-pki/forge-publisher.pem";
forgePublisherClientKeyFile = "/var/lib/swarm-bao-pki/forge-publisher-key.pem";
};
};
```
**A forge host without that leaf still evaluates.** `swarm-bao-forge-oidc`
renders and fails at start, naming both options, and the forge has no
login source until an operator places the leaf. Where the store runs on
the same host, `glue-bao-tls.nix` defaults both options to the leaf it
mints.
## Where each relying party differs
The registration half is identical; what each service does with the
result isn't.
| | forge | matrix |
| ------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| how it learns the config | a oneshot calls `forgejo admin auth`, writing a login-source row into its database | tuwunel reads a `[[global.identity_provider]]` entry from its config file |
| how it reads the secret | a path inside its container | the same path, handed on by `LoadCredential` |
| callback URL | `<root>/user/oauth2/<source>/callback` | `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`, a shape tuwunel fixes rather than accepts |
| cost of a malformed entry | the login source is missing | the homeserver can refuse to start |
Four consequences worth stating plainly:
- **A person's first forge login creates their forge account**, named
after authelia's `preferred_username` (`[oauth2_client]` in the forge
module), provided the subject has an email. It starts as an ordinary user; `swarmctl forge make-admin
<name>` makes it a site admin. A name that already has a local forge
account doesn't get it handed over: forgejo asks for that account's
own password first (`ACCOUNT_LINKING = login`). Agents, `core` and
`swarm-controller` all have local accounts, and `swarmctl user add`
refuses none of those names.
- **A member of `admins` becomes a matrix homeserver admin** once their
first matrix login has created the account: `swarm-controller` promotes
them within five minutes, and never demotes. See
[`../integrations/matrix.md`](../integrations/matrix.md#operators-are-homeserver-admins).
- **tuwunel re-reads its secret file on every OAuth exchange**, not only
at startup, and its own sandboxing hides most paths from it. It gets the
file through `LoadCredential` for the same reason the appservice
registration does — that keeps `DynamicUser` and `PrivateUsers` intact,
with no host-side ownership arrangement to maintain.
- **Matrix SSO lives inside the homeserver.** The client-server API is
spoken by non-browser clients holding matrix access tokens — every
agent's own daemon — as well as by federation, so the homeserver
serves `/_matrix/` directly and authenticates it itself. The
forward-auth vhosts protect browser surfaces; this isn't one of them.
## Machine clients
Everything above is a person in a browser. A swarm also has callers that
hold no session and follow no redirect: the telemetry collector, the
queue's auth-callout responder, and each hive.
**One client per hive, not one per service.** A hive's identity belongs to
the directory rather than to whichever service happens to consume it, so a
hive holds a single OIDC client — `<hiveClientPrefix><hive>` — and mints a
different token per service from it. The alternative, letting each
consuming subsystem declare its own list, collides on the same client id
the moment a second consumer appears.
**The audience is that client id.** A swarm service that has to tell hives
apart needs one name both sides already agree on, and the client id is
already that name. A parallel per-hive naming scheme would be a second
thing to keep in step, and it drifts silently — a mismatch presents as a
valid token refused at the target, which reads like a broken credential
rather than a broken name.
**Authelia signs tokens (`RS256`), not opaque ones**, because a resource server
that can't call the provider back is a real case here: the telemetry
collector verifies offline against `/jwks.json`, and an opaque token gives
it nothing to verify. The queue's responder introspects instead — a
different question asked of the same token, and the reason both
`/api/oidc/introspection` and `/jwks.json` have to stay reachable.
### Machine callers must fail closed
⚠️ **An error page that answers `200` is a security bug, not a cosmetic
one.** The browser surface intercepts upstream errors and serves a
friendly "SSO is unavailable" page; that page is a file, so it returns
`200`. Any machine caller routed through it receives a success carrying
HTML instead of the failure that actually happened:
- `/api/authz/auth-request` — nginx `auth_request` treats **any 2xx as
success**, so a down provider means _access granted_
- `/api/oidc/introspection` — a token check that answers `200`
- `/api/oidc/token`, `/.well-known/openid-configuration` — a client
parsing an error page as its JSON document
The gateway routes authelia's `/api/` and `/.well-known/` prefixes, and
the bridge's `/bridge/`, **without** error interception. The split is by
_audience_, not by an enumerated path list: a human gets the page, every
machine caller gets the status.
Enumerating endpoints individually would leave the next one added
silently intercepted.
The same shape bites any machine route behind a browser-shaped gate: the
caller follows a `302` to a login page, the login page answers `200`,
and the caller reports success while nothing happened. Log ingest hit exactly this
and lost eleven hours of delivery in silence.
**Checking it, if you change this routing.** Point the vhost at a dead
upstream and compare three requests, not one:
1. through `/` — must still serve the friendly page
2. through `/api/` — must deny
3. a direct dial to authelia — must match what (2) did
All three matter. A change that silently deleted the browser page would
pass a deny-only check, and one that quietly stopped denying would pass a
page-only check. The person who introduced the split verified it this way.
## Operator CLI login
`swarmctl login` logs an operator in as their own account, from any
machine, through the public `swarmctl` client authelia always registers: no
secret, authorization code with PKCE, a loopback redirect, and a refresh
token. Its access token carries `authelia.bearer.authz` with the audience
`https://<swarm>/api/`, so the gateway admits it at the controller's API and
nowhere else, under the same access rule as a browser session there.
Revocation is per account: once authelia stops accepting the account, the
gateway refuses its token too. See [`swarmctl/README.md`](../../../swarmctl/README.md#remote-use-swarmctl-login).
## What this doesn't do
- **It doesn't disable local login — except on Grafana.** Forgejo and
tuwunel keep their password databases and gain a second door: an identity
provider that can take a service offline when it hiccups is worse than one
with two ways in, and making authelia the only path there is a separate,
reversible switch (tuwunel's `login_with_password`, forgejo's own setting).
⚠️ **Grafana is the exception, and not a reversible one.** Its module sets
`auth.disable_login_form` unconditionally: its vhost is on the public
gateway, so a password box there would be a way in whatever the
deployment. The form is all this hides. Grafana's API still takes HTTP
basic auth for the built-in `admin` account, whose password is a random
one minted into the swarm secret store, not the upstream `admin`. A
dead provider locks out the SSO accounts but not that one: an operator
reads the password from `swarm/services/<grafana client id>/admin/password`
in the store (or, as root on the Grafana host, from
`/run/swarm-grafana-secrets/admin-password`)
and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana)
and [`credentials.md`](credentials.md). SSO logins are the everyday way in, which
is why `swarm.grafana.oidc.role` defaults to `Admin`.
The three secrets Grafana reads live on the host's tmpfs, so after a reboot
Grafana stays down (no login, no dashboards) until the secret store
answers; the delivery units retry and start it with no manual step.
- **It doesn't provision users.** `swarmctl` creates and destroys agents
continuously today, so the subject set belongs to a program rather
than to a config file.

1092
docs/2-handbook/swarm/ui.md Normal file
View file

File diff suppressed because it is too large Load diff