mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
7 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

152
docs/4-implementation/swarm/bao.md Normal file
View file

@ -0,0 +1,152 @@
# The swarm secret store (OpenBao)
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
credential an agent or service needs comes from it under that
principal's own certificate identity. What lives in it and who reads what:
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md). This page covers the store itself — how it
comes up, who may write its grants, and how it's sealed and reached.
The operator steps (init, the one-time granter bootstrap) are in
[`../getting-started/setup.md`](../../1-getting-started/setup.md#1--secret-store);
this page is the why behind them.
<!-- vale write-good.Passive = NO -->
## The granter
Cert auth answers a _role_, so nothing can authenticate until some role
exists. A short-lived bootstrap token breaks that cycle once, and it's the
only step that needs the root token.
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
covers the auth mounts and the granter's own policy and role, and nothing
else. CI fails when the unit using the token needs a path it lacks. The token
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
names for you; on a store host that isn't all-local, set it and rebuild first.
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
auth methods (and disables `approle` if an older store still has it mounted),
writes the `bao-granter` policy, and creates the `bao-granter` role, which
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
mounts the KV and pki engines and writes the `swarm-controller` role, and each
sibling unit writes its own principal's policy and role. Every one runs on the
host, because every API listener but the loopback UI one demands a client
certificate and the host is the side that has one.
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
the token file is absent, which is the steady state afterwards; the token's
TTL means a forgotten file expires rather than lingering.
A sealed or unreachable store, or one whose listing fails with something other
than a refusal (a node still becoming active), makes the unit retry every
30 s with the store's error in the log. A store that refuses the token (expired,
revoked, or missing the `bao-bootstrap` policy)
makes it exit 4 and stay `failed` with the one-time step in its log; mint a
fresh token or remove the file, then restart it. If the granter's own
certificate login already works, the unit logs that the bootstrap token isn't
needed, deletes the dead token file and exits 0, so a stale file is harmless
and the unit runs once. A successful run deletes the token file too.
After that, a new or changed `swarm-*` grant needs no operator step: the unit
that writes it changes, and the deploy restarts it. A root step comes back only
when the granter itself needs a path it lacks, such as a new mount.
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
them, not on every deploy. When a grant drifts in the store and its unit stays
the same, the next boot re-asserts it, not the next switch.
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
wrapper carries an address, a CA and a client certificate but deliberately
**no token**, so that read answers `403` whether or not the role exists. Read
the unit's journal instead.
### Residual risk
The granter is root-equivalent. It may write any `swarm-*` policy with any
content, and attach it to a role that accepts any certificate; no bao ACL can
constrain what a policy says. What bounds it:
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
module-eval pins each principal's grants. **Merging a change to that policy
text is granting it**: it takes effect on the next deploy with no bao step,
so code review is the only gate.
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
root in a `0700` directory, readable only by root units on the store's host.
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
and `controller-key.pem`, whose policy is already root-equivalent. Root on
that host gains nothing new.
- **Never copy `granter-key.pem` off the host** the way operators copy the
other leaves in that directory. That hands out root-equivalence.
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
subject. Rotate the store's CA, or have root point the `bao-granter` role at
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
one.
## Sealing
`services.hyperhive.deploy.bao.seal`:
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
unseals itself on every restart. `init` is the only manual step.
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
restart, with the keys `init` printed.
⚠️ **A sealed store still answers.** The container is up and the port
responds while every read times out — the failure looks like a stall, not like
a store that was never initialised.
## TLS
The store serves TLS, and on a host that deploys it you need do nothing:
`swarm-bao-pki` mints a CA of the store's own plus the leaves it signs — the
store's server certificate, this host's client certificate, and one per service
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
It runs at every boot and every deploy, and before each start of a container
that loads one of its leaves, and signs only the files that are missing, so a
leaf a deploy adds, or a service a deploy enables, has its file on that deploy.
The store's readers and granting units on this host order after it and want it,
but none requires it: when a run fails, each of them still starts, and one
whose leaf is missing fails its login into its own retry. It has no start
limit, since a mass container start pulls it in once for each unit it starts.
Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about
which. A host that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see
[per-principal identities](../../2-handbook/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
They're the credentials the store can't hand you, being what opens it, so the
store's host signs them and the operator copies them to the host that needs
them and names them in config. The service principals' leaves are at the top of
`/var/lib/swarm-bao-pki/`, named after the principal.
For every hive in `swarm.hives` other than its own, the store's host signs
that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
| file | subject | name it in |
| ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on
the store host's next deploy; a leaf already there is never re-signed.
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
between services and the store**, a separate trust domain, because a store
that took its identity from an authority it itself distributes could never
come up before that authority.
Inside the store's container (`nixos-container root-login swarm-bao`) the
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
and its DNS name resolves to the bridge from in there: export
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
domain>`. The host's `bao` wrapper already carries the address, CA and client
certificate, so the host is the shorter path.
<!-- vale write-good.Passive = YES -->

607
docs/4-implementation/swarm/builder.md Normal file
View file

@ -0,0 +1,607 @@
# Swarm builder
The swarm builder checks that an agent config PR builds before an operator
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
push, the builder builds the PR's head on the agent base, and the result shows as a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a failed build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](../../2-handbook/swarm/services.md#builder).
## What happens on a config PR
1. The forge delivers the `pull_request` event on the `config-pr` webhook
swarm-controller already registers on `agent-configs`. An `opened`,
`reopened` or `synchronized` action on an open PR queues one `build_agent`
node carrying the agent, the head commit and the PR number. An edit, a
label or a close builds nothing. Every `build_agent` node needs the one
`builder` slot, so builds run one at a time and the rest wait pending. A
newer head cancels its PR's pending build; a build already running
finishes and writes its status on its own, older commit.
2. The node sets the status to `pending`, then sends the build as a request on
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
reply. The builder runs one build at a time and refuses a request that has
waited 30 seconds for its turn, such as one behind a build a controller
restart orphaned: a build started later could still run when the 61
minutes end. Reading the forge token before a build has 20 seconds, and
the rest of the minute covers the reply's trip back.
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
with every input overridden:
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
deployed with (`deploy.hyperhiveFlake`, `deploy.builder.nixpkgsFlake`). A hive's meta
flake pins `hyperhive` by forge URL and rev instead. The builder reads the
hyperhive path from `/etc/hyperhive/swarm-builder.json` in its container at
the start of each build, so a deploy that changes only that source doesn't
restart it or end a build in progress.
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
A build has one hour.
4. The reply is the system closure's store path, or the last lines `nix`
printed. The node writes the status on the head commit:
| Outcome | Status | Description | Node |
| ----------------------------------------- | --------- | --------------------------- | ------ |
| built | `success` | the store path | done |
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
| the builder busy for 30 seconds | `failure` | why | failed |
| no builder listening, or no reply in time | `error` | why | failed |
The controller records every build that gets a final status, and the
status links to that record: see [Build records](#build-records). The
full tail is also in the failed node's error on the Jobs page and in the
builder's journal line for the build, as the field `F_ERROR_TAIL`:
```console
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
```
The builder holds the last 64 KiB of each of `nix`'s output streams while the
build runs, and the reply is the last 30 lines of that (at most 4000
bytes), so a long build log never grows its memory.
## Build records
`swarm-controller` records each config PR build when it writes the final status:
built, failed, or no build ran or came back. It also records each [deploy-gate
build](#deploy-gate) the builder answers, in the same list. The newest 100 are one JSON
array under the key `recent` in the `builder-runs` KV bucket on the swarm
queue, which only the controller reads or writes. It creates the bucket on
first use and adds each build with a compare-and-set write, so the records
survive a controller restart. Recording the 101st drops the oldest. A record
the controller can't decode stays in the bucket, out of the list, and counts
toward the 100.
Each record has:
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>` for a
config PR's build, `meta-<agent>-<first 12 characters of the commit>-<started_unix>`
for a deploy's
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`; a deploy's build has
`repo` `internal/meta`, no `pr`, the `internal/meta` commit as `head_sha`,
and `agent`, the agent it built
- `started_unix` and `finished_unix`
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
or `error`
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
`failed`, and `detail` (why) for `error`
`GET /api/builder/runs` lists them, newest first, and
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
when the controller has no swarm queue or can't open the bucket, 500 when it
can't read it.
The controller records a deploy-gate build when the builder answers it, or when
asking it fails (no builder listening, no reply). A build that never got as far
as a request (no forge access, no `internal/meta` head) leaves no record, so a
controller that can't reach the forge doesn't fill the list with retries. Every
ten-minute retry of a failing agent is a record, and a version bump records one
build per agent, so the 100 cover fewer days when builds fail.
The final status links to `https://<swarm domain>/jobs?run=<id>`, the swarm
UI's jobs tab, where the builds are a card under the job graph. Selecting
**Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record
stays at `GET /api/builder/runs/<id>`. A status has no link when the record wasn't
stored (the controller logs the failed write) or when the controller doesn't
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
status has no link.
## What the check is, and isn't
- It composes the config the way a hive's meta flake does
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
what a hive adds: its CA bundle, `claude-code` path, ports and
OTEL settings, and the `follows` that point the config's own inputs at the
hive's. A green check can therefore still fail on a hive, and the reverse.
- The agent base is the builder's host's, not the hive's. Hives on another
hyperhive or nixpkgs pin build something else.
- The check pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
can push to [a binary cache](#binary-cache).
- The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery of a PR push isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
## Archetype evaluation
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
`meta` of every agent archetype its own hyperhive source declares, with no
forge token, and answers with the JSON. An evaluation doesn't wait for a
build in progress and has five minutes; the controller waits six. The builder
logs each evaluation's time as `elapsed_ms`, and a timeout's error carries the last
lines nix printed. What the controller does with the answer: [`archetypes.md`](../../2-handbook/swarm/archetypes.md).
## Meta-flake lock
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
rendered `flake.nix`. The builder writes it into a scratch directory, runs
`nix flake lock` there with the same forge credential helper its builds use,
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
lock doesn't wait for a build in progress and has ten minutes. The builder
commits nothing.
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
with the `hive-meta-flake` renderer, all over `https`.
Each custom agent that a hive places pins to its `agent-configs/<agent>`
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
running rev (`services.hyperhive.deploy.hive-controller.hyperhiveRev`, from `self.rev`):
- the first commit pins hyperhive's forge `main`;
- later, the pin moves to the controller's rev only when the forge's compare
API shows that rev is a strict descendant of the committed pin: commits
ahead of it, and none the pin has that it lacks. The compare endpoint counts
histories with no common ancestor as zero commits ahead, so the check needs
both directions;
- an older controller, a rev on another line, a controller built from a dirty
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
`main` moves nothing.
The controller reads its rev from `/etc/hyperhive/swarm-controller.json` at each
render, not from its unit, so a deploy that changes no Rust doesn't restart it.
A missing file, or one with no rev, counts as a controller without one.
The controller commits `flake.nix` and the returned `flake.lock` to
`internal/meta` in one commit, and gives each hive's forge account read on the
repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run only after the
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
builds only from `internal/meta`, so after a failed update the scheduler skips
the deploy node, and the graph shows the update as failed and the deploy as
skipped. Each ten-minute retry of a failed update declares a fresh build and
deploy behind it for every agent whose latest deploy the failed update skipped,
so the run that succeeds sends them (see
[Config changes](../../2-handbook/agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.
The controller sends a deploy to every agent a hive places whenever the pin in
`internal/meta` differs from the pin it last sent, so they rebuild on the new
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
which only the controller reads or writes, and a `RecordFanout` job writes the
key once every deploy of the fan-out has finished or the deploy gate has skipped
it. The check runs after each update and once when the controller starts.
- The first commit sends a deploy, because the key holds no pin yet.
- A controller that starts with no key, as after the upgrade that added it,
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because its build failed.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
writes no key.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake the swarm builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. A green build here means the
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
this build produced and never builds one itself.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
| What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs |
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
On every ten-minute round the timer declares a build and deploy again for each
agent whose latest build failed, without a meta update when the latest update
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
still records the pin when the gate skips a deploy, so one failing agent doesn't
resend the healthy ones. A controller that updates before its builder host does
is the same case: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each round adds a build and deploy node per failing agent.
A merged config PR whose webhook delivery never arrives would leave the agent
on its old config. Every five minutes, after the config-PR scan, the controller
reads `main` of each `agent-configs/<agent>` repo that has a commit and that
exactly one hive places. It compares that head with the config commit pinned for
the agent in the `flake.nix` of the `internal/meta` commit its `agent-closures`
key names, since the pin in `internal/meta`'s own head also moves for other
agents' merges. When they differ, it declares the same update, build and deploy
a merge declares, and logs at info `config drift: …`. It leaves alone an agent
with no key, a key whose flake pins no config commit for the agent, and a head
the graph already holds a deploy for, in any state: the ten-minute rounds above
build a head whose build failed again, not this pass. The graph is
in memory, so a controller restart tries each such head once more.
When the build belongs to a merged config PR and `nix` exited with an error or hit
its deadline, the controller also comments once on that PR, with its forge
account: the agent, the `internal/meta` commit of the first red build, `nix`'s last lines (at most 4000
bytes), and that the deploy waits and the swarm builds it again every round. A comment
carries a marker line with the agent, and the controller posts only if the PR
holds no comment of its own with that marker; a copy of the marker in someone else's comment is neither counted nor edited. The retry rounds stay quiet whatever commit
they build, and the latest error stays on the Jobs page. A green build of the
agent for that PR rewrites the marker and appends the build's commit to the comment, so a later red comments again.
A `no verdict` failure, and a build that isn't a
merged PR's (a new agent, the pin fan-out), leave no comment. When the forge
refuses the comment, the controller logs a warning and the node ends as before.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
not updating.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key.
The record's `cached` field says whether the builder pushed the closure to
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
The built reply and the record also carry two options of the agent's evaluated
config, `private_users` (`services.hyperhive.agent.user.privateUsers`) and
`nix_builds` (`services.hyperhive.agent.nixBuilds`). The builder reads each with
a `nix eval` in its sandbox, and an eval has three outcomes:
- **A value:** the field holds it.
- **The module doesn't declare the option** (`nix` says the flake doesn't
provide the attribute): the field holds the option's default, off for
`private_users` and `host` for `nix_builds`.
- **Any other failure, or a timeout:** the value is unknown, and the reply and
the record leave the field out. hive-c0re then sends nothing for that option,
and hive-priv keeps what the container's last write chose, as it does for an
agent with no record.
Mixed versions:
| builder | controller and hive-c0re | what happens |
| ------- | ------------------------ | -------------------------------------------------------------------------------------------------------------- |
| new | new | an unread option leaves the field out, and the last write stands |
| old | new | the old builder always sends a value, which the record carries |
| new | old | the old side reads a missing field as the default, so an unread option behaves as it did before this behaviour |
A record or reply from before the fields reads as unknown.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
grant the forge refuses fails the build with no verdict.
## Binary cache
`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
each green meta build's closure to it, or to the cache you override them with,
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
not `attic watch`, and the builder pushes again when the controller runs a build again. The
push has ten minutes, or less when the reply is due sooner, because it must end
inside the 61 minutes the controller waits. `attic` checks the server's certificate
against the system roots and pins nothing, so a Let's Encrypt certificate
works.
Set both options to a name or both to null: evaluation refuses one without the
other. Null in both is how a swarm opts out of pushing.
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
starting with a letter or digit, and evaluation refuses any other name.
The push token is the swarm's own cache's to create, or the operator's to write for an
external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to
a file and an `attic` config naming that file (`token-file`) in the unit's private
runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push.
The token is in no argument or variable, so no process listing shows it.
The reply's `cached` says whether the push finished. A push that fails, times out
or can't read the token leaves the build green with `cached` false and logs a
warning with the reason; the deploy goes ahead and the next build of the agent
tries again. With no attic behind the defaults, every push fails this way. With
both options null, the builder pushes nothing and `cached` is always false.
Mixed versions:
| controller | builder | what happens |
| ---------- | ------- | ------------------------------------------------------------- |
| new | new | `cached` in the reply and in the `agent-closures` record |
| new | old | the reply has no `cached`, which reads as `false` |
| old | new | the controller ignores `cached`; its record has no such field |
A record written before the field existed also reads as `false`.
### Rebuild on a cache miss
The cache drops a closure nobody fetched for its retention period, but the
record in `agent-closures` stays. Every ten minutes `swarm-controller` checks
each agent a hive places: when the record is from `internal/meta`'s head and
says the builder pushed it, the controller fetches the narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token it reads from
`secret/swarm/services/attic/` (the same four leaves it
[copies to the hives](#what-hives-do-with-the-cache); with any missing, or a URL
that isn't `https`, the pass does nothing). A 404 is a miss, and the controller
sends the builder `build-meta` for that agent and the record's own commit. The
builder builds it and pushes as for any meta build. Any other answer or a failed
request isn't a miss, and rebuilds nothing. The next ordinary build handles a record
from an older commit, or one with `cached` false.
A pass sends at most three builds, one after the other. After an agent's rebuild
failed, got no answer or built without pushing, the pass skips it for 30 minutes, then
for twice as long after each further failure, up to 12 hours. The wait is in memory,
so a controller restart tries every agent again. The first pass runs ten
minutes after the controller starts.
The record keeps naming what hives deploy. A rebuild that lands on the record's
path writes nothing. A rebuild that lands on another path means the build isn't
reproducible: the controller logs a warning, and replaces the record only if it
still holds the same commit and no write came since the controller read it. The
record otherwise stays as it was.
The controller counts each rebuild in `hyperhive.builder.cache_miss_rebuilds` with
the label `outcome`: `rebuilt`, `path_changed` (the record now names the new path),
`path_changed_kept` (the record stayed), `not_pushed`, `failed` (`nix` failed) or
`unavailable` (the builder was busy, absent or gave no answer).
Mixed versions: a builder needs no change, since a meta build already names
its commit. An old controller does nothing about a miss, as before. A hive
sees nothing new.
### The swarm's own cache
`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic`
container behind the gateway, at `attic.<swarm domain>`. It defaults on where
you set `deploy.allSwarmServices` and the host holds the cache's store identity
(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write
no secret: on first start, two host units log in to the store as the cache's own
principal and create what's missing.
- `swarm-bao-attic-secret` creates the key atticd signs tokens with at
`secret/swarm/services/attic/jwt-secret` and writes it where the container
reads it.
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
the builder reads, at `secret/swarm/services/attic/push-token`, and a
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
It also stores the facts a hive pulls with: the cache's address at
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
`secret/swarm/services/attic/cache`, and the public key it signs with at
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
A unit creates each secret only while the store holds none, so a second start
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
write the new token to its path in the store, in its UI under the OIDC login:
nothing mints over a value that exists. A new signing key invalidates both tokens.
The unit creates the address, name and key once and never updates them. After
you change `swarm.attic.domain`, or after the server's database goes and the
cache gets a new key, delete the stale leaf and restart
`swarm-bao-attic-tokens`.
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any
host pushes to it.
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
path nobody added or fetched for 14 days goes at the next garbage collection,
which runs every 12 hours (`deploy.attic.retentionPeriod`). The controller
[rebuilds a closure the cache lost](#rebuild-on-a-cache-miss).
The container sits on the hive bridge in a network namespace of its own. atticd
listens on that namespace's loopback (`deploy.attic.port`), and a socket-activated
relay forwards `/run/swarm-attic/atticd.sock` to it, a directory only the
gateway's nginx and the container can open. The gateway terminates TLS with a
swarm services certificate and forwards to that socket with no login in front:
attic checks its own tokens, and a nix client can't follow a login redirect.
The two host units reach the cache through the gateway, at
`https://<swarm.attic.domain>`, like every other client; the host resolves the
name through the hive resolver, and where the gateway serves a self-signed
certificate `swarm-bao-attic-tokens` trusts the swarm's CA. The container logs
to its journal only; its journal doesn't reach the swarm log store.
### External cache
The cache can be an attic tenant on an instance outside the swarm. To point
the swarm at an existing tenant:
1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
instance's host name (no scheme, `host:port` if it isn't on 443) and
`services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
both options on every host that reads them.
2. Seed the push token in the swarm secret store at
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
command line. The token needs push rights on that cache. Until it
exists, every push fails and builds stay green with `cached` false.
Seed the hives' pull token the same way at
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
3. Seed the three facts hives pull with, each as field `value`:
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
cache's page or `attic cache info`). Until all four exist, hives report
`no-credentials`.
The builder checks the instance's certificate against the system roots and
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
reads the token again on every push, so a new token takes effect on the next
build.
To use an external cache, override the two defaults and don't set
`deploy.attic.enable` anywhere. Where you host the shared services with
`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to.
A host that has `deploy.attic.enable` on and `swarm.attic.domain` set to another
name gets an evaluation warning that names this fix.
## What hives do with the cache
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
leaves (`pull-token`, `url`, `cache`, `public-key`) to
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
own prefix. It writes a copy only when the value differs, skips a leaf the
store doesn't hold, and leaves a copy whose source disappeared.
`hive-c0re` reads its four leaves every five minutes under the hive's own store
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
token goes out in a header and in the netrc; a hive that finds another scheme
writes nothing.
`hive-priv` doesn't read those files. The key that must have signed a download and
the pull token come from `hive-priv-cache-facts`, a root unit that reads the
hive's own `attic/public-key` and `attic/pull-token` from the store under the
hive's own store identity every ten minutes and writes them to
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`,
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes
decides nothing `hive-priv` trusts.
The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
It checks that a signature by the public key covers the narinfo, in Rust, with
no help from the host's nix daemon. It downloads nothing. The result is a log
line (at info when it changes) and the gauge
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
latest closure, else `0` with a `reason` label. The first check that fails
decides the reason:
| `reason` | Meaning |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
| `no-record` | `agent-closures` holds no build for the agent. |
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
A failed read of the store or the queue leaves the last result in
place and raises a warning after three failed passes. A hive reports nothing
before its first pass.
Mixed versions:
| controller | hive | what happens |
| ---------- | ---- | ----------------------------------------------------------------------------------- |
| new | new | the copies exist and the hive reports per agent |
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
| new | old | the controller writes copies nobody reads |
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
The signature check follows Nix's fingerprint format but hasn't run against a
live attic. A `bad-signature` on every agent in the first pass after a deploy
means check attic's fingerprint format; it says nothing about an attacker. A
`fetch-failed` on every agent points at the URL, the token or the network.
### Rebuilds download the closure
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
`Swap` then does the same lookup and download, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. A stopped agent skips
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
same and creates the container from that path.
A rebuild or create has two outcomes, by what the lookup and the download find. The hive deploys the record's closure whatever `internal/meta` commit the builder built it from, and reads no forge head to choose one:
| Found | What happens |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| a record that the builder pushed | `MetaSync` reads the record once and pins its store path, so a record that changes during the rebuild doesn't change what `Swap` applies. The hive downloads the pinned closure and deploys it, then writes the store path to the agent's `applied/.<name>.closure` marker. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
The hive never falls back to a local build. The second row has one owner in the
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
an unreachable cache costs is one function.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
| old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
## Credentials
| What | Where | Who reads it |
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo, and on `internal/meta` through the meta update) and mints a
`read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
status with its own forge token. Details per path:
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md).
The builder exports build counts, durations and queue wait; see
[Swarm builder metrics](../scheduler/observability.md#swarm-builder-metrics).
## Isolation
The container has its own network namespace on the bridge, as the CI runner
does, so a config's evaluation can't reach a host-loopback service. Evaluation
is pure and runs in the container. Builds go to the host's nix daemon as an
untrusted client: a config can set no daemon option, and its derivations build
in the host's sandbox. A fixed-output derivation still has the network, as it
does for any build on that host.

289
docs/4-implementation/swarm/ca.md Normal file
View file

@ -0,0 +1,289 @@
# Swarm CA
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
through a role that belongs to that hive. A peer that trusts the mount's
root once validates every hive's gateway and every swarm service name,
present and future, instead of pinning to each one by hand.
That's the point of one issuer — it turns per-peer trust from O(n²)
hand-pinning into one anchor per swarm.
## The gateway leaf comes from the secret store
`hive-tls-resign.service` logs in to the store with this hive's own
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
the files the gateway copies from. The leaf's subject is the hive domain,
and its SANs are the hive domain and `*.<hive domain>`.
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
hive's domain and `*.<hive domain>`, read from
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and `read` on its own `attic/public-key`, `attic/pull-token` and
`squid-selfcheck`
(see [the cache facts for hive-priv](../../2-handbook/swarm/credentials-reference.md) and the
squid self-check password), and nothing
else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every
hive in `swarm.hives`.
**Every hive needs its own store identity.** It's a client certificate
from the store's own client CA, with the subject
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
`baoHiveIdentityKeyFile`:
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
- On every other host, copy `hive-leaf.pem` and `hive-leaf-key.pem` from
`/var/lib/swarm-bao-pki/hives/<hiveName>/` on the store's host, which
`swarm-bao-pki.service` mints for every other hive in `swarm.hives`, and
set both options, before the host's next deploy.
Without it, `hive-tls-resign.service` fails with a message naming both
options, and so does `swarm-services-cert.service` on a hive fronting no
swarm name. The unit checks the identity before the leaf, so it fails even while a
valid leaf keeps the gateway up. The identity is the hive's alone:
it issues this hive's leaf and fetches the store's root, and it's never the
identity a swarm service uses.
**When it issues.** The unit runs at boot, on every deploy, and from
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
the current one is missing, past half the role's 720-hour lifetime, missing
a configured name, doesn't match the key beside it, or doesn't chain to the
store's root (`swarm-services-root.pem`). The first run that reaches the
store replaces a leaf that doesn't chain to that root, whoever signed it.
**The gateway doesn't wait for the store.** Nothing orders the gateway after
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
current leaf in place, says so in the journal, and retries every 30 seconds
for a day. Only a hive with no leaf yet needs the store reachable and
unsealed: its gateway's certificate import fails until the first leaf
lands, and then the unit starts nginx.
nginx serves the leaf alone, without its issuer: the issuer is the
self-signed root a consumer already holds in its trust bundle.
**A gateway host that runs no hive** (`deploy.hive-controller.enable` off,
such as a dedicated swarm-services or secret-store host) has no hive leaf, no
`hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its
gateway serves the swarm-services leaf as its default certificate too. Where it
fronts other swarm names it waits for `swarm-services-cert.service` at start.
Where the store's UI is its only swarm name it starts on a throwaway
self-signed certificate and switches to the services leaf once the store
issues it.
## The trust bundle
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
the file every consumer anchors on. It holds one certificate, a copy of
`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
the swarm-services leaf both chain to. It's the only root the swarm has.
**Without the store's root, the bundle is empty and its readers go red until
it lands.**
While `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem` is
missing or empty, as on a first boot before the store answers,
`hive-tls-ca.service` writes an empty bundle and logs that file's path. Every
`<name>-ca-bundle` unit then fails with `<bundle> contributed no certificate
to the bundle`, and the consumers that `Requires=` it don't start.
`swarm-services-cert.service` reloads `hive-tls-ca.service` when it writes a
new root. That rewrites the bundle in place and restarts no container, the
store's included.
**Each `<name>-ca-bundle` unit converges on its own.** Beside it, a
`<name>-ca-bundle-refresh.path` unit watches the file the bundle derives
from: inside a container the bound `/run/hive-ca/trust-bundle.pem`, on the host
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`. A write to it starts
`<name>-ca-bundle-refresh.service`. That service does nothing while the file
holds no certificate, or while it matches the copy the last assembly took
(`/run/<name>-ca/anchor.pem`). Otherwise it restarts the bundle unit and starts
its consumers without waiting. The restart reaches the running consumers
through their `Requires=`; the start picks up those a first boot left failed.
The same service runs once at boot, after the bundle unit, which covers a root
that landed before the watch began. A first boot needs no second apply, and a
root rotation restarts the consumers once and the container never. The watch
fires for the host's in-place rewrite because inotify keys on the inode and a
bind mount shares it, so `hive-tls-ca` must keep rewriting the file in place
and never rename over it.
A deploy that changes `hive-tls-ca.service` reloads it too, with the same
effect.
⚠️ **A consumer outside `trustBundle` reads its trust when it starts.** A
container whose nix predates the refresh units keeps the bundle it started
with until it restarts, and fails TLS to a gateway serving a store-issued leaf
in the meantime.
## The swarm's service names come from the secret store
The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
**single-label** wildcard over it, and its role allows nothing outside the hive's
domain, so it can't cover them.
The secret store's `pki` mount issues them a leaf of their own, through its
`swarm-services` role. `swarm-services-cert.service` logs in with
the services-issuer identity, calls
`pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
copies from. The store's role, not a key on the host, decides which names a
gateway can serve.
**Every self-signed gateway that fronts a swarm name needs the
services-issuer identity**, whether or not a hive runs on its host. A hive
fronting none needs no copy of it: it fetches the store's root with its own
store identity, as below. It's a client certificate
from the store's own client CA, with the subject
`deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default),
and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile`
and `servicesIssuerClientKeyFile`. Its policy may issue from
`pki/issue/swarm-services` and nothing else, and it's the same certificate on
every gateway host:
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to
it.
- On every other gateway host fronting a swarm name, copy
`services-issuer.pem` and `services-issuer-key.pem` from the store's host
and set both options.
A host fronting a swarm name without both options set still evaluates and deploys, and `swarm-services-cert.service` fails, naming
them. One that names files it can't read fails the same unit, naming the
paths. The old spellings,
`deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are
renames of these two: a config that still sets them evaluates with a warning
naming both paths.
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
same list on every host, populates the role's
`allowed_domains`, so asking for a name nobody configured is a refusal
from the store naming that name — not a certificate quietly issued for
it. A host's own SANs are narrower still: it asks only for the names it
fronts a vhost for (`deploy.hive-controller.tls.localServiceDomains`), so the key on one
gateway can't serve a swarm service that runs behind another.
**The root's public certificate is a file, on every self-signed gateway host:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
swarm name asks for no services leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its own store identity
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
`baoHiveIdentityKeyFile`) for the handshake without logging in with it.
Every hive must hold one: without it, or with either file absent,
`swarm-services-cert.service` fails with a message naming both options.
On the host running
the store it's also at
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
hand a browser, and reading it needs no store login — which matters,
because every store listener but the loopback UI one demands a client
certificate, and authelia gates that one to the `admins` group, not
open to an anonymous browser fetching a trust anchor.
**The granting unit generates the root once, and never again.** It asks
the mount whether it already has an issuer (`bao list pki/issuers`)
before calling `pki/root/generate/internal`, so a rebuild or a reboot
re-asserts the role and the grant without touching the anchor. A root
that changed per boot would invalidate every certificate issued under it
and every browser taught to trust it.
**A daily timer renews the leaf.** The store's role issues it for 720
hours. `swarm-services-cert-renew.timer` runs the same script as
`swarm-services-cert.service` once a day, and that script asks the store
for a new leaf once the current one is past half that window, is
missing a configured name, or doesn't chain to the store's root
(`swarm-services-root.pem`). A rotation re-imports the leaf into the
gateway and reloads nginx. A store that can't answer — still sealed, or
out of reach — fails the unit, which leaves the current leaf in place,
and the next day's run tries again: about fifteen tries before the leaf
lapses. `systemctl
list-timers swarm-services-cert-renew` shows the next run.
## Constraints on the material
No CA private key exists on any hive. The `pki` root's key stays inside the
store's mount (`root/generate/internal`), and each hive's narrowing is its
role's `allowed_domains`, enforced by the store at issue time: a hive's
identity that leaks can get certificates for that hive's names and no
others.
No host holds a CA or its key, except the store's mTLS CA
(`glue-bao-tls.nix`), which signs only the store's server certificate and the
certificates that log in to it. Every host deletes `/var/lib/swarm-ca` and
the `ca.pem`, `ca-previous.pem`, `ca-key.pem`, `ca.srl` and
`.swarm-ca-adopted` in
`<deploy.hive-controller.tls.stateDir>`: `systemd-tmpfiles` removes them at
boot, and on a deploy that changes the host's `tmpfiles.d` rules. The swarm
distributes only certificates.
⚠️ **The deletion is irreversible.** Rolling back to a generation that still
has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a
new one, unrelated to the deleted root. That's harmless: nothing issues
under it.
A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
evaluation, however it's spelled with `//`, `.`, `..` or a trailing `/`. A
symlink to one isn't resolved, so it gets past the check. Those options name
the store's mTLS CA, `/var/lib/swarm-bao-pki/ca.pem` on the host that runs
the store.
## What to hand a peer
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
block a peer operator pastes into their own config. When this hive's
gateway serves a self-signed leaf (detected by the presence of
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints the
secret store's `pki` root itself, read from
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
gateway leaf and the swarm's service names under that root, so it's the one
anchor a peer host needs.
The operator installs the root **once per swarm host**, not once per peer,
so a hive joining later needs no edit on the hives already running. A hive
whose cert already chains to a public CA has nothing to install —
`peer-config` says so instead.
## Distributing the root
The root **certificate** is a runtime file: each hive fetches it from the
store at boot, so nothing can name it at build time. That has a cost worth
naming, because it's subtle and it bites at a distance:
> **Nothing whose trust store nix assembles at build time can reference
> the store's root.** The derivation reads `security.pki.certificateFiles`
> inside itself; the root doesn't exist there.
Two consumers, both reading the root at runtime:
- **Agents fetch it themselves.** `hive-agent-trust-bundle.service` (`nix/agent-modules/hive-ca.nix`) reads `<mount>/ca/pem` from the store with the agent's own store certificate, then writes `/run/hive-agent-trust-bundle/bundle.pem` (system CAs plus the root) and `root.pem` (the root alone). The agent's units point `SSL_CERT_FILE` at the first (Rust, Go and OpenSSL clients) and `NODE_EXTRA_CA_CERTS` at the second (Node and Claude Code), and order after the unit with `Requires=`. The fetch retries inside its start job for up to 14 minutes, with a wait that doubles from 2 to 60 seconds, so a store name that doesn't resolve yet delays those units instead of failing them; after that the job fails and they fail with it. The meta flake names no CA file, and a timer re-reads the root hourly. A consumer reads it at start, so a rotated root reaches an agent unit at its next start.
- **The Matrix container gets a bind mount**, via the shared
`lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
use: the bundle is bind-mounted read-only into the container, and a
oneshot concatenates it with the system CAs before tuwunel starts.
The consumption differs per runtime and is the part worth knowing.
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
— it's not: its rustls-based TLS stack still resolves trust through the
platform certificate store on Linux, and that store honors
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
an OpenSSL-linked binary.
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
> `openssl-probe` uses it _instead of_ the default store, so naming
> just the bundle would drop every public CA and break federation with
> the wider matrix network — a much bigger outage than the one being
> fixed. The same caveat applies to `hive-forge` (Go) for the same
> reason.
A hive in the swarm fetches the root itself. A host outside it gets the
certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)).
One anchor per host, not one per peer: a hive joining later needs no edit
on the hives already running.

103
docs/4-implementation/swarm/ci-metrics.md Normal file
View file

@ -0,0 +1,103 @@
# CI metrics
`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
can show run time and queue wait per repo, workflow and PR, over any range back
to a repo's first run.
## Where it runs
A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
has the poller's store identity, which the host that runs the store has. Run it
on one host per swarm: each poller keeps its own cursor, so two of them push
every run twice.
It reaches the forge, authelia, the store and the collector by name, so it needs
none of them on its host. On a host without the store, its leaf
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
file to copy, as for the CI runner.
With a self-signed gateway, those names serve leaves the swarm's own CA issued.
The poller reads them through `swarm-ci-metrics-ca-bundle.service`, which joins
the system CAs and the host's `trust-bundle.pem` into
`/run/swarm-ci-metrics-ca/trust-bundle.pem` and sets that file as the poller's
`SSL_CERT_FILE` ([CA](ca.md#the-trust-bundle)). Without it, every pass fails
with `invalid peer certificate: UnknownIssuer`.
## What it reads, and as whom
| what | where | identity |
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| its forge token | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass | the same leaf |
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account `ci-metrics`, with that token |
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
swarm-controller creates the `ci-metrics` account and mints its token with the
scope `read:repository`, the way it does for the builder. It also keeps a
`ci-metrics` team in every organisation, with read on the Actions unit of every
repo in it, and the account as its one member. The poller therefore covers:
- every public repo, which needs no grant;
- every private repo an organisation owns, through that team;
- **no** private repo a user owns. The forge grants a user's repo only to a
collaborator, and a collaborator reads every unit, code included.
## What it pushes
Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
forwards them to the metrics store. Each sample carries the run's `stopped`
time as its timestamp, which the store keeps, so a run from last month lands
last month.
| metric | value, in seconds |
| ----------------------------- | ------------------- |
| `hyperhive.ci.run.duration` | `stopped − started` |
| `hyperhive.ci.run.queue_wait` | `started − created` |
Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
them, every run is a series of its own, and the board reads each with
`last_over_time`.
A run becomes no sample when its status is `cancelled` or `skipped`, it has no
start (or the epoch as its start, which the forge writes for a run cancelled
before it started), or it stops no later than it starts. A run still going a day
after its `created` time counts as abandoned. The journal line for each repo counts
these by reason.
## The cursor and the backfill
The poller keeps one file, `cursor.json`, in its state directory
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
poller has pushed or skipped every run, and the run numbers above it already
handled. A run still going holds that number, so a run that finishes after a
newer one still gets pushed.
With no file, the cursor of each repo is at run 0, so the first pass is the
backfill: it walks every run back to the first, a page of 50 a second. For a
repo with 5000 runs that takes a few minutes.
The poller writes the file only after the collector accepted a push, never
before. A crash between the two pushes those runs again on the next pass. The
store then holds the same point twice in the same series, which changes no
panel. Deleting the file re-runs the backfill the same way.
A failed repo leaves its cursor where it was, the pass goes on with the next
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
timer runs it again 5 minutes later.
## The board
The **CI runs** tab of the `forge` board (`nix/host-modules/swarm-grafana/dashboards/forge.json`), with a repo and a workflow
variable:
- mean and 90th percentile run time of successful runs, over time;
- queue wait over time, mean and longest;
- finished runs per day, by outcome;
- a table per PR over the time range: mean and longest successful run time,
runs, failed runs.

133
docs/4-implementation/swarm/inference.md Normal file
View file

@ -0,0 +1,133 @@
# Per-agent inference settings
The swarm can set the model and reasoning effort each agent's turns run on,
plus the models it may start subagents on and how many subagents it may run
at once. A field you set wins over the
agent's own value; a field you leave unset keeps the agent on its own value,
so an agent you set nothing for runs as it does today.
## Setting them
```sh
swarmctl agent inference set iris --model opus --effort high
swarmctl agent inference get iris
swarmctl agent inference clear iris
```
- `set` changes only the flags you pass; the other fields keep what's
stored. A hive must declare the agent.
- `--effort` is one of `low`, `medium`, `high`, `xhigh`, `max`.
- `--subagent-models` takes a comma-separated list. It replaces the
agent's `availableModels` as the models its subagent `start` and
`continue` calls may name, read on each call
([`subagent.md`](../../2-handbook/tools/subagent.md#which-models-a-subagent-may-run-on)).
- You set the subagent cap, `max_subagents`, in the swarm UI's inference
dialog; `set` has no flag for it and keeps the stored one. It's a whole
number from 0 to 32, where 0 allows no subagents; unset means no cap
([`subagent.md`](../../2-handbook/tools/subagent.md#how-many-subagents-may-run-at-once)).
- `get` shows each field, naming the unset ones (or says the swarm sets
nothing for the agent), and then the model and effort the agent reports
running on, each with where it comes from.
- `clear` removes every field; the agent goes back to its own values.
The full flag reference is [`swarmctl-cli.md`](../../2-handbook/reference/swarmctl-cli.md). The
swarm UI's agents page edits the same settings; see [UI](../../2-handbook/swarm/ui.md). The
swarm is the only place to change an agent's model or effort at runtime:
its web UI shows both but has no picker and no `/model` or `/effort`
command.
## What an agent runs on
`get` shows what the agent runs on next to what the swarm sets:
```text
model: opus
effort: (not set; the agent's own)
subagent_models: (not set; the agent's own)
max_subagents: (not set; no cap)
running model: opus (swarm)
running effort: medium (local_file)
```
Each running value names where it comes from:
| source | meaning |
| ------------ | -------------------------------------------------------------------------- |
| `swarm` | the value set here |
| `agent_nix` | `services.hyperhive.agent.model` or `effortLevel` in the agent's agent.nix |
| `local_file` | a `hyperhive-model` or `hyperhive-effort` file in the agent's harness dir |
| `default` | the harness's built-in default (`haiku`, `medium`) |
When the agent starts without a swarm value, its agent.nix model wins over
a `hyperhive-model` file, and a `hyperhive-effort` file wins over its
agent.nix effort. These files come from runtime picks the agent's web UI no
longer offers; the agent still reads them but never writes them, so a
`local_file` value stays until the swarm sets that field or someone deletes
the file.
The agent reports these whenever they change, and again once a minute.
swarm-controller keeps only what it has received since it started, so for up
to a minute after a controller restart `get` shows `running: not reported…`.
## When an agent picks a change up
Each agent reads its settings every 10 seconds and switches on the next
read: its next turn runs on the new model or effort, and its web UI badge
shows it. A turn already running keeps what it started with.
- **Its own value is the one it runs without the swarm**: its agent.nix or a
`hyperhive-model` / `hyperhive-effort` file. Setting a swarm value doesn't
overwrite it.
- **Clearing a field** (or the whole key) puts the agent back on its own
value on the next read.
- **After a restart**, the agent runs its own values until its first read
of its settings succeeds.
<details><summary>Implementation</summary>
The settings live in the `agent-inference` KV bucket, one key per agent,
which swarm-controller alone writes:
```json
{
"model": "opus",
"effort": "high",
"subagent_models": ["haiku", "sonnet"],
"max_subagents": 3
}
```
Every field is optional. The agent runs the same with `{}` as with no key;
`GET` answers `"configured": null` for no key (or a cleared one) and `{}` for
a stored value with no fields. A stored value the controller can't decode,
such as one a newer build wrote, also answers `"configured": null`, with a
warning in its log, until the next `PUT` replaces it.
| method | path | body |
| -------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GET` | `/api/agents/<agent>/inference` | —; answers `{"configured": <the stored value, or null>, "effective": <what the agent reported, or null>}` |
| `PUT` | `/api/agents/<agent>/inference` | the whole value, replacing the stored one; the controller refuses unknown fields, an unknown effort, or a model name with whitespace or a leading `-`, or a `max_subagents` above 32 |
| `DELETE` | `/api/agents/<agent>/inference` | —; answers 204 |
An agent's own queue credential grants it
`$JS.API.DIRECT.GET.KV_agent-inference.$KV.agent-inference.<agent>`, the
direct read of its own key, and nothing else in this bucket — the same shape
as its schedule key ([`schedules.md`](../../2-handbook/swarm/schedules.md)).
The harness (`hive-agent`'s `swarm_inference`) applies a set field to the
in-memory model or effort without writing `hyperhive-model` or
`hyperhive-effort`, which keep the agent's own value.
The subagent daemon (`hive-subagent-mcp`'s `swarm_models`) reads
`subagent_models` and `max_subagents` with the same direct get, under the
same credential, on each `start` or `continue`.
`effective` comes from the agent's turn-state header on
`$SWARM.agent-state.<agent>` ([`README.md`](../../2-handbook/swarm/README.md)), which carries
`model`, `effort`, `model_source` and `effort_source`. swarm-controller
subscribes to every agent's header with the reader grant it already holds for
the state relay and keeps the latest per agent in memory. A header without
those fields, from an agent that doesn't report them, leaves `effective`
`null`.
</details>

317
docs/4-implementation/swarm/notify-router.md Normal file
View file

@ -0,0 +1,317 @@
# Notification router
The swarm's `swarm-notify-router` service turns forge events into todos on the
agents they concern. No hive is in the path, and the agent can be on any hive:
swarm-controller keeps the events, the router decides, and each agent's harness
takes its own todos off the queue.
Where it runs and how it's enabled: [`services.md`](../../2-handbook/swarm/services.md#notification-router).
It runs beside `hive-forge-notify`, which still delivers forge notifications
the way it did. An agent can get the same news from both.
## What an agent gets
| Flow | Event | The agent's todo |
| ------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `assigned` | someone assigns an agent to an issue or PR | `you are assigned to issue <repo>#<n>: <title> (<url>)`, key `forge:<repo>#<n>:assignment` |
| `unassigned` | someone takes an agent off an issue or PR | the same key rewritten: `you are not assigned to issue <repo>#<n> anymore: <title> (<url>)` |
| `closed` | someone closes or merges the issue or PR | the router clears every router todo it queued for it, then queues `closed` on each agent that held one: `<kind> <repo>#<n> was closed: <title> (<url>)`, key `forge:<repo>#<n>:closed` |
The todos are subsystem `router` in the agent's todo list. A todo under the
same key replaces the one before, so an issue gives an agent one assignment
todo however often it changes hands. A new event reopens a todo the agent
marked done.
The `closed` todo has its own key, apart from the assignment todo. A reopened
issue leaves it in place until the next close replaces it; a new assignment
writes the `assignment` key and leaves it alone.
Only agents get todos. An assignee with no agent of that name gets nothing.
An agent assigned before it first reports its status gets its todo on the
issue's next assign event.
Forgejo's assign events carry the assignees after the change, not which one
changed. The router keeps the assignees it saw on the last assign event per
issue and takes the difference. With no earlier assign event for an issue,
every current assignee counts as newly assigned and nobody as removed: an
router notices an unassign only on an issue it has seen an assign event for.
## The path
1. swarm-controller registers two instance-wide hooks, one for the `issues`
and `issue_assign` events at `/webhook/forge/issues` and one for
`pull_request`, `pull_request_assign`, `pull_request_sync` (a push to an
open PR's branch) and the submitted-review events `pull_request_review_approved`,
`pull_request_review_rejected` and `pull_request_review_comment` at `/webhook/forge/pull-request`. At each start it adds
any of these events a registered hook lacks.
2. On each delivery the controller checks the HMAC and publishes the body,
unchanged, to the `SWARM_EVENTS` stream on
`$SWARM.events.forge.<X-Forgejo-Event-Type>`, such as
`$SWARM.events.forge.issue_assign`. The stream keeps a delivery 7 days and
drops a redelivery by its `X-Forgejo-Delivery` id. A delivery the stream did
not keep answers 503, so the forge's delivery log shows it.
3. The router drains its durable consumer, `notify-router`, in order. For each
event it runs every flow, queues the todo ops the flows emit, writes the
issue's snapshot to its `notify-router-state` bucket, then acks the event. A
failed step stops the batch: that event and the rest come back in stream
order after 30 seconds, and the stream drops the ops queued twice by their
message id.
4. Each op goes to the `ROUTER_TODOS` stream on the agent's own subject,
`$SWARM.router.todo.<agent>`. Work-queue retention: an op stays until its
agent acks it, up to 30 days and 1000 per agent.
5. The agent's harness creates its own durable consumer, `todo-<agent>`,
applies each op to its todo list and acks it. An agent that's down, or
whose harness predates this, leaves its ops waiting.
The router holds no forge token and no webhook secret, and registers no hook.
## Flows
A flow is a graph of nodes joined by edges: one `source` node naming the forge
event types it starts from, `match` nodes that pass when every condition
holds, `todo` nodes that emit one todo op per recipient, and `request_reviewer`
nodes that ask for a review. A match node can have more than one child, and
each one runs, so one event can reach more than one leaf. A flow must be a tree:
one source, one parent for every other node, every node reachable, no children
under a `todo` or a `request_reviewer`. The most is 64 nodes, 16
levels deep, and 100 custom flows. A `todo` node sends to the agents an event
field holds (`assignees_added`, `todo_holders`, …) or to one named agent.
The router ships the three flows above, each a source → match on `action` →
todo, and `head-classifier`: source `pull_request` and `pull_request_sync` →
match on `action` (`opened`, `reopened`, `synchronized`) and `draft` = `false`, or on
`undrafted` → `request_reviewer` for `review-classifier` ([below](#requesting-a-review)). The code defines them
(`swarm_wire::notify_flows::builtins`), so a build that changes one reaches
every swarm. Each switches off on the notification router card on `/swarm`. A custom flow has the same type and
the same evaluator, so it can do anything a built-in does.
**The stored document.** swarm-controller keeps the rest in one document, the
`all` key of the `notify-flows` bucket: the ids of the built-in flows you
switched off, and the custom flows. Only the controller writes it, through
`GET` and `PUT /api/notify/flows`. A `PUT` replaces the whole document in one
compare-and-set. The controller answers 400 unless every flow passes the tree
check above and reads only what exists: a source event from `issues`,
`issue_assign`, `pull_request`, `pull_request_assign`, `pull_request_sync` or one of the
three `pull_request_review_*` events, and condition and recipient fields from the event's field list. The same goes for the `{field}`
placeholders in a todo's key and text. A `PUT` must carry the `revision` it read, `0` for no
document, and gets 409 when the stored revision is another. Swarm UI's
[notification router card on `/swarm`](../../2-handbook/swarm/ui.md#notification-flows) shows the document and
switches built-ins.
**The schema.** `GET /api/notify/flows` also returns a `schema` object, built
from the constants the check reads, so an editor offers only what a `PUT`
accepts:
- `source_events`: `[{name, description}]`, the source events.
- `fields`: `[{name, description, values, values_from?, closed}]`, the condition
and recipient fields, and the `{field}` placeholders. `values` lists what the
field holds, as far as the controller knows, as `{name, description}` pairs, such as the `issue` and
`PR` of `kind`.
- `node_kinds`: `[{kind, description, fields}]` for `source`, `match`, `todo`
and `request_reviewer`; each field is
`{name, description, required, values, values_from?, closed}`, and `values`
lists the accepted `{name, description}` pairs (the `upsert` and `clear` of a
todo's `op`), empty for other fields.
- `values_from` on a field says where an editor asks for values that are only
known live: `agents` is the names `GET /api/agents` lists, `repos` the
repositories with an open issue, which `GET /api/repos` lists, so a repository
with none is missing. `closed` says `values` and that source are every value
the field takes, which an editor can show as a dropdown. When false, they
are suggestions: the field still takes any other text, such as an `action` the
forge adds later or the login of a reviewer who is no agent.
- `limits`: `{max_nodes, max_depth, max_flows}`.
A controller that predates `schema` leaves it out of the answer; a client reads
it as optional.
**Requesting a review.** A `request_reviewer` node has one field, `reviewer`,
a forge login. For the PR the event concerns (`repo` and `number`; an event
about an issue sends nothing), the router sends a NATS request on
`$SWARM.router.sink.request-reviewer` with `{repo, number, reviewer}` and
swarm-controller answers with one line:
| Reply | Meaning |
| --------------- | ---------------------------------------------------------------------------------- |
| `requested` | the forge accepted the request |
| `skipped:<why>` | the reviewer's review of the PR's head already stands; the controller sent nothing |
| `error:<msg>` | the forge refused or didn't answer, or the request didn't decode |
The controller asks the forge under its own token, which the router never
holds. It skips a reviewer whose latest review is neither stale (the PR's head
moved past the commit it reviewed) nor dismissed, because requesting a review
from someone who already submitted one makes the forge dismiss it, approval
included. The controller asks again once the review is older than the head.
`hive-forge pr assign-reviewer` applies the same rule (`hive_types::review`).
The router sends the request once, after it queued the event's todo ops and
wrote the snapshot, and logs the answer. A controller that's down or answers
`error:` doesn't hold up the event or any agent's todos, and the request is
not retried: the next push asks again. A request repeated for the same head is
harmless: a reviewer with a pending request is a no-op for the forge, and one
who has reviewed gets `skipped`.
`head-classifier` requests the review classifier's review when someone opens or
reopens a PR and on every push to an open PR that isn't a draft. A draft is a
PR the forge reports with `draft` true, which it does for a title starting
`WIP:` or `[WIP]`. A title edit that drops that prefix (`undrafted` is true:
the event is `edited`, the PR is no draft, and the old title in the payload
started with the prefix) requests the review once, so a PR whose last push
came as a draft gets a classification. The classifier then runs as it does
for any requested review
([`review-classifier.md`](review-classifier.md#trigger-and-lifecycle)), through
its own request queue, which replaces a waiting request for the same PR. A
repo the classifier's account can't read, or a PR that account opened, answers
`error:`.
**Reviews as a source.** A submitted review arrives as a pull request
payload: `pull_request` and `repository` as for any PR event, `action` is
`reviewed`, and `sender` is the reviewer. Approving, requesting changes and
commenting are three event types, so a flow picks the verdict with its source
and reads the reviewer with a condition on `sender`. A flow that asks argus to
review once the classifier has approved:
| node | setting |
| ---------------- | ----------------------------------- |
| source | `pull_request_review_approved` |
| match | `sender` equals `review-classifier` |
| request_reviewer | `reviewer` = `argus` |
The skip rule above applies: the router asks for argus's review once per head.
**What the router runs.** It reads the key at most every 10 seconds, when an
event arrives, and runs the built-ins you left on, then the custom flows that
are on and pass the tree check. It logs a custom flow that fails the check,
leaves it out, and runs the rest.
| the key | the router runs |
| -------------------------------------- | ----------------------- |
| absent, deleted, or its bucket missing | the built-in flows |
| a document it reads | the above |
| a document it can't read | the flows it ran before |
| a read that fails | the flows it ran before |
"Can't read" means not JSON, not a flows document, or a `version` other than 1.
Before the router has read any document, "before" means the built-ins, so a bad
document never leaves it with no flows. The controller refuses to overwrite a
document of a version it doesn't know.
**Deploying in any order.** The controller, the router and the UI can be on
different hosts. A router that predates `request_reviewer` reads a document
holding such a node (a custom flow with one) as one it can't read and keeps the
flows it had; it also has no `head-classifier`. A router that has `head-classifier`
but predates the `draft` and `undrafted` fields requests the review of a draft PR
as before. A router that has `request_reviewer`, with a
controller that predates it, gets no answer to the request and logs that. A controller that predates this leaves the key absent, and the
router runs the built-ins as before. A router that predates this ignores the
document, so the page and the API show a state the router isn't yet running.
## Deliveries and dry runs
**The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG`
stream for each delivery it makes, and the stream keeps the newest 500. A
delivery is a todo op queued for an agent or a review requested from the
controller, whether it went through or not. A row has:
| Field | Holds |
| ------------------------- | ---------------------------------------------------------------------------------- |
| `at` | Unix seconds |
| `flow`, `node` | the flow and the node that delivered |
| `event`, `repo`, `number` | the forge event type and the issue or PR it concerns |
| `kind`, `target` | `todo` and the agent, or `request_reviewer` and the reviewer |
| `ok`, `detail` | whether it went through, and the op and key, the controller's answer, or the error |
A review the controller skipped (`skipped: <why>`) is `ok`: nothing was
wrong. An event the router retries after a failed step writes its rows again,
so a retry shows twice, and the page says so under the table. A row that
takes longer than 2 seconds to write counts as failed: the router logs a
warning and the delivery goes on.
**The journal.** The router also logs each delivery it handles, so
`swarm-logs` answers whether a flow ran for a given PR. The lines carry the
event and sender, never the payload body:
| Message | Level | Fields |
| -------------------------------------------- | ----- | ----------------------------------------------------------------------------------------- |
| `notify router: forge event routed` | info | `event_type`, `action`, `repo`, `number`, `sender`, `delivery_id`, `flows` (at least one) |
| `notify router: forge event matched no flow` | debug | the same fields, with `flows` of 0 |
| `notify router: sink ran` | info | `flow`, `repo`, `number`, `kind`, `target`, `outcome` |
| `notify router: sink failed` | warn | `flow`, `repo`, `number`, `kind`, `target`, `error` |
`flows` counts the flows that produced a todo op or a review request.
`delivery_id` is the forge's delivery id, or `none` when the event carried no
id. A review the controller skipped is a `sink ran` line, as in the log rows.
**The dry run.** The router runs a flows document over the newest 50 events of
the `SWARM_EVENTS` stream and returns, for each flow the document turns on,
how many of those events reached each node. A `match` node counts the events
that passed it. It uses the walk the router delivers with and delivers nothing:
no todo op, no review request, no log row and no snapshot written. It keeps
its own per-issue snapshots, starting empty, so the first assign event it
replays for an issue counts every assignee as newly added. Every recipient
counts as an agent.
**The API.** The router answers two NATS requests from swarm-controller, which
serves them next to `GET /api/notify/flows`:
| Route | Router subject | Answers |
| ---------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------- |
| `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default |
| `GET /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the saved flows document, run over the newest 50 events |
| `POST /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the body's draft flows document, checked as a `PUT` checks it, run over the newest 50 events |
The dry-run request carries a whole document. The `GET` sends the saved one;
the `POST` takes a `PUT` body (it ignores a `revision`), answers 400
with the `PUT` error for a document that fails the check, and forwards a valid one.
[notification router card on `/swarm`](../../2-handbook/swarm/ui.md#notification-flows) shows both: the recent deliveries,
and the counts beside each node.
**Deploying in any order.** A router that predates these subjects doesn't
subscribe to them, and the queue answers the controller's request at once with
no responders. Both routes then answer 503 saying the router doesn't answer
and may be an older build, and the page shows that in place of the table and
the counts. A request the router hears and doesn't answer within 10 seconds is
the same 503. A router that answers with a refusal, or with something the
controller can't read, is 502. A controller that predates the routes has no
route: the page, if newer, shows the failure.
## Credentials and grants
| Principal | Grant |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds |
| swarm-controller | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two
`$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or
`$SWARM.events.forge.*`, so each has one writer, and the controller alone
hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the
controller asks the forge under the token it already holds. The agent's create subject
carries the consumer's filter, and the queue server checks it against the
request, so an agent can't create a consumer that delivers another agent's
todos.
## Observability
The router pushes OTLP metrics through its container's forwarder to the swarm
collector, labelled `service.name=hyperhive-notify-router`. With no
`OTEL_EXPORTER_OTLP_ENDPOINT` it exports nothing. The client-certificate
gauges (`bao.client_cert.*`) come with them.
| Metric | Unit | Kind | Attributes |
| ---------------------------------------------- | ---- | --------- | -------------------------------------------------------------------------------- |
| `hyperhive.notify_router.events` | — | counter | `event_type`; `outcome`: `routed`, `unmatched`, `ignored`, `unusable`, `retried` |
| `hyperhive.notify_router.event.duration` | `s` | histogram | `event_type` |
| `hyperhive.notify_router.todos` | — | counter | `op`: `upsert`, `clear`; `outcome`: `delivered`, `failed` |
| `hyperhive.notify_router.review_requests` | — | counter | `outcome`: `requested`, `skipped`, `failed` |
| `hyperhive.notify_router.pull.failures` | — | counter | — |
| `hyperhive.notify_router.consumer.pending` | — | gauge | events not yet delivered to the router's consumer |
| `hyperhive.notify_router.consumer.ack_pending` | — | gauge | events delivered and not yet acked |
| `hyperhive.notify_router.consumer.redelivered` | — | gauge | events delivered more than once |
`ignored` is an event that names no issue or pull request. `unmatched` is one
no flow matched. `retried` is one the queue or the store failed on: it comes back
and counts again. A growing `consumer.pending` means the router is falling
behind. The router reads the consumer counts at the start of each pull, at most
30 seconds apart.

711
docs/4-implementation/swarm/review-classifier.md Normal file
View file

@ -0,0 +1,711 @@
# Review classifier
The review classifier checks a pull request against short, named yes/no
rules that the repo itself defines, and posts the result as a forge review.
A small model answers one rule at a time, so mundane rule checks come out
the same way on every PR.
It doesn't replace a full review by a human or agent reviewer. A swarm can
request that review only once the classifier has passed the PR, and use a
larger model there. The classifier takes the cheap, repetitive checks off
that reviewer.
## Repo configuration
A repo defines its rules in a file at its root, in YAML as
`.hyperhive/review-classifier.yml` or in JSON as
`.hyperhive/review-classifier.json`. It holds one of the two; with both, the
rules don't load. `swarm-controller` always reads the file from the repo's
**default branch**, never from the PR, so a PR can't edit the rules it's
judged by. It reads it with its own forge token on every load, with no cache,
so a rule merged to the default branch counts from the next load on.
The file holds one key, `classifications`: a list of one to ten entries, each
with exactly these four keys:
- `name`: short and unique in the repo, shown in the result table
- `description`: the yes/no question the model answers about the PR
- `threshold`: the certainty, 0 to 100, a verdict needs to count
- `polarity`: `positive` means a match is the wanted outcome, and
`negative` means a match is a violation
Any other key, at the top or in an entry, is an error: a misspelt key fails
the load instead of vanishing. An empty list, more than ten entries, an empty
name or description, and two entries with one name are errors too. A file
that exists but fails these checks never falls back to a pass: the run posts
no verdict.
A repo whose default branch holds neither file has no rules. When someone
requests the classifier's review on its PR, the account approves at once. The
body reads `no review rules are configured for owner/name; approved without
checks`. The controller makes no model call. The instance has no per-repo
filter yet, so every repo without a rules file gets this approval.
An example:
```yaml
classifications:
- name: datetime
description: All datetimes carry time zone information.
polarity: positive
threshold: 80
- name: no-ads
description: >-
A commit message contains a co-authored-by line or other
advertising for an AI model.
polarity: negative
threshold: 90
- name: bao-first
description: Any added secret reaches its reader through the bao store.
polarity: positive
threshold: 70
- name: docs-updated
description: >-
A change to what an operator or user sees, such as a command, flag,
option, endpoint, service or default, updates the matching page under
docs/ or the crate's README in the same diff. A diff that changes
nothing an operator or user sees makes the question not applicable.
polarity: positive
threshold: 65
```
## Trigger and lifecycle
The classifier has a dedicated forge service account, `review-classifier`
(see [The service account](#the-service-account)). Requesting that account
as a reviewer on a PR starts a run:
1. The forge sends the `pull_request_review_request` delivery to
[`swarm-controller`](../../../swarm-controller/README.md), the swarm's one
webhook owner, on its instance-wide hook `/webhook/forge/review-request`.
A delivery for any other reviewer, or a removed request, does nothing.
2. The controller reads the PR and the repo's rules from the default branch,
and hands the run to the classifier service as one queue request on
`$SWARM.review-classifier.try-out`, the subject a try-out uses.
3. The classifier runs every classification against the PR's commits and
diff.
4. The controller posts the result on the PR as the service account's
review, on the commit it read.
Opening or pushing to a PR starts no run by itself. The notification router's
`head-classifier` flow requests the account's review when someone opens or
reopens a PR and on every push to an open PR, unless its review of the new head
already stands ([`notify-router.md`](notify-router.md#requesting-a-review)). It
requests nothing for a draft PR, one whose title starts with `WIP:` or `[WIP]`
and which the forge reports as `draft`. The edit that takes a PR out of draft
requests the review once, for the head the PR has then. That edit recognises
only the default `WIP:` and `[WIP]` title prefixes: after taking a PR out of
draft that used another prefix, request the review by hand. A request that
reaches the controller is always reviewed, whether or not the PR is a draft.
When a post can't reach the secret store or the forge, because the
connection fails or a try gets no answer within two minutes, the controller
tries it up to six times over about five minutes, enough to outlast a
swarm-bao restart, with the same verdict. It doesn't retry any other failure,
such as the forge refusing the review. A post that still fails ends the run
as `post-failed`, and nothing appears on the PR. The account's review
request is still pending then, and the forge sends no event for a request
that's already standing: remove the service account's review request, then
request it again to start a fresh run.
A verdict belongs to the commit it ran on. A push starts no run. Once the
PR's head moves, requesting the service account as a reviewer again starts a
fresh run on the new head.
Requesting the account again on a head it already reviewed depends on that
head's newest recorded review, as the [run store](#run-store-and-retention)
holds it. When that review approved or requested changes, nothing runs: the
account posts the same verdict and result table again, opening with
"Already reviewed `<sha>`, verdict unchanged." When it had no clean
verdict, or its post failed, the request starts a fresh run. A head with no
recorded review, such as one whose runs have dropped out of the newest 100, or
a run store the controller can't read, gets a fresh run too.
The re-request also retires the account's earlier review. The forge clears
the `official` flag on a reviewer's earlier reviews when it's requested
again, and only official reviews count toward the required approvals or
block a merge. Removing the request before the new review lands restores
the earlier one. A repo whose branch protection also dismisses stale approvals (see
[PR review gate](../../5-project/process/pr-review-gate.md)) stops counting an approval
on an older commit even without a re-request.
How many requested reviews run at once is a swarm-wide setting, two by
default (see [Swarm-level settings](#swarm-level-settings)). The controller
turns no request away. It puts each one in the `review-requests` work-queue
stream on the swarm queue, one subject per PR, and the stream keeps one
message per subject: a newer request for a PR replaces one still waiting.
The controller takes a request once a run place is free and acks it after
writing the run's record. While a run goes, the controller tells the queue
every 20 seconds that the request is still in hand. A request it took and
never acked, because the controller stopped first, comes back within a
minute, and the controller that's running then reviews it. A restart
therefore repeats a run that was going. When it can't queue a request, the
account posts a `**No verdict:**` comment saying so.
When the controller takes a request, it reads the PR first. A request whose
head is no longer the PR's head runs on the PR's current head, and never
re-posts the old head's recorded verdict.
A request for a PR whose review is still going waits in the queue until that
review ends, so a PR never has two runs at once. A request that re-posts an
earlier verdict waits for a run place too, and frees it before it posts.
## Components
### The classifier service
A swarm-level service in its own container, on whichever host the operator
puts it. It talks to `swarm-controller` through async messages only, and
gets its secrets from the swarm's secret store, bao
([`bao.md`](bao.md), [`credentials.md`](../../2-handbook/swarm/credentials.md)).
It looks like an agent from the inside, with three things missing: it has no
full agent identity, no state directory and no tools. Every model session is
ephemeral, and nothing carries over from one run to the next.
On the API key runtime, each classification is one `claude --print` the
service spawns through the `hive-claude` crate directly, with no session to
resume, no tools (`--tools ""`), no MCP server (`--strict-mcp-config` and no
config) and an empty directory of its own. The prompt reaches it on stdin as
one user message in `--input-format stream-json`, so that it can carry content
blocks (see [Shared context](#shared-context)).
The `claude` is the one its host's
`services.hyperhive.deploy.hive-controller.claudeCodePackage` pins for
agents, or its container's own nixpkgs' where the host pins none.
On the ACP runtime, each classification is one prompt on a new `opencode acp`
session. Its config and opencode's state live in a directory of its own,
removed after it. Its working directory is one empty directory that every
classification shares: opencode writes that path into the system message, so
one path keeps the system message the same from one classification to the
next, and the provider's prompt cache can hold. The service drives it through
[`hive-runtime`](../../../hive-runtime/README.md)'s `AcpRuntime`, as the subagent
daemon does a one-run subagent, and stops the agent when the classification
ends. The agent has no tool. opencode runs some tools, `read` among them,
without asking the client, so the service sets opencode's own permission to
deny every tool (`OPENCODE_PERMISSION={"*":"deny"}`), which leaves the model
with no tool offered at all, and refuses any permission request on top. The
config it writes for opencode names one provider and model, from the settings,
and turns off opencode's title call, which would send the prompt a second
time. It also defines the agent the session runs as, whose prompt is the
classifier's system prompt, so opencode sends it as the system message, with
only its environment block (model, directory, platform, date) after it. The
first user message holds the PR alone. That agent has no `permission` of its
own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
there would offer the model tools again.
The service is `swarm-review-classifier`, and its container is the one
[`services.md`](../../2-handbook/swarm/services.md#review-classifier) describes. It holds no forge
credential: `swarm-controller` reads the PR and sends it along.
### Swarm-level settings
The runtime, the model, the effort and how many requested reviews run at once
are one document for the whole swarm. `swarm-controller` keeps it in its
`review-classifier-settings` KV bucket under the key `all`, and no other
service reads it. With no document stored, the classifier runs on the API key
runtime on `haiku` at claude's default effort, two reviews at a time. The
controller ignores a stored field it doesn't know, and a document it still
can't decode counts as none, with a warning in its log, until the next PUT
replaces it.
`GET /api/review-classifier/settings` answers the settings in force:
```json
{
"model": "haiku",
"effort": null,
"max_concurrent_reviews": 2,
"runtime": "api-key",
"acp": null
}
```
- `model`: the model every classification on the API key runtime runs on, as
claude's `--model` takes it: one word with no whitespace, not starting with
`-`
- `effort`: `low`, `medium`, `high`, `xhigh` or `max`; `null` runs at the
runtime's own default. On the ACP runtime it applies only where the agent
offers it
- `max_concurrent_reviews`: 1 to 8, how many requested reviews run at once.
Try-outs don't count toward it. The rest wait in the queue for a place
- `runtime`: `api-key` or `acp` (see [Runtimes](#runtimes))
- `acp`: the provider the ACP runtime runs on, or `null`. It stays stored
while the runtime is `api-key`, so switching back and forth needs no
re-entry. It holds:
- `base_url`: the provider's OpenAI-compatible API root, `https://` only
- `model`: the model as the provider names it, with no whitespace
- `prices`: what the provider charges in USD per million tokens,
`input` and `output`, and optionally `cache_read` and `cache_write`, which
otherwise count as `input`. A price is 0 or more
`PUT /api/review-classifier/settings` takes any of these fields, changes only
those, and answers the whole document it stored. A field left out keeps its
stored value; `null` sets `effort` or `acp` to none. A form that sends the
model, effort and limit alone therefore leaves the runtime as it was. No secret
goes in the document: an `api_key` field is an unknown field.
The controller reads the settings for every run and sends the runtime, model
and effort in the run request, so a change applies from the next run with no
restart. A try-out reads them when it's posted, a review when it gets its
place. Lowering the limit lets the runs already going finish. A run request
with no model runs on `haiku`, and one with no runtime on the API key.
The PUT needs the `Remote-User` header the gateway sets, as the try-out POST
does. Every error is problem+json with the reason in `detail`:
- **400** for a body that isn't JSON or isn't this shape: an unknown field, a
`null` other than for `effort` or `acp`, an effort outside the five levels,
a `max_concurrent_reviews` that isn't a whole number, a runtime other than
the two, or an `acp` without `base_url`, `model` or `prices`. Also **400**
when the document the PUT would store breaks a rule: an empty `model`, one
with whitespace or one starting with `-`, a `max_concurrent_reviews` outside
1 to 8, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
with whitespace, a negative price, or the `acp` runtime with no `acp`
- **401** for a PUT with no `Remote-User`
- **503** when this host has no swarm queue, or it isn't connected
- **500** when the controller can't read or write the settings
When the controller can't read the settings, a try-out answers the same 503
or 500, and a review posts a no-verdict comment. The operator writes each
runtime's key by hand in the secret store
([`services.md`](../../2-handbook/swarm/services.md#review-classifier)).
The per-agent equivalent is [`inference.md`](inference.md), where the
controller writes model and effort to a queue KV bucket
(`swarm-wire/src/agent_inference.rs`) and each agent reads its own
key.
### The service account
A dedicated forge user, separate from swarm-controller's site-admin account
and from every agent's account. It's the reviewer someone requests, it
authors every classifier review, and it pushes the branches the try-out section
creates.
`swarm-controller` creates it as `review-classifier`, with the email
`review-classifier@services.hyperhive.local` and no repo creation, and mints
its one token, `swarm-review-classifier` with scope `write:repository`, into
the store at `swarm/services/review-classifier/forge/token`. A pass at start
and every five minutes re-mints when the token is missing or no longer
matches the forge. The controller refuses an existing account by that name
with any other email. The controller posts the reviews and pushes the try-out
branches with that token, and the classifier service never sees it.
**Giving a repo access is a repo admin's step, by hand.** Nothing grants
it automatically. In the repo's settings, add `review-classifier` as a
collaborator:
- with **read**, so someone can request it as a reviewer, and its reviews show,
but its approval doesn't count
- with **write**, so its approval counts toward the required approvals and
its request-changes blocks the merge where the branch protection blocks on
rejected reviews. Where the branch protection restricts approvals to an
allowlist, the account counts only once it's on that list, and read access
is then enough for reviews.
The try-out's **open a PR** pushes a branch to the repo itself, so it
needs **write** whatever the branch protection says.
The repo also needs a rules file on its default branch (see
[Repo configuration](#repo-configuration)). Without one, a request posts a
comment saying so.
## The classification run
### Shared context
Every classification in a run sees the same context, in the same order:
1. a short system prompt
2. the PR's commits, each with its full message
3. the full diff
Only the last part differs per classification: that classification's
description and the task to classify the PR against it. The identical prefix
lets the runtime's prompt cache reuse it across questions.
The controller builds the diff itself, from each changed file at the PR's
merge base and at its head, with 25 lines of context around each change
rather than the 3 of the forge's `.diff`. An enclosing gate, such as a
`config = lib.mkIf …` or an option's `default`, is in view when it's within
25 lines of the change, so a move that restates it doesn't read as adding one.
A gate further away is still out of view. At review time the controller takes
the merge base from the PR's commits: the one parent that isn't a PR commit.
A rebased or force-pushed PR therefore gets the diff against the base it sits on
now, not the base it started from. The controller sends the forge's `.diff`
instead when the PR changes more than 100 files, when the commits don't name
a single merge base (the head merges the base branch in), when reading a
changed file fails, or when the wide diff is over 256 KiB. The review then says
so, with the reason.
On the API key runtime the commits and diff are one content block of the user
message and the question another, with a cache breakpoint at the end of the
first. The cache writes an entry only where a breakpoint is, and `claude` sets
one only after the last block, so that breakpoint is what lets the next
question read the commits and diff from the cache. The PR stays in the user
message rather than the system prompt, which tells the model to judge it as
data. A shared context shorter than the model's minimum cacheable length is
not cached at all.
No object hash is in that prefix: the header names the repo and the PR number,
not the head commit, each commit's heading gives its position instead of its
sha, and the diff loses each file's `index <hash>..<hash>` line, keeping its
`diff --git` paths and every hunk. A rebase rewrites all of those hashes, so
with them in, a rebased PR with the same messages and hunks would miss the
cache from the first line. The head commit still goes to the controller,
which posts the review against it.
### One question, one answer
The model answers each classification in JSON. The answer holds:
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
- a **certainty** from 0 to 100, how sure the model is of that verdict
- a **reason**; the prompt asks for under 350 characters, and the classifier
truncates one over 500 to 499 plus `…`, keeping the verdict and certainty
An answer:
```json
{
"verdict": "no-match",
"certainty": 92,
"reason": "Two new timestamps use naive datetimes without a zone."
}
```
The verdict field holds `match`, `no-match` or `not-applicable`, and the
certainty field a whole number. The classifier reads the object from the
reply, or from the first code fence in it. When an answer isn't that object,
the classifier asks that classification once more, with a reminder to reply
with the JSON object alone; the other classifications aren't asked again. The
second ask runs on what the first left of the classification's three minutes,
so a retry never lengthens the run, and it doesn't happen when the first ask
leaves less than 20 seconds. The classifier also doesn't ask again after a
runtime error or a timeout. A runtime error, a timeout, or an answer that
still isn't that JSON object after the second ask leaves the classification without a verdict: it's listed with its error, and
doesn't count. A run with no key for its runtime, or in which no
classification got an answer, fails as a whole, and shows its error instead of a
verdict.
### Aggregation into the verdict
The classifier collects the answers into one result, keyed by
classification name, and logs it. A classification counts toward the
overall verdict only if it's applicable and its certainty reaches the
threshold. The table still lists every classification below its threshold or
not applicable, marked as excluded.
A counted classification fails when its verdict lands on the wrong side of
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
failing classification makes the verdict **request changes**. Otherwise the
verdict is **approve**, including when the run counted no classification at
all.
### The review
The service account posts the verdict as a forge review, approve or request
changes, and the table is the review's body. An illustrative table:
| result | check | certainty | reason |
| ----------- | ------------ | --------- | -------------------------------------------- |
| ❌ no match | datetime | 92 / 80 | Two new timestamps use naive datetimes. |
| ✅ no match | no-ads | 99 / 90 | No commit message carries an AI attribution. |
| ❔ unsure | bao-first | 95 / 70 | The PR adds no secret. |
| ❔ match | docs-updated | 41 / 65 | Below threshold. |
**Overall: ❌ request changes**
The result cell is the symbol for what the row did to the review, then the
model's verdict. ✅ is a counted row that passed its polarity and ❌ a counted
row that failed it, so _match_ is ✅ on a positive rule and ❌ on a negative one.
❔ is a row that doesn't count: below its threshold, not applicable (shown as
_unsure_), or without an answer. The certainty cell reads _certainty / threshold_.
The approval is a real one: it counts toward a repo's required approvals
like any other ([`pr-review-gate.md`](../../5-project/process/pr-review-gate.md)). The
Repos that rely on it typically raise their required approval count to
account for it.
A run with no clean verdict never approves. When the PR or the rules don't
load (a repo with no rules file gets an approval instead), the run fails, the classifier
answers a different number of
classifications than the controller sent it, or **any** classification has no answer,
the account posts a **comment** instead, starting with `**No verdict:**` and the reason,
followed by the table when there is one. A comment neither approves nor
blocks the merge: nine timeouts and one pass post a comment, not an
approval.
The controller flattens every cell to one line and backslash-escapes the
characters that could end the cell or start markup, math or raw HTML: `|`, `\`,
backticks, `[`, `]`, `<`, `>`, `&`, `*`, `~`, `$`, and `_` except between two
letters or digits (`ci_metrics.rs` stays as is). Ordinary punctuation such as
`(`, `)` and `.` passes through unescaped. Each `@`, `#`, `!`, `://` and `www.`
gets a zero-width space after it, with no backslash, so a model's `reason`
can't break the table, embed markup, mention anyone, reference an issue or PR,
or carry a link. The prompt asks the model for plain text in `reason`.
## Runtimes
The classifier runs on one of two runtimes, the settings' `runtime`:
- **API key** (`api-key`, the default): the `claude` CLI on the Anthropic API
key at `secret/swarm/services/review-classifier/anthropic-api-key`, at
`model` and `effort`
- **ACP** (`acp`): `opencode` on the provider `acp` names, with the provider
key at `secret/swarm/services/review-classifier/acp-provider`, at
`acp.model`
Model names don't carry across runtimes. A claude alias means nothing to an
ACP provider, so each runtime has a model field of its own.
To switch to ACP, an admin:
1. writes the provider key in the secret store's UI under the OIDC login, at
`secret/swarm/services/review-classifier/acp-provider`, field `api_key`
2. PUTs the runtime and the provider:
```json
{
"runtime": "acp",
"acp": {
"base_url": "https://llm.example/v1",
"model": "Qwen/Qwen3-35B",
"prices": { "input": 0.5, "output": 2 }
}
}
```
The next run goes to that provider. `{"runtime": "api-key"}` switches back and
keeps the provider for next time.
A split deploy runs on the API key: a classifier older than the ACP runtime
ignores `runtime` and runs every classification on `claude`, and a controller
older than it ignores the stored `runtime` and `acp` and sends neither.
A third runtime, **claude subscription** (the `claude` CLI on a Claude
subscription login), comes later.
> **Open point.** The subscription runtime needs a login, which is the one
> piece of durable state the service would keep. A swarm-level claude
> subscription login may have to exist before this runtime works.
## Swarm UI
Two pages in the [swarm UI](../../2-handbook/swarm/ui.md), `admins` only like the rest of it.
### Recent runs
One card per run: the repo, the PR, and how many classifications ran.
Selecting a card expands it to show each classification's result, the same
rows as the review table.
### Try-out
A page for writing classification prompts without touching the repo. Pick a
repo and a PR, write or edit a classification, and run it against that PR;
change the prompt and run it again until it answers the way you want.
Nothing reaches the forge while you iterate. A button the page disables
points at a line that says why, such as the 10-classification cap on **add a
classification**, and a screen reader announces the run's progress once, with
the elapsed seconds outside the announcement.
The page posts the run to `swarm-controller` (`POST
/api/review-classifier/try-out`), which reads the PR with its own forge token
and sends it to the classifier as one queue request on
`$SWARM.review-classifier.try-out`. The page then reads the result back (`GET
/api/review-classifier/try-out/{id}`): a run can take longer than a request
stays open. The controller keeps the result in memory for an hour, and the
finished run goes into the [run store](#run-store-and-retention).
`GET /api/review-classifier/rules?repo=owner/name` returns the repo's own
rules, for the page to load into its editor; a run posts them like typed
ones. The route answers:
- **200** with `path` (the file read), `branch` (the default branch),
`commit` (the commit the route read the file at) and `classifications`, in
the file's order and in the POST's shape
- **400** for a `repo` that's missing or not `owner/name`
- **404** for a repo the forge doesn't know; a default branch with no commit,
in an empty repo or when the default branch doesn't exist; or a default
branch with neither file
- **422** for both files on the default branch, or a file that doesn't parse
or breaks a rule from [Repo configuration](#repo-configuration); the
`detail` names the file and what's wrong
- **503** when this host has no forge token yet, or the forge failed or
didn't answer within two minutes
The page's repo and PR fields offer values to pick from, and stay free text,
since a run also works on a merged or closed PR. Two routes feed them:
- `GET /api/pulls/repos` lists the repos with at least one open PR, as
`owner/name`.
- `GET /api/pulls?repo=owner/name` lists the repo's open PRs, highest number
first, each with `number`, `title` and `head_sha`. It answers **400** for a
`repo` that's missing or not `owner/name`, **404** for a repo the forge
doesn't know, and **503** when this host has no forge token yet, or the
forge failed or didn't answer within two minutes.
Every error is problem+json with the reason in `detail`.
When it's ready, **open a PR** (`POST /api/review-classifier/open-pr`, with
the repo and the classifications) has the service account push a branch that
adds the classifications to the repo's rules, with you, the swarm UI user, as
commit author. The page then links you to the forge's compare page for that
branch, where you log in as usual and open the PR yourself.
- **The branch** is `review-classifier/<you>-<8 hex digits>`, made from the
default branch's head with one commit, through the forge's contents API.
- **The file** is the rules file on the default branch, merged by name: a
classification named like an entry replaces it, a new name goes at the end,
and every other entry stays. A repo with no rules file gets
`.hyperhive/review-classifier.yml`. In a YAML file, the entries the merge
leaves alone keep their text, comments included. The merged file has to
pass the same checks the `rules` route makes.
- **The author** is the forge account named like your swarm UI login. A forge
account made by an SSO login carries the SSO username, so it's yours once
you've logged in to the forge. The commit carries its full name, or the
login when it has none, and its noreply address,
`<login, lowercased>@noreply.<forge domain>`. It's the address the forge
itself uses for an account that keeps its email private, so the commit
links to your account and your email stays out of the repo's history. The
committer is the service account.
- **The token** is the service account's own, read from the store for each
push (see [The service account](#the-service-account)), so the account
needs **write** access to the repo.
The route answers **201** with `branch`, `path` (the file written) and
`compare_url`, or:
- **400** for a body the try-out POST would refuse in its repo or
classifications
- **401** without the `Remote-User` header the gateway sets
- **403** when the forge refuses the push: the service account isn't a
collaborator with write access
- **404** for a repo the forge doesn't know, or one with no commit on its
default branch
- **409** when the forge has no account named like you yet, or the rules
file changed on the default branch during the push
- **422** for a default branch holding both rules files or one the `rules`
route refuses; classifications the file already holds unchanged; or a
merge that breaks a rule, such as more than ten classifications
- **503** when this host has no forge token, the service account's token
isn't minted yet, or the forge failed or didn't answer within two minutes
## Run store and retention
`swarm-controller` records every run when it ends: a try-out when the
classifier's reply arrives, a requested review once the controller has posted
its review or the post has failed, and a request the controller couldn't queue
once its comment is on the forge or the post has failed. The controller records a
re-posted verdict the same way, with the outcome and answer of the run it
repeats and that run's id in its `detail`. A run still going isn't in the list; a
controller restart stops it, and the queue hands its request back.
The newest 100 runs are one JSON array under the key `recent` in the
`review-classifier-runs` KV bucket on the swarm queue, which only the
controller reads or writes. The controller creates the bucket on first use and
adds each run with a compare-and-set write, so the list survives a controller
restart or a move to another host. Recording the 101st run drops the oldest;
there's no time-based retention. The controller leaves a run it can't decode, such
as one a newer build recorded, out of the list with a warning in its log. It
keeps the run in the store as it was and counts it toward the 100. A document
that isn't a list reads as no runs. The controller logs a failed write, and it
changes nothing on the forge.
`GET /api/review-classifier/runs` returns the list, newest first. Like the
try-out GET, the gateway's `admins` rule gates it, and the controller checks no
header. Each run has:
- `id`: a try-out's is the id its POST answered with
- `trigger`: `try-out` or `review`
- `repo` (`owner/name`), `pr`, and `requester`: the swarm UI user who started
the try-out, or the forge user who requested the review
- `head_sha`: the commit the run read, absent when the PR wasn't read, as for
an unreadable PR or a request the controller couldn't queue
- `started_unix` and `finished_unix`: a try-out starts at its POST, a review
once it has a run place and has read the PR
- `outcome`: `approved`, `changes-requested`, `no-verdict` or `post-failed`
for a review, or `queue-full` on a record written before the request queue;
`done` or `failed` for a try-out
- `detail`: the reason, for every outcome but `approved`,
`changes-requested` and `done`. For `post-failed` it's the run's own reason,
when it has one, then why the post failed.
- `rows` and `overall`, as the try-out GET returns them, when the classifier
answered
- `review_url`: the posted review, when the forge names its URL
A run holds no diff, no commit text and no token count or cost. The route
answers 503 when the controller has no swarm queue or can't reach the bucket,
and 500 when it can't read the list, both as problem+json.
Each recorded run is also one `info` event from `swarm-controller` with the
message `review classifier run finished`. The event carries the same fields,
with `rows` as JSON. It reaches VictoriaLogs through the host journal, where
each field arrives as `F_` and the name in capitals (`F_OUTCOME`, `F_REPO`), so
a run whose write failed is still there:
```console
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-controller.service" _msg:="review classifier run finished" | fields _time, F_TRIGGER, F_REPO, F_PR, F_OUTCOME, F_DETAIL, F_REVIEW_URL' --format json
```
Add a filter such as `F_OUTCOME:="post-failed"` before the `|` to narrow it.
## Observability and cost
The swarm UI shows no token counts, cache counts or cost, on any page. On the
claude runtimes, every classifier run starts its `claude` process with
`OTEL_RESOURCE_ATTRIBUTES` naming the repo and the PR, so the cost reads from
the metrics store by repo and PR. The process exports to the OTLP receiver of the
container's own forwarder on loopback, which pushes to the swarm collector as the
classifier's OAuth2 client, on a host with a hive or without one. On a subscription, cost is notional.
On the ACP runtime, the service computes each classification's cost itself,
from the settings' `acp.prices` and the token counts the agent reports for the
prompt, and adds it to the counter `hyperhive.review_classifier.cost.usage`,
labelled `repo`, `pr`, `model` (the provider's) and `currency` (`USD`). It
exports to the same receiver `claude` does. A classification stopped at its
deadline, or whose agent reports no token counts, adds nothing. opencode
reports no cost of its own for a provider without prices in its config, which
is why the prices are a setting.
Each such classification also adds the tokens it used to the counter
`hyperhive.review_classifier.token.usage`, with the same labels bar `currency`
and a `type` of `input`, `output`, `cacheRead` or `cacheCreation`. Those are
the types of `claude_code.token.usage`, which `claude` exports on the claude
runtimes, so the cache hit rate reads the same way on every runtime: `cacheRead`
over the sum of `input`, `cacheRead` and `cacheCreation`.
The controller counts what requests and runs come to as the counters
`hyperhive.review_classifier.requests`, `.verdicts` and `.checks` and the
histogram `.run.duration`, and the service counts each ask it repeats because
an answer failed validation as `hyperhive.review_classifier.ask.retries`. The
[metrics table](../scheduler/observability.md#review-classifier-result-metrics-swarm-controller)
lists their labels, and the `review-classifier` Grafana dashboard charts them.
The classifier logs each run's result to its journal, which ships to
VictoriaLogs like every swarm service container's
([`services.md`](../../2-handbook/swarm/services.md#logs-victorialogs)).
## Still open
These get settled, and filled in here, as the work reaches them:
- **Cache strategy.** Fully independent sessions per classification, or one
warm session and then the rest in parallel. To measure through the cache
token counts before choosing.
- **Oversized diffs.** What happens when even the forge's 3-line diff doesn't fit
the model's context. Lockfiles already collapse to one line each, as in
`hive-forge diff`.
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
cap.
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
goes to a third-party provider. Which repos may use which runtime, and
whether anything gets redacted.
- **Prompt injection.** A PR's diff and commit messages are model input, so
a PR can try to talk the classifier into approving it.
- **Subscription credential.** Where in bao the subscription runtime's login
lives. The service account's forge token is at
`secret/swarm/services/review-classifier/forge/token`.
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
classifier passed a PR before it starts its own review.