mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
22 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/4-implementation/crates/README.md Normal file
View file

@ -0,0 +1,13 @@
# Crate reference
One page per Rust workspace crate — its own `README.md`, unchanged, served
here for browsing alongside the rest of the docs site. **The crate's own
`README.md` is the source of truth; nothing here is hand-maintained.**
`nix/packages/reference-docs.nix` generates every page at
build time straight from the crate's real `README.md`, so it can never
drift out of sync the way a hand-copied mirror would — edit the crate's
own README to change what shows up here.
For the one-line-per-crate index (which crate does what, at a glance)
see the main [`AGENTS.md`](../../../AGENTS.md#repo-map) repo map instead;
this section is each crate's own, longer word on itself.

478
docs/4-implementation/networking/gateway.md Normal file
View file

@ -0,0 +1,478 @@
# hive-gateway
Every host's nginx: the one front door for whatever this host serves. A swarm service running here (forge, matrix, SSO, the swarm UI, the metrics and log stores) declares its own vhost through the gateway; the gateway itself adds the hive's own surface — dashboard, per-agent UIs, and discovery for a server name pinned to the hive domain.
_For the operator configuring `services.hyperhive.gateway.*` on a host._ nginx and the hive resolver (dnsmasq) run on the host next to hive-c0re: they bind `:80`/`:443` and the bridge address.
You rarely switch it on yourself. `gateway.enable` defaults to off, and every module that serves a vhost or needs hive names to resolve sets `gateway.enable` / `gateway.dns.enable` with `mkDefault true` — the hive controller, each swarm service, CI.
## Vhost map
**Swarm services.** Each module declares its vhost on the host that runs the service, under the swarm domain:
| URL | upstream | declared by, when |
| --- | --- | --- |
| `<swarm>/` | swarm-ui dist (static), behind an authelia subrequest | `swarm-ui.nix`, `deploy.swarm-controller.enable` |
| `auth.<swarm>/` | authelia (unix socket `/run/swarm-authelia/authelia.sock`) | `swarm-authelia.nix`, `deploy.authelia.enable` |
| `auth.<swarm>/bridge/*` | swarm-authelia-bridge (unix socket `/run/swarm-authelia/bridge.sock`), prefix stripped; answers swarm-controller's client only | `swarm-authelia.nix`, `deploy.authelia.enable` |
| `forge.<swarm>/` | forgejo (`3000`) | `hive-forge/`, `deploy.forgejo.enable` |
| `chat.<swarm>/_matrix/*` | tuwunel (unix socket `/run/hive-matrix/tuwunel.sock`) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `chat.<swarm>/` | fluffychat-web static | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `chat.<swarm>/config.json` | inline JSON (FluffyChat boot config) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `<queue domain>/metrics` | the queue exporter's `/metrics` on loopback, behind an authelia `auth_request`; every other path, `/` included, returns `403`, so `/varz` and `/connz` are never reachable | `swarm-nats.nix`, the queue's host |
| `grafana.<swarm>`, `metrics.<swarm>`, `logs.<swarm>`, `otel.<swarm>`, `bao.<swarm>` | the matching swarm service | that service's module → [`swarm/services.md`](../../2-handbook/swarm/services.md) |
A service module declares its vhost as `services.hyperhive.gateway.vhosts.<name>`: the locations, `forceSSL` and any `extraConfig` it needs. The gateway renders it into `services.nginx.virtualHosts.<name>` and adds the `listen` set, the certificate for that name and the [security headers](#security-headers) (ahead of the vhost's own `extraConfig`), and drops `addSSL` from a vhost that sets `forceSSL`. The eval fails for a vhost that sets `listen`, `addSSL`, `onlySSL`, `enableACME`, `useACMEHost`, `sslCertificate` or `sslCertificateKey` itself, and for any vhost declared while `gateway.enable` is false. A name with a certificate from outside the gateway's leaves (the queue's) sets `certificate = { cert; key; }` instead. `hive-gateway/vhosts.nix` declares the gateway's own catch-all (`_`) the same way, `hive-c0re/vhosts.nix` the dashboard and the hive domain, and `glue-matrix-hive-discovery.nix` adds the matrix redirect to the hive domain where the homeserver runs, and discovery there when the hive domain is the pinned server name. The catch-all sets `securityHeaders = false` because it sends no response, and the gateway renders a vhost's `default` at `mkDefault` priority.
**The hive's own vhosts**: the dashboard's, named `<name>.hives.<swarm>` (`deploy.hive-controller.dashboardDomain`), and one named for the hive domain. They exist only on a host that runs a hive (`deploy.hive-controller.enable`); a host without a hive has none of these rows:
| URL | upstream | when |
| --- | --- | --- |
| `<name>.hives.<swarm>/` | dashboard dist (static, from `servedFrontend`), behind authelia | always |
| `<name>.hives.<swarm>/api/` | hive-c0re (`7000`), behind authelia | always |
| `<name>.hives.<swarm>/health/` | hive-c0re (`7000`) | always |
| `<name>.hives.<swarm>/api/docs/` | themed Swagger UI dist (static), behind authelia | always |
| `<name>.hives.<swarm>/agent/<name>/` | per-agent harness over its unix socket | `agents.conf` (runtime-generated) |
| `<swarm>/.well-known/matrix/{client,server}` | inline JSON | `deploy.swarm-controller.enable`, and `swarm.matrix.serverName` is the swarm domain (the default) |
| `<hive>/.well-known/matrix/{client,server}` | inline JSON | `swarm.matrix.serverName` is this hive's domain (a pinned homeserver) |
| `<hive>/matrix/` | 301 → `chat.<swarm>/` | `deploy.matrix.enable` and `gatewayHost` set |
| `<hive>/` (every other path) | 301 → `<name>.hives.<swarm>`, same path | always |
The catch-all `_` vhost answers any other `Host` with `444` (connection closed, no response). It's `mkDefault`, so to make your own vhost the default server, set `services.nginx.virtualHosts."_".default = false;` — an eval assertion names both when two claim it.
Only the host that **runs** authelia declares `auth.<swarm>`; a hive that merely uses SSO knows `swarm.authelia.url` but doesn't answer for that name. The server name must be exactly `swarm.authelia.domain` — authelia checks that `authelia_url` sits inside its session cookie domain at startup, and refuses to boot otherwise. The vhost carries no `auth_request` (that would put the login page behind the login it serves) and passes `X-Forwarded-{Proto,Host,Uri,For}`, because authelia decides by the original request.
⚠️ `auth.<swarm>` answering with the **sso unavailable** page means authelia isn't answering at all — check `journalctl -M swarm-authelia -u authelia-swarm`. While the user set is empty, a placeholder user marked `disabled` keeps authelia's user store non-empty, so it boots fine and serves a login page that refuses everyone; add the first account with `swarmctl user add …` ([`setup.md`](../../1-getting-started/setup.md)).
Per-agent UIs stay sub-path; forge and matrix get sub-domains → [Sub-domain shape](#sub-domain-shape-rationale).
## TLS modes
The gateway always terminates TLS. Which certificate it serves depends on what you configure:
| mode | config | cert source | `.well-known` scheme |
|---|---|---|---|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | the secret store issues a gateway leaf through the hive's own role (RSA-4096) | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
`tls.certDir` together with `tls.acme.enable` fails an assertion — pick one.
### ACME / Let's Encrypt (`tls.acme`)
The simplest production path for a public domain:
```nix
services.hyperhive.gateway = {
openFirewall = true;
tls.acme = {
enable = true;
email = "admin@example.com"; # required
};
};
```
nginx obtains and renews certs via the HTTP-01 challenge on `port` (default 80); they land in `/var/lib/acme/`, managed by nixpkgs's `security.acme`. Every name this host serves needs a public DNS record pointing here — the hive domain and each swarm-service name in the [vhost map](#vhost-map) — and `openFirewall = true` so Let's Encrypt reaches `/.well-known/acme-challenge/`.
### Self-signed TLS (default)
On by default, listening on `httpsPort` (default 443) beside the plain-http `port` (default 80). `httpsPort` only sets the listen port: swarm service URLs never carry a port, so clients reach them on 443.
The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `hive-tls-resign.service` (from `hive-tls.nix`) asks the store for a gateway **leaf** through this hive's own role, presenting the hive's own store identity (`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`), and writes it under `…tls.stateDir` (default `/var/lib/hive-tls`). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). On a host that runs no hive the import unit installs the swarm-services leaf there instead, or a throwaway self-signed pair until that leaf lands. [`swarm/ca.md`](../swarm/ca.md) has the role, the identity and what a hive without one does.
⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the source's `0600 root:root` key), and it puts a file at **every cert path the config names**, because nginx refuses a config naming a missing cert file and one missing leaf would take down every vhost, not just one. Where the gateway waits for the swarm-services leaf, a missing one fails the import unit, naming `swarm-services-cert.service`, and nginx stays down until that unit issues it. Where it doesn't — a host fronting no swarm name, or only the store's UI, which starts its passthrough with the store sealed — the hive leaf stands in until the services leaf lands, or on a host that runs no hive a throwaway self-signed pair.
**Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them.
**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. The CI and forge containers and federating peers read the bundle. Agents fetch the root from the store themselves ([`swarm/ca.md`](../swarm/ca.md#distributing-the-root)).
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https://<host>/.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap.
**Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive's role allows nothing outside `<hive>`, so a swarm-service name can't go on this leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
**Rotation**: `hive-tls-resign.service` runs at boot, on every deploy and once a day, and asks the store for a new leaf when the current one is missing, past half its 720-hour lifetime, doesn't cover the configured names, doesn't match its key, or doesn't chain to the store's root. A leaf already on disk lets the gateway start while the store can't answer; a failed renewal leaves it in place. To force a rotation, delete `gateway.pem` under the state dir and restart the unit; it re-imports the leaf and reloads `nginx` itself.
**Cert prompts**: browsers warn once per host until you add the hive's `trust-bundle.pem` (the anchor, not the leaf) to the browser or OS trust store. A container that needs to trust it for its own outbound TLS gets the bundle a different way, bind-mounted read-only at `/run/hive-ca/trust-bundle.pem` (`nix/host-modules/lib/hive-ca-trust.nix`).
### Operator-provided cert (`tls.certDir`)
For a cert from a real CA (Let's Encrypt via your own `security.acme`, a corporate CA):
```nix
services.hyperhive.gateway = {
tls.certDir = "/var/lib/acme/example.com"; # nixpkgs security.acme output dir
# tls.certName = "cert.pem"; # default — matches security.acme layout
# tls.keyName = "key.pem"; # default — matches security.acme layout
};
```
nginx reads the directory directly. Keep the key readable by nginx:
- `security.acme` writes keys `0640 root:acme`, which the `nginx` user can't read. Set `security.acme.certs."example.com".group = "nginx";` (or make the key `0644` if your threat model allows). Otherwise nginx fails at startup with the reason in the journal.
### Fronting with an external TLS terminator
Give the gateway the real cert (`tls.certDir` or `tls.acme`) so it serves proper TLS itself, or front it over a unix socket rather than a plain-http TCP port. `.well-known/matrix/*` responses always advertise `https` ([Discovery flow](#discovery-flow-matrix)).
## Dashboard access
The dashboard answers on `<name>.hives.<swarm>` (`deploy.hive-controller.dashboardDomain`, derived from the hive name and `swarm.domain`), behind an authelia `auth_request`. Log in with your authelia account; only accounts in the `admins` group get in, the same rule as the swarm UI. Add one with `swarmctl user add <name> --group admins` ([`swarm/sso.md`](../../2-handbook/swarm/sso.md)).
The name sits under the swarm domain so authelia's session cookie reaches it, whatever the hive's own domain is. Any path on the hive domain other than the matrix routes redirects there, so old bookmarks still land.
**What it gates:** `/`, `/api/` and `/api/docs/`. **Not gated:** `/health/` (for uptime monitors; status only) and the per-agent `/agent/<name>/` routes, which come from `agents.conf` and carry no `auth_request`.
**Certificate:** in self-signed mode the name is a SAN on the hive's own leaf, issued through its own store role, so it needs nothing extra. With `tls.acme` it needs a public DNS record like any other vhost. With `tls.certDir` your certificate must name it; a `*.<swarm>` wildcard doesn't cover `<name>.hives.<swarm>`.
## Firewall posture (host-level)
`services.hyperhive.gateway.openFirewall = true` opens `port` and `httpsPort` on the host firewall — both, since the gateway always serves TLS. It defaults to off; set it for any reach from outside the host.
nginx is the one external entry point. The per-agent web-port range (`8100`–`8999`) stays closed: agents serve their UI on a unix socket ([Per-agent unix-socket upstream](#per-agent-unix-socket-upstream)), and the hashed TCP port (`lifecycle::agent_web_port`) is only a fallback bind for an agent missing `HIVE_WEB_SOCKET` — the gateway never proxies through it.
The dashboard port (`services.hyperhive.deploy.hive-controller.dashboardPort`, default 7000) binds `127.0.0.1` only, so remote dashboard access goes through the gateway. The dashboard can stop, rebuild and destroy agents; don't expose it without a reverse proxy in front.
nginx serves its `stub_status` counters on `127.0.0.1:<statusPort>` (`services.hyperhive.gateway.statusPort`, default 8085) at `/stub_status`, from a server of its own outside the vhost tree. The hive's collector reads them with its `nginx` receiver, and the `gateway` Grafana dashboard plots them. The receiver only dials `http://`, which is why this is loopback TCP rather than a unix socket.
## `HIVE_FORGE_URL`: agents reach the forge via the gateway by domain
Agents poll `HIVE_FORGE_URL` for Forgejo notifications and run every `hive-forge` call against it. `nix/host-modules/hive-c0re/environment.nix` sets it to `http://<swarm.forge.domain>` (default `forge.<swarm-domain>` — a swarm runs one forge). Agents run in a private netns and can't reach the host's loopback, so they resolve that name through the bridge dnsmasq to the bridge IP and reach nginx on port 80 (the bridge firewall opens 80 and 443) — the same path an operator's browser takes.
## hive-forge container shape
Private Forgejo in a nixos-container named `hive-forge` (not `h-*`, so c0re's lifecycle scanner leaves it alone; manage it with the standard `nixos-container` CLI). The container keeps it from colliding with any `services.forgejo` the operator already runs on the host: separate systemd namespace, separate state dir, separate port.
It shares the host network namespace (`privateNetwork = false`): the container is for state and unit isolation, not network isolation. Agent containers, by contrast, are network-isolated and reach the forge through the gateway ([`HIVE_FORGE_URL`](#hive_forge_url-agents-reach-the-forge-via-the-gateway-by-domain)).
State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/` and survives restarts and reboots; destroy the container to wipe it.
Only the swarm's forge host runs it (`services.hyperhive.deploy.forgejo.enable`, see [`swarm/services.md`](../../2-handbook/swarm/services.md)). Every other hive reaches that host's gateway by `forge.<swarm-domain>`.
### Network and port configuration
```nix
# The SSH port is swarm-wide: every hive dials it. The HTTP port and whether
# THIS host opens them in its firewall are this host's deployment decisions.
services.hyperhive.swarm.forge.sshPort = 2222; # default — git-over-SSH, off the host's openssh on 22
services.hyperhive.deploy.forgejo.httpPort = 3000; # default — outside hyperhive's 7000 / 8100-8999
services.hyperhive.deploy.forgejo.openFirewall = false; # default
```
`sshPort` serves `git clone/push/pull` over SSH (`git@<domain>:owner/repo.git` with `-p 2222`). SSH goes straight to Forgejo, not through nginx.
`openFirewall` (default `false`) opens `httpPort` and `sshPort` on the host. Agents don't need it — they come through the gateway. Set it for a browser reaching `http://<host>:<httpPort>/` directly, or for external git clients pushing over SSH. Forge's gateway vhost doesn't need `openFirewall` — the gateway's own `openFirewall` option covers that path.
### `rootUrl` override
```nix
services.hyperhive.swarm.forge.rootUrl = "https://forge.example.com/";
```
`rootUrl` (default `null`) overrides the Forgejo `ROOT_URL` that's autoderived as `https://<forge.domain>/`. The gateway always terminates TLS, so the forge is always advertised over `https://`. Set `rootUrl` explicitly when `forge.domain` differs from the public URL, or for a bespoke shape such as an external reverse proxy on another host or path. It must end with `/` (an assertion enforces this).
## Security headers
Every named vhost sets these at server scope (the `_` catch-all only closes connections):
| Header | Value |
|--------|-------|
| `X-Frame-Options` | `SAMEORIGIN` |
| `X-Content-Type-Options` | `nosniff` |
| `Referrer-Policy` | `strict-origin-when-cross-origin` |
nginx doesn't merge `add_header`: a `location` that sets its own (the CORS locations `/.well-known/matrix/client` and `/_matrix/`) inherits none of the server-scope headers, so those locations repeat them. Locations with no `add_header` of their own pick them up.
### HSTS (`gateway.hsts`)
Opt-in, off by default:
```nix
services.hyperhive.gateway.hsts = {
enable = true; # default: false
maxAge = 31536000; # default: 1 year (required for preload list)
includeSubDomains = true; # default: true
};
```
When enabled, every vhost adds `Strict-Transport-Security: max-age=…[; includeSubDomains]`. HSTS pins https in the browser: a deployment that later loses TLS locks browsers out until `max-age` expires, so enable it only when TLS is permanent.
## Local dev (`localHostsEntry`)
`services.hyperhive.gateway.localHostsEntry = true` maps to `127.0.0.1` in the host's `/etc/hosts`:
- the hive domain;
- every name a module on this host contributes to `gateway.localNames` — the dashboard's `<name>.hives.<swarm>` where a hive runs, and each swarm service this host runs adds its own (`forge.<swarm>`, `chat.<swarm>`, `auth.<swarm>`, the swarm UI's apex, …).
`services.hyperhive.deploy.singleHostSwarm` turns it on. Leave it off with real DNS.
## Discovery flow (matrix)
The server name's own vhost serves discovery, whichever host runs the homeserver. By default that vhost is the swarm apex on the controller's host; a homeserver pinned to a hive domain gets it on that hive's vhost instead. Both build the JSON from `swarm.matrix.gatewayHost` alone, and the `base_url` carries no port.
The operator points a client at the server name (`<server-name>` below):
1. The client fetches `https://<server-name>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://chat.<swarm>"}}`.
2. It connects to `chat.<swarm>/_matrix/client/...`.
3. The gateway proxies `/_matrix/*` to tuwunel over its unix socket, `/run/hive-matrix/tuwunel.sock`.
matrix-dart-sdk (FluffyChat and others) always fetches the well-known over `https`, which is why [self-signed TLS](#self-signed-tls-default) is on by default.
Federation peers fetch `.well-known/matrix/server` → `{"m.server":"chat.<swarm>:443"}`. The port is explicit even though it's the https default: a delegated host without a port means the federation default 8448, not 443. Peers then reach `/_matrix/` on the chat vhost through the gateway, so the gateway must be reachable from them (`gateway.openFirewall`).
## Sub-domain shape (rationale)
Sub-domain for forge and matrix, sub-path for per-agent UIs:
- forgejo's default `ROOT_URL = http://<host>/` works without `X-Forwarded-Prefix` handling; sub-domain hosting is the canonical Forgejo shape.
- matrix deployments put the API listener on its own name, and federation expects that. `gatewayHost` defaults to `chat.<swarm-domain>` rather than the spec-conventional `matrix.<server_name>`; set it explicitly for the conventional label.
- per-agent UIs are hyperhive-internal and base-path-aware for `/agent/<name>/`. A sub-domain per agent would multiply DNS and TLS cost for nothing.
- cookie and storage isolation: a forge XSS can't reach the dashboard session, because they're different origins.
`services.hyperhive.swarm.forge.domain` and `services.hyperhive.swarm.matrix.gatewayHost` take a full hostname (`forge.darkest.space`, `git.example.com`), not a label glued to a domain.
## Tuning knobs
Per-vhost timeouts and body-size limits live in the location blocks:
- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), websockets on.
- matrix `/_matrix/` (tuwunel): `client_max_body_size` = `deploy.matrix.maxRequestSize` + 1 MiB, so tuwunel stays the tighter limit and returns a matrix error a client can act on; `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*`, websockets on.
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE and WebSocket UIs), websockets on, `X-Forwarded-Prefix` set so the harness can build absolute URLs.
## Internals
How the gateway does what the sections above describe. Read before changing `hive-gateway/`, `gateway_nginx.rs` or `agent_sockets.rs`.
### SPA fallback (Accept-header pattern)
The `chat.<swarm>` vhost (fluffychat) serves a flutter/SPA bundle via the Accept-header pattern below. Per-agent UIs solve the same two requirements differently, with file-existence `try_files` — see [Per-agent static frontend split](#per-agent-static-frontend-split). The dashboard routes by **path** — see [Dashboard: path-based routing](#dashboard-path-based-routing-not-accept-header) below. Two requirements collide for a vhost whose client-side router owns every route under `/`:
- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
- a non-navigation request that isn't an on-disk asset must NOT get HTML with the wrong content-type
Solution: an `nginx http`-context `map $http_accept $matrix_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; everything else (`Accept: image/*`, `*/*`, `application/json`, `text/event-stream`, …) gets a sentinel nonexistent path, so `try_files $uri $uri/ $matrix_spa_target =404` falls through to a plain `404` — a missing asset is just missing.
#### Dashboard: path-based routing (not Accept-header)
hive-c0re serves exactly two prefixes, so the dashboard routes by **path** — deterministic, where a content-type split would let one URL resolve differently by the caller's `Accept` header:
- `location /api/` → hive-c0re (`7000`): all dashboard data, actions, and the two SSE streams (`/api/dashboard/stream`, `/api/build-logs/id/{id}/stream`). `proxy_buffering off` and a 1d read timeout keep the streams live.
- `location /health/` → hive-c0re: liveness and readiness.
- `location /` → the dashboard dist (from the `servedFrontend` nix-store path) with `try_files $uri /index.html`.
The gateway static-serves the dist while hive-c0re stays API-only, so a frontend-only change doesn't restart the core daemon. A new top-level c0re route prefix needs a matching `location` on the dashboard vhost (`hive-c0re/vhosts.nix`).
### Per-agent unix-socket upstream
Every agent binds its web UI on a unix-domain socket at
`/run/hive-agent/<name>/web.sock`. The mechanism:
1. **Agent side**. The nix module sets `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
on every harness service env; `web_ui::serve` binds a
`UnixListener` at that path.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
(`/run/hive-agent/<name>/`) into the agent's container. Dir
bind, not file bind — file bind-mounts don't survive the
harness's `unlink + bind(2)` cycle on socket replace. Per-agent
subdir keeps each agent's container from seeing siblings' sockets.
The dir is `0751`, owned by the agent's container uid/gid (hive-priv
creates it `0751 root` before each start when missing, and the
container's activation hands it to the agent user), so
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
own `0666`. The gateway is one of three principals sharing that dir
and doesn't own its ownership rules — see
[`docs/2-handbook/trust-boundary/boundary.md`](../../2-handbook/trust-boundary/boundary.md#the-per-agent-socket-dir).
3. **Marker gate**. After a successful `bind_unix`, the harness drops
`<dir>/hyperhive-socket-bound` next to the socket. c0re's
`agent_sockets::write` filters its JSON map by marker presence —
only agents whose harness has bound the socket appear there.
It also accepts `.bound`.
4. **Gateway side**. `gateway_nginx::write` generates
`/var/lib/hive-gateway/conf/agents.conf` — a plain nginx include
file with one `location /agent/<name>/` block per agent. Always
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
if the socket isn't yet bound, nginx returns 502 caught by the
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
nginx includes `/var/lib/hive-gateway/conf/agents.conf` — the same
path c0re writes, since both run on the host.
After each write, c0re triggers the appropriate nginx action via
`hive-priv` (which is root; hive-c0re runs as the unprivileged
`hive-core` user and can't act on a system unit).
`hive-priv` queries `ActiveState` and dispatches:
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
- otherwise → `systemctl start nginx`
This is an explicit trigger rather than a path unit watching the
file: the write and the reload belong in one causal chain c0re can
retry and report on (`RELOAD_PENDING`), not two units racing on an
inotify event.
<!-- vale write-good.Passive = NO -->
c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
marker poll tick (`agent_sockets::spawn_poll`). `write()` is
idempotent — skips the rename when content is unchanged.
`gateway_nginx::reload_if_pending` automatically retries failed reloads
on subsequent poll ticks.
<!-- vale write-good.Passive = YES -->
`agents.conf` goes out through a unique tmp file and an atomic `rename()`, so a crashing
c0re process never leaves a partial or unparseable file behind.
### Per-agent static frontend split
hive-c0re's environment sets `HIVE_AGENT_FRONTEND_DIR` to the `agent/`
subdirectory of `services.hyperhive.deploy.hive-controller.servedFrontend`.
The nginx include generator (`gateway_nginx::write`) reads
this variable and, when set, emits split location blocks per agent
instead of a single proxy block.
**Location priority for `/agent/<name>/...`:**
```nginx
# 1. Compiled assets — content-addressed nix store path, cache forever
location ^~ /agent/<name>/static/ {
alias <frontend>/static/;
expires 1y;
add_header Cache-Control "public, immutable, max-age=31536000";
}
# 2. Static dist + proxy fallback for dynamic paths
location /agent/<name>/ {
alias <frontend>/;
try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}
# 3. Proxy catchall — API, events, icon, send, login, …
location @<name>_dynamic {
proxy_pass <upstream>;
proxy_set_header X-Forwarded-Prefix /agent/<name>;
proxy_intercept_errors on;
error_page 502 503 504 = /__hive_agent_unreachable;
# … (full proxy header block)
}
```
**`try_files` resolution** (nginx applies the `alias` mapping before
checking file existence):
| request | resolved | outcome |
| --- | --- | --- |
| `/agent/iris/` | `<frontend>/index.html` | main agent page |
| `/agent/iris/stats` | `<frontend>/stats.html` | stats page |
| `/agent/iris/screen` | `<frontend>/screen.html` | screen page |
| `/agent/iris/static/app.js` | caught by `^~` block first | served with immutable cache |
| `/agent/iris/api/state` | no file match → `@iris_dynamic` | proxied to agent daemon |
| `/agent/iris/events/live` | no file match → `@iris_dynamic` | proxied (SSE) |
Adding a new HTML page to the frontend dist (`dist/<page>.html`)
makes it reachable at `/agent/<name>/<page>` with no generator change.
**Why `^~` for `/static/`**: the `^~` prefix gives this block higher
priority than the plain prefix `location /agent/<name>/`, so compiled
JS/CSS assets skip `try_files` entirely and get the immutable cache
headers. Nix store paths are content-addressed — the hash changes on
any content change — so `max-age=31536000` is safe.
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
store path baked in at hive-c0re build time, and c0re (writing
`agents.conf`) and nginx (serving files from it) are on the same machine,
so they see the same store.
**Unset variable**: if `HIVE_AGENT_FRONTEND_DIR` is empty or unset, each
agent gets a single proxy block and nginx forwards all traffic to the
agent daemon.
**`extraFiles`**: per-agent `services.hyperhive.agent.frontend.extraFiles` sit in
`mergedDist`, not in the base `services.hyperhive.deploy.hive-controller.frontend` dist. They're not under
the nix-store `alias` path, so requests for them fall through
`try_files` to `@<name>_dynamic`, and the agent daemon serves them.
### Per-agent error pages
`/agent/<name>/` requests hit two failure modes; both get static
HTML pages instead of nginx's default error chrome:
- **Agent not found** (`/agent/<unknown>/...`) — no `location` for that
name in `agents.conf`. nginx's prefix match falls back to the bare
`/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
to `/__hive_agent_not_found` → serves `not-found.html` with a link
back to the dashboard.
- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
per-agent harness isn't responding (container restarting, crash
recovery). `proxy_intercept_errors on` + `error_page 502 503
504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.
`hive-gateway/error-pages.nix` renders every page (`notFound`, `unreachable`,
`ssoUnavailable`) from one template with `pkgs.writeText`;
nginx serves each through an `internal` location with `alias` to the file,
so only nginx's own error handling can reach them. They depend on nothing
from the frontend dist and render even when hive-c0re is down. A service
module reaches them through `gateway.lib.errorPages`.
A route earns a custom page when the default status code would blame the
wrong component. The per-agent routes qualify (a 502 there means the
harness is restarting, not a gateway fault), and so does
`auth.<swarm>` — a bare 502 there blames the proxy, but the gateway is
fine; authelia itself isn't answering.
Forge, matrix and fluffychat keep nginx's defaults: a dead upstream there
means what the status code says.
### Dashboard link shape (gateway vs direct)
The dashboard builds per-agent links as same-origin `/agent/<name>/…` URLs
when `StateSnapshot.gateway_enabled` is true, and direct
`http://<host>:<port>/` links otherwise. The flag comes from
`HIVE_GATEWAY_ENABLED`, which `hive-c0re/environment.nix` sets to `1` on
every hive, so the direct shape is a fallback for the variable being unset.
Three render sites follow it: the agent-name link, the favicon fetch
(`<url>/icon`), and the nav-strip `container`-kind links from
`DashboardState.links` (`GET /api/dashboard-state`). Forge links come from
`swarm.forge.publicUrl`; with it unset the dashboard hides them rather than
guess. See `docs/2-handbook/web-ui/dashboard.md::Container row` for the frontend side.
### Dialing another vhost by name (`verifiedProxyTo`)
<!-- vale write-good.Passive = NO -->
`vhost-lib.nix`'s `verifiedProxyTo` builds the `proxy_ssl_*` /
`proxy_set_header` block a module uses to dial another service on this
same gateway BY NAME over https, verified. One definition rather than a
copy per module: nginx verifies nothing by default
(`proxy_ssl_verify` is off), so a `proxy_pass https://…` without these
lines is encrypted and unauthenticated. That failure is invisible — it
works, and keeps working, against any certificate at all.
<!-- vale write-good.Passive = YES -->
Every line earns its place, each confirmed against a real nginx with
the opposite arm run as a control:
- `verify` + `depth` — the chain is leaf -> intermediate -> root.
- `trusted_cert` — the bundle; nginx reads ALL certs in the file, which
the bundle's own doc warns isn't true of every consumer.
- `ssl_name` — checks the HOSTNAME too. Without it a chain-only check
accepts any certificate this CA ever signed, and for an internal CA
that's every service on the hive.
- `server_name on` — sends SNI, or the far end can't pick a cert.
**⚠️ Session-cache footgun**: the module leaves `proxy_ssl_session_reuse` at its
default (on), deliberately — nginx uses this on per-request auth
subrequests, so every request pays the handshake it avoids. Worth
knowing when testing though: nginx keys the session cache by upstream
address and NOT by trust config, so two locations pointing at one
upstream with different trust don't verify independently.
**⚠️ Host-header clobber footgun**: `verifiedProxyTo` also pins `Host`
(and reinstates the rest of nginx's `recommendedProxySettings` header
set) to the target `name` rather than letting nginx fill it in later.
`name` here resolves back to THIS gateway — every consumer dials another
vhost on the same nginx, not a separate host — and nginx picks the vhost
to answer an HTTPS request from the `Host` header, not from the TLS SNI
that `proxy_ssl_name` sends. `recommendedProxySettings`'s own `Host
$host` (the CALLER's host, not the target) is textually appended by
nixpkgs AFTER a location's `extraConfig` — so it always wins over a
`proxy_set_header Host` written in the location body, and the subrequest
loops back into the ORIGINAL vhost instead of reaching the target,
recursing on its own `auth_request` until nginx's subrequest-depth limit
turns it into a plain 500. Every call site sets `recommendedProxySettings
= false` on the location for exactly this reason — nixpkgs' version
would still clobber this one.

580
docs/4-implementation/networking/network.md Normal file
View file

@ -0,0 +1,580 @@
# hive-network
The bridge every container on a host attaches to, and the private-netns
isolation of agent and CI containers behind it. It comes up on any host that runs agents or swarm services:
`services.hyperhive.network.enable` defaults to off, and the modules that
need the bridge set it with `mkDefault true` — the hive controller, CI,
the hive collector, and the gateway's resolver, which every swarm service
turns on. Configured via `services.hyperhive.network.*`.
Agent containers always run in a private netns.
## Network map
One picture of the whole hive — two planes: **infra
containers share the host netns** and bind host ports directly;
**compute containers (agents + CI) each get a private netns** behind
the bridge. The unix-socket control plane rides the VFS, and none of
that touches it.
```
internet
│ uplink NIC — NAT MASQUERADE for the
│ bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge │
│ │
│ nginx :80/:443 [hive-gateway] │
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
│ │
│ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
vb-h-<a> vb-h-<b> vb-<runner> veth pairs
│ │ │
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
│ agent a │ │ agent b │ │ runner │ each; eth0 leases
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
└─────────┘ └─────────┘ └─────────┘
```
| container | netns | IPv4 | listens / reached via |
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `swarm-review-classifier` | private, veth on bridge | DHCP pool | outbound only (classifier → queue, secret store, swarm collector); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end:
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
`dhcpcd` runs with `noipv4ll` and waits for an IPv4 address, so
`network-online.target` means a lease, never a 169.254.x fallback; units
that look up a swarm name at start order after it.
- **DNS** — agents and the service containers query `bridgeIp:53`; dnsmasq
answers hive zones authoritatively with the bridge IP, everything else
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
container points its own `resolv.conf` there, and one that instead
inherits the host's resolves no swarm name at all — those records exist
only on the bridge.
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
own dashboard name resolve to the bridge IP, land on nginx
`:80`/`:443`, and proxy to forgejo
`:3000`, tuwunel's unix socket, hive-c0re `127.0.0.1:7000`, or a per-agent
UI unix socket.
- **Internet egress** — agent default route points at the bridge IP;
the host forwards + masquerades out its uplink.
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
and reach each other's gateway/forge across it
([`docs/swarm/`](../../2-handbook/swarm/README.md)).
- **Control plane (no network)** — per-agent broker socket
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
sockets under `/run/hive-agent/` are unix domain sockets
bind-mounted through the VFS; private netns doesn't affect them.
## Container shape (where dnsmasq lives)
dnsmasq runs on the host itself, next to nginx — both come from the
gateway module, one front door for DNS and HTTP. Its `bind-interfaces`
listener sits on `bridgeIp`, on the host's bridge interface.
## Configuration
```nix
{
services.hyperhive = {
deploy.hive-controller.enable = true;
deploy.hive-controller.hiveName = "pr1ma";
swarm.domain = "darkest.space";
swarm.hives.pr1ma = { }; # -> domain = pr1ma.darkest.space
# network.bridgeIp = "10.42.0.1"; # default
};
}
```
The hive domain (`services.hyperhive.domain`, which the resolver answers
for) comes from this hive's entry in the swarm directory:
`swarm.hives.<hiveName>.domain`, default `<hiveName>.<swarm.domain>`.
Eval fails until `swarm.domain` and that entry exist
([`swarm/README.md`](../../2-handbook/swarm/README.md) § Hive identity config).
## Bridge addressing
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
RFC 1918 space, unlikely to clash with operator's existing setup;
override `bridgeIp` + `bridgePrefixLength` if a different range is
already in use. `/24` gives 254 usable per-agent addresses — enough
for any single-host hive; bigger swarms or tighter addressing
schemes pick their own.
## Resolver behaviour
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
plus whatever swarm-service names this host contributes via
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
(matrix) when this host runs those services, `auth.<swarm-domain>`
when it runs authelia, and `hive-collector.hive` on every hive
— answering each with the bridge IP (where nginx
is reachable, and where the hive's own collector listens). Note forge and
matrix are swarm-domain names, not sub-domains of `<hive-domain>`: a swarm
runs one forge and one homeserver, so their names belong to the swarm rather
than to whichever hive happens to host them.
Every domain answered this way is also `local=` for dnsmasq, so the other
record types (AAAA, TXT, MX, HTTPS) for those names get an empty answer from
dnsmasq itself instead of waiting on the host's upstream resolver. Without it,
dnsmasq 2.86 and later send them upstream, and a host whose upstream doesn't
answer leaves each lookup to its client's own timeout.
dnsmasq forwards everything else to the host's
own resolvers: it runs on the host and reads the host's
`/etc/resolv.conf` directly. Containers don't need to know the
upstream — they query the bridge IP and dnsmasq does the right thing
per-name.
No fallback `server=` exists, by design: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
directly, so a network change (new router, new lease, laptop moving
networks) reaches it the moment openresolv rewrites the file. Nothing
needs synchronising, and no unit watches for it.
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for
container health-checks). External hosts can't reach it — no
DNS-amplification surface even when the operator opens port 80 for
gateway HTTP.
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
resolution stack — the host's resolver (systemd-resolved, plain
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
configured. The hive resolver is purely for inbound queries from
agent containers.
## Firewall posture
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
- Port 53 opens the resolver on the bridge interface only. Other
interfaces stay closed. The hive resolver isn't an external-facing
service.
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
receives DHCP via a regular UDP socket (it doesn't use a
netfilter-bypassing raw socket), so the hole is mandatory — without
it containers never get a lease and fall back to 169.254.x.x.
- Ports 80 and 443 let isolated agents reach nginx (on the host) for the forge sub-domain, per-agent
UI proxies, and any other HTTP services.
The **host** firewall is the only firewall. The swarm service
containers that share the host netns (forge, bao, the
metrics and log stores) run with `networking.firewall.enable = false`
(`nix/container-modules/swarm-container.nix`): a NixOS firewall inside a
shared-netns container runs against the _host_ ruleset — at container
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the
bridge holes above plus the agents' NAT. Private-netns containers
(every container with `privateNetwork = true`: agents and the service containers on the bridge) keep their own firewall — it's scoped to their
namespace.
### Reaching host services (`exposeHostPorts`)
By default agents can only reach the host on 80/443 (+53 DNS), so a
host-side service on another port — for example a dev OTLP collector you want
agents to reach directly — is unreachable. (hyperhive's own services
need none of this: a module that means to be reachable from agents opens
its own port here, which the hive collector and `nats.enable` both do. Note
that `deploy.swarm-otel.upstream.endpoint` is the _upstream_, which no agent ever dials. See
`docs/4-implementation/scheduler/observability.md`.)
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
agent can connect to `<bridgeIp>:P` (for a dev OTLP collector, point its
endpoint at `http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
The hive's own collector needs none of this: agents dial it by
`otel.collector.host`.
<!-- vale write-good.Passive = NO -->
This is **firewall-only**: the host service must bind an address
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
service bound to `127.0.0.1` only stays unreachable; rebind it to
`0.0.0.0`.
<!-- vale write-good.Passive = YES -->
The port is reachable by **every** agent on the bridge subnet (like
DNS/gateway), so only expose services safe for any agent to reach.
## Egress proxy (squid)
Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
(`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
only. An agent sends its outbound HTTP and HTTPS through it once its hive says
the proxy is ready and it has fetched its proxy credential (see _Agents and the
proxy_ below). Any other agent goes straight out through the NAT, as before,
and so does any program that ignores the proxy variables: nothing blocks direct
egress yet.
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
host. It never decrypts TLS, so it holds no signing key. It caches nothing.
- **Credentials.** A request for a listed name must carry a per-agent proxy
credential (HTTP Basic, the agent's name and its secret). `swarm-controller`
mints one per agent at `swarm/agents/<agent>/proxy-credential`, and never
replaces it. `hive-c0re` hashes the credentials of this hive's agents, and
one more for the self-check, into `services.hyperhive.deploy.hive-controller.squid.usersFile`
every 30 seconds. Squid's `basic_ncsa_auth` helper re-reads the file when it
changes. A stopped agent stays in the file. Squid caches a good login for a
minute. Nothing rotates a credential: to replace one, delete its leaf in the
store and the next pass mints a new one. HTTP Basic crosses the bridge in
clear between an agent and the proxy. See
[the credentials reference](../../2-handbook/swarm/credentials-reference.md).
A request without a valid credential gets a 407, once the destination
checks below have passed it.
- **Agents and the proxy.** An agent uses the proxy only when its hive says
this host's squid is ready for it. `hive-c0re` puts two facts in every
container's `/etc/hyperhive-hive-facts`, only on a hive that holds a store
identity: `HIVE_SQUID_PROXY` (`services.hyperhive.deploy.hive-controller.squid.listen`, the bridge IP
and port) and `HIVE_EGRESS_NO_PROXY` (the direct hosts below). A hive that
predates them writes neither, and its agents go direct.
`hive-agent-proxy-credential.service` (`nix/agent-modules/egress-proxy.nix`)
runs in each agent's container, as the agent user, at start and every 10
minutes. With both facts present it logs in to the secret store with the
agent's own certificate, reads `swarm/agents/<agent>/proxy-credential` and
writes `/run/hive-agent-proxy-credential/proxy.env` (`0600`) with
`HTTPS_PROXY`, `HTTP_PROXY`, their lower-case forms, `NODE_USE_ENV_PROXY=1`
and `NO_PROXY`/`no_proxy`. The proxy URL is
`http://<agent>:<credential>@<bridgeIp>:3128`; Claude Code, Node, curl, git
and reqwest send the userinfo as `Proxy-Authorization`. The harness, its bash
tool, the subagents, the MCP servers and the pollers load the file with
`EnvironmentFile=-` when they start. Until the facts and the credential both
exist nothing writes the file, so those units have no proxy variable and go
direct; a failed read keeps the file already in place. Nothing restarts on
deploy: an agent takes the proxy at its next start, which is its next rebuild
or restart. The credential is in the environment of every process those units
start, so `env` in the agent's shell prints it. Login shells
(`nixos-container root-login`, SSH) don't load the file.
- **Direct destinations.** `HIVE_EGRESS_NO_PROXY` is the hosts this agent must
reach without the proxy, and `NO_PROXY` is `localhost,127.0.0.1,::1` followed
by that list. `hive-c0re` computes it from the host's options: the bridge IP,
the hive's domain (`services.hyperhive.domain`), the secret store and queue
names (`services.hyperhive.swarm.bao.domain`, `swarm.nats.domain`),
`services.hyperhive.swarm.serviceDomains` (the swarm UI and controller apex,
the forge, matrix, attic, logs, metrics and otel names) and every name the
gateway answers with the bridge IP (`services.hyperhive.gateway.localNames`).
Squid refuses every destination that resolves to the bridge, and the hive's
resolver answers the bridge address for those names, so they never go through
the proxy. A bare name matches itself and its subdomains. Node's `NO_PROXY`
has no CIDR support, so the list holds the bridge as an address.
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
second file beside the users file, `services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
the users whose agent holds the `unrestricted_internet` flag in the
permissions the hive last read from the swarm, one per line, rewritten when
it changes. The default is permissive: an agent with no record, or a record
that never set the flag, holds it, the same default as the
`hyperhive.agent.unrestricted_internet` metric. While the hive's read of the
permissions is stale or failing, a new agent that already holds a
credential counts as unrestricted. Squid reads the list when it
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
when the file changes. A missing or empty file lists no one, and squid
starts either way. A flag change reaches squid after the hive's next read of
the permissions and its next 30-second pass.
An older `hive-c0re` writes no list, so every credential on that hive is
restricted. Agents must not start sending credentials until every hive runs
the `hive-c0re` that writes it, or a legacy agent loses general internet.
- **Log mode.** The operator can put an agent in `log` mode in the swarm UI.
Its record keeps `unrestricted_internet` at `false` and adds
`internet_log: true`, so a `hive-c0re` that doesn't know the field still
enforces the list. For such an agent `hive-c0re` adds to `agent-allow.conf`,
after the allow rules for every agent's own names,
`http_access allow <user> !allowed_dst log_mark`, where `log_mark` is an
`annotate_transaction` ACL that sets `would_deny=1`. The rule comes last in
the include, so every destination deny and `deny !authed` before it still
applies, and a name the agent may reach never carries the mark. A request the
rule lets through gets a second log line,
`would_deny user=<agent> method=<m> host=<h> port=<p> result=<r>`, from the
`would_denied` `access_log` in `hive-squid.nix`; the usual request line keeps
its format. The **would-deny** tab of the `egress` dashboard in Grafana lists the hosts per agent. Only
the operator changes the mode: nothing moves an agent to `restricted` after a
time or a count. The self-check's users aren't agents,
so `deny-canary.invalid` stays a 403.
- **Modes.** An agent is in one of three modes, and the deny list (next bullet)
applies in all of them:
| mode | the deny list | a name not on the allow list |
| -------------- | ------------- | ---------------------------- |
| `restricted` | blocked | blocked |
| `log` | blocked | allowed and logged |
| `unrestricted` | blocked | allowed |
- **Deny list.** The swarm's deny list names destinations the proxy refuses
whatever else allows them. The swarm controller keeps it as its own value
(the `deny` key of the `egress-allow` bucket), each entry for every agent or
for one, and serves it at `GET`/`POST`/`DELETE /api/egress/denied-domains`
with the same body as the allow-list (`{"domain": ".example.com", "agent":
"atlas"}`). Write a name as in the allow-list: at least two labels, one
leading dot for the name and its subdomains. The controller refuses IP
addresses, other wildcards, ports and paths. Unlike the allow-list it doesn't
refuse the base list's names: denying one is the point. Every 30 seconds
`hive-c0re` reads the list and renders `services.hyperhive.deploy.hive-controller.squid.denyFile`
(`/var/lib/hive-squid/deny.conf`, `hive-core:squid`, `0640`) for the agents it
runs, and `squid-unrestricted-reload.path` reloads squid when it changes.
- **Precedence.** The configuration includes the file after the destination
denies and before `deny !authed` and every allow, so a denied name is
refused even when the base list, the hive's `allowedDomains`, the swarm's
allow-list, the unrestricted flag or `log` mode would allow it. A swarm-wide
entry names no user, so every client gets a 403 for it, one without a
credential too. An agent's entry also names the agent's squid user, so it
binds that agent: a client that sent no credential gets the usual 407 for
it. An agent's entry only adds to the swarm-wide ones; no entry of one scope
lifts another.
- **Log line.** A request a deny rule refused gets a second log line,
`egress_denied user=<agent or -> method=<m> host=<h> port=<p> result=<r>`,
in every mode. It differs from the `would_deny` line of `log` mode and
from the `TCP_DENIED/403` line every refusal has, so you can tell a
deny-list hit from a name that isn't on the allow-list.
- **Who writes it.** Only an operator's request to the controller writes the
list. It has no automatic entries: nothing derives a deny from an agent's
API-key environment or a linked account, and the stored type has no such
field. An agent has no queue grant that reaches the key, and the controller's
API sits behind the operator's SSO login: nginx checks it before it hands
a request to the socket, which only nginx and root can open.
- **Empty list.** With no entry `deny.conf` is empty and squid configures as
it did without the list: the configuration gains one `include` line and the
`egress_denied` log lines, and no rule. A hive on a `hive-c0re` that
predates the list never reads the key and keeps its old configuration, so
nothing on its wire changes.
- **Applied.** Each hive reports the revision of the list it last rendered as
`egress_deny_applied` in its status snapshot (see [the hive
snapshot](../../2-handbook/swarm/README.md)). `GET /api/egress/denied-domains` answers the
list's `revision` and `behind`, the hives that haven't reported it: an older
`hive-c0re` never does, so it shows there rather than silently allowing the
names. A hive that reads the list reports within about a minute. Reporting
means the file holds the rules; squid reloads moments later.
- **Base list.** `services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains` is open to every
authenticated user, whatever its flag: what the agent harness needs off the
swarm. The default is the hosts Claude Code requires
(<https://code.claude.com/docs/en/network-config>): `api.anthropic.com` for
the model API and the usage probe (`hive-agent/src/claude_usage_watch.rs`),
`claude.ai` for the account login, and `platform.claude.com` for the OAuth
token exchange and refresh. An agent whose `ANTHROPIC_BASE_URL` points
elsewhere needs no entry here: the swarm controller adds that host to the
agent's own swarm entries (see _Swarm allow-list_ below). The list omits the plugin marketplace clone from `github.com`, since a failed add
isn't fatal. The rules below still refuse an entry that resolves to a
private address.
- **Allow-list.** `services.hyperhive.deploy.hive-controller.squid.allowedDomains` defaults to empty,
plus the self-check canary (`services.hyperhive.deploy.hive-controller.squid.selfcheck.canary`), so
the canary widens the restricted list by that one name. The swarm's own
service names aren't on it: they resolve to the bridge or a private address,
which the rules below refuse, and agents reach them directly. Each entry
matches that exact name, or with a leading dot the name and its subdomains. A user on the unrestricted list reaches any
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
`allowedDomains` is the hive's own list, always allowed whether or not the
swarm has a list of its own (next bullet).
- **Swarm allow-list.** The operator adds domains in the swarm UI
([allowed domains](../../2-handbook/swarm/ui.md#allowed-domains)); the swarm controller
keeps them in one value of the `egress-allow` bucket, each for every agent
or for one agent. Every 30 seconds `hive-c0re` reads it and writes two files
beside the users file, `services.hyperhive.deploy.hive-controller.squid.allowedFile`
(`/var/lib/hive-squid/allowed`) and `services.hyperhive.deploy.hive-controller.squid.agentAllowFile`
(`/var/lib/hive-squid/agent-allow.conf`), `hive-core:squid`, `0640`, and
logs `squid allow list: 2 swarm, 3 agent entries` when they change. The
swarm-wide names join the allow ACL, so a restricted user reaches them. A
per-agent entry is a rule for that agent's squid user, `http_access allow
<user> <names>`, in the generated `agent-allow.conf` that the configuration
includes at the point where the base list gets its allow: after every
destination deny and
`deny !authed`, so the rules under _Denied before any allow_ still bind it,
and before the unrestricted rule. Only this hive's agents get a rule.
`squid-unrestricted-reload.path` reloads squid when either file changes, so
there is no deploy or restart.
A name is a DNS name of at least two labels. One leading dot (`.example.com`)
also allows every subdomain, and is the only wildcard; the controller refuses
IP addresses, other wildcards, ports, paths and the base list's names when
you add the entry, and `hive-c0re` checks each name again and skips one that
fails, or that the base list or the canary covers, or that another entry of
the same list covers. A name that resolves to a private address stays a 403.
The controller also adds entries itself, and only from the operator's own
actions in the swarm UI. Saving an agent's API-key environment makes it take
the host from `ANTHROPIC_BASE_URL` and from every variable named `*_BASE_URL` (the
address an ACP agent that loads the same file reads), and keeps one
`derived` entry for that agent per host. Each host passes the same check as
an operator's entry, so an IP address, `localhost` and a base-list name get
none. A save or a delete of the environment replaces the agent's derived
entries, and destroying the agent drops all its entries. The agent can
create its own `backend-env` leaf while the leaf is empty, so text found in
the store never creates an entry: when the controller starts, it only
drops derived entries whose exact host the stored environment no longer
names, and logs a host it didn't derive. An environment saved before this
existed gets its entries when the operator saves it once more. An agent on
the hand-placed `backendEnvironmentFile` alone, and the `opencode` preset's
`acp.opencode.provider.baseUrl`, get no derived entry, so add one in the
swarm UI. A `hive-c0re` that predates the field renders a derived entry
like any other.
Linking a GitHub token, an external matrix account or an external forge
account adds a derived entry for that agent per host it reaches, and
deleting the account removes it: `github.com`, `api.github.com`,
`raw.githubusercontent.com`, `codeload.github.com` and
`objects.githubusercontent.com` for GitHub, and the host of the matrix account's `homeserver` or the forge
account's `url`. Each entry's `source` (`link:github`,
`link:matrix:<account>`, `link:forge:<label>`) keeps it apart from the
environment's, which a save of the environment leaves alone; an entry
without a `source` counts as the environment's. Only the controller writes
these leaves, so when it starts it also adds the entries of accounts linked
before this existed and drops those whose account no longer exists, and logs
the counts (`link egress: entries match the linked accounts`). A store it
can't read changes nothing. An `agent.nix` `matrixAccounts` homeserver gets
no entry, since the controller never reads it; add one in the swarm UI.
The swarm's list adds to the hive's `allowedDomains`: `allowedFile` holds
those names plus the swarm-wide entries, so removing an entry never removes a
name the hive lists itself. Until the controller has a list (it predates the
feature, or nobody added an entry) `allowedFile` holds `allowedDomains` and
`agent-allow.conf` is empty. A `hive-c0re` older than this rule replaces
`allowedDomains` with the swarm's list once the controller has one, so every
hive must run the new `hive-c0re` before the controller's first entry exists. A queue that stops answering for the bucket
leaves a file the swarm wrote untouched, and a failed read leaves both
files untouched. The first line of a file written from the swarm's list is a
`# source: swarm` comment. The queue grant a hive needs to read the list
comes with the controller's `swarm-nats-auth`; a hive that can't read it
times out, keeps its files and warns after about 90 seconds.
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
after the destination checks, which look the name up on the host.
- **Denied before any allow.** Destinations that resolve to loopback,
link-local, the bridge subnet, a private range (`10.0.0.0/8`,
`172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`), shared address space
(`100.64.0.0/10`), `0.0.0.0/8`, `::`, multicast or reserved space, and IP-literal
hosts, for every user including the unrestricted ones and for listed names.
Squid resolves names on the host and connects from it, so without these a
name pointing at `127.0.0.1`, at the host's own LAN address or at a private
neighbour would walk around the bridge→loopback DROP rule and the swarm's
network. Squid checks the address it connects to, so a name that changes its
answer between a check and the connect gains nothing. A listed name must
therefore resolve to a public address on the host's resolver: a split-horizon
name that answers a LAN address gets a 403, and so does the self-check
canary, which is why it isn't a swarm service name. Only `CONNECT` to port
443 and plain HTTP to ports 80 and 443 pass.
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
`squid_client_http_requests_total` are the two to watch.
- **Logs.** One syslog line per request reaches the journal as
`squid.service`, and `services.hyperhive.otel.journalUnits` forwards that
unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the
client's source address. A request that a `log`-mode agent's rule let
through also logs a `would_deny` line (see _Log mode_). A request a deny-list rule refused logs an `egress_denied` line (see _Deny list_).
- **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service`
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
to `services.hyperhive.deploy.hive-controller.squid.selfcheck.canary` (default: `cache.nixos.org`,
a public name that's also on the allow-list)
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
the same secret and is always on the unrestricted list, and a `CONNECT` to
`services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary` (default:
`example.com`, a name outside the allow-list) must answer 200. It writes one
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
per run and fails the unit when any is wrong. The password is the hive's
`swarm/hives/<hive>/squid-selfcheck` secret, which
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't
written the users file, the line reads `credential=absent` or
`allow_status=407` and `ok=false`, and nothing else breaks. The
host's resolver looks up the canary, so a host whose resolver maps
that name to a loopback, bridge or private address fails the check.
## Container isolation
Each agent container runs in a private network namespace with a dedicated
veth pair attached to the bridge. The following table summarises what
the nix side sets up unconditionally:
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
`10.42.0.1/24`), **not** the canonical network address. The Rust side
must normalise (bitwise-AND with mask) before subnet membership checks or
address arithmetic.
### What the Rust side does
`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
and `HOST_BRIDGE=<bridgeName>` when creating or updating containers:
`lifecycle::set_nspawn_flags` hands them to hive-priv, which rewrites
`/etc/nixos-containers/<container>.conf`. hive-c0re validates both variables **once at
daemon startup**, not per container: they're process-global, so a
missing or malformed value is a misconfigured daemon rather than one bad
container, and failing at boot gives a single diagnostic instead of one
per agent. No non-isolated mode exists to fall back to. hive-c0re leaves `LOCAL_ADDRESS` empty so the
container's dhcpcd acquires an address from the bridge dnsmasq pool
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies to
every agent container, manager included.
`HOST_ADDRESS` is the bridge gateway IP (the address part of
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
so a non-`.1` operator override still resolves to wherever the bridge
actually lives). It's **load-bearing**: nixos-container's container-side
network setup only installs a default route (`ip route add default via
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode
nixos-container skips the host-side address/route setup, so writing it only affects the
container's default route — without it the container comes up with an IP
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
### How the isolated container gets its resolver
nixos-container copies the **host's** `/etc/resolv.conf` into the container
at every start. The host resolver (for example `127.0.0.53` from systemd-resolved,
or a LAN router) is unreachable from a private netns and isn't
authoritative for the hive's own zones, so it's replaced with the
bridge dnsmasq at boot. Because the copy happens on every start, it
would clobber a declarative `environment.etc."resolv.conf"` — so the
wiring is runtime:
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
gateway IP) into each container's `/etc`.
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
It's ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
the forge-token fetch so the resolver is correct before the first DNS lookup.
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
domain sockets bind-mounted into containers, not network listeners — see
the _Control plane (no network)_ bullet under [Network
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
that never touches the network stack.
The nix side also enables IP forwarding + NAT (agents reach the internet
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP.
## Cross-references
- `docs/4-implementation/networking/gateway.md` — vhost map + the gateway's other duties

238
docs/4-implementation/networking/snapshot-store.md Normal file
View file

@ -0,0 +1,238 @@
# Snapshot store
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
over the WireGuard mesh; a destination hive later pulls one back to
complete a migration.
Two things it's not, both worth stating because both are common
assumptions:
- **It's not the swarm controller**, and doesn't depend on one. It's
a NixOS host role: a btrfs subvolume tree, a socket-activated
receiver, and the `wg-hive` interface the swarm module already brings
up. That's why you can deploy it before any controller exists.
- **It's not a backup product.** It happens to hold the data a backup
would hold, and you should operate it accordingly (see
[Operating it](#operating-it)) --- but nothing in it does scheduling,
verification, or restore orchestration.
## Enabling it
```nix
services.hyperhive.snapshotStore = {
enable = true;
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
port = 51821;
};
# The mesh is a hard requirement, and is asserted:
services.hyperhive.deploy.wireguard = {
enable = true;
address = "10.100.0.9/24";
privateKeyFile = "/etc/wireguard/hive.key";
};
```
The store host is a swarm member like any other: it gets an entry in
`services.hyperhive.swarm.hives`, the same directory every host holds. See
[swarm/](../../2-handbook/swarm/README.md) for the mesh itself.
Note that `deploy.wireguard.enable` gates the mesh, **not**
`deploy.hive-controller.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## Pointing a hive at it
The block above configures the host that _receives_. You must tell every
hive that _pushes_ separately where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
address = "10.100.0.9"; # the store's mesh address, no prefix
port = 51821; # optional; must match the receiver's port
};
```
Two deliberate asymmetries in that pair, both liable to be misread as
inconsistency:
- **`address` has no default.** It's a deployment fact a pushing hive
can't derive, and a wrong guess means streaming an agent's state at
whatever happens to answer. Unset, a push fails naming this option.
- **`port` does default** (`51821`), because it's a convention both
ends read from the same option docs --- a default there is
coordination, not a guess.
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That's the distinction the
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
from this host, and a bare `services.hyperhive.<service>` describes _a
role this host performs_. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
[--parent <label>]` streams a snapshot straight into the store. No
destination argument exists, because a swarm has exactly one store (see
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
and no credential argument, because the mesh is the authentication.
## The mesh is the authentication
No certificates live here, and no key material of its own. That's
deliberate rather than an omission.
WireGuard's cryptokey routing already binds a peer's source address to
its public key: the swarm module configures each peer with
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
that address provably came from the holder of that private key. By the
time a packet reaches the receiver, the kernel has already
authenticated it.
Layering TLS client certs on top would authenticate _the same fact_ a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
## One subvolume per agent, not per hive
The receiver keys the destination by **agent**.
This isn't cosmetic. After a migration, an agent's next incremental
send arrives from a _different_ hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and can't choose
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
and the subvolume name inside it's chosen by the sender. The
protocol is one `agent <name>` header line, then the raw stream.
The receiver waits 30 s for the header, and systemd stops a transfer
that runs past 6 h in total. The sender gives connect and header write 15 s
each; it puts no limit on the stream itself.
The rule that matters:
> **The receiver owns the destination root. The sender-supplied name is
> validated, never used as a path.**
Validation is an allowlist --- `[A-Za-z0-9_-]+` and nothing else. No
slash and no dot means neither directory traversal nor an absolute path
can survive it. It's deliberately an allowlist and not a list of
forbidden characters: a blocklist only ever excludes the attacks
somebody already thought of.
## Reachability
The receiver is socket-activated, and the socket binds **this host's
mesh address**, never a wildcard. The module asserts three things: the
mesh is on, the address isn't empty, and it isn't `0.0.0.0` or `::` ---
none is just documentation: bound to a wildcard this socket is an
unauthenticated remote write into agent state.
The socket sets `FreeBind`, so the bind succeeds before `wg-hive` has
its address. Nothing orders the wireguard unit before `sockets.target`,
and without it the bind fails with `EADDRNOTAVAIL` at boot and the
receiver never listens.
Binding isn't sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, _before_ a packet reaches a bound socket, so
the module explicitly opens the port --- scoped to the mesh interface:
```nix
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
```
A host-wide `allowedTCPPorts` would open the port on every interface
including a public NIC, leaving only the socket's bind address between
the internet and a root `btrfs receive`.
## Operating it
### Confinement is the deployment's job
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
narrow `ReadWritePaths` --- but those are **defence in depth, not a
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
undo the namespace they set up.
The boundary is the machine. The intended deployments are:
- **a swarm**: the store is its own small VM. The machine is the
boundary, which is stronger than anything the unit could assert about
itself.
- **all-in-one / local**: the store runs as a container on the c0re
host.
The second is worth keeping deliberately, and not only for
convenience: it means every local deployment exercises the confined
path. The usual failure mode for an isolated variant is that
nobody runs it day to day, so it rots and turns up broken in
production.
⚠️ **The assumption to keep true over time:** the store host runs
nothing else. That's true on day one and quietly false the day someone
notices the box has spare disk. Nothing in the config objects when it
stops being true.
### It holds every agent's state from every hive
Which makes it the highest-value target in the swarm by a wide margin,
and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The pitfall is the label: this box holds backup-grade data while nobody
calls it a backup, so it can end up with backup-grade _exposure_ and
non-backup-grade _controls_. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
The snapshot covers an agent's **state subvolume**, which is the parent
of `state/`, `claude/` and `harness/` (see
[`docs/2-handbook/agent-lifecycle/persistence.md`'s btrfs subvolume
section](../../2-handbook/agent-lifecycle/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
for how and when hive-c0re creates that subvolume). Consequences:
<!-- vale write-good.Passive = NO -->
- The Claude session (`claude/`) travels, so a restored agent keeps its
live `--continue` session rather than needing to log in again.
- `harness/` travels too, including `harness/bash-tasks/`. Task output
is part of an agent's working continuity, so this is wanted --- but it
means anything that has ever leaked into a task's captured output is
in the retained snapshots as well.
<!-- vale write-good.Passive = YES -->
It does **not** cover the agent's config (the forge's `agent-configs/<name>`) or
its topology entry, both of which live outside the subvolume. A restore
therefore yields an agent's memory without its definition.
### Retention
<!-- vale write-good.Passive = NO -->
Retention lives on the _sending_ side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.
<!-- vale write-good.Passive = YES -->
Per-agent or per-hive `btrfs qgroup` quotas aren't configured yet.
Without them one runaway hive can fill the store and take out every
other hive's snapshots.
## Not built yet
**The pull side.** Push is safe with minimal authorisation because a
hive can only ever write to a chain it owns. Pull is the direction that
needs a policy: unrestricted, any compromised hive could read every
agent's state from every other hive. It needs a notion of which hive
currently owns which agent, and that ownership record lands with the
swarm controller work.
With a single hive the question is trivial --- the only peer owns
everything it sends --- which is why the receive half ships first.

363
docs/4-implementation/scheduler/ci.md Normal file
View file

@ -0,0 +1,363 @@
# swarm-ci-runner: Forgejo Actions Runner
The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
`swarm-ci-runner` nixos-container, executing CI jobs from
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
whichever host sets it, and needs neither the forge nor a hive on that host.
## For operators
**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
in the NixOS config of the host that should run it. No manual token
provisioning: swarm-controller publishes the forge's runner registration token
to the secret store, and the runner reads it with a store identity of its own.
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
`/var/lib/swarm-bao-pki/` and name them in
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
`ciRunnerClientKeyFile`.
Two things worth knowing before you rely on it:
- **Only run CI for contributors you trust.** Builds run **unsandboxed**
(an nspawn limitation, not a choice) — a malicious build script in a
PR could make network requests or write to the container filesystem
during the build. Fine for a small trusted-team hive where everyone
already has forge access; if you take outside/fork contributions,
gate CI behind Forgejo's fork-PR-approval setting or restrict the
workflow to push-only triggers (forks can't push to your branches).
See _Security_ below for the full threat model.
- **Watch your disk.** CI builds through the host's nix store with no
automatic garbage collection of its own — a busy CI day can fill
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
disk-pressure GC config from _Host store maintenance_ below to your
host's NixOS config (not optional if you plan to actually use this).
Everything below this point is implementation detail: registration
internals, container design, and the full security writeup.
## CI checks
<!-- vale write-good.Passive = NO -->
The jobs run on every PR (and on `workflow_dispatch` for manual
re-triggers), defined in
[`.forgejo/workflows/ci.yml`](../../../.forgejo/workflows/ci.yml). The first four
below are required checks (forge branch protection) — a hit on any of them
blocks merge. The two vale jobs split what used to be a single check: the
error-only one exists so it _can_ become a required check without also
gating on the full job's standing warning/suggestion backlog, but flipping
that switch on the forge is a separate operator action, not something this
table can assert ahead of it happening.
<!-- vale write-good.Passive = YES -->
| Job | What it runs | Required? |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| **nix flake check** | `scripts/ci-flake-check.sh`: nix-eval-jobs, then builds each check (treefmt, clippy, tests, module eval); no VM tests | yes |
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) | yes |
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) | yes |
| **doc-pointer lint** | flags `docs/…md` paths and relative markdown links that no longer resolve (`scripts/check-doc-refs.sh`) | yes |
| **attribution-trailer lint** | flags `Co-Authored-By` trailers with Claude/Anthropic attribution in PR commits (`scripts/check-attribution-trailers.sh`) | not yet — intended to be |
| **prose lint (vale, errors)** | vale, `docs/` tree, error-severity rule hits only | not yet — intended to be |
| **prose lint (vale)** | vale, `docs/` tree, full severity range (warnings + suggestions too) | no — advisory, standing backlog |
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
without an empty commit.
<!-- vale write-good.Passive = NO -->
**`ci-rerun --pr` verifies the code but doesn't reliably move the PR's own
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
JSON): a genuine PR-triggered run carries `event: pull_request` with the
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
carries `event: workflow_dispatch` with the real branch name instead.
Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
runs specifically, so a dispatched run — even a genuinely successful one —
never writes to that status slot. Reproduced with three dispatches on one
PR over 20+ minutes, all genuinely green, the PR's own status unmoved
throughout. **Always re-check `pr status` after a dispatch and believe
what it says — don't push a commit just to unwedge it, that's the
empty-commit anti-pattern this verb exists to avoid.** If `hive-forge
ci-log --run <n> --job 0` shows the code green (verdict is the
last line) but the status is stuck, ask the operator to select "rerun" in
the forge web UI — it's CSRF-gated, so only they can do it.
<!-- vale write-good.Passive = YES -->
### Running checks locally
Don't run `nix flake check` directly — it dispatches to the shared build farm and
wastes a remote-builder slot. Use the devshell equivalents instead:
```sh
nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix fmt # authoritative; the formatter set is nix/treefmt.nix
sh scripts/check-issue-refs.sh # tracker-tag lint
sh scripts/check-comment-blocks.sh # comment-block lint
```
`scripts/pre-push` provides a git pre-push hook that automates the two
lint checks. Install it once per clone:
```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push
```
After that, any `git push` automatically runs both lints and aborts with a
diagnostic if either fails — catching the issue locally before CI sees it.
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
slow); run those manually before pushing Rust changes.
### VM tests
Tests that boot NixOS VMs are the `vmTests.<system>.<name>` flake output
(`nix/vm-tests.nix`), not `checks`, so the per-PR **nix flake check** job never
builds them. [`.forgejo/workflows/vm-nightly.yml`](../../../.forgejo/workflows/vm-nightly.yml)
builds all of them nightly at 02:30; a failing test fails that run. To run one
by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
workflow, optional `test` input), or run
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
must not handshake).
## Configuration reference
`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
(see _For operators_ above). The runner reaches the forge at
`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
gateway's dnsmasq answers it locally where the forge runs, and forwards it to
the operator's DNS everywhere else.
Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
(workflow targeting; the default `hive-ci:host` is what this repo's workflows
name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
can't hold the runner's single slot indefinitely).
The old `services.hyperhive.deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout}`
and `services.hyperhive.swarm.forge.ci.{enable,name,concurrency,labels,package}`
are still accepted: each forwards to the same-named option under
`deploy.ci-runner`, and evaluation warns naming both paths. Set the new name to
silence the warning. The two `name` defaults differ (`hive-ci` before,
`swarm-ci-runner` now), so a host that never set the old `name` shows up under
the new default.
The forwarded `enable` carries the new module's requirements, not the old
module's: evaluation refuses a host that sets only the old `enable` unless
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
`ciRunnerClientKeyFile` are both set. They default to the identity
`glue-bao-tls.nix` mints when `services.hyperhive.deploy.bao.enable` is true;
elsewhere place them as described under _For operators_. The runner no longer
needs `deploy.forgejo.enable` or a hive on its host.
## Container design
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/5-project/process/gotchas.md`.
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.
## Registration
1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
the forge's instance-wide token from
`GET /api/v1/admin/runners/registration-token` at start and every five
minutes, and writes it to the secret store at
`swarm/services/ci-runner/registration-token` when it differs from the
value it last wrote. Its store grant on that path is write-only.
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
and nothing else, retrying every 30 seconds until the token is there. It
writes the token to a root-only file in `/run/swarm-ci-runner` on the host,
which the container sees read-only, and restarts the runner when the value
changes. `/run` is tmpfs, so no copy reaches persistent disk; after a reboot
the unit fetches the token again, and until the store answers only a
runner that has no `.runner` yet waits. It runs at boot, on deploy, and every
ten minutes on a timer.
3. **`swarm-ci-runner-register.service`** in the container is the only unit
that loads that file, with `LoadCredential=`. systemd hands a unit's
credentials to all of its processes, and for a `host` label act_runner
runs every job step as a process of the daemon with the daemon's whole
environment, so the daemon unit carries the token in neither. The
register unit is a oneshot under the daemon's dynamic user and state
directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs
`act_runner register` only when `.runner` is missing, so the `.runner` it
writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`)
starts it in a pre-start step and runs the daemon once it succeeds.
A root step of the register unit, ahead of registering, removes `.runner`
in three cases:
- The forge has no record of the runner. Forgejo answers every call from an
unknown runner with `unregistered runner`, and the daemon exits 1 when its
startup `Declare` gets that answer. On the next start the step finds the
message in the daemon's journal since its last start, and registers again.
A daemon that's already running when an admin deletes its runner keeps polling
and logging the error without exiting; the next restart of the unit or
container registers it again.
- `.runner` names another forge address than the configured one.
- `.runner` names other labels than the configured ones, compared in the
form the daemon writes back: `<name>:<scheme>`, with `host` for a
label that names no scheme.
A changed token alone doesn't register again: the daemon authenticates with
the per-runner token in `.runner`. Without `.runner` and with no token yet
the register unit fails and the runner unit's pre-start step with it;
`Restart=on-failure` on the runner unit starts both again every 15 seconds,
with no start limit.
Host jobs run as the runner's own user, so they can still read `.runner`
and act as this runner; see the security section below.
A new registration is a new runner on the forge: the previous entry stays in
the forge admin panel as offline until an admin deletes it. The runner the old
`hive-ci` module registered is one such entry, since this runner registers
under its own name and token. Delete it under Site administration, Actions,
Runners.
## Actions mirrors
<!-- vale write-good.Passive = NO -->
With `services.hyperhive.deploy.forgejo.actionsFromSelf` enabled, the swarm
controller seeds `actions/checkout` and `actions/upload-artifact` pull-mirrors
(from `github.com/actions/*`) on the forge and Forgejo's `DEFAULT_ACTIONS_URL`
points at the forge itself. CI `uses: actions/<name>@vN` steps then resolve on
the forge, with no external DNS on the CI critical path. A workflow that names
any other bare `actions/*` action fails at job setup with "repository not
found" until that action joins the list in `nix/host-modules/hive-forge/default.nix`.
A forge host that runs the swarm controller therefore resolves those actions
from itself, with or without a runner on that host.
Set `actionsFromSelf = false` there to resolve actions from `data.forgejo.org`
and seed no mirror.
The option defaults to whether the forge's host also runs the swarm controller
(`services.hyperhive.deploy.swarm-controller.enable`; set by `glue-forge-controller.nix`): on when it does, off
otherwise. Setting it `true` on a forge host without the controller leaves
the actions unresolved, because nothing seeds the mirrors; evaluation
warns about it.
<!-- vale write-good.Passive = YES -->
The **swarm controller** seeds the mirror in its swarm-wide forge-objects
pass (`swarm-controller/src/forge/objects.rs`), beside the orgs it ensures.
`glue-forge-controller.nix` forwards the effective mirror list as
`SWARM_CONTROLLER_FORGE_MIRRORS` in the `swarm-controller` service
environment (JSON-encoded `[{upstream, dest}]` list). It reads the list of
the host the controller runs on: mirrors declared on any other host aren't
seeded, and evaluation warns about them.
**General-purpose mirrors**: you can pre-seed any external repo as a
pull-mirror via `services.hyperhive.deploy.forgejo.mirrors`:
```nix
services.hyperhive.deploy.forgejo.mirrors = [
{ upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
{ upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; }
];
```
The controller creates each entry as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. The controller
autocreates the `<owner>` org in `dest`. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions.
## Security: unsandboxed builds and trusted contributors
**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
### What unsandboxed builds mean
nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
A malicious `default.nix` or build script in a PR can therefore:
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
Note: `nix flake check --no-build` (eval-only) reduces the exposure but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The CI workflow's `nix flake check` job builds every check derivation, which is the higher-risk path.
### Mitigation
For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.
For repos with external contributors or fork PRs:
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.
## Host store maintenance (recommended)
The CI runner builds derivations through the **host** nix-daemon — the
`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
outputs accumulate in `/nix/store` with no automatic collection, and a busy
CI day can fill the disk until every job fails fast with `ENOSPC`.
Store GC is a **host-level** concern, so it belongs in the host's own NixOS
configuration, not in the hyperhive service modules — a single service should
not reach out and change the host's global nix-daemon options. Add the
following to your host config:
```nix
{
# Daily GC: delete store paths not referenced by a live root and older
# than a day. Keeps the store bounded between builds.
#
# `randomizedDelaySec` because a fixed daily time lands on the same wall
# clock as whatever else runs nightly. A collection that overlaps a CI
# job doesn't fail it — the runner builds through the host nix-daemon,
# so an in-flight build's temp roots are protected — but the first build
# after a collection rebuilds what the collection deleted, so a run that
# lands just behind one pays for it in wall time.
nix.gc = {
automatic = true;
dates = "daily";
randomizedDelaySec = "45min";
options = "--delete-older-than 1d";
};
# Disk-pressure GC: when free space drops below min-free mid-build, the
# daemon collects garbage up to max-free before continuing. This is the
# real-time net the daily timer can't provide — a same-day build burst is
# what fills the disk. Tune to your disk size.
nix.settings.min-free = 20 * 1024 * 1024 * 1024; # 20 GiB
nix.settings.max-free = 50 * 1024 * 1024 * 1024; # 50 GiB
}
```
**Measure a collection on your own hosts before relying on that stagger.**
`randomizedDelaySec` spreads the _start_, not the run, and the sweep over
unused `.links` entries dominates the run — which scales with the size of the
store, not with what this collection deletes. On a large store it can take most
of an hour, so two hosts both set to `daily` may overlap however much jitter you
give them. `journalctl -u nix-gc.service` reports each run's wall clock.
**Remote builders:** if CI dispatches builds to a remote builder (for example via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so apply the same GC config wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder.
⚠️ Two stores also means the temp-root protection above **doesn't span
them**. Each daemon holds roots only for the builds it runs, so a
collection on the _builder_ knows nothing about a coordinator build
waiting on a path there. Stagger the two hosts' GC windows rather than
giving them the same `dates`, and prefer `min-free`/`max-free` on the
builder — disk-pressure collection at least runs while nothing is mid-copy.
## References
- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
- `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers.

535
docs/4-implementation/scheduler/coordinator.md Normal file
View file

@ -0,0 +1,535 @@
# hive-c0re coordinator internals
Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
For the public API surface (dashboard, socket protocol) see `docs/5-project/process/conventions.md`
and `docs/2-handbook/agent-lifecycle/persistence.md`.
---
## Job queue
**For the job queue as a general idea — graph of steps, shared resource
slots, no hive-c0re specifics — see [`jobq.md`](jobq.md) instead.** This
section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics).
hive-c0re submits every container operation (rebuild, first-spawn, power
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of basic nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers.
Exceptions like graceful stop, deferred starts, and the boot sweep
need no bespoke code paths — each takes the shape of a DAG built
from the same basic nodes.
### Two levels: DAG and node
The **DAG** is the unit of cancel and the
dashboard group; the **node** is the unit of scheduling / execution /
build-log. Deps are intra-DAG edges only (`AfterOk` by default:
the dep must succeed, a failed/cancelled dep cancels the dependent —
cancel-downstream). Cross-DAG ordering comes from the per-agent lease,
never from edges between DAGs.
A DAG is **declared, not described**: a template builds it through
`hive_jobq::JobBuilder`, naming each node it depends on via the handle
`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
only exists for a node already declared, so every edge points backwards and
nothing can write a cycle down — there is no submit-time validation pass,
because there is no malformed spec to reject.
### Node inventory (primitives)
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied |
Cheap — no build slot:
<!-- vale write-good.Passive = NO -->
| Node | Behavior |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`), then reads the agent's closure record once and pins its store path for `Prebuild` and `Swap`; fails when the swarm has no closure to deploy; reads nothing from the forge. Declares no resources |
| `Provision` | first-spawn pre-create provisioning — the state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev markers, forge/matrix sync, manager kick, rescan; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state dir. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. The destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `EmitRebuilt` | tail of a rebuild — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `KnowledgePull` | `/knowledge` pull (`knowledge::pull`): at boot (commits that landed while `hive-c0re` was down), on the swarm knowledge-changed event, and hourly as a fallback. Holds `Resource::KnowledgeTree` (capacity 1), so two pulls never overlap on the working tree; each trigger queues its own pass, which waits for the resource if one is already live. Agentless |
| `WantedPull` | pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. Runs at boot, waiting up to a minute in total for the queue connection, and again after every attach of the wanted-state watch (which retries with a backoff when the attach fails or the stream ends); the deploy event (`swarm_status`) and the watch are the fast path, this repairs a missed one. A first deploy already queued or running for an agent isn't queued again. Agentless |
<!-- vale write-good.Passive = YES -->
Nothing in the queue commits to a meta repo: the hive has none to commit to,
and `agent_set::refresh` serializes itself with its own mutex.
### Every operation as a DAG
The `stop` / `start` power ops write the durable `wanted` intent via a head
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
deliberately stopped agent isn't forced back up by a hive-wide restart. The
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs.
**`job_queue/power.rs` builds these dynamically from each agent's live
running state** (an async `lifecycle::is_running` read) — that's why they
live there rather than in the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
closes the TOCTOU window: if an agent flips state between the `is_running`
read and node exec, the tail `Reconcile` still converges it in-DAG (with
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
sweep. `start` folds the per-agent declared-closure upgrade in (a _down + stale_
agent's subgraph is a rebuild-then-start).
```text
rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) ∥ [Signal(a)→Drain(a) if graceful]; both →(after-ok) StopForUpdate(a) → Swap(a) →(after-ok) RebuildBookkeeping(a) } →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
boot: «graceful rebuild subgraph» per stale agent (closure marker ≠ the record's store path);
plus Reconcile(a) for every drifted agent (all ONE DAG)
```
Notable collapses:
- **`rebuild` is one uniform shape** — no `was_running` branch.
`StopForUpdate` noops when already down; the tail `Reconcile` autonoops the
start when `wanted = Offline` (a rebuild of a deliberately stopped agent
leaves it stopped).
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
terminal ok _or_ fail, bringing a wanted-up agent back on its old config.
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
whole-hive graceful stop fires every agent's signal immediately and all
drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The boot sweep is one DAG**: it declares one graceful `rebuild` subgraph
per stale agent, each an independent root, beside a `Reconcile` per drifted
agent. Not child DAGs — one DAG, no `parent_id`.
### Desired-state (spec vs status)
Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
`container_view` remains the observed _status_; `Reconcile` nodes converge the
two. Setting `wanted` is never a queued node: the power layer
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
Every operator power surface —
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts
left are the gateway's (no lease, no
harness). Cancelling a still-queued power DAG reverts `wanted` to the
observed state — a cancel means "don't do it," not "do it later." hive-c0re
seeds agents without a row from observed state on first touch (running ⇒
`Up`); destroy removes the row.
The admin-socket responses carry the submitted DAG ids; `hivectl` polls
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
glyph, template, agent, node chain — so
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
non-zero). Nodes appended in-DAG (a `Reconcile` fanning its `Start`/`Stop`)
join the same DAG, so they surface under that DAG's id in the same loop — no
separate child DAGs.
### Scheduler semantics
<!-- vale write-good.Passive = NO -->
A node is **ready** when it's `Queued`, every dep is satisfied, and its
resources are free. Resources:
<!-- vale write-good.Passive = YES -->
1. **Build slots** — `services.hyperhive.deploy.hive-controller.buildSlots` permits (default 1),
held by nix-heavy nodes for the node's duration.
2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is
per-node; a DAG can span agents) and globally exclusive per agent across
all DAGs: acquired either at a container-affecting node (`SetWanted`,
`Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`) on
behalf of a whole coordinated subtree; held by the owning
DAG until it's terminal, so two DAGs never interleave container ops on the
same agent. A DAG touching multiple agents holds one lease per agent.
(`SetWanted` is a store write, not a container op, but takes the lease anyway
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
`Prebuild`, `Provision` — they
touch the store, not the running container, which is exactly why a
stop can land while another DAG's prebuild is still downloading. Also exempt, for a different
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
container, but their `AgentWindow` brace holds the lease above them — see
_Braces_ below.
#### Braces
Templates otherwise declare a resource on **every** node that needs it, even
when a parent already holds it, so the requirement belongs to the node rather
than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
exception: a pure-resource-holder root that declares on behalf of a subtree
coordinated with itself, whose members then declare nothing.
It's forced rather than stylistic. Declaring a resource means _"I need this
exclusively"_, and the agent lease is single-unit — so **two siblings that both
declared it could never run concurrently.** For a subtree whose whole point is
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
the requirement truthfully on every node and running those nodes in parallel are
mutually exclusive. One holder above them speaks for the subtree.
This is the opposite of the failure the declare-your-own rule exists to prevent,
not a relapse into it: there the requirement was _implicit_, inferred from a
node's kind and true only by accident of placement. Here it's explicit, on one
node, with the omission below it documented on the brace itself.
Two consequences worth knowing:
<!-- vale write-good.Passive = NO -->
- **Flattening a chain under a brace is safe.** A brace (`AgentWindow`)
holds the lease continuity for the stop chain's subtree: as independent
siblings, `Signal`, `Drain`, and `StopForUpdate` would each acquire the
lease separately and leave a gap another DAG could claim the agent in,
mid-bounce.
- **Observability is unaffected.** `running_transients` keys off a node's
_payload_ agent, not off a declared lease edge, so every child still lights its
own dashboard pill and still reports its own `takes_container_down` to the
crash watcher. A brace itself reports `false`: it parents the stopping nodes
but doesn't stop anything, and claiming otherwise would widen crash
suppression across the build and tail, where a vanished container is still a
real crash.
<!-- vale write-good.Passive = YES -->
Among simultaneously ready nodes competing for a resource, DAG-submit order
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
created on lease acquisition and dropped when the DAG settles terminal.
The queue is in-memory only and lost on hive-c0re restart — deliberate:
desired state is re-derived at boot from the DB + rev markers (see _Boot
reconcile_), so there is no durable-recovery machinery to go wrong.
### Cancel, history
Cancel only applies to DAGs that are still fully queued (an in-flight nix build isn't
interruptible).
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
DAGs — a flat cap over the whole sorted list, not per template, since
the dashboard renders one recent-builds list and one number bounds it.
### Wire shape
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
payload.** It says _the queue changed_; a client that wants to know how
re-fetches `GET /api/jobq/graph`.
That endpoint serves the graph generically (`hive-jobq-wire`): every node
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
string — the kind _is_ the phase label, there is no separate sub-step
string) and free-form `data` for what only some kinds have (`agent`,
`inputs`, `build_log_id`). Group roots ride as ordinary
nodes, so a group's state is just the root's own `state`.
No group-level `agent` exists — agent is per-node, so one group can
span agents; consumers derive a group's agents from its nodes. Build logs
are likewise **per-node**: the dashboard renders the node tree and keys the
live-log panel off the running node.
The event carries no payload by design: shipping a typed projection of
the whole queue in the event itself would be a second rendering of the
same graph that someone has to keep in agreement by hand with the
endpoint every consumer actually reads. Telling a client _when_ to refetch is
the event's whole job.
---
## Container view
`container_view.rs` maintains an in-memory snapshot of every nixos-container's
systemd service state. It's polled on coordinator startup and re-scanned after
every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
the actual container status without a live `nixos-container list` call on each
render.
---
## Boot reconcile
On startup, `auto_update::run` classifies every agent by closure freshness (the
per-agent `.{name}.closure` marker under `/var/lib/hyperhive/applied/` vs the
store path the agent's `agent-closures` record declares) and persisted
`wanted` intent, then:
1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
a hive that missed the swarm's deploy request catches up. Stale but wanted-offline
agents get no boot-time nix work — their rebuild happens on their next start
(the start submit path upgrades a stale start to rebuild+start). Each
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
running-stale and stopped-but-wanted-up cases. An agent whose declaration
the hive can't read or compare (no queue connection, no record, a closure the
builder didn't push, a running closure it can't read) rebuilds nothing. An
agent with no marker takes the closure its container profile points at as the
marker, so a first boot after the marker arrived rebuilds exactly the
agents that run something else than declared, each holding a build slot.
2. **Power path** — every agent whose observed state drifted from `wanted`
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
Booting with every agent on its declared closure performs no rebuild — only
reconciles. After the boot pass, `auto_update::run` repeats the rebuild half
every five minutes. It skips an agent with a rebuild waiting or running
(`JobQueue::rebuild_live`) and one whose declared store path this process
already attempted, so a closure that fails to deploy gets one try per process.
The boot log line records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. `auto_update::run`
seeds agents without an `agent_power` row from observed state during
classification (the one-time migration; thereafter the DB is authoritative).
## Meta flake
Agents build from the swarm's `internal/meta` flake, which swarm-controller
commits and the swarm builder builds from (see
[Config changes](../../2-handbook/agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
The hive keeps no flake of its own and deploys the closure the swarm builds
from it, as the agent's `agent-closures` record declares (`attic/closure.rs`).
`agent_set.rs` keeps the
host-side files that follow the agent set (`topology.json`, the gateway's
agent-socket map and nginx include) in line with the live agents, and
`meta.rs` removes the permission seed files once the swarm has them, and
`migrate.rs` deletes the legacy `meta/` repo's leftover `.git`.
---
## Container lifecycle (`lifecycle.rs`)
Every container operation ultimately calls into `lifecycle.rs`. Two paths exist:
**rebuild** (existing container) and **spawn** (first-time creation).
### Rebuild path (existing container)
Goal: apply the new system profile and any `EXTRA_NSPAWN_FLAGS` / drop-in changes
in a single start, with minimum downtime.
`nixos-container update` only runs `systemctl reload container@<c>` when the
container is already up (per `isContainerRunning` in `nixos-container.pl`). Stopping
first turns `update` into a boot-style operation: it builds + `nix-env --set`s the
new profile and skips the in-container `switch-to-configuration`. The subsequent
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node):
1. `MetaSync` — rebuild-dir prep, the archetype refresh, the agent-set
refresh, and the pin of the swarm's declared closure (one read of the record;
`Prebuild` and `Swap` use the pin). Short, and it declares no
resources.
2. `Prebuild` — download the swarm's closure for the agent **before** stopping.
The container keeps serving the previous generation while the download
happens out-of-band. A missing or unpullable closure fails the DAG here,
before `Swap` touches the running container, and the container keeps
running. Skipped when the container is already down — a stopped agent has
no uptime to preserve, so `Swap` downloads it.
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — profile-swap onto the swarm's closure (near-instant after the
prebuild), then it writes the closure marker.
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot.
### Cold-start fallback
`start` after `update` can exit non-zero when packages are **removed** between
generations: the old-generation activation script references units that no longer
exist in the new closure, causing systemd to exit non-zero. The container may be
half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
`lifecycle::start_with_fallback` preserves and surfaces both errors if the
cold-start also fails. Every `Reconcile` node's start action uses this
fallback.
### Spawn path (new container)
For a first-time `create`, `nixos-container create` is atomic: it leaves no container
record to clean up if it fails. The `Create` node downloads the swarm's closure first,
so a spawn with no closure to deploy fails before it creates the container. Sequence:
download the closure → `create` from it → write nspawn flags → `systemctl daemon-reload` → `start`.
---
## Host-level resource + performance options
A handful of `services.hyperhive.deploy.hive-controller.*` options tune container resource
limits, build parallelism, and first-spawn latency.
### Build slots
`buildSlots` (default `1`) sets how many closure-fetching job-queue nodes
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
concurrently. The default serializes all of them; raise it on hosts with
the bandwidth and disk to take more agents at once. Per-agent correctness is independent of the count —
each agent's container-affecting ops serialize on its lifecycle lease
regardless.
### Container resource limits
`agentCpuQuota` and `agentMemoryMax` map directly to systemd
`CPUQuota=` and `MemoryMax=`. hive-c0re writes a
`container@h-<name>.service.d/` drop-in file on each spawn and
rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild.
The answer per agent and per field is, first match wins:
1. the swarm's cap for the agent (see below);
2. `agentCpuQuota` / `agentMemoryMax`.
A swarm cap replaces the hive's answer, so a hive can't raise an agent
above it. Like the rest of the drop-in, a changed swarm cap reaches a
running agent at its next spawn or rebuild. A hive has no way to set a cap
for one agent: you set caps per agent in the swarm UI.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the swarm caps file,
the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
`hive-subagent-daemon` (two thirds). Neither is in the agent's build, so a
changed cap reaches them at the next container start without a rebuild.
`infinity` sets neither. A missing or malformed file sets neither as well, and
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
When the container reaches its cap, the kernel kills in this order:
1. a subagent run (`oom_score_adj` +1000);
2. a bash task (0);
3. the harness and the agent's own `claude` (-500).
Each subagent run and each bash task has its own cgroup with
`memory.oom.group=1`, so the kernel kills all of a run or task and nothing
outside it. Inside the container these are
`/sys/fs/cgroup/system.slice/hive-subagent-daemon.service/runs/<name>-<n>/`
and `/sys/fs/cgroup/system.slice/hive-bash-daemon.service/tasks/<id>/`; the
daemon itself sits in `daemon/` beside them. A daemon that can't set these
up logs one warning at start and runs everything in its own cgroup; subagent
runs still get +1000.
<details><summary>Implementation: swarm caps</summary>
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
agent and no hive in it, so an agent that moves to another hive keeps its
caps. swarm-controller alone writes it:
| method | path | body |
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
A stored value the controller can't decode, such as one a newer build wrote,
answers `{}`, with a warning in its log, until the next `PUT` replaces it.
`PUT` refuses an agent no hive's wanted state declares, unknown fields, and
any value systemd would ignore, which would leave the agent uncapped. A
percentage has at most two decimals, no leading zero in its whole part (it
refuses `08%`), and is above zero, up to `100%` for
`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an
optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at
least one byte and below 2^64 - 1 bytes. The controller stores each value as
given.
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the validators first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a hive
warning (the `hyperhive.hive.warning` metric and the `hh-hive-warning` alert).
A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
hive's own agents would need a roster in the auth responder.
</details>
The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle
an agent even on an idle host, while the weights are cgroup
v2 **relative shares** that only decide who yields _under contention_.
A low-weight container still gets the whole machine when nothing else
wants it.
| Option | Default | Description |
| ---------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `services.hyperhive.deploy.hive-controller.agentCpuQuota` | `"200%"` | CPU cap per agent, as a percentage of one core (`"200%"` = 2 cores). Raise if agents hit CPU limits during builds or heavy tool use. |
| `services.hyperhive.deploy.hive-controller.agentMemoryMax` | `"4G"` | Memory cap per agent. Raise for agents that run large nix builds or hold big in-memory data. |
| `services.hyperhive.deploy.hive-controller.agentCpuWeight` | `80` | `cpu.weight` share per agent, `1`–`10000` or `null` to omit the setting. Kernel default is `100`, so `80` makes agents yield. |
| `services.hyperhive.deploy.hive-controller.agentIoWeight` | `80` | `io.weight` share per agent, same range and `null` handling. See the caveat below — it's a no-op on many hosts. |
Two things to know about the weights:
- They're **hive-wide** — unlike the caps there is no per-agent
setting, so every agent carries the same value and the weight does _not_
rank agents against each other.
What `80` buys is that agents yield to everything **not** on this
drop-in path: host services (nginx and dnsmasq among them) and the
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
stay at the kernel default of `100`.
- `IOWeight=` is only honoured when the backing device runs the BFQ
scheduler or has blk-iocost QoS enabled. On a host using
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
and the kernel ignores it — harmless, but it will measure as nothing.
Check with `cat /sys/fs/cgroup/io.cost.qos`, and set the option to
`null` if you would rather not write a setting nothing reads.
For a hive-wide cap across all containers together, set
`systemd.slices.machine.serviceConfig.CPUQuota` in your NixOS
config — all nspawn containers live in `machine.slice`.
---
## See also
- `docs/2-handbook/agent-lifecycle/config-changes.md` — config PRs, meta flake, system messages
- `docs/2-handbook/agent-lifecycle/persistence.md` — SQLite schema, state-dir layout
- `docs/5-project/process/conventions.md` — wire protocol, recipient sentinels
- `docs/2-handbook/agent-lifecycle/agent-roster.md` — the agent roster and privilege

61
docs/4-implementation/scheduler/jobq.md Normal file
View file

@ -0,0 +1,61 @@
# The job queue, for operators
Long-running work runs through a job graph. The swarm controller keeps one
for swarm-level work — creating an agent's identity, forge user and config
repo. Each hive's hive-c0re keeps its own for container operations —
rebuild, first-spawn, power changes. This page explains
what the job queue _is_, as a general idea, independent of what either uses
it for. For the hive-c0re step catalogue and the engineering internals
(scheduler, leases, resource windows) see [`coordinator.md`](coordinator.md)
instead.
## What the job queue is, in the abstract
"jobq" is a generic engine for running many interdependent jobs under
limited concurrency — it has no idea what a "container" or a "rebuild" is.
Two ideas are all there is to it:
- **A job is a small graph of steps**, not one opaque blob. Steps can
depend on each other (this step only starts once that one finishes), so
a big operation is really a short, ordered sequence — not a single
black box that's either "done" or "not done."
- **A step can need a shared resource**, which only so many steps can hold
at once (a "slot"). If every currently running step already holds the
slots it needs, a new step that wants the same one waits its turn —
that's the whole reason things queue instead of all firing at once.
The engine's whole job is: whenever a step's ordering and resource needs
are both satisfied, run it. It has no opinion on what the steps _do_ —
that's supplied by whoever builds the graph. The engine is generic: the
swarm controller and hive-c0re each build their own graph on it.
## Watching it happen
Two views, one per graph, drawn by the same component: the swarm UI's
`/jobs` page shows the swarm controller's graph, and the hive dashboard's
**R3BU1LD QU3U3** tab (see
[`web-ui/dashboard.md`](../../2-handbook/web-ui/dashboard.md)) shows that hive's. Each
**row** is one job; the rows nested under it are that job's steps, in
order (occasionally a couple run side by side). A step shows one of:
<!-- vale write-good.Passive = NO -->
| Glyph | Meaning |
| ----- | ------------------------------------------------------- |
| `⏸` | queued, waiting its turn |
| `▶` | running |
| `◐` | its own work is done, waiting on a step nested under it |
| `✔` | finished successfully |
| `✖` | failed |
| `⊘` | cancelled |
| `·` | skipped (not needed for this run) |
<!-- vale write-good.Passive = YES -->
A step that isn't needed for a given run stays in the graph as `·` rather
than being absent from it, so the same kind of operation keeps a
recognizable shape run to run, whichever steps it actually needed.
⚠️ **The view hides `✔` and `·` by default**, so that full shape isn't what
you see first — tick them back on in the state filter. The selection is
part of the request, not a display toggle: hidden states are never fetched.

756
docs/4-implementation/scheduler/observability.md Normal file
View file

@ -0,0 +1,756 @@
# Observability (OpenTelemetry)
The swarm has one telemetry pipeline. The swarm's collector writes the
swarm's metrics and log stores and, if you name one, exports upstream; each
hive runs a collector of its own that forwards its agents' telemetry there.
What flows through it: each agent's Claude Code statistics (token usage,
cost, tool call counts) via Claude Code's built-in OpenTelemetry
integration, each agent container's journal, the hive tier's own
`hive-c0re.service` journal line for a detected container crash (see
[Container-crash alerting](#container-crash-alerting-hive-c0re)), and the
hyperhive metrics catalogued below.
## The two collectors
Telemetry crosses two collectors, and which one you configure depends on what
the host is:
| | runs where | receives from | does |
| ------------------------------------ | -------------- | --------------------------------- | ----------------------------------------------- |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
| **hive tier** — always on | every hive | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
An all-local host runs both; the hop between them configures itself. A host that
runs the swarm's services without a hive runs only the swarm tier: the hive tier
labels a hive's telemetry outside its agents' control, and that host has no hive.
The swarm services there push to the swarm collector directly, each as its own
OAuth2 client.
```nix
services.hyperhive.deploy.swarm-otel.upstream = {
endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
};
```
## Enabling export
Every hive runs its collector and every agent exports to it, with nothing to
switch on: there is no per-hive or per-agent opt-in.
`upstream.endpoint` is where telemetry ends up after it leaves the swarm — optional,
because the swarm's own metrics store (`deploy.victoriametrics`) is a
destination in its own right. With both, telemetry goes to both. See
[`swarm/services.md`](../../2-handbook/swarm/services.md#metrics-victoriametrics--grafana).
**Telemetry leaves a hive exactly one way: through the collector that
runs on the host.** Agents never talk to `upstream.endpoint` themselves —
they export unauthenticated to a collector address only their own containers
can reach. They dial it as
`http://hive-collector.hive:4318`, a name identical on
every hive and every swarm, which each hive's own dnsmasq answers with its bridge IP, so an
agent's build carries no hive-specific address and no swarm domain. The otel module opens that
port on the bridge itself. That collector forwards to the swarm's
([`swarm/services.md`](../../2-handbook/swarm/services.md#telemetry-collector-otel)), the
single process holding the upstream credential and the only writer to the
swarm's stores. No agent holds a copy, and neither does the hive.
The hive collector reaches the swarm collector by its gateway name
(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
shape every hive-to-swarm-service hop uses. On the host running the swarm
collector the hive's dnsmasq answers that name; elsewhere it resolves through
ordinary DNS.
⚠️ **The hive collector carries every bit of that hive's telemetry.** It
runs on the same host as the agents and restarts on failure. Telemetry isn't
the control plane, so degraded telemetry isn't degraded operation.
Both collector tiers queue outgoing batches on disk (`file_storage`) and retry
a failing batch until the receiving end accepts it, so a restart or a short
outage of the next hop delays telemetry instead of dropping it. The queue holds
at most 5000 batches. Two limits remain:
- A full queue makes the exporter refuse new batches and log the drop. The
journald receiver has already moved its cursor past a refused batch, so an
outage longer than the queue can hold still loses data, later and a bounded
amount.
- Retry has no time limit, so the exporter keeps resending a batch that the
store rejects with a retryable error, and enough of those fill the queue.
### Why two tiers
**Agents export only to the hive tier.** Exporting straight to `upstream.endpoint` would mean
every agent needs the credential — and the only place to hand it to an agent
container is somewhere the agent itself can read, its own claude settings
among them. `0600` protects a secret from other containers, not from the
agent it belongs to.
**The tiers stay separate on one box.** An all-local hive is a statement about
_where_ processes run, not about the shape of the deployment. A boundary that
disappears locally is one the local deployment stops testing.
### What the agent→collector hop is and isn't
**It has no application-level auth.** The receiver takes any OTLP that reaches
it; what bounds who can reach it's the firewall — `exposeHostPorts` opens the
port on the bridge interface only — "unauthenticated to a bridge address"
means _reachable from an agent container_, not _presents a credential_.
The consequence, stated because it's a choice rather than an oversight: **any
agent can push arbitrary OTLP, and it's forwarded on under the operator's
credential.** Neither tier can tell a container's genuine Claude Code stats
from anything else shaped like OTLP arriving on that port — including data
smuggled out in resource attributes on an otherwise-legitimate export.
That's a **different risk from the one the collector fixes**: an agent that
held the upstream credential could do everything above _and_ use the token
anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust
boundary (`docs/2-handbook/trust-boundary/security.md`: capability = accepted risk), so an agent being
able to _send_ is an accepted extension of that boundary — but it's not
closed by this design, and don't read anything here as closing it.
**The `agent` label is self-reported, and the hive→swarm authentication below
doesn't change that.** Treat it as a convenience for grouping dashboards, never
as evidence of which container produced a sample: any agent that can reach this
hive's collector can label its data as any other agent.
<!-- vale write-good.Passive = NO -->
**Logs ride the same hop, and a journal carries more than a counter does.** Each
agent container forwards its own journal through this port — every unit in it at
`info` and above, not an allowlist. That's the harness, the MCP daemons and
whatever a tool call spawned, so command lines and error text leave the
container, not just counts. Destination, credential and trust boundary are the
same as for any OTLP the agent sends.
<!-- vale write-good.Passive = YES -->
Worth spelling out, because two different hops are in play and only one of them
carries a credential:
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
downstream can tell one agent's export from another's.
- **hive→swarm** authenticates. The swarm tier gives each hive its own receiver
and stamps `hive=` from whichever one accepted the sample, so _that_ label is
unforgeable — see [Authenticated ingest](#authenticated-ingest) below.
A verified `hive` is reachable and a verified `agent` isn't — and that falls
out of the topology rather than being a gap someone forgot to close. The swarm
runs one collector, and the mechanism gives it no finer grain: a bearer-token
check never reveals _which_ token matched, and a receiver reads request metadata
rather than the claims it authenticated with.
If you need per-agent numbers you can act on, take them from the agent's own
turn-stats rather than from a metric label.
### Authenticated ingest
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
from whichever receiver accepted a sample. A hive therefore can't report
metrics as another hive, and can't relabel its own by editing what it sends —
the label isn't taken from the payload at all.
**Nothing to set where the hive holds its store identity.** The hive's collector
reads its secret from the swarm's secret store under the hive's own leaf
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client`, which authelia's mint, in its container,
writes, into `/run/hive-otel-oidc/` (tmpfs, so never on persistent disk). It
runs at boot and retries every 30s until the store answers; until then the
collector runs on an empty file and the swarm refuses its exports. When a
run lands a changed value it restarts the collector. A timer runs it again
every ten minutes, so a secret rotated later reaches the collector within
about ten minutes. On a hive that
runs the store the leaf is already there; elsewhere it's the credential you
place — see [secret placement](../../2-handbook/swarm/credentials.md).
To deliver the secret some other way, name the file. The fetch unit isn't
rendered, and the collector waits up to 120s for the file to appear:
```nix
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
```
Setting the option to `/var/lib/hive-otel-oidc/hive-<name>.secret`, the path
the fetch used to write, now counts as delivering the file yourself: nothing
refreshes it, and the build warns. Remove the setting to fetch from the store.
**No unauthenticated mode exists.** A hive always presents an identity, so a
missing credential is a build error rather than a quieter fallback — the
collector has no anonymous route to accept samples on, and every path it serves
belongs to exactly one hive.
Getting the secret wrong shows up as the hive's collector logging 401s from the
swarm tier and no metrics appearing for that hive.
⚠️ **`upstream.endpoint` must be valid for `upstream.protocol`.** The upstream exporter follows
`upstream.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an _address_: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
address"_. nix validates the collector's config at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.
## Network access
Agent containers reach the host only on ports 80 and 443 (plus DNS). The
collector port needs nothing from you: the hive collector opens it on the bridge.
To let agents reach some other host-local service of your own →
[`networking/network.md`](../networking/network.md#reaching-host-services-exposehostports).
## Options reference
The nix modules (`nix/host-modules/otel.nix`, and `nix/host-modules/swarm-otel.nix`
for the `services.hyperhive.deploy.swarm-otel.upstream.*` export) generate every
`services.hyperhive.otel.*` option's full type, default, description and
example into the [options reference](https://hyperhive.darkest.space/options/)
(host options — `nix build .#docs-host` for a local render). The same module
fixes the collector port every hive and its agents share,
`services.hyperhive.swarm.hiveOtel.collectorPort`, and each hive's own
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
That's the reference; this page covers what a per-option listing can't: the
two-tier architecture, the security model, and how the options interact.
## Built-in resource labels
The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container
wide**, for every unit and in `/etc/profile`, so every
process in an agent container exports to the hive's collector without any
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
yourself from a tool call or a shell you open in the container.
Every agent's export therefore includes these resource attributes
automatically:
| Attribute | Value |
| -------------- | ------------------------------------------------------------ |
| `service.name` | `hyperhive-agent` (constant) |
| `agent` | agent logical name (for example `iris`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
The swarm's collector adds `hive`, the name of the hive whose receiver accepted
the push (see [Authenticated ingest](#authenticated-ingest)). The container
doesn't set it.
Append additional labels for one hive via `extraResourceAttributes` (see
option reference above). Agents get them at container start, so a change
reaches an agent at its next start, not through a rebuild. Pass custom
per-data-point labels with `hive-metric --labels` (see below).
## Log severity
Every journald receiver in the pipeline maps the journal's `PRIORITY` onto an
OpenTelemetry severity, so a line arrives in the log store already carrying its
level and `severity_text:ERROR` is a query you can write:
| `PRIORITY` | syslog | severity |
| ---------- | ------------------ | -------- |
| 0, 1, 2 | emerg, alert, crit | `FATAL` |
| 3 | err | `ERROR` |
| 4 | warning | `WARN` |
| 5, 6 | notice, info | `INFO` |
| 7 | debug | `DEBUG` |
Both tiers that read a journal do this — the agent container's forwarder over
its own journal, and the swarm collector over the host's — and both import the
same mapping from `nix/journald-severity.nix`, which holds the table once.
Note the direction. Syslog's priority counts **down** in urgency, where
OpenTelemetry's severity number counts **up**. Neither is a passthrough of the
other, and a pipeline that hands one straight to the other renders debug lines
as critical without erroring anywhere.
A record that reaches the store from somewhere other than a journal has no
priority to map. Claude Code's own telemetry is the live example: it exports
OTLP log records directly (`OTEL_LOGS_EXPORTER`, see
`nix/agent-modules/claude-settings.nix`) and sets no severity on them, so they
store as `Unspecified`. The **Log rows with no severity** panel on the
`hyperhive · logs` dashboard counts both cases and keeps them apart: lines that
had a priority and arrived without a severity anyway signal a broken mapping,
while lines that never had one signal nothing of the sort.
### Why Explore's level buttons need the datasource told
The stored field is `severity_text`, and Grafana's log-level buttons filter on
a field called `level` — a name no row here carries. That's not something the
pipeline can fix: VictoriaLogs' OTLP ingester names the stored field
`severity_text` unconditionally (v1.52.0, `app/vlinsert/opentelemetry/pb.go`),
and its ingest parameters have no `_level_field` sibling to rename it with.
`nix/host-modules/swarm-grafana.nix` tells the datasource instead,
provisioning it with a `logLevelRules` entry per severity in the table
above, each matching `severity_text` against one of
those values — which is the plugin's way of saying "the level lives in this
field." `logLevelRules` is the datasource plugin's own jsonData key and the
only level-related one it has — there is no single "the level lives in field
X" string to set, so mapping every level takes one rule each. A level button
then filters on `level` **or** the matching `severity_text`, and the store
answers the second half. Each enabled rule appends an `OR
<field>:<op>"<value>"` term to the query a level button emits, beside the
`level:…` term that matches nothing here. Selecting "info" goes from
```
level:contains_common_case("info","information","informational","notice")
```
to that OR `severity_text:="INFO"`, which the rows here do match.
`Unspecified` is deliberately left out of those rules. It's VictoriaLogs' own
rendering of an absent severity, and it's what the **Log rows with no
severity** panel counts — giving it a level would dress the missing data up
as a colour and retire the instrument that measures it.
#### Three ways to get a `logLevelRules` entry wrong, all of them silent
Grafana accepts any jsonData it doesn't recognise, so a bad rule provisions
cleanly and the affected level button goes on returning zero rows:
- `enabled` must be literally `true`, not merely not-false. The query
builder keeps rules on `rule.enabled` being truthy while the row colouring
path keeps them on `!== false`, so an omitted flag colours rows correctly
and leaves the buttons broken — working in the half nobody is looking at.
- `level` must be a canonical Grafana `LogLevel` value. The builder groups
rules by it and only ever looks up `critical`, `error`, `warning`, `info`,
`debug`, `trace`; `warn` and `fatal` are enum aliases that resolve to
other spellings and match no group.
- the builder compares `value` with `===`, so it must be the severity text
exactly as stored: the uppercase OpenTelemetry short names `overwrite_text`
writes in `nix/journald-severity.nix`, not that table's lowercase keys.
## Host-emitted container-resource metrics (hive-c0re)
<!-- vale write-good.Passive = NO -->
**hive-c0re itself** also exports each agent
container's resource load — the same cgroup gauges —
to this hive's own collector, exactly like an agent does and with
no separate toggle. These come from the host, not the in-container Claude SDK,
so they cover containers even when their agent is idle.
<!-- vale write-good.Passive = YES -->
Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
metric names + the standard `container.name` attribute where a spec metric
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source |
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
alongside the spec `container.cpu.time` counter for convenience). The `swarm`
label rides on the resource, and the swarm collector adds `hive` from the
receiver that accepted the sample. Any other label goes in the hive's
`extraResourceAttributes`.
<!-- vale write-good.Passive = NO -->
**An absent `hyperhive.agent.unrestricted_internet` series doesn't mean off.**
A hive that predates the metric exports no series at all, and neither does a
hive that can't read its stored permissions. Only a `0` means the flag is off.
Count the agents still on with
`sum({"hyperhive.agent.unrestricted_internet"} == 1)`; dotted names need the
quoted-name selector.
The export cadence is a fixed 60s. Transport is OTLP/HTTP
(JSON) to the hive collector's bridge address, with no auth header — that
first hop is unauthenticated for every producer on this host, and the upstream
credential stays on the swarm tier.
## Container-crash alerting (hive-c0re)
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
classifies a container that stopped without an operator-initiated action as
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
no broker event, no per-hive dashboard banner. Both of those used to exist;
this change removed them in favour of the log line.
The hive tier's own collector runs a `journald` receiver scoped to the units
in `services.hyperhive.otel.journalUnits`: `hive-c0re.service`, the `container@*.service`
units (the start-time rootfs check lines), plus
`squid.service` and `squid-selfcheck.service` (see [Egress proxy](../networking/network.md#egress-proxy-squid)).
A host that runs the cache without the swarm collector adds `swarm-bao-attic-secret.service`
and `swarm-bao-attic-tokens.service`.
A host that also runs the swarm collector names none of these: its whole-host journal receiver
ships them already, and the hive collector then has no `journald` receiver.
No bind mount is needed. Unlike an agent's journal, this collector already
shares a filesystem with the units it reads, because it's a host systemd
service rather than a container. It forwards the crash log line alongside the
rest of the hive's telemetry. A
swarm-level grafana alert, `hh-container-crashed` in
[`nix/host-modules/swarm-grafana.nix`](../../../nix/host-modules/swarm-grafana.nix),
matches it directly and pages the operator — no agent, and no per-hive
dashboard, in the loop.
Requires `services.journald.storage = "persistent"` (the NixOS default) on
every hive host — asserted at eval time.
## Agent-emitted per-turn metrics (`hive-agent`)
<!-- vale write-good.Passive = NO -->
The harness itself (`hive-agent`) exports one small set
of metrics per claude turn, recorded the moment the turn ends (not polled).
These are deliberately the fields Claude Code's own built-in export (see
above) can't know about — the harness's own wall-clock timing, what woke the
turn, its own outcome classification, the loose-ends backlog, session
boundaries, how full the context window is, the tools a turn called, the shell
commands bash tasks ran, and reminders. Token usage is **not** duplicated here;
that's already covered by Claude's own export, which has no tool-call counter.
Cost is here only for agents on an ACP runtime, which have no export
of their own — see [ACP-reported cost](#acp-reported-cost).
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Attributes |
| ---------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------- |
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them) — polled every 5 minutes, **not** per turn |
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window` — unix seconds at which that window resets; same 5-minute poll |
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
| `hyperhive.agent.tool.calls` | — | counter | `tool` — the tool's name, or `Skill:<skill>` for a skill; counted when a turn ends |
| `hyperhive.agent.bash.commands` | — | counter | `command` — the command name a bash task ran (the first real command, past `cd` and `sudo`), counted when a turn ends |
| `hyperhive.agent.reminders` | — | counter | `event` (`scheduled` or `delivered`) |
| `hyperhive.agent.first_turn.context` | `{token}` | gauge | none — input tokens of the first turn of a fresh session, the cold cost of the system prompt and `AGENTS.md` |
An agent with no `subagent_roles/` directory exports no
`subagent_role.lines` series. A role file deleted while the harness runs
keeps its last value until the harness restarts.
For the two `claude_usage` gauges the harness polls the Claude subscription
usage endpoint (`GET /api/oauth/usage`, the one behind claude's own `/usage`)
with the OAuth access token from the agent's own
`~/.claude/.credentials.json`. The harness only reads that token and never
refreshes it — claude owns refresh-token rotation. An agent with no OAuth
session (API-key backends, not yet logged in) or an expired token skips the
poll, so a gauge keeps its last value until the next successful poll — a
`resets_at` in the past means the paired `percent` is stale.
The two `context` gauges hold the two numbers the web UI divides for its ctx%:
the last turn's context tokens (input, cache read and cache creation), and the
API-reported window, else the model's default. A turn that parsed no usage
leaves both where the previous turn put them. The dashboard's **Context
window used by agent** panel (health tab) divides one by the other.
Resource attributes (`service.name`, `agent`, `swarm`) come from the
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
section, and the swarm's collector adds `hive` — nothing extra to configure.
The export cadence is a fixed 60s, the same
as `hive-c0re`'s container-resource export above. That only controls how often
the harness flushes the batched points to the collector, not how often it
records them (every turn, always).
### ACP-reported cost
An ACP agent reports what its session has cost so far in the `cost` field of
its `usage_update` notifications — a running total, not a per-turn figure
(opencode sums every assistant message in the session). The harness counts
what each report adds to the last one and exports that as
`hyperhive.agent.cost.usage`, so `increase()` over it gives the money spent
in a range. It follows a few rules:
- A new session's first report counts in full, since the session started at
zero.
- A session loaded into a freshly started agent (after a harness restart or
an agent crash) has an unknown total until it reports one, so that first
report only sets the baseline — the turn it covers goes uncounted.
- A total lower than the previous one means the agent dropped part of the
session's history; it adds nothing and becomes the new baseline.
- Cost from a compaction the harness runs between turns counts toward the
next turn.
The value is in whatever `currency` the agent names; opencode always sends
`USD`, and the `hyperhive · agents` dashboard's ACP cost panels (cost tab)
filter on it. A claude agent never records this metric: claude's own export
already carries its cost as `claude_code.cost.usage`, and keeping the two
apart means no turn is ever counted twice. Add both names together for a
whole-swarm spend figure.
## Hive-scoped metrics (hive-c0re)
<!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these five per **hive** instead, and they carry no
`agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with
`{hive!="",agent=""}`.
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning |
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
included, so a healthy hive is visible as zeros rather than as missing
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above.
## VCS activity metrics (`swarm-controller`)
`swarm-controller` registers a single instance-wide Forgejo webhook (a
"global/system" hook, not scoped to any one org or repo) and counts commit
and push activity as deliveries arrive — occurrence-driven, not polled.
Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
counts of durable rows (issues, comments, repos), and Forgejo doesn't
store either a commit or a push anywhere as a row to count.
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ------- | ------------------- |
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
A push with zero commits (a branch delete, or a force-push that doesn't add
new commits) still increments `push.count`; `commit.count` only advances
when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_OTLP_ENDPOINT`), fixed 60s cadence
and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
crate's own doc comment) — `swarm-controller` sets `service.name =
swarm-controller` directly rather than reading it from the container
environment, since it's a standalone daemon, not a per-agent harness process.
Both exporters push through `swarm-controller`'s own authenticated client,
and the collector's `oidc/swarm` receiver checks the bearer. When a push
fails, `swarm-controller` logs `otel export: push to the collector failed`
at warn, with the kind of failure (`token`, `transport` or `status <code>`)
and the start of the response body. It repeats a persisting failure every
10 minutes with a count, and logs once at info when pushes succeed again.
## Review classifier result metrics (`swarm-controller`)
`swarm-controller` counts what review requests and review runs come to, from
the same exporter as the VCS metrics above. The classifier service adds the
retry counter beside its cost counters
([`review-classifier.md`](../swarm/review-classifier.md#observability-and-cost)).
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ------------------------------------------------------------------------------------ |
| `hyperhive.review_classifier.requests` | — | counter | `outcome`: `queued`, `queue_failed`, `unparsable`, `deferred`, `started`, `reposted` |
| `hyperhive.review_classifier.verdicts` | — | counter | `outcome`: `approve`, `request_changes`, `no_verdict` |
| `hyperhive.review_classifier.checks` | — | counter | `check` (the rule's name), `result`: `match`, `no_match`, `unsure`, `no_answer` |
| `hyperhive.review_classifier.run.duration` | `s` | histogram | `outcome`: the verdict names, or `post_failed` |
| `hyperhive.review_classifier.ask.retries` | — | counter | `runtime`: `api_key`, `acp` |
A verdict means the classifier decided, before it posts the review: a run
whose post fails still adds to `verdicts`, and shows as `post_failed` in
`run.duration`. A repo with no rules file gets an approval with no model call and
counts as an `approve` verdict and run, with no `checks`.
`deferred` counts each time the queue hands over a request for a PR whose
review is still going, so a long review adds one per retry wait. `reposted`
is a request for a head already reviewed: it posts the earlier verdict and
adds to `verdicts` but runs no checks, so it adds nothing to `checks` or
`run.duration`. A request the queue refuses posts a comment and counts
as `queue_failed` and a `no_verdict`. `no_answer` is a classification that
failed, so a `no_verdict` run with a `no_answer` check is the usual cause of
a comment instead of a review. The `check` values are the names in a repo's
rules file, at most 10 per repo.
The `review-classifier` dashboard in Grafana charts these.
## Swarm builder metrics
`swarm-builder` exports what it builds through the OTLP receiver of its
container's own forwarder, which pushes to the swarm collector as the builder's
client. `nix/host-modules/swarm-builder.nix` sets
`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` for the daemon. The exporter is the same setup as the review
classifier's: HTTP/JSON, cumulative, one export a minute, and nothing exported
without `OTEL_EXPORTER_OTLP_ENDPOINT`.
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ---------------------------------------------------------------- |
| `hyperhive.swarm_builder.builds` | — | counter | `kind`: `pr`, `meta`; `outcome`: `success`, `failure`, `refused` |
| `hyperhive.swarm_builder.build.duration` | `s` | histogram | `kind`, `outcome`: `success`, `failure` |
| `hyperhive.swarm_builder.queue.depth` | — | gauge | none |
| `hyperhive.swarm_builder.queue.wait` | `s` | histogram | `kind` |
| `hyperhive.swarm_builder.publish.failures` | — | counter | none |
`kind` `pr` is an agent config PR's head and `meta` an agent built from the
swarm meta flake. Lock and archetype requests aren't counted. `refused` is a
request that waited out the one-build slot, which the builder then answers busy
without building, so it has a `builds` point and no `build.duration` point. A build that
can't start (no forge token) counts as a `failure` with a duration; a payload
that doesn't decode counts as a `failure` in `builds` only.
`build.duration` runs from holding the slot to the reply, including the cache
push of a meta build. `queue.wait` runs from the request's arrival to the slot,
or to the refusal. `queue.depth` is the requests waiting for the slot now,
exported every minute, 0 when idle, so it doubles as the liveness signal; the
`bao.client_cert.*` gauges below ride the same exporter.
`publish.failures` counts replies the queue refused to take, which the
controller sees as a missing answer.
## Store client-certificate metrics
Every service that logs in to the secret store with a client certificate can
report when that certificate stops being valid. Two observable gauges, one
data point each, unix seconds:
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ----- | ---------- |
| `bao.client_cert.not_before` | `s` | gauge | `cert` |
| `bao.client_cert.not_after` | `s` | gauge | `cert` |
`cert` names the identity (`client`, `controller`, `agent`, `review-classifier`,
`ci-metrics`, `builder`); the resource labels (`service.name`, `hive`, `agent`) say who
holds it. `swarm_secret_client::cert_metrics` (cargo feature `otel`) registers both gauges on the service's
existing exporter and reads `BAO_CLIENT_CERT` at every collection, so a
renewed leaf shows up on the next export. A file that's missing, or that isn't a
certificate yields no data point and a debug log line. The parser is
`swarm_secret_client::cert`.
| Service | `cert` | Exporter it rides |
| ------------------------- | ------------------- | --------------------------------------------------- |
| `hive-c0re` | `client` | the container-stats exporter |
| `hive-agent` | `agent` | the per-turn exporter, built on its first recording |
| `swarm-controller` | `controller` | the job-graph exporter |
| `swarm-review-classifier` | `review-classifier` | the cost exporter, built on its first recording |
| `swarm-ci-metrics` | `ci-metrics` | a push of its own at the start of each timer pass |
| `swarm-builder` | `builder` | the build exporter, started with the daemon |
Not covered yet: the other six Rust services that hold a store certificate
(`swarm-claude-login`, `swarm-notify-router`,
`swarm-nats-auth`, `swarm-logs-auth`, `swarm-metrics-auth`,
`swarm-matrix-ctl`) have no exporter, and the ten identities that only shell
units use report nothing. Neither does any CA certificate. Each leaf's own
window is all the gauges show, and a CA that expires before its leaves is
invisible to them.
The swarm Grafana alerts on the lifetime fraction:
| Alert | Fires when |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `hh-bao-client-cert-lifetime` | `(time() - not_before) / (not_after - not_before)` stays above 0.8 for 10 minutes, per `service.name`, `cert`, `hive` and `agent` |
The two gauges of a series carry identical labels, so MetricsQL divides them
one-to-one without `on()` or `group_left`. The query wraps each in
`last_over_time(...[15m])` so `swarm-ci-metrics`, which reports every five
minutes, keeps a series. A series that stops reporting resolves the alert
rather than firing it (no data counts as OK).
An agent leaf lives 7 days and renews at half its life, so a fraction over 0.8
means the renewal stalled about two days ago. The host leaves (`client`,
`controller`, and the others minted by `swarm-bao-pki`) live 3650 days and never
renew; for them 0.8 comes about eight years after minting. A guard on the days
left would suit them better and doesn't exist yet.
## Agent-emitted custom metrics (`hive-metric`)
Agents can push arbitrary labeled metrics to the same OTEL collector via the
`hive-metric` CLI tool, available in every agent container.
### Usage
```text
hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]
```
- `<name>` — metric name (for example `tasks_completed`, `latency_ms`).
- `<value>` — numeric value (f64; integers and floats both accepted).
- `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or
`gauge` (instantaneous point-in-time value).
- `--temporality delta|cumulative` — counter reporting mode (`counter` only,
ignored for `gauge`): `delta` (this call's own contribution, default — send
`1` each time and the collector accumulates) or `cumulative` (this call
reports the running total, which a stateless one-shot CLI can't track
itself).
- `--labels key=value` — extra per-data-point labels. Repeat the flag for
more than one.
`hive-metric` inherits the resource labels (agent, swarm, service.name)
automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them.
### Examples
```text
# Counter: one more task finished (delta is the default — no flag needed)
hive-metric tasks_completed 1 --labels phase=scan
# Gauge: current queue depth (absolute value — must use --type gauge)
hive-metric queue_depth 17 --type gauge
# Float gauge with multiple labels (instantaneous measurement)
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
```
### Error when an agent turns OTEL off
An agent whose own config sets `services.hyperhive.agent.otel.enable = false`
has no `OTEL_EXPORTER_OTLP_ENDPOINT` env var, and `hive-metric` exits with an
informative error message. No silently dropped metrics.
### Wire format
`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. `hive-metric` forwards auth headers from
`OTEL_EXPORTER_OTLP_HEADERS` verbatim.
## Metrics temporality
The harness configures OTEL export with **cumulative** temporality by default
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
overriding Claude Code's default of DELTA. This avoids silent metric drops in
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
delta-to-cumulative processor.
<!-- vale write-good.Passive = NO -->
**`hive-metric` counters are the one exception**, reporting delta by default
(see above) — programmatically set on the exporter, which overrides this
container-wide env var for that tool specifically. `--type gauge` is
unaffected either way; gauges have no temporality. The hive-tier collector
runs a `deltatocumulative` processor ahead of export, so a delta
`hive-metric` counter still lands in VictoriaMetrics as a cumulative
series — the standard `rate()`/`increase()` idioms work on it exactly like
any other counter in this system, no extra query needed.
<!-- vale write-good.Passive = YES -->

152
docs/4-implementation/swarm/bao.md Normal file
View file

@ -0,0 +1,152 @@
# The swarm secret store (OpenBao)
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
credential an agent or service needs comes from it under that
principal's own certificate identity. What lives in it and who reads what:
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md). This page covers the store itself — how it
comes up, who may write its grants, and how it's sealed and reached.
The operator steps (init, the one-time granter bootstrap) are in
[`../getting-started/setup.md`](../../1-getting-started/setup.md#1--secret-store);
this page is the why behind them.
<!-- vale write-good.Passive = NO -->
## The granter
Cert auth answers a _role_, so nothing can authenticate until some role
exists. A short-lived bootstrap token breaks that cycle once, and it's the
only step that needs the root token.
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
covers the auth mounts and the granter's own policy and role, and nothing
else. CI fails when the unit using the token needs a path it lacks. The token
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
names for you; on a store host that isn't all-local, set it and rebuild first.
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
auth methods (and disables `approle` if an older store still has it mounted),
writes the `bao-granter` policy, and creates the `bao-granter` role, which
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
mounts the KV and pki engines and writes the `swarm-controller` role, and each
sibling unit writes its own principal's policy and role. Every one runs on the
host, because every API listener but the loopback UI one demands a client
certificate and the host is the side that has one.
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
the token file is absent, which is the steady state afterwards; the token's
TTL means a forgotten file expires rather than lingering.
A sealed or unreachable store, or one whose listing fails with something other
than a refusal (a node still becoming active), makes the unit retry every
30 s with the store's error in the log. A store that refuses the token (expired,
revoked, or missing the `bao-bootstrap` policy)
makes it exit 4 and stay `failed` with the one-time step in its log; mint a
fresh token or remove the file, then restart it. If the granter's own
certificate login already works, the unit logs that the bootstrap token isn't
needed, deletes the dead token file and exits 0, so a stale file is harmless
and the unit runs once. A successful run deletes the token file too.
After that, a new or changed `swarm-*` grant needs no operator step: the unit
that writes it changes, and the deploy restarts it. A root step comes back only
when the granter itself needs a path it lacks, such as a new mount.
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
them, not on every deploy. When a grant drifts in the store and its unit stays
the same, the next boot re-asserts it, not the next switch.
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
wrapper carries an address, a CA and a client certificate but deliberately
**no token**, so that read answers `403` whether or not the role exists. Read
the unit's journal instead.
### Residual risk
The granter is root-equivalent. It may write any `swarm-*` policy with any
content, and attach it to a role that accepts any certificate; no bao ACL can
constrain what a policy says. What bounds it:
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
module-eval pins each principal's grants. **Merging a change to that policy
text is granting it**: it takes effect on the next deploy with no bao step,
so code review is the only gate.
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
root in a `0700` directory, readable only by root units on the store's host.
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
and `controller-key.pem`, whose policy is already root-equivalent. Root on
that host gains nothing new.
- **Never copy `granter-key.pem` off the host** the way operators copy the
other leaves in that directory. That hands out root-equivalence.
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
subject. Rotate the store's CA, or have root point the `bao-granter` role at
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
one.
## Sealing
`services.hyperhive.deploy.bao.seal`:
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
unseals itself on every restart. `init` is the only manual step.
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
restart, with the keys `init` printed.
⚠️ **A sealed store still answers.** The container is up and the port
responds while every read times out — the failure looks like a stall, not like
a store that was never initialised.
## TLS
The store serves TLS, and on a host that deploys it you need do nothing:
`swarm-bao-pki` mints a CA of the store's own plus the leaves it signs — the
store's server certificate, this host's client certificate, and one per service
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
It runs at every boot and every deploy, and before each start of a container
that loads one of its leaves, and signs only the files that are missing, so a
leaf a deploy adds, or a service a deploy enables, has its file on that deploy.
The store's readers and granting units on this host order after it and want it,
but none requires it: when a run fails, each of them still starts, and one
whose leaf is missing fails its login into its own retry. It has no start
limit, since a mass container start pulls it in once for each unit it starts.
Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about
which. A host that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see
[per-principal identities](../../2-handbook/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
They're the credentials the store can't hand you, being what opens it, so the
store's host signs them and the operator copies them to the host that needs
them and names them in config. The service principals' leaves are at the top of
`/var/lib/swarm-bao-pki/`, named after the principal.
For every hive in `swarm.hives` other than its own, the store's host signs
that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
| file | subject | name it in |
| ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on
the store host's next deploy; a leaf already there is never re-signed.
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
between services and the store**, a separate trust domain, because a store
that took its identity from an authority it itself distributes could never
come up before that authority.
Inside the store's container (`nixos-container root-login swarm-bao`) the
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
and its DNS name resolves to the bridge from in there: export
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
domain>`. The host's `bao` wrapper already carries the address, CA and client
certificate, so the host is the shorter path.
<!-- vale write-good.Passive = YES -->

607
docs/4-implementation/swarm/builder.md Normal file
View file

@ -0,0 +1,607 @@
# Swarm builder
The swarm builder checks that an agent config PR builds before an operator
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
push, the builder builds the PR's head on the agent base, and the result shows as a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a failed build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](../../2-handbook/swarm/services.md#builder).
## What happens on a config PR
1. The forge delivers the `pull_request` event on the `config-pr` webhook
swarm-controller already registers on `agent-configs`. An `opened`,
`reopened` or `synchronized` action on an open PR queues one `build_agent`
node carrying the agent, the head commit and the PR number. An edit, a
label or a close builds nothing. Every `build_agent` node needs the one
`builder` slot, so builds run one at a time and the rest wait pending. A
newer head cancels its PR's pending build; a build already running
finishes and writes its status on its own, older commit.
2. The node sets the status to `pending`, then sends the build as a request on
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
reply. The builder runs one build at a time and refuses a request that has
waited 30 seconds for its turn, such as one behind a build a controller
restart orphaned: a build started later could still run when the 61
minutes end. Reading the forge token before a build has 20 seconds, and
the rest of the minute covers the reply's trip back.
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
with every input overridden:
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
deployed with (`deploy.hyperhiveFlake`, `deploy.builder.nixpkgsFlake`). A hive's meta
flake pins `hyperhive` by forge URL and rev instead. The builder reads the
hyperhive path from `/etc/hyperhive/swarm-builder.json` in its container at
the start of each build, so a deploy that changes only that source doesn't
restart it or end a build in progress.
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
A build has one hour.
4. The reply is the system closure's store path, or the last lines `nix`
printed. The node writes the status on the head commit:
| Outcome | Status | Description | Node |
| ----------------------------------------- | --------- | --------------------------- | ------ |
| built | `success` | the store path | done |
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
| the builder busy for 30 seconds | `failure` | why | failed |
| no builder listening, or no reply in time | `error` | why | failed |
The controller records every build that gets a final status, and the
status links to that record: see [Build records](#build-records). The
full tail is also in the failed node's error on the Jobs page and in the
builder's journal line for the build, as the field `F_ERROR_TAIL`:
```console
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
```
The builder holds the last 64 KiB of each of `nix`'s output streams while the
build runs, and the reply is the last 30 lines of that (at most 4000
bytes), so a long build log never grows its memory.
## Build records
`swarm-controller` records each config PR build when it writes the final status:
built, failed, or no build ran or came back. It also records each [deploy-gate
build](#deploy-gate) the builder answers, in the same list. The newest 100 are one JSON
array under the key `recent` in the `builder-runs` KV bucket on the swarm
queue, which only the controller reads or writes. It creates the bucket on
first use and adds each build with a compare-and-set write, so the records
survive a controller restart. Recording the 101st drops the oldest. A record
the controller can't decode stays in the bucket, out of the list, and counts
toward the 100.
Each record has:
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>` for a
config PR's build, `meta-<agent>-<first 12 characters of the commit>-<started_unix>`
for a deploy's
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`; a deploy's build has
`repo` `internal/meta`, no `pr`, the `internal/meta` commit as `head_sha`,
and `agent`, the agent it built
- `started_unix` and `finished_unix`
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
or `error`
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
`failed`, and `detail` (why) for `error`
`GET /api/builder/runs` lists them, newest first, and
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
when the controller has no swarm queue or can't open the bucket, 500 when it
can't read it.
The controller records a deploy-gate build when the builder answers it, or when
asking it fails (no builder listening, no reply). A build that never got as far
as a request (no forge access, no `internal/meta` head) leaves no record, so a
controller that can't reach the forge doesn't fill the list with retries. Every
ten-minute retry of a failing agent is a record, and a version bump records one
build per agent, so the 100 cover fewer days when builds fail.
The final status links to `https://<swarm domain>/jobs?run=<id>`, the swarm
UI's jobs tab, where the builds are a card under the job graph. Selecting
**Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record
stays at `GET /api/builder/runs/<id>`. A status has no link when the record wasn't
stored (the controller logs the failed write) or when the controller doesn't
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
status has no link.
## What the check is, and isn't
- It composes the config the way a hive's meta flake does
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
what a hive adds: its CA bundle, `claude-code` path, ports and
OTEL settings, and the `follows` that point the config's own inputs at the
hive's. A green check can therefore still fail on a hive, and the reverse.
- The agent base is the builder's host's, not the hive's. Hives on another
hyperhive or nixpkgs pin build something else.
- The check pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
can push to [a binary cache](#binary-cache).
- The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery of a PR push isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
## Archetype evaluation
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
`meta` of every agent archetype its own hyperhive source declares, with no
forge token, and answers with the JSON. An evaluation doesn't wait for a
build in progress and has five minutes; the controller waits six. The builder
logs each evaluation's time as `elapsed_ms`, and a timeout's error carries the last
lines nix printed. What the controller does with the answer: [`archetypes.md`](../../2-handbook/swarm/archetypes.md).
## Meta-flake lock
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
rendered `flake.nix`. The builder writes it into a scratch directory, runs
`nix flake lock` there with the same forge credential helper its builds use,
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
lock doesn't wait for a build in progress and has ten minutes. The builder
commits nothing.
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
with the `hive-meta-flake` renderer, all over `https`.
Each custom agent that a hive places pins to its `agent-configs/<agent>`
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
running rev (`services.hyperhive.deploy.hive-controller.hyperhiveRev`, from `self.rev`):
- the first commit pins hyperhive's forge `main`;
- later, the pin moves to the controller's rev only when the forge's compare
API shows that rev is a strict descendant of the committed pin: commits
ahead of it, and none the pin has that it lacks. The compare endpoint counts
histories with no common ancestor as zero commits ahead, so the check needs
both directions;
- an older controller, a rev on another line, a controller built from a dirty
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
`main` moves nothing.
The controller reads its rev from `/etc/hyperhive/swarm-controller.json` at each
render, not from its unit, so a deploy that changes no Rust doesn't restart it.
A missing file, or one with no rev, counts as a controller without one.
The controller commits `flake.nix` and the returned `flake.lock` to
`internal/meta` in one commit, and gives each hive's forge account read on the
repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run only after the
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
builds only from `internal/meta`, so after a failed update the scheduler skips
the deploy node, and the graph shows the update as failed and the deploy as
skipped. Each ten-minute retry of a failed update declares a fresh build and
deploy behind it for every agent whose latest deploy the failed update skipped,
so the run that succeeds sends them (see
[Config changes](../../2-handbook/agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.
The controller sends a deploy to every agent a hive places whenever the pin in
`internal/meta` differs from the pin it last sent, so they rebuild on the new
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
which only the controller reads or writes, and a `RecordFanout` job writes the
key once every deploy of the fan-out has finished or the deploy gate has skipped
it. The check runs after each update and once when the controller starts.
- The first commit sends a deploy, because the key holds no pin yet.
- A controller that starts with no key, as after the upgrade that added it,
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because its build failed.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
writes no key.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake the swarm builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. A green build here means the
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
this build produced and never builds one itself.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
| What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs |
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
On every ten-minute round the timer declares a build and deploy again for each
agent whose latest build failed, without a meta update when the latest update
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
still records the pin when the gate skips a deploy, so one failing agent doesn't
resend the healthy ones. A controller that updates before its builder host does
is the same case: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each round adds a build and deploy node per failing agent.
A merged config PR whose webhook delivery never arrives would leave the agent
on its old config. Every five minutes, after the config-PR scan, the controller
reads `main` of each `agent-configs/<agent>` repo that has a commit and that
exactly one hive places. It compares that head with the config commit pinned for
the agent in the `flake.nix` of the `internal/meta` commit its `agent-closures`
key names, since the pin in `internal/meta`'s own head also moves for other
agents' merges. When they differ, it declares the same update, build and deploy
a merge declares, and logs at info `config drift: …`. It leaves alone an agent
with no key, a key whose flake pins no config commit for the agent, and a head
the graph already holds a deploy for, in any state: the ten-minute rounds above
build a head whose build failed again, not this pass. The graph is
in memory, so a controller restart tries each such head once more.
When the build belongs to a merged config PR and `nix` exited with an error or hit
its deadline, the controller also comments once on that PR, with its forge
account: the agent, the `internal/meta` commit of the first red build, `nix`'s last lines (at most 4000
bytes), and that the deploy waits and the swarm builds it again every round. A comment
carries a marker line with the agent, and the controller posts only if the PR
holds no comment of its own with that marker; a copy of the marker in someone else's comment is neither counted nor edited. The retry rounds stay quiet whatever commit
they build, and the latest error stays on the Jobs page. A green build of the
agent for that PR rewrites the marker and appends the build's commit to the comment, so a later red comments again.
A `no verdict` failure, and a build that isn't a
merged PR's (a new agent, the pin fan-out), leave no comment. When the forge
refuses the comment, the controller logs a warning and the node ends as before.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
not updating.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key.
The record's `cached` field says whether the builder pushed the closure to
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
The built reply and the record also carry two options of the agent's evaluated
config, `private_users` (`services.hyperhive.agent.user.privateUsers`) and
`nix_builds` (`services.hyperhive.agent.nixBuilds`). The builder reads each with
a `nix eval` in its sandbox, and an eval has three outcomes:
- **A value:** the field holds it.
- **The module doesn't declare the option** (`nix` says the flake doesn't
provide the attribute): the field holds the option's default, off for
`private_users` and `host` for `nix_builds`.
- **Any other failure, or a timeout:** the value is unknown, and the reply and
the record leave the field out. hive-c0re then sends nothing for that option,
and hive-priv keeps what the container's last write chose, as it does for an
agent with no record.
Mixed versions:
| builder | controller and hive-c0re | what happens |
| ------- | ------------------------ | -------------------------------------------------------------------------------------------------------------- |
| new | new | an unread option leaves the field out, and the last write stands |
| old | new | the old builder always sends a value, which the record carries |
| new | old | the old side reads a missing field as the default, so an unread option behaves as it did before this behaviour |
A record or reply from before the fields reads as unknown.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
grant the forge refuses fails the build with no verdict.
## Binary cache
`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
each green meta build's closure to it, or to the cache you override them with,
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
not `attic watch`, and the builder pushes again when the controller runs a build again. The
push has ten minutes, or less when the reply is due sooner, because it must end
inside the 61 minutes the controller waits. `attic` checks the server's certificate
against the system roots and pins nothing, so a Let's Encrypt certificate
works.
Set both options to a name or both to null: evaluation refuses one without the
other. Null in both is how a swarm opts out of pushing.
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
starting with a letter or digit, and evaluation refuses any other name.
The push token is the swarm's own cache's to create, or the operator's to write for an
external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to
a file and an `attic` config naming that file (`token-file`) in the unit's private
runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push.
The token is in no argument or variable, so no process listing shows it.
The reply's `cached` says whether the push finished. A push that fails, times out
or can't read the token leaves the build green with `cached` false and logs a
warning with the reason; the deploy goes ahead and the next build of the agent
tries again. With no attic behind the defaults, every push fails this way. With
both options null, the builder pushes nothing and `cached` is always false.
Mixed versions:
| controller | builder | what happens |
| ---------- | ------- | ------------------------------------------------------------- |
| new | new | `cached` in the reply and in the `agent-closures` record |
| new | old | the reply has no `cached`, which reads as `false` |
| old | new | the controller ignores `cached`; its record has no such field |
A record written before the field existed also reads as `false`.
### Rebuild on a cache miss
The cache drops a closure nobody fetched for its retention period, but the
record in `agent-closures` stays. Every ten minutes `swarm-controller` checks
each agent a hive places: when the record is from `internal/meta`'s head and
says the builder pushed it, the controller fetches the narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token it reads from
`secret/swarm/services/attic/` (the same four leaves it
[copies to the hives](#what-hives-do-with-the-cache); with any missing, or a URL
that isn't `https`, the pass does nothing). A 404 is a miss, and the controller
sends the builder `build-meta` for that agent and the record's own commit. The
builder builds it and pushes as for any meta build. Any other answer or a failed
request isn't a miss, and rebuilds nothing. The next ordinary build handles a record
from an older commit, or one with `cached` false.
A pass sends at most three builds, one after the other. After an agent's rebuild
failed, got no answer or built without pushing, the pass skips it for 30 minutes, then
for twice as long after each further failure, up to 12 hours. The wait is in memory,
so a controller restart tries every agent again. The first pass runs ten
minutes after the controller starts.
The record keeps naming what hives deploy. A rebuild that lands on the record's
path writes nothing. A rebuild that lands on another path means the build isn't
reproducible: the controller logs a warning, and replaces the record only if it
still holds the same commit and no write came since the controller read it. The
record otherwise stays as it was.
The controller counts each rebuild in `hyperhive.builder.cache_miss_rebuilds` with
the label `outcome`: `rebuilt`, `path_changed` (the record now names the new path),
`path_changed_kept` (the record stayed), `not_pushed`, `failed` (`nix` failed) or
`unavailable` (the builder was busy, absent or gave no answer).
Mixed versions: a builder needs no change, since a meta build already names
its commit. An old controller does nothing about a miss, as before. A hive
sees nothing new.
### The swarm's own cache
`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic`
container behind the gateway, at `attic.<swarm domain>`. It defaults on where
you set `deploy.allSwarmServices` and the host holds the cache's store identity
(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write
no secret: on first start, two host units log in to the store as the cache's own
principal and create what's missing.
- `swarm-bao-attic-secret` creates the key atticd signs tokens with at
`secret/swarm/services/attic/jwt-secret` and writes it where the container
reads it.
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
the builder reads, at `secret/swarm/services/attic/push-token`, and a
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
It also stores the facts a hive pulls with: the cache's address at
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
`secret/swarm/services/attic/cache`, and the public key it signs with at
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
A unit creates each secret only while the store holds none, so a second start
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
write the new token to its path in the store, in its UI under the OIDC login:
nothing mints over a value that exists. A new signing key invalidates both tokens.
The unit creates the address, name and key once and never updates them. After
you change `swarm.attic.domain`, or after the server's database goes and the
cache gets a new key, delete the stale leaf and restart
`swarm-bao-attic-tokens`.
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any
host pushes to it.
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
path nobody added or fetched for 14 days goes at the next garbage collection,
which runs every 12 hours (`deploy.attic.retentionPeriod`). The controller
[rebuilds a closure the cache lost](#rebuild-on-a-cache-miss).
The container sits on the hive bridge in a network namespace of its own. atticd
listens on that namespace's loopback (`deploy.attic.port`), and a socket-activated
relay forwards `/run/swarm-attic/atticd.sock` to it, a directory only the
gateway's nginx and the container can open. The gateway terminates TLS with a
swarm services certificate and forwards to that socket with no login in front:
attic checks its own tokens, and a nix client can't follow a login redirect.
The two host units reach the cache through the gateway, at
`https://<swarm.attic.domain>`, like every other client; the host resolves the
name through the hive resolver, and where the gateway serves a self-signed
certificate `swarm-bao-attic-tokens` trusts the swarm's CA. The container logs
to its journal only; its journal doesn't reach the swarm log store.
### External cache
The cache can be an attic tenant on an instance outside the swarm. To point
the swarm at an existing tenant:
1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
instance's host name (no scheme, `host:port` if it isn't on 443) and
`services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
both options on every host that reads them.
2. Seed the push token in the swarm secret store at
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
command line. The token needs push rights on that cache. Until it
exists, every push fails and builds stay green with `cached` false.
Seed the hives' pull token the same way at
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
3. Seed the three facts hives pull with, each as field `value`:
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
cache's page or `attic cache info`). Until all four exist, hives report
`no-credentials`.
The builder checks the instance's certificate against the system roots and
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
reads the token again on every push, so a new token takes effect on the next
build.
To use an external cache, override the two defaults and don't set
`deploy.attic.enable` anywhere. Where you host the shared services with
`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to.
A host that has `deploy.attic.enable` on and `swarm.attic.domain` set to another
name gets an evaluation warning that names this fix.
## What hives do with the cache
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
leaves (`pull-token`, `url`, `cache`, `public-key`) to
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
own prefix. It writes a copy only when the value differs, skips a leaf the
store doesn't hold, and leaves a copy whose source disappeared.
`hive-c0re` reads its four leaves every five minutes under the hive's own store
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
token goes out in a header and in the netrc; a hive that finds another scheme
writes nothing.
`hive-priv` doesn't read those files. The key that must have signed a download and
the pull token come from `hive-priv-cache-facts`, a root unit that reads the
hive's own `attic/public-key` and `attic/pull-token` from the store under the
hive's own store identity every ten minutes and writes them to
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`,
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes
decides nothing `hive-priv` trusts.
The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
It checks that a signature by the public key covers the narinfo, in Rust, with
no help from the host's nix daemon. It downloads nothing. The result is a log
line (at info when it changes) and the gauge
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
latest closure, else `0` with a `reason` label. The first check that fails
decides the reason:
| `reason` | Meaning |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
| `no-record` | `agent-closures` holds no build for the agent. |
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
A failed read of the store or the queue leaves the last result in
place and raises a warning after three failed passes. A hive reports nothing
before its first pass.
Mixed versions:
| controller | hive | what happens |
| ---------- | ---- | ----------------------------------------------------------------------------------- |
| new | new | the copies exist and the hive reports per agent |
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
| new | old | the controller writes copies nobody reads |
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
The signature check follows Nix's fingerprint format but hasn't run against a
live attic. A `bad-signature` on every agent in the first pass after a deploy
means check attic's fingerprint format; it says nothing about an attacker. A
`fetch-failed` on every agent points at the URL, the token or the network.
### Rebuilds download the closure
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
`Swap` then does the same lookup and download, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. A stopped agent skips
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
same and creates the container from that path.
A rebuild or create has two outcomes, by what the lookup and the download find. The hive deploys the record's closure whatever `internal/meta` commit the builder built it from, and reads no forge head to choose one:
| Found | What happens |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| a record that the builder pushed | `MetaSync` reads the record once and pins its store path, so a record that changes during the rebuild doesn't change what `Swap` applies. The hive downloads the pinned closure and deploys it, then writes the store path to the agent's `applied/.<name>.closure` marker. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
The hive never falls back to a local build. The second row has one owner in the
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
an unreachable cache costs is one function.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
| old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
## Credentials
| What | Where | Who reads it |
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo, and on `internal/meta` through the meta update) and mints a
`read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
status with its own forge token. Details per path:
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md).
The builder exports build counts, durations and queue wait; see
[Swarm builder metrics](../scheduler/observability.md#swarm-builder-metrics).
## Isolation
The container has its own network namespace on the bridge, as the CI runner
does, so a config's evaluation can't reach a host-loopback service. Evaluation
is pure and runs in the container. Builds go to the host's nix daemon as an
untrusted client: a config can set no daemon option, and its derivations build
in the host's sandbox. A fixed-output derivation still has the network, as it
does for any build on that host.

289
docs/4-implementation/swarm/ca.md Normal file
View file

@ -0,0 +1,289 @@
# Swarm CA
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
through a role that belongs to that hive. A peer that trusts the mount's
root once validates every hive's gateway and every swarm service name,
present and future, instead of pinning to each one by hand.
That's the point of one issuer — it turns per-peer trust from O(n²)
hand-pinning into one anchor per swarm.
## The gateway leaf comes from the secret store
`hive-tls-resign.service` logs in to the store with this hive's own
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
the files the gateway copies from. The leaf's subject is the hive domain,
and its SANs are the hive domain and `*.<hive domain>`.
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
hive's domain and `*.<hive domain>`, read from
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and `read` on its own `attic/public-key`, `attic/pull-token` and
`squid-selfcheck`
(see [the cache facts for hive-priv](../../2-handbook/swarm/credentials-reference.md) and the
squid self-check password), and nothing
else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every
hive in `swarm.hives`.
**Every hive needs its own store identity.** It's a client certificate
from the store's own client CA, with the subject
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
`baoHiveIdentityKeyFile`:
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
- On every other host, copy `hive-leaf.pem` and `hive-leaf-key.pem` from
`/var/lib/swarm-bao-pki/hives/<hiveName>/` on the store's host, which
`swarm-bao-pki.service` mints for every other hive in `swarm.hives`, and
set both options, before the host's next deploy.
Without it, `hive-tls-resign.service` fails with a message naming both
options, and so does `swarm-services-cert.service` on a hive fronting no
swarm name. The unit checks the identity before the leaf, so it fails even while a
valid leaf keeps the gateway up. The identity is the hive's alone:
it issues this hive's leaf and fetches the store's root, and it's never the
identity a swarm service uses.
**When it issues.** The unit runs at boot, on every deploy, and from
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
the current one is missing, past half the role's 720-hour lifetime, missing
a configured name, doesn't match the key beside it, or doesn't chain to the
store's root (`swarm-services-root.pem`). The first run that reaches the
store replaces a leaf that doesn't chain to that root, whoever signed it.
**The gateway doesn't wait for the store.** Nothing orders the gateway after
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
current leaf in place, says so in the journal, and retries every 30 seconds
for a day. Only a hive with no leaf yet needs the store reachable and
unsealed: its gateway's certificate import fails until the first leaf
lands, and then the unit starts nginx.
nginx serves the leaf alone, without its issuer: the issuer is the
self-signed root a consumer already holds in its trust bundle.
**A gateway host that runs no hive** (`deploy.hive-controller.enable` off,
such as a dedicated swarm-services or secret-store host) has no hive leaf, no
`hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its
gateway serves the swarm-services leaf as its default certificate too. Where it
fronts other swarm names it waits for `swarm-services-cert.service` at start.
Where the store's UI is its only swarm name it starts on a throwaway
self-signed certificate and switches to the services leaf once the store
issues it.
## The trust bundle
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
the file every consumer anchors on. It holds one certificate, a copy of
`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
the swarm-services leaf both chain to. It's the only root the swarm has.
**Without the store's root, the bundle is empty and its readers go red until
it lands.**
While `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem` is
missing or empty, as on a first boot before the store answers,
`hive-tls-ca.service` writes an empty bundle and logs that file's path. Every
`<name>-ca-bundle` unit then fails with `<bundle> contributed no certificate
to the bundle`, and the consumers that `Requires=` it don't start.
`swarm-services-cert.service` reloads `hive-tls-ca.service` when it writes a
new root. That rewrites the bundle in place and restarts no container, the
store's included.
**Each `<name>-ca-bundle` unit converges on its own.** Beside it, a
`<name>-ca-bundle-refresh.path` unit watches the file the bundle derives
from: inside a container the bound `/run/hive-ca/trust-bundle.pem`, on the host
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`. A write to it starts
`<name>-ca-bundle-refresh.service`. That service does nothing while the file
holds no certificate, or while it matches the copy the last assembly took
(`/run/<name>-ca/anchor.pem`). Otherwise it restarts the bundle unit and starts
its consumers without waiting. The restart reaches the running consumers
through their `Requires=`; the start picks up those a first boot left failed.
The same service runs once at boot, after the bundle unit, which covers a root
that landed before the watch began. A first boot needs no second apply, and a
root rotation restarts the consumers once and the container never. The watch
fires for the host's in-place rewrite because inotify keys on the inode and a
bind mount shares it, so `hive-tls-ca` must keep rewriting the file in place
and never rename over it.
A deploy that changes `hive-tls-ca.service` reloads it too, with the same
effect.
⚠️ **A consumer outside `trustBundle` reads its trust when it starts.** A
container whose nix predates the refresh units keeps the bundle it started
with until it restarts, and fails TLS to a gateway serving a store-issued leaf
in the meantime.
## The swarm's service names come from the secret store
The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
**single-label** wildcard over it, and its role allows nothing outside the hive's
domain, so it can't cover them.
The secret store's `pki` mount issues them a leaf of their own, through its
`swarm-services` role. `swarm-services-cert.service` logs in with
the services-issuer identity, calls
`pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
copies from. The store's role, not a key on the host, decides which names a
gateway can serve.
**Every self-signed gateway that fronts a swarm name needs the
services-issuer identity**, whether or not a hive runs on its host. A hive
fronting none needs no copy of it: it fetches the store's root with its own
store identity, as below. It's a client certificate
from the store's own client CA, with the subject
`deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default),
and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile`
and `servicesIssuerClientKeyFile`. Its policy may issue from
`pki/issue/swarm-services` and nothing else, and it's the same certificate on
every gateway host:
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to
it.
- On every other gateway host fronting a swarm name, copy
`services-issuer.pem` and `services-issuer-key.pem` from the store's host
and set both options.
A host fronting a swarm name without both options set still evaluates and deploys, and `swarm-services-cert.service` fails, naming
them. One that names files it can't read fails the same unit, naming the
paths. The old spellings,
`deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are
renames of these two: a config that still sets them evaluates with a warning
naming both paths.
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
same list on every host, populates the role's
`allowed_domains`, so asking for a name nobody configured is a refusal
from the store naming that name — not a certificate quietly issued for
it. A host's own SANs are narrower still: it asks only for the names it
fronts a vhost for (`deploy.hive-controller.tls.localServiceDomains`), so the key on one
gateway can't serve a swarm service that runs behind another.
**The root's public certificate is a file, on every self-signed gateway host:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
swarm name asks for no services leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its own store identity
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
`baoHiveIdentityKeyFile`) for the handshake without logging in with it.
Every hive must hold one: without it, or with either file absent,
`swarm-services-cert.service` fails with a message naming both options.
On the host running
the store it's also at
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
hand a browser, and reading it needs no store login — which matters,
because every store listener but the loopback UI one demands a client
certificate, and authelia gates that one to the `admins` group, not
open to an anonymous browser fetching a trust anchor.
**The granting unit generates the root once, and never again.** It asks
the mount whether it already has an issuer (`bao list pki/issuers`)
before calling `pki/root/generate/internal`, so a rebuild or a reboot
re-asserts the role and the grant without touching the anchor. A root
that changed per boot would invalidate every certificate issued under it
and every browser taught to trust it.
**A daily timer renews the leaf.** The store's role issues it for 720
hours. `swarm-services-cert-renew.timer` runs the same script as
`swarm-services-cert.service` once a day, and that script asks the store
for a new leaf once the current one is past half that window, is
missing a configured name, or doesn't chain to the store's root
(`swarm-services-root.pem`). A rotation re-imports the leaf into the
gateway and reloads nginx. A store that can't answer — still sealed, or
out of reach — fails the unit, which leaves the current leaf in place,
and the next day's run tries again: about fifteen tries before the leaf
lapses. `systemctl
list-timers swarm-services-cert-renew` shows the next run.
## Constraints on the material
No CA private key exists on any hive. The `pki` root's key stays inside the
store's mount (`root/generate/internal`), and each hive's narrowing is its
role's `allowed_domains`, enforced by the store at issue time: a hive's
identity that leaks can get certificates for that hive's names and no
others.
No host holds a CA or its key, except the store's mTLS CA
(`glue-bao-tls.nix`), which signs only the store's server certificate and the
certificates that log in to it. Every host deletes `/var/lib/swarm-ca` and
the `ca.pem`, `ca-previous.pem`, `ca-key.pem`, `ca.srl` and
`.swarm-ca-adopted` in
`<deploy.hive-controller.tls.stateDir>`: `systemd-tmpfiles` removes them at
boot, and on a deploy that changes the host's `tmpfiles.d` rules. The swarm
distributes only certificates.
⚠️ **The deletion is irreversible.** Rolling back to a generation that still
has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a
new one, unrelated to the deleted root. That's harmless: nothing issues
under it.
A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
evaluation, however it's spelled with `//`, `.`, `..` or a trailing `/`. A
symlink to one isn't resolved, so it gets past the check. Those options name
the store's mTLS CA, `/var/lib/swarm-bao-pki/ca.pem` on the host that runs
the store.
## What to hand a peer
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
block a peer operator pastes into their own config. When this hive's
gateway serves a self-signed leaf (detected by the presence of
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints the
secret store's `pki` root itself, read from
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
gateway leaf and the swarm's service names under that root, so it's the one
anchor a peer host needs.
The operator installs the root **once per swarm host**, not once per peer,
so a hive joining later needs no edit on the hives already running. A hive
whose cert already chains to a public CA has nothing to install —
`peer-config` says so instead.
## Distributing the root
The root **certificate** is a runtime file: each hive fetches it from the
store at boot, so nothing can name it at build time. That has a cost worth
naming, because it's subtle and it bites at a distance:
> **Nothing whose trust store nix assembles at build time can reference
> the store's root.** The derivation reads `security.pki.certificateFiles`
> inside itself; the root doesn't exist there.
Two consumers, both reading the root at runtime:
- **Agents fetch it themselves.** `hive-agent-trust-bundle.service` (`nix/agent-modules/hive-ca.nix`) reads `<mount>/ca/pem` from the store with the agent's own store certificate, then writes `/run/hive-agent-trust-bundle/bundle.pem` (system CAs plus the root) and `root.pem` (the root alone). The agent's units point `SSL_CERT_FILE` at the first (Rust, Go and OpenSSL clients) and `NODE_EXTRA_CA_CERTS` at the second (Node and Claude Code), and order after the unit with `Requires=`. The fetch retries inside its start job for up to 14 minutes, with a wait that doubles from 2 to 60 seconds, so a store name that doesn't resolve yet delays those units instead of failing them; after that the job fails and they fail with it. The meta flake names no CA file, and a timer re-reads the root hourly. A consumer reads it at start, so a rotated root reaches an agent unit at its next start.
- **The Matrix container gets a bind mount**, via the shared
`lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
use: the bundle is bind-mounted read-only into the container, and a
oneshot concatenates it with the system CAs before tuwunel starts.
The consumption differs per runtime and is the part worth knowing.
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
— it's not: its rustls-based TLS stack still resolves trust through the
platform certificate store on Linux, and that store honors
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
an OpenSSL-linked binary.
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
> `openssl-probe` uses it _instead of_ the default store, so naming
> just the bundle would drop every public CA and break federation with
> the wider matrix network — a much bigger outage than the one being
> fixed. The same caveat applies to `hive-forge` (Go) for the same
> reason.
A hive in the swarm fetches the root itself. A host outside it gets the
certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)).
One anchor per host, not one per peer: a hive joining later needs no edit
on the hives already running.

103
docs/4-implementation/swarm/ci-metrics.md Normal file
View file

@ -0,0 +1,103 @@
# CI metrics
`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
can show run time and queue wait per repo, workflow and PR, over any range back
to a repo's first run.
## Where it runs
A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
has the poller's store identity, which the host that runs the store has. Run it
on one host per swarm: each poller keeps its own cursor, so two of them push
every run twice.
It reaches the forge, authelia, the store and the collector by name, so it needs
none of them on its host. On a host without the store, its leaf
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
file to copy, as for the CI runner.
With a self-signed gateway, those names serve leaves the swarm's own CA issued.
The poller reads them through `swarm-ci-metrics-ca-bundle.service`, which joins
the system CAs and the host's `trust-bundle.pem` into
`/run/swarm-ci-metrics-ca/trust-bundle.pem` and sets that file as the poller's
`SSL_CERT_FILE` ([CA](ca.md#the-trust-bundle)). Without it, every pass fails
with `invalid peer certificate: UnknownIssuer`.
## What it reads, and as whom
| what | where | identity |
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| its forge token | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass | the same leaf |
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account `ci-metrics`, with that token |
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
swarm-controller creates the `ci-metrics` account and mints its token with the
scope `read:repository`, the way it does for the builder. It also keeps a
`ci-metrics` team in every organisation, with read on the Actions unit of every
repo in it, and the account as its one member. The poller therefore covers:
- every public repo, which needs no grant;
- every private repo an organisation owns, through that team;
- **no** private repo a user owns. The forge grants a user's repo only to a
collaborator, and a collaborator reads every unit, code included.
## What it pushes
Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
forwards them to the metrics store. Each sample carries the run's `stopped`
time as its timestamp, which the store keeps, so a run from last month lands
last month.
| metric | value, in seconds |
| ----------------------------- | ------------------- |
| `hyperhive.ci.run.duration` | `stopped − started` |
| `hyperhive.ci.run.queue_wait` | `started − created` |
Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
them, every run is a series of its own, and the board reads each with
`last_over_time`.
A run becomes no sample when its status is `cancelled` or `skipped`, it has no
start (or the epoch as its start, which the forge writes for a run cancelled
before it started), or it stops no later than it starts. A run still going a day
after its `created` time counts as abandoned. The journal line for each repo counts
these by reason.
## The cursor and the backfill
The poller keeps one file, `cursor.json`, in its state directory
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
poller has pushed or skipped every run, and the run numbers above it already
handled. A run still going holds that number, so a run that finishes after a
newer one still gets pushed.
With no file, the cursor of each repo is at run 0, so the first pass is the
backfill: it walks every run back to the first, a page of 50 a second. For a
repo with 5000 runs that takes a few minutes.
The poller writes the file only after the collector accepted a push, never
before. A crash between the two pushes those runs again on the next pass. The
store then holds the same point twice in the same series, which changes no
panel. Deleting the file re-runs the backfill the same way.
A failed repo leaves its cursor where it was, the pass goes on with the next
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
timer runs it again 5 minutes later.
## The board
The **CI runs** tab of the `forge` board (`nix/host-modules/swarm-grafana/dashboards/forge.json`), with a repo and a workflow
variable:
- mean and 90th percentile run time of successful runs, over time;
- queue wait over time, mean and longest;
- finished runs per day, by outcome;
- a table per PR over the time range: mean and longest successful run time,
runs, failed runs.

133
docs/4-implementation/swarm/inference.md Normal file
View file

@ -0,0 +1,133 @@
# Per-agent inference settings
The swarm can set the model and reasoning effort each agent's turns run on,
plus the models it may start subagents on and how many subagents it may run
at once. A field you set wins over the
agent's own value; a field you leave unset keeps the agent on its own value,
so an agent you set nothing for runs as it does today.
## Setting them
```sh
swarmctl agent inference set iris --model opus --effort high
swarmctl agent inference get iris
swarmctl agent inference clear iris
```
- `set` changes only the flags you pass; the other fields keep what's
stored. A hive must declare the agent.
- `--effort` is one of `low`, `medium`, `high`, `xhigh`, `max`.
- `--subagent-models` takes a comma-separated list. It replaces the
agent's `availableModels` as the models its subagent `start` and
`continue` calls may name, read on each call
([`subagent.md`](../../2-handbook/tools/subagent.md#which-models-a-subagent-may-run-on)).
- You set the subagent cap, `max_subagents`, in the swarm UI's inference
dialog; `set` has no flag for it and keeps the stored one. It's a whole
number from 0 to 32, where 0 allows no subagents; unset means no cap
([`subagent.md`](../../2-handbook/tools/subagent.md#how-many-subagents-may-run-at-once)).
- `get` shows each field, naming the unset ones (or says the swarm sets
nothing for the agent), and then the model and effort the agent reports
running on, each with where it comes from.
- `clear` removes every field; the agent goes back to its own values.
The full flag reference is [`swarmctl-cli.md`](../../2-handbook/reference/swarmctl-cli.md). The
swarm UI's agents page edits the same settings; see [UI](../../2-handbook/swarm/ui.md). The
swarm is the only place to change an agent's model or effort at runtime:
its web UI shows both but has no picker and no `/model` or `/effort`
command.
## What an agent runs on
`get` shows what the agent runs on next to what the swarm sets:
```text
model: opus
effort: (not set; the agent's own)
subagent_models: (not set; the agent's own)
max_subagents: (not set; no cap)
running model: opus (swarm)
running effort: medium (local_file)
```
Each running value names where it comes from:
| source | meaning |
| ------------ | -------------------------------------------------------------------------- |
| `swarm` | the value set here |
| `agent_nix` | `services.hyperhive.agent.model` or `effortLevel` in the agent's agent.nix |
| `local_file` | a `hyperhive-model` or `hyperhive-effort` file in the agent's harness dir |
| `default` | the harness's built-in default (`haiku`, `medium`) |
When the agent starts without a swarm value, its agent.nix model wins over
a `hyperhive-model` file, and a `hyperhive-effort` file wins over its
agent.nix effort. These files come from runtime picks the agent's web UI no
longer offers; the agent still reads them but never writes them, so a
`local_file` value stays until the swarm sets that field or someone deletes
the file.
The agent reports these whenever they change, and again once a minute.
swarm-controller keeps only what it has received since it started, so for up
to a minute after a controller restart `get` shows `running: not reported…`.
## When an agent picks a change up
Each agent reads its settings every 10 seconds and switches on the next
read: its next turn runs on the new model or effort, and its web UI badge
shows it. A turn already running keeps what it started with.
- **Its own value is the one it runs without the swarm**: its agent.nix or a
`hyperhive-model` / `hyperhive-effort` file. Setting a swarm value doesn't
overwrite it.
- **Clearing a field** (or the whole key) puts the agent back on its own
value on the next read.
- **After a restart**, the agent runs its own values until its first read
of its settings succeeds.
<details><summary>Implementation</summary>
The settings live in the `agent-inference` KV bucket, one key per agent,
which swarm-controller alone writes:
```json
{
"model": "opus",
"effort": "high",
"subagent_models": ["haiku", "sonnet"],
"max_subagents": 3
}
```
Every field is optional. The agent runs the same with `{}` as with no key;
`GET` answers `"configured": null` for no key (or a cleared one) and `{}` for
a stored value with no fields. A stored value the controller can't decode,
such as one a newer build wrote, also answers `"configured": null`, with a
warning in its log, until the next `PUT` replaces it.
| method | path | body |
| -------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GET` | `/api/agents/<agent>/inference` | —; answers `{"configured": <the stored value, or null>, "effective": <what the agent reported, or null>}` |
| `PUT` | `/api/agents/<agent>/inference` | the whole value, replacing the stored one; the controller refuses unknown fields, an unknown effort, or a model name with whitespace or a leading `-`, or a `max_subagents` above 32 |
| `DELETE` | `/api/agents/<agent>/inference` | —; answers 204 |
An agent's own queue credential grants it
`$JS.API.DIRECT.GET.KV_agent-inference.$KV.agent-inference.<agent>`, the
direct read of its own key, and nothing else in this bucket — the same shape
as its schedule key ([`schedules.md`](../../2-handbook/swarm/schedules.md)).
The harness (`hive-agent`'s `swarm_inference`) applies a set field to the
in-memory model or effort without writing `hyperhive-model` or
`hyperhive-effort`, which keep the agent's own value.
The subagent daemon (`hive-subagent-mcp`'s `swarm_models`) reads
`subagent_models` and `max_subagents` with the same direct get, under the
same credential, on each `start` or `continue`.
`effective` comes from the agent's turn-state header on
`$SWARM.agent-state.<agent>` ([`README.md`](../../2-handbook/swarm/README.md)), which carries
`model`, `effort`, `model_source` and `effort_source`. swarm-controller
subscribes to every agent's header with the reader grant it already holds for
the state relay and keeps the latest per agent in memory. A header without
those fields, from an agent that doesn't report them, leaves `effective`
`null`.
</details>

317
docs/4-implementation/swarm/notify-router.md Normal file
View file

@ -0,0 +1,317 @@
# Notification router
The swarm's `swarm-notify-router` service turns forge events into todos on the
agents they concern. No hive is in the path, and the agent can be on any hive:
swarm-controller keeps the events, the router decides, and each agent's harness
takes its own todos off the queue.
Where it runs and how it's enabled: [`services.md`](../../2-handbook/swarm/services.md#notification-router).
It runs beside `hive-forge-notify`, which still delivers forge notifications
the way it did. An agent can get the same news from both.
## What an agent gets
| Flow | Event | The agent's todo |
| ------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `assigned` | someone assigns an agent to an issue or PR | `you are assigned to issue <repo>#<n>: <title> (<url>)`, key `forge:<repo>#<n>:assignment` |
| `unassigned` | someone takes an agent off an issue or PR | the same key rewritten: `you are not assigned to issue <repo>#<n> anymore: <title> (<url>)` |
| `closed` | someone closes or merges the issue or PR | the router clears every router todo it queued for it, then queues `closed` on each agent that held one: `<kind> <repo>#<n> was closed: <title> (<url>)`, key `forge:<repo>#<n>:closed` |
The todos are subsystem `router` in the agent's todo list. A todo under the
same key replaces the one before, so an issue gives an agent one assignment
todo however often it changes hands. A new event reopens a todo the agent
marked done.
The `closed` todo has its own key, apart from the assignment todo. A reopened
issue leaves it in place until the next close replaces it; a new assignment
writes the `assignment` key and leaves it alone.
Only agents get todos. An assignee with no agent of that name gets nothing.
An agent assigned before it first reports its status gets its todo on the
issue's next assign event.
Forgejo's assign events carry the assignees after the change, not which one
changed. The router keeps the assignees it saw on the last assign event per
issue and takes the difference. With no earlier assign event for an issue,
every current assignee counts as newly assigned and nobody as removed: an
router notices an unassign only on an issue it has seen an assign event for.
## The path
1. swarm-controller registers two instance-wide hooks, one for the `issues`
and `issue_assign` events at `/webhook/forge/issues` and one for
`pull_request`, `pull_request_assign`, `pull_request_sync` (a push to an
open PR's branch) and the submitted-review events `pull_request_review_approved`,
`pull_request_review_rejected` and `pull_request_review_comment` at `/webhook/forge/pull-request`. At each start it adds
any of these events a registered hook lacks.
2. On each delivery the controller checks the HMAC and publishes the body,
unchanged, to the `SWARM_EVENTS` stream on
`$SWARM.events.forge.<X-Forgejo-Event-Type>`, such as
`$SWARM.events.forge.issue_assign`. The stream keeps a delivery 7 days and
drops a redelivery by its `X-Forgejo-Delivery` id. A delivery the stream did
not keep answers 503, so the forge's delivery log shows it.
3. The router drains its durable consumer, `notify-router`, in order. For each
event it runs every flow, queues the todo ops the flows emit, writes the
issue's snapshot to its `notify-router-state` bucket, then acks the event. A
failed step stops the batch: that event and the rest come back in stream
order after 30 seconds, and the stream drops the ops queued twice by their
message id.
4. Each op goes to the `ROUTER_TODOS` stream on the agent's own subject,
`$SWARM.router.todo.<agent>`. Work-queue retention: an op stays until its
agent acks it, up to 30 days and 1000 per agent.
5. The agent's harness creates its own durable consumer, `todo-<agent>`,
applies each op to its todo list and acks it. An agent that's down, or
whose harness predates this, leaves its ops waiting.
The router holds no forge token and no webhook secret, and registers no hook.
## Flows
A flow is a graph of nodes joined by edges: one `source` node naming the forge
event types it starts from, `match` nodes that pass when every condition
holds, `todo` nodes that emit one todo op per recipient, and `request_reviewer`
nodes that ask for a review. A match node can have more than one child, and
each one runs, so one event can reach more than one leaf. A flow must be a tree:
one source, one parent for every other node, every node reachable, no children
under a `todo` or a `request_reviewer`. The most is 64 nodes, 16
levels deep, and 100 custom flows. A `todo` node sends to the agents an event
field holds (`assignees_added`, `todo_holders`, …) or to one named agent.
The router ships the three flows above, each a source → match on `action` →
todo, and `head-classifier`: source `pull_request` and `pull_request_sync` →
match on `action` (`opened`, `reopened`, `synchronized`) and `draft` = `false`, or on
`undrafted` → `request_reviewer` for `review-classifier` ([below](#requesting-a-review)). The code defines them
(`swarm_wire::notify_flows::builtins`), so a build that changes one reaches
every swarm. Each switches off on the notification router card on `/swarm`. A custom flow has the same type and
the same evaluator, so it can do anything a built-in does.
**The stored document.** swarm-controller keeps the rest in one document, the
`all` key of the `notify-flows` bucket: the ids of the built-in flows you
switched off, and the custom flows. Only the controller writes it, through
`GET` and `PUT /api/notify/flows`. A `PUT` replaces the whole document in one
compare-and-set. The controller answers 400 unless every flow passes the tree
check above and reads only what exists: a source event from `issues`,
`issue_assign`, `pull_request`, `pull_request_assign`, `pull_request_sync` or one of the
three `pull_request_review_*` events, and condition and recipient fields from the event's field list. The same goes for the `{field}`
placeholders in a todo's key and text. A `PUT` must carry the `revision` it read, `0` for no
document, and gets 409 when the stored revision is another. Swarm UI's
[notification router card on `/swarm`](../../2-handbook/swarm/ui.md#notification-flows) shows the document and
switches built-ins.
**The schema.** `GET /api/notify/flows` also returns a `schema` object, built
from the constants the check reads, so an editor offers only what a `PUT`
accepts:
- `source_events`: `[{name, description}]`, the source events.
- `fields`: `[{name, description, values, values_from?, closed}]`, the condition
and recipient fields, and the `{field}` placeholders. `values` lists what the
field holds, as far as the controller knows, as `{name, description}` pairs, such as the `issue` and
`PR` of `kind`.
- `node_kinds`: `[{kind, description, fields}]` for `source`, `match`, `todo`
and `request_reviewer`; each field is
`{name, description, required, values, values_from?, closed}`, and `values`
lists the accepted `{name, description}` pairs (the `upsert` and `clear` of a
todo's `op`), empty for other fields.
- `values_from` on a field says where an editor asks for values that are only
known live: `agents` is the names `GET /api/agents` lists, `repos` the
repositories with an open issue, which `GET /api/repos` lists, so a repository
with none is missing. `closed` says `values` and that source are every value
the field takes, which an editor can show as a dropdown. When false, they
are suggestions: the field still takes any other text, such as an `action` the
forge adds later or the login of a reviewer who is no agent.
- `limits`: `{max_nodes, max_depth, max_flows}`.
A controller that predates `schema` leaves it out of the answer; a client reads
it as optional.
**Requesting a review.** A `request_reviewer` node has one field, `reviewer`,
a forge login. For the PR the event concerns (`repo` and `number`; an event
about an issue sends nothing), the router sends a NATS request on
`$SWARM.router.sink.request-reviewer` with `{repo, number, reviewer}` and
swarm-controller answers with one line:
| Reply | Meaning |
| --------------- | ---------------------------------------------------------------------------------- |
| `requested` | the forge accepted the request |
| `skipped:<why>` | the reviewer's review of the PR's head already stands; the controller sent nothing |
| `error:<msg>` | the forge refused or didn't answer, or the request didn't decode |
The controller asks the forge under its own token, which the router never
holds. It skips a reviewer whose latest review is neither stale (the PR's head
moved past the commit it reviewed) nor dismissed, because requesting a review
from someone who already submitted one makes the forge dismiss it, approval
included. The controller asks again once the review is older than the head.
`hive-forge pr assign-reviewer` applies the same rule (`hive_types::review`).
The router sends the request once, after it queued the event's todo ops and
wrote the snapshot, and logs the answer. A controller that's down or answers
`error:` doesn't hold up the event or any agent's todos, and the request is
not retried: the next push asks again. A request repeated for the same head is
harmless: a reviewer with a pending request is a no-op for the forge, and one
who has reviewed gets `skipped`.
`head-classifier` requests the review classifier's review when someone opens or
reopens a PR and on every push to an open PR that isn't a draft. A draft is a
PR the forge reports with `draft` true, which it does for a title starting
`WIP:` or `[WIP]`. A title edit that drops that prefix (`undrafted` is true:
the event is `edited`, the PR is no draft, and the old title in the payload
started with the prefix) requests the review once, so a PR whose last push
came as a draft gets a classification. The classifier then runs as it does
for any requested review
([`review-classifier.md`](review-classifier.md#trigger-and-lifecycle)), through
its own request queue, which replaces a waiting request for the same PR. A
repo the classifier's account can't read, or a PR that account opened, answers
`error:`.
**Reviews as a source.** A submitted review arrives as a pull request
payload: `pull_request` and `repository` as for any PR event, `action` is
`reviewed`, and `sender` is the reviewer. Approving, requesting changes and
commenting are three event types, so a flow picks the verdict with its source
and reads the reviewer with a condition on `sender`. A flow that asks argus to
review once the classifier has approved:
| node | setting |
| ---------------- | ----------------------------------- |
| source | `pull_request_review_approved` |
| match | `sender` equals `review-classifier` |
| request_reviewer | `reviewer` = `argus` |
The skip rule above applies: the router asks for argus's review once per head.
**What the router runs.** It reads the key at most every 10 seconds, when an
event arrives, and runs the built-ins you left on, then the custom flows that
are on and pass the tree check. It logs a custom flow that fails the check,
leaves it out, and runs the rest.
| the key | the router runs |
| -------------------------------------- | ----------------------- |
| absent, deleted, or its bucket missing | the built-in flows |
| a document it reads | the above |
| a document it can't read | the flows it ran before |
| a read that fails | the flows it ran before |
"Can't read" means not JSON, not a flows document, or a `version` other than 1.
Before the router has read any document, "before" means the built-ins, so a bad
document never leaves it with no flows. The controller refuses to overwrite a
document of a version it doesn't know.
**Deploying in any order.** The controller, the router and the UI can be on
different hosts. A router that predates `request_reviewer` reads a document
holding such a node (a custom flow with one) as one it can't read and keeps the
flows it had; it also has no `head-classifier`. A router that has `head-classifier`
but predates the `draft` and `undrafted` fields requests the review of a draft PR
as before. A router that has `request_reviewer`, with a
controller that predates it, gets no answer to the request and logs that. A controller that predates this leaves the key absent, and the
router runs the built-ins as before. A router that predates this ignores the
document, so the page and the API show a state the router isn't yet running.
## Deliveries and dry runs
**The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG`
stream for each delivery it makes, and the stream keeps the newest 500. A
delivery is a todo op queued for an agent or a review requested from the
controller, whether it went through or not. A row has:
| Field | Holds |
| ------------------------- | ---------------------------------------------------------------------------------- |
| `at` | Unix seconds |
| `flow`, `node` | the flow and the node that delivered |
| `event`, `repo`, `number` | the forge event type and the issue or PR it concerns |
| `kind`, `target` | `todo` and the agent, or `request_reviewer` and the reviewer |
| `ok`, `detail` | whether it went through, and the op and key, the controller's answer, or the error |
A review the controller skipped (`skipped: <why>`) is `ok`: nothing was
wrong. An event the router retries after a failed step writes its rows again,
so a retry shows twice, and the page says so under the table. A row that
takes longer than 2 seconds to write counts as failed: the router logs a
warning and the delivery goes on.
**The journal.** The router also logs each delivery it handles, so
`swarm-logs` answers whether a flow ran for a given PR. The lines carry the
event and sender, never the payload body:
| Message | Level | Fields |
| -------------------------------------------- | ----- | ----------------------------------------------------------------------------------------- |
| `notify router: forge event routed` | info | `event_type`, `action`, `repo`, `number`, `sender`, `delivery_id`, `flows` (at least one) |
| `notify router: forge event matched no flow` | debug | the same fields, with `flows` of 0 |
| `notify router: sink ran` | info | `flow`, `repo`, `number`, `kind`, `target`, `outcome` |
| `notify router: sink failed` | warn | `flow`, `repo`, `number`, `kind`, `target`, `error` |
`flows` counts the flows that produced a todo op or a review request.
`delivery_id` is the forge's delivery id, or `none` when the event carried no
id. A review the controller skipped is a `sink ran` line, as in the log rows.
**The dry run.** The router runs a flows document over the newest 50 events of
the `SWARM_EVENTS` stream and returns, for each flow the document turns on,
how many of those events reached each node. A `match` node counts the events
that passed it. It uses the walk the router delivers with and delivers nothing:
no todo op, no review request, no log row and no snapshot written. It keeps
its own per-issue snapshots, starting empty, so the first assign event it
replays for an issue counts every assignee as newly added. Every recipient
counts as an agent.
**The API.** The router answers two NATS requests from swarm-controller, which
serves them next to `GET /api/notify/flows`:
| Route | Router subject | Answers |
| ---------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------- |
| `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default |
| `GET /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the saved flows document, run over the newest 50 events |
| `POST /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the body's draft flows document, checked as a `PUT` checks it, run over the newest 50 events |
The dry-run request carries a whole document. The `GET` sends the saved one;
the `POST` takes a `PUT` body (it ignores a `revision`), answers 400
with the `PUT` error for a document that fails the check, and forwards a valid one.
[notification router card on `/swarm`](../../2-handbook/swarm/ui.md#notification-flows) shows both: the recent deliveries,
and the counts beside each node.
**Deploying in any order.** A router that predates these subjects doesn't
subscribe to them, and the queue answers the controller's request at once with
no responders. Both routes then answer 503 saying the router doesn't answer
and may be an older build, and the page shows that in place of the table and
the counts. A request the router hears and doesn't answer within 10 seconds is
the same 503. A router that answers with a refusal, or with something the
controller can't read, is 502. A controller that predates the routes has no
route: the page, if newer, shows the failure.
## Credentials and grants
| Principal | Grant |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds |
| swarm-controller | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two
`$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or
`$SWARM.events.forge.*`, so each has one writer, and the controller alone
hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the
controller asks the forge under the token it already holds. The agent's create subject
carries the consumer's filter, and the queue server checks it against the
request, so an agent can't create a consumer that delivers another agent's
todos.
## Observability
The router pushes OTLP metrics through its container's forwarder to the swarm
collector, labelled `service.name=hyperhive-notify-router`. With no
`OTEL_EXPORTER_OTLP_ENDPOINT` it exports nothing. The client-certificate
gauges (`bao.client_cert.*`) come with them.
| Metric | Unit | Kind | Attributes |
| ---------------------------------------------- | ---- | --------- | -------------------------------------------------------------------------------- |
| `hyperhive.notify_router.events` | — | counter | `event_type`; `outcome`: `routed`, `unmatched`, `ignored`, `unusable`, `retried` |
| `hyperhive.notify_router.event.duration` | `s` | histogram | `event_type` |
| `hyperhive.notify_router.todos` | — | counter | `op`: `upsert`, `clear`; `outcome`: `delivered`, `failed` |
| `hyperhive.notify_router.review_requests` | — | counter | `outcome`: `requested`, `skipped`, `failed` |
| `hyperhive.notify_router.pull.failures` | — | counter | — |
| `hyperhive.notify_router.consumer.pending` | — | gauge | events not yet delivered to the router's consumer |
| `hyperhive.notify_router.consumer.ack_pending` | — | gauge | events delivered and not yet acked |
| `hyperhive.notify_router.consumer.redelivered` | — | gauge | events delivered more than once |
`ignored` is an event that names no issue or pull request. `unmatched` is one
no flow matched. `retried` is one the queue or the store failed on: it comes back
and counts again. A growing `consumer.pending` means the router is falling
behind. The router reads the consumer counts at the start of each pull, at most
30 seconds apart.

711
docs/4-implementation/swarm/review-classifier.md Normal file
View file

@ -0,0 +1,711 @@
# Review classifier
The review classifier checks a pull request against short, named yes/no
rules that the repo itself defines, and posts the result as a forge review.
A small model answers one rule at a time, so mundane rule checks come out
the same way on every PR.
It doesn't replace a full review by a human or agent reviewer. A swarm can
request that review only once the classifier has passed the PR, and use a
larger model there. The classifier takes the cheap, repetitive checks off
that reviewer.
## Repo configuration
A repo defines its rules in a file at its root, in YAML as
`.hyperhive/review-classifier.yml` or in JSON as
`.hyperhive/review-classifier.json`. It holds one of the two; with both, the
rules don't load. `swarm-controller` always reads the file from the repo's
**default branch**, never from the PR, so a PR can't edit the rules it's
judged by. It reads it with its own forge token on every load, with no cache,
so a rule merged to the default branch counts from the next load on.
The file holds one key, `classifications`: a list of one to ten entries, each
with exactly these four keys:
- `name`: short and unique in the repo, shown in the result table
- `description`: the yes/no question the model answers about the PR
- `threshold`: the certainty, 0 to 100, a verdict needs to count
- `polarity`: `positive` means a match is the wanted outcome, and
`negative` means a match is a violation
Any other key, at the top or in an entry, is an error: a misspelt key fails
the load instead of vanishing. An empty list, more than ten entries, an empty
name or description, and two entries with one name are errors too. A file
that exists but fails these checks never falls back to a pass: the run posts
no verdict.
A repo whose default branch holds neither file has no rules. When someone
requests the classifier's review on its PR, the account approves at once. The
body reads `no review rules are configured for owner/name; approved without
checks`. The controller makes no model call. The instance has no per-repo
filter yet, so every repo without a rules file gets this approval.
An example:
```yaml
classifications:
- name: datetime
description: All datetimes carry time zone information.
polarity: positive
threshold: 80
- name: no-ads
description: >-
A commit message contains a co-authored-by line or other
advertising for an AI model.
polarity: negative
threshold: 90
- name: bao-first
description: Any added secret reaches its reader through the bao store.
polarity: positive
threshold: 70
- name: docs-updated
description: >-
A change to what an operator or user sees, such as a command, flag,
option, endpoint, service or default, updates the matching page under
docs/ or the crate's README in the same diff. A diff that changes
nothing an operator or user sees makes the question not applicable.
polarity: positive
threshold: 65
```
## Trigger and lifecycle
The classifier has a dedicated forge service account, `review-classifier`
(see [The service account](#the-service-account)). Requesting that account
as a reviewer on a PR starts a run:
1. The forge sends the `pull_request_review_request` delivery to
[`swarm-controller`](../../../swarm-controller/README.md), the swarm's one
webhook owner, on its instance-wide hook `/webhook/forge/review-request`.
A delivery for any other reviewer, or a removed request, does nothing.
2. The controller reads the PR and the repo's rules from the default branch,
and hands the run to the classifier service as one queue request on
`$SWARM.review-classifier.try-out`, the subject a try-out uses.
3. The classifier runs every classification against the PR's commits and
diff.
4. The controller posts the result on the PR as the service account's
review, on the commit it read.
Opening or pushing to a PR starts no run by itself. The notification router's
`head-classifier` flow requests the account's review when someone opens or
reopens a PR and on every push to an open PR, unless its review of the new head
already stands ([`notify-router.md`](notify-router.md#requesting-a-review)). It
requests nothing for a draft PR, one whose title starts with `WIP:` or `[WIP]`
and which the forge reports as `draft`. The edit that takes a PR out of draft
requests the review once, for the head the PR has then. That edit recognises
only the default `WIP:` and `[WIP]` title prefixes: after taking a PR out of
draft that used another prefix, request the review by hand. A request that
reaches the controller is always reviewed, whether or not the PR is a draft.
When a post can't reach the secret store or the forge, because the
connection fails or a try gets no answer within two minutes, the controller
tries it up to six times over about five minutes, enough to outlast a
swarm-bao restart, with the same verdict. It doesn't retry any other failure,
such as the forge refusing the review. A post that still fails ends the run
as `post-failed`, and nothing appears on the PR. The account's review
request is still pending then, and the forge sends no event for a request
that's already standing: remove the service account's review request, then
request it again to start a fresh run.
A verdict belongs to the commit it ran on. A push starts no run. Once the
PR's head moves, requesting the service account as a reviewer again starts a
fresh run on the new head.
Requesting the account again on a head it already reviewed depends on that
head's newest recorded review, as the [run store](#run-store-and-retention)
holds it. When that review approved or requested changes, nothing runs: the
account posts the same verdict and result table again, opening with
"Already reviewed `<sha>`, verdict unchanged." When it had no clean
verdict, or its post failed, the request starts a fresh run. A head with no
recorded review, such as one whose runs have dropped out of the newest 100, or
a run store the controller can't read, gets a fresh run too.
The re-request also retires the account's earlier review. The forge clears
the `official` flag on a reviewer's earlier reviews when it's requested
again, and only official reviews count toward the required approvals or
block a merge. Removing the request before the new review lands restores
the earlier one. A repo whose branch protection also dismisses stale approvals (see
[PR review gate](../../5-project/process/pr-review-gate.md)) stops counting an approval
on an older commit even without a re-request.
How many requested reviews run at once is a swarm-wide setting, two by
default (see [Swarm-level settings](#swarm-level-settings)). The controller
turns no request away. It puts each one in the `review-requests` work-queue
stream on the swarm queue, one subject per PR, and the stream keeps one
message per subject: a newer request for a PR replaces one still waiting.
The controller takes a request once a run place is free and acks it after
writing the run's record. While a run goes, the controller tells the queue
every 20 seconds that the request is still in hand. A request it took and
never acked, because the controller stopped first, comes back within a
minute, and the controller that's running then reviews it. A restart
therefore repeats a run that was going. When it can't queue a request, the
account posts a `**No verdict:**` comment saying so.
When the controller takes a request, it reads the PR first. A request whose
head is no longer the PR's head runs on the PR's current head, and never
re-posts the old head's recorded verdict.
A request for a PR whose review is still going waits in the queue until that
review ends, so a PR never has two runs at once. A request that re-posts an
earlier verdict waits for a run place too, and frees it before it posts.
## Components
### The classifier service
A swarm-level service in its own container, on whichever host the operator
puts it. It talks to `swarm-controller` through async messages only, and
gets its secrets from the swarm's secret store, bao
([`bao.md`](bao.md), [`credentials.md`](../../2-handbook/swarm/credentials.md)).
It looks like an agent from the inside, with three things missing: it has no
full agent identity, no state directory and no tools. Every model session is
ephemeral, and nothing carries over from one run to the next.
On the API key runtime, each classification is one `claude --print` the
service spawns through the `hive-claude` crate directly, with no session to
resume, no tools (`--tools ""`), no MCP server (`--strict-mcp-config` and no
config) and an empty directory of its own. The prompt reaches it on stdin as
one user message in `--input-format stream-json`, so that it can carry content
blocks (see [Shared context](#shared-context)).
The `claude` is the one its host's
`services.hyperhive.deploy.hive-controller.claudeCodePackage` pins for
agents, or its container's own nixpkgs' where the host pins none.
On the ACP runtime, each classification is one prompt on a new `opencode acp`
session. Its config and opencode's state live in a directory of its own,
removed after it. Its working directory is one empty directory that every
classification shares: opencode writes that path into the system message, so
one path keeps the system message the same from one classification to the
next, and the provider's prompt cache can hold. The service drives it through
[`hive-runtime`](../../../hive-runtime/README.md)'s `AcpRuntime`, as the subagent
daemon does a one-run subagent, and stops the agent when the classification
ends. The agent has no tool. opencode runs some tools, `read` among them,
without asking the client, so the service sets opencode's own permission to
deny every tool (`OPENCODE_PERMISSION={"*":"deny"}`), which leaves the model
with no tool offered at all, and refuses any permission request on top. The
config it writes for opencode names one provider and model, from the settings,
and turns off opencode's title call, which would send the prompt a second
time. It also defines the agent the session runs as, whose prompt is the
classifier's system prompt, so opencode sends it as the system message, with
only its environment block (model, directory, platform, date) after it. The
first user message holds the PR alone. That agent has no `permission` of its
own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
there would offer the model tools again.
The service is `swarm-review-classifier`, and its container is the one
[`services.md`](../../2-handbook/swarm/services.md#review-classifier) describes. It holds no forge
credential: `swarm-controller` reads the PR and sends it along.
### Swarm-level settings
The runtime, the model, the effort and how many requested reviews run at once
are one document for the whole swarm. `swarm-controller` keeps it in its
`review-classifier-settings` KV bucket under the key `all`, and no other
service reads it. With no document stored, the classifier runs on the API key
runtime on `haiku` at claude's default effort, two reviews at a time. The
controller ignores a stored field it doesn't know, and a document it still
can't decode counts as none, with a warning in its log, until the next PUT
replaces it.
`GET /api/review-classifier/settings` answers the settings in force:
```json
{
"model": "haiku",
"effort": null,
"max_concurrent_reviews": 2,
"runtime": "api-key",
"acp": null
}
```
- `model`: the model every classification on the API key runtime runs on, as
claude's `--model` takes it: one word with no whitespace, not starting with
`-`
- `effort`: `low`, `medium`, `high`, `xhigh` or `max`; `null` runs at the
runtime's own default. On the ACP runtime it applies only where the agent
offers it
- `max_concurrent_reviews`: 1 to 8, how many requested reviews run at once.
Try-outs don't count toward it. The rest wait in the queue for a place
- `runtime`: `api-key` or `acp` (see [Runtimes](#runtimes))
- `acp`: the provider the ACP runtime runs on, or `null`. It stays stored
while the runtime is `api-key`, so switching back and forth needs no
re-entry. It holds:
- `base_url`: the provider's OpenAI-compatible API root, `https://` only
- `model`: the model as the provider names it, with no whitespace
- `prices`: what the provider charges in USD per million tokens,
`input` and `output`, and optionally `cache_read` and `cache_write`, which
otherwise count as `input`. A price is 0 or more
`PUT /api/review-classifier/settings` takes any of these fields, changes only
those, and answers the whole document it stored. A field left out keeps its
stored value; `null` sets `effort` or `acp` to none. A form that sends the
model, effort and limit alone therefore leaves the runtime as it was. No secret
goes in the document: an `api_key` field is an unknown field.
The controller reads the settings for every run and sends the runtime, model
and effort in the run request, so a change applies from the next run with no
restart. A try-out reads them when it's posted, a review when it gets its
place. Lowering the limit lets the runs already going finish. A run request
with no model runs on `haiku`, and one with no runtime on the API key.
The PUT needs the `Remote-User` header the gateway sets, as the try-out POST
does. Every error is problem+json with the reason in `detail`:
- **400** for a body that isn't JSON or isn't this shape: an unknown field, a
`null` other than for `effort` or `acp`, an effort outside the five levels,
a `max_concurrent_reviews` that isn't a whole number, a runtime other than
the two, or an `acp` without `base_url`, `model` or `prices`. Also **400**
when the document the PUT would store breaks a rule: an empty `model`, one
with whitespace or one starting with `-`, a `max_concurrent_reviews` outside
1 to 8, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
with whitespace, a negative price, or the `acp` runtime with no `acp`
- **401** for a PUT with no `Remote-User`
- **503** when this host has no swarm queue, or it isn't connected
- **500** when the controller can't read or write the settings
When the controller can't read the settings, a try-out answers the same 503
or 500, and a review posts a no-verdict comment. The operator writes each
runtime's key by hand in the secret store
([`services.md`](../../2-handbook/swarm/services.md#review-classifier)).
The per-agent equivalent is [`inference.md`](inference.md), where the
controller writes model and effort to a queue KV bucket
(`swarm-wire/src/agent_inference.rs`) and each agent reads its own
key.
### The service account
A dedicated forge user, separate from swarm-controller's site-admin account
and from every agent's account. It's the reviewer someone requests, it
authors every classifier review, and it pushes the branches the try-out section
creates.
`swarm-controller` creates it as `review-classifier`, with the email
`review-classifier@services.hyperhive.local` and no repo creation, and mints
its one token, `swarm-review-classifier` with scope `write:repository`, into
the store at `swarm/services/review-classifier/forge/token`. A pass at start
and every five minutes re-mints when the token is missing or no longer
matches the forge. The controller refuses an existing account by that name
with any other email. The controller posts the reviews and pushes the try-out
branches with that token, and the classifier service never sees it.
**Giving a repo access is a repo admin's step, by hand.** Nothing grants
it automatically. In the repo's settings, add `review-classifier` as a
collaborator:
- with **read**, so someone can request it as a reviewer, and its reviews show,
but its approval doesn't count
- with **write**, so its approval counts toward the required approvals and
its request-changes blocks the merge where the branch protection blocks on
rejected reviews. Where the branch protection restricts approvals to an
allowlist, the account counts only once it's on that list, and read access
is then enough for reviews.
The try-out's **open a PR** pushes a branch to the repo itself, so it
needs **write** whatever the branch protection says.
The repo also needs a rules file on its default branch (see
[Repo configuration](#repo-configuration)). Without one, a request posts a
comment saying so.
## The classification run
### Shared context
Every classification in a run sees the same context, in the same order:
1. a short system prompt
2. the PR's commits, each with its full message
3. the full diff
Only the last part differs per classification: that classification's
description and the task to classify the PR against it. The identical prefix
lets the runtime's prompt cache reuse it across questions.
The controller builds the diff itself, from each changed file at the PR's
merge base and at its head, with 25 lines of context around each change
rather than the 3 of the forge's `.diff`. An enclosing gate, such as a
`config = lib.mkIf …` or an option's `default`, is in view when it's within
25 lines of the change, so a move that restates it doesn't read as adding one.
A gate further away is still out of view. At review time the controller takes
the merge base from the PR's commits: the one parent that isn't a PR commit.
A rebased or force-pushed PR therefore gets the diff against the base it sits on
now, not the base it started from. The controller sends the forge's `.diff`
instead when the PR changes more than 100 files, when the commits don't name
a single merge base (the head merges the base branch in), when reading a
changed file fails, or when the wide diff is over 256 KiB. The review then says
so, with the reason.
On the API key runtime the commits and diff are one content block of the user
message and the question another, with a cache breakpoint at the end of the
first. The cache writes an entry only where a breakpoint is, and `claude` sets
one only after the last block, so that breakpoint is what lets the next
question read the commits and diff from the cache. The PR stays in the user
message rather than the system prompt, which tells the model to judge it as
data. A shared context shorter than the model's minimum cacheable length is
not cached at all.
No object hash is in that prefix: the header names the repo and the PR number,
not the head commit, each commit's heading gives its position instead of its
sha, and the diff loses each file's `index <hash>..<hash>` line, keeping its
`diff --git` paths and every hunk. A rebase rewrites all of those hashes, so
with them in, a rebased PR with the same messages and hunks would miss the
cache from the first line. The head commit still goes to the controller,
which posts the review against it.
### One question, one answer
The model answers each classification in JSON. The answer holds:
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
- a **certainty** from 0 to 100, how sure the model is of that verdict
- a **reason**; the prompt asks for under 350 characters, and the classifier
truncates one over 500 to 499 plus `…`, keeping the verdict and certainty
An answer:
```json
{
"verdict": "no-match",
"certainty": 92,
"reason": "Two new timestamps use naive datetimes without a zone."
}
```
The verdict field holds `match`, `no-match` or `not-applicable`, and the
certainty field a whole number. The classifier reads the object from the
reply, or from the first code fence in it. When an answer isn't that object,
the classifier asks that classification once more, with a reminder to reply
with the JSON object alone; the other classifications aren't asked again. The
second ask runs on what the first left of the classification's three minutes,
so a retry never lengthens the run, and it doesn't happen when the first ask
leaves less than 20 seconds. The classifier also doesn't ask again after a
runtime error or a timeout. A runtime error, a timeout, or an answer that
still isn't that JSON object after the second ask leaves the classification without a verdict: it's listed with its error, and
doesn't count. A run with no key for its runtime, or in which no
classification got an answer, fails as a whole, and shows its error instead of a
verdict.
### Aggregation into the verdict
The classifier collects the answers into one result, keyed by
classification name, and logs it. A classification counts toward the
overall verdict only if it's applicable and its certainty reaches the
threshold. The table still lists every classification below its threshold or
not applicable, marked as excluded.
A counted classification fails when its verdict lands on the wrong side of
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
failing classification makes the verdict **request changes**. Otherwise the
verdict is **approve**, including when the run counted no classification at
all.
### The review
The service account posts the verdict as a forge review, approve or request
changes, and the table is the review's body. An illustrative table:
| result | check | certainty | reason |
| ----------- | ------------ | --------- | -------------------------------------------- |
| ❌ no match | datetime | 92 / 80 | Two new timestamps use naive datetimes. |
| ✅ no match | no-ads | 99 / 90 | No commit message carries an AI attribution. |
| ❔ unsure | bao-first | 95 / 70 | The PR adds no secret. |
| ❔ match | docs-updated | 41 / 65 | Below threshold. |
**Overall: ❌ request changes**
The result cell is the symbol for what the row did to the review, then the
model's verdict. ✅ is a counted row that passed its polarity and ❌ a counted
row that failed it, so _match_ is ✅ on a positive rule and ❌ on a negative one.
❔ is a row that doesn't count: below its threshold, not applicable (shown as
_unsure_), or without an answer. The certainty cell reads _certainty / threshold_.
The approval is a real one: it counts toward a repo's required approvals
like any other ([`pr-review-gate.md`](../../5-project/process/pr-review-gate.md)). The
Repos that rely on it typically raise their required approval count to
account for it.
A run with no clean verdict never approves. When the PR or the rules don't
load (a repo with no rules file gets an approval instead), the run fails, the classifier
answers a different number of
classifications than the controller sent it, or **any** classification has no answer,
the account posts a **comment** instead, starting with `**No verdict:**` and the reason,
followed by the table when there is one. A comment neither approves nor
blocks the merge: nine timeouts and one pass post a comment, not an
approval.
The controller flattens every cell to one line and backslash-escapes the
characters that could end the cell or start markup, math or raw HTML: `|`, `\`,
backticks, `[`, `]`, `<`, `>`, `&`, `*`, `~`, `$`, and `_` except between two
letters or digits (`ci_metrics.rs` stays as is). Ordinary punctuation such as
`(`, `)` and `.` passes through unescaped. Each `@`, `#`, `!`, `://` and `www.`
gets a zero-width space after it, with no backslash, so a model's `reason`
can't break the table, embed markup, mention anyone, reference an issue or PR,
or carry a link. The prompt asks the model for plain text in `reason`.
## Runtimes
The classifier runs on one of two runtimes, the settings' `runtime`:
- **API key** (`api-key`, the default): the `claude` CLI on the Anthropic API
key at `secret/swarm/services/review-classifier/anthropic-api-key`, at
`model` and `effort`
- **ACP** (`acp`): `opencode` on the provider `acp` names, with the provider
key at `secret/swarm/services/review-classifier/acp-provider`, at
`acp.model`
Model names don't carry across runtimes. A claude alias means nothing to an
ACP provider, so each runtime has a model field of its own.
To switch to ACP, an admin:
1. writes the provider key in the secret store's UI under the OIDC login, at
`secret/swarm/services/review-classifier/acp-provider`, field `api_key`
2. PUTs the runtime and the provider:
```json
{
"runtime": "acp",
"acp": {
"base_url": "https://llm.example/v1",
"model": "Qwen/Qwen3-35B",
"prices": { "input": 0.5, "output": 2 }
}
}
```
The next run goes to that provider. `{"runtime": "api-key"}` switches back and
keeps the provider for next time.
A split deploy runs on the API key: a classifier older than the ACP runtime
ignores `runtime` and runs every classification on `claude`, and a controller
older than it ignores the stored `runtime` and `acp` and sends neither.
A third runtime, **claude subscription** (the `claude` CLI on a Claude
subscription login), comes later.
> **Open point.** The subscription runtime needs a login, which is the one
> piece of durable state the service would keep. A swarm-level claude
> subscription login may have to exist before this runtime works.
## Swarm UI
Two pages in the [swarm UI](../../2-handbook/swarm/ui.md), `admins` only like the rest of it.
### Recent runs
One card per run: the repo, the PR, and how many classifications ran.
Selecting a card expands it to show each classification's result, the same
rows as the review table.
### Try-out
A page for writing classification prompts without touching the repo. Pick a
repo and a PR, write or edit a classification, and run it against that PR;
change the prompt and run it again until it answers the way you want.
Nothing reaches the forge while you iterate. A button the page disables
points at a line that says why, such as the 10-classification cap on **add a
classification**, and a screen reader announces the run's progress once, with
the elapsed seconds outside the announcement.
The page posts the run to `swarm-controller` (`POST
/api/review-classifier/try-out`), which reads the PR with its own forge token
and sends it to the classifier as one queue request on
`$SWARM.review-classifier.try-out`. The page then reads the result back (`GET
/api/review-classifier/try-out/{id}`): a run can take longer than a request
stays open. The controller keeps the result in memory for an hour, and the
finished run goes into the [run store](#run-store-and-retention).
`GET /api/review-classifier/rules?repo=owner/name` returns the repo's own
rules, for the page to load into its editor; a run posts them like typed
ones. The route answers:
- **200** with `path` (the file read), `branch` (the default branch),
`commit` (the commit the route read the file at) and `classifications`, in
the file's order and in the POST's shape
- **400** for a `repo` that's missing or not `owner/name`
- **404** for a repo the forge doesn't know; a default branch with no commit,
in an empty repo or when the default branch doesn't exist; or a default
branch with neither file
- **422** for both files on the default branch, or a file that doesn't parse
or breaks a rule from [Repo configuration](#repo-configuration); the
`detail` names the file and what's wrong
- **503** when this host has no forge token yet, or the forge failed or
didn't answer within two minutes
The page's repo and PR fields offer values to pick from, and stay free text,
since a run also works on a merged or closed PR. Two routes feed them:
- `GET /api/pulls/repos` lists the repos with at least one open PR, as
`owner/name`.
- `GET /api/pulls?repo=owner/name` lists the repo's open PRs, highest number
first, each with `number`, `title` and `head_sha`. It answers **400** for a
`repo` that's missing or not `owner/name`, **404** for a repo the forge
doesn't know, and **503** when this host has no forge token yet, or the
forge failed or didn't answer within two minutes.
Every error is problem+json with the reason in `detail`.
When it's ready, **open a PR** (`POST /api/review-classifier/open-pr`, with
the repo and the classifications) has the service account push a branch that
adds the classifications to the repo's rules, with you, the swarm UI user, as
commit author. The page then links you to the forge's compare page for that
branch, where you log in as usual and open the PR yourself.
- **The branch** is `review-classifier/<you>-<8 hex digits>`, made from the
default branch's head with one commit, through the forge's contents API.
- **The file** is the rules file on the default branch, merged by name: a
classification named like an entry replaces it, a new name goes at the end,
and every other entry stays. A repo with no rules file gets
`.hyperhive/review-classifier.yml`. In a YAML file, the entries the merge
leaves alone keep their text, comments included. The merged file has to
pass the same checks the `rules` route makes.
- **The author** is the forge account named like your swarm UI login. A forge
account made by an SSO login carries the SSO username, so it's yours once
you've logged in to the forge. The commit carries its full name, or the
login when it has none, and its noreply address,
`<login, lowercased>@noreply.<forge domain>`. It's the address the forge
itself uses for an account that keeps its email private, so the commit
links to your account and your email stays out of the repo's history. The
committer is the service account.
- **The token** is the service account's own, read from the store for each
push (see [The service account](#the-service-account)), so the account
needs **write** access to the repo.
The route answers **201** with `branch`, `path` (the file written) and
`compare_url`, or:
- **400** for a body the try-out POST would refuse in its repo or
classifications
- **401** without the `Remote-User` header the gateway sets
- **403** when the forge refuses the push: the service account isn't a
collaborator with write access
- **404** for a repo the forge doesn't know, or one with no commit on its
default branch
- **409** when the forge has no account named like you yet, or the rules
file changed on the default branch during the push
- **422** for a default branch holding both rules files or one the `rules`
route refuses; classifications the file already holds unchanged; or a
merge that breaks a rule, such as more than ten classifications
- **503** when this host has no forge token, the service account's token
isn't minted yet, or the forge failed or didn't answer within two minutes
## Run store and retention
`swarm-controller` records every run when it ends: a try-out when the
classifier's reply arrives, a requested review once the controller has posted
its review or the post has failed, and a request the controller couldn't queue
once its comment is on the forge or the post has failed. The controller records a
re-posted verdict the same way, with the outcome and answer of the run it
repeats and that run's id in its `detail`. A run still going isn't in the list; a
controller restart stops it, and the queue hands its request back.
The newest 100 runs are one JSON array under the key `recent` in the
`review-classifier-runs` KV bucket on the swarm queue, which only the
controller reads or writes. The controller creates the bucket on first use and
adds each run with a compare-and-set write, so the list survives a controller
restart or a move to another host. Recording the 101st run drops the oldest;
there's no time-based retention. The controller leaves a run it can't decode, such
as one a newer build recorded, out of the list with a warning in its log. It
keeps the run in the store as it was and counts it toward the 100. A document
that isn't a list reads as no runs. The controller logs a failed write, and it
changes nothing on the forge.
`GET /api/review-classifier/runs` returns the list, newest first. Like the
try-out GET, the gateway's `admins` rule gates it, and the controller checks no
header. Each run has:
- `id`: a try-out's is the id its POST answered with
- `trigger`: `try-out` or `review`
- `repo` (`owner/name`), `pr`, and `requester`: the swarm UI user who started
the try-out, or the forge user who requested the review
- `head_sha`: the commit the run read, absent when the PR wasn't read, as for
an unreadable PR or a request the controller couldn't queue
- `started_unix` and `finished_unix`: a try-out starts at its POST, a review
once it has a run place and has read the PR
- `outcome`: `approved`, `changes-requested`, `no-verdict` or `post-failed`
for a review, or `queue-full` on a record written before the request queue;
`done` or `failed` for a try-out
- `detail`: the reason, for every outcome but `approved`,
`changes-requested` and `done`. For `post-failed` it's the run's own reason,
when it has one, then why the post failed.
- `rows` and `overall`, as the try-out GET returns them, when the classifier
answered
- `review_url`: the posted review, when the forge names its URL
A run holds no diff, no commit text and no token count or cost. The route
answers 503 when the controller has no swarm queue or can't reach the bucket,
and 500 when it can't read the list, both as problem+json.
Each recorded run is also one `info` event from `swarm-controller` with the
message `review classifier run finished`. The event carries the same fields,
with `rows` as JSON. It reaches VictoriaLogs through the host journal, where
each field arrives as `F_` and the name in capitals (`F_OUTCOME`, `F_REPO`), so
a run whose write failed is still there:
```console
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-controller.service" _msg:="review classifier run finished" | fields _time, F_TRIGGER, F_REPO, F_PR, F_OUTCOME, F_DETAIL, F_REVIEW_URL' --format json
```
Add a filter such as `F_OUTCOME:="post-failed"` before the `|` to narrow it.
## Observability and cost
The swarm UI shows no token counts, cache counts or cost, on any page. On the
claude runtimes, every classifier run starts its `claude` process with
`OTEL_RESOURCE_ATTRIBUTES` naming the repo and the PR, so the cost reads from
the metrics store by repo and PR. The process exports to the OTLP receiver of the
container's own forwarder on loopback, which pushes to the swarm collector as the
classifier's OAuth2 client, on a host with a hive or without one. On a subscription, cost is notional.
On the ACP runtime, the service computes each classification's cost itself,
from the settings' `acp.prices` and the token counts the agent reports for the
prompt, and adds it to the counter `hyperhive.review_classifier.cost.usage`,
labelled `repo`, `pr`, `model` (the provider's) and `currency` (`USD`). It
exports to the same receiver `claude` does. A classification stopped at its
deadline, or whose agent reports no token counts, adds nothing. opencode
reports no cost of its own for a provider without prices in its config, which
is why the prices are a setting.
Each such classification also adds the tokens it used to the counter
`hyperhive.review_classifier.token.usage`, with the same labels bar `currency`
and a `type` of `input`, `output`, `cacheRead` or `cacheCreation`. Those are
the types of `claude_code.token.usage`, which `claude` exports on the claude
runtimes, so the cache hit rate reads the same way on every runtime: `cacheRead`
over the sum of `input`, `cacheRead` and `cacheCreation`.
The controller counts what requests and runs come to as the counters
`hyperhive.review_classifier.requests`, `.verdicts` and `.checks` and the
histogram `.run.duration`, and the service counts each ask it repeats because
an answer failed validation as `hyperhive.review_classifier.ask.retries`. The
[metrics table](../scheduler/observability.md#review-classifier-result-metrics-swarm-controller)
lists their labels, and the `review-classifier` Grafana dashboard charts them.
The classifier logs each run's result to its journal, which ships to
VictoriaLogs like every swarm service container's
([`services.md`](../../2-handbook/swarm/services.md#logs-victorialogs)).
## Still open
These get settled, and filled in here, as the work reaches them:
- **Cache strategy.** Fully independent sessions per classification, or one
warm session and then the rest in parallel. To measure through the cache
token counts before choosing.
- **Oversized diffs.** What happens when even the forge's 3-line diff doesn't fit
the model's context. Lockfiles already collapse to one line each, as in
`hive-forge diff`.
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
cap.
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
goes to a third-party provider. Which repos may use which runtime, and
whether anything gets redacted.
- **Prompt injection.** A PR's diff and commit messages are model input, so
a PR can try to talk the classifier into approving it.
- **Subscription credential.** Where in bao the subscription runtime's login
lives. The service account's forge token is at
`secret/swarm/services/review-classifier/forge/token`.
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
classifier passed a PR before it starts its own review.

259
docs/4-implementation/turn-loop/README.md Normal file
View file

@ -0,0 +1,259 @@
# Turn loop + MCP
How the harness wakes up, what it asks the agent's runtime to do, and what
tools the agent has access to in return.
## Runtimes
Every turn runs through `hive-runtime`, on the runtime
`services.hyperhive.agent.runtime` selects:
- **`claude`** (default) — one `claude --print` process per turn.
→ [claude-invocation.md](claude-invocation.md)
- **`acp`** — any Agent Client Protocol agent, run as one long-lived child
of the harness. → [`hive-runtime/README.md`](../../../hive-runtime/README.md)
Both report the turn in claude's `stream-json` shape, so the loop below is
the same for either.
## The loop
Each agent harness (`hive-agent` — one serve-loop binary for all
agents) runs:
0. Check the pause marker (`<harness>/paused`). While it exists the
loop does nothing but re-stat it every 5 s — no broker poll, no
claude process. Because step 1 is never reached, messages stay
queued and unacked, so a resume drains the backlog instead of
losing it; reminders and todo wakes buffer in their channels. Set
it with `hivectl agent <name> pause` or the dashboard toggle; see
[persistence](../../2-handbook/agent-lifecycle/persistence.md#-harnesspaused-per-agent).
1. Long-poll `Recv` on its socket. The host-side broker
(`broker.rs::recv_blocking_batch`) returns immediately if there's
a pending message, otherwise waits up to 30 s for a broker `Sent`
event for this recipient. After a failed `Recv` (a socket error, or
a broker error, which comes back at once) the loop waits before the
next one: 250 ms, doubling up to 30 s, and no wait again once a
`Recv` succeeds. Todo wakes and local messages don't wait for it.
2. Pop one message. Peek the remaining inbox depth with `Status`.
3. Emit `LiveEvent::TurnStart { from, body, unread }` onto the SSE
bus.
4. Hand the wake prompt to the runtime: on `claude`, spawn
`claude --print` and pipe it over stdin; on `acp`, send it as a
prompt on the agent's session.
5. Stream the turn's `stream-json` lines into the bus as
`LiveEvent::Stream(value)`. Pump stderr as `Note`.
6. Wait for the turn to end and classify its outcome from the
stream + result — success, compaction, rate-limit, auth-failure, stall,
or hard failure. The outcome drives the post-turn action (see
[Turn outcomes](#turn-outcomes)); the session handles compaction
internally (see
[Compaction](claude-invocation.md#compaction)). This page describes
rate-limit and auth-failure detection [below](#failure-detection-and-login-claude).
7. Emit `LiveEvent::TurnEnd { ok, note }`. Sleep `poll_ms` to avoid
tight loops on transient failures.
### Failure detection and login (claude)
- **Rate limit** — a `429` / `rate_limit` marker on stderr, or a parsed
`{"type":"error"}` rate-limit event on stdout (conversation-text
mentions don't count), sets the `rate_limited` sentinel, parks for
`HIVE_RATE_LIMIT_SLEEP_SECS` (default 300), then retries. The UI shows
a `⊘ rate limited` badge while parked.
- **Auth failure (401)** — `drive_turn` retries once (transient
token-refresh races clear on retry); a second `AuthFailed` writes
`{state_dir}/hyperhive-needs-login`, requeues the message, and parks in
`wait_for_login` — the same path as a cold boot with no session. The
operator re-auths via the per-agent web UI; the queued message then
drives the next turn.
- **Login detection** — both boot (`login::has_session`, Online vs
NeedsLogin) and `wait_for_login`'s resume check key off the credential
files in `login::CRED_FILE_NAMES` (the set `/logout` deletes).
`wait_for_login` takes a `since: SystemTime` baseline (the instant of
the 401 that parked it, or `login::NO_PRIOR_FAILURE` at cold boot) and
resumes only once a credential file's mtime postdates it — so stale
credentials already on disk at the 401 don't trigger an instant
false-resume, and a login that lands *before* `wait_for_login` even
starts polling still resumes correctly (baselining on a fixed instant
rather than an entry-time directory snapshot is what closes that race).
Leftover session-history files still don't read as a live session after
a logout + container recreate (`has_session`/`is_cred_file` scope to
`CRED_FILE_NAMES` either way).
## Harness binary shape
Two sibling binaries over one runtime library, all role-agnostic (there
is one role: agent — the privilege boundary lives server-side at the
broker socket (`/run/hive/mcp.sock`), which refuses privileged `Request`
variants regardless of who sends them):
- `hive-agent` — long-running harness loop (the inbox poll +
runtime turn + ack/requeue cycle described above).
- `hive-runtime` — the library every turn runs through, one `Runtime`
impl per runtime (`ClaudeRuntime`, `AcpRuntime`); `hive-subagent-mcp`
drives its nested sessions through it too.
- `hive-agent-mcp` — MCP server for the built-in `hyperhive` surface.
Run with `--http <addr>` as a persistent streamable-HTTP daemon (the
`hive-mcp-http` systemd unit, on `services.hyperhive.agent.mcp.httpPort`, default
8790); claude connects to its URL via `--mcp-config`. HTTP is the sole
transport — no per-turn stdio child (eliminates the re-registration
race).
A small `Surface` trait, with one zero-sized impl, factors
`hive-agent`'s wire types (`hive_core_agent_sock::{Request, Response}` —
one unified enum shared by every agent socket) and its turn
loop, so the loop itself has no per-role branches. See
`hive-agent/src/main.rs`'s module
doc for the trait shape.
### Boot wiring
`serve_main` reads `HIVE_PORT` (default `DEFAULT_WEB_PORT`) +
`HIVE_LABEL` (default `"hive"` for standalone runs; the meta
flake sets it unconditionally for any container-deployed agent;
see `docs/5-project/process/conventions.md::Hive identity` for the env stack),
opens turn-stats sqlite, prepares the on-boot files (see
[claude-invocation](claude-invocation.md#on-boot-files)),
installs claude plugins, spawns `web_ui::serve` + `vacuum::run`,
and either drops into `serve_loop` directly (`Online`) or parks on
the login flow first (`NeedsLogin`). `hive-forge-notify` polls forge
notifications on its own, not this loop — see
[`forge.md`](../../2-handbook/integrations/forge.md).
Boot also opens the todos store and the socket in-container producers
dial. Matrix / bash / forge-notify daemons and the in-process
`disk_watch` todo producer (low state-disk space) are the built-in
producers, but the socket accepts any `subsystem` marker — a
user-configured MCP server can push its own todos the same way. See
[`docs/2-handbook/agent-lifecycle/persistence.md`](../../2-handbook/agent-lifecycle/persistence.md#state-dirs-per-agent) for
what each built-in todo producer watches and how the store + `get_loose_ends`
merge work.
Plugin install failures aren't fatal: the harness logs each at `warn` and skips it.
### Turn outcomes
`turn::TurnOutcome` (`Result<bool, TurnError>` — `Ok(compacted)` on success,
else a `TurnError`) drives the post-claude branch:
| Outcome | Action |
| --- | --- |
| `Ok(_)` (`false` normal / `true` compacted) | `ack_turn` |
| `Err(PromptTooLong)` | `drive_turn` archived the session (the lib already compacted + retried and it still overflowed); requeue inflight so the message redelivers into a fresh session that fits — no status park |
| `Err(RateLimited)` | sleep `HIVE_RATE_LIMIT_SLEEP_SECS` (default 300), requeue inflight, status back to `online` |
| `Err(AuthFailed)` | emit `needs_login_idle` sentinel, requeue inflight, park in `wait_for_login` |
| `Err(SessionNotFound)` | resume + create self-heal both missed ("shouldn't happen"); requeue inflight so the next turn creates fresh — no status park, message not dropped |
| `Err(ApiStall)` | idle watchdog sent claude SIGINT after `HIVE_TURN_IDLE_SECS` (default 600) of output silence and claude exited; sleep `HIVE_STALL_SLEEP_SECS` (default 60), requeue inflight, status back to `online` |
| `Err(AgentStall(note))` | the same, for an ACP agent: after `HIVE_TURN_IDLE_SECS` with no `session/update` the harness sent the agent `session/cancel`, and SIGINT if it ignored that; `note` says which |
| `Err(Failed(err))` | the harness logs it at `warn` as `turn failed`, the line the Grafana alert rule `hh-agent-turn-failed` (`nix/host-modules/swarm-grafana.nix`) fires on |
`ApiStall` catches an Anthropic API stall — a multi-retry connection storm where
the stream goes silent for minutes. On `claude` the idle watchdog lives in
`hive-claude`'s driver (`Config::idle_timeout`, enforced around `child.wait()`): the timer
resets on every stdout line, so a large but still-streaming turn is never cut —
only complete output silence for the window trips it. The harness sets the
window from `HIVE_TURN_IDLE_SECS` (`0` disables) and maps the driver's
`Error::IdleTimeout` onto `TurnError::ApiStall`. The watchdog sends claude's
process group SIGINT, so claude stops the commands it started, and then
waits for claude to exit. Nothing escalates to SIGKILL: a claude that
ignores the SIGINT keeps the turn open until someone kills it or the unit
stops.
`hive-runtime` applies the same window to an ACP agent's turns. That's
also how a provider HTTP 429 ends an ACP turn when the agent retries it without
reporting it: the turn goes silent until the watchdog stops it.
After the outcome handler, the stats sink records a row. `handle_turn`
reports the result to `serve_loop` via `TurnControl { auth_failed }` —
on auth failure the loop parks in `wait_for_login`; otherwise it loops
straight back to the idle wait (step 1). No same-turn
self-continue mechanism exists: every multi-step continuation rides an
external wake instead — a new inbox message, a `remind`, or an
in-container todo wake (bash-task completion, forge notification,
matrix activity). Ending the turn and letting one of
those drive the next one is strictly better than parking in-process:
it checkpoints the session and observes wakes that only reach the
harness between turns.
## Harness systemd unit shape
One harness serve binary (`hive-agent`, with its `hive-agent-mcp`
sibling), one shared `nix/agent-modules/` tree, one service unit
(`systemd.services.hive-agent`) for all agents. Privilege
differences live in the agent's swarm permissions and the broker socket (which tools
and manager-surface calls each agent receives).
`nix/templates/agent.nix` is the one entry point every agent builds from, and
it imports the shared `nix/agent-modules/`.
`services.hyperhive.agent.docs.enable` defaults to `false` for every agent,
ruth included; an agent opts in from its own `agent.nix` (see
[config.md](config.md), "Reference docs").
### Environment variables set on the unit
- `HOME = /home/<userName>` — systemd defaults `HOME` to `/` for
services without `User=` set; with the per-agent user the harness
needs the right home so claude finds its bind-mounted `~/.claude/`
session dir.
- `HIVE_STATIC_DIR = <mergedDist>` — `tower_http::ServeDir` root for
the per-agent web UI; merged dist = agent default + every
`services.hyperhive.agent.frontend.extraFiles` overlay.
- `HIVE_ASSETS_DIR = "${config.services.hyperhive.agent.packages.assets}/share/hyperhive"`
(`nix/agent-modules/agent-service.nix:480`; `packages.assets` resolves
to the `hyperhive-assets` derivation) — set directly on the unit,
**not** via `environment.variables`, because the latter only
populates `/etc/profile` which systemd services don't inherit.
### `PATH` setup (the wrapper-dir trick)
```nix
path = [ "/run/wrappers" "/run/current-system/sw" ];
```
<!-- vale write-good.Passive = NO -->
`/run/wrappers` (not `/run/wrappers/bin`) comes first so setuid
wrappers — notably `sudo` — resolve before bare nix-store binaries; see
[`docs/5-project/process/gotchas.md`](../../5-project/process/gotchas.md) ("`systemd.services.*.path` appends
`/bin` to every entry") for why the trailing `/bin` matters in
general. It's load-bearing here because the harness runs as the
per-agent user: without the wrapper dir on `PATH`, `sudo` resolves to
the non-setuid nix-store binary and every
`services.hyperhive.agent.user.passwordlessSudo` grant fails with "must be owned by
uid 0 and have the setuid bit set."
<!-- vale write-good.Passive = YES -->
### `serviceConfig` highlights
- `ExecStart = "${config.services.hyperhive.agent.packages.hive-agent}/bin/${binary}"`
(`nix/agent-modules/agent-service.nix:535`); `packages.hive-agent` is
wired to the flake's `hyperhive.packages.<system>.hive-agent` output
(`nix/agent-modules/packages.nix:8-22`) — same binary for every agent.
- `RuntimeDirectory = "hive-config"` → `/run/hive-config/` owned by
`User=`, autocleared on stop. The harness writes regenerated
`claude-{mcp-config,settings,system-prompt}` files there
(`paths::config_dir`). Deliberately separate from `/run/hive`, which
the host bind-mounts in root-owned and which holds hive-c0re's
`mcp.sock`.
- `User = Group = userName` — drops root inside the container; sudo is
the explicit escalation surface (`services.hyperhive.agent.user.passwordlessSudo`).
## Sub-pages
The rest lives alongside this page, in three topic files:
- **[claude-invocation.md](claude-invocation.md)** — how the harness
spawns `claude --print` each turn, the two-pronged compaction (reactive +
proactive), and the on-boot files it materialises (`--mcp-config`,
`--system-prompt-file`).
- **[config.md](config.md)** — the optional per-agent knobs the meta
flake wires in (reference docs, icon, passwordless sudo, dashboard links,
custom static files, connectivity overrides, claude plugins, cargo message
filtering).
- **[mcp.md](mcp.md)** — the MCP tool surface claude sees: core tools,
permission-gated tools, self-wake, authoritative state, the tool envelope,
and the built-in tool allowlist.
Per-subsystem impl detail lives in each module's `//!` doc-comment; these pages
describe present-state behaviour + wiring, not line-level mechanics.

317
docs/4-implementation/turn-loop/claude-invocation.md Normal file
View file

@ -0,0 +1,317 @@
# The claude invocation
How a turn runs on the `claude` runtime, the default
(`services.hyperhive.agent.runtime`). An `acp` agent's turns:
→ [`hive-runtime/README.md`](../../../hive-runtime/README.md).
```
claude --print --verbose --output-format stream-json --model <name> \
--effort <level> --resume <title> # or --name <title> on first use \
--system-prompt-file /run/hive-config/claude-system-prompt.md \
--mcp-config /run/hive-config/claude-mcp-config.json --strict-mcp-config \
--tools <builtins> --allowedTools <builtins+mcp>
# wake prompt piped over stdin
```
**Crate split.** `hive-agent` runs every turn through `hive-runtime`,
whose `ClaudeRuntime` is a pass-through to the external **`hive-claude`**
crate (a crates.io dependency, not in this repo). `hive-claude` holds the
generic subprocess mechanics — spawning `claude --print`, streaming +
classifying stream-json, session lookup/archive, and the durable-session
compaction loop (`hive_claude::{Claude, InfiniteSession, Attach,
CompactionPolicy, PercentPolicy, Telemetry, Sink, SessionStore}`).
`hive-agent`'s `turn` module is the
hyperhive **policy layer** on top: it builds the per-turn config from
the bus, bridges the output stream onto the event bus (`BusSink`), and
owns the compaction / autoreset / retry decisions in `drive_turn`. The
lib returns everything it parsed from a turn (usage, cost, context
window, resolved model) as `Telemetry`, which the policy layer applies
to the bus.
**Which `claude` binary.** The bare name `claude`, resolved off the
harness unit's PATH. By default that's the `claude-code` in the agent's
own nixpkgs (the meta flake's `nixpkgs` input) via
`environment.systemPackages`. Since that's typically a release channel and
this package moves fast, the operator can pin one per hive with
`services.hyperhive.deploy.hive-controller.claudeCodePackage`. `hive-priv`
writes its store path into the container's `/etc/hyperhive-hive-facts` as
`HYPERHIVE_CLAUDE_CODE_PATH` before every start. `claude` on PATH is a wrapper
that runs that build when the path exists, and the container's own
`claude-code` when the variable is unset or the path no longer exists, saying
so on stderr in the second case. A bump reaches an agent at its next container
start, not at a rebuild. See docs/5-project/process/gotchas.md::`claude-code` is unfree.
Hive-enforced settings ship at `/etc/claude-code/managed-settings.json`
(claude-code's canonical managed-settings path — precedence #1,
read-only, un-overridable), wired in `nix/agent-modules/claude-settings.nix`
from the `prompts/claude-settings.json` asset. `effortLevel` is
deliberately not in that file — the `--effort` flag
(`HIVE_DEFAULT_EFFORT` / the swarm's effort setting) controls effort
live, which managed scope would otherwise lock.
The harness reads `<name>` from `Bus::model()` on each turn.
`services.hyperhive.agent.model` in the agent's `agent.nix` sets the initial
default (NixOS option; propagates via `HIVE_DEFAULT_MODEL` env var; falls
back to an existing `/harness/hyperhive-model` file, then `"haiku"`, if
unset). The swarm changes the model at runtime, not the web
terminal: a model or effort the swarm sets for the agent replaces the
`Bus` value from the next turn, and the harness doesn't persist it;
clearing it goes back to the local value →
[`swarm/inference.md`](../swarm/inference.md).
`harness_state::context_window_tokens(model)` looks up
context-window size per-model. Resolution order (first
match wins):
1. The harness's per-family defaults, matched when the family name
is a case-insensitive substring of the active model name:
haiku=200k, sonnet=1M, opus=1M.
2. `HIVE_CONTEXT_WINDOW_TOKENS` — single global override for any
other model (useful in dev / test).
3. Hard fallback: `200_000`.
Once a turn completes, the window the model reports
(`modelUsage.*.contextWindow`) replaces this value.
The effective window drives watermarks; the harness exposes it at
runtime via `/api/state.context_window_tokens` so the UI can show a
percentage-of-window ctx badge.
**Session identity — a constant title.** Every turn keys on one fixed,
harness-owned session title (`turn::session_title()`, default
`hive-session`, override `HIVE_SESSION_TITLE`). The durable
`hive_claude::InfiniteSession` (built once by the serve loop via
`turn::make_session`, then reused) `--resume <title>`s it; the _first_ use
(bootstrap, post-archive, post-purge) misses and the session re-runs the
same prompt once with `--name <title>` to mint it. That single self-heal
rule is the whole identity system — there is **no** scraped session-id
file. Because the
title is constant, `/compact` and its post-compact retry provably target
the same session (killing the old "compact ran on a different/empty
session" bug), and a `choom` invocation in the same cwd can't hijack the
context (it won't carry that title). claude stores sessions in
`~/.claude/projects/<cwd-slug>/<uuid>.jsonl` (bind-mounted persistently);
`--name` writes the title into the file as a `custom-title` event, which
is what `--resume <title>` resolves against. The harness never passes
bare `--continue` (it resumes the _latest_ session in the cwd — the hijack
vector). The managed settings at `/etc/claude-code/managed-settings.json`
disable automemory and dynamic workflows (the `/workflows` feature):
`disableWorkflows` keeps the `/workflows` machinery from spawning sub-runs
that burn usage on the harness's autonomous turns. Claude's own native
autocompact is **on**, at the widest window claude-code allows — see
[Compaction](#compaction) below for why (it's a safety net, not the
primary mechanism; hyperhive still owns that). `outputStyle` is fleet-wide
**Concise** — output tokens are real spend on an always-on agent, and the
preamble/recap Concise drops carries no information a turn-loop consumer
needs.
**Session reset** is available via `POST /api/new-session` (the state
badge's `new session` menu row). It does _not_ touch the session inline —
that would race a mid-write claude process. Instead `Bus::request_session_reset()`
sets a one-shot flag consumed at the next turn boundary by `drive_turn`,
which **archives** the current session: it renames the backing
`<uuid>.jsonl` to `<uuid>.jsonl.archived` (dropped out of claude's `*.jsonl`
resolution glob, history preserved on disk, only the file carrying _our_
title — it leaves any `choom` session sharing the cwd alone). The next
turn's `--resume <title>` then misses and self-heals into a fresh session.
## Compaction
hyperhive owns compaction as policy: the `hive_claude::InfiniteSession`
keeps the session alive across the context window with two triggers baked
into its `run`:
<!-- vale write-good.Passive = NO -->
- **Reactive** — claude-code prints `Prompt is too long`. The session is
_already_ past the window, so no turn can run on it — the session
`/compact`s straight away and retries the same wake-up prompt once. No
notes-checkpoint turn is possible here: the detail is gone. If the retry
_still_ overflows, `run` surfaces `Error::PromptTooLong`; `drive_turn`
then archives the session (session lifecycle stays hive-side) and the
serve loop requeues the message so it redelivers into a fresh session
(see [Turn outcomes](README.md#turn-outcomes) — the wake prompt itself is small, so
the overflow was the accumulated context the archive clears).
- **Proactive** — a turn finishes cleanly but the last inference's context
size crossed the policy watermark. While the session is still healthy it
runs one synthetic _notes-checkpoint_ turn (`CHECKPOINT_PROMPT` —
"context is filling up, flush durable state into `/state` now") and
_then_ `/compact`s, so the agent can persist in-flight state before the
detail collapses into a summary.
<!-- vale write-good.Passive = YES -->
**hive-agent checks both triggers above between its own turns** —
`PercentPolicy::should_compact` reads the percent watermark from the
last completed turn's telemetry (checked after `attempt()` returns), and
the reactive path only fires once claude-code has already refused a whole
turn as too long. Neither can see context growth happening _inside_ a
single long turn's own tool-calling before it returns — a turn whose tool
calls alone blow past the window hits the reactive path with no preceding
checkpoint, losing whatever wasn't already persisted (a known, root-caused gap).
**Safety net for that gap: claude-code's own native autocompact is on**,
via the managed settings (`"autoCompactEnabled": true`,
`"autoCompactWindow": 1000000` — the CLI's documented max, so it fires only
right before the real hard limit, not as an earlier watermark). This is
deliberately _not_ the primary mechanism — it runs with no checkpoint and
none of the `CHECKPOINT_PROMPT` notes-flush machinery above, so a session
saved by it still loses whatever wasn't already durable — but it can act
_mid-turn_, underneath `InfiniteSession::run`, which the two triggers above
structurally can't. Don't confuse this with the `services.hyperhive.agent.autoCompact`
nix option below: that one is a per-agent on/off for hyperhive's _own_
proactive watermark and has no effect on claude's native mechanism, which
is fleet-wide and set once in `hive-agent/prompts/claude-settings.json`.
The **when** for hyperhive's own policy is a `hive_claude::CompactionPolicy` injected by the harness:
`turn::make_session` builds a `PercentPolicy` that compacts once the
model-reported context fill reaches `HIVE_COMPACT_WATERMARK_PERCENT`
(default **75%**), falling back to `harness_state::context_window_tokens(model)`
for the window on turns the model didn't report one. `0` disables proactive
compaction (the reactive path always applies). The proactive path is
best-effort — a failed checkpoint or `/compact` never fails the turn that
already succeeded. An ACP agent gets the same policy; how it compacts is in
[`hive-runtime/README.md`](../../../hive-runtime/README.md#acp-backend-compaction).
The operator can force a compaction any time via `POST /api/compact`. It's
**deferred**: the handler sets `Bus::request_compact()` and returns
immediately; the harness runs the `/compact` at the next turn boundary
(end of the in-flight turn in `drive_turn`, or — when the agent is idle —
in `turn::run_pending_compact` on the serve loop's next empty poll). This
lets `/api/compact` work mid-turn instead of only when idle, without racing
a live claude process.
To disable proactive compaction for a specific agent, use the nix option:
```nix
services.hyperhive.agent.autoCompact = false; # default true
```
Setting `autoCompact = false` sets `HIVE_COMPACT_WATERMARK_TOKENS=0`, which
the percent resolver still honours as a disable. Useful for large-context
models (sonnet/opus) where the 75% heuristic fires before the session is
actually full — the reactive path (compact-on-overflow at the hard limit)
still applies.
- **Auto session-reset** — a third path (`turn::maybe_auto_reset`,
pre-turn) that fires when both conditions hold: context is ≥ a watermark
(`HIVE_AUTO_RESET_WATERMARK_TOKENS`, default **50% of
`context_window_tokens(model)`**) AND the time since the last turn
exceeds the assumed prompt-cache TTL (`HIVE_CACHE_TTL_SECS`, default
`3600`). Claude's prompt cache goes cold after a while; once it's cold,
`--resume`-ing a large session pays the full re-upload cost with no
benefit over starting fresh. `drive_turn` **archives** the current
session (same mechanism as the operator reset — rename `<uuid>.jsonl` →
`.archived`) so the next turn's `--resume <title>` misses and starts
fresh. Unlike proactive compaction, `drive_turn` drops the session
entirely rather than compacting it — and _no_ preceding checkpoint turn runs, because any turn
before the reset would just re-warm the cache and defeat the purpose.
Set `HIVE_AUTO_RESET_WATERMARK_TOKENS=0` to disable. Autoreset and the
operator reset are mutually exclusive per turn (both archive → fresh
turn), so an explicit operator reset short-circuits the heuristic.
The child runs with `cwd = /state` (when the bind exists; falls
back to the parent's cwd in dev), so any relative path in a tool
call (`Read foo.md`, `Bash ls`, `Write notes.md`) lands in the
agent's durable bind-mounted dir. AGENTS.md autoload walks
upward from `/state` — drop a per-agent AGENTS.md there if you
want long-term hints that survive destroy/recreate. A `CLAUDE.md`
in `/state` or above it shadows `AGENTS.md`: claude then loads only
the `CLAUDE.md` (see `docs/2-handbook/agent-lifecycle/persistence.md` for the
boot-time rename).
The wake prompt is intentionally minimal: the popped message's
`from`/`body`, prefixed with a `[msg #<id>]` broker-row-id marker
(so the agent knows what id to pass to `ack_until` for bulk triage;
transient pings show no marker because their sentinel id 0 has nothing
to ack), plus an inline `({unread} more pending — drain via …)` hint
when `unread > 0`. Claude drives any further `recv`/`send` itself via
the embedded MCP server.
`hive-agent` self-announces once on every boot:
`boot_announce::announce` pushes a one-line "you were just
(re)started, check /state/ for your notes, your session is intact"
todo the moment the in-agent todo store opens, before the serve loop
starts. This replaced `hive-c0re`'s old `Coordinator::kick_agent`
calls at the tail of the rebuild and start DAG nodes — those fired a
broker message per DAG node, so one rebuild (always a real
`StopForUpdate` → `Swap` → `RebuildBookkeeping` → `Reconcile` →
`Start` cycle) queued two separate restart messages for a single
actual boot. The harness itself is the one side that knows for
certain whether a boot happened, so it dedups for free: exactly one
todo per real process start, surfaced the normal todo-wake way
rather than as an inbox message.
## On-boot files
`hive-agent`'s `turn` module writes two files into its own per-service
runtime dir, `/run/hive-config/` (`RuntimeDirectory = "hive-config"`,
deliberately separate from `/run/hive/`, the host-owned dir holding the
per-agent socket — so the harness owns this write surface and nothing
needs to `chown` a bind mount), once at startup:
- `claude-mcp-config.json` — points claude at the persistent
`hive-mcp-http` daemon (`http://127.0.0.1:{port}/mcp`, port from
`services.hyperhive.agent.mcp.httpPort`, default 8790) for the built-in `hyperhive`
surface. HTTP is the sole transport for it — no per-turn stdio child,
so the URL survives the per-turn claude re-spawn (no re-registration
race), trading that for a hard dependency on the daemon's uptime
(`Restart=always`, no stdio fallback). Extra servers stay stdio.
- `claude-system-prompt.md` — rendered from
`hive-agent/prompts/system.md` by `hive-agent`'s `prompt::render`:
HTML-comment markers (`<!-- role:agent -->...<!-- /role:agent -->`,
same for `role:manager`) gate the role-specific blocks; both flavors
share everything else. `<!-- config:repo -->` and
`<!-- config:archetype -->` blocks gate the config-repo paragraphs: an
agent with `HIVE_ARCHETYPE` set (the meta flake sets it on an agent built
from an [archetype](../../2-handbook/swarm/archetypes.md)) gets the archetype block and
every other agent the repo block, with `{archetype}` filled in with the
archetype's name. Five placeholders are then
substituted: `{label}` (short agent name), `{qualified_label}`
(hive-qualified `name@domain` form),
`{hive_identity}` (for example `` on hive `pr1ma` ``; empty when
`HYPERHIVE_HIVE_NAME` is unset), `{swarm_identity}` (same
shape for the swarm), and `{operators}`: a paragraph listing the
swarm's operators with the pronouns from each one's forge profile,
for example `Operators: mara (she/her), sol.`. The harness reads
them with one direct get on the one key of the `operator-profiles`
KV bucket, which `swarm-controller` writes. The harness waits up to five seconds for
them. When the bucket is empty or doesn't answer in time,
`{operators}` is empty and the prompt names no operator. Either way
the template tells agents to say "the operator" in third person. A
pronoun change reaches an agent at its next harness start, since
the prompt renders once per start.
When you enable `services.hyperhive.agent.docs.enable`, `HIVE_DOCS_DIR` is present
in the environment and `render()` appends a one-sentence pointer
telling the agent where the docs live.
Passed via `--system-prompt-file`.
**Marker grammar.** `<!-- KIND:X -->` opens a block; any
`<!-- /KIND:X -->` closes the current block of that kind. The renderer
filters the two kinds one after the other. The renderer always uses
role `agent`, so it elides blocks with other role tags, and config `repo`
or `archetype` as above. Nesting **isn't**
supported — a stray opener with no closer runs until end of file.
The renderer tolerates whitespace inside markers (`<!--role:foo-->` parses the
same as `<!-- role:foo -->`). Content outside any marker is always
included. Today's `system.md` carries no role markers (single agent
role), and the role grammar stays wired for a future manager / multi-role
prompt; it carries one pair of config blocks.
**`hive_identity` / `swarm_identity` shape.** Each carries a
leading space + backticked name (` on hive \`pr1ma\``,
` in swarm \`constellat1on\``) when the corresponding env var
is set, otherwise empty string. The independence lets the
template drop one or both into the opener prose without
breaking single-hive deployments that never set the option;
the renderer also treats `Some("")`from a caller as`None` so
empty-string env vars and missing env vars round-trip the
same way.
The per-turn plumbing described on this page — on-boot files, session
identity, the reset/autoreset/retry state machine, and the
telemetry-to-bus bridge — lives in `hive-agent`'s `turn` module; see its
`//!` doc comment (`hive-agent/src/turn.rs`) for the exact call shape.
The actual claude spawn, stream classification, and the
reactive/proactive compaction loop are in the `hive-claude` crate, reached
through `hive-runtime`'s `ClaudeRuntime`.
Login-wait lives in `hive-agent`'s `login` module.

348
docs/4-implementation/turn-loop/config.md Normal file
View file

@ -0,0 +1,348 @@
# Agent config knobs
Optional per-agent knobs, set under `services.hyperhive.agent.*` in the
agent's own `agent.nix` and read at boot or per turn by the harness.
Absent means the default. (The claude spawn + compaction themselves live in
[claude-invocation](claude-invocation.md).)
## Reference docs (`services.hyperhive.agent.docs.enable`)
```nix
services.hyperhive.agent.docs.enable = true; # default: false
```
Makes the hyperhive `docs/` tree available inside the container at a nix
store path read from `$HIVE_DOCS_DIR`, and injects a single pointer
sentence into the agent's system prompt so it knows the docs exist and
where to find them. `claude --add-dir` serves the tree so the full
markdown is readable during every turn.
Off by default for every agent, ruth included. An agent opts in by adding the
line above to its `agent.nix`.
The docs come from the pinned `hyperhive` input (the `reference-docs` package
built from its `docs/` tree), so they match the hyperhive rev the host runs.
## Agent icon
```nix
services.hyperhive.agent.icon = ./icon.svg; # default: null (falls back to shared hyperhive logo)
```
Path to an SVG file used as this agent's visual identity — shown in
the per-agent page header, as the page favicon, and uploaded to the
agent's Forgejo profile avatar (via the `forge-avatar-sync` boot
unit) and Matrix profile avatar (set by `hive-matrix-daemon` over its
live Client). Commit the SVG next to `agent.nix` in the config repo
and reference it as a relative path.
When `null` (the default), the agent falls back to the shared
hyperhive branding mark. The harness serves whichever icon is active
at `GET /icon` on the per-agent web port.
## `user.passwordlessSudo`
```nix
services.hyperhive.agent.user.passwordlessSudo = true; # default
```
Grants the per-agent unix user passwordless `sudo` (`NOPASSWD: ALL`).
Enabled by default so claude's shell tools work for operations that
need root inside the container (`systemctl`, package managers in dev
shells, etc.) — the agent user gets root explicitly via `sudo` rather
than running as root itself.
Set to `false` for agents that should be strictly unprivileged.
Any tool invocation that needs root then fails loudly with the standard
sudo rejection rather than silently succeeding — easier to audit.
`services.hyperhive.agent.user.name` is the unix user the harness and
its co-process daemons run as inside the container (default `"agent"`
for a standalone evaluation; the meta flake rebinds it to the agent's
own name). `services.hyperhive.agent.user.uid` / `.gid` are optional
fixed UID/GID (`null` default lets NixOS assign one automatically from
the normal-user range) — set them only when something outside the
container needs a stable numeric id across a full destroy + recreate.
## Dashboard links
```nix
services.hyperhive.agent.dashboardLinks = [
{ label = "Stats"; icon = "📊"; url = "http://localhost:9001/stats"; }
{ label = "Scratchpad"; url = "http://localhost:8080"; }
];
```
Declares extra navigation links that appear in the per-agent page
header alongside the built-in forge / config / container links. Each
entry has:
| Field | Required | Description |
| ------- | -------- | ----------------------------------------------------------------------------------------- |
| `label` | yes | Display text shown in the icon strip tooltip and meta-nav. |
| `url` | yes | Absolute URL — may include a different port (the dashboard renders it as a plain anchor). |
| `icon` | no | Emoji or short glyph prefix. Defaults to empty string. |
A one-shot systemd unit writes the list to
`<state>/hyperhive-dashboard-links.json` at container boot. The harness's own web UI
(`agent_links` in `hive-agent/src/web_ui/state.rs`) reads the file on
each `/api/state` snapshot and appends the entries to the per-agent
nav as `kind = External` links — no `hive-c0re` / operator-dashboard
involvement, and no code change needed to pick up a new entry.
Omitting the option (default empty) produces no extra links.
## Custom static files
```nix
services.hyperhive.agent.frontend.extraFiles = {
"games/bitburner" = {
source = ./bitburner-dist; # path relative to agent.nix
# target defaults to attribute name: "games/bitburner"
};
"my-page" = {
source = ./my-page.html;
target = "my-page.html"; # explicit override
};
};
```
Layers additional files over the default per-agent web UI dist. Each
attribute defines one overlay entry:
- **`source`** — a Nix path (file or directory) copied into the merged
static tree. Evaluated at nix build time; the resulting derivation is
pointed at by `HIVE_STATIC_DIR`.
- **`target`** — destination path within the merged tree, used as both
the served URL prefix (`/<target>/…`) and the on-disk layout.
Defaults to the attribute name. Forward slashes create nested layouts
(`"games/bitburner"` serves at `/games/bitburner/…`).
Constraints: `target` must start with an alphanumeric or `_` and
contain only alphanumerics, `_`, `.`, `/`, `-`. A config assertion
rejects `..` segments. The merge step refuses to overwrite
files already present in the default dist — pick a target name that
doesn't collide with existing paths (`static/`, `index.html`, etc.).
The default dist ships at `services.hyperhive.agent.frontend.dist` (the
`hyperhive-frontend` package output, read-only). To replace the
entire UI rather than layer on top, override `frontend.dist` directly.
## Connectivity overrides
Two `services.hyperhive.agent.forge.*` / `services.hyperhive.agent.matrix.*` options override where
the per-agent daemons connect. Both rarely need changing on a standard
single-host deploy, but are useful for multi-hive or custom-network
setups.
```nix
services.hyperhive.agent.forge.url = "http://forge.example:3000"; # default: null
services.hyperhive.agent.matrix.url = "https://matrix.example"; # default: null
```
**`services.hyperhive.agent.forge.url`** — base URL of the Forgejo instance. It scopes
the git credential helper to this forge and is where `forge-avatar-sync`
uploads the agent's icon. Override when the agent should connect to a
Forgejo on a different host or port (for example a swarm peer's forge).
Validated: must be an `http://` or `https://` URL, or `null`.
**Defaults to `null`, meaning "no forge" — not a guessed address.** A
loopback default would only ever be correct when the forge shares the
agent's network namespace, and inside a container `localhost` is the
agent itself, so the default was a value that built fine and then talked
to the wrong machine. With `null` the credential helper and `forge-avatar-sync`
units aren't generated at all: an absent integration rather than a
misdirected one. You don't normally set this — hive-c0re renders the
host's real forge URL into every agent, and refuses to write a meta
flake without one, so `null` only survives where nix evaluates the
agent modules outside a hive.
**`services.hyperhive.agent.matrix.url`** — homeserver URL used by
`hive-matrix-daemon` when connecting via the matrix-sdk. hive-c0re
writes it into every agent at deploy time as the gateway-routed
`chat.<swarm-domain>` URL, so isolated agents can reach the homeserver.
Override per-agent when an agent should talk to a different homeserver
— for example a remote hive's tuwunel reached over a VPN, or an
external Matrix server for a federation-only agent.
**Defaults to `null`, meaning "no matrix" — same reasoning as
`forge.url` above** (a loopback default would resolve inside the
agent's own netns to the agent itself, not the homeserver). With
`null` the daemon has no homeserver and no-ops exactly as it does
without a token. The hive only forwards `HIVE_MATRIX_URL` when it
actually has a matrix vhost to name, so `null` survives where a hive
runs no homeserver, or where nix evaluates the agent modules outside a
hive.
## Claude Code plugins
The harness installs Claude Code plugins before the serve loop opens.
Three per-agent `agent.nix` options control this:
```nix
services.hyperhive.agent.claudeMarketplaces = [ # default
"anthropics/claude-plugins-official"
"${services.hyperhive.agent.packages.claude-plugins}" # hive's own local marketplace, registered as `hyperhive`
];
services.hyperhive.agent.claudePlugins = [ # default
"skill-creator@claude-plugins-official"
"base@hyperhive"
];
services.hyperhive.agent.claudePluginsAutoUpdate = false; # default
```
- **`claudeMarketplaces`** — list of marketplace sources passed to
`claude plugin marketplace add <source>`. Defaults to Anthropic's
official marketplace plus hyperhive's own `claude-plugins` nix
package (see `nix/packages/claude-plugins.nix`) — a local-path
marketplace registered under the name `hyperhive`, so a hive-authored
skill needs no forge repo or git remote to ship. Override or extend
to add custom marketplaces. Idempotent — re-adding an existing
source is a no-op.
- **`claudePlugins`** — list of plugin specs passed to
`claude plugin install <spec>`. The harness installs each spec on
every boot, assuming `install` is idempotent; failures log a warning
but don't abort boot. Boot waits for the marketplace adds, the optional
update and the installs (serial `claude plugin` calls). The harness kills
and logs any call that hasn't exited within 2 minutes, and caps the whole
sequence at 4 minutes: after that the remaining calls fail at once with
the same warning and the web UI starts. Defaults to Anthropic's
`skill-creator` (so every
agent can author, refine, and evaluate its own skills) plus
hyperhive's own `base` plugin — skills that apply to every agent
regardless of role (currently just `state-hygiene`) — all without
any per-agent wiring.
> Both plugin lists follow ordinary NixOS list-option semantics: a
> per-agent definition **replaces** the default, it doesn't extend it.
> An agent that sets `claudePlugins` and still wants the defaults has
> to list `skill-creator@claude-plugins-official` and `base@hyperhive`
> explicitly alongside its own entries — likewise for the two default
> entries in `claudeMarketplaces`.
- **`claudePluginsAutoUpdate`** — when `true`, runs
`claude plugin marketplace update` before installing plugins to pull
the latest index. Off by default to keep boot times short and
plugin versions pinned.
## `cargo.shortMessages`
```nix
services.hyperhive.agent.cargo.shortMessages = true; # default
```
When enabled (the default), the harness injects a `cargo` shell
function into `/etc/hyperhive/bash-env.sh` that transparently appends
`--message-format short` to compile subcommands (`build`, `check`,
`clippy`, `test`, `run`, `doc`, `bench`, `install`, `rustc`, `fix`).
This suppresses the per-crate progress lines that flood the response
window, leaving only warnings and errors.
The function handles `+toolchain` selectors (`cargo +nightly build`)
and passes through cleanly when `--message-format` is already present.
It leaves non-compile subcommands (`new`, `add`, third-party
`cargo-*`) untouched.
Set to `false` for agents that parse cargo's JSON output
programmatically and don't pass `--message-format json` themselves.
## API-key provider (`useApiKey` / `backendEnvironmentFile`)
```nix
services.hyperhive.agent.useApiKey = true; # default: false
services.hyperhive.agent.backendEnvironmentFile =
"/agents/myagent/harness/openrouter.env"; # default: null
services.hyperhive.agent.model = "anthropic/claude-3.5-sonnet"; # provider-specific model string
```
Runs this agent's `claude` against an API-key provider (for example OpenRouter)
instead of a Claude subscription via OAuth. Two options, paired — each
is a no-op without the other:
- **`useApiKey`** tells the harness itself not to wait for a Claude OAuth
session: at boot, `LoginState::from_dir` reports `Online` without
checking `~/.claude/` (`hive_agent::login::using_api_key`, reads
`HIVE_USE_API_KEY`), and the harness stamps the fact into the
consolidated harness state file so the operator dashboard also stops reading this
agent's empty `~/.claude/` as "needs login." An api-key agent that hits
a real 401 (the key itself is bad) still surfaces `needs_login` — the
harness skips only the boot-time check for an earlier login, not
the auth-failure path.
- **`backendEnvironmentFile`** points at an operator-managed file
(outside the nix store, one `KEY=value` per line, systemd
`EnvironmentFile` syntax) supplying the credentials `claude` itself
reads from the environment — typically `ANTHROPIC_API_KEY` and
`ANTHROPIC_BASE_URL`. Loaded as an _optional_ `EnvironmentFile`
(leading `-`), so setting the option before the file exists doesn't
strand the harness at boot. With `useApiKey` set, the subagent daemon
(`hive-subagent-daemon`) loads the same file, so the agent's claude
subagents run on the same key.
On an agent with a secret store (`services.hyperhive.agent.bao.addr`) that uses
`useApiKey` or the `acp` runtime, `hive-agent-backend-env.service` also keeps the
file's text in the store at `swarm/agents/<agent>/backend-env`, field `env`, and
writes it to `/run/hive-agent-backend-env/env` every 2 minutes. When the store
holds nothing there, the unit creates the leaf from `backendEnvironmentFile`,
byte for byte and once; it never rotates a value and never rewrites or deletes
the file. The harness and the subagent daemon load `backendEnvironmentFile`
first and the `/run` file after it, so a variable the store holds wins, and with
the store down at a start the file alone carries the agent. Both read it at
their own start, so a changed value reaches a running agent at its next
restart.
To set or replace a key, use **set the API-key environment** on the agent in the swarm UI:
it sends `PUT /api/hives/{hive}/agents/{agent}/backend-env` with the file text,
which must be blank lines, `#` or `;` comments, and `NAME=VALUE` lines of at
most 16 KiB. A save overwrites the stored text, including text the agent wrote
into an empty leaf, and the store's value wins over the file. The box never
shows the stored text. **Remove** (`DELETE` on the same path) deletes the stored
text. An agent whose file has content creates the leaf from it again within 2
minutes; one without a file keeps its copy in `/run` until the container
restarts.
A save also keeps the host of `ANTHROPIC_BASE_URL` (and of every `*_BASE_URL`
variable) as an automatic allowed-domain entry for the agent, so a restricted
agent reaches its provider through the hive's proxy; **Remove** drops it. The
controller derives only from an operator's save, never from text it finds in the
store, so an environment saved before this existed needs one more save. An
agent that has just the file gets no entry, and an operator adds its
provider's host in the swarm UI (see
[allowed domains](../../2-handbook/swarm/ui.md#allowed-domains)).
Provision the file once, out of band (never through nix — an API key in
the store is world-readable and travels with the flake closure):
```sh
# on the host, once per agent that should use an api-key backend
sudo install -m 0600 -o root /dev/stdin \
/var/lib/hyperhive/agents/<name>/harness/openrouter.env <<KEYS
ANTHROPIC_BASE_URL=https://openrouter.ai/api/v1
ANTHROPIC_API_KEY=sk-or-...
KEYS
```
The file lives in the agent's bind-mounted **harness** dir, not `state/`
— `harness/` survives container rebuilds exactly like `state/` does, but
(unlike `state/`) is never bind-mounted into a parent agent's container
(see `docs/2-handbook/agent-lifecycle/persistence.md`'s "Parent access to child state"), so an API
key placed here is reachable by nothing but this agent and the host. The
file doesn't stay root-owned, though: the container's activation chowns the
whole harness dir to the agent, so after the next boot the file is agent-owned
whatever owner you installed it with, and the agent can rewrite it. That holds
until the swarm secret store replaces the file.
<!-- vale write-good.Passive = NO -->
⚠️ Verified end-to-end against OpenRouter hasn't happened as of this
writing — `ANTHROPIC_BASE_URL` support in the shipped Claude CLI is
documented behavior, not something this hive has run a live turn
against yet. Tool use, streaming, and MCP all need to keep working
through a non-Anthropic base URL; treat the first real agent on this
path as the actual verification, not this doc.
<!-- vale write-good.Passive = YES -->
Switching an already-provisioned OAuth agent to `useApiKey` leaves
`~/.claude/credentials.json` in place but unused — harmless, not
cleaned up automatically. Cost shape also changes: subscription pricing
→ per-request billing with no built-in monthly cap, worth knowing before
pointing a busy agent at a metered provider.

240
docs/4-implementation/turn-loop/mcp.md Normal file
View file

@ -0,0 +1,240 @@
# MCP surface
The harness ships an embedded MCP server (rmcp 2). A persistent
`hive-mcp-http` daemon serves the built-in `hyperhive` surface over
streamable HTTP (loopback, `127.0.0.1:<services.hyperhive.agent.mcp.httpPort>`,
per-container private netns). Claude connects to its stable URL via
`--mcp-config` rather than respawning a stdio child each turn, so the
URL survives the per-turn claude re-spawn (and a host-side hive-c0re
restart) — there is no per-turn MCP re-registration race for the
built-in surface. HTTP is the sole transport for it (no stdio fallback).
Extra servers (`services.hyperhive.agent.extraMcpServers`) pick their own transport per
entry (`type = "stdio" | "http"`, default `"stdio"`): `matrix` and
`subagent` each run their own persistent streamable-http listener
(`hive-matrix-daemon`, `hive-subagent-daemon`), same reasoning as the
built-in surface and `bash`'s `hive-bash-daemon`. The GUI bridge
(`screen`, gated on `services.hyperhive.agent.gui.enable`) is the one
entry on the schema's stdio default — it spawns
`hive-screen-mcp` fresh each turn. `rea` (gated on
`services.hyperhive.agent.rea.enable`) is a second stdio entry, spawning `rea mcp`. The server name is `hyperhive`, so
the tools land in claude as
`mcp__hyperhive__<tool>`. Each entry also has an `availableToSubagents`
toggle (default `false`) — see
[`docs/2-handbook/tools/subagent.md`](../../2-handbook/tools/subagent.md#mcp-servers-available-to-a-subagent)
for what reaches a subagent's own `--mcp-config` and what doesn't.
An `"http"` entry that needs request headers (a bearer-authenticated remote
server) sets `headersFile` to an absolute path: a file holding a JSON object
of header name to value. It renders as claude's `headersHelper` (`cat` of the
file), so the secret never enters the nix store, and claude re-reads a rotated token
on its next reconnect. ACP agents don't read it.
The agent's permissions gate tool access
([`docs/5-project/process/conventions.md#permissions`](../../5-project/process/conventions.md)).
The default (`Permission::DEFAULT`) is `messaging`, `meta`, `inbox` and
`execution`; `swarmctl agent permissions set` changes it, and the next turn
runs on the change.
## Core tools (always available)
**Messaging** (`messaging` group): `send(to, body, in_reply_to?)`,
`recv(max?)`, `ack_until(up_to)`.
- `send` — hive-local broker message to a peer on this hive (logical name). With
the matrix MCP tools, talk to peers over matrix instead (a DM to the `user_id` in
`get_agent_meta`'s `matrix_accounts`); `send` is for peers without a matrix
account and agents without matrix tools. Optional
`in_reply_to: i64` links the message to a prior id for thread
rendering. Per-agent `services.hyperhive.agent.allowedRecipients`
(default: empty = unrestricted) limits which names `send` accepts —
useful for sandboxing. The broker refuses `to: "operator"`. The error
points an agent that has the matrix MCP tools at matrix
([matrix](../../2-handbook/tools/matrix.md)), and tells one without them that the
operator reads its `set_status` line, its turn output on the dashboard,
and its logs.
- `recv` — drain inbox. Always an immediate peek, never blocks. `max`
(default 1, cap 5) drains up to N rows. `recv` prefixes each returned row
with `[msg #<id>]` (broker row id; note the highest id seen, then pass
it to `ack_until` to bulk-triage the batch). **Graceful shutdown**: when the harness
receives a stop signal, the inbox becomes fenced and `recv` returns an
explicit `from: "graceful-stop"` message instead of an empty inbox.
This unmissably directs the agent to flush durable state (`/state`
files) and end the turn — the container exits when the turn completes.
The graceful-stop turn takes the same post-turn compaction path as any
other turn: if the context crossed the watermark the harness runs a
notes-checkpoint turn and then `/compact`. Compacting before shutdown
keeps a later cold start cheap instead of re-uploading a large transcript.
- `ack_until(up_to)` — bulk-mark inbox rows handled: it stamps every row
with broker id `<= up_to` as acked in a single UPDATE.
Recipient-scoped (agents can only ack their own rows). Use when a
restart redelivers a large backlog of already-handled messages: read
the highest `[msg #N]` from the set you've actually processed, then
`ack_until(N)` to prevent re-pop. Acked rows never redeliver.
Transient pings (sentinel id 0) have nothing to ack and show no marker.
**System messages** (from sender `system`): the broker delivers them as regular inbox
messages (same `recv` path). A container crash never reaches an
agent — it only produces a log line + swarm-level grafana
alert, no agent turn at all (`docs/4-implementation/scheduler/observability.md`). Lifecycle
transitions the job-queue scheduler or crash watcher drive directly —
stop/kill, destroy, a flake-rev login or logout state change — reach
no individual agent: they publish onto a swarm-wide NATS
JetStream stream (`swarm_notices::notify`,
`hive-c0re/src/swarm_notices.rs`); every hive is swarm-controlled.
**Inbox** (`inbox` group): `get_loose_ends()`,
`cancel_loose_end(kind, id)`, `remind(message, delay_seconds? |
at_unix_timestamp?)`.
- `get_loose_ends()` — list this agent's own open threads: pending
reminders, open todos pushed by an in-container subsystem (matrix,
bash, forge, or any user-configured MCP server), an undelivered-inbox
count, and unread matrix notifications. Always the caller's own rows. Full per-variant wire shape in
[`docs/5-project/process/conventions.md` § Loose-ends wire
shape](../../5-project/process/conventions.md#loose-ends-wire-shape).
- `cancel_loose_end` — hard-delete a `reminder` or clear a `todo` row
(loose-ends-v2). Agents may only cancel rows they own.
- `remind` — schedule a reminder in this agent's own inbox. Large
payloads spill to `/agents/<self>/state/reminders/`. Pending count
capped at 50 per agent (`HIVE_REMIND_MAX_PENDING_PER_AGENT`).
No same-turn self-continue tool exists — see
[Turn outcomes](README.md#turn-outcomes) for why. Multi-step work rides
`remind` for a durable self-wake, or an in-container todo wake
(bash-task completion, forge notification, matrix activity) for work
already in flight.
**Meta** (`meta` group): `get_agent_meta(name?)`.
- `get_agent_meta` — fetch identity + status metadata for an agent:
`{ name, hyperhive_rev, running, status_text, status_set_at,
hive_name?, swarm_name?, matrix_accounts? }`. `matrix_accounts` is a
list of matrix identities the agent can act as (`name`, `user_id?`,
`homeserver`); omitted for agents with no matrix provisioning. Omit
`name` to query self.
**Always-on, no permission needed**: `set_status(text)`, `compact()`,
`mark_todos_done(ids)`.
- `set_status` — set a free-text status string visible on the
dashboard. Single line, ≤ 200 chars. Persisted to
`{state_dir}/hyperhive-status`. Pass `""` to clear. Not gated by the
`meta` group — every agent needs to keep its dashboard status chip
current regardless of which optional groups it holds.
- `compact` — agent self-service equivalent of the operator dashboard's
`/compact` button. No args. Gated server-side on the last completed
turn's context usage: refused (with an explanation, no side effect)
unless usage is above 66% of the effective context window. On a
pass, queues the same deferred `compact_pending` flag the dashboard
button sets — consumed at the end of the current turn, so it never
races a live claude process, and the usual pre-compaction
notes-checkpoint turn still fires first. Dispatched through the
in-agent socket (`hive-agent-sock::Request::Compact`), not the
broker — see `hive-agent/src/todo_server.rs`.
- `mark_todos_done(ids)` — bulk-clear specific loose-ends-v2 todo rows
by id in one call, instead of `cancel_loose_end`ing each one.
List-based: pass the ids `get_loose_ends` actually showed you reviewed;
unknown or already-done ids are silently skipped. Dials the same
in-agent todo socket as `cancel_loose_end`'s `todo` kind.
## Privileged tools (by permission)
- **Bash execution** (`execution`) — background shell tasks. See
[`docs/2-handbook/tools/bash.md`](../../2-handbook/tools/bash.md).
- **Subagent spawning** — nested sessions on the agent's own runtime
(claude or ACP, via `hive-runtime`) as background tasks, shipped
default-on like bash execution; no permission gates it. See
[`docs/2-handbook/tools/subagent.md`](../../2-handbook/tools/subagent.md).
- **Config changes** — no permission: they go through a forge PR on
`agent-configs/<name>` — see
[`docs/2-handbook/agent-lifecycle/config-changes.md`](../../2-handbook/agent-lifecycle/config-changes.md).
- **Web egress** (`web_tools`) — enables Claude's built-in `WebFetch`
and `WebSearch` tools (not MCP tools; added directly to the
`--allowedTools` list). Off by default.
- **Recovery mounts** (`manage_root_agent`) — no tool: every agent's state
and config mounted into this agent's container. See
[`docs/5-project/process/conventions.md#permissions`](../../5-project/process/conventions.md).
- **Matrix MCP + extra servers** — `mcp__matrix__*` tools and
per-agent extra MCP config. See
[`docs/2-handbook/tools/matrix.md`](../../2-handbook/tools/matrix.md).
## Waking the agent from inside the container
The built-in producers (matrix, bash, forge) upsert a todo on the
harness's in-agent socket (`HIVE_AGENT_SOCKET`, `UpsertTodo` — see
[`docs/2-handbook/tools/bash.md` § Completion as a todo
(loose-ends v2)](../../2-handbook/tools/bash.md#completion-as-a-todo-loose-ends-v2)
for the full upsert/signal/clear mechanism): a new or changed summary
makes the harness signal its own turn loop, with no hive-c0re
round-trip. An external MCP server (or any other in-container process)
can push its own todo the same way — `subsystem` is any string.
The host-served per-agent socket at `/run/hive/mcp.sock` carries a
lower-level `Wake` request (`hive_core_agent_sock::Request::Wake {
from, body }` — JSON-line, `{"cmd":"wake","from":"matrix","body":"new dm
from @alice"}\n`) that drops the body straight into the broker inbox as
`{from:<label>, to:<agent>, body}`, bypassing the todo layer entirely.
`hive-c0re` handles it; matrix, bash and forge don't call it. Identity = socket: anything that can connect to
`/run/hive/mcp.sock` is implicitly trusted to inject one — the
bind-mount is the agent's own container only.
## Authoritative state
`hive-agent`'s `events::Bus` carries the current turn-loop state in
addition to the broadcast channel and the events history. Variants:
- `Idle` — sitting on `Recv` waiting for mail.
- `Thinking` — `claude --print` is running for a turn.
- `Compacting` — operator-triggered `/compact` is in flight.
The harness flips state at the relevant transitions
(`set_state(Thinking)` before `drive_turn`, `set_state(Idle)`
after; `set_state(Compacting)` around an idle operator compact in
`turn::run_pending_compact`). Exposed via `/api/state.turn_state` +
`turn_state_since` (unix seconds); the agent page renders this rather
than deriving from SSE events.
## Tool envelope
`mcp::run_tool_envelope`: every MCP tool handler logs the request,
runs the body, logs the result. Pre-/post-log only — the inbox
status hint lives in the wake prompt + UI header, not here.
## Tool allowlist (`hive_sh4re::permissions::ALLOWED_BUILTIN_TOOLS`)
The built-in list and the `--tools` value it resolves to live in
`hive-sh4re` (the `Permission` enum itself is in `hive-types`), not in the
harness, because the
subagent daemon spawns its own `claude` and must resolve the same set —
see [`docs/2-handbook/tools/subagent.md`](../../2-handbook/tools/subagent.md).
- Allowed built-ins: `Edit`, `Glob`, `Grep`, `Read`, `Skill`, `Write`.
`Skill` is what makes an installed plugin's `SKILL.md` invokable —
without it, a skill's `description` frontmatter never gets seen by
the model no matter how well it matches the task.
- Permission-gated built-ins: `WebFetch`, `WebSearch` (added once the
agent holds `web_tools`).
- Denied by omission (absent from the harness `--tools` /
`--allowedTools`, so they "literally don't exist" in a harness turn):
`Bash`, `Task`, `NotebookEdit`, `TodoWrite`.
- Additionally in the managed-settings deny list
(`/etc/claude-code/managed-settings.json`, un-overridable): `Task`,
`TodoWrite`. `Bash` **isn't** in the managed deny — see below.
- Allowed MCP tools: as listed above (by permission).
The autonomous harness disallows `Bash` — shell execution goes
through `mcp__bash__run` (background tasks with structured output +
task-id tracking) instead of an interactive shell. The harness gate is
`--tools` / `--allowedTools` (Bash absent from `ALLOWED_BUILTIN_TOOLS`),
so Bash never exists in a harness turn regardless of managed settings.
`Bash` is deliberately **not** in the managed-settings deny so that the
operator-driven `hivectl agent <name> choom` session — which passes neither `--tools`
nor `--allowedTools` — gets claude's built-in synchronous `Bash` tool
(inline, human-approved). That sidesteps the async `mcp__bash__run`
completion wake landing in the wrong session (the harness inbox) for a
choom-started task; `choom` is an operator action (root or `hive-admin`),
so built-in shell there stays within the existing trust boundary. The bash MCP server
(`run` / `status` / `kill`) uses `allowedTools = ["*"]` so all
`mcp__bash__*` tools are available whenever the agent holds `execution`.
`WebFetch` / `WebSearch` are off by default; grant `web_tools`
(`swarmctl agent permissions set`) to enable them from the next turn.

183
docs/4-implementation/web-ui/css-vars.md Normal file
View file

@ -0,0 +1,183 @@
# CSS custom properties (theme variables)
Colour variables live in **two standalone stylesheets**, split so a theme
swap touches only the first:
1. **`colors.css`** — the 16 `--base00`…`--base0F` base16 slots. **This is
the entire theme swap contract.** A generator (for example one fed a stylix
base16 scheme, which is natively base00–base0F) replaces _only this
file_.
2. **`theme.css`** — the semantic layer: `--bg`, `--fg`, `--purple`, …
_derived_ from the base16 slots (`--bg: var(--base00)` etc.). This is
what the app references and it **never changes on a swap**.
Both files live in `frontend/packages/shared/src/`. Per-page stylesheets
(`common.css`, `dashboard.css`, `agent.css`)
reference the **semantic** names and must **not** redeclare them or reach
for the raw `--baseNN` slots directly. (`base.css` holds only the shared
`body` typography — it references the palette but no longer defines it.)
`colors.css` declares each palette's 16 hex values exactly once —
`--mocha-baseNN` (dark, Catppuccin Mocha) and `--latte-baseNN` (light,
Catppuccin-Latte-derived; `base08`-`base0F` run darker than stock
Latte to actually clear WCAG AA against how this app uses them — real
contrast failures found in review, see the comment above those
declarations in `colors.css` for the numbers and why). Everything else
in the file just points the active `--baseNN` slot at one of those two
raw palettes via `var()`, never a second copy of a hex value: an
unconditional default (Mocha), a row gated behind `@media
(prefers-color-scheme: light)` (Latte, the OS/browser-level default),
and two `:root[data-theme='light'|'dark']` blocks a per-user override
sets (swarm-ui's `SettingsMenu`) — an attribute selector always
outranks the plain `:root` inside the media query, so the override wins
regardless of the actual OS preference. Absent an override, the media
query decides; that's what a client with no more specific theme
decision gets (see "Theme swapping" below for how a stylix deployment
supersedes all of this at once).
Both are deliberately standalone, not `@import`ed into the page bundles:
each package re-exports them (`src/{colors,theme}.css` →
`@import "@hive/shared/…"`) so esbuild emits its own
`dist/static/colors.css` + `dist/static/theme.css`, and every page links
them **first** (`colors.css` then `theme.css`) ahead of the page CSS. Note
`theme.css` does **not** `@import` `colors.css` — that would bake the
slots back into it; they're two separate output files so a swap replaces
just `colors.css`. (Load order doesn't actually affect resolution —
custom properties resolve at computed-value time — but each page still
links `colors.css` first for clarity.)
## Semantic palette (`theme.css`)
Each semantic var derives from a base16 slot (or, for the three that have
no clean slot, a `color-mix()` over base16 — pixel-identical under the
default palette). Default hexes shown are Catppuccin Mocha.
| Variable | Derives from | Default hex | Use |
| -------------- | --------------------------- | ----------- | --------------------------------------------------------------- |
| `--bg` | `base00` | `#1e1e2e` | page background |
| `--bg-elev` | `base01` | `#181825` | elevated surfaces: floating dropdowns, popovers |
| `--border` | `base02` | `#313244` | general borders, hover/active backgrounds |
| `--purple-dim` | `base03` | `#45475a` | subtle borders, terminal chrome, badge backgrounds |
| `--fg` | `base05` | `#cdd6f4` | primary text colour |
| `--red` | `base08` | `#f38ba8` | errors, fail state |
| `--amber` | `base09` | `#fab387` | warnings, pending / running state |
| `--yellow` | `base0A` | `#f9e2af` | flash messages, mild warnings |
| `--green` | `base0B` | `#a6e3a1` | success, ok state |
| `--cyan` | `base0C` | `#89dceb` | tool-use events, info accents |
| `--blue` | `base0D` | `#89b4fa` | links, interactive accent (distinct from cyan) |
| `--purple` | `base0E` | `#cba6f7` | accent — active tabs, links, highlights |
| `--pink` | `base0F` | `#f5c2e7` | thinking events |
| `--crust` | `mix(base00 58%, #000000)` | `#11111b` | terminal / code block background (below `--bg`) |
| `--muted` | `mix(base05 55.5%, base00)` | `#7f849c` | secondary / dimmed text |
| `--subtext0` | `mix(base05 77.7%, base00)` | `#a6adc8` | toolbar/status text; dimmer than `--fg`, lighter than `--muted` |
The shares in the `--muted` and `--subtext0` rows are the dark palette's. The light
palette keeps more of the text colour (85% and 93%, set in `colors.css` as
`--muted-share` and `--subtext0-share`), because the dark shares leave its dimmed
text at 2.4:1 against a card, short of the 4.5:1 AA asks for. A palette that sets
neither variable, such as a generated one, keeps the dark shares.
## Common mistakes
The Catppuccin colour names **don't** map 1:1 to the variable names.
Variables to avoid (undefined — they will silently resolve to transparent / inherited):
| Wrong | Correct |
| -------------------------- | --------------------------------------------------- |
| `--text` | `--fg` |
| `--mauve` | `--purple` |
| `--surface0` | `--bg-elev` (float bg) or `--border` (border/hover) |
| `--surface1` | `--border` |
| `--surface2` | `--purple-dim` |
| `--overlay0`, `--overlay1` | `--muted` |
| `--base`, `--mantle` | `--bg`, `--bg-elev` |
## Usage guide
**Floating menus and dropdowns** (for example agent context menu, tabbar overflow):
```css
background: var(--bg-elev);
border: 1px solid var(--purple-dim);
```
**Hover / active state backgrounds**:
```css
background: var(--border);
```
**Active tab text / accent elements**:
```css
color: var(--purple);
```
**Muted / meta text**:
```css
color: var(--muted);
```
**Error / warning / success badges**:
```css
color: var(--red); /* error */
color: var(--amber); /* warning / running */
color: var(--green); /* ok */
```
## Theme swapping — the base16 contract
**The swap interface is `colors.css` — the 16 base16 slots, not the
semantic names.** A theme generator (for example one reading a stylix base16
scheme) overrides only `colors.css`; the semantic layer in `theme.css`
derives everything else, so the whole UI re-themes with nothing else to
template or regenerate. The base16 slot → semantic mapping is _internal_
to `theme.css` (the "Derives from" column above) — a generator never
needs to know the semantic var names, and `theme.css` + the page bundles stay
untouched.
**Two palettes, one contract.** `colors.css` ships both a dark default
(Catppuccin Mocha, unconditional) and a light default (Catppuccin Latte,
behind `@media (prefers-color-scheme: light)`), plus a per-user override
path (`:root[data-theme]`) — a theme generator that overrides the file
wholesale (for example the stylix path, which writes a single unconditional
`:root` block with no media query, no `--mocha-*`/`--latte-*` vars, no
`[data-theme]` blocks) supersedes all of it at once, same as today.
`colors.css` base16 slot defaults (Catppuccin Mocha, dark default):
| Slot | Default | Standard base16 role | Mapped to |
| -------- | --------- | -------------------- | ---------------------------------------------------------- |
| `base00` | `#1e1e2e` | default bg | `--bg`, (darkened) `--crust` |
| `base01` | `#181825` | lighter bg | `--bg-elev` |
| `base02` | `#313244` | selection/surface | `--border` |
| `base03` | `#45475a` | comments/dim surface | `--purple-dim` |
| `base04` | `#585b70` | dark foreground | _(unused; kept for completeness)_ |
| `base05` | `#cdd6f4` | default foreground | `--fg`, (blended) `--muted`/`--subtext0` |
| `base06` | `#f5e0dc` | light foreground | _(unused)_ |
| `base07` | `#b4befe` | lightest | _(unused)_ |
| `base08` | `#f38ba8` | red | `--red` |
| `base09` | `#fab387` | orange | `--amber` |
| `base0A` | `#f9e2af` | yellow | `--yellow` |
| `base0B` | `#a6e3a1` | green | `--green` |
| `base0C` | `#89dceb` | cyan | `--cyan` (named _sky_ here; Catppuccin's `base0C` is teal) |
| `base0D` | `#89b4fa` | blue | `--blue` |
| `base0E` | `#cba6f7` | magenta | `--purple` |
| `base0F` | `#f5c2e7` | extra accent | `--pink` |
Notes for theme authors:
- **`--crust`, `--muted`, `--subtext0` have no own slot** — they're
derived via `color-mix()` over base16 (`--crust` = a darkened `base00`;
`--muted`/`--subtext0` = `base05`↔`base00` blends). They still track a
swap automatically; no generator action needed.
- A standard Catppuccin base16 scheme uses **teal** for `base0C`;
`colors.css` defaults it to **sky** (`#89dceb`) to preserve the historical
accent. A
stylix Catppuccin scheme will shift `--cyan` to teal — that's the
operator's chosen scheme, working as intended.
- `base04`/`base06`/`base07` aren't consumed by a semantic var today;
they're kept at their standard values so the 16-slot contract is
complete (a partial scheme override still works).

Some files were not shown because too many files have changed in this diff Show more