mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
62 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

96
docs/1-getting-started/nixos.md Normal file
View file

@ -0,0 +1,96 @@
# Adding hyperhive to a NixOS flake
How to take a NixOS machine you already manage with a flake to the point
where `nixos-rebuild switch` brings hyperhive up. After the switch, continue
with [`setup.md`](setup.md) for the steps that stay by hand.
## 1 · Add the flake input
In your host's `flake.nix`, add hyperhive next to nixpkgs:
```nix
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
hyperhive.url = "git+https://forge.darkest.space/hyperhive/hyperhive";
};
```
Add `hyperhive` to the arguments of `outputs` too, so the next step can
reach it.
## 2 · Import the module
Import `hyperhive.nixosModules.default`. It's the whole host stack in one
import: the hive (`hive-c0re` and its gateway, bridge and resolver) and every
swarm service. It also wires in hyperhive's own packages, so you set no
package options.
Importing `default` turns nothing on. Every service stays off until you
enable it, and a host that imports it and enables nothing evaluates without
any of the options below.
```nix
nixosConfigurations.my-host = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
hyperhive.nixosModules.default
{ services.hyperhive = { /* step 3 */ }; }
];
};
```
`claude-code` is unfree. Hyperhive scopes the allowance to itself, and only
for agents that run claude (`services.hyperhive.agent.runtime = "claude"`, the
default), so you don't set `allowUnfree`.
## 3 · Set the required options
All of these live under `services.hyperhive`. Evaluation fails with a message
naming the option when one is missing. None of them has a default, because a
guessed value here is a wrong hostname that evaluates cleanly and deploys.
| option | what it does | without it |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| `deploy.hive-controller.enable` | Runs a hive on this host: `hive-c0re`, which owns every agent container, plus the bridge and gateway for the agents. | No agents can run here. Leave it off on a host that only runs shared swarm services. |
| `swarm.domain` | The swarm's DNS domain. Each hive lives in a sub-domain of it. Use the same value on every host of the swarm. | Eval fails on any host that runs a hyperhive service. |
| `deploy.hive-controller.hiveName` | This hive's label inside the swarm, and the first part of its address, `<hiveName>.<swarm.domain>`. It's also the hive's identity in the secret store. | Eval fails on a host that runs a hive. |
| `swarm.hives` | The directory of every hive in the swarm, this one included. Entries need no fields: each `domain` defaults to `<name>.<swarm.domain>`. Every host of the swarm shares one identical copy. | Eval fails, naming the missing entry for `hiveName`. |
An all-local swarm, with every shared service and the hive on this one box,
adds one switch:
| option | what it does |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.singleHostSwarm` | Turns on the shared swarm services, the swarm controller, and `/etc/hosts` entries so the swarm's names resolve on the box itself. Leave it off if the services run on other hosts. |
The smallest working config, as in the README's
[all-local quick start](../../README.md#quick-start-an-all-local-swarm):
```nix
services.hyperhive = {
deploy.singleHostSwarm = true;
deploy.hive-controller.enable = true;
swarm.domain = "hive.example";
deploy.hive-controller.hiveName = "pr1ma";
swarm.hives.pr1ma = { };
};
```
Run `nixos-rebuild switch --flake .#my-host`. The swarm UI then answers at
`https://<swarm.domain>/`.
## Everything else is optional
Leave the other options at their defaults until you need them:
- Spreading a swarm over more than one host, and the credentials a split swarm
needs: [`swarm/README.md`](../2-handbook/swarm/README.md) and
[`swarm/credentials.md`](../2-handbook/swarm/credentials.md).
- Every option, with its type and default:
[the options reference](https://hyperhive.darkest.space/options/).
- The secret store's seal and the granter: [`swarm/bao.md`](../4-implementation/swarm/bao.md).
## Next
[`setup.md`](setup.md): initialize the secret store, create your SSO account
and your first agents.

138
docs/1-getting-started/setup.md Normal file
View file

@ -0,0 +1,138 @@
# First-run setup
What's left to do by hand after the first `nixos-rebuild switch` of the
[README's all-local quick start](../../README.md#quick-start-an-all-local-swarm)
— from a freshly built host to your first agent. Do the steps in order;
each one assumes the ones before it. Step 0, the NixOS flake, module and
required options, is [`nixos.md`](nixos.md).
Every command runs as **root on the host**.
> **Not all-local?** Each step says which host it runs on. A split swarm
> also has credentials you can't generate where they're read — each
> host's mTLS identity at the secret store. Read [`swarm/credentials.md`](../2-handbook/swarm/credentials.md) first; it
> says which files you place, and where.
## 1 · Secret store
_On the host running `swarm-bao`._ Everything else fetches its credentials
from the store, so it comes first.
```bash
sudo bao operator init # keep the keys it prints and the root token OFF this host
```
Then bootstrap the **granter**, the one principal that writes every
`swarm-*` policy and role from then on. This is the only step that needs
the root token:
```bash
sudo -i
read -rs BAO_TOKEN && export BAO_TOKEN # paste the root token from `bao operator init`
bao policy write bao-bootstrap /etc/hyperhive/bao-bootstrap-policy.hcl
bao token create -policy=bao-bootstrap -ttl=24h -orphan -display-name=bao-bootstrap -field=token \
| install -D -m 0600 /dev/stdin /var/lib/swarm-bao-bootstrap/grant.token
unset BAO_TOKEN
systemctl restart swarm-bao-granter-role
```
Check `systemctl status swarm-bao-granter-role` logs
`Uploaded policy: bao-granter`, then restart the granting units that
failed while they waited. The granter unit deletes the token itself after a
successful run:
```bash
systemctl reset-failed 'swarm-bao-*-policy.service' swarm-bao-agent-pki.service
systemctl restart 'swarm-bao-*-policy.service' swarm-bao-agent-pki.service
systemctl status swarm-bao-controller-policy # Uploaded policy, Data written to: auth/cert/certs/swarm-controller
```
⏱️ A first attempt right after a rebuild may fail with `local node not
active` while the store comes up; the units retry every 30s for a day.
With the default `pkcs11` seal the store unseals itself from here on. With
`deploy.bao.seal = "shamir"`, run `bao operator unseal` after every restart.
What the granter is, why it's root-equivalent, and the store's TLS:
[`swarm/bao.md`](../4-implementation/swarm/bao.md).
## 2 · Your SSO account
_On the host running authelia._ Until this runs, `auth.<swarm.domain>` serves
a login page that refuses everyone: its only subject is a placeholder marked `disabled`.
```bash
swarmctl user add mara --display-name Mara --email mara@example.com --group admins
```
It prints a generated password once — record it. Keep both flags:
- **`--group admins`** — the swarm UI and other operator surfaces gate on
it. Without it you log in fine and are then refused.
- **`--email`** — the forge won't create an account without one.
Fix either afterwards with `swarmctl user update mara --add-group admins
--email …`. Details: [`swarm/sso.md`](../2-handbook/swarm/sso.md).
## 3 · Swarm UI login
Open `https://<swarm.domain>/` and sign in with the account from step 2. The
name resolves on the box itself through `/etc/hosts`; from anywhere else it
needs a real DNS record. → [`swarm/ui.md`](../2-handbook/swarm/ui.md)
Signing in to the forge once through SSO creates your forge account under the
same username. Then, _on the swarm-controller's host_:
```bash
swarmctl forge make-admin mara
```
If the forge already has a local account with your username, the first
SSO login asks for its forge password once, to link the two.
Agents learn your pronouns from the **Pronouns** field of your forge profile
settings. Leave it empty and agents refer to you by name only.
## 4 · Clippy, then ruth
hive-c0re creates no agent at startup. On the swarm UI's start page, the
paperclip (clippy, the setup assistant) offers to set up ruth, the manager
agent. Select it: it opens the create form with `ruth` filled in and the
`manage_root_agent` permission checked. That permission mounts the other
agents into her container so she can recover them. The form declares her,
creates her config repo and deploys her to the hive you pick.
The swarm declares a new agent `paused`, so without a further step ruth runs
but drives no turns. Open `/agents` in the swarm UI and set her `up` from her
card.
On a hive that already has a ruth container from an earlier release, create
her once the same way, so that her config repo and store identity exist. The
container keeps starting with the hive meanwhile.
## 5 · Ask ruth for further help
From here on, ask ruth. Open her card on `/agents` in the swarm UI and use the
message box under her terminal. She is the manager agent: her MCP surface is the
privileged one that configures agents, and she can recover the others. Changes to
an agent's config arrive as PRs you merge. → [`agent-lifecycle/agent-roster.md`](../2-handbook/agent-lifecycle/agent-roster.md)
To create an agent yourself instead, use the swarm UI, or:
```bash
swarmctl agent create iris --hive pr1ma
```
The controller provisions iris's identity, forge user and config repo
(`agent-configs/iris`, from the default template), then has the hive build
and start the container. It returns once the scheduler queues the job — watch the
swarm UI's job view for progress.
Later config changes are PRs on `agent-configs/iris`; you merge them on the
forge, and the merge deploys them. →
[`agent-lifecycle/config-changes.md`](../2-handbook/agent-lifecycle/config-changes.md#config-changes)
## After setup: use the swarm UI
Day-to-day operation happens in the swarm UI at `https://<swarm.domain>/`:
agents, hives, jobs and messages. Swarm-level operations take over from the
per-hive dashboard and `hivectl`. → [`swarm/ui.md`](../2-handbook/swarm/ui.md)

133
docs/2-handbook/agent-lifecycle/agent-roster.md Normal file
View file

@ -0,0 +1,133 @@
# Agent roster & privileges
<!-- vale write-good.Passive = NO -->
Agents are a **flat set**: the `manage_root_agent` permission scopes
which agents can manage which others.
<!-- vale write-good.Passive = YES -->
This doc covers what the roster file holds, the map-shaped alternative
`topology::all_agents` accepts, and what's hard-coded for the manager.
## Where the roster lives
"The roster" means two different, non-overlapping things depending on
scope. The [README](../../../README.md)'s control-plane row ("`swarm-controller`
holds the hive directory, the agent roster and the job graph") is the
**swarm-wide** one: `GET /api/agents` on swarm-controller reads
straight from the swarm's authelia identity store over
`swarm-authelia-bridge` (`swarm-controller/src/auth.rs::list_agent_identities`)
— "every agent the swarm holds an identity for" (`swarm-controller/src/routes/agents.rs`'s
`get_agents` doc comment). It's authoritative: an agent exists in the
swarm if and only if it has an identity there, and `swarmctl agent
create` (or the swarm UI, via `POST /api/agents`) writes that identity
once, by calling `ensure_agent_identity`.
This page is about a different, **hive-local** file: the hive-c0re-owned
`meta/` directory's `topology.json`, at
`/var/lib/hyperhive/meta/topology.json`:
```json
["alice", "bob", "ruth"]
```
One entry per agent _this hive_ has state/config for, in name order.
Unlike the swarm roster, it's a derived cache: the reconcile pass below
rebuilds it from what the hive observes locally (config repos cloned, containers spawned), and it
answers a narrower question than "does this agent exist": _which of
this hive's local agents the `ManageRootAgent` permission's bind-mounts
should cover._ `topology::all_agents` is the only reader that matters.
That reader is a permission boundary: the set it returns is what an agent
holding the `ManageRootAgent` permission gets bind-mounted (each other
agent's `state` read-write and `config` read-only; never `harness`). An
agent holding no permission sees its own dirs and nothing else:
`ManageRootAgent` is the only grant that reaches another agent's state
dir. See
[`persistence.md`](persistence.md)'s _Cross-agent access to state._
### Reading the map-shaped format
`topology.json` may also be a map of `name → parent | null`; the reader
accepts that shape and takes its keys as the roster. An empty
roster costs more than a cosmetic gap: every permission holder loses its
mounts until the next reconcile pass writes the array form.
### Why meta, not per-agent `agent.nix`
An agent shouldn't be able to add itself to a set that governs who may
reach its state dir. The roster IS a system-level fact; meta is where
system-level facts live.
### How `topology.json` gets updated
- **Read** — parsed into a set of names; a missing or unparsable file
degrades safely to "no agents" (covers a fresh install that hasn't
synced yet).
- **Reconcile** — runs alongside the periodic meta/flake regeneration.
Adds newly spawned agents, drops removed ones. Reconcile keeps agents
whose config repo exists but that haven't spawned yet, so the gap until
the container appears doesn't churn the file.
No write API and no operator verb reach this file. Reconcile derives it
from which agents exist, so the next pass overwrites a hand edit.
See `hive-c0re/src/agent_config/topology.rs` and `hive-c0re/src/meta.rs`'s
module docs for the exact call chain.
## Hard-coded manager behaviour
The **manager (`ruth`)** differs from other agents in naming/bootstrap
and its socket flavour; its permissions, prompt, tool allow-list and
state dirs work as for every other agent.
<!-- vale write-good.Passive = NO -->
- **Naming/bootstrap** — the manager's broker recipient name, state-dir
key, and nixos-container name are all `ruth` (container `h-ruth`).
Like every agent it comes from the swarm dashboard or
`swarmctl agent create`; `hive-c0re` creates nothing at boot.
Roster-wise, `ruth` is just another entry.
- **Wire-protocol** — `hive-core-agent-sock`'s unified `Request` enum
is the same on both socket flavours. Container lifecycle
ops (kill/start/restart/rebuild) go through the separate host-admin
socket `hivectl` speaks. `Wake` (inject a `from: <X>` message straight
into the caller's own inbox) is on this socket too, unprivileged on
both flavours; matrix, bash and forge don't call it — their
notifications push a todo on the harness's in-agent socket (see
[`docs/4-implementation/turn-loop/mcp.md`](../../4-implementation/turn-loop/mcp.md#waking-the-agent-from-inside-the-container)).
- **Storage/mounts** — the `manage_root_agent` permission, which the swarm
UI's setup assistant creates ruth with, gets a container
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
manage any agent's state dir; config isn't mounted, since a real
config change is a PR from a clone), plus an RO mount for
`/meta` (the hive's topology and limits files). Any agent holding it gets the same mounts.
- **Prompt** — identical for every container but one block (below):
`prompts/system.md`
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker
blocks, and `prompt::render` filters for `"agent"` unconditionally,
manager included ("always `agent` role — there is only one role",
`hive-agent/src/prompt.rs`'s own module doc). The `role:manager`
blocks are dead in production, exercised only by a unit test
(`filter_blocks(SAMPLE, "role", "manager")`). The one block that differs
is `<!-- config:repo -->` vs `<!-- config:archetype -->`: an agent built
from an [archetype](../swarm/archetypes.md) is told it has no config repo.
- **Tool allow-list** — the MCP tools claude sees come from the agent's
permissions alone, the same mechanism for every agent
([`docs/4-implementation/turn-loop/mcp.md`](../../4-implementation/turn-loop/mcp.md)). Ruth's permissions
are the default plus `manage_root_agent`, stored for her like any grant.
- **State dirs** — _not_ treated as an exception: `HYPERHIVE_STATE_DIR` is
injected uniformly via `systemd.globalEnvironment` for every
container including the manager, so all token/state paths resolve
through it the same way everywhere.
<!-- vale write-good.Passive = YES -->
None of the above is a stable interface — treat the module doc
comments as the source of truth for exactly which checks exist today.
## Cross-references
- Milestone: "Agent privileges and sub-agents" (tracked internally)
- Audit table source: milestone comment (tracked internally)
- Operator/agent trust boundary (orthogonal axis): [`boundary.md`](../trust-boundary/boundary.md)

383
docs/2-handbook/agent-lifecycle/config-changes.md Normal file
View file

@ -0,0 +1,383 @@
# Config changes + system messages
Config changes are pull requests on the agent's config repo, which an
operator merges on the forge; that merge deploys them. The "System messages"
section covers the plain-text `system` messages hive-c0re sends agents.
## For operators
- **Config change** — an agent proposed a change to another agent's
config (or its own, via a sub-manager) as a pull request on
`agent-configs/<agent>`. Review it on the forge like any other PR, and
merge it there to deploy it. See [Config changes](#config-changes).
Everything below this point is the implementation detail behind that
flow.
## Config changes
Config changes flow through a **forge pull request** on the agent's
`agent-configs/<name>` repo — the same surface agents use for code PRs.
No MCP tool exists for config changes: opening the PR IS the request.
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
tracked path, but `agent.nix` is the contract entry point), commits
with its own git identity, and pushes a branch + opens a PR with
`hive-forge` — the same way it would change any other repo.
The container has no config mount: the clone is how an agent reads its
own config as well as how it changes it.
Branch protection (the agent isn't on the `main` push allowlist;
merge and approvals allowlist = `operators` team; see "Forge config repos"
below) makes the agent a write collaborator that **can't merge its own
config PR**.
2. An operator reviews the PR **on the forge** (native diff, threaded
comments, CI status) and merges it there. Merging needs membership of
the `operators` team in `agent-configs`. The team starts empty: add
each operator by hand in the forge UI.
3. swarm-controller gets the `agent-configs` org's `pull_request`
delivery. A `closed` event with `merged: true` and base branch `main`
names the commit in `merge_commit_sha`.
4. The controller looks up which hive's wanted state places the agent
and publishes a deploy request carrying that commit on that hive's
deploy subject. With no such hive, or more than one, it deploys
nothing and logs a warning naming them.
5. If the closure the swarm declares for the agent is the one the hive last
applied (the `applied/.<n>.closure` marker), the hive does nothing.
Otherwise it queues a rebuild, which downloads the declared closure. The
request's commit plays no part: the hive keeps no copy of the config and
reads no forge head. An agent with no container on that hive yet gets its
first deploy from the declared closure.
This path runs **no eval-verify**. A merged config that doesn't
evaluate or build fails the rebuild, so the agent's rebuilds keep failing
until a fix merges. A failed rebuild shows on the hive like any other. A merge
whose webhook delivery never reaches the controller deploys nothing.
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
`POST /api/agents` on the swarm controller), and no hive can originate
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
agent's config repo with a default `agent.nix` template, then asks the
target hive to deploy it.
An agent created from an [archetype](../swarm/archetypes.md) gets no
config repo: the swarm builds it from the archetype's module.
Changing what the template seeded isn't an exception: like every
later change, it's a PR on that config repo, made from a clone and
merged by an operator on the forge (see [Config changes](#config-changes)).
## Agent lifecycle
### Reminder delivery: file-path semantics
A reminder may carry a `file_path` (the agent-visible path inside its
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
1. **Translates** the container path to the host path
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
from outside the container.
2. **Validates** the path: rejects anything outside the agent's own state
subtree, containing `..` (path traversal), or with an empty relative
tail. On rejection hive-c0re skips the write and delivers the
original message inline with a warning — the reminder still fires.
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
canonicalizes the parent dir and re-verifies it lives under the agent's host
state root. hive-c0re opens the final file with
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
basename can't redirect the write to an arbitrary host path.
4. **Writes the body to disk** and delivers a short pointer message in its
place, keeping the agent's inbox / wake-prompt small while the agent
reads the bulky payload out of band.
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
body strings before calling it.
### Destroy semantics
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
Stops + removes the nspawn container and drops the systemd drop-in.
Persistent state (claude credentials, `/state/` notes) is
**kept by default** — recreating the agent with the same name
reuses prior notes + login. With `purge = true` the agent's
`/var/lib/hyperhive/agents/<name>/` tree is also
wiped (creds + notes gone forever). The
root/bootstrap container is destroyable like any other, and stays gone
until you create ruth again from the swarm dashboard.
## Meta flake
A hive has no meta flake of its own. It builds every agent from the swarm's
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
It declares one flake input per agent (`agent-<n>.url =
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
agent's forge config repo, the one config PRs merge into) and one
`nixosConfigurations.<n>` output per agent. Each output wraps
`inputs.agent-<n>.nixosModules.default` with the identity +
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
(the dashboard port, the pinned claude-code, extra OTEL labels, the queue
address) aren't in it:
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
start (see `docs/5-project/process/conventions.md`).
A config that sits only in a local checkout, not on forge `main`, never
deploys.
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
state, bound read-only into the root agent's container as `/meta`:
`topology.json`, the swarm resource limits, and the permission seed files of a
hive that predates the swarm's permissions. `agent_set::refresh(agents:
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
nginx include in line with the live agents. `provision_container`, `destroy`,
`rebuild` and the startup migration call it. It refuses an empty agent list,
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
is unset, since every agent would come up with no forge.
## The swarm's meta flake
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
every agent any hive places (the pins:
[`docs/4-implementation/swarm/builder.md`](../../4-implementation/swarm/builder.md#meta-flake-lock)). The swarm
builder builds every agent from it, and a hive deploys that build. While the
swarm has recorded no build of an agent, that agent's deploy fails and names the
agent. The controller queues
its update of the repo before the deploy request for a merged config commit
and for a new agent, and sends the deploy only when that update succeeded and
the swarm builder built the agent from the updated repo (the
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
agent's record in `agent-closures` declares, whichever `internal/meta` commit
the builder built it from, and reads no forge head. The rebuild's `MetaSync`
node prepares it (the agent's archetype, its runtime directories, the agent
set), reads the record once and pins its store path. `Prebuild` and `Swap` use
the pin, so a record that changes mid-rebuild can't make `Swap` pull another
closure than `Prebuild` fetched, and the marker names the closure it applied. The
next autoupdate pass picks up the newer record. A first deploy has no
`MetaSync`; its `Create` reads the record once.
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
the agent's record in the swarm's `agent-closures` bucket (the controller
writes it from the builder's reply) and have `hive-priv` download the closure
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
`CreateContainer`), which accepts only the path its last `RealiseClosure` for
that agent returned. On the wire the
`hive_priv_sock::PrivRequest` variants carry a required field:
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
A rebuild with no record, or none the builder pushed, fails with an error
that names the agent; see
[`docs/4-implementation/swarm/builder.md`](../../4-implementation/swarm/builder.md#rebuilds-download-the-closure).
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
runs the hyperhive of the controller's deploy, not its hive's.
- **Marker.** After a successful swap or create `hive-c0re` writes the closure's
store path to `applied/.<n>.closure`. The autoupdate pass
([below](#autoupdate-on-startup)) compares it with the record.
## Deploy markers per agent
```
/var/lib/hyperhive/applied/ deploy markers - core-only
└── .<name>.closure # store path of the last closure applied
/var/lib/hyperhive/meta/ plain hive state - core
├── topology.json # the live agent set
└── swarm-resource-limits.json # the swarm's per-agent caps
```
hive-c0re keeps no copy of an agent's config: the forge's `agent-configs/<name>`
`main` is the only one, and a config change is a PR against it. The marker
records the closure last applied, and nothing else. Hives that ran before the
marker existed have `.<name>.hyperhive-rev`, `.<name>.config-rev` and
`meta-rev/<name>` files instead. Nothing reads them any more, and you can
delete them.
Older hives also kept an *applied* repo at `applied/<name>` and a *proposed*
clone at `agents/<name>/config`. Nothing reads either any more, and you can
delete them.
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
The agent's `agent-configs/<n>/flake.nix`
is a fixed boilerplate that exports `nixosModules.default =
import ./agent.nix`; the meta flake imports that module and
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
### Forge config repos
The bundled `hive-forge` container runs on the swarm's forge host
(`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)).
Every agent created through the swarm has a repo in the private
`agent-configs` Forgejo org, created and seeded by swarm-controller.
hive-c0re pushes nothing to it, keeps no clone and reads nothing from it: forge
`main` advances only by an operator's merge.
Each agent is a **write collaborator on its own** `agent-configs/<name>`
repo — so it can push a branch and open a config PR — but not a member
of any other agent's, so it can't reach another agent's config through
the forge. Branch protection keeps the agent off the `main` push
allowlist and allowlists merging to the `operators` team only, so an
agent can't fast-forward its own config or self-merge its PR (see
[Config changes](#config-changes) above). hive-c0re passes the credential to
each `git` call inline, never writing it into a repo's config, so
no stored token sits in a repo an agent might one day read.
The dashboard deep-links into this org — a `config repo` link
per container row. See `docs/2-handbook/web-ui/dashboard.md`.
### Submitting agent's view of config repos
No agent has a config repo mounted, not even one holding
`ManageRootAgent`: reading another agent's config means cloning its repo
from the forge, subject to the forge's own access rules.
An agent submits a change to another agent's config the same way it
makes any other change: **clone the target agent's config repo from the
forge into its own state dir, commit on a branch, open a PR**, and let
the operator review and merge it. By design, no second, mount-shaped
path reaches the same file without the review.
Agents holding the `manage_root_agent` permission (granted per agent in
the swarm's `agent-permissions` record; see
`docs/5-project/process/conventions.md#permissions`) get additional host-side bind
mounts via `set_nspawn_flags`:
- `/var/lib/hyperhive/agents/<n>/state` → `/agents/<n>/state` (RW) for
**every** agent. The permission means "may manage any agent," so the mounted
set is every agent.
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state
(topology, swarm limits).
An agent without the permission only has its direct children's config
dirs.
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
an agent, after its current turn, when its `manage_root_agent` changes.
No agent holds the permission by default: an agent whose record leaves
`permissions` unset gets the default, which doesn't include
`manage_root_agent`, whatever the agent's name. The swarm UI's setup
assistant creates ruth with it. For a ruth that predates that, whose
permissions were unset, swarm-controller writes the default plus
`manage_root_agent` into her key once at start, so she keeps her mounts;
it never touches a key that's already set.
## Startup migrations (older hosts)
hive-c0re runs a couple of idempotent migrations on every startup so a
host set up before the current deploy scheme (described above)
converges to it automatically. Each phase is a no-op
once already applied:
- **Agent files**: refreshes the files that follow the agent set
(`agent_set::refresh`) from the current agent list. Set
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
- **Legacy meta repo**: deletes the leftover `.git` in
`/var/lib/hyperhive/meta/` and the
unread `/var/lib/hyperhive/forge-core-token`. The files beside the `.git`
stay. The forge-side `core/meta` repo and `core` account aren't touched.
No state loss in either migration: claude creds, `/state/` notes, the
events DB all survive. The root
agent keeps its session; sub-agents stay logged in.
## The root/bootstrap container
The root agent container runs through the **same lifecycle as
sub-agents**. Nothing creates it at hive boot: you create ruth from the
swarm dashboard like any agent, and an existing ruth container starts with
the hive like every other. The root agent can clone
`agent-configs/ruth` to read its own `agent.nix` and open a config PR there
for operator review, same as any other agent.
Differences from sub-agents:
- Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`.
Her container builds from `agent-base` like every agent, and the seeded
`agent.nix` is the same for every name, so a newly created ruth has no docs
until a config PR adds that line.
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
FNV-1a hash as every other agent (8100..8999 range).
- The root agent holds `manage_root_agent` (the swarm UI's setup
assistant creates her with it), for which `set_nspawn_flags` adds extra
binds: every agent's `state` dir (RW) under `/agents/<n>/state`, and the
hive's plain state under `/meta` (RO).
- Its socket comes from the generic per-agent registration, in the same
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
not a separate manager-only path.
**Migration note** (for older hosts): drop any `containers.root =
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
definition, then create ruth from the swarm dashboard.
## Root-agent policy
The system prompt (`hive-agent/prompts/system.md`, rendered by
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
varies is which MCP tools it surfaces (gated by the agent's permissions,
which the swarm sets). No `role:manager` block renders only
for the root agent. The root agent's role-specific
behaviour comes from its AGENTS.md / agent-specific instructions, not
the system prompt template.
## System messages
Hive-c0re has no typed event that drives an immediate claude turn. A
message from sender `system` is plain inbox text, and the harness logs it
(`log_system_event` in `hive-agent/src/main.rs`) without parsing the body.
Today hive-c0re sends two: the `/knowledge updated` broadcast
(`workers/knowledge.rs`) and the wake message for a graceful stop
(`Coordinator::kick_agent`).
**A previously running container crashing** drives no agent turn.
`crash_watch` only `tracing::warn!`s `"container crash
detected"`; a swarm-level grafana alert matches that log line directly
and pages the operator, with no agent (and no per-hive dashboard
banner) in the loop. See
[`docs/4-implementation/scheduler/observability.md`](../../4-implementation/scheduler/observability.md#container-crash-alerting-hive-c0re).
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
when convenient" events with no reason to drive an immediate turn, so
they go out via `swarm_notices::notify` instead: `subsystem = "core"`,
`key = "<event>:<agent>"`, and a single free-text `summary`
(`rebuilt_todo_summary` renders `Rebuilt`'s `ok`/`note`/`sha`/`tag`
fields into that string).
To add a new lifecycle notice: if it's "FYI, check when convenient,"
call `swarm_notices::notify` — no new wire type needed. If it needs to
drive an immediate turn, decide first whether it actually needs an
*agent* in the loop at all (the crash case above didn't, once a
swarm-level alert could carry the same signal).
## Autoupdate on startup
`hive-c0re serve` runs `auto_update::run` in a background task right
after opening the coordinator. It enumerates managed containers and
rebuilds any whose closure marker differs from the store path the agent's
record in `agent-closures` declares. That's how a hive that missed the
swarm's deploy request catches up, and it repeats every five minutes, which
covers a request lost while the hive is up. Sub-agents and the root agent go
through the same graceful rebuild subgraph.
The comparison reads the record and the marker
(`/var/lib/hyperhive/applied/.<name>.closure`, written after a successful
swap or create) and nothing on the forge. Where the marker is missing, as for
every agent on the first boot after the marker arrived, the hive reads the
closure the container's profile
(`/nix/var/nix/profiles/per-container/h-<name>/system`) points at and writes
that as the marker, so the first pass catches an agent that doesn't run its declared
closure. Those first-boot rebuilds go through the same job queue as
any other: each holds one of the `buildSlots`, so they don't swap all at once.
It rebuilds nothing when the hive can't tell: no connection to the swarm queue,
no record, a closure the builder didn't push, or a profile it can't read or that
leaves the nix store. The periodic pass
attempts a declared store path once per process, and skips an agent while a
rebuild of it waits or runs, so a closure that fails to deploy doesn't
restart its agent every five minutes. Restarting `hive-c0re`, or a rebuild from
the agent's menu, tries again.
A stopped agent gets its rebuild at its next start: the start path upgrades
the start of a stale, stopped agent to a rebuild followed by the start. In
the window after a config merge, before the swarm's builder has recorded the
new closure, the rebuild deploys the previous one.
The dashboard shows no pending-update state: the hive applies a changed
declaration itself. The agent menu's rebuild queues the same
`job_queue::templates::rebuild` DAG, which downloads and applies the declared
closure again.

850
docs/2-handbook/agent-lifecycle/persistence.md Normal file
View file

@ -0,0 +1,850 @@
# Persistence + retention
Where state lives, what survives what, and how it's bounded.
## For operators
The short answer to "does anything disappear": **destroying an agent
keeps its state, purging it doesn't.**
- **`DESTR0Y`** (the default action) stops and removes the container
but keeps everything on disk — config history, claude login, `/state/`
notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on
the C0R3 page); `swarmctl agent create` with the same name and hive
recreates it from the kept state, **no re-login needed**.
- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
history, claude credentials, `/state/` notes, everything. **No
undo.** Only reach for this when you actually want the agent gone
for good. The agent's matrix encryption keys go with the state dir,
but an account whose key backup is on (see
[matrix](../integrations/matrix.md)) gets them back from its
homeserver when it next comes up, so its encrypted history stays
readable.
Beyond that:
- **Broker messages**: acked ones vacuum after 30 days; anything
undelivered or delivered-but-not-yet-acked is always kept, however
old.
- **An agent's own `/state/` notes and claude login survive every
restart and rebuild** — only an explicit purge (or a hive
`--purge`-style host operation, or the agent's own choices) touches
them.
- The **root/bootstrap agent has no guard**: destroying ruth is as final as
for any agent, and she stays gone until you create her again from the
swarm dashboard.
Everything below this point is implementation detail: exact table
schemas, file layouts, and internal migration mechanics.
## Sqlite databases
### `/var/lib/hyperhive/db/broker.sqlite` (host)
Three tables, all in one file — the message queue, a small key/value
table, and the per-agent power-intent registry:
- `messages` — every inter-agent / operator-authored message.
`sender / recipient / body / sent_at / delivered_at / acked_at /
in_reply_to / priority`. `in_reply_to` links a reply to its parent
row id; the dashboard and per-agent inbox render these as threaded
rows.
- `kv` — small persistent key/value store (`key PK / value`) for
host-side bookkeeping that doesn't warrant its own table.
⚠️ The `mcp__hyperhive__remind` queue lives in a harness-local,
per-agent store — see [`/harness/` contents
below](#state-dirs-per-agent) for where reminders (and todos)
live.
- `agent_power` — one row per agent: `agent PK / wanted (up |
offline) / updated_at`, owned by `hive-c0re/src/stores/power.rs`.
This is the durable power *intent* the job queue reconciles the
observed container state against; intent survives hive-c0re
restarts even though in-flight queue work doesn't. See
[`docs/4-implementation/scheduler/coordinator.md`'s Desired-state
section](../../4-implementation/scheduler/coordinator.md#desired-state-spec-vs-status) for who
writes and reads it and how reconciliation works.
On a host whose hive-c0re once kept scheduled prompts, the file also
holds `scheduled_prompts` and `scheduled_prompt_targets`. Nothing opens
them; scheduled prompts live at swarm level
([`swarm/schedules.md`](../swarm/schedules.md)). They're safe to drop by
hand.
Retention:
- `Broker::vacuum_delivered` runs hourly via a tokio task in
`hive-c0re::main`. Drops acked message rows older than 30 days
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
rows are always kept — the harness `ack_turn`s only after a
successful turn, so `requeue_inflight` can still requeue
an unacked row on a crash.
- `agent_power` rows live until the operator destroys the agent (one row per
agent — nothing to vacuum).
### `/harness/hyperhive-events.sqlite` (per agent)
Lives inside each container's bind-mounted `/harness/` dir (host
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
One table:
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
harness emits during turn loop execution.
The db runs in WAL mode, so `-wal` and `-shm` files sit beside it while
the harness is up. Copy or back up the three together, or the recent rows
sit in the `-wal` file until a checkpoint.
<!-- vale write-good.Passive = NO -->
The harness both writes and vacuums it. Cleanup runs in-container
because hive-c0re runs as the unprivileged `hive-core` user under
privsep and can't delete agent-owned files (host-side deletes hit
`PermissionDenied` on the bash-task trio and a readonly database error
here). `hive-agent`'s `vacuum::run`
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
**type-scoped**: it deletes only the verbose `stream` rows (the raw
claude `stream-json` deltas — one per text chunk / tool use, the bulk
of the file's size) older than 14 days, and keeps every other kind
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
small and carry the semantic per-turn history the operator scrolls
back through when debugging a regression. Age-only within the
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
history sooner than a quiet one. The trade-off (accepted): a
misbehaving harness can skip its own cleanup — but a compromised harness is
already inside the container trust boundary
([`docs/2-handbook/trust-boundary/security.md`](../trust-boundary/security.md)), and these are ephemeral local
artifacts, so cleaning them up where they live is the honest fix.
<!-- vale write-good.Passive = YES -->
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
setups). On open failure the `Bus` falls back to no-store mode
rather than crashing the harness — events still broadcast over SSE,
just nothing persisted.
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
Per-turn analytics sink. One row per claude turn captures
identity (`model`, `wake_from`, `result_kind`), timing
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
cache_read / cache_creation token counts), behaviour
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
snapshot metrics (`open_threads_count`,
`open_reminders_count` — fetched via the same socket the harness
already uses for `GetOpenThreads` + `CountPendingReminders`).
Bin-loop helpers `build_row` + `record` land each row at
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
the turn loop continue.
The `hive-bash-daemon` (not the harness) writes a sibling
`bash_commands(ts INTEGER, head TEXT)` table in the same
file: one row per executed bash task recording the normalised command head -
the basename of the first real command, looking past `cd repo &&`
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
backs the "favorite tools" view on the /stats page. Best-effort and created on first write
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
task runs.
turn-stats.sqlite has **no vacuum** — it's one small row per turn
(~hundreds of KB even over months), read directly by the `/stats` page,
so pruning it would only lose trend
history for no space gain.
### `/state/hyperhive-harness.json` (per agent)
Consolidated harness state file written atomically (`.tmp` + rename) by
`Bus::emit_status` whenever rate-limited or login-failed flags change.
Shape:
```json
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
```
- `rate_limited` — set when the harness detects a 429 from the Claude
API; cleared by any subsequent status emit. Drives
`ContainerView.rate_limited` on the dashboard.
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
cleared by `"online"` status (re-auth completed). Drives the
`needs_login` flag alongside the `claude_has_session` check.
- `active_model` — the resolved Claude model for the dashboard badge.
The turn loop is the only writer today, but it still goes
read-modify-write under a shared in-process lock and merges into the
existing object rather than reconstructing it — so a second writer
would preserve fields it doesn't own, and the lock closes the
lost-update window between a writer's read and its rename. The lock
is in-process only, so it wouldn't serialise a writer running as a
separate process; none of today's writers are.
hive-c0re reads this file on each `build_all` sweep (~10s) via
`container_view::read_harness_flags`. Falls back to the individual
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) when the
JSON is absent.
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
Full stdout + stderr capture for every `nixos-container` / `nix
build` invocation the lifecycle layer fires. One row per invocation;
the row accumulates lines as the child runs.
Capturing the full stream (rather than a short tail buffer) matters
because real eval errors routinely run long — "tried alternatives"
blocks alone are often 30+ lines — so a truncated tail would cut off
the actual failure and leave only the host journal holding the
complete output. With this table the dashboard can surface the entire
log.
Three indices:
- `(agent, started_at)` — backs the per-agent latest-N lookup used
by the agent card chip.
- `(status, finished_at)` — backs the retention sweep that runs
as part of the existing hourly vacuum.
- `(node_id)` — added by a later migration so a build log row can be
looked up by the job-queue node it belongs to (a `hive_jobq` node is
immutable after insert, so hive-c0re records the link on the log row
instead); `node_id` is nullable.
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
log a warning on sqlite error and let the build continue. A failed
log row never blocks a rebuild.
Each operation also logs to the host journal, target `nixos-container`, and the
swarm log store collects those lines. Every line carries `build_id` (this
table's row id; empty without the store), `agent`, `kind` and `cmdline`. A
`start` line (`event=start`) opens the build, one line follows per line of
output, and a `finish` line (`event=finish`) closes it with `status` (`ok` or
`fail`) and `runtime_secs`. hive-c0re logs a nix warning or error at warn, and a failed
finish too.
A row is open (`finished_at IS NULL`) only while its child runs. At startup
`Coordinator::open` closes every row still open as `fail` and appends a
line saying hive-c0re stopped: the previous process died mid-build, and
the retention sweep never deletes open rows.
### `/harness/hyperhive-model` (per agent)
Single-line text file holding a claude model name. The harness only
reads it, at boot in `Bus::new` and again when the swarm clears the
model it set. It holds the model the agent runs when neither the swarm nor
`HIVE_DEFAULT_MODEL` (agent.nix) names one. Nothing in the harness
writes it. Path overridable via `HYPERHIVE_MODEL_FILE`. Survives
destroy/recreate, gone on `--purge`.
### `/harness/paused` (per agent)
Empty marker file. Its presence parks the agent's turn loop: the
harness keeps serving its web UI and MCP daemons but drives no turns,
and inbox messages queue unacked until it's removed (see
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
<!-- vale write-good.Passive = NO -->
Unusually, it's read and written from **both** sides of the harness
bind-mount, and that's the whole design: the harness stats it
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
stats it on the host (`Coordinator::is_paused`) to populate the
`paused` field on the agent card, and creates/removes it
(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
dashboard toggle. Because the file itself is the only shared state
there's no protocol between them, no round-trip into the container, and
pause keeps working when the harness is wedged or the container is
stopped.
<!-- vale write-good.Passive = YES -->
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
is the agent's own space to fill, and this is harness control state.
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
back paused after a restart, which is the intended behaviour rather
than an accident of storage.
## State dirs (per agent)
Under `/var/lib/hyperhive/agents/<name>/`:
- `config/` — unused. Older hives kept a proposed nix repo here; nothing
reads it any more, and you can delete it. An agent reads its config by
cloning `agent-configs/<name>` from the forge.
- `claude/` — claude OAuth credentials, bind-mounted RW to
`/home/<name>/.claude` inside the container.
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
to `/agents/<name>/state` inside the container (uniform for
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
the same path to in-container scripts. Notable files written here
by the harness:
- `hyperhive-status` — single-line free-text status string written
by `set_status`; cleared on explicit `set_status("")`. Read by
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
surface the status chip on the dashboard. Absent when no status
has a value.
- `hyperhive-harness.json` — rate-limited / needs-login flags read
by the dashboard's async container-state fetch. See
`docs/2-handbook/web-ui/dashboard.md::Container row`.
The agent writes its own instructions to `AGENTS.md` here; claude and
opencode load it automatically from their cwd, `state/`. At boot,
`agents_md_migrate::run` renames a `state/CLAUDE.md` to `AGENTS.md`
when no `AGENTS.md` exists. When both exist it renames neither and
logs an error naming both paths: claude loads only `CLAUDE.md` while
one exists in its cwd or above. The harness also publishes the file
to the swarm, which reads that copy, and applies the git diffs the
swarm sends to edit it. A new agent's first start writes the text given at
create, only when no file exists. This file stays the only copy that counts:
[`docs/2-handbook/swarm/README.md`](../swarm/README.md#an-agents-agentsmd).
- `harness/` — harness-internal ephemeral state; not intended for
agent consumption. Bind-mounted to `/agents/<name>/harness`
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
- `bash-tasks/` — task JSON + stdout/stderr files for
background `mcp__bash__run` jobs. JSON files are
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
(full captured output). The harness's own hourly sweep
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
event rows above) deletes terminal task trios older than 48
hours; non-terminal (still-running) tasks are never deleted. This
sweep runs in-container, for the same privsep-ownership reason as
the events vacuum above.
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
and reminders, one small table each in a single file (in-container
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
`hive-forge-notify` — upsert keyed todos here over the harness's
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
`get_loose_ends` output and clears a row on `mark_todo_done`).
At boot, `db_migrate::run` creates this file from
`hyperhive-todos.sqlite` and `hyperhive-reminders.sqlite` when those
are present. Also backs the `mcp__hyperhive__remind` queue.
The harness itself is also a producer, not just the socket server:
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
socket the out-of-process daemons above dial) alongside
`disk_watch::run` — an *in-process* todo producer that shares the
store + wake `Notify` directly rather than dialling its own socket.
`disk_watch` raises a keyed `disk` todo when the filesystem backing
this agent's state gets tight, naming the agent's own biggest
directories; it buckets the summary and carries no raw byte
counts, so an unchanged situation re-upserts as `changed == false`
and never re-wakes.
Retention, same hourly `hive-agent::vacuum::run` sweep as
`hyperhive-events.sqlite` below: it reaps delivered (soft-deleted) reminder
rows 14 days after delivery, kept that long only to serve
the trailing-window `ReminderRollup` stats, and reaps acked todo rows
30 days after acking (long enough that only a genuinely quiet
month triggers the "one spurious re-announcement" fallback a
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
module doc). Un-acked todos and undelivered reminders are never
swept — same "audit trail, not cache" treatment as the c0re-side
tables above.
- `hyperhive-events.sqlite` — turn-loop event log.
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
effort level files, read-only fallbacks.
- `work/` — the agent's working space for git worktrees, build dirs
and other large working data. Bind-mounted RW to
`/agents/<name>/work` inside the container and owned by the agent
user. It sits outside `state/` so a claude or subagent started under
`work/` doesn't walk up into `state/AGENTS.md` and load the agent's
own identity. Survives `destroy`/recreate, gone on `--purge`. Counted
in the per-agent disk metric (`measure_agent_disk`) together with
`state/` and the container rootfs. The dashboard doesn't serve it.
### The container root (per agent)
An agent container starts with `systemd-nspawn --ephemeral`
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
clones and builds go in `work/`.
The nix profile needs one extra step. nix keeps a non-root user's profile in
`~/.local/state/nix`, so the agent's tmpfiles rules
(`nix/agent-modules/user.nix`) point that directory into
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
`nix profile` survive a restart, and go with the container on `destroy`.
**The one-time salvage.** The first start of an agent whose root has content
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
`WriteNspawnFlags` that precedes every start, while the container is down:
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
modes and mtimes kept. The agent finds its old `/home/<name>` there.
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
logs the path. Nothing deletes it; the operator removes it when ready.
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
id, so the journal directory and DHCP identity hold) and the marker
`etc/hyperhive-ephemeral-root`.
The salvage runs again only if the template holds content outside the cache
paths, which happens when a hive without this change started the container
persistently in between; the new copy gets a numbered directory beside the
first. A failed salvage leaves the root where it was and the container starts
without the flag (hive-priv logs the error), so a failure costs the switch,
never data.
**Checked before every start.** nixpkgs' start script runs as host root on
the root template, and its `mkdir -p`, `chmod`, `touch` and
`cp --remove-destination` follow a link the container planted at the paths
they touch. The container's limits drop-in
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
script, on systemd's own restarts too. It looks at each path with `lstat`,
never following a link, and fails the start with one line on stderr naming the
path and its type when:
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
exists and isn't a real directory;
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
(`etc/os-release` may also be a link, which the script skips and a booted
NixOS guest creates);
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
exists and isn't a link that resolves into `/nix/store`, or the `system`
file inside the resolved directory isn't a regular file.
Absent paths pass, so a first start works. The check neither deletes nor
repairs: the operator removes the offending entry, and the next start goes
through.
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
points at a build without the sub-command. A drop-in written by an older
points at a build without the sub-command. That build's store path has a GC
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
drop-in and replaced on each write, so a garbage collection can't remove the
binary a restart would run. The directory sits beside `gcroots/per-container`,
which is the only gcroots path bound into a container, so only root writes it.
Destroying the container removes the drop-in and then its root; the build stays
until the next collection. A drop-in written by an older hive-priv has no
`ExecStartPre=` and no root, and the container starts as before.
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
the swarm services (15 containers, declared by their own modules) run the same
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
their own `container@<name>` unit, through the symlink `/run/hive-rootfs-check`
that `hive-priv.nix` points at the current build, so a new hyperhive build does
not change the unit or restart the container. It evaluates every rule, logs each
violation
as a warning, and exits 0, so the container starts either way. The line reads
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
Strict refusal needs the live roots of all 15 to pass first, and the warnings
show which ones would not. A container the operator runs that hyperhive doesn't
declare gets no check at all: the modules name their own containers, and
nothing enumerates the host's `containers`.
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and
copies the data of a file the filesystem refuses to clone. Before it creates
anything, hive-priv sizes the root and compares it with the free space under
`work/`. A salvage whose files clone (tested with one file of the root) must
leave 2 GiB free. One that has to copy data must leave 15 % of the filesystem
free. When the free space is short, hive-priv skips the salvage, deletes
nothing, logs `salvage skipped: <container> need <size> free <size>`, and the
container starts persistent; the next start tries again. hive-priv checks a file
that fails to clone against the 2 GiB reserve again before it copies the data. A
salvage that fails partway removes the directory it created, and leaves the root
and any earlier salvage as they were. After a salvage, hive-priv logs `salvage: <container> <size> reflinked=<n>/<m> files`.
**Concurrency.** hive-priv holds an `flock` on
`/run/hive-priv/salvage-<container>.lock` (the unit's `RuntimeDirectory`, which
no container can write) for the whole `WriteNspawnFlags`, so a second request for
the same container waits and then finds the finished template. Across
containers, one salvage copies at a time, so the free space each one checks
excludes any copy still in flight. The request runs on the blocking
thread pool, so a long copy holds no async worker. hive-c0re's per-agent lease
already serialises the requests it sends; the lock covers hive-priv's own
guarantee.
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
in ephemeral mode without failing. The journal vanishes at stop, and with it
the lines written after the last push.
### User namespace (opt-in, per agent)
An agent whose config sets `services.hyperhive.agent.user.privateUsers = true`
runs in its own user namespace. Every other agent keeps the host's uids.
**Approving the config PR that sets it authorises a host-level change:** the
first start afterwards has hive-priv, as root, chown the agent's `state`,
`harness`, `work` and `claude` dirs once to the host's data owner. It deletes nothing.
**How the hive learns the flag.** It never evaluates the agent's nix. The swarm
builder reads `config.services.hyperhive.agent.user.privateUsers` from the
config it evaluates in its sandbox and puts it in the agent's closure record
(`private_users`). hive-c0re reads the record when it writes the container's
flags and sends the value to hive-priv with each start. An agent whose module
doesn't declare the option reads false. When the builder can't read the
option (its eval fails or times out), the record carries no flag, and so does a
record or builder from before the field. A record without a flag, a record
hive-c0re can't read, or none, leaves the agent as it is.
- **On disk the data belongs to one host uid:gid**,
`services.hyperhive.deploy.hive-controller.userns.dataOwner`
(default `60000:60000`). Inside the container that owner shows as the agent
user. hive-priv mounts the agent's `state/`, `harness/`, `work/` and
`claude/` dirs and its socket dir with `owneridmap`, and every other bind too
(config, `/knowledge`, the runtime dir), whose owner shows as root inside.
- **`/nix/store` has to be on ext4, btrfs, xfs or erofs.** nspawn mounts the
store into a namespaced container idmapped, and the kernel refuses that on
overlayfs and on some other filesystems. `WriteNspawnFlags` for an opted-in
agent checks the `statfs` type of `/nix/store` against an allowlist of those
four; the VM test's erofs store idmaps on the hive's kernel. The check
refuses every other filesystem, zfs and tmpfs included, even where a newer
kernel could idmap it, because the gate exists to stop a container that would
not boot. The error names the type and the fix, and
comes before anything moves. Check a host with
`findmnt -no FSTYPE /nix/store`.
- **The container starts with `--private-users=pick --private-users-ownership=map`.**
nspawn idmaps the root at mount time, so a different range for `pick` never
walks or chowns it.
- **The move happens once.** The first `WriteNspawnFlags` that finds the
agent's container stopped chowns those four dirs to `dataOwner` (ownership
only; it removes or copies nothing), writes
`/var/lib/hive-priv/userns/<name>`, and creates the mount points in the root
template owned by the agent user. A later request with the marker present
chowns nothing. A running container waits for its next stop. The socket dir
is on a tmpfs, so `EnsureAgentSocketDir` hands it to `dataOwner` at every
start. The walk works by open directory fd and follows no link, so an agent
or manager that can write the dirs can't redirect it to a host path.
- **A move that fails starts over.** hive-priv opens all four dirs and
checks the agent's ids before the first chown, so a link or file where a dir
goes refuses with nothing changed. It then writes the ids to
`/var/lib/hive-priv/userns/<name>.pending`. If the walk fails after that (a
chown error, a tree nested too deep), the next start repeats it from those
ids, then writes the marker and removes the pending file. hive-priv refuses a
pending file that doesn't hold `<owner> <uid> <gid>`, and names it.
- **The agent's gid must equal its uid.** `owneridmap` maps the gid of a bind
with the uid map, so a group other than the uid's has no mapping and writes
fail. The agent module asserts that `user.uid` and `user.gid` hold the
same number when `privateUsers` is on. Start the agent once with the pin
before turning it on: the move refuses an agent whose `state` dir has another
gid than uid, naming the agent.
- **Turning it off is manual.** hive-priv refuses to start an agent whose
marker exists while its record says false, and one whose `dataOwner`
changed. Chown the four dirs back, then delete the marker and any
`<name>.pending` beside it.
- **Keep a `ManageRootAgent` holder off it.** The other agents' dirs it mounts
aren't remapped.
The pause marker needs nothing extra: hive-priv gives it the owner of the
`harness/` dir. hive-c0re still creates agent dirs and new subvolumes as
`hive-core`, and only the first namespaced start chowns them. A dir created
after the move, such as a recreated subvolume, stays `hive-core`'s and the
agent can't write to it.
### Cross-agent access to state
An agent holding the `manage_root_agent` permission gets every other
agent's `state` dir bind-mounted **read-write** (`bind_child_agent_dirs` in
`lifecycle/host_config.rs`). The RW is deliberate, not an oversight: the
holder recovers other agents, which includes writing into their state (for
example seeding notes, clearing a stuck sentinel) as well as reading it.
This is the **only** cross-agent mount. An agent without that permission
sees its own dirs and nothing else.
**`harness` isn't mounted at all.** It holds that agent's own runtime
material - `bash-tasks/`, the turn-stats and event sqlite dbs - and
nothing argues for anyone else reading it, let alone writing it.
hive-c0re reads a harness dir **directly on the host** when it wants
those stats, which needs no mount into another container.
**`config` isn't mounted either.** A config change is a PR on that agent's
config repo, made from a clone and merged after review. Anyone who wants to
read the config clones `agent-configs/<name>` from the forge. A mount, even a
read-only one, would show an agent a tree that's not the one that governs it
whenever the forge has moved on.
Isolation still holds by default: a container has its *own* dirs and,
unless it holds the permission, nothing else.
Under `/var/lib/hyperhive/applied/` — the hive-c0re-only deploy markers
`.<name>.closure`: the store path of the closure the agent was last swapped to
or created from, or, where the file was missing, the closure the container's
profile pointed at when the hive first read it. No config lives here; the
forge's `agent-configs/<name>` is the only copy. A hive that predates this
finds per-agent `applied/<name>` repos here. Nothing reads them any more, and
you can delete them.
Under `/var/lib/hyperhive/meta/` — system-level config files, plain state
with no git history. The flake agents build from is the
swarm's `internal/meta`, not here. A hive that predates this finds a
leftover `.git` here; `migrate::run` deletes it at startup. Contents:
- `topology.json` — the agent roster (`["alice", "bob", "ruth"]`).
Written by `topology::reconcile` from `agent_set::refresh`; read by
`topology::all_agents`, which is the set the `manage_root_agent`
permission grants mounts over. The reader also accepts a map-shaped
file carrying a `parent` per agent, and keeps its keys.
- `swarm-resource-limits.json` — the caps the swarm sets for this
hive's agents
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
Written only by `workers::swarm_limits`, which copies each agent's key of
the swarm's `agent-limits` bucket here every 30 seconds. Read where
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
into the container — these are host-side caps on the container, so
the capped party never sets them. It sees only the resulting memory cap,
as a byte count or `infinity` in its runtime dir (below). Fallback is per
*field*: an absent file, absent agent, or absent field falls back to the
hive-wide `services.hyperhive.deploy.hive-controller.agentCpuQuota` / `agentMemoryMax`,
so an agent can have only its memory capped and still track the hive
default for CPU. Every value passes the validators first, and one bad
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
shares in the same drop-in have **no** per-agent setting — they're
hive-wide only and come straight off `HiveEnv`, so this file has no field
for them.
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
to live here; nothing reads or writes it now. An old copy in this directory
is harmless and you can delete it.
An agent holding `manage_root_agent`, normally ruth, has the meta dir
RO-mounted at `/meta/`.
Outside this directory, `/var/lib/hyperhive/swarm-agent-permissions.json`
holds the permissions the swarm grants this hive's agents, as
`workers::swarm_permissions` last read them from the swarm's
`agent-permissions` bucket (every 30 seconds): a copy, not the
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
reads it at every container start for `manage_root_agent`.
`/var/lib/hyperhive/swarm-agent-archetypes.json` holds the archetype each of
this hive's archetype agents runs, as last read from the swarm's
`agent-archetype` bucket at the agent's provision and each rebuild. An agent
with an entry has no config repo and no `agent-<n>` meta input; see
[archetypes](../swarm/archetypes.md).
hive-c0re creates containers from the closure the swarm builds from `internal/meta` (`nixosConfigurations.<n>`).
## Destroy vs purge
See [For operators](#for-operators) above for what each action does to
an agent's state. The mechanics, for completeness:
- `DESTR0Y` also drops the systemd drop-in; reviving the agent is
`swarmctl agent create` with the same name and hive, which reuses the
kept state.
- `PURG3` wipes `/var/lib/hyperhive/agents/<name>/` — the
union of everything `DESTR0Y` left behind.
`actions::destroy` has no guard for the root/bootstrap agent: it destroys ruth
like any other agent, and she stays gone until you create her again from the
swarm dashboard.
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
On a btrfs host, `lifecycle::ensure_agent_state_subvolume` creates a brand-new agent's state root as a
**btrfs subvolume** instead of a plain directory (progressive
enhancement). This is a no-op fallback on
non-btrfs hosts and for any agent whose root already exists, so
hive-c0re automigrates nothing: existing agents keep their plain dirs
until an explicit opt-in upgrade.
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
hive-c0re creates the per-agent subdirs (spawn / rebuild).
It skips the work when the root already exists; otherwise it asks
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
root when the FS is btrfs (`statfs` magic gate) and chown it to the
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
succeed inside it.
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
reuses it untouched.
- **PURG3 deletes it correctly:** `rmdir`/`remove_dir_all` can't remove
a subvolume root, so purge first calls hive-priv
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
actually a subvolume, then the normal `remove_dir_all` sweep covers
plain-dir agents.
Per-subvolume disk-usage accounting and optional quotas have since
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
`hivectl agent <name> quota show|set` reads/limits one agent's
subvolume usage through the same hive-priv-mediated path as
subvolume creation/deletion above.
This is the same subvolume `hivectl agent <name> subvol snapshot push`
sends to the swarm's snapshot store — see
[`docs/4-implementation/networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) for what a pushed
snapshot contains and how the store authenticates a sender.
The subvolume is the whole agent state root, so `work/` is part of every
snapshot and migration of it and can make them much larger.
## `/var/lib/swarm-controller/` (swarm-controller host only)
Only present on the one host running
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
so it survives restarts and redeploys.
- `webhook-secret` — the HMAC key Forgejo signs the swarm's forge webhooks
with, as it was before the swarm secret store held it. **Keep it.** The
controller stores its value in the swarm secret store at
`swarm/controller/swarm-controller/forge/webhook-secret` the first time it
starts with that path empty, and reads the store from then on. The file is the
fallback while the store is unreachable or its policy doesn't grant the path
yet, and nothing else reads or removes it. A host with neither the file nor a
store value gets a new secret in the store, with nothing to configure.
<!-- vale write-good.Passive = NO -->
If the store can't answer and the file is unreadable or missing at startup, the daemon still starts and logs
`webhook secret unavailable`; the webhook endpoint then answers 503
rather than accepting deliveries it can't verify. Everything else the
controller serves is unaffected. The daemon keeps reading the store in the
background, at most five minutes apart, and the endpoint and the forge hook
registration start working when it answers. It writes a new secret only when
the store answers that it holds none, never because a read failed.
<!-- vale write-good.Passive = YES -->
- `operator-matrix-admins.json` — the matrix user ids the operator admin
sync has made homeserver admin, a JSON array. The sync revokes admin
only from these, so deleting the file leaves every current admin in
place; a missing file is an empty set. A file the controller can't
read or parse stops the sync, grants included, and the controller logs
`operator matrix admin: pass failed` on every pass. See
[`docs/2-handbook/integrations/matrix.md`](../integrations/matrix.md#operators-are-homeserver-admins).
## Run-time dirs
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
Without that, every restart wipes bind sources and existing
containers can't start.
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
(bind-mounted into the container as `/run/hive/mcp.sock`).
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
`MemoryMax=` as a byte count (a percentage resolved against the host's
physical memory) or `infinity`, written by `lifecycle::write_dropins`
before every start. Absent only when hive-c0re can't tell the cap. The
container reads it at boot as `/run/hive/memory-max-bytes`
(`nix/agent-modules/memory-cap.nix`).
On startup, `Coordinator::register_agent` drops any prior socket
task before rebinding — idempotent so a hive-c0re restart followed
by `rebuild alice` recreates the agent's socket without a clean
reinstall.
## First-boot agent-user migration
The harness runs as a per-agent unix user inside the container
(`services.hyperhive.agent.user.name`, defaults to the agent's logical label so each
container has a uniquely named user). The activation script below
moves root-owned state to the agent user so the claude session survives.
`system.activationScripts.hive-agent-user-migrate` (in
`nix/agent-modules/user.nix`) runs on every activation,
marker-guarded so the substantive moves only happen once per
container lifetime:
1. **`${homeDir}` exists with the right ownership** — covers the
the first boot before `useradd`'s `createHome` has had a
chance to chown. Also re-applies on every rebuild in case the
meta-flake's per-agent name evolves (rare).
2. **Migrate any leftover `/root/.claude` content into
`${homeDir}/.claude`.** Marker
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
only written once there's nothing left to migrate, so a `cp`
failure leaves it absent and the next boot retries. The root is
ephemeral, so the marker vanishes at every stop and the step
re-runs; with no `/root/.claude` it only writes the marker again.
`cp -an` (no-clobber) so any pre-existing files at the new
location win — never blow over data already there, and so a
retry after a partial copy is as safe as the first attempt.
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
recursively so the agent user can read/write it. Wildcard
matches the single agent that container sees; `-h` skips
symlinks the agent might have planted.
4. **Chown the `~/.claude/` bind-mount** recursively. The harness reads
`~/.claude/` as the agent user to decide Online vs NeedsLogin in
`login::has_session`. Without the chown, credentials owned by
another uid get silently treated as "no session" and the operator
re-prompts every boot.
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
user**, `0751`, not recursive. hive-priv creates it `0751 root`
on the host before every start; the harness binds its sockets
there as the agent user. Why the mode matters:
[`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
A namespaced agent has the dir bound at `/agents/<name>/sockets` and
linked from `/run/hive-agent/<name>`; the activation chowns nothing, because
a mount point under `/run` would belong to root and the id mapping would
give the agent user no ids there. A later in-container activation sees the
link and logs `is a symlink or not a directory; not handing it`. The log
is harmless.
## Matrix per-agent daemon + token-arrival trigger
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
process per agent. Serves its MCP tools directly over
streamable-http (`services.hyperhive.agent.mcp.matrixHttpPort`, no stdio bridge —
same shape as `hive-bash-daemon`), emits hyperhive wake signals
on incoming room events via `/run/hive/mcp.sock`. Conditional on the
agent having at least one `services.hyperhive.agent.matrixAccounts`
entry — there is no separate enable switch, and the same condition gates
the daemon, its path watcher AND the autoinjected
`extraMcpServers.matrix` entry. The module declares the hive-internal
`main` entry whenever `services.hyperhive.agent.matrix.url` is non-null,
so on a real hive (where hive-c0re renders that URL per agent) every
agent has one; a `null` URL with no operator-declared account is the
"this agent has no matrix" state.
**First-boot ordering**: a token can arrive after the container comes
up — a declared token file, or a token the swarm mints into the store.
Without the path-trigger sibling
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
<this agent's state dir>/matrix-token*` — the trailing `*` also catches
a declared secondary token file like `matrix-token-ccc`), the daemon
would exit 0 quietly the first time it ran and the MCP would have no
daemon until the next restart. The `.path` unit makes the appearance
of the token re-fire the service so the daemon comes alive in the
same boot cycle as
provisioning. A token in the store changes no file, so a store-backed agent
also gets a timer that restarts the daemon five minutes after it last
exited. A running daemon re-lists its linked accounts in the store
every two minutes and, when the set changed, exits with status
75, which the unit's `RestartForceExitStatus` restarts. On the same
two-minute tick it tries again each secondary account that didn't come
up — a declared account with no token yet, a token the homeserver
rejected, or a homeserver that
was down — waiting two minutes after its first failure and twice as long
after each further one, up to sixteen minutes. It tries one account at a
time and counts a try that hasn't finished after two minutes as a failure,
so one hung account doesn't hold the others back. One that comes up joins
the running daemon; the primary and the other accounts aren't
restarted. The same token watcher also drives avatar setting: on a
restart the daemon re-runs each account's bring-up, which sets the
avatar (see below).
### matrix avatar (set by the daemon over the live Client)
`hive-matrix-daemon` itself publishes the agent icon (`services.hyperhive.agent.icon`, an SVG) as each matrix
account's profile avatar
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
the daemon builds + restores an account's `Client` (authenticated,
pointed at that account's resolved homeserver), it calls matrix-sdk's
`account().upload_avatar()` — one call that uploads the media and sets
`avatar_url`. Because it reuses the live Client, there is no hardcoded
homeserver URL, no token re-read, and no token-file globbing: the daemon
already iterates every configured + dashboard-discovered account in its
bring-up loop, so it sets the avatar for **every** account.
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
unit, gated on `services.hyperhive.agent.icon != null`. No icon configured → the env is
unset → `sync_avatar` returns early and sets no avatar.
<!-- vale write-good.Passive = NO -->
Idempotency is **per-account**: an `avatar-icon-hash` file in each
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
skips the upload when unchanged, because every upload mints a fresh
`mxc://` URI that emits a profile state event in every joined room —
re-uploading identical bytes is timeline spam. A swarm-UI-linked
account gets its avatar when the daemon restarts on a changed account
listing (exit 75, which re-runs the per-account bring-up), so no separate
avatar trigger is needed. The daemon swallows avatar failures
(logged, non-fatal) so they never break account bring-up or sync.
<!-- vale write-good.Passive = YES -->

395
docs/2-handbook/integrations/forge.md Normal file
View file

@ -0,0 +1,395 @@
# Swarm forge (Forgejo)
Private Forgejo instance running in a nixos-container (`hive-forge`, the
container and CLI name), used as the
swarm's persistent code-collaboration surface (issues, PRs, reviews,
attachments). Configured via `services.hyperhive.swarm.forge.*` (what the
forge IS to every hive — names, ports, URLs) plus
`services.hyperhive.deploy.forgejo.*` (what the host running it decides —
which build it runs, how it's served, what it mirrors, where its
host-local secrets sit).
Container
shape, ROOT_URL / sub-domain routing, and operator-vs-in-cluster URL
handling live in [`docs/4-implementation/networking/gateway.md`](../../4-implementation/networking/gateway.md); this file owns the
per-agent integration story and the notification pump that wakes
each agent on relevant activity.
## Token scopes
Two scope sets:
**`AGENT_TOKEN_SCOPES`** (swarm-controller's `forge::agent_token`, pinned
by a test). swarm-controller mints every agent token with it:
| Scope | Why |
| -------------------- | ------------------------------------------------------------------------------------------------------- |
| `write:repository` | Create, clone, push, delete repos; merge PRs. |
| `write:issue` | Open / comment / review issues **and** pull requests (Forgejo namespaces PR conversation under issues). |
| `write:user` | Edit own profile, create repos under own user. |
| `write:organization` | Create + manage orgs (lets agents share a forge namespace). |
| `read:user` | Token-owner endpoint used for self-identification at harness startup. |
| `write:misc` | Hooks, attachments, the rest of the long tail. |
| `read:notification` | Poll `GET /notifications` for unread events. |
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
**`HIVE_TOKEN_SCOPES`** (swarm-controller's `forge::hive_token`, pinned by a
test): `read:repository` alone, for each hive's read-only account (next
section).
---
## How a hive reads the forge
Each hive in swarm-controller's hive directory has its own forge account,
`hive-<hive>`. swarm-controller creates it, makes it a member of the `hives`
team in `agent-configs`, and mints one token named `swarm-hive` into the
swarm secret store at `swarm/hives/<hive>/forge/token`. The team has read
permission on `repo.code` and `repo.pulls` of every repo in the org,
including repos created later. A pass at start and every five minutes
creates what's missing and re-mints a token that's missing or no longer
matches the forge. A pass whose store or forge reads fail queues nothing, so
an outage never rotates every hive's token.
swarm-controller refuses to create an agent whose name starts with `hive-`:
these accounts own that prefix. An agent that already had such a
name keeps it, so the accounts are also told apart by email. swarm-controller
creates the hive's account with the email `hive-<hive>@hives.hyperhive.local`
and refuses an account by that name with any other email: it never adds an
agent's account to the team or mints a token for it. An agent's own forge
account setup refuses the hive's account the same way.
hive-c0re reads the token with `forge::reader_token()`, under the hive's own
store certificate, every time it reads the forge. A hive whose store holds
none reads nothing. These reads go through it, and none of them needs a local
forge container:
- cloning and pulling `/knowledge`
The hive builds nothing, so no nix process reads the forge as the hive's
account. hive-c0re hands the token to each `git` command it runs
(`http.extraHeader`) and keeps no copy on disk. A hive whose store holds no token
reads nothing private from the forge. This also holds on a hive with no forge
container.
Roll a change to any of this out store host first (its granter writes the
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
then the hives. A hive that starts before its token exists reads nothing, and
builds nothing, until the controller's next pass.
---
## Per-agent forge accounts
Each agent gets its own Forgejo user and access token. swarm-controller
creates the user when it creates the agent, and mints one token named
`swarm-agent` with the admin API into the swarm secret store at
`swarm/agents/<agent>/forge-token`. A pass at start and every five minutes
covers every agent with a store identity: it creates a missing forge user and
re-mints any token that's missing or no longer matches the forge;
a rotation deletes the old `swarm-agent` token first, so each agent holds at
most one. The agent fetches the token under its own store certificate into
`/run/hive-agent-forge-token/token` (`nix/agent-modules/forge-token.nix`),
and `hive-forge`, the git credential helper, the `forge_notify` poller and
the avatar sync read it from there, falling back to `<state>/forge-token`,
the file hive-c0re wrote before. hive-c0re no longer creates agent users or
mints agent tokens. swarm-controller deletes the `hyperhive-<unix-seconds>`
tokens it left behind, for each agent whose current `swarm-agent` token is
live; tokens of other names are never touched.
Two things live in the `agent-configs` Forgejo organization:
- A config repo per agent (`agent-configs/<name>`). The
agent is a **write collaborator on its own** repo — it can push
config-change branches and open config PRs — but `main` is
branch-protected by swarm-controller: the merge and approval
allowlists are the `operators` team, and the agent can neither push
`main` directly nor self-merge. An operator's merge in the Forgejo UI
deploys the merged commit (see
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes)).
hive-c0re pushes nothing to this org and reads nothing from it.
Repos stay private, so an agent can't read another
agent's config. The agent's **persistent meta flake input** is this repo's `main` head,
pinned by sha in `internal/meta` (`agent-<n>.url =
git+<forge>/agent-configs/<n>.git?rev=<sha>`; see
[config-changes.md § Meta flake](../agent-lifecycle/config-changes.md#meta-flake)).
The `hive-forge` CLI (separate workspace crate, see
[`README.md`](../../README.md) file map) wraps the Forgejo REST API
with the per-agent token; agents call it for issue / PR / comment
ops as if it were a peer. All REST calls across the workspace
(`hive-forge` verbs, hive-c0re provisioning, this poller) go through
the typed `forgejo-api` crate; only non-`/api/v1` web-router routes
(attachment / artifact downloads, log streaming) and the poller's
enrichment fetches of server-provided subject URLs stay on raw
reqwest.
## Notification poller (`hive-forge-notify/src/notify.rs`)
Its own long-running per-agent daemon (`hive-forge-notify`, a sibling of
`hive-bash-daemon` / `hive-matrix-daemon`) — it used
to be a background task inside the `hive-agent` serve loop. Polls
`GET /api/v1/notifications?all=false` every 30 seconds (Forgejo's
unread-only filter), formats each notification as a broker
`Wake { from: "forge" }` message, and delivers it to the agent's own
inbox so claude's normal turn loop picks it up.
The crate builds a second, independent binary for a different host — see
[github.md](github.md#notifications).
The host-specific calls — list unread, mark read, resolve own login —
live behind `Source` in `hive-forge-notify/src/source.rs`;
classification, formatting, dedupe and todo delivery live in `notify.rs`,
common to both binaries. Todo
keys here are the bare thread ids, and must stay that way: renaming them
would orphan every in-flight forge todo on the first restart after a
deploy.
### Mark-read on delivery
On a **successful** broker delivery, `forge_notify` marks the thread
read on forge straight away (`PATCH /notifications/threads/{id}`). The
broker inbox is the durable work queue now — each delivered wake is a
sqlite row with its own ack lifecycle — so the forge unread flag no
longer needs to track whether the agent has _processed_ a
notification. Clearing it on delivery keeps forge's unread set **small
by construction**: at rest it holds only threads that failed to
deliver plus whatever arrived since the last 30s poll.
That size property is the whole point. A container rebuild starts the
poller with no memory of what it delivered, re-scans `?all=false`, and
finds nothing stale — the delivered threads are already read on forge.
Forge's own read-state is thus the durable, cross-rebuild record of
what `forge_notify` has delivered; there is **no persisted cursor**.
**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
that leaves threads unread and tracks delivery via a separately persisted
cursor is fragile — losing that cursor across a rebuild re-delivers the
agent's entire still-unread backlog as a flood of fresh wakes. Forge's
own read-state is the only durable record this design needs.
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
read-before-comment guard (which keys off forge unread-state) doesn't
force the agent to view a thread before commenting on it: the broker
wake already carries the notification body, so _delivery is the read_.
An agent that wants the full thread still runs `hive-forge comments` /
`view`.
**In-process dedupe (small, ephemeral).** A single-process map (thread
id → last-delivered `updated_at`) guards the narrow window where a
mark-read call _transiently fails_ and the thread reappears unread in
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
re-fire the wake. it's **not persisted** and resets on restart (forge
read-state covers the durable case). Each poll prunes it to the ids in
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
exceed that many entries (a debug assertion pins the invariant; the
fetch limit and the bound are the same constant). `forge_notify` leaves
a failed _delivery_ unread and out of the map, so it resurfaces next tick.
`forge_notify` marks self-echo notifications (the agent's own writes, see below)
read directly without a delivery — same `mark_read` call, no wake.
### Activation gates (graceful no-ops)
The poller starts switched off and stays that way for any of:
- `HIVE_FORGE_URL` not set or not parseable as a URL (a hand-run
harness; hive-c0re sets it from `swarm.forge.domain` on every hive).
- no token in `$HIVE_FORGE_TOKEN_FILE` or `<state>/forge-token` (agent has no forge
account — pre-provisioning or destroy-without-purge race).
- Initial client construction fails (the typed `forgejo-api` client
for the API calls, or the plain reqwest client kept for the
best-effort enrichment fetches of server-provided subject URLs;
both unlikely; treated as fatal-to-the-task only).
Switched off = the spawned task returns immediately. All other failure
modes (HTTP errors, parse errors, mark-read failures) are
best-effort: logged at debug/warn and retried next tick.
### Self-notification filtering
Forgejo fires notifications for the agent's own actions (it opened a
PR, posted a comment, submitted a review). Surfacing those would
loop claude on its own writes. `forge_notify` drops the comment/review case
silently (mark-read without delivery):
- **Self-authored comments / reviews** — comment payload's
`user.login` matches `own_login`.
- **Self-authored creations** (an agent opening its own PR/issue) — the
already-fetched subject payload's poster `user.login` matches
`own_login`. `forge_notify` drops only _creations_ this way; a later state
change on the agent's own subject comes from someone else, so it still surfaces.
`forge_notify` fetches `own_login` at startup via `GET /api/v1/user`. On fetch
failure the filter degrades open (no filtering) rather than crashing
the task — a noisy inbox beats a silently stuck poller — but the fetch
is **re-attempted on each poll tick** until it succeeds, so a boot-time
failure (the forge not yet reachable) self-heals instead of leaving
self-echo filtering off for the whole process lifetime.
### Body excerpt + truncation + heading escape
The wake message embeds the comment / review body so the agent sees
actual content without a follow-up fetch — **not** a new issue/PR's own
description, which the wrapper table below deliberately omits. Three
pipeline steps in order:
1. **Truncate** to `BODY_TRUNCATE = 500` chars at a char-boundary;
appends `…` when cut. Truncation happens BEFORE escape so the
mention-overflow diff (next step) compares like-for-like against
the raw body.
2. **Mention overflow extraction** — when truncation actually
trimmed content, walk the full body line-by-line and surface any
`@username` lines that fell outside the embed window. Rendered as
a trailing `mentions (truncated from body):\n > <line>` block.
Mention detection requires the `@` to be at line start or
following a non-username byte, so email-style `foo@bar.com` does
NOT count.
3. **ATX heading escape** — for each line that's a strict
`CommonMark` ATX heading (1-6 leading `#`s followed by a space,
tab, or end-of-line), prepend `\` so the embedded body doesn't
blow into a top-level h1/h2 inside the wrapper message when the
dashboard renders it. Lines like `#tag`, `#123`, `#!/bin/bash` <!-- lint:allow: syntax examples, not real tags -->
**aren't** headings — no escape, no cosmetic noise. Indented
"headings" inside lists / nested quotes keep their leading
whitespace.
The strict ATX rule is deliberate: `\#tag` and `#tag` render
identically, so an over-eager escape just adds visual clutter
without changing behavior. Setext-style headings (`title\n====`)
aren't handled — rarer in practice, would need multi-line
lookahead.
### Wrapper format
Five shapes, distinguished by the notification's classification:
| Trigger | Wrapper |
| ----------------------------------- | ------------------------------------------------------------------------------ |
| Comment on issue / PR | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
| Review submission | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
The issue/PR's own description is deliberately never embedded here (only a
comment/review body is — that's the actual new content a notification
exists to surface): repeating a subject's own description on every "new"
or "activity on" wake for it's noise the agent already has via the URL,
not context (mara, triage: "just the issue title should be enough").
Review labels come from the Forgejo `state` field: `APPROVED` →
`approved`, `REQUEST_CHANGES` → `changes requested`, `COMMENT` →
`review comment`. `forge_notify` drops `PENDING` (review saved but not
submitted yet — no peer-visible event). Unknown states fall back to
the generic comment wrapper.
A review submitted with **no body** renders `reviewed by: <author>` in
place of the `<author>: <body>` line — deliberately worded to not collide
with the meta-suffix `reviewer:` line (requested reviewers, below).
### Inline review comments
A review's line comments belong to neither the review body nor the
`latest_comment_url`. A review with only inline comments has no such url
and would otherwise land as a bare `activity on PR`. Every PR wake therefore
fetches the PR's reviews (`/pulls/N/reviews`, up to four pages of 50) and,
for the reviews new to the agent, appends an
`inline review comments (read them with `hive-forge pr reviews N`):` block
saying where they sit: one `author: path:line, path, ...` line per review
author (bare `path` when Forgejo dropped the position because the line left
the diff; control characters in a path become spaces). The comment text stays
out of the wake, and the agent reads the full review with the CLI.
A review counts as new when it landed strictly after the thread's previously
delivered `updated_at`. Without a baseline every review counts, and so does
one with no submit time, so nothing disappears. It doesn't count when the
agent wrote it, when it's a `PENDING` draft, or when `comments_count` is 0.
The notifier fetches the newest 5 such reviews and lists at most 10 locations
(`(+N more)` for the rest). Best effort: a failed fetch yields less text and
never drops the wake. Cost: one reviews fetch per PR wake, plus one per new
commented review.
### Merge/close vs a later comment
A notification carrying a `latest_comment_url` normally takes the comment
path. But a merged/closed subject **keeps** its `latest_comment_url` set,
so a just-merged PR that had any prior discussion would route to the
comment path and render `[comment on PR]` (with a stale pre-merge comment
body) instead of `[PR merged]` — the agent never learns its PR merged.
When the notification IS the merge/close transition — its
event time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` of the
subject's `closed_at` (set for both `merged` and `closed`) — the
state-change path wins even with a comment url present
(`state_change_is_current`). A genuine **later** comment on an
already-closed subject bumps `updated_at` well past `closed_at`, so it
stays on the comment path and keeps its comment body. Missing/unparseable
timestamps default to the state-change path, so a merge is never silently
hidden behind a stale comment.
#### Merge racing a comment
The one gap the timestamp cut leaves: a genuine comment posted **within
`NEW_ITEM_TOLERANCE_SECS` of the merge** bumps `updated_at` close enough
to `closed_at` that `state_change_is_current` returns `true` — so it takes
the state-change path, dropping its body. Best of both worlds:
on the merge/close path `forge_notify` fetches the `latest_comment_url` comment and, when
its `created_at` is strictly **after** the subject's `closed_at`
(`comment_is_after_close`) — in other words, it raced the merge rather than being the
pre-merge last comment the subject keeps — append it as a
`comment by <author>: <excerpt>` block before the meta suffix
(`fresh_post_close_comment_tail`). The wake carries **both** `[PR merged]`
and the racing comment. `forge_notify` leaves the kept pre-merge comment (created before
`closed_at`) off, drops a self-authored racing comment (don't
echo the agent's own write), and appends nothing for a missing/unparseable `created_at`/
`closed_at` (conservative — only surface a comment it can
positively place after the close). Cost: one extra comment fetch on
merge/close notifications, acceptable given how rare they are.
### "new" vs "activity on"
A review submitted with **no body** carries no `latest_comment_url`,
so it misses the comment path and lands on the state-change path with
`state == "open"` — exactly like a freshly opened PR. Labeling that
`new PR` is misleading: agents dismiss it as a duplicate of the
original open notification and miss the review. The `open`
state only earns the `new <kind>` label when the notification's event
time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` (120s) of the
subject's `created_at`. `forge_notify` labels anything later `activity on <kind>`
— neutral and non-misleading, since it can't cheaply say _what_ the
activity was without an extra reviews fetch. Missing/unparseable
timestamps default to `new` (preserve prior behavior rather than mask a
genuine new item). A small dependency-free
RFC 3339 helper (`parse_rfc3339`) parses timestamps.
`forge_notify` extracts the number from `subject.html_url`'s last path segment
(strips `#anchor` first), and the repo slug from `repository.full_name`.
Both degrade gracefully when absent (number → blank, repo → blank)
so unexpected Forgejo shapes don't crash the formatter.
### Meta suffix
Every wrapper ends with one or more of:
- `assignee: <list>` — always present; `unassigned` when empty so
the line shape is stable.
- `reviewer: <list>` — PR notifications only, present only when
`requested_reviewers` is non-empty.
### Review-request override
For new PRs, the kind label flips to `[review requested #N
owner/repo]` when `own_login` appears in `requested_reviewers`,
regardless of the Forgejo `reason` field. Forgejo doesn't reliably
set `reason == "review_requested"` (often null instead), so the
fallback checks the subject payload directly. `is_new` gates detection
so the label only fires once on PR creation, not on every
subsequent comment.
### Subscription management
The poller does **not** autounsubscribe from repo watches — it
delivers every unread notification it's handed. A hive-forge CLI
subscription verb bounds the firehose explicitly (dropping broad repo
watches an agent doesn't need) — the poller never guesses which
watches to drop. See the `subscription` verb in
[`docs/2-handbook/tools/forge.md`](../tools/forge.md).

150
docs/2-handbook/integrations/github.md Normal file
View file

@ -0,0 +1,150 @@
# GitHub accounts
Give an agent a managed GitHub identity — a `gh` CLI and `git push` over
HTTPS, both authenticated by an operator-supplied personal access token
(PAT) — so it can run GitHub API calls and push commits without any manual
`gh auth login`.
Provisioning is UI-driven: link a PAT to the agent in the swarm UI and it
works. No per-agent nix declaration, no rebuild — the agent fetches the token
from the swarm secret store itself.
## Enabling
<!-- vale write-good.Passive = NO -->
The integration is **on by default** for every agent (`services.hyperhive.agent.github.enable
= true`), inert until the operator provisions a PAT. No per-agent declaration is
needed — an agent gains GitHub by having a PAT stored for it.
<!-- vale write-good.Passive = YES -->
To turn it off for the whole swarm, set the swarm option:
```nix
services.hyperhive.swarm.github.enable = false;
```
hive-c0re's meta-flake renderer then injects `services.hyperhive.agent.github.enable = false`
into every agent, so no agent ships the `gh` wrapper, the credential helper
or the token fetch.
(`services.hyperhive.agent.github.enable` also exists per-agent for completeness, but the
swarm-wide switch is the intended control.)
github.com only. The token **value** never touches nix — the agent writes it
to `<state>/github-token` at runtime (see [Provisioning](#provisioning)).
## Provisioning
The PAT is operator-supplied. In the [swarm UI](../swarm/ui.md#linking-external-accounts),
open the agent on `/agents`, choose **link github account** and paste the PAT
(`PUT /api/hives/{hive}/agents/{agent}/github-account`). swarm-controller
stores it at `swarm/agents/<agent>/github-token` in the swarm secret store;
no hive writes it. One token per agent: swarm-controller refuses to link
another until you delete the stored one, and no route hands it back.
The agent's `hive-agent-github-token` unit reads that path under the
agent's own store certificate and writes `<state>/github-token` (`0600`,
owned by the agent), on boot and every two minutes, replacing the file only
when the token changed. It needs a store identity
(`services.hyperhive.agent.bao.addr`); an agent without one gets no token.
The unit never deletes the file: a `github-token` already in place stays
when the store holds none or doesn't answer. Where the token lives and who
reads it: [credentials-reference.md](../swarm/credentials-reference.md).
## Security
- Use a **dedicated bot account**, never a human's.
- Mint a **minimally scoped PAT** — only the repos/scopes the agent's
workflow needs. Agents have passwordless sudo, so a compromised or
hallucinating agent can act as the account within the token's scopes;
scope is the real blast-radius limiter, and the container boundary is
the enforcement. See [security.md](../trust-boundary/security.md).
## Implementation
<details>
<summary>How the agent's `gh`/`git push` actually authenticate, and how
the notification poller works</summary>
### How the agent uses it
When enabled, the container gets:
- **A `gh` wrapper** on `PATH` (shadowing the raw `gh`) that exports
`GH_TOKEN` from the token file at invocation, then execs real `gh`. So
`gh pr create`, `gh api …`, etc. just work — `gh` derives the identity
from the token.
- **A git credential helper** (`git-credential-hive-github`), wired via a
host-scoped `/etc/gitconfig` entry for `https://github.com`, so
`git push https://github.com/<owner>/<repo>` authenticates as
`x-access-token` + the PAT (GitHub ignores the username for PAT auth).
Host-scoped, so it never touches the forge (`localhost:3000`) or any
other remote.
Both scripts read the token from `<state>/github-token` **at invocation
time**, so a PAT written (or rotated) mid-session takes effect immediately
— no container rebuild or restart. Until the file exists, `gh` / `git push`
fail unauthenticated. The build bakes the token path into the scripts
(rather than reading it from an env var), because claude's Bash tool runs in a
minimal environment that wouldn't carry one.
### Notifications
`hive-github-notify` polls github.com for the agent, turning each unread
notification thread into a todo. it's a **separate binary and a
separate systemd unit** from the internal forge's poller
(`hive-forge-notify`, see [forge.md](forge.md#notification-poller-hive-forge-notifysrcnotifyrs)),
installed by `nix/agent-modules/github.nix` under
`services.hyperhive.agent.github.enable`. Both binaries ship from the one
`hive-forge-notify` derivation, so the unit is a second `ExecStart`
path, not a new package.
Two units rather than one daemon with two loops, because it puts the
decision in nix: a hive built without this module has **no github poller
in its closure at all**, which is what makes GitHub access separable
rather than merely switched off. it's also why this isn't a cargo
feature — a feature would unify across the workspace and cost every
crate its build cache.
At runtime the poller needs the PAT above. No PAT, no polling: the unit
logs why and exits 0, which is why it's `Restart = on-failure` and
never `always` — a clean exit on a PAT-less agent must not become a
restart loop. Once polling, it re-reads `<state>/github-token` on every
tick, so it picks up a rotated PAT on the next tick without a restart.
Forgejo models its notifications API on GitHub's, so one tolerant
parse serves both: `id`, `repository.full_name`,
`subject {title,url,latest_comment_url}` and `updated_at` line up field
for field. Lenient deserializers absorb the two real differences —
GitHub sends the thread id as a _string_ where Forgejo
sends a number, and says `PullRequest` where Forgejo says `Pull`.
`hive-github-notify` prefixes todo keys `gh:<id>` so a github thread id can't collide with a
forge one.
Two host differences worth knowing before touching this code:
- **Auth scheme, not just value.** Forgejo takes
`Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
application/vnd.github+json`, `X-GitHub-Api-Version` and a
`User-Agent`. Sending Forgejo's form to GitHub doesn't error — it
authenticates as _nobody_ and silently drops to the unauthenticated
rate limit. The cheap way to tell the two apart is the rate-limit
header: `x-ratelimit-remaining` near 5000 is an authenticated user,
near 60 is anonymous.
- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
practice, slower than the poller's own tick) and rate-limits callers who
ignore it, so the loop re-arms to the server's interval whenever that's
_slower_ than the poller's. A hint faster than the poller's own tick
isn't a reason to poll harder.
⚠️ **This needs the `notifications` scope on the PAT.** A token minted
for `gh` + `git push` typically carries `repo` only, which is enough to
push and open PRs but **not** to read the notification stream (nor to
mark a thread read, which is the same scope). A PAT without it doesn't
break anything: the poller logs the refusal and stays quiet, and the
agent never gets GitHub wakes. If an agent's GitHub
notifications never arrive, check the token's scopes first — the
symptom is silence, not an error.
</details>

91
docs/2-handbook/integrations/gitlab.md Normal file
View file

@ -0,0 +1,91 @@
# GitLab accounts
Give an agent managed accounts on one or more GitLab instances: a `glab` CLI
and `git push` over HTTPS, both authenticated by operator-supplied access
tokens, with no manual `glab auth login`.
An agent can hold any number of accounts, each under a label you choose. Two labels
can name two instances, such as `gitlab.com` and a self-hosted one. Provisioning
needs no per-agent nix declaration and no rebuild: the agent fetches its tokens
from the swarm secret store itself.
## Linking an account
In the swarm UI, open an agent, choose **link account**, then **gitlab
account**. Fill in a label, the instance URL (it starts as `https://gitlab.com`)
and the token. The form shows the recommended scopes and the controller's
answer if it refuses the input. The linked account appears in the agent's list
of [linked accounts](../swarm/ui.md#linking-external-accounts) with a delete
action. The UI calls the controller's API, which you can also call directly:
```text
PUT /api/hives/{hive}/agents/{agent}/gitlab-accounts/{label}
{"url": "https://gitlab.example.org", "token": "<access token>"}
```
- `label` is lowercase letters, digits and hyphens. It names the account in the
store and in the account list.
- `url` is the instance's base URL. The host must be a DNS name: an IP
address, `localhost` and a single-label name get a 400, because the agent
reaches the instance through its hive's egress proxy, which takes names
only. A port and a subfolder (`https://example.org:8443/gitlab`) are part of
the address. The path may hold letters, digits and `. _ ~ / -`; a URL with
user information, a query, a fragment or any other character gets a 400, so
every stored account is one the agent uses.
- A plain `http://` URL works, and then the token crosses the network in
cleartext. Use `https://` unless the instance sits on a trusted network.
- `token` is a GitLab personal, group or project access token. Scope it
`api` and `write_repository`: `glab` needs the first, and `git push` over
HTTPS needs the second. Use a dedicated bot account and the narrowest token
that does the agent's work, because the agent has passwordless sudo and can
act within the token's scopes.
A label that holds an account already answers 409; delete it first
(`DELETE .../gitlab-accounts/{label}`), then link again. No route hands a
token back. `GET /api/hives/{hive}/agents/{agent}/linked-accounts` lists the
account as kind `gitlab`, with its label and URL.
Linking also allows the instance's host in the agent's egress, as a
`link:gitlab:<label>` entry on the swarm's allowed-domains list. swarm-controller
derives the host from the URL it stored, so nothing the agent writes can widen
its egress. Deleting the account removes the entry.
## What the agent gets
An agent with a store identity (`services.hyperhive.agent.bao.addr`) has:
- the `glab` command, which reads its configuration from `<state>/glab/`;
- a git credential helper, `git-credential-hive-gitlab`, for each linked
instance;
- the `hive-agent-gitlab-accounts` unit and its timer.
The unit logs in to the store with the agent's own certificate, lists
`swarm/agents/<agent>/gitlab/`, reads each account, and writes two files in
`<state>/glab/` (mode `0600`) on boot and every two minutes, replacing a file
only when its content changed:
- `config.yml`, `glab`'s own configuration with one `hosts` entry per instance,
keyed by host and port, holding the token, the API protocol and the
subfolder. It sets `git_protocol: https`. The `host` default is the first
label's instance; inside a repository `glab` uses the host of its remote.
- `gitconfig`, one `[credential "<scheme>://<host>"]` block per instance, which
`/etc/gitconfig` includes. The helper answers only for a host that has an
account, so it never touches GitHub or the swarm's forge.
The unit generates both files. A deleted account leaves them on the next run, and
a hand edit doesn't survive. `glab` keeps one token per host: when two labels
name the same host, the first label in alphabetical order wins. If the unit
can't read one of the accounts, it replaces nothing and retries, so the files
keep the last complete set.
The unit fails fast and restarts itself every 30 seconds for up to a day, as
the other store fetches do, so a store that's down or sealed never holds the
container's boot open.
## Rolling out
The endpoint, the store path and the agent unit are new, and each end ignores
what it doesn't know. An old controller has no GitLab route, so no account gets
linked. An old agent has no unit, so a stored account sits unused. A new agent
under an old controller lists a store directory that doesn't exist and writes an
empty `glab` configuration.

156
docs/2-handbook/integrations/knowledge.md Normal file
View file

@ -0,0 +1,156 @@
# Swarm-wide knowledge repository
`internal/knowledge` on the forge is a shared reference doc repo
readable by every agent. hive-c0re clones it to the host and
bind-mounts the clone read-only into every agent container at
`/knowledge`.
## Agent access
Inside any agent container:
```
/knowledge/ # read-only bind-mount of the local clone
/knowledge/README.md # table of contents (seeded on first use)
```
<!-- vale write-good.Passive = NO -->
Agents read documents directly from that path. The mount is
read-only — agents never write through it. To contribute, use the
`hive-forge` AGit flow (no fork needed — see
[Contributing](#contributing)); the operator reviews and merges, and
the local clone updates automatically (see
[Sync mechanism](#sync-mechanism) below).
<!-- vale write-good.Passive = YES -->
## Repository layout
Canonical forge location: `internal/knowledge` (org `internal`,
repo `knowledge`). The repo is public, so every agent's forge account
has read access without an explicit per-agent collaborator grant;
only site admins can push: `core`, and swarm-controller's own account,
which seeds the README.
The swarm controller creates the repo if it doesn't exist, at startup
and every few minutes after, and seeds an empty one with a `README.md`
containing a contribution guide and a blank table of contents. Add an entry to that ToC each time you
create a new document.
## Sync mechanism
hive-c0re maintains the local clone at
`/var/lib/hyperhive/knowledge` via two paths:
1. **Swarm event** — the swarm controller holds the single push hook on
`internal/knowledge` (see `docs/2-handbook/swarm/README.md` § Swarm-wide forge
webhooks). On any push to main, including merge commits, it sends an
event to every hive over the swarm queue and each hive runs `git
pull`, so agents see the new content on their next turn.
A hive that's offline when the swarm controller sends the event
doesn't get it on reconnect — the periodic pull below is what closes that gap. One
hive briefly showing older `/knowledge` content than another is
expected, and resolves by itself within the fallback interval.
**don't add a per-hive hook.** A webhook has exactly one target
URL, so a second registration against the same repo doesn't add a
recipient — it takes delivery away from whoever registered first.
Earlier versions had each hive register its own, and later ones
removed that leftover at startup. A hive upgraded straight past those
versions still holds its old hook: delete any hook on the repo that
points at a hive's own `/webhook/knowledge`.
2. **Periodic pull** — a background task in `hive-c0re::main`
queues a pull on a fixed cadence as a fallback (webhook missed, c0re
restarted between pushes). The pull is best-effort — a failure
shows as a failed `KnowledgePull` node on the job queue, counts toward
the `knowledge_pull` hive warning, and doesn't affect the rest of the daemon.
Both paths, and the one pull at boot, queue the same `KnowledgePull` job
node. It holds the working tree for its duration, so two pulls never run
over each other — each trigger queues its own pass, and a pass that lands
while another is already queued or running waits its turn, since the
one ahead of it may have fetched before the push. The node runs
`knowledge::pull()`, which also handles the change notice below.
### Change notice
When a pull actually moves the local clone's `HEAD` (a real change,
not a no-op — for example the periodic pull finding nothing new), hive-c0re
broadcasts a short notice to every currently registered agent's inbox:
sender `system`, body `[system] /knowledge updated:` followed by a
`git diff --stat <old>..<new>` summary of what changed (or a generic
"see the repo" fallback if computing the diff itself fails). This is
the same broadcast mechanism used for other hive-wide notices — inbox
message only, no forced wake, and it carries the standard "this was a
broadcast" hint. hive-c0re logs a diff or per-agent send failure
without blocking the pull itself.
On a hive with no local clone yet, `knowledge::pull()` takes the clone
instead of pulling, so every hive gets `/knowledge` from its first pull,
whether or not it runs the forge. It authenticates with the hive's forge
read token when it has one (see
[forge.md](forge.md#how-a-hive-reads-the-forge)) and clones the public
repo anonymously otherwise.
## State
<!-- vale write-good.Passive = NO -->
- **Host clone**: `/var/lib/hyperhive/knowledge` — persists across
hive-c0re restarts and agent destroy/recreate. Deleted only by
manual operator action.
- **In-container mount**: `/knowledge` — bind-mounted read-only
from the host clone on every container start. Gone when container
is stopped; reappears on next start with the current clone state.
<!-- vale write-good.Passive = YES -->
The mount deliberately **excludes `.git`**: hive-priv overlays an empty
tmpfs at `/knowledge/.git`, so agents see the documents, not the branches or
history. An older clone also kept a token in `.git/config`; `knowledge::pull()`
rewrites that remote to the plain URL.
## Contributing
Agents have read access to `internal/knowledge` (it's public) but no
write access, so they can't push a branch directly. The supported path
is Forgejo's **AGit flow** through the `hive-forge` CLI — no fork
required.
1. Clone the repo (`hive-forge` injects credentials automatically;
the `-r` flag selects the repo, the clone lands in `./knowledge`):
```sh
hive-forge -r internal/knowledge clone
cd knowledge
```
2. Create a branch and add or update a document, then commit normally.
3. Open (or update) a PR with `--agit`. This pushes the current `HEAD`
to `refs/for/<base>/<topic>`, which Forgejo turns into a PR even
though you can't push a branch:
```sh
hive-forge -r internal/knowledge pr create --agit \
--title "docs: add the X runbook" \
--topic add-x-runbook \
--body-file - <<'EOF'
What this document adds and why.
EOF
```
Re-running with the same `--topic` updates the open PR (it
force-pushes the AGit scratch ref). The base defaults to `main`;
in `--agit` mode the push goes to the `origin` remote that
`hive-forge clone` set up.
4. The operator reviews and merges. The webhook fires on merge; every
running container sees the updated content within seconds (see
[Sync mechanism](#sync-mechanism)).
Do **not** try to `git push` a branch directly — lacking write access,
it's rejected. The `--agit` flow above is the no-fork path that works
from any agent.

442
docs/2-handbook/integrations/matrix.md Normal file
View file

@ -0,0 +1,442 @@
# Swarm Matrix homeserver
Private Matrix homeserver (matrix-tuwunel — the conduwuit
successor) wrapped in a nixos-container (`hive-matrix`), plus optional fluffychat-web
client at `chat.<swarm-domain>/` (the `gatewayHost` vhost). A swarm runs one
homeserver, on one host. Two namespaces configure it:
- `services.hyperhive.swarm.matrix.*` — what the homeserver **is**, as
every hive sees it: `serverName`, `gatewayHost`.
- `services.hyperhive.deploy.matrix.*` — what the host running it decides:
`enable`, `gui.package`, `trustedServers`,
`maxRequestSize`, `sso.clientSecretFile`.
Vhost routing lives in [`gateway.md`](../../4-implementation/networking/gateway.md).
## Container shape
Same shape as [`gateway.md::hive-forge container shape`](../../4-implementation/networking/gateway.md):
- Container name `hive-matrix` (not `h-*`) so c0re's lifecycle
scanner ignores it; operator manages via the standard
`nixos-container` CLI.
- Keeps hive-matrix from fighting any `services.matrix-*` the
operator already runs on the host — separate systemd namespace,
separate state dir.
- Container runs in a network namespace of its own
(`privateNetwork = true`) on the hive bridge, and binds no TCP port. Agents
reach the homeserver at `chat.<swarm-domain>` via the gateway, which
dials its unix socket (see [Reach](#reach)).
- Persistent state at
`/var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/`
survives container restart / host reboot. To wipe, destroy the
container.
## Identity vs API listener: `serverName` vs `gatewayHost`
Two distinct hostnames:
- **`serverName`** — matrix-spec `server_name`, embedded
*irrevocably* in every `@user:<server_name>` and `!room:<server_name>`
identifier minted on this homeserver. You can't change it later
without abandoning every account and chat history. Defaults to the
bare `services.hyperhive.swarm.domain`.
- **`gatewayHost`** — the API listener hostname, where the gateway's
matrix vhost proxies `/_matrix/*` to tuwunel. Defaults to
`chat.<services.hyperhive.swarm.domain>`. Set to `null` to skip the
gateway vhost, which leaves nothing able to reach the homeserver.
Both default under the **swarm** domain, because a swarm runs one
homeserver: tying its identity to a single hive's domain would make
relocating the container between hives look like a different
homeserver.
⚠️ **they're still not interchangeable, and the difference is the
cost of changing one.** `gatewayHost` is a routing detail clients
rediscover through `.well-known`, so it's safe to move on a running
deployment. The controller's host serves that `.well-known` on the swarm
domain, and a homeserver pinned to a hive domain gets it on that hive's
vhost. The matrix id format bakes `serverName` into every user and
room id, so adopting a new one does **not** rename the existing users and rooms —
it strands them, because their ids still name a homeserver that no
longer answers.
<details><summary>Pinning <code>serverName</code> on a homeserver with existing ids</summary>
`serverName` defaults to the bare `services.hyperhive.swarm.domain`. A
homeserver that changes `serverName` after minting ids strands its
existing users and rooms — keep the value it minted them under:
```nix
services.hyperhive.swarm.matrix = {
# whichever this deployment already uses
serverName = config.services.hyperhive.domain;
gatewayHost = "matrix.${config.services.hyperhive.domain}";
};
```
A rebuild on a host that already has a homeserver prints a
`hive-matrix: WARNING — … serverName is unset` line when this is missing,
naming the value it's about to default to. It never fails the rebuild, so
pin the value before the homeserver mints any ids under the default.
A pinned hive domain answers `.well-known/matrix/*` itself, built from
`gatewayHost`, on any host that runs that hive's gateway. It doesn't need
the homeserver on the same host.
</details>
## Reach
tuwunel binds no TCP port. It listens on the unix socket
`/run/hive-matrix/tuwunel.sock`, in a host directory (`root:nginx`,
`0770`) bind-mounted into the container, and the gateway's
`/_matrix/` location proxies to that socket. Clients, agents, hive-c0re
and federation peers all reach the homeserver at `chat.<swarm-domain>`
through the gateway; from outside the host that takes
`gateway.openFirewall`.
hive-c0re doesn't talk to the homeserver: matrix is swarm-level, and
the hive has no matrix API URL to set.
Federation peers find the homeserver through `.well-known/matrix/server`
at the server name, which names `<gatewayHost>:443`
([`gateway.md::Discovery flow`](../../4-implementation/networking/gateway.md#discovery-flow-matrix)).
Port 8448 isn't used.
Outbound, the container sits on the hive bridge like the CI runner: it
leases an address from the gateway's DHCP, resolves through its dnsmasq,
and leaves through the bridge's NAT. It reaches the secret store,
authelia and federation peers by name.
## Provisioning flow (appservice)
<!-- vale write-good.Passive = NO -->
Registration is closed. **The swarm's appservice** creates accounts: agents
never see an appservice token, and an agent only ever receives its own
`access_token`.
<!-- vale write-good.Passive = YES -->
The appservice has no URL (`url: null` in its registration), so the
homeserver never calls out to it and there is no service to run. What the
registration buys is an identity the homeserver recognises — which is why
no secret has to be equal on both sides of the wire, and why account
creation doesn't depend on registration being open to anyone who learns
a token.
1. **`swarm-matrix-ctl appservice render`**, inside the `hive-matrix`
container and before tuwunel starts, fetches the `as_token` and
spec-required `hs_token` from the store and renders the registration to
`/run/swarm-matrix-appservice/swarm.yaml` (mode `0600 root:root`) on the
container's tmpfs, so no token sits on disk. It mints a pair only when
the store answers and holds none. While the store doesn't answer, `render`
fails and retries, and tuwunel stays down.
2. **systemd `LoadCredential=`** copies the registration into
`/run/credentials/tuwunel.service/swarm-appservice.yaml`, owned by
tuwunel's dynamic user with mode `0400`, at service start. The source
file stays `root:root 0600` — no `chown :tuwunel` / `chmod 0640` /
GID-pin gymnastics required. Keeps `DynamicUser = true` +
`PrivateUsers = true` intact.
3. tuwunel's `appservice_dir` points at the credentials directory. It
reads only `.yaml`/`.yml` entries from there, so the sibling
credentials are invisible to it. The `.yaml` suffix on the credential
id is what makes this work.
4. **hive-c0re** holds no appservice token, creates no account and does
not talk to matrix.
5. **`swarm-controller` creates every account**: each agent's, with its
token stored at `swarm/agents/<agent>/matrix/main` for the agent's
`hive-matrix-daemon` to read as the agent itself, and each hive's
sender account (below).
A homeserver whose `swarm-matrix-ctl` has no store identity
(`services.hyperhive.deploy.matrix.ctlBaoClientCertFile`) loads no
registration, and nothing creates accounts on it.
On the store's host `ctlBaoClientCertFile` / `ctlBaoClientKeyFile` default
to the leaf `glue-bao-tls.nix` mints there. A homeserver host without the
store gets no default, even when it holds the hive's own reader
`deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.matrixCtlCommonName`, place it, and name it in both options.
Until you do, `swarm-bao-matrix-oidc` fails naming both.
### The swarm's appservice, and its admin sender
The registration's id is `swarm`, its sender `@swarm`, and its user
namespace non-exclusive. It's the swarm's identity on the homeserver:
`swarm-controller` creates every agent's account and every hive's sender
account with it.
The swarm secret store holds the registration's tokens, at
`swarm/controller/swarm-controller/matrix/appservice-registration`.
`swarm-matrix-ctl appservice render` fetches them and renders the
registration tuwunel loads as a credential onto the `hive-matrix`
container's tmpfs, before tuwunel starts.
A render that can't reach the store fails and retries, and tuwunel, which fails to start without the registration, retries until one has landed.
`appservice sync` runs beside tuwunel on a timer. The store's tokens win:
when they differ from the rendered copy, it rewrites it. It
restarts tuwunel whenever the registration is newer than tuwunel's last
start, so the next run retries one that failed after writing, and starts
one that failed because `render` couldn't reach the store. A store that
holds none takes the tokens an earlier version kept in
`/var/lib/swarm-matrix-appservice` with a create-only write, so a running
swarm keeps its value; the files stay, unused. A host with neither files nor stored tokens mints a pair. `sync` then writes the `as_token`
to `swarm/controller/swarm-controller/matrix/appservice-token`. No hive's
policy reaches either path: only matrix-ctl (which holds `create` and
`read` on the registration, and writes the token) and `swarm-controller`
(which reads only the token) hold a grant.
`@swarm` is the **one** account the homeserver promotes to admin at boot
(`admin_execute`). Only those two principals read its token, which is
the difference from the hive sender below: every hive reads its own
sender's token. `swarm` is a reserved name, so nothing can create an agent
as this account.
`swarm-controller` mints each agent's account with the device id
`hyperhive-<agent>`, and a login on that device replaces its token. The controller therefore reads
the stored token back with `whoami` every five minutes and re-mints only
when it's missing, unknown to the homeserver, or someone else's. An
agent's daemon picks a re-minted token up on its next start, and a timer
restarts it while it's down.
### Operators are homeserver admins
Every member of authelia's `admins` group who has a matrix account is a
homeserver admin. `swarm-controller` reads the group through
`swarm-authelia-bridge` at start and every five minutes after, and for
each member not yet in `#admins:<server_name>` it sends
`!admin users make-user-admin @<name>:<server_name>` there as `@swarm`.
`<name>` is the authelia username lowercased, the account an SSO login
lands on.
- **Leaving `admins` revokes what the sync granted.** The controller
records each account it sends a grant for in
`/var/lib/swarm-controller/operator-matrix-admins.json`. When one of
them is no longer in `admins` but still in `#admins`, it sends
`!admin users force-leave-room @<name>:<server_name> <admin room id>`
there as `@swarm`; leaving that room is what ends homeserver admin.
- **Admins the sync didn't make keep admin.** An admin missing from that
file, `@swarm`, the server user `@conduit:<server_name>`, every agent's
account, and every user id in
`services.hyperhive.deploy.swarm-controller.matrixAdminSync.ignoreUsers` are never
revoked.
- **The last admin stays.** A revoke that would leave no admin in
`#admins` besides `@swarm` and the server user isn't sent; the
controller logs a warning on each pass instead.
- **No account, no grant.** A member who has never logged in to matrix
has no account to promote. The first pass after their first SSO login
promotes them.
- **One command per account per controller run.** A member still missing
from `#admins` after the controller sends its grant gets a
`make-user-admin … failed to take` warning in the controller log on
each later pass, and no second command until the controller restarts;
a revoke that leaves its account in `#admins` gets
`force-leave-room … failed to take` the same way.
- Runs only on a controller with both
`services.hyperhive.deploy.swarm-controller.authBridgeUrl` and
`services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl` set.
### A hive's sender account, and why it isn't an admin
`@hive-<hive>:<server_name>` is the account a hive acts as.
`swarm-controller` creates it with the swarm's appservice token, for every
hive in its directory. It's an **ordinary account**: nothing promotes it;
the homeserver's `admin_execute` promotes only `@swarm`.
**One account per hive.** The localpart carries the hive's name, so a swarm
whose hives share a homeserver gives each of them its own identity: the
homeserver attributes an action to the hive that took it, and revoking one
hive's standing leaves the others alone.
Its access token is the **sender token**. `swarm-controller`, its only
minter, mints it for every hive with the swarm's appservice token and
publishes it to `swarm/hives/<hive>/matrix/sender-token`, inside the hive's own
read grant (`swarm/hives/<hive>/*`). Nothing on the hive uses it any more:
hive-c0re used to present it for its Space and invite calls and no longer
makes any. The name says what it authenticates as — an account the
swarm's appservice creates — rather than any privilege level, because it
carries none.
Promoting a user to homeserver admin and resetting a password both need an
admin **sender**: `!admin …` messages into `#admins:<server_name>`, and
tuwunel only treats a message as a command when its sender is already an
admin. Of the swarm's own accounts only `@swarm` is a homeserver admin
(operators are too, see
[Operators are homeserver admins](#operators-are-homeserver-admins)), so
both are swarm-level operations.
<details><summary>Appservice identity</summary>
`swarm-matrix-ctl appservice render` renders the registration from the
store's copy of the appservice tokens before tuwunel starts, so every
boot has both halves. With the store down or sealed, tuwunel waits for it.
<!-- vale write-good.Passive = NO -->
- **Access tokens live on the device that minted them.**
`login_with_password` stays on, so the password fallback works
too.
- **The homeserver's `admin_execute` promotes only `@swarm` at boot**
(above); a hive's sender account is never promoted.
- **The value that matters lives at
`swarm/controller/swarm-controller/matrix/appservice-token`.**
`appservice sync` writes it there whenever the store's differs from the
registration's `as_token`, and `swarm-controller` reads it from there.
<!-- vale write-good.Passive = YES -->
</details>
Initial rollout settings:
- `allow_federation = true` at the protocol level so swarms can be
wired up later by extending `trustedServers` without a homeserver
restart. `trusted_servers = []` keeps it effectively closed
until you list peers.
- `allow_registration = false`. tuwunel checks this flag only for
requests that arrive **without** an appservice token, so the swarm's
appservice still creates accounts and tuwunel refuses everyone else. With no
registration token configured, `allow_registration = true` makes
tuwunel refuse to start unless
`yes_i_am_very_very_sure_…_open_registration_…` is also set.
- `allow_encryption` — server-side E2EE switch, sourced from
`services.hyperhive.deploy.matrix.allowEncryption` (**default `false`**, opt-in).
Off by default because on the swarm's own homeserver the operator
already controls the transport; turn it on for encrypted rooms on
external / federated homeservers or to keep contents opaque to the
homeserver admin. **The agent matrix client always supports decryption
regardless of this flag** — it uses the `e2e-encryption` feature of
`matrix-sdk` so it can read encrypted rooms it's invited to even when
this homeserver doesn't permit room encryption. matrix-sdk stores
crypto keys in the per-agent sqlite store under the state dir; they persist across
restarts. `read_room` decrypts via
`room.messages()` — UTD events surface as `event_type =
"m.room.encrypted"` with `body = "[unable to decrypt]"`.
The daemon bootstraps cross-signing on the bearer token alone, which
works where the homeserver skips re-auth for a first key upload
(MSC3967, as tuwunel does).
- **Key backup.** `swarm-controller` mints a random recovery passphrase
for every matrix account an agent has, at
`swarm/agents/<agent>/matrix-recovery/<account>` in the swarm secret
store, and never replaces it. When `hive-matrix-daemon` brings an
account up it reads that passphrase: with no backup on the account's
homeserver it turns on server-side key backup and secret storage
(log line `matrix recovery enabled`), unless the account already has
secret storage, which it never replaces (a `WARN` instead). With a
backup there it recovers from it (`recovered from the server-side key backup`), which downloads the
room keys and signs the new device with the account's existing
cross-signing identity. A crypto store lost to `--purge` or to a
primary-token reset therefore comes back with its history. With no passphrase
stored yet the account comes up without backup and picks it up on the
next daemon start. A failure is a `WARN` and never stops the daemon.
## Retired hive-side matrix code
Earlier versions of hive-c0re created a private "hive" Matrix Space and a
`hive-chat` room, invited every agent to both, kept the hive's sender token
on disk and offered `hivectl matrix invite`. Matrix is swarm-level, so
hive-c0re dropped all of it and no longer talks to the homeserver. The Space, the
room and the files `/var/lib/hyperhive/matrix/{access-token,space-room-id,chat-room-id}`
are harmless leftovers, so delete them by hand if you like.
## Configuration tuning
```nix
services.hyperhive.deploy.matrix = {
trustedServers = [ "matrix.org" "example.com" ]; # default: []
maxRequestSize = 20000000; # default: 20 MB
};
```
**`trustedServers`** (default `[]`) — list of peer homeserver names
whose signing keys tuwunel fetches and trusts. tuwunel enables
federation at the protocol level from first boot (`allow_federation =
true`) but trusts no remote homeserver until you list it here. For a closed
single-hive deployment the default empty list is correct — add peer
hive domains here when connecting hives into a swarm (see
[`docs/swarm/`](../swarm/README.md)).
**`maxRequestSize`** (default `20_000_000` bytes = 20 MB) — maximum
size of a single matrix client request body. Matches the matrix-spec
recommendation for media uploads and the upstream tuwunel default.
Raise for deployments that need large file transfers; lower for
resource-constrained hosts where a 20 MB request is unexpectedly large.
## Assertion rationale
`config.assertions` in this module fail eval early rather than ship
surprising behaviour:
- **`services.hyperhive.swarm.matrix.gatewayHost != ""`** — same footgun as `forge.domain`:
empty string renders `.<hive>`-shaped garbage in both nginx
`server_name` (treated as wildcard catch-all, surprising) and
`/etc/hosts` (malformed entry). `null` is the right opt-out shape;
a config assertion rejects empty string explicitly.
SSO is unconditional, so the three below are requirements of running a
homeserver at all rather than of a setting:
- **Keep `deploy.matrix.sso.clientSecretFile` set** — it defaults to
`/run/hive-matrix-oidc/<client id>.secret` wherever the homeserver runs,
and `swarm-bao-matrix-oidc` writes the secret there from the swarm secret
store under matrix-ctl's leaf. The path is on the host's tmpfs, so every
reboot fetches the secret again and it never reaches persistent disk. After
a reboot the homeserver doesn't start until the store answers; the
store-retry timer then starts it with no manual step. A path
outside `/run/hive-matrix-oidc/`, or a null, fails at eval, not at boot:
tuwunel reads its identity providers from the config file, so a half-configured
one can stop the homeserver from starting outright rather than merely
hiding a login button.
- **Set `swarm.authelia.url`** — without a provider URL there
is nothing to discover against.
- **Set `swarm.matrix.gatewayHost != null`** — the SSO callback URL is
format-locked to `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`,
and the identity provider needs a public name to redirect the
browser to.
`server_name`'s own bogus-value guard lives in `hive-network.nix`
(`services.hyperhive.domain != null`), not here — see
[`docs/4-implementation/networking/network.md`](../../4-implementation/networking/network.md).
## fluffychat-web build fixes
`pkgs.fluffychat-web` ships from `flutter341.buildFlutterApplication`,
which has two upstream gaps for fluffychat's web target:
- The dart web-worker entry point (`web/native_executor.dart`) isn't
compiled — `buildFlutterApplication` only runs `flutter build web`
on the main entry.
- `native_imaging`'s C source isn't built — emscripten isn't a
flutter-builder native build input.
Both fixed in `nix/host-modules/hive-matrix.nix` via two derivations:
- **`fluffychat-web-imaging`** builds `Imaging.{js,wasm}` from the
`native_imaging` C source via `pkgs.emscripten`. Source comes
from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging`
— already in the build closure of the flutter app, so no parallel
hash pin and version autosyncs with nixpkgs bumps. Build closure
is ~3.6 GiB (emscripten LLVM); runtime closure is just the two
output files. `dontConfigure = true` because cmake runs inside
`js/Makefile` via `emcmake cmake`, not at the package root. The
build script needs `HOME` + `EM_CACHE` writable for emscripten's
on-demand sysroot build (libc, libc++ → wasm).
- **`fluffychat-web-fixed`** is `pkgs.fluffychat-web` plus a
`postInstall` patch that (a) compiles `web/native_executor.dart`
via `dart compile js` (dart from the flutter341 closure, no
incremental cost) and (b) installs `fluffychat-web-imaging`'s
outputs into `$out`.
Two subtle details worth knowing before touching either derivation:
- **`make -C js`** instead of `cd js; make` — keeps the build-phase
pwd at the source root so `installPhase` doesn't have to know
about the cd. Robust against future reorders / `dontBuild`.
- **`web/native_executor.dart`** as a build-CWD-relative path,
*not* `$src/web/...` — `dart`'s `package_config.json` walk-up
needs to hit `buildFlutterApplication`'s pub-get output
(`.dart_tool/` in the build CWD). Walking up from a read-only
`$src/` store path finds no `.dart_tool/` and errors with
"Couldn't resolve the package 'matrix'."
Drop both derivations when nixpkgs's flutter builder grows worker
+ emcc support upstream.
Mount point is `chat.<swarm-domain>/` (the `gatewayHost` vhost);
upstream `--base-href "/"` is correct at sub-domain root, no override.

1432
docs/2-handbook/reference/forge-cli.md Normal file
View file

File diff suppressed because it is too large Load diff

530
docs/2-handbook/reference/hivectl-cli.md Normal file
View file

@ -0,0 +1,530 @@
# Command-Line Help for `hivectl`
This document contains the help content for the `hivectl` command-line program.
**Command Overview:**
* [`hivectl`↴](#hivectl)
* [`hivectl agent`↴](#hivectl-agent)
* [`hivectl agent restart`↴](#hivectl-agent-restart)
* [`hivectl agent pause`↴](#hivectl-agent-pause)
* [`hivectl agent resume`↴](#hivectl-agent-resume)
* [`hivectl agent start`↴](#hivectl-agent-start)
* [`hivectl agent stop`↴](#hivectl-agent-stop)
* [`hivectl agent kill`↴](#hivectl-agent-kill)
* [`hivectl agent destroy`↴](#hivectl-agent-destroy)
* [`hivectl agent rebuild`↴](#hivectl-agent-rebuild)
* [`hivectl agent choom`↴](#hivectl-agent-choom)
* [`hivectl agent watch`↴](#hivectl-agent-watch)
* [`hivectl agent quota`↴](#hivectl-agent-quota)
* [`hivectl agent quota show`↴](#hivectl-agent-quota-show)
* [`hivectl agent quota set`↴](#hivectl-agent-quota-set)
* [`hivectl agent subvol`↴](#hivectl-agent-subvol)
* [`hivectl agent subvol upgrade`↴](#hivectl-agent-subvol-upgrade)
* [`hivectl agent subvol snapshot`↴](#hivectl-agent-subvol-snapshot)
* [`hivectl agent subvol snapshot create`↴](#hivectl-agent-subvol-snapshot-create)
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
* [`hivectl agent subvol snapshot push`↴](#hivectl-agent-subvol-snapshot-push)
* [`hivectl list-agents`↴](#hivectl-list-agents)
* [`hivectl quota-enable`↴](#hivectl-quota-enable)
* [`hivectl wg`↴](#hivectl-wg)
* [`hivectl wg init`↴](#hivectl-wg-init)
* [`hivectl wg peer`↴](#hivectl-wg-peer)
* [`hivectl wg status`↴](#hivectl-wg-status)
* [`hivectl peer-config`↴](#hivectl-peer-config)
* [`hivectl stop`↴](#hivectl-stop)
* [`hivectl start`↴](#hivectl-start)
* [`hivectl restart`↴](#hivectl-restart)
* [`hivectl open`↴](#hivectl-open)
* [`hivectl completions`↴](#hivectl-completions)
## `hivectl`
Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that don't go through the broker, plus future recovery / debugging verbs.
**Usage:** `hivectl [OPTIONS] <COMMAND>`
###### **Subcommands:**
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
* `list-agents` — Show all managed agents with their status and technical state
* `quota-enable` — Enable btrfs qgroup accounting on the agent-state filesystem
* `wg` — WireGuard inter-hive mesh setup helpers
* `peer-config` — Generate the federation peer-config block for THIS hive
* `stop` — Stop containers hive-wide in one operator action
* `start` — Start containers hive-wide — the inverse of `hivectl stop`
* `restart` — Restart containers hive-wide — `stop` then `start` over one scope
* `open` — Print (and best-effort open in a browser) a hive web surface URL
* `completions` — Generate a shell completion script for `hivectl` and print it to stdout
###### **Options:**
* `--socket <SOCKET>` — Path to the hive-c0re host admin socket, used by the daemon-assisted verbs (`agent`, `list-agents`, `stop`, `start`). Global: accepted before or after the subcommand. Verbs that don't talk to the daemon ignore it
Default value: `/run/hyperhive/host.sock`
## `hivectl agent`
Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running.
Everything here targets a single named agent (`hivectl agent foo restart`, `hivectl agent foo choom`, …) — anything that acts hive-wide lives at the top level instead (`list-agents`, `restart`/`stop`/`start` with a scope, `quota-enable`).
**Usage:** `hivectl agent <NAME> <COMMAND>`
###### **Subcommands:**
* `restart` — Stop and start this agent container without rebuilding config
* `pause` — Park this agent's turn loop, leaving the container running
* `resume` — Resume this paused agent — it drains whatever queued up while parked
* `start` — Start this EXISTING agent container. Fails immediately if `name` has no config/topology entry at all — it never attempts first-time creation, which is swarm-level (`swarmctl agent create`)
* `stop` — Gracefully stop this agent container: signal → drain → reconcile. Never escalates to a hard kill — use `kill` for that
* `kill` — Hard-stop this managed container
* `destroy` — Tear down this sub-agent container, keeping its state by default. No undo
* `rebuild` — Apply pending config to this managed container
* `choom` — Open an interactive Claude session inside this agent's container
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups
* `subvol` — btrfs subvolume management for this agent's state dir
###### **Arguments:**
* `<NAME>` — Agent name (for example `damocles`, `iris`)
## `hivectl agent restart`
Stop and start this agent container without rebuilding config
**Usage:** `hivectl agent restart [OPTIONS]`
###### **Options:**
* `--no-wait` — Return immediately after hivectl queues the restart DAG
## `hivectl agent pause`
Park this agent's turn loop, leaving the container running.
The harness stops driving turns but keeps serving its web UI and MCP daemons, so the container, its mounts and its warm caches stay up while it burns no tokens. Inbox messages queue unacked and the backlog drains on `resume`. Sticky: it survives a restart, and pausing a stopped agent makes it come up paused.
**Usage:** `hivectl agent pause [OPTIONS]`
###### **Options:**
* `--no-wait` — Return immediately after hivectl queues the pause DAG instead of waiting for the harness to acknowledge it
## `hivectl agent resume`
Resume this paused agent — it drains whatever queued up while parked
**Usage:** `hivectl agent resume`
## `hivectl agent start`
Start this EXISTING agent container. Fails immediately if `name` has no config/topology entry at all — it never attempts first-time creation, which is swarm-level (`swarmctl agent create`)
**Usage:** `hivectl agent start [OPTIONS]`
###### **Options:**
* `--paused` — Start (or leave) the agent paused: if it's currently down, hivectl writes the pause marker before the container boots, so it comes up paused instead of driving turns immediately. If it's already running, this pauses it in place and doesn't attempt a (re)start
## `hivectl agent stop`
Gracefully stop this agent container: signal → drain → reconcile. Never escalates to a hard kill — use `kill` for that
**Usage:** `hivectl agent stop`
## `hivectl agent kill`
Hard-stop this managed container
**Usage:** `hivectl agent kill`
## `hivectl agent destroy`
Tear down this sub-agent container, keeping its state by default. No undo
**Usage:** `hivectl agent destroy [OPTIONS]`
###### **Options:**
* `--purge` — Also wipe the agent's state dirs (config + creds + notes)
## `hivectl agent rebuild`
Apply pending config to this managed container
**Usage:** `hivectl agent rebuild`
## `hivectl agent choom`
Open an interactive Claude session inside this agent's container.
A fresh session by default, or resume a prior one. Requires root or `hive-admin` group membership, plus a running container.
**Usage:** `hivectl agent choom [OPTIONS]`
###### **Options:**
* `--resume <SESSION>` — Resume a prior claude session by its session id, passed through as `claude --resume <value>` (claude's `--continue` takes no value — it resumes the cwd's latest session, which is the harness's, so choom never uses it; this flag matches the claude flag it maps to). Omit for a fresh blank session. Requires a value
## `hivectl agent watch`
Follow this agent's live turn/tool-call event stream from the CLI.
Prints one compact line per event as they happen. `Ctrl-C` to stop. Requires the agent to be running.
**Usage:** `hivectl agent watch`
## `hivectl agent quota`
This agent's disk accounting + optional quota via btrfs qgroups.
Needs `hivectl quota-enable` run once hive-wide first. No-op on non-btrfs hosts.
**Usage:** `hivectl agent quota <COMMAND>`
###### **Subcommands:**
* `show` — Report this agent's disk usage from btrfs qgroups
* `set` — Set or clear this agent's disk-usage quota
## `hivectl agent quota show`
Report this agent's disk usage from btrfs qgroups
**Usage:** `hivectl agent quota show`
## `hivectl agent quota set`
Set or clear this agent's disk-usage quota
**Usage:** `hivectl agent quota set <SIZE>`
###### **Arguments:**
* `<SIZE>` — Size cap (`5G`, `500M`, `1073741824`) or `none` to clear
## `hivectl agent subvol`
btrfs subvolume management for this agent's state dir.
Upgrade an existing plain-dir agent's state into a btrfs subvolume so it gains snapshots and per-subvol usage/quota.
**Usage:** `hivectl agent subvol <COMMAND>`
###### **Subcommands:**
* `upgrade` — Convert a plain-dir agent state root into a btrfs subvolume in place, so it gains snapshots and per-subvol usage/quota
* `snapshot` — Read-only snapshots of this agent's state subvolume
## `hivectl agent subvol upgrade`
Convert a plain-dir agent state root into a btrfs subvolume in place, so it gains snapshots and per-subvol usage/quota.
Bounces the agent to migrate its state, so it requires `--yes`.
**Usage:** `hivectl agent subvol upgrade [OPTIONS]`
###### **Options:**
* `--yes` — Confirm: this stops the agent, migrates its state dir, and restarts it. Required — the command refuses without it
## `hivectl agent subvol snapshot`
Read-only snapshots of this agent's state subvolume
**Usage:** `hivectl agent subvol snapshot <COMMAND>`
###### **Subcommands:**
* `create` — Create a read-only snapshot (agent must already be a subvolume)
* `delete` — Delete a snapshot created by `subvol snapshot create`
* `send` — Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
* `push` — Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport
## `hivectl agent subvol snapshot create`
Create a read-only snapshot (agent must already be a subvolume)
**Usage:** `hivectl agent subvol snapshot create --label <LABEL>`
###### **Options:**
* `--label <LABEL>` — Snapshot label. Mandatory, and must start with `hive-` — the prefix doubles as an allow-list hive-priv checks so only hivectl-issued snapshot names can reach the `btrfs subvolume snapshot` shellout
## `hivectl agent subvol snapshot delete`
Delete a snapshot created by `subvol snapshot create`
**Usage:** `hivectl agent subvol snapshot delete <LABEL>`
###### **Arguments:**
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
## `hivectl agent subvol snapshot send`
Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
**Usage:** `hivectl agent subvol snapshot send [OPTIONS] --dest <DEST> <LABEL>`
###### **Arguments:**
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
###### **Options:**
* `--parent <PARENT>` — Optional parent snapshot label for an incremental send (`btrfs send -p`) — must be an existing, older snapshot of the same agent. Omit for a full send
* `--dest <DEST>` — Destination filename (not a path) under the migrate-staging dir. Refused if it already exists
## `hivectl agent subvol snapshot push`
Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport.
Stages nothing locally: `btrfs send` writes straight into the connection, so a multi-gigabyte agent needs no scratch space on this host. The mesh is the authentication (cryptokey routing binds the sender's address to its key), so there is no credential to pass here.
No destination argument exists: a swarm has one store, read from `services.hyperhive.swarm.snapshotStore`.
**Usage:** `hivectl agent subvol snapshot push [OPTIONS] <LABEL>`
###### **Arguments:**
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
###### **Options:**
* `--parent <PARENT>` — Optional parent snapshot label for an incremental send (`btrfs send -p`) — must be an existing, older snapshot of the same agent, and must already be present on the receiver. Omit for a full send
## `hivectl list-agents`
Show all managed agents with their status and technical state.
Global — not scoped to one agent, so it lives at the top level rather than under `hivectl agent <name>`. Needs the hive-c0re daemon running.
**Usage:** `hivectl list-agents [OPTIONS]`
###### **Options:**
* `--json` — Emit the raw JSON rows instead of the padded table (for scripting). The table is the default human-readable shape
## `hivectl quota-enable`
Enable btrfs qgroup accounting on the agent-state filesystem.
Global one-shot toggle (not per-agent), hence top-level rather than under `hivectl agent <name>`. Idempotent — safe to re-run. Once enabled, `hivectl agent <name> quota show`/`quota set` work.
**Usage:** `hivectl quota-enable`
## `hivectl wg`
WireGuard inter-hive mesh setup helpers.
Generate this hive's mesh key and print the nix to enable the mesh, add a peer, or inspect live interface state.
**Usage:** `hivectl wg <COMMAND>`
###### **Subcommands:**
* `init` — Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
* `peer` — Print the nix to add a peer hive to the mesh
* `status` — Show the live mesh interface state
## `hivectl wg init`
Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
**Usage:** `hivectl wg init [OPTIONS]`
###### **Options:**
* `--address <ADDRESS>` — This hive's mesh address (for example `10.42.0.1/32`) to bake into the printed snippet. Omit to get a placeholder you fill in
## `hivectl wg peer`
Print the nix to add a peer hive to the mesh
**Usage:** `hivectl wg peer [OPTIONS] --pubkey <PUBKEY> --address <ADDRESS> <DOMAIN>`
###### **Arguments:**
* `<DOMAIN>` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
###### **Options:**
* `--pubkey <PUBKEY>` — Peer's WireGuard public key (from its `hivectl wg init`)
* `--address <ADDRESS>` — Peer's mesh address (for example `10.42.0.2/32`)
* `--endpoint <ENDPOINT>` — Peer's `host:port` endpoint (omit for a peer that only dials out, for example one behind NAT — it must set an endpoint pointing back here)
## `hivectl wg status`
Show the live mesh interface state
**Usage:** `hivectl wg status`
## `hivectl peer-config`
Generate the federation peer-config block for THIS hive.
Prints the nix a peer operator pastes into their swarm config to trust and reach this hive.
**Usage:** `hivectl peer-config [OPTIONS]`
###### **Options:**
* `--wg-address <WG_ADDRESS>` — This hive's WireGuard mesh address (for example `10.42.0.1/32`), emitted as `wireguardAddress`. Omit when not running the mesh
* `--wg-endpoint <WG_ENDPOINT>` — This hive's public WireGuard endpoint (`host:port`), emitted as `wireguardEndpoint`. Omit when peers dial in / no mesh
## `hivectl stop`
Stop containers hive-wide in one operator action.
Bare `hivectl stop` stops every agent and the gateway; scope flags narrow it to specific sub-agents or the gateway. The swarm's forge and matrix are never in scope: systemd on the services host runs them.
**Usage:** `hivectl stop [OPTIONS]`
###### **Options:**
* `--agents` — All sub-agent containers
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: it signals the harness, which runs one stop-checkpoint turn to flush durable `/state` and drains, then the DAG stops the container (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only
* `--no-wait` — Return immediately after hivectl queues the stop DAGs instead of waiting for them with live per-node progress
## `hivectl start`
Start containers hive-wide — the inverse of `hivectl stop`.
Bare `hivectl start` restores the agents stopped by the last broad-scope `stop` (or starts everything if none); scope flags narrow it. The swarm's forge and matrix are never in scope.
**Usage:** `hivectl start [OPTIONS]`
###### **Options:**
* `--agents` — All sub-agent containers
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
* `--no-wait` — Return immediately after hivectl queues the start DAGs instead of waiting for them with live per-node progress
## `hivectl restart`
Restart containers hive-wide — `stop` then `start` over one scope.
Bare `hivectl restart` restarts everything; scope flags narrow it. The swarm's forge and matrix are never in scope.
**Usage:** `hivectl restart [OPTIONS]`
###### **Options:**
* `--agents` — All sub-agent containers
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
* `--graceful` — Gracefully quiesce each agent on the stop half (see `stop --graceful`). Applies to agents only
## `hivectl open`
Print (and best-effort open in a browser) a hive web surface URL.
Resolves the URL from the running daemon so custom forge / matrix domains work. Bare `hivectl open` opens the operator dashboard.
**Usage:** `hivectl open [TARGET]`
###### **Arguments:**
* `<TARGET>` — Which surface to open. Defaults to the operator dashboard
Default value: `home`
Possible values:
- `home`:
The operator dashboard (`https://<hive>.hives.<swarm domain>/`)
- `forge`:
The forge (Forgejo) web UI
- `matrix`:
The matrix GUI (fluffychat)
## `hivectl completions`
Generate a shell completion script for `hivectl` and print it to stdout.
Supports bash, zsh, fish, elvish, and powershell. The NixOS module already installs the zsh script system-wide; this is for ad-hoc or other-shell use.
**Usage:** `hivectl completions <SHELL>`
###### **Arguments:**
* `<SHELL>` — Shell to emit completions for
Possible values: `bash`, `elvish`, `fish`, `powershell`, `zsh`

51
docs/2-handbook/reference/swarm-logs-cli.md Normal file
View file

@ -0,0 +1,51 @@
# Command-Line Help for `swarm-logs`
This document contains the help content for the `swarm-logs` command-line program.
**Command Overview:**
* [`swarm-logs`↴](#swarm-logs)
* [`swarm-logs query`↴](#swarm-logs-query)
## `swarm-logs`
query the swarm's log store from an agent
**Usage:** `swarm-logs <COMMAND>`
###### **Subcommands:**
* `query` — Run a LogsQL query and print the matched log lines
## `swarm-logs query`
Run a LogsQL query and print the matched log lines
**Usage:** `swarm-logs query [OPTIONS] <LOGSQL>`
###### **Arguments:**
* `<LOGSQL>` — The LogsQL query. A bare word is a full-text search for it.
Quote it: LogsQL uses characters the shell also claims, and an unquoted `|` pipes this command into the rest of your query instead of sending it.
###### **Options:**
* `--limit <LIMIT>` — Stop after this many records.
Unset means the store's own default. Worth setting on a broad query — nothing between here and the store narrows one, so a bare word matches across every hive in the swarm.
* `--format <FORMAT>` — What to print for each matched record
Default value: `message`
Possible values:
- `message`:
One log message per line — the `_msg` field and nothing else
- `json`:
The store's response body, unmodified, one JSON object per line

123
docs/2-handbook/reference/swarm-metrics-cli.md Normal file
View file

@ -0,0 +1,123 @@
# Command-Line Help for `swarm-metrics`
This document contains the help content for the `swarm-metrics` command-line program.
**Command Overview:**
* [`swarm-metrics`↴](#swarm-metrics)
* [`swarm-metrics query`↴](#swarm-metrics-query)
* [`swarm-metrics query-range`↴](#swarm-metrics-query-range)
* [`swarm-metrics series`↴](#swarm-metrics-series)
* [`swarm-metrics labels`↴](#swarm-metrics-labels)
* [`swarm-metrics label-values`↴](#swarm-metrics-label-values)
## `swarm-metrics`
query the swarm's metrics store from an agent
**Usage:** `swarm-metrics <COMMAND>`
###### **Subcommands:**
* `query` — Evaluate a `MetricsQL` (`PromQL`) expression at one instant
* `query-range` — Evaluate an expression over a window, one point per step
* `series` — List the series that match one or more selectors
* `labels` — List label names, optionally only those on matching series
* `label-values` — List one label's values; `__name__` lists every metric name
## `swarm-metrics query`
Evaluate a `MetricsQL` (`PromQL`) expression at one instant
**Usage:** `swarm-metrics query [OPTIONS] <PROMQL>`
###### **Arguments:**
* `<PROMQL>` — The expression. Quote it: `{`, `|` and `"` are the shell's too.
OTLP metric names keep their dots, so match them by `{__name__="jobq.nodes"}` rather than as a bare name.
###### **Options:**
* `--time <TIME>` — The instant: RFC 3339, a unix timestamp, or relative (`-5m`). Unset means now
## `swarm-metrics query-range`
Evaluate an expression over a window, one point per step
**Usage:** `swarm-metrics query-range [OPTIONS] --start <START> <PROMQL>`
###### **Arguments:**
* `<PROMQL>` — The expression, quoted as for `query`
###### **Options:**
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`). Required, so a range is always bounded
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--step <STEP>` — Seconds or a duration (`1m`) between points. Unset means the store's default
## `swarm-metrics series`
List the series that match one or more selectors
**Usage:** `swarm-metrics series [OPTIONS] <MATCHERS>...`
###### **Arguments:**
* `<MATCHERS>` — A series selector, such as `{"service.name"="nats"}`. Repeatable
###### **Options:**
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries
## `swarm-metrics labels`
List label names, optionally only those on matching series
**Usage:** `swarm-metrics labels [OPTIONS]`
###### **Options:**
* `--match <MATCHERS>` — A series selector to narrow to. Repeatable
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries
## `swarm-metrics label-values`
List one label's values; `__name__` lists every metric name
**Usage:** `swarm-metrics label-values [OPTIONS] <NAME>`
###### **Arguments:**
* `<NAME>` — The label name
###### **Options:**
* `--match <MATCHERS>` — A series selector to narrow to. Repeatable
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries

620
docs/2-handbook/reference/swarmctl-cli.md Normal file
View file

@ -0,0 +1,620 @@
# Command-Line Help for `swarmctl`
This document contains the help content for the `swarmctl` command-line program.
**Command Overview:**
* [`swarmctl`↴](#swarmctl)
* [`swarmctl agent`↴](#swarmctl-agent)
* [`swarmctl agent create`↴](#swarmctl-agent-create)
* [`swarmctl agent mint-forge-token`↴](#swarmctl-agent-mint-forge-token)
* [`swarmctl agent inference`↴](#swarmctl-agent-inference)
* [`swarmctl agent inference get`↴](#swarmctl-agent-inference-get)
* [`swarmctl agent inference set`↴](#swarmctl-agent-inference-set)
* [`swarmctl agent inference clear`↴](#swarmctl-agent-inference-clear)
* [`swarmctl agent permissions`↴](#swarmctl-agent-permissions)
* [`swarmctl agent permissions get`↴](#swarmctl-agent-permissions-get)
* [`swarmctl agent permissions set`↴](#swarmctl-agent-permissions-set)
* [`swarmctl agent permissions clear`↴](#swarmctl-agent-permissions-clear)
* [`swarmctl agent agents-md`↴](#swarmctl-agent-agents-md)
* [`swarmctl agent agents-md get`↴](#swarmctl-agent-agents-md-get)
* [`swarmctl agent agents-md apply`↴](#swarmctl-agent-agents-md-apply)
* [`swarmctl schedule`↴](#swarmctl-schedule)
* [`swarmctl schedule add`↴](#swarmctl-schedule-add)
* [`swarmctl schedule list`↴](#swarmctl-schedule-list)
* [`swarmctl schedule edit`↴](#swarmctl-schedule-edit)
* [`swarmctl schedule pause`↴](#swarmctl-schedule-pause)
* [`swarmctl schedule resume`↴](#swarmctl-schedule-resume)
* [`swarmctl schedule rm`↴](#swarmctl-schedule-rm)
* [`swarmctl user`↴](#swarmctl-user)
* [`swarmctl user add`↴](#swarmctl-user-add)
* [`swarmctl user reset-password`↴](#swarmctl-user-reset-password)
* [`swarmctl user update`↴](#swarmctl-user-update)
* [`swarmctl user list`↴](#swarmctl-user-list)
* [`swarmctl forge`↴](#swarmctl-forge)
* [`swarmctl forge make-admin`↴](#swarmctl-forge-make-admin)
* [`swarmctl login`↴](#swarmctl-login)
* [`swarmctl completions`↴](#swarmctl-completions)
## `swarmctl`
swarm-level operator CLI
**Usage:** `swarmctl [OPTIONS] <COMMAND>`
###### **Subcommands:**
* `agent` — Manage agents across the swarm
* `schedule` — Manage scheduled prompts
* `user` — Manage subjects in the swarm's SSO provider
* `forge` — Manage human accounts on the swarm's forge
* `login` — Log in to the swarm named by `--swarm` as your own SSO account
* `completions` — Generate a shell completion script for `swarmctl` and print it to stdout
###### **Options:**
* `--authelia-bin <PATH>` — authelia binary used to hash passwords. The argon2 parameters must match the verifier's, so this has to be the *configured* package rather than whatever is on `PATH`
* `--users-file <PATH>` — Host-side path of authelia's users database — that is, the path inside the container, prefixed with the container's root.
This is the only user store: it's read before every change and written in place, and `swarm-authelia-bridge` writes the same file.
* `--swarm <URL>` — The swarm's domain, as `https://<domain>` or the bare domain. Given, the `agent`, `schedule` and `forge` verbs go through its gateway as the account `swarmctl login` logged in, instead of over the controller's socket. Falls back to `SWARMCTL_SWARM`; `login` then falls back to `SWARMCTL_LOGIN_DEFAULT_SWARM`, which the swarm-controller nix module sets to its host's own swarm
* `--ca-file <PATH>` — A PEM CA certificate to trust beside the system roots, for a swarm whose gateway serves a certificate from the swarm's own CA. Falls back to `SWARMCTL_CA_FILE`
## `swarmctl agent`
Manage agents across the swarm
**Usage:** `swarmctl agent <COMMAND>`
###### **Subcommands:**
* `create` — Queue creation of a new agent on a hive in this swarm
* `mint-forge-token` — Check one agent's forge token, and mint it if it's missing or stale
* `inference` — Show the model, effort, subagent models and subagent cap the swarm sets for an agent, or change the model, effort and subagent models
* `permissions` — Show or change the permissions the swarm grants an agent
* `agents-md` — Show an agent's AGENTS.md, or edit it with a diff
## `swarmctl agent create`
Queue creation of a new agent on a hive in this swarm.
Asks the swarm-controller to insert its agent-creation job graph — SSO identity, forge user, config repo, and the deploy message that puts the agent on `--hive` — and prints the queued job's node id.
**This returns as soon as it queues the work.** It doesn't wait, and a finished graph would not mean the agent is up either: the last node publishes a deploy, after which the hive converges on its own clock. Watch the swarm UI's job view, or the hive itself, for the rest.
No approval gate guards this: the caller is root on the controller's host, or logged in through the swarm domain's SSO gate.
**Usage:** `swarmctl agent create [OPTIONS] --hive <HIVE> <NAME>`
###### **Arguments:**
* `<NAME>` — Name for the new agent: 1–63 characters of `[a-z0-9-]`.
Becomes an SSO subject, a forge user and a repository name, so it's validated here before queuing. The controller refuses a name the swarm has already placed on a different hive; the same name on the same hive re-creates that agent.
###### **Options:**
* `--hive <HIVE>` — Hive in this swarm to deploy the agent to.
Required, and deliberately not defaulted: it's an *address* — the hive that gets the deploy message — and only the operator knows which one they mean. The controller checks it against the swarm's hive roster and names the known hives if it misses.
* `--agents-md <FILE>` — A text file the agent's `AGENTS.md` starts with.
The agent writes it on its first start, and only when it has no `AGENTS.md` yet: re-creating an agent never replaces an existing file. The controller refuses a file over 64 KiB.
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent mint-forge-token`
Check one agent's forge token, and mint it if it's missing or stale.
swarm-controller does this for every agent with a store identity at start and every five minutes; this is for when waiting isn't an option. It leaves a current token alone.
Queues and returns, the same way `agent create` does — watch the swarm UI's job view for the outcome.
**Usage:** `swarmctl agent mint-forge-token [OPTIONS] <NAME>`
###### **Arguments:**
* `<NAME>` — Name of an agent that already exists
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent inference`
Show the model, effort, subagent models and subagent cap the swarm sets for an agent, or change the model, effort and subagent models.
`get` prints the subagent cap (`max_subagents`); `set` has no flag for it and keeps a stored one. Only swarm-ui sets it.
A field the swarm sets wins over the agent's own value (its agent.nix or a `hyperhive-model`/`hyperhive-effort` file) from its next turn on; the agent reads its settings every ten seconds. A field the swarm doesn't set, or clears, leaves the agent on its own value.
**Usage:** `swarmctl agent inference <COMMAND>`
###### **Subcommands:**
* `get` — Show what the swarm sets for an agent, naming the fields it doesn't set, and the model and effort the agent runs on, each with where it comes from
* `set` — Set fields for an agent. Only the flags given change; the others keep what's stored
* `clear` — Clear every field for an agent, so it goes back to its own values
## `swarmctl agent inference get`
Show what the swarm sets for an agent, naming the fields it doesn't set, and the model and effort the agent runs on, each with where it comes from
**Usage:** `swarmctl agent inference get [OPTIONS] <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent inference set`
Set fields for an agent. Only the flags given change; the others keep what's stored
**Usage:** `swarmctl agent inference set [OPTIONS] <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name. A hive must declare it
###### **Options:**
* `--model <MODEL>` — The model its turns run on, as claude's `--model` takes it
* `--effort <LEVEL>` — The reasoning effort its turns run at
Possible values: `low`, `medium`, `high`, `xhigh`, `max`
* `--subagent-models <MODELS>` — The models it may start subagents on, comma-separated. Stored and shown; the agent doesn't apply it yet
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent inference clear`
Clear every field for an agent, so it goes back to its own values
**Usage:** `swarmctl agent inference clear [OPTIONS] <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent permissions`
Show or change the permissions the swarm grants an agent.
The agent applies its tool permissions at its next turn; the hive running it restarts it, after its current turn, when `manage_root_agent` changes. Unset leaves the agent on the default: `messaging`, `meta`, `inbox`, `execution`, whatever its name.
**Usage:** `swarmctl agent permissions <COMMAND>`
###### **Subcommands:**
* `get` — Show the permissions the swarm grants an agent, or that it sets none
* `set` — Replace the permissions the swarm grants an agent; the internet flag stays unless given
* `clear` — Clear the agent's permissions, so the default applies again
## `swarmctl agent permissions get`
Show the permissions the swarm grants an agent, or that it sets none
**Usage:** `swarmctl agent permissions get [OPTIONS] <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent permissions set`
Replace the permissions the swarm grants an agent; the internet flag stays unless given
**Usage:** `swarmctl agent permissions set [OPTIONS] --permissions <PERMISSIONS> <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name. A hive must declare it
###### **Options:**
* `--permissions <PERMISSIONS>` — The permissions to grant, comma-separated: any of `messaging`, `meta`, `inbox`, `execution`, `web_tools`, `manage_root_agent`, `logs`, `metrics`. `--permissions ''` grants none
* `--unrestricted-internet <BOOL>` — Whether the agent's traffic may leave the host unfiltered. Left out, the stored value stays
Possible values: `true`, `false`
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent permissions clear`
Clear the agent's permissions, so the default applies again
**Usage:** `swarmctl agent permissions clear [OPTIONS] <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent agents-md`
Show an agent's AGENTS.md, or edit it with a diff.
`get` shows the copy the agent last published, so it works while the agent isn't running. `apply` hands the diff to the agent, which applies it to the file with `git apply`; it needs the agent running.
**Usage:** `swarmctl agent agents-md <COMMAND>`
###### **Subcommands:**
* `get` — Print an agent's AGENTS.md on stdout, and its revision on stderr
* `apply` — Apply a unified diff to an agent's AGENTS.md
## `swarmctl agent agents-md get`
Print an agent's AGENTS.md on stdout, and its revision on stderr
**Usage:** `swarmctl agent agents-md get [OPTIONS] <AGENT>`
###### **Arguments:**
* `<AGENT>` — Agent name
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl agent agents-md apply`
Apply a unified diff to an agent's AGENTS.md.
The diff names `a/AGENTS.md` and `b/AGENTS.md`, as `diff -u a/AGENTS.md b/AGENTS.md` writes it. The agent refuses it when the file is no longer at `--base`, or when it doesn't apply, and leaves the file as it was.
**Usage:** `swarmctl agent agents-md apply [OPTIONS] --base <REVISION> <AGENT> <DIFF_FILE>`
###### **Arguments:**
* `<AGENT>` — Agent name
* `<DIFF_FILE>` — The file holding the diff
###### **Options:**
* `--base <REVISION>` — The revision you made the diff against, as `get` printed it. It's the git blob id of that text, so `git hash-object` of the unedited file prints it too
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl schedule`
Manage scheduled prompts.
The swarm keeps each prompt, and every agent it names fires it in its own container. Fire times are the schedule's start time plus whole intervals: changing the body never moves them, and an agent that was down fires the latest missed one once.
**Usage:** `swarmctl schedule <COMMAND>`
###### **Subcommands:**
* `add` — Add a scheduled prompt and print it with its new id
* `list` — List scheduled prompts, one row each, with their agents and next fire time
* `edit` — Change a scheduled prompt
* `pause` — Stop a scheduled prompt firing. Times that pass while it's paused aren't fired when it resumes
* `resume` — Let a paused scheduled prompt fire again, from its next time on
* `rm` — Remove a scheduled prompt from every agent it names
## `swarmctl schedule add`
Add a scheduled prompt and print it with its new id.
Without `--every` it fires once. Without `--at` its first fire is now.
**Usage:** `swarmctl schedule add [OPTIONS] --name <TEXT> --agent <AGENT> --body <TEXT>`
###### **Options:**
* `--name <TEXT>` — The prompt's title
* `--agent <AGENT>` — Agent to fire it in. Repeatable; at least one
* `--body <TEXT>` — The prompt, delivered to each agent verbatim
* `--at <TIME>` — First fire time: an RFC 3339 time with its offset, such as `2026-10-05T09:00:00+02:00`, or unix seconds. Defaults to now
* `--every <DURATION>` — Time between fires: a count with a unit of `s`, `m`, `h` or `d`, such as `1h` or `7d`. Leave out for a one-shot
* `--paused` — Add it paused
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl schedule list`
List scheduled prompts, one row each, with their agents and next fire time
**Usage:** `swarmctl schedule list [OPTIONS]`
###### **Options:**
* `--agent <AGENT>` — Show only the prompts that fire in this agent
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl schedule edit`
Change a scheduled prompt.
Only the flags given change, and the id stays. `--at` and `--every` move the fire times; `--name`, `--body` and the agent flags don't.
**Usage:** `swarmctl schedule edit [OPTIONS] <ID>`
###### **Arguments:**
* `<ID>` — The prompt's id, as `schedule list` shows it
###### **Options:**
* `--name <TEXT>` — New title
* `--body <TEXT>` — New prompt text
* `--at <TIME>` — New first fire time, in the form `schedule add --at` takes
* `--every <DURATION>` — New time between fires, in the form `schedule add --every` takes
* `--once` — Make it a one-shot
* `--add-agent <AGENT>` — Agent to start firing it in. Repeatable
* `--remove-agent <AGENT>` — Agent to stop firing it in. Repeatable. Fails if the agent isn't a target, or if no agent would remain
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl schedule pause`
Stop a scheduled prompt firing. Times that pass while it's paused aren't fired when it resumes
**Usage:** `swarmctl schedule pause [OPTIONS] <ID>`
###### **Arguments:**
* `<ID>` — The prompt's id, as `schedule list` shows it
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl schedule resume`
Let a paused scheduled prompt fire again, from its next time on
**Usage:** `swarmctl schedule resume [OPTIONS] <ID>`
###### **Arguments:**
* `<ID>` — The prompt's id, as `schedule list` shows it
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl schedule rm`
Remove a scheduled prompt from every agent it names
**Usage:** `swarmctl schedule rm [OPTIONS] <ID>`
###### **Arguments:**
* `<ID>` — The prompt's id, as `schedule list` shows it
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl user`
Manage subjects in the swarm's SSO provider
**Usage:** `swarmctl user <COMMAND>`
###### **Subcommands:**
* `add` — Add a user, generating a password for them
* `reset-password` — Set a new password for an existing user, generating it the same way `user add` does
* `update` — Change an existing user's attributes
* `list` — List every user in authelia's users database
## `swarmctl user add`
Add a user, generating a password for them
**Usage:** `swarmctl user add [OPTIONS] <USERNAME>`
###### **Arguments:**
* `<USERNAME>` — Login name. Conservative ASCII only — it's a YAML map key and reaches access-control rules and logs
###### **Options:**
* `--display-name <TEXT>` — Name shown in the SSO UI. Defaults to the username
* `--email <ADDRESS>`
* `--group <GROUP>` — Repeatable
## `swarmctl user reset-password`
Set a new password for an existing user, generating it the same way `user add` does.
authelia's file store has no self-service reset (no SMTP notifier), so this is the only way a human account gets a new password once an operator forgets the old one. Refuses if the user doesn't exist — there's no separate "create" path here.
**Usage:** `swarmctl user reset-password <USERNAME>`
###### **Arguments:**
* `<USERNAME>` — Login name of an existing user
## `swarmctl user update`
Change an existing user's attributes.
Every flag is optional and they compose, so one call can set multiple things at once. Deliberately **doesn't** touch the password: regenerating a credential is a different intent from editing an attribute, and folded together an attribute edit can invalidate a login by accident.
**Usage:** `swarmctl user update [OPTIONS] <USERNAME>`
###### **Arguments:**
* `<USERNAME>` — Login name of an existing user
###### **Options:**
* `--display-name <TEXT>` — Name shown in the SSO UI
* `--email <ADDRESS>`
* `--add-group <GROUP>` — Repeatable. Adding a group the user is already in isn't an error
* `--remove-group <GROUP>` — Repeatable. Fails if the user isn't in the group — a revocation that reports success without revoking is the failure nobody re-checks
## `swarmctl user list`
List every user in authelia's users database.
Read-only: it never writes the file. Shows every subject in it, including agent identities `swarm-authelia-bridge` created — one line per user: username, display name, email (if set), groups (if any).
**Usage:** `swarmctl user list`
## `swarmctl forge`
Manage human accounts on the swarm's forge
**Usage:** `swarmctl forge <COMMAND>`
###### **Subcommands:**
* `make-admin` — Make an existing forge user a site admin
## `swarmctl forge make-admin`
Make an existing forge user a site admin.
The forge creates a human's account on their first SSO login, and this fails until that login has happened. Running it on a site admin changes nothing. Refused for an agent.
**Usage:** `swarmctl forge make-admin [OPTIONS] <NAME>`
###### **Arguments:**
* `<NAME>` — Forge username, the same as the user's SSO username
###### **Options:**
* `--controller-socket <PATH>` — swarm-controller's unix socket.
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
## `swarmctl login`
Log in to the swarm named by `--swarm` as your own SSO account.
With no `--swarm` or `SWARMCTL_SWARM`, logs in to `SWARMCTL_LOGIN_DEFAULT_SWARM`: on a swarm-controller host, the swarm that host belongs to.
Opens the swarm's SSO login in your browser (and prints the URL) and receives the answer on a loopback port, so run it where that browser runs. swarmctl keeps the login, mode 0600, under `$XDG_STATE_HOME/swarmctl/` and refreshes it as it's used. To log out, delete that file.
**Usage:** `swarmctl login`
## `swarmctl completions`
Generate a shell completion script for `swarmctl` and print it to stdout.
Supports bash, zsh, fish, elvish and powershell. The nix package already installs bash/zsh/fish system-wide; this is for ad-hoc or other-shell use.
Dispatched before `PathArgs::resolve()` for the same reason as `markdown-docs`: emitting a completion script needs none of the `SWARMCTL_AUTHELIA_*` deployment env vars, and requiring them would make the package's own build-time invocation fail.
**Usage:** `swarmctl completions <SHELL>`
###### **Arguments:**
* `<SHELL>` — Shell to emit completions for
Possible values: `bash`, `elvish`, `fish`, `powershell`, `zsh`

682
docs/2-handbook/swarm/README.md Normal file
View file

@ -0,0 +1,682 @@
# The swarm
The **swarm** is where things live: agent identities and accounts, secrets,
the job graph that creates and places agents, telemetry, and the UI you
drive it all from. **Hives are the substrate** — NixOS hosts that run agent
containers on the swarm's behalf. Every hive belongs to a swarm; a single
host is a swarm of one.
This page is for the operator. It covers the control plane, how a hive
joins the directory, and how hives and agents report upward. The steps for a
fresh swarm are in [`setup.md`](../../1-getting-started/setup.md); the all-local
config is the [README quick start](../../../README.md#quick-start-an-all-local-swarm).
Option reference: `services.hyperhive.swarm.*` (swarm-wide facts, identical
on every host) and `services.hyperhive.deploy.*` (whether _this_ host runs
grafana, the controller, authelia, …) →
[options reference](https://hyperhive.darkest.space/options/), or
`nix build .#docs-swarm` / `.#docs-deploy`.
## Where each piece lives
- **control plane** — `swarm-controller`: hive directory, agent roster, job
graph, agent creation. → [below](#swarm-controller)
- **scheduled prompts** — named prompts for one or more agents at set times,
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
over its own. → [`inference.md`](../../4-implementation/swarm/inference.md)
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
over its hive's. →
[`coordinator.md`](../../4-implementation/scheduler/coordinator.md#container-resource-limits)
- **review classifier** — checks a PR against its repo's own yes/no rules
and posts the result as a review. →
[`review-classifier.md`](../../4-implementation/swarm/review-classifier.md)
- **notification router** — forge events turned into todos on the agents
they concern: assigned, unassigned, cleared on close; and the classifier's
review requested on a push. →
[`notify-router.md`](../../4-implementation/swarm/notify-router.md)
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
`admins` only. → [`ui.md`](ui.md)
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
stack, each on whichever host you put it. → [`services.md`](services.md)
- **SSO** — the secrets authelia generates and how each reaches its reader.
→ [`sso.md`](sso.md)
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
every credential the swarm holds, who mints it, who reads it, how it renews
and where it lives → [`credentials-reference.md`](credentials-reference.md) ·
how the store comes up, who writes its grants and how it unseals →
[`bao.md`](../../4-implementation/swarm/bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer. → [`ca.md`](../../4-implementation/swarm/ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint,
`swarm.snapshotStore.{address,port}`. →
[`snapshot-store.md`](../../4-implementation/networking/snapshot-store.md)
## Swarm controller
`swarm-controller` is the swarm's control plane: it holds the hive directory,
the agent roster and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent,
the step it stopped at and the error, which the Grafana alert rule
`hh-agent-identity-mint-failed` fires on. The agent is still created, without a
store identity, and the controller's five-minute credential pass mints one for
it (also when only the certificate exists and its cert-auth role is missing) as
long as the agent isn't `Destroyed`. A mint that runs after a hive declares the
agent `Destroyed` logs `identity mint skipped` and writes nothing.
`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
runs the controller.** A swarm has one controller, so enabling it states a
fact about swarm topology, not about whether this host runs a hive.
What it serves, why it's a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../../swarm-controller/README.md).
### Per-hive status (`GET /api/hives/status`)
One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.
A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.
| freshness | what to do about it |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive removed from the roster but still running |
The snapshot is the hive's `/health/ready` document plus `capabilities`, a list
of names for what the hive's release can do. Two exist: `tolerant-permissions`, and `no-config-checkout`, which
[turning an archetype agent into a custom one](archetypes.md#turn-into-custom)
needs. A hive on an older release lists none.
The snapshot also carries `egress_deny_applied`, the revision of the egress deny
list the hive last rendered into its proxy (`0` when the swarm has none). A hive
on an older release, or one that hasn't read the list yet, leaves it out; see
[the egress proxy](../../4-implementation/networking/network.md#egress-proxy-squid).
Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload. A roster hive's row
carries `dashboard`, the host its dashboard answers on; the swarm UI's
hive list links there.
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.
### Making a hive report
Two options on the **hive**, plus its store identity. They sit in two
namespaces, because the queue's URL is a fact about _this machine_ and the
token endpoint is the swarm's single address:
| option | what to set it to |
| ---------------------------------------------- | --------------------------------------------- |
| `deploy.hive-controller.statusPublish.natsUrl` | `tls://<swarm.nats.domain>:<swarm.nats.port>` |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). Authelia's mint, in its
container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config`
warning and tries again, waiting 5 seconds after the first failure and doubling
up to 5 minutes. It starts publishing as soon as a read succeeds, with no
restart.
The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.
The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue
host without the store gets no default, even when it holds the hive's own
reader `deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.natsCommonName`, place it, and name it in both options. Until you
do, `swarm-bao-nats-tls` fails naming both whenever the queue's certificate is
missing or due for renewal.
The queue host's gateway also serves the queue's metrics, at
`https://<swarm.nats.domain>/metrics`, so a swarm collector on any host
scrapes them. Only the collector's authelia client gets through; any other
session gets `deny`, and every other path on that name is 403. With the
gateway's default self-signed TLS, that vhost serves the queue's own
certificate, so a host whose only swarm service is the queue needs no
swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.
The endpoint answers **503** when this host has no swarm queue
configured, or has one and can't read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
can't see. The body says which. Status survives a controller restart:
it's stored in the queue, not in the daemon.
### Giving the agents the queue too
An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:
| option | what to set it to |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
It reaches each container at start, as `HIVE_AGENT_NATS_URL` in
`/etc/hyperhive-hive-facts`, so no agent build carries it. On every hive,
`agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. The option takes one
`<scheme>://<host>:<port>` (a hostname or IPv4 address), not an IPv6 literal or a
list: evaluation refuses anything else. On the queue's
host the name resolves inside a container to the bridge address, where the
firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.
**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:
- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits. The error reads
`no swarm queue connection; exiting, since this agent cannot run without one`,
and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.
Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading
the credential from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for a `swarm-logs` built before it read the agent's
own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.
<details><summary>What an agent publishes over the queue</summary>
What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. The queue keeps the rows in
the stream `term-main-<agent>` for 24 hours or 64 MiB, the same limits and the
same creator as the subagent stream below, so a swarm terminal that attaches
later replays the newest of them. The two streams hold disjoint subjects.
An agent's **subagents** publish their terminals too, from the agent's
subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`,
classified the same way. The queue grants that family to the agent's own queue
credential alone, so the daemon reads it from the store under the agent's store
identity, exactly as the harness does, and publishes nothing without it. The
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
whichever comes first, dropping the oldest rows once either limit kicks in so
a publish never fails because of it, and the swarm lists an agent's subagents
from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for
every agent a hive's wanted state names, within a minute of the agent
appearing there. The agent's grant is publish on its own
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
subjects and limits are the controller's and never the agent's. Subagents
publish output only and read nothing.
The queue would refuse a row too large for its `max_payload` outright and
take the connection down with it, so the harness drops such a row's body before
sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.
The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model and effort (`model`, `effort`, where each comes from as
`model_source` and `effort_source`, and the resolved id the last turn
actually ran on), the
context budget and the last turn's context and cost token blocks,
`agent_state`, and `needs_login`: `true` while the agent's claude has no
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.
`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
spellings. ⚠️ From inside the container only two of those four are sayable: the
harness reports `up`, or `paused` when the pause marker is present. `offline` and
`destroyed` are hive-c0re's observations — a stopped agent publishes nothing and
a destroyed one doesn't exist — so a view that needs the full four-state picture
takes them from the `agent-status` bucket and uses this subject to sharpen the
rest.
Headers go out **on transition**: the harness rebuilds the header whenever its
event bus moves and publishes when the result differs from what it last sent.
That's the whole point of the subject — the `agent-status` bucket already
republishes once a minute, which is far too slow for "is this agent thinking
right now." A core subject keeps nothing for a late subscriber, so the harness
also re-sends the current header once a minute: a subscriber attaching
mid-idle sees it within that minute. The swarm's stream sends a new client the
latest header it holds first, when that's under 150 seconds old. A renderer opens with the bucket's
snapshot and lets this stream refine it.
Swarm-side, `GET /api/agents/<name>/state/stream` relays that subject as SSE,
resolving the agent's hive at request time exactly as the terminal stream does.
The payload passes through opaquely. Separately, the controller keeps each
agent's latest `model`, `effort` and their sources for
`GET /api/agents/<name>/inference` ([`inference.md`](../../4-implementation/swarm/inference.md)), and each
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key of that bucket. The harness writes
once per start, because a config change reaches an agent by restarting its
container. An agent with no icon deletes its key. Swarm-side,
`GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.
The fourth is its **`AGENTS.md`**, which goes into the `agents-md` bucket under
the key `<agent>`, writable by the agent's own credential alone
(`$KV.agents-md.<agent>`). The swarm reads that copy and edits the file through
the agent; [An agent's AGENTS.md](#an-agents-agentsmd) has the details.
The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>`, its operator-input subject `$SWARM.input.<agent>` and
its command subject `$SWARM.control.<agent>`, and nothing else. Besides the
subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies.
Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
sets as their inbox prefix. A KV watch or key listing delivers there as well. A
hive may also subscribe to `$SWARM.knowledge` and its own `$SWARM.deploy.<hive>`,
so it can't read another hive's deploys or any agent's terminal. The controller
may also subscribe to `$SWARM.term.<agent>`, `$SWARM.term.<agent>.sub.<subagent>`
and `$SWARM.agent-state.<agent>`, the streams it relays to swarm-ui. A hive or
controller that connects with the library's default `_INBOX.<nuid>` prefix gets
no replies, so every request it makes times out.
</details>
### An agent's AGENTS.md
The file in the agent's state dir, `state/AGENTS.md`, is the only copy that
counts, and only the agent writes it: itself, or its harness applying an edit
from the swarm. No hive is in either path.
**Reading.** The harness publishes the file's text and its revision, the git
blob id that `git hash-object` prints for it, at start, within ten seconds of
any change, and straight after an edit it applied. An agent with no `AGENTS.md`
publishes `""`. `GET /api/agents/<name>/agents-md` answers `{content, revision}`
from that copy, so it works while the agent isn't running. A 404 means the agent
hasn't published one: its harness hasn't run since it began to.
**Initial text.** `POST /api/agents` and `swarmctl agent create --agents-md <file>`
take an optional `agents_md`, the text the file starts with, up to 64 KiB; a
longer text is a 400. The controller stores it under the key `<agent>` of the
`agents-md-initial` bucket, which only it writes, before it queues the creation.
At start the harness reads its own key, with one direct get on
`$JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>`, and
writes the text to `AGENTS.md` only when no file exists there. It does this
before the first mirror publish and the first turn. An existing file is never
replaced, so a restart or a re-create keeps the agent's edits; a later change
to the file never reaches the key. Destroying the agent deletes the key. When
the harness can't read the key it starts without the text and logs a warning.
The text holds no secret, so nothing goes through the secret store.
**Editing.** `PUT /api/agents/<name>/agents-md` takes `{base_revision, diff}`.
The diff is a unified diff that names `a/AGENTS.md` and `b/AGENTS.md`, made
against the text at `base_revision`. swarm-controller sends it as a request on
the agent's command subject `$SWARM.control.<name>`, as it sends a cancel. The
harness refuses it when the file isn't at `base_revision`. Otherwise it runs
`git apply --check` and then `git apply` on a copy of the file, and writes the
result over the file only when both pass:
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| 200 | applied; the body is `{revision}`, the file's new revision |
| 409 | the file isn't at `base_revision` any more; nothing applied, and `revision` in the body is the one it's at |
| 422 | `git apply` refused the diff, or it touches a file other than `AGENTS.md`; `detail` carries git's message, and the file stays as it was |
| 503 | the agent isn't running or isn't connected to the queue |
| 504 | the harness didn't answer within ten seconds, so whether the edit applied is unknown |
| 500 | the harness couldn't read or write the file or run git, or the queue failed the request |
The harness refuses a diff that sets any file mode other than `100644`, so an
edit can't turn `AGENTS.md` into a symlink. `PUT` is operator-only, behind
`Remote-User` like cancel: whoever can call it can rewrite the agent's
instructions, so don't widen who holds it without revisiting this.
An edit needs the agent running, and a read of a stopped agent shows the text
it last published. A harness older than this route doesn't know the command and
drops it unanswered, which reads as a 504. `swarmctl agent agents-md get` and
`apply` call the two routes.
### Swarm-wide forge objects
The controller also keeps the forge objects that are one per swarm, not
one per hive. It ensures them at start and every five minutes after
(`swarm-controller/src/forge/objects.rs`):
- the orgs `agent-configs`, `internal` and `agents`, plus each mirror's
owner org;
- the empty `operators` merge-gate team in `agents` and `agent-configs`;
- the `main` merge gate on every `agent-configs` repo: merge and approval
whitelists = the `operators` team, and no user. The controller leaves a
repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
host (with the `actions/checkout` and `actions/upload-artifact` ones
`deploy.forgejo.actionsFromSelf` adds);
- `internal/docs` (private) and `internal/knowledge` (public, with a
seed `README.md` while empty);
- the `agent-configs` org avatar
(`deploy.swarm-controller.configOrgAvatarPng`).
A pass that can't finish logs a
`warn` line per object plus `swarm forge objects: pass incomplete` in
`journalctl -u swarm-controller`, and retries on the next tick. While the
controller is down the objects stay as they are.
The controller gives each agent **read** on `internal/docs`. A `GrantDocsRead`
node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the
store's `hive-agent-*` roster (`forge/docs_grant.rs`, at start and every five
minutes) adds any agent that isn't one, so an agent whose create-time grant failed
gets it within five minutes. The pass only adds: a roster it can't read, a
forge read that fails, or an `internal/docs` that doesn't exist (checked once per
pass, logged at warn) changes nothing. The grant doesn't depend on the hive
the agent runs on, or on a forge container there.
### Swarm-wide forge webhooks
At startup the controller registers its Forgejo hooks pointing at
itself, all under `https://<swarm.domain>/webhook/forge/`:
- a `push` hook on `internal/knowledge`
- a `pull_request` hook on the `agent-configs` org
- an instance-wide `push` system hook, for commit metrics
- an instance-wide `pull_request_review_request` system hook, which starts
the [review classifier](../../4-implementation/swarm/review-classifier.md#trigger-and-lifecycle)
- instance-wide `issues` + `issue_assign` and `pull_request` +
`pull_request_assign` + `pull_request_sync` + `pull_request_review_approved` +
`pull_request_review_rejected` + `pull_request_review_comment` system hooks, whose deliveries it
keeps in the `SWARM_EVENTS` stream for the
[notification router](../../4-implementation/swarm/notify-router.md#the-path)
The controller **interprets** a delivery and sends hives a specific
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.
The `issues` and `pull_request` deliveries are the exception: no hive gets
them, and the notification router, not the controller, decides what they mean.
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
of its `merge_commit_sha` on the one hive whose wanted state places the
agent; with no such hive, or more than one, the controller deploys nothing. See
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes).
A merge whose `closed` delivery never arrives deploys nothing either; the
operator recovers by redelivering that delivery from the forge hook page,
which re-enters the same handler and re-queues the deploy.
**`internal/knowledge` is on that path.** The controller's is the only
hook on it ([`knowledge.md`](../integrations/knowledge.md) covers clearing a
leftover). A webhook has exactly one target URL, so a second registration
would take delivery away from the first rather than add a recipient.
<!-- vale write-good.Passive = NO -->
**The `agent-configs` org is on it too.** The controller's hook is the
only one that acts on config PRs. Each hive deletes its own
`/webhook/config-pr` org hook on startup, since a route no hive serves
would otherwise sit on the org collecting failed deliveries. Removing
the controller's hook stops forge-UI merges from deploying until its
next start recreates it.
<!-- vale write-good.Passive = YES -->
Nothing to configure. The controller registers the hooks only when this
host also serves the swarm UI vhost — that's what publishes the
endpoint, and a hook the forge can't reach would collect failed
deliveries while looking healthy. The controller generates the HMAC
secret on first start and keeps it
(see [`docs/2-handbook/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)).
To check it's working, push to `internal/knowledge` and look for
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
refused delivery logs `webhook: refused delivery` with the reason.
The controller registers the hooks within five minutes of having both the forge
token and the secret. It keeps trying, at most five minutes apart, for as long
as the forge or the secret store is down. A restart isn't needed to recover.
## Hives: the substrate
- **hive** — one host running `hive-c0re` and its agent containers
(`deploy.hive-controller.enable`). Addressed as `<hiveName>.<swarm.domain>`.
- **swarm** — every hive in `services.hyperhive.swarm.hives`, plus the
shared services and controller. You can qualify an agent as
`agent@hive-domain`.
- **peer hive** — any hive in the directory other than this one. Peers are
_derived_, not declared: the directory lists every hive including
yourself, and `hiveName` says which one you are.
### Hive identity config
```nix
services.hyperhive = {
swarm.domain = "example.com"; # required — the swarm's DNS domain
deploy.hive-controller.hiveName = "pr1ma"; # required on a hive host — this hive's label in it
swarm.name = "constellat1on"; # shared swarm display name (optional)
# required — the directory, identical on every host in the swarm.
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
swarm.hives = {
pr1ma = { }; # this host, per hiveName
lab = { }; # a second hive
edge = { domain = "edge.elsewhere.example"; }; # addressed off-convention
};
};
```
<!-- vale write-good.Passive = NO -->
Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.
<!-- vale write-good.Passive = YES -->
**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.
```
# hive A # hive B
deploy.hive-controller.hiveName = "pr1ma"; deploy.hive-controller.hiveName = "edge";
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
```
One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.
`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`iris@pr1ma.example.com`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.
`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
The env-var chain and `qualify()` semantics:
[`conventions.md`](../../5-project/process/conventions.md) § Hive identity.
Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](../../4-implementation/swarm/ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.
### What the directory feeds
1. **Swarm-wide hive roster** — swarm-controller reads this same
directory and serves it at `GET /api/hives`; the swarm UI's front page
renders it ([`ui.md`](ui.md)).
2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
federates with the peer's matrix server (discovered via the peer's
`.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
Federation validates the peer's TLS certificate against the matrix
**container's** trust bundle, independent of this directory.
⚠️ On a hive with self-signed gateway certificates, the container
also trusts this hive's `trust-bundle.pem`, bound in at runtime and
added to the public CAs; it holds the store's `pki` root, so a peer whose
certificate chains to the same root validates. A peer outside this
swarm's root needs a CA-issued certificate (ACME). Federation firewall
and TLS requirements: [`integrations/matrix.md`](../integrations/matrix.md).
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).
### WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.
#### Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
#### Config example (two hives)
```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
pr1ma = {
domain = "pr1ma.example.com";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
edge = {
domain = "edge.corp";
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = { /* … identical to hive A's … */ };
};
```
#### What the mesh does
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
host (not inside agent containers; containers reach peers via the
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
from `services.hyperhive.swarm.hives`, minus this host's own hive, to
build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
#### NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.
## Cross-references
- [`ui.md`](ui.md) — the swarm UI, the operator surface for "what hives exist"
- [`../networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) — the
swarm's `btrfs receive` endpoint and the `swarm.snapshotStore` option
- [`../process/conventions.md`](../../5-project/process/conventions.md) § Hive identity —
env vars, qualified labels
- [`../integrations/matrix.md`](../integrations/matrix.md) — matrix
federation, TLS cert autogeneration, firewall posture
- [`../networking/gateway.md`](../../4-implementation/networking/gateway.md) — nginx vhosts and
the `.well-known/matrix/` autodiscovery scheme

195
docs/2-handbook/swarm/archetypes.md Normal file
View file

@ -0,0 +1,195 @@
# Agent archetypes
An archetype is a named agent setup: settings to create the agent with, and
a NixOS module its container runs. The swarm UI's create form lists the
archetypes the swarm knows. Picking one fills the form's permissions,
resource caps and inference fields from it, and you can still change each
field before you create the agent.
An agent created from an archetype has no `agent-configs/<name>` repo of its
own. The swarm's `internal/meta` flake builds it from the archetype's module,
so every agent of that archetype runs the same configuration. To change what it runs, change the
archetype. Picking "none" creates a custom agent, with its own config repo. The
swarm can also turn an archetype agent into a custom one later
([below](#turn-into-custom)).
## The built-in archetypes
The hyperhive flake ships these:
| archetype | what it sets |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reviewer` | permissions `messaging`, `inbox`, `execution`: the default set without `meta`. With `execution` it can run `hive-forge`, `git` and other commands. |
| `android-dev` | installs `android-tools` (`adb`, `fastboot`), Gradle and a JDK; default permissions. |
| `reverse-engineer` | sets `services.hyperhive.agent.rea.enable`: installs REA, Ghidra 12.1 and procps, and allows all REA MCP tools; default permissions and the agent's normal memory cap. |
## Where an archetype comes from
A flake declares archetypes in its `hyperhiveArchetypes` output:
```nix
hyperhiveArchetypes.reviewer = {
module = { };
meta = {
description = "Reviews what it is sent.";
permissions = [ "messaging" "inbox" "execution" ];
limits = { memory_max = "4G"; };
inference = { effort = "high"; };
};
};
```
- `meta` is plain data: no derivation, no path, no function. The swarm
evaluates `meta` alone.
- Every archetype needs a `description`. `permissions`, `limits` and
`inference` are optional and take the shapes `POST /api/agents` takes. The swarm ignores
other keys in `meta`.
- `module` is a NixOS module for the agent's container. The meta flake
imports it into each agent created from the archetype.
Today the swarm reads one flake: the hyperhive source the
[builder](../../4-implementation/swarm/builder.md)'s host runs, so only the built-ins appear. The meta
flake has one archetype source too, its own `hyperhive` input, and refuses
an archetype from any other source name.
## How the list stays current
1. swarm-controller asks the builder for the list on the queue subject
`$SWARM.builder.archetypes` once the queue connects, and every 10 minutes
after that.
2. The builder evaluates the `meta` of each `hyperhiveArchetypes` entry on
its own hyperhive and nixpkgs sources, and answers with the JSON or the
last lines `nix` printed:
```sh
nix eval --json <hyperhive>#hyperhiveArchetypes \
--apply 'builtins.mapAttrs (_: a: a.meta)'
```
3. The controller checks each archetype the way `POST /api/agents` checks
its settings and stores the result in the `agent-archetypes` KV bucket,
under the key `hyperhive`, when it differs from the stored one. The list
leaves out an archetype that fails the check, and its name and reason go
into the record's `refused` map and the controller's journal.
4. `GET /api/archetypes` answers from that record, and the create form lists
from it.
A builder host redeploy can change the list. The next pass, within 10
minutes, picks it up. When no builder answers, the controller keeps the
stored list and logs a warning on each pass. A swarm with no builder enabled
shows no archetype select.
The stored value:
```json
{
"source": "/nix/store/…-hyperhive-flake-source",
"archetypes": {
"reviewer": {
"description": "…",
"permissions": ["messaging", "inbox", "execution"]
}
},
"refused": { "broken": "limits: …" }
}
```
## Creating an agent from an archetype
The create form sends the picked archetype as `archetype` on
`POST /api/agents`, `{ "source": "hyperhive", "name": "reviewer" }`. The
controller refuses with 400 an archetype the stored list doesn't hold, then:
1. stores the choice in the `agent-archetype` KV bucket, keyed by agent name,
next to the agent's permissions, caps and inference;
2. queues the creation graph without `CreateRepo`, `AddRepoMember` and
`InitAgentConfigRepo`. Identity, forge user, docs read, the mints and the
wanted state stay, and the deploy waits on those alone.
Destroying the agent deletes its key.
Every hive may read any key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-archetype.>`) and write none; only the
controller writes it.
Deploy `swarm-nats-auth` before the hives: every provision reads the
agent's key, and a hive whose queue connection predates the grant gets a
refused read, so its `Provision` nodes fail until it reconnects.
swarm-controller renders `internal/meta`'s `flake.nix` with the
`hive-meta-flake` crate, and reads each agent's archetype from the
`agent-archetype` bucket to do so. For an archetype agent the render:
- adds no `agent-<name>` input and reads no config repo, and lists the agent
as `mkAgent { …; archetype = { source = "hyperhive"; name = "<name>"; }; }`;
- makes `mkAgent` import `hyperhive.hyperhiveArchetypes.<name>.module` in
place of the agent's own flake, and set `HIVE_ARCHETYPE` in the harness
environment.
The swarm builds from `internal/meta` alone and the hive deploys that closure. The hive reads the agent's key when it
provisions the agent and on every rebuild, and keeps the last value read in
`/var/lib/hyperhive/swarm-agent-archetypes.json`. When the read fails, a rebuild keeps the stored value. For an
archetype agent the hive also:
- deploys the closure its record declares, as every agent does.
The agent's system prompt says it has no config repo and names its
archetype.
The module comes from the `hyperhive` input that `internal/meta` pins, not
from the hive's own. When that pin predates the archetype, the agent's build
fails with `this hive's hyperhive input has no archetype <name>; update that
input`. Update the `hyperhive` pin in `internal/meta` to fix it.
## Turn into custom
An archetype agent can become a custom agent with its own
`agent-configs/<name>` repo, either from the agent's overview in the swarm UI
(the "turn into custom" button asks first) or with `POST /api/agents/{name}/custom`.
The agent keeps its state, permissions, caps, inference settings and accounts.
Its configuration stays the same, except that it no longer has `HIVE_ARCHETYPE`
in its environment. Keeping the config current is then up to the repo's owner.
Nothing here undoes it.
The controller refuses with 409 an agent that no hive or more than one hive
places, and one whose hive doesn't list the
`no-config-checkout` capability (see below). Otherwise it queues a job graph:
1. `CreateRepo`, `CreateForgeUser` and `AddRepoMember`: the repo and the
agent's access to the repo, as for any custom agent. Each is a no-op for what
exists.
2. `SeedArchetypeConfig` commits two files to the repo. It copies nothing from
the archetype:
- `flake.nix` declares the archetype's source as an input, at the URL
`internal/meta` holds for that source today (a pinned rev, which is the input's only pin: the repo has no lock, and a source meta doesn't pin to a commit fails the node), and imports
`inputs.<source>.hyperhiveArchetypes.<name>.module` next to `./agent.nix`;
- `agent.nix` is an empty module for the owner's edits.
The node leaves a repo whose `flake.nix` already is this file alone, whatever
URL it pins. Any other `flake.nix` fails the node.
3. `DropArchetype` deletes the agent's `agent-archetype` key, only after the
seed and the member grant succeeded. A failure before it leaves the agent
archetype-built and working, and you can send the same request again.
4. `UpdateMeta`, then the build and the deploy, as for a merged config PR.
If the build fails, the agent is custom and has no key. Sending the request
again then queues only this step, as the earlier ones have run.
When the source is `hyperhive`, the meta flake sets
`agent-<name>.inputs.hyperhive.follows = "hyperhive"` for an agent whose
`flake.nix` or `flake.lock` declares a `hyperhive` input, so the lock holds one
hyperhive, and the module comes from the pin `internal/meta` holds. Any other
source keeps the pinned URL from its seeded `flake.nix`, and the meta lock
records that rev. The `follows` applies to every custom agent whose `flake.nix`
declares a `hyperhive` input, not only converted ones. A `hyperhive` node left
in a lock doesn't count.
### Which hives may convert
A hive that keeps a checkout of a custom agent's config would create one for
the converted agent, under the container's running mount. A hive that lists
`no-config-checkout` under `capabilities` in the status it publishes (the
document `GET /api/hives/status` serves as `snapshot`) keeps none: it deploys a
custom agent's closure from the swarm alone, and the agent clones its own repo from
the forge. The controller reads the agent's hive's latest status and refuses
unless it's `fresh` and lists the capability; an older hive lists none. Deploy
the release to every hive that places archetype agents before converting theirs.

104
docs/2-handbook/swarm/claude-login.md Normal file
View file

@ -0,0 +1,104 @@
# Claude login from the swarm UI
An operator logs an agent in to claude from the agent's card in the swarm UI.
No hive is in the path: the login runs in the swarm's `swarm-claude-login`
service, the credentials go to the agent's own leaf in the secret store, and the
agent's harness installs them from there.
Where it runs and how it's enabled: [`services.md`](services.md#claude-login-minter).
The service runs the `claude-code` that its host's
`services.hyperhive.deploy.hive-controller.claudeCodePackage`
pins for agents, and its container's own nixpkgs' where the host pins none.
## What happens on a login
1. The agent publishes `needs_login` in its turn-state header, and its card
shows **login needed** with a **log in** button.
2. **log in** opens a dialog, which calls
`POST /api/agents/<agent>/claude-login`. swarm-controller relays the start as
a request on the queue subject `$SWARM.claude-login.request`.
3. The minter runs `claude auth login` with `HOME` and `CLAUDE_CONFIG_DIR` in a
fresh scratch directory, over plain stdio pipes, and answers with the first
`https://` URL it prints. The dialog shows that URL.
4. The operator opens the URL, signs in to the agent's own claude account, and
pastes the code the page shows into the dialog, which posts it to
`POST /api/agents/<agent>/claude-login/code`.
5. The minter writes the code to claude's stdin and waits for it to exit. It
stores the `.credentials.json` claude wrote at
`swarm/agents/<agent>/claude-credentials` (field `value`), as a new version
over any stored before, and removes the scratch directory.
6. The agent's harness, while it waits for a login, reads that leaf every 30
seconds under the agent's own certificate. When the stored version is newer
than the one it installed last, it writes the file to
`~/.claude/.credentials.json`, records the version, and resumes its turn
loop as it does after a login from the agent's web UI.
A started login waits 15 minutes for its code, then the minter stops it.
Starting another login for the same agent replaces the running one. Each step
has a minute to finish; a refused code fails the login, and the dialog shows
what claude printed.
## Why the harness takes only a newer version
`claude` rotates the refresh token in `.credentials.json` as it uses the
session, and the agent can't write the store. After the first refresh the
stored copy is stale. A harness that installed it whenever the bytes differed
would replace the agent's live session with a stale copy and log the agent
out. The harness therefore compares the store's KV version with the last
version it installed (`claude-credentials-version` under the harness dir):
- it installs a newer version, that is, a new login;
- it leaves an equal or older version alone, even when the agent needs a
login again, so it never puts back a session that has gone bad;
- it reads nothing while the agent has a session.
## Credentials
| Principal | Grant |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `swarm-claude-login`, in its container | `create`, `update` on `swarm/agents/+/claude-credentials`; `read` on `swarm/services/<minter client id>/oidc/client` |
| the agent, under its own certificate | `read` on its own `swarm/agents/<agent>/*`, which covers the leaf |
The minter has no read on the leaf and nothing else under an agent. Every
hive's policy reads `swarm/agents/*`, so every hive can read every agent's
claude session, as it can read the GitHub token.
The `operatorsOnly` authelia rule on the swarm domain, not the controller,
keeps these routes to operators: the controller checks only that the gateway
named a user. Without that rule any account authelia admits could store its
own claude session as any agent's, so no agent account may exist before the
rule reaches the swarm.
The minter refuses a step whose agent name isn't an identifier before it
makes a directory or runs `claude`.
## Caveat
The installed-version record lives under the harness dir and the credentials
file under `~/.claude`. If only the record survives a rebuild, the agent
ignores the stored version until the operator logs it in again, which writes a
newer one.
## What stays
The agent's own web UI login (`/login`) still works, unchanged. It goes away
once the operator has confirmed the swarm login works.
## Observability
The minter pushes OTLP metrics through its container's forwarder to the swarm
collector, labelled `service.name=hyperhive-claude-login`. With no
`OTEL_EXPORTER_OTLP_ENDPOINT` it exports nothing. The client-certificate
gauges (`bao.client_cert.*`) come with them.
| Metric | Unit | Kind | Attributes |
| ----------------------------------------- | ---- | --------- | --------------------------------------------------------------------------------------------------- |
| `hyperhive.claude_login.steps` | — | counter | `step`: `start`, `code`, `cancel`, `invalid`; `outcome`: `started`, `stored`, `cancelled`, `failed` |
| `hyperhive.claude_login.step.duration` | `s` | histogram | `step` |
| `hyperhive.claude_login.sessions.expired` | — | counter | — |
| `hyperhive.claude_login.sessions.active` | — | gauge | logins waiting for a code, as of the last step |
`invalid` is a request that didn't decode or named no agent. A `start` spawns
`claude auth login` and a `code` stores the credentials, so those two are the
slow steps. `expired` counts logins stopped because their code never came.

189
docs/2-handbook/swarm/credentials-reference.md Normal file
View file

@ -0,0 +1,189 @@
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
# Credentials: the per-secret reference
Every credential in the swarm's secret store, in one table: who mints it, who
reads it, and whether it renews on its own. The overview and the rules a
credential follows are in [`credentials.md`](credentials.md); this page is
the detail, so it changes whenever someone adds a credential or moves its minting.
The second half, [where each secret lives on disk](#where-each-secret-lives-on-disk), maps the
files that exist today.
In the table, **automatic re-mint** is something replacing the stored value
without an operator. **Automatic re-pull** is the reader picking up a replaced
value without a restart. ✅ means it happens today, ❌ means it doesn't, and
the rest of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice render` in `hive-matrix`, when the store answers and holds none; before that, the tokens an earlier version kept in the container, stored unchanged | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` render the new pair and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read once; a store down at start is re-read in the background until it answers |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation, and by the renewal pass when none is stored or its cert-auth role is missing | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity; minted when none is stored or its cert-auth role is missing, for agents not `Destroyed` | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/observability` | `swarm-controller`, when an operator sets, clears or creates an agent's permissions with `logs` or `metrics` in the old or new list, and when it destroys the agent; the leaf stays absent for an agent that never held either | the agent (read-only, its own prefix); `swarm-logs-auth` and `swarm-metrics-agent-auth`, which deny when it's absent or unreadable | ✅ the controller rewrites it with the permission, and every five minutes makes it equal the record's | ✅ the stores honour a changed grant for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/gitlab/<label>` | `swarm-controller`, when you link a GitLab account through the API | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes; a deleted account leaves the agent's `glab` configuration on the next run |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/backend-env` | `hive-agent-backend-env`, in the agent's container, from the agent's `backendEnvironmentFile` when the leaf holds nothing (field `env`, the file's exact text); or `swarm-controller`, when you set the API-key environment in the swarm UI | the agent | ❌ nothing replaces it without you; a save in the swarm UI overwrites it, including text an agent wrote into an empty leaf; the agent's file stays as it was | ❌ the file in `/run` refreshes every 2 minutes, and the harness and the subagent daemon load it at their next start |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/agents/<agent>/proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds |
| `swarm/hives/<hive>/squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute, and while they can't reach the store they keep accepting the value they last read, for an hour at most; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, to copy it and to check narinfos | ❌ write a new token | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ❌ as the URL. A cache recreated with a new key leaves the stored one stale: every hive then reports `bad-signature` | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->
## Operating notes
The name `main` belongs to the swarm's own homeserver account, which the
controller mints, so the controller rejects any account you link under it.
`swarm-controller` checks every forge token itself: it looks at every agent
that has a store identity at start and every five minutes. For any whose
stored token is missing or stale it creates the forge user if there isn't one,
then mints the token. To check one agent now:
```sh
swarmctl agent mint-forge-token <agent>
```
An agent without a store identity gets no swarm token and keeps using the
`forge-token` file in its state dir, if it has one; re-create it with
`swarmctl agent create` to give it one. The swarm leaves an agent it already
declares as it is. For an agent with no declaration, such as one only its hive
knows about, `agent create` declares it `paused`; set it `up` from its card in
the swarm UI.
⚠️ **Re-create existing agents before deploying a hive-side change that makes a
container require a credential it may not have.** A container whose store
identity is absent doesn't start, so create the identities first, setting any
newly paused agent `up` as above.
**Revoking an agent's queue credential.** Declaring an agent `destroyed` —
`PUT /api/hives/{hive}/agents/{agent}/state` — deletes
`swarm/agents/<agent>/queue` and every version it held. An open queue
connection closes within an hour. An agent recreated under the same name gets a
fresh secret. It leaves the agent's mTLS leaf alone. If the store refuses the
delete, `swarm-controller` logs an `error` naming the agent and the teardown
continues; declare the agent destroyed again to re-run the revocation.
A failed store login retries every 30 seconds for a day. When a later attempt
lands a changed value, the service that loaded it restarts. If the fetch unit
can't compare file times it logs `cannot compare the mtime of <path> … —
leaving <unit> as it is` at `err`, and the service keeps the old value until
the next timer tick.
## Where each secret lives on disk
One row per secret, by the place that owns it: who generates it, where the file
or store path is, and what an operator of a hive elsewhere places. How the
pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.md).
### Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store is down, refuses the login, or fails the read of a leaf (a refusal, a timeout, an error), the unit runs on the files, logs the store's error, and writes nothing to the store; the unit seeds or mints only a leaf the store reports as absent. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting |
| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services/<grafana client id>/secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk, unused once the store holds the key. A store that's down or doesn't grant the leaf fails the unit, and Grafana doesn't start | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it |
| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 until the store answers: the controller re-reads it in the background, at most five minutes apart, and registers the forge hooks once it has the secret |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once. An attempt that fails or runs past 30 seconds retries after 5 seconds, then at growing intervals up to 5 minutes | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the fetch fails fast while the store is down and retries with a delay that grows to 5 minutes; the attempt that succeeds starts the collector, which isn't part of the container's boot, and it resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these leaves has
a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the cache's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, `swarm-bao-grafana-secret-key`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` and `swarm/services/<grafana client id>/secret-key` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services/<runner forwarder client id>/oidc/client` |
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The option names are `deploy.bao.<pair>`; the common names, what a missing leaf
costs and how to place a leaf by hand are in
[per-principal identities](credentials.md#per-principal-identities).

639
docs/2-handbook/swarm/credentials.md Normal file
View file

@ -0,0 +1,639 @@
# Credentials: the target shape
<!-- vale write-good.Passive = NO -->
The swarm's credential store is bao. This page is the overview of how the swarm
treats credentials; [`credentials-reference.md`](credentials-reference.md) lists
each one: who mints it, who reads it, and how it renews, with the notes for
operating them. It describes the target, not what's on disk today; the second half of the
reference maps [the files that exist right now](credentials-reference.md#where-each-secret-lives-on-disk).
<!-- vale write-good.Passive = YES -->
**Store secrets stay off disk.** A process pulls a value from the store at
runtime and holds it in memory only.
**The mTLS client certificate is the only credential on disk.** Each
identity (an agent, a hive, a swarm-level service) has one. It's what logs a
principal in to the store, so you place a hive's or service's certificate by
hand. The nix option for it takes a path to the certificate, never its bytes,
because a literal in a nix expression lands in the world-readable nix store.
A hive hands each agent its certificate; the agent pulls everything else
itself.
**Each credential has a minter, a reader and a renewal story.** Who mints it, who
reads it, and whether it renews without an operator are what the
[reference table](credentials-reference.md) gives for every credential, with the
operating notes that go with them.
## Where a secret lives
A swarm generates its credentials in three different places and reads them
in a fourth, so "where does this file go" has a different answer per deployment.
This page is that answer, one row per secret.
Two rules run through all of it.
**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.
The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
the queue callout's user key, reaches every client that connects,
so the store is a perfectly good place for it.
**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.
### Topologies, by who places secrets
Read every row of the [reference tables](credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level secrets
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
test worth applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
each with the secret publisher's identity and writes it to the file authelia loads. A store
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
database, and a new issuer key invalidates every token issued. A container without the
publisher's identity generates the files as before.
The same unit mints the client secrets, straight into the store. It reads
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from
`swarm/services/swarm-authelia-bridge/oidc/client` into
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
answer keeps the bridge down while it retries. A container without the publisher's
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.
The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
### The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Hive-level secrets
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
mint in authelia's container has stored the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the mint in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.
All agents share one policy, `hive-agent`, which the agent's cert-auth role
attaches and no other. It states the rule with `{{identity.entity.name}}`
where an agent's name would go, so the rule is one document and a change to it
is one write. `swarm-controller` writes it at agent creation and at its own
start.
For that the controller names each agent's identity entity after the agent: at
each certificate mint or renewal, before it writes the agent's cert-auth role,
and for every agent at its own start. It reads the cert mount's accessor and
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
entity a first login created under a generated name. At a mint or renewal this is
a precondition: when the store refuses a call, or the alias sits on any other
entity, the mint fails and the role stays as it was, so a live agent keeps
working until an operator resolves it. The start pass only logs the same
failures, because it writes no role. The controller needs the `identity/` and
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.
A cert-auth role that predates the shared policy may still name a policy
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
objects stay in the store and keep granting what they did, to a token that
already carries them and to a role not yet rewritten. A live agent's role is
rewritten at its next certificate renewal.
A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and rewrites the roles to attach it. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. The leaves, their option names
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](../../4-implementation/swarm/bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token.
The other roles issue service tokens that live an hour (`token_ttl=1h` and
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
agent and hive roles the controller writes through
`SecretStore::write_cert_role`, which takes the same value from
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
expires, and the mount default of 32 days leaves one lease per login for a
month; at the 17 thousand logins a day this swarm made, that came to half a
million leases. A reader logs in per operation and holds the token for seconds.
The longest hold is the controller's forge work, which stops after two minutes.
A client that keeps a `SecretStore` handle longer than an hour must log in
again, because nothing renews the token. A role the controller writes takes the
TTL at its next write: a hive's at the controller's start, an agent's at its
next certificate mint or renewal. At its start the controller also reads every
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
differs from the TTL, writes those two fields and no others. The store keeps the
fields a write leaves out, so the role's certificate, allowed names and
policies stay. A role already at the TTL isn't written. The start logs one line
with `rewritten`, `already_current` and `failed` counts, and a role that fails
gets a warning that names it, then the pass moves on. Tokens issued before
expire on their own; the controller revokes nothing. At that rate an hour keeps
about 700 tokens live.
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
it, and a deployed store never rewrites it, so the longer TTL would not reach
it; the granter logs in and finishes within seconds.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
### OIDC login
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
`url`, `cache` and `public-key`). That shows the key tree and each key's
versions and timestamps, and no value: the store refuses every `secret/data/`
read. The writes are the leaves the operator fills in by hand: the review
classifier's keys, and the cache's tokens and facts, which configure an external
cache or replace the in-swarm cache's own. The cache's host units create those
leaves only while the store holds none, so a saved value survives their next run
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
Anything more needs a token.
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
`services` > `attic`, pick the leaf and choose **Create new version**. The
form starts empty because the login can't read the current value: type the
key `value` and the new value, and save. A new version replaces the whole
secret, so `value` is the only key to enter. `bao kv put` with the same login
works too.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../../1-getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.
## Adding a secret
State three things, in the row you add to the [reference](credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.

Some files were not shown because too many files have changed in this diff Show more