hyperhive/docs/swarm/README.md

# The swarm

The **swarm** is where things live: agent identities and accounts, secrets,
the job graph that creates and places agents, telemetry, and the UI you
drive it all from. **Hives are the substrate** — NixOS hosts that run agent
containers on the swarm's behalf. Every hive belongs to a swarm; a single
host is a swarm of one.

This page is for the operator. It covers the control plane, how a hive
joins the directory, and how hives and agents report upward. The steps for a
fresh swarm are in [`setup.md`](../getting-started/setup.md); the all-local
config is the [README quick start](../../README.md#quick-start-an-all-local-swarm).

Option reference: `services.hyperhive.swarm.*` (swarm-wide facts, identical
on every host) and `services.hyperhive.deploy.*` (whether _this_ host runs
grafana, the controller, authelia, …) →
[options reference](https://hyperhive.darkest.space/options/), or
`nix build .#docs-swarm` / `.#docs-deploy`.

## Where each piece lives

- **control plane** — `swarm-controller`: hive directory, agent roster, job
  graph, agent creation. → [below](#swarm-controller)
- **scheduled prompts** — named prompts for one or more agents at set times,
  kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
  over its own. → [`inference.md`](inference.md)
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
  over its hive's. →
  [`coordinator.md`](../scheduler/coordinator.md#container-resource-limits)
- **review classifier** — planned, not implemented yet: checks a PR against
  its repo's own yes/no rules and posts the result as a review. →
  [`review-classifier.md`](review-classifier.md)
- **notification router** — forge events turned into todos on the agents
  they concern: assigned, unassigned, cleared on close. →
  [`notify-router.md`](notify-router.md)
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
  `admins` only. → [`ui.md`](ui.md)
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
  stack, each on whichever host you put it. → [`services.md`](services.md)
- **SSO** — the secrets authelia generates and how each reaches its reader.
  → [`sso.md`](sso.md)
- **secrets** — every credential the swarm holds, who mints it and where it
  lives → [`secrets.md`](secrets.md) · the per-secret minter/reader/renewal
  contract → [`credentials.md`](credentials.md) · how the store comes up,
  who writes its grants and how it unseals → [`bao.md`](bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
  hand a peer. → [`ca.md`](ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint,
  `swarm.snapshotStore.{address,port}`. →
  [`snapshot-store.md`](../networking/snapshot-store.md)

## Swarm controller

`swarm-controller` is the swarm's control plane: it holds the hive directory,
the agent roster and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent
and the error, which the Grafana alert rule `hh-agent-identity-mint-failed`
fires on. The agent is still created, without a store identity.

`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
runs the controller.** A swarm has one controller, so enabling it states a
fact about swarm topology, not about whether this host runs a hive.

What it serves, why it's a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../swarm-controller/README.md).

### Per-hive status (`GET /api/hives/status`)

One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.

A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.

| freshness        | what to do about it                                                                                                                          |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh`          | nothing — reported within `staleAfterSeconds`                                                                                                |
| `stale`          | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say               |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage                                              |
| `unknown`        | something is publishing under a name that's not in `swarm.hives` — a typo in the roster, or a hive removed from the roster but still running |

Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload. A roster hive's row
carries `dashboard`, the host its dashboard answers on; the swarm UI's
hive list links there.

Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.

### Making a hive report

Two options on the **hive**, plus its store identity. They sit in two
namespaces, because the queue's URL is a fact about _this machine_ and the
token endpoint is the swarm's single address:

| option                                         | what to set it to                             |
| ---------------------------------------------- | --------------------------------------------- |
| `deploy.hive-controller.statusPublish.natsUrl` | `tls://<swarm.nats.domain>:<swarm.nats.port>` |
| `swarm.statusPublish.tokenEndpoint`            | the swarm IdP's `/api/oidc/token`             |

Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, in
authelia's container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.

Until the secret is in the store, hive-c0re shows a `swarm_queue_config`
warning and tries again, waiting 5 seconds after the first failure and doubling
up to 5 minutes. It starts publishing as soon as a read succeeds, with no
restart.

The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.

The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue
host without the store gets no default, even when it holds the hive's own
reader `deploy.bao.clientCertFile`: issue a leaf whose subject is
`deploy.bao.natsCommonName`, place it, and name it in both options. Until you
do, `swarm-bao-nats-tls` fails naming both whenever the queue's certificate is
missing or due for renewal.

The queue host's gateway also serves the queue's metrics, at
`https://<swarm.nats.domain>/metrics`, so a swarm collector on any host
scrapes them. Only the collector's authelia client gets through; any other
session gets `deny`, and every other path on that name is 403. With the
gateway's default self-signed TLS, that vhost serves the queue's own
certificate, so a host whose only swarm service is the queue needs no
swarm-services certificate.

The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.

If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.

The endpoint answers **503** when this host has no swarm queue
configured, or has one and can't read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
can't see. The body says which. Status survives a controller restart:
it's stored in the queue, not in the daemon.

### Giving the agents the queue too

An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:

| option                                      | what to set it to                                             |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |

On every hive, `agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. On the queue's
host the name resolves inside a container to the bridge address, where the
firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.

**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:

- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
  store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
  certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
  wait each time, and logs a warning for each attempt that fails. After the
  seventh, about five minutes in, it logs an error and exits. The error reads
  `no swarm queue connection; exiting, since this agent cannot run without one`,
  and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.

Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading
the credential from the store again for each attempt.

The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for a `swarm-logs` built before it read the agent's
own queue secret: the harness never connects with them, and a current
`swarm-logs` presents that secret to the log store instead.

<details><summary>What an agent publishes over the queue</summary>

What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. The queue keeps the rows in
the stream `term-main-<agent>` for 24 hours or 64 MiB, the same limits and the
same creator as the subagent stream below, so a swarm terminal that attaches
later replays the newest of them. The two streams hold disjoint subjects.

An agent's **subagents** publish their terminals too, from the agent's
subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`,
classified the same way. The queue grants that family to the agent's own queue
credential alone, so the daemon reads it from the store under the agent's store
identity, exactly as the harness does, and publishes nothing without it. The
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
whichever comes first, dropping the oldest rows once either limit kicks in so
a publish never fails because of it, and the swarm lists an agent's subagents
from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for
every agent a hive's wanted state names, within a minute of the agent
appearing there. The agent's grant is publish on its own
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
subjects and limits are the controller's and never the agent's. Subagents
publish output only and read nothing.

The queue would refuse a row too large for its `max_payload` outright and
take the connection down with it, so the harness drops such a row's body before
sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.

The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model and effort (`model`, `effort`, where each comes from as
`model_source` and `effort_source`, and the resolved id the last turn
actually ran on), the
context budget and the last turn's context and cost token blocks,
`agent_state`, and `needs_login`: `true` while the agent's claude has no
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.

`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
spellings. ⚠️ From inside the container only two of those four are sayable: the
harness reports `up`, or `paused` when the pause marker is present. `offline` and
`destroyed` are hive-c0re's observations — a stopped agent publishes nothing and
a destroyed one doesn't exist — so a view that needs the full four-state picture
takes them from the `agent-status` bucket and uses this subject to sharpen the
rest.

Headers go out **on transition**: the harness rebuilds the header whenever its
event bus moves and publishes when the result differs from what it last sent.
That's the whole point of the subject — the `agent-status` bucket already
republishes once a minute, which is far too slow for "is this agent thinking
right now." A core subject keeps nothing for a late subscriber, so the harness
also re-sends the current header once a minute: a subscriber attaching
mid-idle sees it within that minute. A renderer opens with the bucket's
snapshot and lets this stream refine it.

Swarm-side, `GET /api/agents/<name>/state/stream` relays that subject as SSE,
resolving the agent's hive at request time exactly as the terminal stream does.
The payload passes through opaquely. Separately, the controller keeps each
agent's latest `model`, `effort` and their sources for
`GET /api/agents/<name>/inference` ([`inference.md`](inference.md)), and each
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it.

The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key of that bucket. The harness writes
once per start, because a config change reaches an agent by restarting its
container. An agent with no icon deletes its key. Swarm-side,
`GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.

The fourth is its **`AGENTS.md`**, which goes into the `agents-md` bucket under
the key `<agent>`, writable by the agent's own credential alone
(`$KV.agents-md.<agent>`). The swarm reads that copy and edits the file through
the agent; [An agent's AGENTS.md](#an-agents-agentsmd) has the details.

The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>`, its operator-input subject `$SWARM.input.<agent>` and
its command subject `$SWARM.control.<agent>`, and nothing else. Besides the
subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies. The queue refuses the hive's shared
agent client outright.

Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
sets as their inbox prefix. A KV watch or key listing delivers there as well. A
hive may also subscribe to `$SWARM.knowledge` and its own `$SWARM.deploy.<hive>`,
so it can't read another hive's deploys or any agent's terminal. The controller
may also subscribe to `$SWARM.term.<agent>`, `$SWARM.term.<agent>.sub.<subagent>`
and `$SWARM.agent-state.<agent>`, the streams it relays to swarm-ui. A hive or
controller that connects with the library's default `_INBOX.<nuid>` prefix gets
no replies, so every request it makes times out.

</details>

### An agent's AGENTS.md

The file in the agent's state dir, `state/AGENTS.md`, is the only copy that
counts, and only the agent writes it: itself, or its harness applying an edit
from the swarm. No hive is in either path.

**Reading.** The harness publishes the file's text and its revision, the git
blob id that `git hash-object` prints for it, at start, within ten seconds of
any change, and straight after an edit it applied. An agent with no `AGENTS.md`
publishes `""`. `GET /api/agents/<name>/agents-md` answers `{content, revision}`
from that copy, so it works while the agent isn't running. A 404 means the agent
hasn't published one: its harness hasn't run since it began to.

**Initial text.** `POST /api/agents` and `swarmctl agent create --agents-md <file>`
take an optional `agents_md`, the text the file starts with, up to 64 KiB; a
longer text is a 400. The controller stores it under the key `<agent>` of the
`agents-md-initial` bucket, which only it writes, before it queues the creation.
At start the harness reads its own key, with one direct get on
`$JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>`, and
writes the text to `AGENTS.md` only when no file exists there. It does this
before the first mirror publish and the first turn. An existing file is never
replaced, so a restart or a re-create keeps the agent's edits; a later change
to the file never reaches the key. Destroying the agent deletes the key. When
the harness can't read the key it starts without the text and logs a warning.
The text holds no secret, so nothing goes through the secret store.

**Editing.** `PUT /api/agents/<name>/agents-md` takes `{base_revision, diff}`.
The diff is a unified diff that names `a/AGENTS.md` and `b/AGENTS.md`, made
against the text at `base_revision`. swarm-controller sends it as a request on
the agent's command subject `$SWARM.control.<name>`, as it sends a cancel. The
harness refuses it when the file isn't at `base_revision`. Otherwise it runs
`git apply --check` and then `git apply` on a copy of the file, and writes the
result over the file only when both pass:

| Status | Meaning                                                                                                                                 |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| 200    | applied; the body is `{revision}`, the file's new revision                                                                              |
| 409    | the file isn't at `base_revision` any more; nothing applied, and `revision` in the body is the one it's at                              |
| 422    | `git apply` refused the diff, or it touches a file other than `AGENTS.md`; `detail` carries git's message, and the file stays as it was |
| 503    | the agent isn't running or isn't connected to the queue                                                                                 |
| 504    | the harness didn't answer within ten seconds, so whether the edit applied is unknown                                                    |
| 500    | the harness couldn't read or write the file or run git, or the queue failed the request                                                 |

The harness refuses a diff that sets any file mode other than `100644`, so an
edit can't turn `AGENTS.md` into a symlink. `PUT` is operator-only, behind
`Remote-User` like cancel: whoever can call it can rewrite the agent's
instructions, so don't widen who holds it without revisiting this.

An edit needs the agent running, and a read of a stopped agent shows the text
it last published. A harness older than this route doesn't know the command and
drops it unanswered, which reads as a 504. `swarmctl agent agents-md get` and
`apply` call the two routes.

### Swarm-wide forge objects

The controller also keeps the forge objects that are one per swarm, not
one per hive. It ensures them at start and every five minutes after
(`swarm-controller/src/forge/objects.rs`):

- the orgs `agent-configs`, `internal` and `agents`, plus each mirror's
  owner org;
- the empty `operators` merge-gate team in `agents` and `agent-configs`;
- the `main` merge gate on every `agent-configs` repo: merge and approval
  whitelists = the `operators` team, and no user. The controller leaves a
  repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
  host (with the `actions/checkout` and `actions/upload-artifact` ones
  `deploy.forgejo.actionsFromSelf` adds);
- `internal/docs` (private) and `internal/knowledge` (public, with a
  seed `README.md` while empty);
- the `agent-configs` org avatar
  (`deploy.swarm-controller.configOrgAvatarPng`).

A pass that can't finish logs a
`warn` line per object plus `swarm forge objects: pass incomplete` in
`journalctl -u swarm-controller`, and retries on the next tick. While the
controller is down the objects stay as they are.

The controller gives each agent **read** on `internal/docs`. A `GrantDocsRead`
node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the
store's `hive-agent-*` roster (`forge/docs_grant.rs`, at start and every five
minutes) adds any agent that isn't one, so an agent whose create-time grant failed
gets it within five minutes. The pass only adds: a roster it can't read, a
forge read that fails, or an `internal/docs` that doesn't exist (checked once per
pass, logged at warn) changes nothing. The grant doesn't depend on the hive
the agent runs on, or on a forge container there.

### Swarm-wide forge webhooks

At startup the controller registers its Forgejo hooks pointing at
itself, all under `https://<swarm.domain>/webhook/forge/`:

- a `push` hook on `internal/knowledge`
- a `pull_request` hook on the `agent-configs` org
- an instance-wide `push` system hook, for commit metrics
- an instance-wide `pull_request_review_request` system hook, which starts
  the [review classifier](review-classifier.md#trigger-and-lifecycle)
- instance-wide `issues` + `issue_assign` and `pull_request` +
  `pull_request_assign` system hooks, whose deliveries it
  keeps in the `SWARM_EVENTS` stream for the
  [notification router](notify-router.md#the-path)

The controller **interprets** a delivery and sends hives a specific
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.
The `issues` and `pull_request` deliveries are the exception: no hive gets
them, and the notification router, not the controller, decides what they mean.

A `config-pr` delivery reporting a PR merged into `main` queues a deploy
of its `merge_commit_sha` on the one hive whose wanted state places the
agent; with no such hive, or several, the controller deploys nothing. See
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes).
A merge whose `closed` delivery never arrives deploys nothing either; the
operator recovers by redelivering that delivery from the forge hook page,
which re-enters the same handler and re-queues the deploy.

**`internal/knowledge` is on that path.** The controller's is the only
hook on it ([`knowledge.md`](../integrations/knowledge.md) covers clearing a
leftover). A webhook has exactly one target URL, so a second registration
would take delivery away from the first rather than add a recipient.

<!-- vale write-good.Passive = NO -->

**The `agent-configs` org is on it too.** The controller's hook is the
only one that acts on config PRs. Each hive deletes its own
`/webhook/config-pr` org hook on startup, since a route no hive serves
would otherwise sit on the org collecting failed deliveries. Removing
the controller's hook stops forge-UI merges from deploying until its
next start recreates it.

<!-- vale write-good.Passive = YES -->

Nothing to configure. The controller registers the hooks only when this
host also serves the swarm UI vhost — that's what publishes the
endpoint, and a hook the forge can't reach would collect failed
deliveries while looking healthy. The controller generates the HMAC
secret on first start and keeps it
(see [`docs/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)).

To check it's working, push to `internal/knowledge` and look for
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
refused delivery logs `webhook: refused delivery` with the reason.

## Hives: the substrate

- **hive** — one host running `hive-c0re` and its agent containers
  (`deploy.hive-controller.enable`). Addressed as `<hiveName>.<swarm.domain>`.
- **swarm** — every hive in `services.hyperhive.swarm.hives`, plus the
  shared services and controller. You can qualify an agent as
  `agent@hive-domain`.
- **peer hive** — any hive in the directory other than this one. Peers are
  _derived_, not declared: the directory lists every hive including
  yourself, and `hiveName` says which one you are.

### Hive identity config

```nix
services.hyperhive = {
  swarm.domain = "example.com";     # required — the swarm's DNS domain
  hiveName = "pr1ma";               # required — this hive's label in it
  swarm.name = "constellat1on";     # shared swarm display name (optional)

  # required — the directory, identical on every host in the swarm.
  # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
  swarm.hives = {
    pr1ma = { };                                     # this host, per hiveName
    lab   = { };                                     # a second hive
    edge  = { domain = "edge.elsewhere.example"; };  # addressed off-convention
  };
};
```

<!-- vale write-good.Passive = NO -->

Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive, the secret store or the homeserver. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.

<!-- vale write-good.Passive = YES -->

**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.

```
# hive A                          # hive B
hiveName = "pr1ma";               hiveName = "edge";
swarm.hives = { … };              swarm.hives = { … };   # byte-identical
```

One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.

`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`iris@pr1ma.example.com`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.

`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.

The env-var chain and `qualify()` semantics:
[`conventions.md`](../process/conventions.md) § Hive identity.

Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.

### What the directory feeds

1. **Swarm-wide hive roster** — swarm-controller reads this same
   directory and serves it at `GET /api/hives`; the swarm UI's front page
   renders it ([`ui.md`](ui.md)).

2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
   federates with the peer's matrix server (discovered via the peer's
   `.well-known/matrix/server` delegation, which the gateway serves).
   Federation validates the peer's TLS certificate against the matrix
   **container's** trust bundle, independent of this directory.

   ⚠️ On a hive with self-signed gateway certificates, the container
   also trusts this hive's `trust-bundle.pem`, bound in at runtime and
   added to the public CAs; it holds the store's `pki` root, so a peer whose
   certificate chains to the same root validates. A peer outside this
   swarm's root needs a CA-issued certificate (ACME). Federation firewall
   and TLS requirements: [`integrations/matrix.md`](../integrations/matrix.md).

3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
   entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
   to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).

### WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.

#### Generating keys

On each hive host:

```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators
```

#### Config example (two hives)

```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  deploy.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    pr1ma = {
      domain             = "pr1ma.example.com";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint  = "198.51.100.1:51820";
      wireguardAddress   = "10.100.0.1/32";
    };
    edge = {
      domain             = "edge.corp";
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint  = "203.0.113.42:51820";
      wireguardAddress   = "10.100.0.2/32";
    };
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  deploy.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.hives = { /* … identical to hive A's … */ };
};
```

#### What the mesh does

- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
  host (not inside agent containers; containers reach peers via the
  host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
  from `services.hyperhive.swarm.hives`, minus this host's own hive, to
  build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
  the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
  disable (not needed when both sides have public IPs and no NAT).

#### NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.

## Cross-references

- [`ui.md`](ui.md) — the swarm UI, the operator surface for "what hives exist"
- [`../networking/snapshot-store.md`](../networking/snapshot-store.md) — the
  swarm's `btrfs receive` endpoint and the `swarm.snapshotStore` option
- [`../process/conventions.md`](../process/conventions.md) § Hive identity —
  env vars, qualified labels
- [`../integrations/matrix.md`](../integrations/matrix.md) — matrix
  federation, TLS cert autogeneration, firewall posture
- [`../networking/gateway.md`](../networking/gateway.md) — nginx vhosts and
  the `.well-known/matrix/` autodiscovery scheme