hyperhive/docs/2-handbook/swarm/README.md

# The swarm

The **swarm** is where things live: agent identities and accounts, secrets,
the job graph that creates and places agents, telemetry, and the UI you
drive it all from. **Hives are the substrate** — NixOS hosts that run agent
containers on the swarm's behalf. Every hive belongs to a swarm; a single
host is a swarm of one.

This page is for the operator. It covers the control plane, how a hive
joins the directory, and how hives and agents report upward. The steps for a
fresh swarm are in [`setup.md`](../../1-getting-started/setup.md); the all-local
config is the [README quick start](../../../README.md#quick-start-an-all-local-swarm).

Option reference: `services.hyperhive.swarm.*` (swarm-wide facts, identical
on every host) and `services.hyperhive.deploy.*` (whether _this_ host runs
grafana, the controller, authelia, …) →
[options reference](https://hyperhive.darkest.space/options/), or
`nix build .#docs-swarm` / `.#docs-deploy`.

## Where each piece lives

- **control plane** — `swarm-controller`: hive directory, agent roster, job
  graph, agent creation. → [below](#swarm-controller)
- **scheduled prompts** — named prompts for one or more agents at set times,
  kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
  over its own. → [`inference.md`](../../4-implementation/swarm/inference.md)
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
  over its hive's. →
  [`coordinator.md`](../../4-implementation/scheduler/coordinator.md#container-resource-limits)
- **review classifier** — checks a PR against its repo's own yes/no rules
  and posts the result as a review. →
  [`review-classifier.md`](../../4-implementation/swarm/review-classifier.md)
- **notification router** — forge events turned into todos on the agents
  they concern: assigned, unassigned, cleared on close; and the classifier's
  review requested on a push. →
  [`notify-router.md`](../../4-implementation/swarm/notify-router.md)
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
  `admins` only. → [`ui.md`](ui.md)
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
  stack, each on whichever host you put it. → [`services.md`](services.md)
- **forge** — the swarm's Forgejo: per-agent accounts, notifications and the
  CLI agents use. → [`forge.md`](forge.md)
- **matrix** — the swarm's homeserver, several accounts per agent, and
  federation. → [`matrix.md`](matrix.md)
- **backup archive** — where the hives back up each agent's state, how long it
  stays and how to check it. → [`archive.md`](archive.md)
- **knowledge repo** — the reference docs every agent reads, and how to add
  one. → [`knowledge.md`](knowledge.md)
- **`swarmctl`** — the swarm operator's CLI. → [`swarmctl-cli.md`](swarmctl-cli.md)
- **SSO** — the secrets authelia generates and how each reaches its reader.
  → [`sso.md`](sso.md)
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
  every credential the swarm holds, who mints it, who reads it, how it renews
  and where it lives → [`credentials-reference.md`](credentials-reference.md) ·
  how the store comes up, who writes its grants and how it unseals →
  [`bao.md`](../../4-implementation/swarm/bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
  hand a peer. → [`ca.md`](../../4-implementation/swarm/ca.md)
- **agent-state backups** — the old snapshot store no longer exists; swarm-archive will replace it.

## Swarm controller

`swarm-controller` is the swarm's control plane: it holds the hive directory,
the agent roster and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent,
the step it stopped at and the error, which the Grafana alert rule
`hh-agent-identity-mint-failed` fires on. The agent is still created, without a
store identity, and the controller's five-minute credential pass mints one for
it (also when only the certificate exists and its cert-auth role is missing) as
long as the agent isn't `Destroyed`. A mint that runs after a hive declares the
agent `Destroyed` logs `identity mint skipped` and writes nothing.

`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
runs the controller.** A swarm has one controller, so enabling it states a
fact about swarm topology, not about whether this host runs a hive.

What it serves, why it's a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../../swarm-controller/README.md).

### What the controller keeps

- **Hive status.** Each hive publishes its state upward through the swarm queue.
  The swarm UI's hive list shows when each last reported: `fresh`, `stale`,
  `never_reported` or `unknown`.
  `services.hyperhive.swarm.controller.staleAfterSeconds` (default `120`) sets
  how long a hive may stay silent before it reads `stale`.
- **Agent queue access.** Each agent connects to the queue as itself, with a
  credential the controller mints when it creates the agent.
- **An agent's `AGENTS.md`.** The swarm holds the revisions and the harness
  applies them.
- **Forge objects and webhooks.** Swarm-wide labels, repos and webhooks the
  controller registers on the forge.

The options, wire formats and failure modes of each:
[`controller.md`](../../4-implementation/swarm/controller.md).

## Hives: the substrate

- **hive** — one host running `hive-c0re` and its agent containers
  (`deploy.hive-controller.enable`). Addressed as `<hiveName>.<swarm.domain>`.
- **swarm** — every hive in `services.hyperhive.swarm.hives`, plus the
  shared services and controller. You can qualify an agent as
  `agent@hive-domain`.
- **peer hive** — any hive in the directory other than this one. Peers are
  _derived_, not declared: the directory lists every hive including
  yourself, and `hiveName` says which one you are.

### Hive identity config

```nix
services.hyperhive = {
  swarm = {
    domain = "myswarm.example"; # required — the swarm's DNS domain
    name = "myswarm"; # shared swarm display name (optional)

    # required — the directory, identical on every host in the swarm.
    # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
    hives = {
      firsthive = { }; # this host, per hiveName
      secondhive = { }; # a second hive
      thirdhive.domain = "thirdhive.elsewhere.example"; # addressed off-convention
    };
  };

  # required on a hive host — this hive's label in the directory
  deploy.hive-controller.hiveName = "firsthive";
};
```

<!-- vale write-good.Passive = NO -->

Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.

<!-- vale write-good.Passive = YES -->

**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.

```
# hive A                                   # hive B
deploy.hive-controller.hiveName = "firsthive";   deploy.hive-controller.hiveName = "secondhive";
swarm.hives = { … };                       swarm.hives = { … };   # byte-identical
```

One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.

`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`agent1@firsthive.myswarm.example`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.

`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.

The env-var chain and `qualify()` semantics:
[`conventions.md`](../../5-project/process/conventions.md) § Hive identity.

Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](../../4-implementation/swarm/ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.

### What the directory feeds

1. **Swarm-wide hive roster** — swarm-controller reads this same
   directory and serves it at `GET /api/hives`; the swarm UI's front page
   renders it ([`ui.md`](ui.md)).

2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
   federates with the peer's matrix server (discovered via the peer's
   `.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
   Federation validates the peer's TLS certificate against the matrix
   **container's** trust bundle, independent of this directory.

   ⚠️ On a hive with self-signed gateway certificates, the container
   also trusts this hive's `trust-bundle.pem`, bound in at runtime and
   added to the public CAs; it holds the store's `pki` root, so a peer whose
   certificate chains to the same root validates. A peer outside this
   swarm's root needs a CA-issued certificate (ACME). Federation firewall
   and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).

3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
   entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
   to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).

### Reaching the queue and the store on a routable address

The mesh isn't the only way to reach the queue and the secret store from
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
queue's port on every interface, and `deploy.bao.openFirewall` together with
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
on the addresses it listens on. Both default to off. Each exposes a service
beyond the tunnel, which is the operator's decision. The queue requires TLS and
its `auth_callout` refuses every client it can't identify. The store requires a
client certificate on every listener, and evaluation fails when you
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
`nats.<swarm.domain>` and `bao.<swarm.domain>` in your upstream DNS at the
address the other hives reach.

### Keeping a vhost off the public address

The gateway is one nginx per host, and a vhost answers on every address of the
host by default, IPv4 and IPv6. To serve some vhosts only inside a private
network (a VPN, say), list the host's addresses on that network in
`services.hyperhive.gateway.privateListenAddresses` and set `public = false`
on the vhosts that should stay inside:

```nix
services.hyperhive.gateway = {
  privateListenAddresses = [ "10.100.0.1" "fd00:100::1" ];
  vhosts."grafana.myswarm.example".public = false;
};
```

A `public = false` vhost listens on those addresses only. The public address
never serves it, and a request for its name there closes without a response.
Vhosts that keep the default `public = true` answer on the private addresses
too. The addresses can be IPv4 or IPv6 (`fd00::1` or `[fd00::1]`), and each
must exist on the host when nginx starts. The firewall stays your call: this
decides which vhosts nginx serves on which address, not which ports are open.

The gateway also listens on `[::]` by default, so the vhosts are reachable
over IPv6. `gateway.listenIPv6` follows `networking.enableIPv6`; turn it off
on a host whose kernel can't open an IPv6 socket. Details are in
[`gateway.md`](../../4-implementation/networking/gateway.md#listen-addresses).

### WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.

#### Generating keys

On each hive host:

```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators
```

#### Config example (two hives)

```nix
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
services.hyperhive = {
  deploy.wireguard = {
    enable = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address = "10.100.0.1/24";
    listenPort = 51820; # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    firsthive = {
      domain = "firsthive.myswarm.example";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint = "198.51.100.1:51820";
      wireguardAddress = "10.100.0.1/32";
    };
    secondhive = {
      domain = "secondhive.elsewhere.example";
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint = "203.0.113.42:51820";
      wireguardAddress = "10.100.0.2/32";
    };
  };
};

# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
services.hyperhive = {
  deploy.wireguard = {
    enable = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address = "10.100.0.2/24";
  };

  swarm.hives = {
    # … identical to hive A's …
  };
};
```

#### What the mesh does

- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
  host (not inside agent containers; containers reach peers via the
  host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
  from `services.hyperhive.swarm.hives`, minus this host's own hive, to
  build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
  the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
  disable (not needed when both sides have public IPs and no NAT).

#### NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.

## Cross-references

- [`ui.md`](ui.md) — the swarm UI, the operator surface for "what hives exist"
- [`5-project/process/conventions.md`](../../5-project/process/conventions.md) § Hive identity —
  env vars, qualified labels
- [`2-handbook/swarm/matrix.md`](matrix.md) — matrix
  federation, TLS cert autogeneration, firewall posture
- [`4-implementation/networking/gateway.md`](../../4-implementation/networking/gateway.md) — nginx vhosts and
  the `.well-known/matrix/` autodiscovery scheme