hyperhive/docs/swarm/README.md

# Multi-hive swarms

A **swarm** is a collection of agents that share an identity and
coordinate across one or more hives. A single hyperhive instance
running on one host is already a swarm (one hive). This doc covers
the additional config needed when the swarm spans multiple hosts.

## Terminology

- **hive** — a single hyperhive installation on one host. Has its
  own `services.hyperhive.domain` DNS name and its own set of agent
  containers.
- **swarm** — one or more hives whose operators have declared them
  as peers. Agents can be qualified as `agent@hive-domain`.
- **peer hive** — any hive in `services.hyperhive.swarm.hives` other
  than this one. Peers are *derived*, not declared: the directory lists
  every hive including yourself, and `hiveName` says which one you are.

## Hive identity config

```nix
services.hyperhive = {
  swarm.domain = "example.com";     # required — the swarm's DNS domain
  hiveName = "pr1ma";               # required — this hive's label in it
  swarm.name = "constellat1on";     # shared swarm display name (optional)

  # required — the directory, identical on every host in the swarm.
  # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
  swarm.hives = {
    pr1ma = { };
    edge = { };
  };
};
```

`swarm.domain` and `hiveName` are **required** whenever hyperhive is
enabled; eval fails with a hint naming each. Neither is defaulted,
because a guessed value here is a wrong hostname that evaluates cleanly
and deploys — an eval failure asking the operator to write the address
down is the cheaper outcome. **Upgrading past this release means setting
both once.**

`domain` is required too, but you no longer *write* it: it is read from
this hive's own entry in the directory, whose `domain` defaults to
`<name>.<swarm.domain>`. So a conventional swarm states no addresses at
all, and a hive addressed by something else states it in the one place
the other hives read — `swarm.hives.edge.domain = "edge.elsewhere.example";`.

Setting `services.hyperhive.domain` directly still works and still wins,
with a **deprecation warning**. The reason it's deprecated is not tidiness:
that option is local to one host, the directory is copied to every host,
so a value written only there leaves every peer pointing somewhere else
with nothing detecting the disagreement.

⚠️ **Upgrading:** a hive that has been running on `swarm.domain` +
`hiveName` alone now needs its own directory entry —
`services.hyperhive.swarm.hives.<hiveName> = { };`, one line, no value.
Eval fails naming it if you forget.

`domain` drives `HYPERHIVE_HIVE_DOMAIN` in every container so agents can
form qualified labels (`iris@pr1ma.example.com`).

`swarm.name` is purely display — it surfaces in the dashboard chrome
header and per-agent system prompts, and federated hives at different
domains can share one. `hiveName` surfaces in the same places but is
*not* only display: it is the leftmost label of the hive's domain. That
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
distinction — one names this hive, the other names the group it belongs
to.

See `docs/conventions.md` § Hive identity for the env-var chain
and `qualify()` / `qualified_label()` semantics.

## Swarm CA

A hive's internal TLS chains to a **swarm root CA**, so a peer that
trusts the root validates every hive in the swarm rather than being
pinned to each one by hand. Provisioning modes, what to hand a peer
(`trust-bundle.pem`, never `ca.pem`), the name constraints on a hive
CA, and how an existing hive adopts the hierarchy: [`ca.md`](ca.md).

## Running the swarm's shared services

One authelia, one matrix, one forge per swarm — which host runs them,
and what a hive that runs none of them configures instead:
[`services.md`](services.md).

## Single sign-on

Which secrets the SSO provider generates, which one has a reader in
another container, and the three ways that one gets delivered:
[`sso.md`](sso.md).

## Secrets

Every credential the swarm holds, who mints it, where it must live, and
which of the three topologies makes it the operator's job to place:
[`secrets.md`](secrets.md).

## Swarm UI

The operator-only web surface on the swarm apex, why reaching it needs
the `admins` group rather than just a session, and the four sites a
swarm service name has to be wired into: [`ui.md`](ui.md).

## The swarm's hive directory

```nix
services.hyperhive.swarm.hives = {
  pr1ma = { domain = "pr1ma.example.com"; };                          # this host, per hiveName
  lab   = { domain = "lab.example.com"; };                            # CA-trusted (Let's Encrypt etc.)
  edge  = { domain = "edge.corp"; certFingerprint = "sha256:…"; };    # self-signed leaf, pinned
};
```

One attrset describing **every** hive in the swarm, **including this
one**, keyed by that hive's `hiveName`. It is meant to be *identical on
every host* — write it once, share it, and each host reads it correctly
because `services.hyperhive.hiveName` says which entry is itself.

Empty (the default) means this host isn't in a swarm. Once non-empty it
**must** contain an entry for `hiveName`; eval fails naming the missing
hive. That assertion is load-bearing rather than pedantic — "my peers"
is derived as *everything that isn't me*, so a directory that doesn't
contain you derives every hive as a peer and you peer with yourself.

`domain` is required per entry and deliberately undefaulted: it is
conventionally `<name>.<swarm.domain>`, but a wrong domain that
evaluates cleanly points at a real machine that isn't the one you meant.

**`certFingerprint`** (`"sha256:…"`, optional) pins that hive's TLS
_leaf_. Scopes **only** to hive-c0re's own peer HTTPS checks (the P33RS
dashboard links + agent peer discovery below); matrix federation never
consults it. Omit it for any hive under the swarm root CA or a public
CA — which is the normal case.

> **There is no per-hive CA field.** Trust inside a swarm comes from the
> swarm root ([`ca.md`](ca.md)): every hive chains to it, so one anchor
> replaces the O(n²) pinning. What that genuinely drops is trusting a
> hive whose root this swarm does *not* own — another swarm's, or one
> keeping its own CA. That is a cross-swarm problem and wants a
> mechanism designed for it, not a field that happened to work.

### Fingerprint format

The value is the string `sha256:` followed by exactly 64 hexadecimal
digits — the SHA-256 digest of the peer's DER-encoded TLS leaf
certificate. The hex is case-insensitive (upper or lower both parse),
carries no colon separators between bytes, and any value not matching
this shape is ignored with a warning rather than weakening trust.

```
sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12
```

Generate it from the peer's certificate with openssl. The
`-fingerprint -sha256` output is uppercase and colon-separated, so
strip the colons, lowercase, and prepend the `sha256:` prefix:

```sh
# from a PEM/CRT file
openssl x509 -in peer.crt -noout -fingerprint -sha256 \
  | sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'

# straight from the live endpoint (port 443)
echo | openssl s_client -connect peer.example.com:443 -servername peer.example.com 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 \
  | sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
```

Pin the leaf certificate, not an intermediate or the CA — the
digest must match the exact cert the peer serves on its HTTPS
endpoint. When the peer rotates its cert, update the pin to the new
fingerprint (or switch the peer to a CA-trusted cert and drop the
field).

The nix module serialises the attrset to a `HYPERHIVE_PEERS` JSON
array (`[{ domain, cert_fingerprint }]`) injected into the c0re
environment and forwarded to agent containers.

## What the config does at runtime

1. **Dashboard P33RS tab** — hive-c0re reads `HYPERHIVE_PEERS` and
   surfaces it as the peer list in the dashboard's state API. The
   dashboard shows a P33RS tab (hidden when the list is empty) with a
   card per peer linking to `https://{domain}/`. Wire format + module
   pointer: `docs/web-ui/dashboard.md` § P33RS tab.

2. **Agent identity** — the same `HYPERHIVE_PEERS` env var is
   forwarded to agent containers, so agent code can discover peer
   hives and address them with qualified names (`agent@domain`). See
   `hive-agent/src/identity.rs`'s module doc for the label/domain
   helpers.

3. **Matrix federation** — when `matrix.enable` is on, tuwunel
   federates with the peer's matrix server (discovered via the peer's
   `.well-known/matrix/server` delegation, which the gateway serves).
   Federation validates the peer's TLS certificate against the matrix
   **container's** trust bundle — independently of `certFingerprint`,
   which it never consults.

   ⚠️ **That container currently trusts no swarm-internal CA**, so a
   self-signed gateway certificate does not federate. The swarm root
   can't simply be listed there: `security.pki.certificateFiles` is
   read when the system is *built*, and the root is a runtime file (its
   key must never enter the store), so there is no build-time name for
   it. Bridging that needs a runtime mechanism and is tracked as its own
   issue. Until then, federation needs CA-issued certs (ACME). See
   `docs/matrix.md` for federation firewall + TLS requirements.

## One directory, not a bilateral declaration

Both hives hold the **same** `hives` attrset; neither declares the
other. What differs between the two hosts is only `hiveName`:

```
# hive A                          # hive B
hiveName = "pr1ma";               hiveName = "edge";
swarm.hives = { … };              swarm.hives = { … };   # byte-identical
```

That is the point of the shape, and it removes a class of bug rather
than saving typing: a per-host peer list let two hosts hold *different*
facts about the same third hive — a stale endpoint, a rotated
fingerprint — with nothing to detect the disagreement. One entry per
hive makes it unrepresentable.

## WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hive-c0re can configure a host-to-host
WireGuard mesh.

### Generating keys

On each hive host:

```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators
```

### Config example (two hives)

```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    pr1ma = {
      domain             = "pr1ma.example.com";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint  = "198.51.100.1:51820";
      wireguardAddress   = "10.100.0.1/32";
    };
    edge = {
      domain             = "edge.corp";
      certFingerprint    = "sha256:…";           # TLS trust (unchanged)
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint  = "203.0.113.42:51820";
      wireguardAddress   = "10.100.0.2/32";
    };
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.hives = { /* … identical to hive A's … */ };
};
```

### What the mesh does

- `networking.wireguard.interfaces.wg-hive` is configured on the host
  (not inside agent containers; containers reach peers via the host's
  routing table).
- UDP port 51820 (or `listenPort`) is opened on the host firewall.
- `HYPERHIVE_PEERS` gains a `wireguard_address` field for each mesh
  peer so hive-c0re can reach intra-swarm services without a public
  DNS round-trip.
- `persistentKeepalive = 25` is set by default; override or null to
  disable (not needed when both sides have public IPs and no NAT).

### NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, a STUN relay or a third host (exit node)
is required. Out of scope for v0.

## Snapshot store

One further option lives in this namespace but is documented with the
service it points at: `services.hyperhive.swarm.snapshotStore.{address,
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.

It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
one store, because the receiver keys destinations by *agent* so a
migrating agent keeps one unbroken incremental chain. See
[snapshot-store.md](../snapshot-store.md).

## Swarm controller

`services.hyperhive.swarm.controller.enable` runs the `swarm-controller`
daemon on this host. **Off by default and deliberately not derived from
`services.hyperhive.enable`**: a swarm has one controller, so enabling it
is a statement about swarm topology, not about whether hyperhive is
installed. Every hive runs `hive-c0re` (the agents on that host); one
hive additionally runs this (what is true across hives).

What it serves, why it is a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../swarm-controller/README.md).

### Per-hive status (`GET /api/hives/status`)

What each hive last **offered** about itself. Hives publish upward; the
controller never reaches down to collect. That direction is deliberate:
during the #3097 gateway outage every recovery channel ran through the
one broken thing, so a status path that depended on the controller would
have gone dark exactly when it was needed to diagnose the controller's
own network. A hive computes its own status locally either way — this
endpoint is a view of what was published, never the source.

**The queue is the store.** A hive publishes into the `hive-status`
JetStream KV bucket (`history: 1` — the last thing each hive said), and
the controller reads that bucket per request, keeping no copy. A cache
here would be a second answer free to disagree with the first, and the
disagreement would surface as a hive reading healthy on a dashboard
while the bucket says otherwise. The bucket is created by whichever side
gets there first.

**Absence is what the endpoint is built around**:

| freshness | means |
|---|---|
| `fresh` | published within `staleAfterSeconds` |
| `stale` | published longer ago than that — the payload is still returned, because "old" and "absent" are different answers |
| `never_reported` | in the roster, has never published. Distinct from `stale`: went quiet is a fault, never spoke is usually a deployment that hasn't happened |
| `unknown` | published but not in `swarm.hives` — surfaced rather than dropped |

Rows come from the **roster**, not from the bucket, so a hive that has
never reported appears rather than not appearing, and an empty bucket
renders as a swarm nobody has heard from instead of a healthy one.
Freshness is derived at read time from a timestamp and never stored as a
flag — a stored `healthy` boolean goes stale silently the moment nothing
arrives, which is the failure this is designed against. Each row also
carries `last_seen_unix` and `age_seconds`, so a consumer that disagrees
with `staleAfterSeconds` can apply its own threshold.

`last_seen_unix` is the **bucket's** timestamp, applied by the NATS
server when the value landed, not a field inside the payload — a
publisher cannot make itself look fresher than it is, and a hive with a
wrong clock skews its own payload rather than its freshness.

Because the bucket outlives a controller restart, a restarted controller
reports what it reads: `stale, age_seconds: 10800` rather than
`never_reported`. That is the more honest of the two — it genuinely
knows when the hive last spoke. Losing the bucket degrades in the same
direction: every hive reads `never_reported` until its next publish,
which is the true answer and not a remembered "healthy".

The endpoint answers **503**, not an empty list, when no queue is
configured on this host or its store cannot be read. "I cannot reach the
store" and "every hive is silent" are different answers, and rendering
the second when the first is true would turn a local fault into an
apparent swarm-wide outage.

⚠️ **Nothing writes to the bucket yet.** The transport and the
controller's read path are in place; the hive-side publisher is a later
slice. Until one lands, every hive reads `never_reported` — the correct
answer for a controller that has been told nothing.

## Cross-references

- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
  the `swarm.snapshotStore` option that points a hive at it
- `docs/conventions.md` § Hive identity — env vars, qualified labels
- `docs/matrix.md` — matrix federation, TLS cert auto-generation,
  firewall posture
- `docs/web-ui/dashboard.md` § P33RS tab — dashboard surface
- `docs/gateway.md` — nginx vhosts and the `.well-known/matrix/`
  auto-discovery scheme