hyperhive/docs/swarm/README.md

# Multi-hive swarms

A **swarm** is a collection of agents that share an identity and
coordinate across one or more hives. A single hyperhive instance
running on one host is already a swarm (one hive). This doc covers
the additional config needed when the swarm spans multiple hosts.

## Terminology

- **hive** — a single hyperhive installation on one host. Has its
  own `services.hyperhive.domain` DNS name and its own set of agent
  containers.
- **swarm** — one or more hives whose operators have declared them
  as peers. Agents can be qualified as `agent@hive-domain`.
- **peer hive** — any hive in `services.hyperhive.swarm.hives` other
  than this one. Peers are *derived*, not declared: the directory lists
  every hive including yourself, and `hiveName` says which one you are.

## Hive identity config

```nix
services.hyperhive = {
  swarm.domain = "example.com";     # required — the swarm's DNS domain
  hiveName = "pr1ma";               # required — this hive's label in it
  swarm.name = "constellat1on";     # shared swarm display name (optional)

  # required — the directory, identical on every host in the swarm.
  # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
  swarm.hives = {
    pr1ma = { };
    edge = { };
  };
};
```

`swarm.domain` and `hiveName` are **required** whenever hyperhive is
enabled; eval fails with a hint naming each. Neither is defaulted,
because a guessed value here is a wrong hostname that evaluates cleanly
and deploys — an eval failure asking the operator to write the address
down is the cheaper outcome. **Upgrading past this release means setting
both once.**

`domain` is required too, but you no longer *write* it: it is read from
this hive's own entry in the directory, whose `domain` defaults to
`<name>.<swarm.domain>`. So a conventional swarm states no addresses at
all, and a hive addressed by something else states it in the one place
the other hives read — `swarm.hives.edge.domain = "edge.elsewhere.example";`.

Setting `services.hyperhive.domain` directly still works and still wins,
with a **deprecation warning**. The reason it's deprecated is not tidiness:
that option is local to one host, the directory is copied to every host,
so a value written only there leaves every peer pointing somewhere else
with nothing detecting the disagreement.

⚠️ **Upgrading:** a hive that has been running on `swarm.domain` +
`hiveName` alone now needs its own directory entry —
`services.hyperhive.swarm.hives.<hiveName> = { };`, one line, no value.
Eval fails naming it if you forget.

`domain` drives `HYPERHIVE_HIVE_DOMAIN` in every container so agents can
form qualified labels (`iris@pr1ma.example.com`).

`swarm.name` is purely display — it surfaces in the dashboard chrome
header and per-agent system prompts, and federated hives at different
domains can share one. `hiveName` surfaces in the same places but is
*not* only display: it is the leftmost label of the hive's domain. That
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
distinction — one names this hive, the other names the group it belongs
to.

See `docs/conventions.md` § Hive identity for the env-var chain
and `qualify()` / `qualified_label()` semantics.

## Swarm CA

A hive's internal TLS chains to a **swarm root CA**, so a peer that
trusts the root validates every hive in the swarm rather than being
pinned to each one by hand. Provisioning modes, what to hand a peer
(`trust-bundle.pem`, never `ca.pem`), the name constraints on a hive
CA, and how an existing hive adopts the hierarchy: [`ca.md`](ca.md).

## Running the swarm's shared services

One authelia, one matrix, one forge per swarm — which host runs them,
and what a hive that runs none of them configures instead:
[`services.md`](services.md).

## Single sign-on

Which secrets the SSO provider generates, which one has a reader in
another container, and the three ways that one gets delivered:
[`sso.md`](sso.md).

## Secrets

Every credential the swarm holds, who mints it, where it must live, and
which of the three topologies makes it the operator's job to place:
[`secrets.md`](secrets.md).

## Swarm UI

The operator-only web surface on the swarm apex, why reaching it needs
the `admins` group rather than just a session, and the four sites a
swarm service name has to be wired into: [`ui.md`](ui.md).

## The swarm's hive directory

```nix
services.hyperhive.swarm.hives = {
  pr1ma = { domain = "pr1ma.example.com"; };   # this host, per hiveName
  lab   = { domain = "lab.example.com"; };     # a second hive in the swarm
};
```

One attrset describing **every** hive in the swarm, **including this
one**, keyed by that hive's `hiveName`. It is meant to be *identical on
every host* — write it once, share it, and each host reads it correctly
because `services.hyperhive.hiveName` says which entry is itself.

Empty (the default) means this host isn't in a swarm. Once non-empty it
**must** contain an entry for `hiveName`; eval fails naming the missing
hive. That assertion is load-bearing rather than pedantic — "my peers"
is derived as *everything that isn't me*, so a directory that doesn't
contain you derives every hive as a peer and you peer with yourself.

`domain` is required per entry and deliberately undefaulted: it is
conventionally `<name>.<swarm.domain>`, but a wrong domain that
evaluates cleanly points at a real machine that isn't the one you meant.

> **There is no per-hive CA field, and no per-hive cert pinning.** Trust
> inside a swarm comes from the swarm root ([`ca.md`](ca.md)): every
> hive chains to it, so one anchor replaces per-hive pinning entirely.
> What that genuinely drops is trusting a hive whose root this swarm
> does *not* own — another swarm's, or one keeping its own CA. That is
> a cross-swarm problem and wants a mechanism designed for it. (An
> earlier `certFingerprint` field existed for exactly that gap, pinning
> a peer's TLS leaf for hive-c0re's own peer HTTPS checks — removed
> along with the dashboard feature it existed to serve, since nothing
> else ever consumed it.)

## What the config does at runtime

1. **Swarm-wide hive roster** — swarm-controller reads this same
   directory and serves it at `GET /api/hives`; `swarm-ui`'s overview
   page renders it (`docs/swarm/ui.md`). This is the operator-facing
   "what hives exist" surface — a per-hive dashboard "peer hives"
   display existed here once and was removed in favour of it.

2. **Matrix federation** — when `matrix.enable` is on, tuwunel
   federates with the peer's matrix server (discovered via the peer's
   `.well-known/matrix/server` delegation, which the gateway serves).
   Federation validates the peer's TLS certificate against the matrix
   **container's** trust bundle, independent of this directory.

   ⚠️ **That container currently trusts no swarm-internal CA**, so a
   self-signed gateway certificate does not federate. The swarm root
   can't simply be listed there: `security.pki.certificateFiles` is
   read when the system is *built*, and the root is a runtime file (its
   key must never enter the store), so there is no build-time name for
   it. Bridging that needs a runtime mechanism and is tracked as its own
   issue. Until then, federation needs CA-issued certs (ACME). See
   `docs/matrix.md` for federation firewall + TLS requirements.

3. **WireGuard mesh** (optional) — `swarm.wireguard.enable` reads each
   entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
   to configure `wg-hive`. See "WireGuard inter-hive mesh" below.

## One directory, not a bilateral declaration

Both hives hold the **same** `hives` attrset; neither declares the
other. What differs between the two hosts is only `hiveName`:

```
# hive A                          # hive B
hiveName = "pr1ma";               hiveName = "edge";
swarm.hives = { … };              swarm.hives = { … };   # byte-identical
```

That is the point of the shape, and it removes a class of bug rather
than saving typing: a per-host peer list let two hosts hold *different*
facts about the same third hive — a stale endpoint, a rotated
fingerprint — with nothing to detect the disagreement. One entry per
hive makes it unrepresentable.

## WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hive-c0re can configure a host-to-host
WireGuard mesh.

### Generating keys

On each hive host:

```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators
```

### Config example (two hives)

```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    pr1ma = {
      domain             = "pr1ma.example.com";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint  = "198.51.100.1:51820";
      wireguardAddress   = "10.100.0.1/32";
    };
    edge = {
      domain             = "edge.corp";
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint  = "203.0.113.42:51820";
      wireguardAddress   = "10.100.0.2/32";
    };
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.hives = { /* … identical to hive A's … */ };
};
```

### What the mesh does

- `networking.wireguard.interfaces.wg-hive` is configured on the host
  (not inside agent containers; containers reach peers via the host's
  routing table).
- UDP port 51820 (or `listenPort`) is opened on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
  from `services.hyperhive.swarm.peerHives` to build `wg-hive`'s
  `allowedIPs`, so intra-swarm traffic can route over the mesh address
  rather than the public domain.
- `persistentKeepalive = 25` is set by default; override or null to
  disable (not needed when both sides have public IPs and no NAT).

### NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, a STUN relay or a third host (exit node)
is required. Out of scope for v0.

## Snapshot store

One further option lives in this namespace but is documented with the
service it points at: `services.hyperhive.swarm.snapshotStore.{address,
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.

It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
one store, because the receiver keys destinations by *agent* so a
migrating agent keeps one unbroken incremental chain. See
[snapshot-store.md](../snapshot-store.md).

## Swarm controller

`services.hyperhive.swarm.controller.enable` runs the `swarm-controller`
daemon on this host. **Off by default and deliberately not derived from
`services.hyperhive.enable`**: a swarm has one controller, so enabling it
is a statement about swarm topology, not about whether hyperhive is
installed. Every hive runs `hive-c0re` (the agents on that host); one
hive additionally runs this (what is true across hives).

What it serves, why it is a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../swarm-controller/README.md).

### Per-hive status (`GET /api/hives/status`)

One row per hive in `swarm.hives`, saying when it last reported and what
it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that cannot reach the swarm
still knows its own state — you just cannot see it from here.

⚠️ **Nothing publishes yet.** The read path is in place; the hive-side
publisher lands in a later change. Until it does, every hive reads
`never_reported`.

| freshness | what to do about it |
|---|---|
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |

Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
arrival, not one the hive put in its own payload.

Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
`120`) **above the rate hives publish at**, or everything reads `stale`
between reports. It takes effect on the next request; nothing has to
re-publish.

The endpoint answers **503** when this host has no swarm queue
configured, or has one and cannot read it — deliberately not an empty
list, which would look like a silent swarm rather than a controller that
cannot see. The body says which. Status survives a controller restart:
it is stored in the queue, not in the daemon.

## Cross-references

- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
  the `swarm.snapshotStore` option that points a hive at it
- `docs/conventions.md` § Hive identity — env vars, qualified labels
- `docs/matrix.md` — matrix federation, TLS cert auto-generation,
  firewall posture
- `docs/swarm/ui.md` — the swarm-wide hive roster, now the operator
  surface for "what hives exist" (superseded the per-hive dashboard's
  old "peer hives" display)
- `docs/gateway.md` — nginx vhosts and the `.well-known/matrix/`
  auto-discovery scheme