hyperhive/docs/swarm.md

# Multi-hive swarms

A **swarm** is a collection of agents that share an identity and
coordinate across one or more hives. A single hyperhive instance
running on one host is already a swarm (one hive). This doc covers
the additional config needed when the swarm spans multiple hosts.

## Terminology

- **hive** — a single hyperhive installation on one host. Has its
  own `services.hyperhive.domain` DNS name and its own set of agent
  containers.
- **swarm** — one or more hives whose operators have declared them
  as peers. Agents can be qualified as `agent@hive-domain`.
- **peer hive** — a remote hive declared under
  `services.hyperhive.swarm.peers` on the local host.

## Hive identity config

```nix
services.hyperhive = {
  domain   = "pr1ma.example.com";   # machine-addressable DNS domain
  hiveName = "pr1ma";               # human display name (optional)
  swarm.name = "constellat1on";     # shared swarm display name (optional)
};
```

`domain` is required when matrix federation is on (`matrix.enable`);
it drives `HYPERHIVE_HIVE_DOMAIN` in every container so agents can
form qualified labels (`iris@pr1ma.example.com`). `hiveName` and
`swarm.name` are purely display — they surface in the dashboard chrome
header and per-agent system prompts. Federated hives at different
domains can share a `swarm.name`; that it sits under `swarm` and
`hiveName` does not is the whole distinction — one names this hive, the
other names the group it belongs to.

See `docs/conventions.md` § Hive identity for the env-var chain
and `qualify()` / `qualified_label()` semantics.

## Declaring peer hives

```nix
services.hyperhive.swarm.peers = {
  "lab.example.com"  = { };                               # CA-trusted (Let's Encrypt etc.)
  "edge.corp"        = { certFingerprint = "sha256:…"; }; # self-signed TLS, c0re peer checks only
  "mesh.internal"    = { caCert = ./mesh-ca.pem; };       # self-signed, trusted for matrix federation
};
```

The attrset key is the peer's DNS domain. Two independent, optional
trust knobs — pick by what you need to trust:

- **`certFingerprint`** (`"sha256:…"`) — pin the peer's TLS _leaf_
  fingerprint. Scopes **only** to hive-c0re's own peer HTTPS checks
  (the P33RS dashboard links + agent peer discovery below). It is
  **not** consulted by matrix federation — tuwunel validates a peer's
  federation certificate against the system CA bundle independently
  (see _Matrix federation_ below), so a fingerprint pin does nothing
  for a self-signed matrix cert.
- **`caCert`** (path to the peer's root CA PEM) — embeds that CA (at
  build time, into the nix store — no runtime file on the host) and
  trusts it **everywhere the hive's own internal CA is**: it rides
  alongside `hive-ca.pem` in every agent's
  `security.pki.certificateFiles` (via the meta-flake renderer) **and**
  in the matrix container's trust bundle, so tuwunel validates the
  peer's _federation_ TLS when it chains to that CA. Trust stays
  **inside the hive** (agents + the matrix container), never the host
  system trust store. **This is the knob that unblocks federation with
  a self-signed peer hive** — use it instead of `certFingerprint` when
  you control the peer's CA. (It does not affect hive-c0re's own peer
  HTTPS checks — those stay on `certFingerprint` / the system bundle.)
- **Both omitted** — the stock system CA bundle validates the peer
  (correct for Let's Encrypt / any publicly-trusted peer).

### Fingerprint format

The value is the string `sha256:` followed by exactly 64 hexadecimal
digits — the SHA-256 digest of the peer's DER-encoded TLS leaf
certificate. The hex is case-insensitive (upper or lower both parse),
carries no colon separators between bytes, and any value not matching
this shape is ignored with a warning rather than weakening trust.

```
sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12
```

Generate it from the peer's certificate with openssl. The
`-fingerprint -sha256` output is uppercase and colon-separated, so
strip the colons, lowercase, and prepend the `sha256:` prefix:

```sh
# from a PEM/CRT file
openssl x509 -in peer.crt -noout -fingerprint -sha256 \
  | sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'

# straight from the live endpoint (port 443)
echo | openssl s_client -connect peer.example.com:443 -servername peer.example.com 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 \
  | sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
```

Pin the leaf certificate, not an intermediate or the CA — the
digest must match the exact cert the peer serves on its HTTPS
endpoint. When the peer rotates its cert, update the pin to the new
fingerprint (or switch the peer to a CA-trusted cert and drop the
field).

The nix module serialises the attrset to a `HYPERHIVE_PEERS` JSON
array (`[{ domain, cert_fingerprint }]`) injected into the c0re
environment and forwarded to agent containers.

## What the config does at runtime

1. **Dashboard P33RS tab** — `parse_peer_hives()` in `dashboard.rs`
   reads `HYPERHIVE_PEERS` and includes
   `peer_hives: Vec<{ name, url }>` in `/api/state`. The dashboard
   shows a P33RS tab (hidden when the list is empty) with a card per
   peer linking to `https://{domain}/`. See
   `docs/web-ui/dashboard.md` § P33RS tab.

2. **Agent identity** — the same `HYPERHIVE_PEERS` env var is
   forwarded to agent containers by `meta.rs`; agent code can call
   `identity::peers()` to discover peer hives and address them with
   qualified names (`agent@domain`).

3. **Matrix federation** — when `matrix.enable` is on, tuwunel
   federates with the peer's matrix server (discovered via the peer's
   `.well-known/matrix/server` delegation, which the gateway serves).
   Federation validates the peer's TLS certificate against the matrix
   **container's** trust bundle — independently of `certFingerprint`,
   which it never consults. A self-signed gateway certificate therefore
   won't federate unless the peer's root CA is trusted: set `caCert`
   above (embeds the peer CA into the matrix container's trust bundle),
   or give the peers CA-issued certs (ACME). See `docs/matrix.md` for
   federation firewall + TLS requirements.

## Bilateral setup

Each hive must declare the other. If hive A lists hive B as a peer,
B must also list A for agents on B to see A in their peer list:

```
# hive A (pr1ma.example.com)
services.hyperhive.swarm.peers."edge.corp" = { };

# hive B (edge.corp)
services.hyperhive.swarm.peers."pr1ma.example.com" = { certFingerprint = "sha256:…"; };
```

Mixed trust is fine: A trusts B via CA bundle (no fingerprint), B
pins A's self-signed cert.

## WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hive-c0re can configure a host-to-host
WireGuard mesh.

### Generating keys

On each hive host:

```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators
```

### Config example (two hives)

```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  swarm.peers."edge.corp" = {
    certFingerprint    = "sha256:…";   # TLS trust (unchanged)
    wireguardPublicKey = "base64key="; # peer's wg pubkey
    wireguardEndpoint  = "203.0.113.42:51820"; # peer's public IP:port
    wireguardAddress   = "10.100.0.2/32";      # peer's mesh IP
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.peers."pr1ma.example.com" = {
    wireguardPublicKey = "base64key="; # hive A's wg pubkey
    wireguardEndpoint  = "198.51.100.1:51820";
    wireguardAddress   = "10.100.0.1/32";
  };
};
```

### What the mesh does

- `networking.wireguard.interfaces.wg-hive` is configured on the host
  (not inside agent containers; containers reach peers via the host's
  routing table).
- UDP port 51820 (or `listenPort`) is opened on the host firewall.
- `HYPERHIVE_PEERS` gains a `wireguard_address` field for each mesh
  peer so hive-c0re can reach intra-swarm services without a public
  DNS round-trip.
- `persistentKeepalive = 25` is set by default; override or null to
  disable (not needed when both sides have public IPs and no NAT).

### NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, a STUN relay or a third host (exit node)
is required. Out of scope for v0.

## Snapshot store

One further option lives in this namespace but is documented with the
service it points at: `services.hyperhive.swarm.snapshotStore.{address,
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.

It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
one store, because the receiver keys destinations by *agent* so a
migrating agent keeps one unbroken incremental chain. See
[snapshot-store.md](snapshot-store.md).

## Swarm controller

`services.hyperhive.swarm.controller.enable` runs the `swarm-controller`
daemon on this host. It is **off by default and deliberately not derived
from `services.hyperhive.enable`**: a swarm has one controller, so
enabling it is a statement about swarm topology, not about whether
hyperhive is installed. Every hive runs `hive-c0re` (which owns the
agents on that host); one hive additionally runs this (which owns what is
true across hives).

It serves HTTP over a unix socket — `socketPath`, default
`/run/swarm-controller/controller.sock` — rather than a TCP port. The
gateway's nginx is the only intended client and reaches the socket
through a bind-mount, and a listener that is never bound to an address
cannot be reached from off-host by mistake.

⚠️ **The socket's directory is its access control.** The socket itself is
`0666`, because nginx runs as a different user and `connect(2)` needs
write — the same arrangement hive-c0re uses for the per-agent sockets.
What keeps that safe is that the directory holds one socket and is
bind-mounted into exactly one container. Pointing `socketPath` at a
directory that carries anything else — `/run/hyperhive`, which holds the
host **admin** socket, above all — exposes everything in it to every
consumer that mounts it. Changing `socketPath` therefore means
re-checking the gateway bind-mount, not just the daemon.

Today the daemon serves a single `/health` endpoint and holds no state:
the unit exists so the swarm-level surfaces that follow have somewhere to
land.

## Cross-references

- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
  the `swarm.snapshotStore` option that points a hive at it
- `docs/conventions.md` § Hive identity — env vars, qualified labels
- `docs/matrix.md` — matrix federation, TLS cert auto-generation,
  firewall posture
- `docs/web-ui/dashboard.md` § P33RS tab — dashboard surface
- `docs/gateway.md` — nginx vhosts and the `.well-known/matrix/`
  auto-discovery scheme