hyperhive/docs/swarm.md

Multi-hive swarms

A swarm is a collection of agents that share an identity and coordinate across one or more hives. A single hyperhive instance running on one host is already a swarm (one hive). This doc covers the additional config needed when the swarm spans multiple hosts.

Terminology

• hive — a single hyperhive installation on one host. Has its own services.hyperhive.domain DNS name and its own set of agent containers.
• swarm — one or more hives whose operators have declared them as peers. Agents can be qualified as agent@hive-domain.
• peer hive — a remote hive declared under services.hyperhive.swarm.peers on the local host.

Hive identity config

services.hyperhive = {
  swarm.domain = "example.com";     # required — the swarm's DNS domain
  hiveName = "pr1ma";               # required — this hive's label in it
  # domain = "pr1ma.example.com";   # derived from the two above
  swarm.name = "constellat1on";     # shared swarm display name (optional)
};

swarm.domain and hiveName are required whenever hyperhive is enabled; eval fails with a hint naming each. Neither is defaulted, because a guessed value here is a wrong hostname that evaluates cleanly and deploys — an eval failure asking the operator to write the address down is the cheaper outcome. Upgrading past this release means setting both once.

domain is required too, but you no longer have to write it: it defaults to <hiveName>.<swarm.domain>, since every hive in a swarm occupies its own sub-domain of it. A hive that pins domain explicitly keeps exactly the value it has today — that's why this is a default and not a rename: re-rooting where a value comes from must not reinterpret the values already deployed.

domain drives HYPERHIVE_HIVE_DOMAIN in every container so agents can form qualified labels (iris@pr1ma.example.com).

swarm.name is purely display — it surfaces in the dashboard chrome header and per-agent system prompts, and federated hives at different domains can share one. hiveName surfaces in the same places but is not only display: it is the leftmost label of the hive's domain. That swarm.name sits under swarm and hiveName does not is the whole distinction — one names this hive, the other names the group it belongs to.

See docs/conventions.md § Hive identity for the env-var chain and qualify() / qualified_label() semantics.

Swarm CA

A hive's internal TLS chains to a swarm root CA: the root signs each hive's own CA, and that hive CA signs the gateway leaf. A peer that trusts the root once validates every hive in the swarm, present and future, instead of being pinned to each one by hand.

Two provisioning modes, one structure — what differs is who puts the artifacts on disk:

	swarm root	this hive's CA
default	operator-provided	operator-provided, else self-signed as before
autoConfigure = true (all on one host)	generated by swarm-ca.service on first boot	issued by hive-tls-ca.service under the root

services.hyperhive.swarm.ca.autoConfigure selects between them, and is off by default: a swarm's services and its hives can live on different hosts, and a host cannot tell whether it is the one holding the root, so setting the swarm CA up is an operator action rather than something a host assumes. Turn it on for an all-on-one-host deployment and the hierarchy costs no configuration.

A hive given neither artifact keeps the self-signed CA it has always had. It serves TLS exactly as before and simply isn't part of a swarm's trust hierarchy — the right outcome for a hive nobody has federated yet. Only autoConfigure issues a hive sub-CA, because only that case can: signing one needs the root's private key.

Moving the swarm CA onto its own host is then a matter of moving services.hyperhive.swarm.ca.stateDir and leaving autoConfigure off — there is no second code path to switch to.

The root's private key never reaches the nix store: the store is world-readable, so a key committed to a flake is a key published to everyone who builds it. Only certificates are distributed.

Each hive CA is name-constrained (X.509 nameConstraints) to that hive's own domain, so a hive CA that leaks can only mint names inside its own subdomain — enforced by every verifier rather than by convention.

What to hand a peer

hivectl peer-config prints the cp line. The file is <tls.stateDir>/trust-bundle.pem — the hive CA plus the swarm root — not ca.pem: an intermediate on its own is not something a verifier can build a chain to. On a hive that predates the swarm root the bundle is just that hive's self-signed CA, so the recipe does not change.

Adopting the hierarchy on an existing hive

An existing ca.pem is never re-rooted automatically — swapping it would break every consumer that already trusts it, and agents only pick up new trust when their container restarts. To adopt, delete ca.pem + ca-key.pem under tls.stateDir and restart hive-tls-ca.service; the CA is re-issued under the root and the leaf re-signed. Until then the hive serves TLS exactly as before and is simply not part of the swarm's trust hierarchy.

Running the swarm's shared services

Some things exist once per swarm, not once per hive: the forge, the matrix homeserver, the SSO provider, the CA. Two options say where the optional ones live, and everything else derives:

services.hyperhive.enableAllLocalDefaults = true;   # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.swarm.enableRequiredServices = true;

enableAllLocalDefaults is the all-on-one-box switch: it defaults both swarm.enableRequiredServices (the shared services run here) and swarm.ca.autoConfigure (the swarm CA is generated here). Each derived toggle can still be set on its own, so "all local except X" needs no further option.

Both default to off, and that is deliberate: a host cannot tell whether it is meant to be the swarm's service host, so this is an operator saying so rather than something inferred. With them off, a hive is a client of those services — it configures how to reach them and runs none of them.

The forge is the exception, and not because it is per-hive: it is swarm-wide but not optional, being the canonical store for the meta flake and every agent's config repo, so it deploys with hyperhive itself and has no enable to derive from anything.

SSO (authelia)

One authelia per swarm, in a swarm-authelia container, at auth.<hive-domain>. Operator and agents are both subjects of the same provider, differentiated by roles and claims rather than by mechanism — there is one IdP and one auth path.

• swarm.authelia.enable — run the container here. Defaults from swarm.enableRequiredServices.
• swarm.authelia.url — where clients are sent to authenticate. Present on every hive, defaulting to this host's own instance only when this module is the thing running it; otherwise null, and a hive joining someone else's swarm sets it explicitly. Null means "no SSO configured", and consumers say so rather than guessing an address.

The users database is written by swarm-controller, not by hand: agents are created and destroyed continuously, so the subject set is dynamic. This module only guarantees the file exists and parses, so authelia starts with nobody in it rather than failing to start — a provider with no subjects yet is the correct state before anything has provisioned them. Session and storage keys are generated in the container on first boot and never rotated automatically; replacing one invalidates data already written (sessions, the encrypted store), so that is an operator action.

Storage is local sqlite and the notifier writes to a file. Both are small-deployment choices, and the scope is the justification: redis buys shared session state across replicas and there is one instance; SMTP exists to mail humans, and provisioning here is programmatic.

Declaring peer hives

services.hyperhive.swarm.peers = {
  "lab.example.com"  = { };                               # CA-trusted (Let's Encrypt etc.)
  "edge.corp"        = { certFingerprint = "sha256:…"; }; # self-signed TLS, c0re peer checks only
  "mesh.internal"    = { caCert = ./mesh-ca.pem; };       # self-signed, trusted for matrix federation
};

The attrset key is the peer's DNS domain. Two independent, optional trust knobs — pick by what you need to trust:

• certFingerprint ("sha256:…") — pin the peer's TLS leaf fingerprint. Scopes only to hive-c0re's own peer HTTPS checks (the P33RS dashboard links + agent peer discovery below). It is not consulted by matrix federation — tuwunel validates a peer's federation certificate against the system CA bundle independently (see Matrix federation below), so a fingerprint pin does nothing for a self-signed matrix cert.
• caCert (path to the peer's trust bundle / root CA PEM, i.e. what its hivectl peer-config told you to copy) — embeds that CA (at build time, into the nix store — no runtime file on the host) and trusts it everywhere the hive's own internal CA is: it rides alongside hive-ca.pem in every agent's security.pki.certificateFiles (via the meta-flake renderer) and in the matrix container's trust bundle, so tuwunel validates the peer's federation TLS when it chains to that CA. Trust stays inside the hive (agents + the matrix container), never the host system trust store. This is the knob that unblocks federation with a self-signed peer hive — use it instead of certFingerprint when you control the peer's CA. (It does not affect hive-c0re's own peer HTTPS checks — those stay on certFingerprint / the system bundle.)
• Both omitted — the stock system CA bundle validates the peer (correct for Let's Encrypt / any publicly-trusted peer).

Fingerprint format

The value is the string sha256: followed by exactly 64 hexadecimal digits — the SHA-256 digest of the peer's DER-encoded TLS leaf certificate. The hex is case-insensitive (upper or lower both parse), carries no colon separators between bytes, and any value not matching this shape is ignored with a warning rather than weakening trust.

sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12

Generate it from the peer's certificate with openssl. The -fingerprint -sha256 output is uppercase and colon-separated, so strip the colons, lowercase, and prepend the sha256: prefix:

# from a PEM/CRT file
openssl x509 -in peer.crt -noout -fingerprint -sha256 \
  | sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'

# straight from the live endpoint (port 443)
echo | openssl s_client -connect peer.example.com:443 -servername peer.example.com 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 \
  | sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'

Pin the leaf certificate, not an intermediate or the CA — the digest must match the exact cert the peer serves on its HTTPS endpoint. When the peer rotates its cert, update the pin to the new fingerprint (or switch the peer to a CA-trusted cert and drop the field).

The nix module serialises the attrset to a HYPERHIVE_PEERS JSON array ([{ domain, cert_fingerprint }]) injected into the c0re environment and forwarded to agent containers.

What the config does at runtime

1. Dashboard P33RS tab — parse_peer_hives() in dashboard.rs reads HYPERHIVE_PEERS and includes peer_hives: Vec<{ name, url }> in /api/state. The dashboard shows a P33RS tab (hidden when the list is empty) with a card per peer linking to https://{domain}/. See docs/web-ui/dashboard.md § P33RS tab.

2. Agent identity — the same HYPERHIVE_PEERS env var is forwarded to agent containers by meta.rs; agent code can call identity::peers() to discover peer hives and address them with qualified names (agent@domain).

3. Matrix federation — when matrix.enable is on, tuwunel federates with the peer's matrix server (discovered via the peer's .well-known/matrix/server delegation, which the gateway serves). Federation validates the peer's TLS certificate against the matrix container's trust bundle — independently of certFingerprint, which it never consults. A self-signed gateway certificate therefore won't federate unless the peer's root CA is trusted: set caCert above (embeds the peer CA into the matrix container's trust bundle), or give the peers CA-issued certs (ACME). See docs/matrix.md for federation firewall + TLS requirements.

Bilateral setup

Each hive must declare the other. If hive A lists hive B as a peer, B must also list A for agents on B to see A in their peer list:

# hive A (pr1ma.example.com)
services.hyperhive.swarm.peers."edge.corp" = { };

# hive B (edge.corp)
services.hyperhive.swarm.peers."pr1ma.example.com" = { certFingerprint = "sha256:…"; };

Mixed trust is fine: A trusts B via CA bundle (no fingerprint), B pins A's self-signed cert.

WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic. For private deployments — or to reduce latency and TLS overhead on intra-swarm traffic — hive-c0re can configure a host-to-host WireGuard mesh.

Generating keys

On each hive host:

wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators

Config example (two hives)

# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  swarm.peers."edge.corp" = {
    certFingerprint    = "sha256:…";   # TLS trust (unchanged)
    wireguardPublicKey = "base64key="; # peer's wg pubkey
    wireguardEndpoint  = "203.0.113.42:51820"; # peer's public IP:port
    wireguardAddress   = "10.100.0.2/32";      # peer's mesh IP
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.peers."pr1ma.example.com" = {
    wireguardPublicKey = "base64key="; # hive A's wg pubkey
    wireguardEndpoint  = "198.51.100.1:51820";
    wireguardAddress   = "10.100.0.1/32";
  };
};

What the mesh does

• networking.wireguard.interfaces.wg-hive is configured on the host (not inside agent containers; containers reach peers via the host's routing table).
• UDP port 51820 (or listenPort) is opened on the host firewall.
• HYPERHIVE_PEERS gains a wireguard_address field for each mesh peer so hive-c0re can reach intra-swarm services without a public DNS round-trip.
• persistentKeepalive = 25 is set by default; override or null to disable (not needed when both sides have public IPs and no NAT).

NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only that host needs a null wireguardEndpoint on the peer config — the other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, a STUN relay or a third host (exit node) is required. Out of scope for v0.

Snapshot store

One further option lives in this namespace but is documented with the service it points at: services.hyperhive.swarm.snapshotStore.{address, port} tells this hive where the swarm's btrfs receive endpoint is, so hivectl agent <name> subvol snapshot push has somewhere to stream to.

It is genuinely swarm-scoped rather than per-peer — a swarm has exactly one store, because the receiver keys destinations by agent so a migrating agent keeps one unbroken incremental chain. See snapshot-store.md.

Swarm controller

services.hyperhive.swarm.controller.enable runs the swarm-controller daemon on this host. Off by default and deliberately not derived from services.hyperhive.enable: a swarm has one controller, so enabling it is a statement about swarm topology, not about whether hyperhive is installed. Every hive runs hive-c0re (the agents on that host); one hive additionally runs this (what is true across hives).

What it serves, why it is a unix socket rather than a port, and the socket-directory constraint that governs where socketPath may point: swarm-controller/README.md.

Cross-references

• docs/snapshot-store.md — the swarm's btrfs receive endpoint, and the swarm.snapshotStore option that points a hive at it
• docs/conventions.md § Hive identity — env vars, qualified labels
• docs/matrix.md — matrix federation, TLS cert auto-generation, firewall posture
• docs/web-ui/dashboard.md § P33RS tab — dashboard surface
• docs/gateway.md — nginx vhosts and the .well-known/matrix/ auto-discovery scheme