vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: split swarm.md into a directory, starting with the services page

Browse source 
`docs/swarm.md` becomes `docs/swarm/README.md` and the shared-services
material moves to `docs/swarm/services.md`, following the shape
`docs/turn-loop/` and `docs/web-ui/` already use. The README keeps a
pointer so the reading path is unbroken.

Every referrer moved with it — five docs pages, two option descriptions
in swarm.nix, and CLAUDE.md's reading path. A pointer to a file that
moved is worse than one to a file that was deleted: the content still
exists, so the reader concludes the note is wrong rather than the path.
This commit is contained in:
atlas 2026-08-05 18:07:04 +02:00
commit b3b1ed19c6
9 changed files with 70 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

343
docs/swarm/README.md Normal file
View file

@ -0,0 +1,343 @@
# Multi-hive swarms
A **swarm** is a collection of agents that share an identity and
coordinate across one or more hives. A single hyperhive instance
running on one host is already a swarm (one hive). This doc covers
the additional config needed when the swarm spans multiple hosts.
## Terminology
- **hive** — a single hyperhive installation on one host. Has its
own `services.hyperhive.domain` DNS name and its own set of agent
containers.
- **swarm** — one or more hives whose operators have declared them
as peers. Agents can be qualified as `agent@hive-domain`.
- **peer hive** — a remote hive declared under
`services.hyperhive.swarm.peers` on the local host.
## Hive identity config
```nix
services.hyperhive = {
swarm.domain = "example.com"; # required — the swarm's DNS domain
hiveName = "pr1ma"; # required — this hive's label in it
# domain = "pr1ma.example.com"; # derived from the two above
swarm.name = "constellat1on"; # shared swarm display name (optional)
};
```
`swarm.domain` and `hiveName` are **required** whenever hyperhive is
enabled; eval fails with a hint naming each. Neither is defaulted,
because a guessed value here is a wrong hostname that evaluates cleanly
and deploys — an eval failure asking the operator to write the address
down is the cheaper outcome. **Upgrading past this release means setting
both once.**
`domain` is required too, but you no longer have to *write* it: it
defaults to `<hiveName>.<swarm.domain>`, since every hive in a swarm
occupies its own sub-domain of it. A hive that pins `domain` explicitly
keeps exactly the value it has today — that's why this is a default and
not a rename: re-rooting where a value *comes from* must not reinterpret
the values already deployed.
`domain` drives `HYPERHIVE_HIVE_DOMAIN` in every container so agents can
form qualified labels (`iris@pr1ma.example.com`).
`swarm.name` is purely display — it surfaces in the dashboard chrome
header and per-agent system prompts, and federated hives at different
domains can share one. `hiveName` surfaces in the same places but is
*not* only display: it is the leftmost label of the hive's domain. That
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
distinction — one names this hive, the other names the group it belongs
to.
See `docs/conventions.md` § Hive identity for the env-var chain
and `qualify()` / `qualified_label()` semantics.
## Swarm CA
A hive's internal TLS chains to a **swarm root CA**: the root signs each
hive's own CA, and that hive CA signs the gateway leaf. A peer that
trusts the root once validates every hive in the swarm, present and
future, instead of being pinned to each one by hand.
Two provisioning modes, one structure — what differs is who puts the
artifacts on disk:
| | swarm root | this hive's CA |
| --- | --- | --- |
| default | operator-provided | operator-provided, else self-signed as before |
| `autoConfigure = true` (all on one host) | generated by `swarm-ca.service` on first boot | issued by `hive-tls-ca.service` under the root |
`services.hyperhive.swarm.ca.autoConfigure` selects between them, and is
**off by default**: a swarm's services and its hives can live on
different hosts, and a host cannot tell whether it is the one holding
the root, so setting the swarm CA up is an operator action rather than
something a host assumes. Turn it on for an all-on-one-host deployment
and the hierarchy costs no configuration.
**A hive given neither artifact keeps the self-signed CA it has always
had.** It serves TLS exactly as before and simply isn't part of a
swarm's trust hierarchy — the right outcome for a hive nobody has
federated yet. Only `autoConfigure` issues a hive sub-CA, because only
that case can: signing one needs the root's private key.
Moving the swarm CA onto its own host is then a matter of moving
`services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off —
there is no second code path to switch to.
The root's private key never reaches the nix store: the store is
world-readable, so a key committed to a flake is a key published to
everyone who builds it. Only certificates are distributed.
Each hive CA is **name-constrained** (X.509 `nameConstraints`) to that
hive's own domain, so a hive CA that leaks can only mint names inside
its own subdomain — enforced by every verifier rather than by
convention.
### What to hand a peer
`hivectl peer-config` prints the `cp` line. The file is
`<tls.stateDir>/trust-bundle.pem` — the hive CA plus the swarm root —
not `ca.pem`: an intermediate on its own is not something a verifier can
build a chain to. On a hive that predates the swarm root the bundle is
just that hive's self-signed CA, so the recipe does not change.
### Adopting the hierarchy on an existing hive
An existing `ca.pem` is never re-rooted automatically — swapping it
would break every consumer that already trusts it, and agents only pick
up new trust when their container restarts. To adopt, delete `ca.pem` +
`ca-key.pem` under `tls.stateDir` and restart `hive-tls-ca.service`;
the CA is re-issued under the root and the leaf re-signed. Until then
the hive serves TLS exactly as before and is simply not part of the
swarm's trust hierarchy.
## Running the swarm's shared services
One authelia, one matrix, one forge per swarm — which host runs them,
and what a hive that runs none of them configures instead:
[`services.md`](services.md).
## Declaring peer hives
```nix
services.hyperhive.swarm.peers = {
"lab.example.com" = { }; # CA-trusted (Let's Encrypt etc.)
"edge.corp" = { certFingerprint = "sha256:…"; }; # self-signed TLS, c0re peer checks only
"mesh.internal" = { caCert = ./mesh-ca.pem; }; # self-signed, trusted for matrix federation
};
```
The attrset key is the peer's DNS domain. Two independent, optional
trust knobs — pick by what you need to trust:
- **`certFingerprint`** (`"sha256:…"`) — pin the peer's TLS _leaf_
fingerprint. Scopes **only** to hive-c0re's own peer HTTPS checks
(the P33RS dashboard links + agent peer discovery below). It is
**not** consulted by matrix federation — tuwunel validates a peer's
federation certificate against the system CA bundle independently
(see _Matrix federation_ below), so a fingerprint pin does nothing
for a self-signed matrix cert.
- **`caCert`** (path to the peer's trust bundle / root CA PEM, i.e. what
its `hivectl peer-config` told you to copy) — embeds that CA (at
build time, into the nix store — no runtime file on the host) and
trusts it **everywhere the hive's own internal CA is**: it rides
alongside `hive-ca.pem` in every agent's
`security.pki.certificateFiles` (via the meta-flake renderer) **and**
in the matrix container's trust bundle, so tuwunel validates the
peer's _federation_ TLS when it chains to that CA. Trust stays
**inside the hive** (agents + the matrix container), never the host
system trust store. **This is the knob that unblocks federation with
a self-signed peer hive** — use it instead of `certFingerprint` when
you control the peer's CA. (It does not affect hive-c0re's own peer
HTTPS checks — those stay on `certFingerprint` / the system bundle.)
- **Both omitted** — the stock system CA bundle validates the peer
(correct for Let's Encrypt / any publicly-trusted peer).
### Fingerprint format
The value is the string `sha256:` followed by exactly 64 hexadecimal
digits — the SHA-256 digest of the peer's DER-encoded TLS leaf
certificate. The hex is case-insensitive (upper or lower both parse),
carries no colon separators between bytes, and any value not matching
this shape is ignored with a warning rather than weakening trust.
```
sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12
```
Generate it from the peer's certificate with openssl. The
`-fingerprint -sha256` output is uppercase and colon-separated, so
strip the colons, lowercase, and prepend the `sha256:` prefix:
```sh
# from a PEM/CRT file
openssl x509 -in peer.crt -noout -fingerprint -sha256 \
| sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
# straight from the live endpoint (port 443)
echo | openssl s_client -connect peer.example.com:443 -servername peer.example.com 2>/dev/null \
| openssl x509 -noout -fingerprint -sha256 \
| sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
```
Pin the leaf certificate, not an intermediate or the CA — the
digest must match the exact cert the peer serves on its HTTPS
endpoint. When the peer rotates its cert, update the pin to the new
fingerprint (or switch the peer to a CA-trusted cert and drop the
field).
The nix module serialises the attrset to a `HYPERHIVE_PEERS` JSON
array (`[{ domain, cert_fingerprint }]`) injected into the c0re
environment and forwarded to agent containers.
## What the config does at runtime
1. **Dashboard P33RS tab** — `parse_peer_hives()` in `dashboard.rs`
reads `HYPERHIVE_PEERS` and includes
`peer_hives: Vec<{ name, url }>` in `/api/state`. The dashboard
shows a P33RS tab (hidden when the list is empty) with a card per
peer linking to `https://{domain}/`. See
`docs/web-ui/dashboard.md` § P33RS tab.
2. **Agent identity** — the same `HYPERHIVE_PEERS` env var is
forwarded to agent containers by `meta.rs`; agent code can call
`identity::peers()` to discover peer hives and address them with
qualified names (`agent@domain`).
3. **Matrix federation** — when `matrix.enable` is on, tuwunel
federates with the peer's matrix server (discovered via the peer's
`.well-known/matrix/server` delegation, which the gateway serves).
Federation validates the peer's TLS certificate against the matrix
**container's** trust bundle — independently of `certFingerprint`,
which it never consults. A self-signed gateway certificate therefore
won't federate unless the peer's root CA is trusted: set `caCert`
above (embeds the peer CA into the matrix container's trust bundle),
or give the peers CA-issued certs (ACME). See `docs/matrix.md` for
federation firewall + TLS requirements.
## Bilateral setup
Each hive must declare the other. If hive A lists hive B as a peer,
B must also list A for agents on B to see A in their peer list:
```
# hive A (pr1ma.example.com)
services.hyperhive.swarm.peers."edge.corp" = { };
# hive B (edge.corp)
services.hyperhive.swarm.peers."pr1ma.example.com" = { certFingerprint = "sha256:…"; };
```
Mixed trust is fine: A trusts B via CA bundle (no fingerprint), B
pins A's self-signed cert.
## WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hive-c0re can configure a host-to-host
WireGuard mesh.
### Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
### Config example (two hives)
```nix
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
swarm.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
swarm.peers."edge.corp" = {
certFingerprint = "sha256:…"; # TLS trust (unchanged)
wireguardPublicKey = "base64key="; # peer's wg pubkey
wireguardEndpoint = "203.0.113.42:51820"; # peer's public IP:port
wireguardAddress = "10.100.0.2/32"; # peer's mesh IP
};
};
# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
swarm.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.peers."pr1ma.example.com" = {
wireguardPublicKey = "base64key="; # hive A's wg pubkey
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
};
```
### What the mesh does
- `networking.wireguard.interfaces.wg-hive` is configured on the host
(not inside agent containers; containers reach peers via the host's
routing table).
- UDP port 51820 (or `listenPort`) is opened on the host firewall.
- `HYPERHIVE_PEERS` gains a `wireguard_address` field for each mesh
peer so hive-c0re can reach intra-swarm services without a public
DNS round-trip.
- `persistentKeepalive = 25` is set by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
### NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, a STUN relay or a third host (exit node)
is required. Out of scope for v0.
## Snapshot store
One further option lives in this namespace but is documented with the
service it points at: `services.hyperhive.swarm.snapshotStore.{address,
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
one store, because the receiver keys destinations by *agent* so a
migrating agent keeps one unbroken incremental chain. See
[snapshot-store.md](snapshot-store.md).
## Swarm controller
`services.hyperhive.swarm.controller.enable` runs the `swarm-controller`
daemon on this host. **Off by default and deliberately not derived from
`services.hyperhive.enable`**: a swarm has one controller, so enabling it
is a statement about swarm topology, not about whether hyperhive is
installed. Every hive runs `hive-c0re` (the agents on that host); one
hive additionally runs this (what is true across hives).
What it serves, why it is a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../swarm-controller/README.md).
## Cross-references
- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
the `swarm.snapshotStore` option that points a hive at it
- `docs/conventions.md` § Hive identity — env vars, qualified labels
- `docs/matrix.md` — matrix federation, TLS cert auto-generation,
firewall posture
- `docs/web-ui/dashboard.md` § P33RS tab — dashboard surface
- `docs/gateway.md` — nginx vhosts and the `.well-known/matrix/`
auto-discovery scheme

59
docs/swarm/services.md Normal file
View file

@ -0,0 +1,59 @@
# Swarm-wide services
Some things exist once per **swarm**, not once per hive: the forge, the
matrix homeserver, the SSO provider, the CA. Two options say where the
optional ones live, and everything else derives:
```nix
services.hyperhive.enableAllLocalDefaults = true; # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.swarm.enableRequiredServices = true;
```
`enableAllLocalDefaults` is the all-on-one-box switch: it defaults both
`swarm.enableRequiredServices` (the shared services run here) and
`swarm.ca.autoConfigure` (the swarm CA is generated here). Each derived
toggle can still be set on its own, so "all local except X" needs no
further option.
**Both default to off**, and that is deliberate: a host cannot tell
whether it is meant to be the swarm's service host, so this is an
operator saying so rather than something inferred. With them off, a hive
is a *client* of those services — it configures how to reach them and
runs none of them.
The forge is the exception, and not because it is per-hive: it is
swarm-wide but **not optional**, being the canonical store for the meta
flake and every agent's config repo, so it deploys with hyperhive itself
and has no `enable` to derive from anything.
### SSO (authelia)
One authelia per swarm, in a `swarm-authelia` container, at
`auth.<hive-domain>`. Operator and agents are both subjects of the same
provider, differentiated by roles and claims rather than by mechanism —
there is one IdP and one auth path.
- **`swarm.authelia.enable`** — run the container here. Defaults from
`swarm.enableRequiredServices`.
- **`swarm.authelia.url`** — where clients are sent to authenticate.
Present on **every** hive, defaulting to this host's own instance only
when this module is the thing running it; otherwise `null`, and a hive
joining someone else's swarm sets it explicitly. Null means "no SSO
configured", and consumers say so rather than guessing an address.
The users database is written by swarm-controller, not by hand: agents
are created and destroyed continuously, so the subject set is dynamic.
This module only guarantees the file exists and parses, so authelia
starts with nobody in it rather than failing to start — a provider with
no subjects yet is the correct state before anything has provisioned
them. Session and storage keys are generated in the container on first
boot and never rotated automatically; replacing one invalidates data
already written (sessions, the encrypted store), so that is an operator
action.
Storage is local sqlite and the notifier writes to a file. Both are
small-deployment choices, and the scope is the justification: redis
buys shared session state across replicas and there is one instance;
SMTP exists to mail humans, and provisioning here is programmatic.