diff --git a/docs/gateway.md b/docs/gateway.md index 03f6e195..1d2c26e9 100644 --- a/docs/gateway.md +++ b/docs/gateway.md @@ -188,7 +188,7 @@ nginx inside the gateway container obtains and auto-renews certs via the ACME HT Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails an assertion — pick one external TLS source (or neither, for the self-signed default). -**Swarm peers**: CA-signed certs are trusted by default — remote hives need no `certFingerprint` in `swarm.peers`. +**Swarm peers**: CA-signed certs are trusted by default — this hive's entry in `swarm.hives` needs no `certFingerprint`. ### Self-signed TLS (default) @@ -232,10 +232,10 @@ security.acme.certs."example.com".group = "nginx"; or make the key world-readable (`0644`) if your threat model allows it. nginx errors out at startup on a key it can't read — the error is explicit in the journal, not a silent failure. -**Peer hive config**: when using a CA-signed cert, peer hives can declare this hive without `certFingerprint` in `swarm.peers` — the standard CA bundle validates: +**Swarm directory entry**: when using a CA-signed cert, this hive's entry needs no `certFingerprint` — the standard CA bundle validates: ```nix -services.hyperhive.swarm.peers."example.com" = { }; # no certFingerprint needed +services.hyperhive.swarm.hives.example = { domain = "example.com"; }; # no certFingerprint needed ``` ### Fronting with an external TLS terminator diff --git a/docs/snapshot-store.md b/docs/snapshot-store.md index c82ffbba..2ea3c35b 100644 --- a/docs/snapshot-store.md +++ b/docs/snapshot-store.md @@ -33,8 +33,8 @@ services.hyperhive.swarm.wireguard = { }; ``` -The store host is a swarm member like any other: peers declare it, and -it declares them, through `services.hyperhive.swarm.peers`. See +The store host is a swarm member like any other: it gets an entry in +`services.hyperhive.swarm.hives`, the same directory every host holds. See [swarm/](swarm/README.md) for the mesh itself. Note that the mesh is gated on `swarm.wireguard.enable`, **not** on diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 1e0c1bc5..574ae569 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -12,8 +12,9 @@ the additional config needed when the swarm spans multiple hosts. containers. - **swarm** — one or more hives whose operators have declared them as peers. Agents can be qualified as `agent@hive-domain`. -- **peer hive** — a remote hive declared under - `services.hyperhive.swarm.peers` on the local host. +- **peer hive** — any hive in `services.hyperhive.swarm.hives` other + than this one. Peers are *derived*, not declared: the directory lists + every hive including yourself, and `hiveName` says which one you are. ## Hive identity config @@ -71,41 +72,43 @@ One authelia, one matrix, one forge per swarm — which host runs them, and what a hive that runs none of them configures instead: [`services.md`](services.md). -## Declaring peer hives +## The swarm's hive directory ```nix -services.hyperhive.swarm.peers = { - "lab.example.com" = { }; # CA-trusted (Let's Encrypt etc.) - "edge.corp" = { certFingerprint = "sha256:…"; }; # self-signed TLS, c0re peer checks only - "mesh.internal" = { caCert = ./mesh-ca.pem; }; # self-signed, trusted for matrix federation +services.hyperhive.swarm.hives = { + pr1ma = { domain = "pr1ma.example.com"; }; # this host, per hiveName + lab = { domain = "lab.example.com"; }; # CA-trusted (Let's Encrypt etc.) + edge = { domain = "edge.corp"; certFingerprint = "sha256:…"; }; # self-signed leaf, pinned }; ``` -The attrset key is the peer's DNS domain. Two independent, optional -trust knobs — pick by what you need to trust: +One attrset describing **every** hive in the swarm, **including this +one**, keyed by that hive's `hiveName`. It is meant to be *identical on +every host* — write it once, share it, and each host reads it correctly +because `services.hyperhive.hiveName` says which entry is itself. -- **`certFingerprint`** (`"sha256:…"`) — pin the peer's TLS _leaf_ - fingerprint. Scopes **only** to hive-c0re's own peer HTTPS checks - (the P33RS dashboard links + agent peer discovery below). It is - **not** consulted by matrix federation — tuwunel validates a peer's - federation certificate against the system CA bundle independently - (see _Matrix federation_ below), so a fingerprint pin does nothing - for a self-signed matrix cert. -- **`caCert`** (path to the peer's trust bundle / root CA PEM, i.e. what - its `hivectl peer-config` told you to copy) — embeds that CA (at - build time, into the nix store — no runtime file on the host) and - trusts it **everywhere the hive's own internal CA is**: it rides - alongside `hive-ca.pem` in every agent's - `security.pki.certificateFiles` (via the meta-flake renderer) **and** - in the matrix container's trust bundle, so tuwunel validates the - peer's _federation_ TLS when it chains to that CA. Trust stays - **inside the hive** (agents + the matrix container), never the host - system trust store. **This is the knob that unblocks federation with - a self-signed peer hive** — use it instead of `certFingerprint` when - you control the peer's CA. (It does not affect hive-c0re's own peer - HTTPS checks — those stay on `certFingerprint` / the system bundle.) -- **Both omitted** — the stock system CA bundle validates the peer - (correct for Let's Encrypt / any publicly-trusted peer). +Empty (the default) means this host isn't in a swarm. Once non-empty it +**must** contain an entry for `hiveName`; eval fails naming the missing +hive. That assertion is load-bearing rather than pedantic — "my peers" +is derived as *everything that isn't me*, so a directory that doesn't +contain you derives every hive as a peer and you peer with yourself. + +`domain` is required per entry and deliberately undefaulted: it is +conventionally `.`, but a wrong domain that +evaluates cleanly points at a real machine that isn't the one you meant. + +**`certFingerprint`** (`"sha256:…"`, optional) pins that hive's TLS +_leaf_. Scopes **only** to hive-c0re's own peer HTTPS checks (the P33RS +dashboard links + agent peer discovery below); matrix federation never +consults it. Omit it for any hive under the swarm root CA or a public +CA — which is the normal case. + +> **There is no per-hive CA field.** Trust inside a swarm comes from the +> swarm root ([`ca.md`](ca.md)): every hive chains to it, so one anchor +> replaces the O(n²) pinning. What that genuinely drops is trusting a +> hive whose root this swarm does *not* own — another swarm's, or one +> keeping its own CA. That is a cross-swarm problem and wants a +> mechanism designed for it, not a field that happened to work. ### Fingerprint format @@ -163,27 +166,33 @@ environment and forwarded to agent containers. `.well-known/matrix/server` delegation, which the gateway serves). Federation validates the peer's TLS certificate against the matrix **container's** trust bundle — independently of `certFingerprint`, - which it never consults. A self-signed gateway certificate therefore - won't federate unless the peer's root CA is trusted: set `caCert` - above (embeds the peer CA into the matrix container's trust bundle), - or give the peers CA-issued certs (ACME). See `docs/matrix.md` for - federation firewall + TLS requirements. + which it never consults. -## Bilateral setup + ⚠️ **That container currently trusts no swarm-internal CA**, so a + self-signed gateway certificate does not federate. The swarm root + can't simply be listed there: `security.pki.certificateFiles` is + read when the system is *built*, and the root is a runtime file (its + key must never enter the store), so there is no build-time name for + it. Bridging that needs a runtime mechanism and is tracked as its own + issue. Until then, federation needs CA-issued certs (ACME). See + `docs/matrix.md` for federation firewall + TLS requirements. -Each hive must declare the other. If hive A lists hive B as a peer, -B must also list A for agents on B to see A in their peer list: +## One directory, not a bilateral declaration + +Both hives hold the **same** `hives` attrset; neither declares the +other. What differs between the two hosts is only `hiveName`: ``` -# hive A (pr1ma.example.com) -services.hyperhive.swarm.peers."edge.corp" = { }; - -# hive B (edge.corp) -services.hyperhive.swarm.peers."pr1ma.example.com" = { certFingerprint = "sha256:…"; }; +# hive A # hive B +hiveName = "pr1ma"; hiveName = "edge"; +swarm.hives = { … }; swarm.hives = { … }; # byte-identical ``` -Mixed trust is fine: A trusts B via CA bundle (no fingerprint), B -pins A's self-signed cert. +That is the point of the shape, and it removes a class of bug rather +than saving typing: a per-host peer list let two hosts hold *different* +facts about the same third hive — a stale endpoint, a rotated +fingerprint — with nothing to detect the disagreement. One entry per +hive makes it unrepresentable. ## WireGuard inter-hive mesh (optional) @@ -213,11 +222,22 @@ services.hyperhive = { listenPort = 51820; # optional, default 51820 }; - swarm.peers."edge.corp" = { - certFingerprint = "sha256:…"; # TLS trust (unchanged) - wireguardPublicKey = "base64key="; # peer's wg pubkey - wireguardEndpoint = "203.0.113.42:51820"; # peer's public IP:port - wireguardAddress = "10.100.0.2/32"; # peer's mesh IP + # The same `hives` attrset both hosts hold — mesh fields included, + # since "where this hive can be dialled" is a fact about that hive. + swarm.hives = { + pr1ma = { + domain = "pr1ma.example.com"; + wireguardPublicKey = "base64keyA="; + wireguardEndpoint = "198.51.100.1:51820"; + wireguardAddress = "10.100.0.1/32"; + }; + edge = { + domain = "edge.corp"; + certFingerprint = "sha256:…"; # TLS trust (unchanged) + wireguardPublicKey = "base64keyB="; + wireguardEndpoint = "203.0.113.42:51820"; + wireguardAddress = "10.100.0.2/32"; + }; }; }; @@ -229,11 +249,7 @@ services.hyperhive = { address = "10.100.0.2/24"; }; - swarm.peers."pr1ma.example.com" = { - wireguardPublicKey = "base64key="; # hive A's wg pubkey - wireguardEndpoint = "198.51.100.1:51820"; - wireguardAddress = "10.100.0.1/32"; - }; + swarm.hives = { /* … identical to hive A's … */ }; }; ``` diff --git a/docs/swarm/ca.md b/docs/swarm/ca.md index 9d176946..750a16de 100644 --- a/docs/swarm/ca.md +++ b/docs/swarm/ca.md @@ -126,3 +126,35 @@ named in the message. That is a decision, and it is recorded as one. A hive with **no** root configured at all is not affected by any of this: it self-signs exactly as it always has. + +## Distributing the root + +The root **key** is a runtime file for an obvious reason: the nix store +is world-readable and content-addressed, so a key committed to a flake +is a key *published* to every consumer of that flake. + +The root **certificate** is a runtime file as a *consequence* — it lives +beside the key under `swarm.ca.stateDir` — and that has a cost worth +naming, because it is not obvious and it bites at a distance: + +> **Nothing whose trust store is assembled at build time can reference +> the swarm root.** `security.pki.certificateFiles` is read inside the +> derivation; the root does not exist there. + +Two consumers, and only one of them is fine: + +- **Agents are covered.** `hive-tls.nix` folds the root into this hive's + `trust-bundle.pem`, hive-c0re receives that path as + `HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one file + next to every agent's flake. The bundle is the runtime-to-build-time + bridge. +- **The Matrix container is not.** It has no equivalent bridge, so it + trusts no swarm-internal CA and federation with a self-signed peer + does not validate. Giving it the root needs a runtime mechanism — + bind-mount plus a bundle assembled at unit start, appending to the + system bundle rather than replacing it — which is tracked separately. + +To put the root on another host, copy the certificate to the same path +there (`scp /root.pem :/root.pem`). One anchor +per host, not one per peer: a hive joining later needs no edit on the +hives already running, which is the whole point of the hierarchy. diff --git a/docs/tools/hivectl-cli.md b/docs/tools/hivectl-cli.md index a3bab285..fc8333c4 100644 --- a/docs/tools/hivectl-cli.md +++ b/docs/tools/hivectl-cli.md @@ -761,7 +761,7 @@ Print the nix to add a peer hive to the mesh ###### **Arguments:** -* `` — Peer hive's DNS domain (the `swarm.peers` attrset key) +* `` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`) ###### **Options:** diff --git a/docs/web-ui/dashboard.md b/docs/web-ui/dashboard.md index 84c3ad33..bc38230b 100644 --- a/docs/web-ui/dashboard.md +++ b/docs/web-ui/dashboard.md @@ -607,10 +607,10 @@ prices. Models not covered fall back to hive-c0re's built-in estimate. ## P33RS tab Peer hives in this swarm. The tab is hidden when the -`state.peer_hives` array from `/api/state` is empty (i.e. no -`services.hyperhive.swarm.peers` are configured). When at least -one peer is present the `hidden` attribute is removed and the tab -becomes active. +`state.peer_hives` array from `/api/state` is empty — i.e. when +`services.hyperhive.swarm.hives` holds no hive other than this one. +When at least one peer is present the `hidden` attribute is removed +and the tab becomes active. **P33R H1V3S** — each peer renders as a card row: a hexagon icon (`⬡`), the peer's DNS domain as the primary name, and the peer @@ -619,15 +619,16 @@ URL opens the peer hive's dashboard in a new tab. ### Backend wiring -The host daemon reads `services.hyperhive.swarm.peers` from the -nix config (an attrset keyed by peer domain), serialises each -entry as `{ name, url }` into `state.peer_hives: Vec`, +The host daemon reads `services.hyperhive.swarm.peerHives` from the +nix config (the `swarm.hives` directory minus this hive), serialises +each entry as `{ name, url }` into `state.peer_hives: Vec`, and includes the field in the `/api/state` snapshot. `tabs.js` reads `state.peer_hives` on every `refreshState` call and calls `swarm.js::renderPeerHives(peers)`, which rebuilds the `#peers-section` div from scratch. -The `name` field is the peer's DNS domain (the attrset key); `url` +The `name` field is the peer's DNS domain (its entry's `domain`, not +the attrset key — the key is the hive's name); `url` is `https://{domain}/`. Both are derived from the env var `HYPERHIVE_PEERS` (a JSON array of `{ domain, cert_fingerprint }` objects) that the nix module writes into the c0re container diff --git a/hive-c0re/src/dashboard/state_snapshot.rs b/hive-c0re/src/dashboard/state_snapshot.rs index 69cc274c..d98aafbf 100644 --- a/hive-c0re/src/dashboard/state_snapshot.rs +++ b/hive-c0re/src/dashboard/state_snapshot.rs @@ -122,7 +122,8 @@ pub(super) struct StateSnapshot { swarm_name: Option, /// Peer hives in the same swarm. Parsed from `HYPERHIVE_PEERS` /// (JSON array of `{domain,cert_fingerprint}` objects, emitted by - /// the c0re NixOS module from `services.hyperhive.swarm.peers`). + /// the c0re NixOS module from `services.hyperhive.swarm.peerHives` + /// — the swarm's `hives` directory minus this hive). /// Empty on single-hive deploys. Feeds the P33RS dashboard tab. peer_hives: Vec, /// Server-level warnings for the dashboard's top-of-page banner @@ -162,9 +163,9 @@ async fn infra_container_views() -> Vec { /// One peer hive for the P33RS dashboard tab. Derived from /// `HYPERHIVE_PEERS` env; `url` is the peer's HTTPS dashboard root. -/// `cert_fingerprint` is `Some("sha256:")` when the peer uses a -/// self-signed cert and the operator pinned its fingerprint in -/// `services.hyperhive.swarm.peers`. +/// `cert_fingerprint` is `Some("sha256:")` when the operator +/// pinned the peer's leaf in `services.hyperhive.swarm.hives`, which a +/// hive under the swarm root CA does not need. #[derive(Serialize)] struct PeerHiveView { name: String, @@ -450,7 +451,7 @@ pub(super) async fn api_state( /// Parse `HYPERHIVE_PEERS` env var into dashboard-ready `PeerHiveView` /// entries. The env var is a JSON array of `{domain, cert_fingerprint}` /// objects emitted by the c0re NixOS module from -/// `services.hyperhive.swarm.peers`. Each entry becomes +/// `services.hyperhive.swarm.peerHives`. Each entry becomes /// `{ name: domain, url: "https://domain/" }` for the P33RS tab. /// Returns empty vec when unset (single-hive deploy). fn parse_peer_hives() -> Vec { diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index 4e7856b3..ae2909cd 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -108,11 +108,10 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> { let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default(); let initial = !dir.join(".git").exists(); - // Embedded-CA list (self-signed hive CA + peer CAs): keep the - // `./hive-ca.pem` / `./peer-ca-.pem` files at the meta root in - // lockstep with their host sources so the build-time `certificateFiles` - // list render_flake emits always resolves. Empty when neither a - // self-signed hive CA nor any peer CA is configured. + // Embedded-CA list: keep the `./hive-ca.pem` file at the meta root in + // lockstep with its host source so the build-time `certificateFiles` + // list render_flake emits always resolves. Empty when no self-signed + // hive CA is configured. let (ca_files, ca_changed) = ca_embed_state(&dir); // Skip only when both the flake AND the embedded CA are unchanged — a @@ -217,6 +216,7 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> { "flake.nix" => Some("flake"), "flake.lock" => Some("lock"), "hive-ca.pem" => Some("hive-ca"), + // Only ever a REMOVAL now; see `is_embedded_ca_name`. f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"), "topology.json" => Some("topology"), "capabilities.json" => Some("capabilities"), @@ -809,10 +809,10 @@ fn forwarded_env_vars() -> Vec<(&'static str, String)> { .collect() } -/// Filename the hive's own self-signed CA cert is embedded under at the -/// meta-flake root. One entry of the embedded-CA list `render_flake` -/// references in `security.pki.certificateFiles` (see `embedded_ca_files`); -/// peer CAs sit alongside it as `peer-ca-.pem`. +/// Filename the hive's own trust anchors are embedded under at the +/// meta-flake root — the only entry of the embedded-CA list +/// `render_flake` references in `security.pki.certificateFiles` (see +/// `embedded_ca_files`). const HIVE_CA_FILE: &str = "hive-ca.pem"; /// Host path of the hive's TLS trust anchors, when self-signed TLS is @@ -833,23 +833,6 @@ fn hive_ca_source() -> Option { Some(path) } -/// Host paths of peer-hive root CA certificates, from `HIVE_PEER_CA_PATHS` -/// (colon-separated; set by hive-c0re.nix from `swarm.peers..caCert`). -/// Each is embedded alongside the hive CA so a peer's CA is trusted -/// everywhere the hive's own internal CA is — i.e. by every agent. Empty -/// segments and paths that don't resolve to a file are dropped, so we -/// never reference a `certificateFiles` entry we couldn't embed. -fn peer_ca_sources() -> Vec { - let Ok(raw) = std::env::var("HIVE_PEER_CA_PATHS") else { - return Vec::new(); - }; - raw.split(':') - .map(str::trim) - .filter(|p| !p.is_empty() && std::path::Path::new(p).is_file()) - .map(ToOwned::to_owned) - .collect() -} - /// Hive-wide OTEL config injected into every agent's build, read off /// hive-c0re's own unit env (set from `services.hyperhive.otel.*` in /// `nix/modules/hive-c0re.nix`). A present, non-empty @@ -906,9 +889,16 @@ fn otel_config() -> Option { } /// The ordered set of CA certs embedded next to the meta flake, as -/// `(filename, host_source_path)`. The self-signed hive CA (when active) -/// is `hive-ca.pem`; each peer CA is `peer-ca-.pem` in declaration -/// order. `render_flake` emits exactly these filenames into +/// `(filename, host_source_path)` — now just the hive's own trust +/// anchors as `hive-ca.pem`, when self-signed TLS is active. +/// +/// That single file already carries the swarm root (`hive-tls.nix` +/// writes the hive CA *and* the root it is issued under into the trust +/// bundle), so every hive under the swarm root validates from it. Which +/// is why the per-peer CAs this used to append are gone: they said the +/// same thing once per peer. +/// +/// `render_flake` emits exactly these filenames into /// `security.pki.certificateFiles` and `sync_agents` materialises them, /// so the rendered reference and the embedded files always agree. fn embedded_ca_files() -> Vec<(String, String)> { @@ -916,9 +906,6 @@ fn embedded_ca_files() -> Vec<(String, String)> { if let Some(p) = hive_ca_source() { out.push((HIVE_CA_FILE.to_owned(), p)); } - for (i, p) in peer_ca_sources().into_iter().enumerate() { - out.push((format!("peer-ca-{i}.pem"), p)); - } out } @@ -949,9 +936,16 @@ fn materialise_ca_files(dir: &Path, ca_files: &[(String, String)]) -> Result.pem`. Lets `sync_agents` find stale CA files to clean -/// up (a CA dropped from config) without touching unrelated meta files. +/// True for a filename the embedded-CA machinery owns. Lets +/// `sync_agents` find stale CA files to clean up (a CA dropped from +/// config) without touching unrelated meta files. +/// +/// ⚠️ Still matches `peer-ca-.pem`, which `embedded_ca_files` no +/// longer produces — deliberately. Cleanup is driven by +/// "recognised but not desired", so this arm is exactly what removes the +/// peer CAs a hive embedded before per-peer pinning was replaced by the +/// swarm root. Drop it and those files are orphaned at every meta root +/// forever, referenced by nothing and cleaned by nobody. fn is_embedded_ca_name(name: &str) -> bool { name == HIVE_CA_FILE || (name.starts_with("peer-ca-") && has_pem_ext(name)) } @@ -967,13 +961,12 @@ fn has_pem_ext(name: &str) -> bool { /// Embedded-CA state for the meta repo: `(desired_files, changed)`. /// `desired_files` is `(filename, contents)` for every CA that should sit -/// next to flake.nix (the hive CA + each peer CA). `changed` is true when -/// the on-disk set differs in any way — a file's contents changed, a new -/// CA appeared, or a previously-embedded CA (`hive-ca.pem` / -/// `peer-ca-*.pem`) is no longer wanted (stale, to be removed). Drives -/// both the re-commit decision and the materialise/cleanup in -/// `sync_agents`, so a CA rotation or a peer-set change re-commits even -/// when the flake itself is byte-identical. +/// next to flake.nix. `changed` is true when the on-disk set differs in +/// any way — the file's contents changed, a CA appeared, or a +/// previously-embedded CA (`hive-ca.pem` / a legacy `peer-ca-*.pem`) is +/// no longer wanted (stale, to be removed). Drives both the re-commit +/// decision and the materialise/cleanup in `sync_agents`, so a CA +/// rotation re-commits even when the flake itself is byte-identical. fn ca_embed_state(dir: &std::path::Path) -> (Vec<(String, String)>, bool) { let desired: Vec<(String, String)> = embedded_ca_files() .into_iter() @@ -1170,16 +1163,16 @@ where if let Some(path) = claude_code_path { let _ = writeln!(out, " hyperhive.claudeCodePath = \"{path}\";"); } - // CA trust: embed every hive-trusted CA so each agent validates them at - // build time. The list is the hive's own self-signed CA (when active) - // plus every peer-hive root CA (`swarm.peers..caCert`) — a peer CA is - // trusted everywhere the hive's own internal CA is. `certificateFiles` is - // build-time, so the certs travel with the flake source: `sync_agents` - // writes `./hive-ca.pem` + `./peer-ca-.pem` next to flake.nix and - // stages them. Only public CA certs are embedded; no private key ever - // leaves the host. The filename list matches `sync_agents` exactly (both - // derive it from `embedded_ca_files`), so we never reference a file we - // didn't embed; emitted only when the list is non-empty. + // CA trust: embed the hive's trust anchors so each agent validates + // them at build time — the hive's own self-signed CA when active, + // together with the swarm root it is issued under (one file; see + // `hive_ca_source`). `certificateFiles` is build-time, so the certs + // travel with the flake source: `sync_agents` writes `./hive-ca.pem` + // next to flake.nix and stages it. Only public CA certs are embedded; + // no private key ever leaves the host. The filename list matches + // `sync_agents` exactly (both derive it from `embedded_ca_files`), so + // we never reference a file we didn't embed; emitted only when the + // list is non-empty. let ca_refs: Vec = embedded_ca_files() .into_iter() .map(|(name, _)| format!("./{name}")) @@ -2169,20 +2162,16 @@ mod tests { ) }; - // Two peer-hive CA temp files for the list cases. - let peer0 = std::env::temp_dir().join(format!("peer-ca0-test-{}.pem", std::process::id())); - let peer1 = std::env::temp_dir().join(format!("peer-ca1-test-{}.pem", std::process::id())); + // A leftover peer-CA file + the env var that used to name it. Both + // must now be inert: the swarm root rides in the hive's own trust + // bundle, so nothing per-peer is embedded any more. + let stale_peer = + std::env::temp_dir().join(format!("peer-ca0-test-{}.pem", std::process::id())); std::fs::write( - &peer0, + &stale_peer, "-----BEGIN CERTIFICATE-----\np0\n-----END CERTIFICATE-----\n", ) - .expect("write peer CA 0"); - std::fs::write( - &peer1, - "-----BEGIN CERTIFICATE-----\np1\n-----END CERTIFICATE-----\n", - ) - .expect("write peer CA 1"); - let peer_paths = format!("{}:{}", peer0.display(), peer1.display()); + .expect("write stale peer CA"); // All env mutations are serialised within this one test (no other // test asserts on these vars), restored before returning. @@ -2191,38 +2180,47 @@ mod tests { std::env::set_var("HIVE_TLS_CA_PATH", &ca_file); } let with_ca = render(); - // Hive CA + peer CAs: the list carries all three, hive CA first. unsafe { - std::env::set_var("HIVE_PEER_CA_PATHS", &peer_paths); + std::env::set_var("HIVE_PEER_CA_PATHS", stale_peer.display().to_string()); } - let with_peers = render(); - // Peers only (this hive on ACME, federating with self-signed peers). + let with_stale_peer_env = render(); + // The stale var alone, with no hive CA: must produce nothing. unsafe { std::env::remove_var("HIVE_TLS_CA_PATH"); } - let peers_only = render(); + let stale_peer_env_only = render(); unsafe { std::env::remove_var("HIVE_PEER_CA_PATHS"); } let without_ca = render(); let _ = std::fs::remove_file(&ca_file); - let _ = std::fs::remove_file(&peer0); - let _ = std::fs::remove_file(&peer1); + let _ = std::fs::remove_file(&stale_peer); assert!( with_ca.contains("security.pki.certificateFiles = [ ./hive-ca.pem ]"), "CA cert must be wired into certificateFiles when signalled:\n{with_ca}" ); + // The regression guard, scoped to the CA list rather than the whole + // render. Comparing the two flakes wholesale looks stronger and is + // actually FLAKY: `cargo test` runs these in parallel threads and + // sibling tests mutate process env (OTEL, forge URLs) between the + // two `render()` calls, so a whole-output equality assertion fails + // on changes that have nothing to do with this test. assert!( - with_peers.contains( - "security.pki.certificateFiles = [ ./hive-ca.pem ./peer-ca-0.pem ./peer-ca-1.pem ]" - ), - "hive CA + peer CAs must all appear in the certificateFiles list:\n{with_peers}" + with_stale_peer_env.contains("security.pki.certificateFiles = [ ./hive-ca.pem ]"), + "HIVE_PEER_CA_PATHS must not change the CA list — per-peer CA \ + embedding was replaced by the swarm root inside the hive's own \ + trust bundle:\n{with_stale_peer_env}" ); assert!( - peers_only - .contains("security.pki.certificateFiles = [ ./peer-ca-0.pem ./peer-ca-1.pem ]"), - "peer CAs must be trusted even when this hive has no self-signed CA:\n{peers_only}" + !with_stale_peer_env.contains("peer-ca-"), + "no peer-ca-.pem may be embedded, however HIVE_PEER_CA_PATHS \ + is set:\n{with_stale_peer_env}" + ); + assert!( + !stale_peer_env_only.contains("security.pki.certificateFiles"), + "a stale HIVE_PEER_CA_PATHS must not resurrect a certificateFiles \ + reference on its own:\n{stale_peer_env_only}" ); assert!( !without_ca.contains("security.pki.certificateFiles"), diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 81687317..5e87597c 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -1030,7 +1030,7 @@ fn is_broad_scope(scope: &LifecycleScope) -> bool { /// convention (gateway terminates TLS, so https). fn hive_urls() -> hive_host_sock::HiveUrls { // Treat an empty env value as unset everywhere — an empty domain would - // otherwise render `swarm.peers."" = …` (invalid nix) and `https:///`. + // otherwise render `swarm.hives."" = …` (invalid nix) and `https:///`. let env = |k: &str| std::env::var(k).ok().filter(|v| !v.is_empty()); let domain = env("HYPERHIVE_HIVE_DOMAIN"); hive_host_sock::HiveUrls { diff --git a/hivectl/src/cli.rs b/hivectl/src/cli.rs index 9f0761dc..6a32d3ad 100644 --- a/hivectl/src/cli.rs +++ b/hivectl/src/cli.rs @@ -425,7 +425,7 @@ pub enum WgCmd { }, /// Print the nix to add a peer hive to the mesh. Peer { - /// Peer hive's DNS domain (the `swarm.peers` attrset key). + /// Peer hive's DNS domain (its `swarm.hives` entry's `domain`) domain: String, /// Peer's WireGuard public key (from its `hivectl wg init`). #[arg(long)] diff --git a/hivectl/src/wg.rs b/hivectl/src/wg.rs index 3a715fa1..933f47f3 100644 --- a/hivectl/src/wg.rs +++ b/hivectl/src/wg.rs @@ -16,9 +16,10 @@ const WG_INTERFACE: &str = "wg-hive"; /// Host path of this hive's TLS trust bundle (matches the /// `services.hyperhive.tls.stateDir` default in hive-tls.nix). Its -/// existence means the gateway serves a self-signed, hive-CA-signed leaf, -/// so a federating peer needs this via `swarm.peers..caCert`. Absent -/// = ACME / operator cert (trusted by the default CA bundle, no `caCert`). +/// existence means the gateway serves a self-signed, hive-CA-signed +/// leaf, so a federating peer needs an anchor for it — which is now the +/// swarm root, not a per-hive CA. Absent = ACME / operator cert, trusted +/// by the default CA bundle with nothing to distribute. /// /// The bundle rather than `ca.pem`: the hive CA is an intermediate under /// the swarm root, so `ca.pem` alone is not a chain a peer can validate @@ -28,6 +29,26 @@ const WG_INTERFACE: &str = "wg-hive"; /// path with no mode to branch on. const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem"; +/// Host path of the swarm root CA cert (matches the +/// `services.hyperhive.swarm.ca.stateDir` default in swarm-ca.nix). The +/// anchor a whole swarm shares: install it once per host and every +/// present *and future* hive under it validates, which is what replaced +/// the per-hive CA pinning. +const SWARM_CA_ROOT_PATH: &str = "/var/lib/swarm-ca/root.pem"; + +/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived +/// from its domain's first DNS label. +/// +/// A hive occupies `.`, so the first label *is* +/// the hive name in any deployment that hasn't overridden `domain` by +/// hand. Where it has, a wrong key fails loudly rather than quietly: on +/// that hive's own host the `hives.` assertion fires, because +/// the directory is supposed to be the same attrset everywhere. The +/// snippets below say so rather than presenting the guess as fact. +fn hive_key(domain: &str) -> &str { + domain.split('.').next().unwrap_or(domain) +} + /// Best-effort query for this hive's domain from the running daemon /// (`HostRequest::Urls`, which reads `HYPERHIVE_HIVE_DOMAIN` from c0re's /// service env). `None` when the daemon is unreachable or the domain is @@ -143,25 +164,29 @@ fn wg_pubkey(privkey: &[u8]) -> Result { /// Pure output (no fallible work), so it returns `()`; the dispatch arm /// wraps it in `Ok` to match the sibling verbs. pub(crate) fn wg_peer(domain: &str, pubkey: &str, address: &str, endpoint: Option<&str>) { + let key = hive_key(domain); println!("Add to this hive's NixOS config:"); - println!(" services.hyperhive.swarm.peers.\"{domain}\" = {{"); + println!(" # `hives` describes the whole swarm and is meant to be the same"); + println!(" # attrset on every host — add this entry to all of them."); + println!(" services.hyperhive.swarm.hives.\"{key}\" = {{"); + println!(" domain = \"{domain}\";"); println!(" wireguardPublicKey = \"{pubkey}\";"); println!(" wireguardAddress = \"{address}\";"); if let Some(ep) = endpoint { println!(" wireguardEndpoint = \"{ep}\";"); } println!(" }};"); + println!(" # the key must be that hive's services.hyperhive.hiveName"); } -/// `peer-config` — print the `swarm.peers.""` block a peer -/// operator pastes to federate with THIS hive, plus a `cp` line for the -/// CA when this hive is self-signed. Reads local state only (the TLS CA -/// cert presence + the wg key); prints, never mutates. +/// `peer-config` — print the `swarm.hives.""` block a peer +/// operator pastes to federate with THIS hive, plus the swarm-root +/// install step when this hive serves a self-signed chain. Reads local +/// state only (the TLS trust bundle's presence + the wg key); prints, +/// never mutates. pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: Option<&str>) { let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists(); - // CA filename derived from the first DNS label so multiple peers' - // certs don't collide in the operator's config dir. - let ca_file = format!("{}-ca.pem", domain.split('.').next().unwrap_or("peer")); + let key = hive_key(domain); // WireGuard public key, when this hive has a mesh key. Best-effort: // a missing key or absent `wg` binary just omits the mesh lines. let wg_pub = std::fs::read(WG_KEY_PATH) @@ -169,17 +194,19 @@ pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: O .and_then(|k| wg_pubkey(&k).ok()); if self_signed { - println!("# 1. copy this hive's CA cert next to the peer's config:"); - println!("cp {HIVE_TLS_TRUST_BUNDLE_PATH} ./{ca_file}"); + // One anchor for the whole swarm, installed once per host — not + // a file per peer. That is the point of the hierarchy: a hive + // joining later needs no edit on the hives already running. + println!("# 1. install the SWARM ROOT on the peer host (once, not per hive):"); + println!("scp {SWARM_CA_ROOT_PATH} :{SWARM_CA_ROOT_PATH}"); + println!("# (skip if that host already has the swarm root)"); println!(); println!("# 2. paste into the peer hive's NixOS config:"); } else { println!("# paste into the peer hive's NixOS config:"); } - println!("services.hyperhive.swarm.peers.\"{domain}\" = {{"); - if self_signed { - println!(" caCert = ./{ca_file};"); - } + println!("services.hyperhive.swarm.hives.\"{key}\" = {{"); + println!(" domain = \"{domain}\";"); if let Some(pk) = &wg_pub { println!(" wireguardPublicKey = \"{pk}\";"); } @@ -190,9 +217,11 @@ pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: O println!(" wireguardEndpoint = \"{ep}\";"); } println!("}};"); + println!("# the key must be this hive's services.hyperhive.hiveName, and the"); + println!("# same entry belongs in every hive's config — `hives` is the swarm."); if !self_signed { println!( - "# (this hive's cert chains to a public CA — no `caCert` needed; \ + "# (this hive's cert chains to a public CA — nothing to install; \ it's trusted by the default bundle.)" ); } diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index e303ac84..56f21137 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -28,6 +28,7 @@ ./swarm-snapshot-store.nix ./swarm-wireguard.nix ./swarm.nix + ./swarm-peers-removed.nix ./swarm-required-services.nix ]; } diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 5d4caad5..66973cdb 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -187,41 +187,25 @@ in in "${s.address}:${toString s.port}"; } -// lib.optionalAttrs (config.services.hyperhive.swarm.peers != { }) { +// lib.optionalAttrs (config.services.hyperhive.swarm.peerHives != { }) { # Peer hives serialised as a JSON array of {domain, cert_fingerprint, # wireguard_address?} objects. Consumed by hive-agent::identity::peers() - # + the dashboard's peer_hives StateSnapshot field (P33RS tab). Domain - # is the attrset key; cert_fingerprint is null for CA-trusted peers; - # wireguard_address is omitted when not part of the mesh. + # + the dashboard's peer_hives StateSnapshot field (P33RS tab). + # `cert_fingerprint` is null for CA-trusted hives; `wireguard_address` + # is omitted when not part of the mesh. + # + # Reads `peerHives` — `swarm.hives` minus this hive — so the "not me" + # filter is the one derived in ../swarm.nix rather than a fifth copy. HYPERHIVE_PEERS = builtins.toJSON ( lib.mapAttrsToList ( - domain: p: + _name: p: { - inherit domain; + inherit (p) domain; cert_fingerprint = p.certFingerprint; } // lib.optionalAttrs (p.wireguardAddress != null) { wireguard_address = p.wireguardAddress; } - ) config.services.hyperhive.swarm.peers + ) config.services.hyperhive.swarm.peerHives ); } -// - lib.optionalAttrs - (lib.any (p: p.caCert != null) (lib.attrValues config.services.hyperhive.swarm.peers)) - { - # Peer-hive root CA file paths (colon-joined), one per peer that - # declares `swarm.peers..caCert`. hive-c0re's meta-flake - # renderer (meta.rs) embeds each next to every agent's flake and - # adds it to `security.pki.certificateFiles`, so a peer CA is - # trusted everywhere the hive's own internal CA (`hive-ca.pem`) - # is — i.e. by every agent. The matrix container trusts the same - # CAs separately for federation TLS. The `caCert` files are - # copied into the nix store at build, so these are store paths — - # nothing mutable lives on the host. - HIVE_PEER_CA_PATHS = lib.concatStringsSep ":" ( - lib.filter (c: c != null) ( - lib.mapAttrsToList (_domain: p: p.caCert) config.services.hyperhive.swarm.peers - ) - ); - } diff --git a/nix/host-modules/hive-gateway/options.nix b/nix/host-modules/hive-gateway/options.nix index 9f3ed563..41e30a77 100644 --- a/nix/host-modules/hive-gateway/options.nix +++ b/nix/host-modules/hive-gateway/options.nix @@ -147,8 +147,8 @@ in config.security.acme.certs."example.com".directory; ``` - When using an external CA cert, peer hives can declare this - hive in `services.hyperhive.swarm.peers` without + When using an external CA cert, other hives can declare this + one in `services.hyperhive.swarm.hives` without `certFingerprint` — the standard CA bundle validates. Mutual exclusion with `tls.acme.enable` — set one or the other, @@ -207,8 +207,8 @@ in }; ``` - After enabling, peer hives can omit `certFingerprint` in - `swarm.peers` — Let's Encrypt certs are CA-trusted + After enabling, this hive's entry in `swarm.hives` can omit + `certFingerprint` — Let's Encrypt certs are CA-trusted by default. ''; }; diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index 6e98114a..96aca15e 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -395,20 +395,20 @@ in # all filtering; never run one in here. networking.firewall.enable = false; - # Peer-hive root CAs (`swarm.peers..caCert`) added to THIS - # container's trust bundle so tuwunel validates *federation* TLS - # from a self-signed peer hive (it checks the peer's federation - # cert against its trust bundle). Peer CAs are trusted everywhere - # the hive's own internal CA is — agents get them via the - # meta-flake renderer (`HIVE_PEER_CA_PATHS` → each agent's - # `security.pki.certificateFiles`); this block is the matrix - # container's copy, since the host `security.pki` store doesn't - # cross the container boundary. They are never installed in the - # HOST trust store. Null entries (CA-bundle / fingerprint-pinned - # peers) drop out. - security.pki.certificateFiles = lib.filter (c: c != null) ( - lib.mapAttrsToList (_domain: p: p.caCert) config.services.hyperhive.swarm.peers - ); + # ⚠️ This container trusts no swarm-internal CA. It used to take + # per-hive root CAs, hand-pinned as nix paths, so tuwunel could + # validate *federation* TLS from a self-signed peer hive; that + # field is gone, and the swarm root that replaces it cannot be + # substituted here. `security.pki.certificateFiles` is read when + # the system is BUILT, and the swarm root is a runtime file + # (`swarm.ca.stateDir`) precisely because its key must never + # reach the store — so there is nothing build-time to name. + # + # Giving the container the swarm root therefore needs a runtime + # mechanism (bind-mount + a bundle assembled at start), which is + # a different shape than this line and is tracked as its own + # issue. Federation with a peer whose cert chains to the swarm + # root does not validate until then. # tuwunel hard-fails to boot if `/etc/resolv.conf` has no # `nameserver` line (`Failed to configure DNS resolver ... no diff --git a/nix/host-modules/swarm-ca.nix b/nix/host-modules/swarm-ca.nix index d0ae1dc7..08341269 100644 --- a/nix/host-modules/swarm-ca.nix +++ b/nix/host-modules/swarm-ca.nix @@ -1,17 +1,19 @@ # The swarm root CA: the anchor a whole swarm shares, and the issuer of # each hive's own CA (which is where it gets used — see ./hive-tls.nix). # -# Why a hierarchy at all: cross-hive trust is hand-pinned today -# (`swarm.peers..caCert`), so every hive must name every peer — O(n²) -# configuration that a new hive can only join by editing all the others. -# One root makes it O(1): trust the root once and every present *and -# future* peer validates. +# Why a hierarchy at all: cross-hive trust used to be hand-pinned per +# peer, so every hive had to name every other one — O(n²) configuration +# that a new hive could only join by editing all the others. One root +# makes it O(1): trust the root once and every present *and future* peer +# validates. The per-peer CA field is gone precisely because this +# replaced it. # # Two provisioning modes, ONE structure — what differs is who puts the # artifacts on disk, never what the artifacts are: # -# - autoconfigured (the default while the hive declares no peers): the -# unit below generates the root here on first boot. +# - autoconfigured (`services.hyperhive.enableAllLocalDefaults`, or +# `autoConfigure` set directly): the unit below generates the root +# here on first boot. # - operator-provided (multi-host): the operator installs the root cert # — and, on a host that does not hold the root key, the hive CA too — # into the state dirs, and this unit does nothing. Splitting a @@ -21,10 +23,11 @@ # # The root KEY is the reason this is a runtime file and not a nix option: # the store is world-readable and content-addressed, so a key committed -# to a flake is a key *published* to every consumer of that flake. The -# root CERT has the opposite property, and is already distributed at -# build time via `swarm.peers..caCert`. That asymmetry is what makes -# the layering work at all. +# to a flake is a key *published* to every consumer of that flake. +# +# ⚠️ The root CERT is therefore a runtime file too, which costs something: +# nothing whose trust store is built at build time can name it. See +# docs/swarm/ca.md § "Distributing the root". { lib, config, @@ -83,8 +86,8 @@ in default = "/var/lib/swarm-ca"; description = '' Host directory holding the swarm root CA: `root.pem` (the - anchor, safe to distribute — this is what a peer hive is - pointed at via `swarm.peers..caCert`) and `root-key.pem` + anchor, safe to distribute — copy it to this same path on every + other host in the swarm) and `root-key.pem` (0600, the one file that must never reach the nix store or another host). The directory itself is 0700: nothing reads out of it but the hive CA issuance in `hive-tls.nix`. diff --git a/nix/host-modules/swarm-peers-removed.nix b/nix/host-modules/swarm-peers-removed.nix new file mode 100644 index 00000000..b4866ad2 --- /dev/null +++ b/nix/host-modules/swarm-peers-removed.nix @@ -0,0 +1,70 @@ +# Migration shim for `services.hyperhive.swarm.peers`, replaced by the +# `swarm.hives` directory in ./swarm.nix. +# +# ⚠️ This could not be a `mkRenamedOptionModule`. `hives` is not `peers` +# under a new name, it is **`peers` ∪ {self}**: the set gains a member no +# existing config has ever written down, because a host's own identity +# lived in *other* options entirely. A rename migrates a name and a +# default can re-root a meaning; neither can conjure a new member. +# +# The whole module is self-contained and deletable — one file to remove +# when the deprecation window closes, with nothing else referring to it. +# +# Deliberately a warning rather than a hard failure, with the loudness +# coming from elsewhere: a config that set only `peers` leaves `hives` +# empty, so the self-entry assertion in ./swarm.nix fails the build +# anyway. The warning is what explains it; the assertion is what stops +# it. +{ + lib, + config, + ... +}: +let + peers = config.services.hyperhive.swarm.peers; + withCaCert = lib.attrNames (lib.filterAttrs (_: p: p ? caCert && p.caCert != null) peers); +in +{ + options.services.hyperhive.swarm.peers = lib.mkOption { + type = lib.types.attrsOf lib.types.anything; + default = { }; + visible = false; + internal = true; + description = '' + Removed — use `services.hyperhive.swarm.hives` instead, which + describes every hive in the swarm including this one. Kept only so + an existing definition produces a warning that says where to move + it, rather than an "option does not exist" error that says nothing. + ''; + }; + + config.warnings = + lib.optional (peers != { }) '' + services.hyperhive.swarm.peers is removed and ignored. Move these + entries to services.hyperhive.swarm.hives, keyed by hive NAME + rather than domain, and add an entry for this hive itself + (services.hyperhive.hiveName) — `hives` is the swarm's full + directory, identical on every host. + + services.hyperhive.swarm.hives. = { + domain = ""; + # certFingerprint / wireguard* carry over unchanged + }; + + Still set: ${lib.concatStringsSep ", " (lib.attrNames peers)} + '' + ++ lib.optional (withCaCert != [ ]) '' + services.hyperhive.swarm.peers..caCert is removed and + ignored, and has no replacement in services.hyperhive.swarm.hives. + + Trust inside a swarm now derives from the swarm root CA + (services.hyperhive.swarm.ca — see docs/swarm/ca.md): every hive + under it chains to it, so a per-hive CA is dead weight. What this + genuinely drops is trusting a hive whose root this swarm does NOT + own — another swarm's, or one keeping its own CA. certFingerprint + does not cover that: it pins a leaf for hive-c0re's own HTTPS + checks and does not reach Matrix federation. + + Still set on: ${lib.concatStringsSep ", " withCaCert} + ''; +} diff --git a/nix/host-modules/swarm-wireguard.nix b/nix/host-modules/swarm-wireguard.nix index 4433de13..1f393131 100644 --- a/nix/host-modules/swarm-wireguard.nix +++ b/nix/host-modules/swarm-wireguard.nix @@ -108,9 +108,12 @@ networking.wireguard.interfaces = let wgCfg = config.services.hyperhive.swarm.wireguard; + # `peerHives` is `swarm.hives` minus this hive (../swarm.nix) — + # a mesh that included our own entry would configure a tunnel to + # ourselves. meshPeers = lib.filterAttrs ( _: p: p.wireguardPublicKey != null && p.wireguardAddress != null - ) config.services.hyperhive.swarm.peers; + ) config.services.hyperhive.swarm.peerHives; in { wg-hive = { @@ -118,7 +121,7 @@ listenPort = wgCfg.listenPort; privateKeyFile = wgCfg.privateKeyFile; peers = lib.mapAttrsToList ( - _domain: p: + _name: p: { publicKey = p.wireguardPublicKey; allowedIPs = [ p.wireguardAddress ]; diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 31b8c7a1..41add894 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -1,68 +1,73 @@ -# Swarm peering: who the peer hives are. Serialised into hive-c0re's -# environment (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS — see ./hive-c0re) -# and consumed by identity.rs + the dashboard's P33RS tab. +# The swarm's directory: one entry per hive, **including this one**, +# identical on every host in the swarm. `services.hyperhive.hiveName` +# says which entry is us, and `peerHives` below derives the rest. # -# Declaration only — this module has no `config` block. The mesh that -# uses the `wireguard*` fields below lives in ./swarm-wireguard.nix, -# because bringing up an interface is host networking rather than swarm +# Why a directory rather than a per-host peer list: every field here is +# intrinsic to the hive it describes — none of them says anything about +# the *pair*. A list where every field is intrinsic is a directory each +# host was keeping its own copy of, which is O(n²) duplication that +# deduplicates without loss. It is also a correctness gain: two hosts +# could hold different endpoints for the same third hive and nothing +# detected it. One entry per hive makes that unrepresentable. +# +# Consumed by hive-c0re's environment (HYPERHIVE_PEERS — see +# ./hive-c0re), identity.rs + the dashboard's P33RS tab, and the mesh in +# ./swarm-wireguard.nix. The mesh lives there rather than here because +# bringing up an interface is host networking rather than swarm # bookkeeping, and a host that runs no hive still needs it. { lib, config, ... }: +let + cfg = config.services.hyperhive; + swarmCfg = cfg.swarm; +in { - # Peer hives in the same swarm. Each entry declares a remote hive - # reachable from this host. - options.services.hyperhive.swarm.peers = lib.mkOption { + options.services.hyperhive.swarm.hives = lib.mkOption { type = lib.types.attrsOf ( lib.types.submodule { options = { + domain = lib.mkOption { + type = lib.types.str; + example = "lab.example.com"; + description = '' + Public DNS domain this hive occupies — used for dashboard + links, peer HTTPS checks and Matrix federation discovery. + + Deliberately has no default. It is conventionally + `.`, but defaulting to that would let + a typo'd swarm domain produce a name that resolves + somewhere real; an eval failure naming the hive is the + better outcome. + ''; + }; + certFingerprint = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; example = "sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12"; description = '' - Expected TLS certificate fingerprint for this peer's HTTPS - endpoint. Null = trust the system CA bundle (for Let's - Encrypt peers). Set to pin a self-signed cert. + Expected TLS certificate fingerprint for this hive's HTTPS + endpoint. Null = trust the CA bundle — which for a hive + inside the swarm CA hierarchy is the normal case, since + every hive under the swarm root already chains to it. + Set it to pin a leaf that no CA in the bundle vouches for. Format: the literal `sha256:` followed by exactly 64 hex digits (case-insensitive, no colon separators) — the - SHA-256 digest of the peer's DER-encoded leaf certificate. + SHA-256 digest of the DER-encoded leaf certificate. Generate with `openssl x509 -noout -fingerprint -sha256`, then strip the colons and prepend `sha256:`. A malformed value is ignored with a warning rather than weakening trust. See docs/swarm/README.md for the full recipe. Scopes only to hive-c0re's own peer HTTPS checks — it does - NOT help Matrix federation (tuwunel validates against its - container trust bundle). For a self-signed peer whose root - CA you want trusted hive-wide (every agent + Matrix - federation), set `caCert` below. - ''; - }; - - caCert = lib.mkOption { - type = lib.types.nullOr lib.types.path; - default = null; - example = "./peers/edge-ca.pem"; - description = '' - Path to this peer hive's root CA certificate (PEM). When - set, the CA is embedded (at build time, into the nix store - — no runtime file on the host) and trusted **everywhere the - hive's own internal CA is**: it rides alongside `hive-ca.pem` - in each agent's `security.pki.certificateFiles` (via the - meta-flake renderer), and is added to the Matrix homeserver - container's trust bundle so tuwunel validates *federation* - TLS from a self-signed peer hive whose cert chains to it. - This is the CA-trust path that `certFingerprint` - (leaf-pinning, c0re-only) can't cover, and is what unblocks - Matrix federation with a self-signed peer hive. Trust stays - inside the hive (agents + the Matrix container), never the - host system trust store. Mutually complementary with - `certFingerprint`; set `caCert` for the federation case. See - docs/swarm/README.md. + NOT help Matrix federation, which validates against the + container's trust bundle. There is no per-hive CA field to + cover that case any more: the swarm root is the trust path + (see ./swarm-ca.nix). ''; }; @@ -71,9 +76,9 @@ default = null; example = "base64pubkey="; description = '' - WireGuard public key for this peer host. Required when + WireGuard public key for this hive's host. Required when `services.hyperhive.swarm.wireguard.enable = true` and - you want this peer reachable over the mesh. Null = TLS- + you want this hive reachable over the mesh. Null = TLS- only peering (public internet, no mesh tunnel). ''; }; @@ -83,11 +88,13 @@ default = null; example = "203.0.113.1:51820"; description = '' - WireGuard endpoint for this peer in `host:port` form. - Required when the peer host is behind a firewall and - this host needs to initiate the tunnel. Null = this host - waits for the peer to connect (peer-initiates; peer must - have an endpoint pointing back at this host). + WireGuard endpoint for this hive in `host:port` form. + Null = this hive has no reachable endpoint, so the tunnel + is initiated from the other side. + + Reads like a fact about the relationship and is not: it + says whether *this* hive can be dialled, which every other + hive in the swarm needs the same answer to. ''; }; @@ -96,36 +103,82 @@ default = null; example = "10.100.0.2/32"; description = '' - IP address (with prefix) of the peer host on the - WireGuard mesh. Used as the `allowedIPs` for the peer's + IP address (with prefix) of this hive's host on the + WireGuard mesh. Used as the `allowedIPs` for its WireGuard config entry and injected into `HYPERHIVE_PEERS` so hive-c0re can route intra-swarm traffic to the mesh address rather than the public domain. Required to include - the peer in the WireGuard mesh (peers missing this field - are silently excluded from `wg-hive`). + a hive in the mesh (entries missing this field are + silently excluded from `wg-hive`). ''; }; - }; } ); default = { }; example = { - "lab.example.com" = { - certFingerprint = "sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12"; + pr1ma = { + domain = "pr1ma.example.com"; + wireguardAddress = "10.100.0.1/32"; + wireguardEndpoint = "203.0.113.1:51820"; + }; + edge = { + domain = "edge.example.com"; + wireguardAddress = "10.100.0.2/32"; }; - "edge.corp" = { }; }; description = '' - Peer hives in the same swarm. The attrset key is the peer's DNS - domain — used for dashboard links and Matrix federation discovery. - Null `certFingerprint` trusts the system CA bundle; set it to pin - a self-signed TLS cert. Add `wireguardPublicKey` + `wireguardAddress` - (and optionally `wireguardEndpoint`) to include the peer in the - WireGuard mesh when `swarm.wireguard.enable = true`. + Every hive in this swarm, keyed by `hiveName` — **including this + host's own hive**. The same attrset is meant to be identical on + every host in the swarm, so it can be written once and shared; + `services.hyperhive.hiveName` is what makes a given host read it + as "me and four others" rather than "five peers". + + Empty (the default) means this host is not part of a swarm. Once + non-empty it must contain an entry for `hiveName`, which is + asserted — a hive that lists everyone but itself would otherwise + derive its own peer set as *everything* and peer with itself. ''; }; + options.services.hyperhive.swarm.peerHives = lib.mkOption { + type = lib.types.attrsOf (lib.types.attrsOf lib.types.unspecified); + readOnly = true; + internal = true; + description = '' + Read-only: `hives` minus this host's own entry. Derived once here + rather than in each consumer, because "everything that isn't me" + is a filter four different modules were re-implementing and only + one of them has to be wrong for a hive to peer with itself. + ''; + }; + + config = { + services.hyperhive.swarm.peerHives = lib.filterAttrs (name: _: name != cfg.hiveName) swarmCfg.hives; + + assertions = [ + { + # Guarded on `hiveName != null` so the required-hiveName + # assertion in ./hyperhive.nix is what fires for that case — + # two assertions naming the same missing value is noise. + assertion = swarmCfg.hives == { } || cfg.hiveName == null || swarmCfg.hives ? ${cfg.hiveName}; + message = '' + services.hyperhive.swarm.hives has no entry for this hive + (services.hyperhive.hiveName = "${toString cfg.hiveName}"). + + `hives` describes every hive in the swarm including this one, + so that every host can share one identical attrset. Add: + + services.hyperhive.swarm.hives."${toString cfg.hiveName}" = { + domain = "${toString cfg.domain}"; + }; + + Declared hives: ${lib.concatStringsSep ", " (lib.attrNames swarmCfg.hives)} + ''; + } + ]; + }; + # `enableRequiredServices` is declared in ./swarm-required-services.nix # together with the per-service `enable`s it asserts — it is a # deployment-shape switch rather than swarm bookkeeping, so it lives