vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

refactor(nix): swarm.peers becomes swarm.hives, a directory of every hive

Browse source 
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.

Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.

Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.

The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.

Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
This commit is contained in:
atlas 2026-08-05 20:18:22 +02:00
commit 433b294099
19 changed files with 484 additions and 293 deletions
Download patch file Download diff file Expand all files Collapse all files

1
nix/host-modules/default.nix
View file

@ -28,6 +28,7 @@
./swarm-snapshot-store.nix
./swarm-wireguard.nix
./swarm.nix
./swarm-peers-removed.nix
./swarm-required-services.nix
];
}

36
nix/host-modules/hive-c0re/environment.nix
View file

@ -187,41 +187,25 @@ in
in
"${s.address}:${toString s.port}";
}
// lib.optionalAttrs (config.services.hyperhive.swarm.peers != { }) {
// lib.optionalAttrs (config.services.hyperhive.swarm.peerHives != { }) {
# Peer hives serialised as a JSON array of {domain, cert_fingerprint,
# wireguard_address?} objects. Consumed by hive-agent::identity::peers()
# + the dashboard's peer_hives StateSnapshot field (P33RS tab). Domain
# is the attrset key; cert_fingerprint is null for CA-trusted peers;
# wireguard_address is omitted when not part of the mesh.
# + the dashboard's peer_hives StateSnapshot field (P33RS tab).
# `cert_fingerprint` is null for CA-trusted hives; `wireguard_address`
# is omitted when not part of the mesh.
#
# Reads `peerHives` — `swarm.hives` minus this hive — so the "not me"
# filter is the one derived in ../swarm.nix rather than a fifth copy.
HYPERHIVE_PEERS = builtins.toJSON (
lib.mapAttrsToList (
domain: p:
_name: p:
{
inherit domain;
inherit (p) domain;
cert_fingerprint = p.certFingerprint;
}
// lib.optionalAttrs (p.wireguardAddress != null) {
wireguard_address = p.wireguardAddress;
}
) config.services.hyperhive.swarm.peers
) config.services.hyperhive.swarm.peerHives
);
}
//
lib.optionalAttrs
(lib.any (p: p.caCert != null) (lib.attrValues config.services.hyperhive.swarm.peers))
{
# Peer-hive root CA file paths (colon-joined), one per peer that
# declares `swarm.peers.<domain>.caCert`. hive-c0re's meta-flake
# renderer (meta.rs) embeds each next to every agent's flake and
# adds it to `security.pki.certificateFiles`, so a peer CA is
# trusted everywhere the hive's own internal CA (`hive-ca.pem`)
# is — i.e. by every agent. The matrix container trusts the same
# CAs separately for federation TLS. The `caCert` files are
# copied into the nix store at build, so these are store paths —
# nothing mutable lives on the host.
HIVE_PEER_CA_PATHS = lib.concatStringsSep ":" (
lib.filter (c: c != null) (
lib.mapAttrsToList (_domain: p: p.caCert) config.services.hyperhive.swarm.peers
)
);
}

8
nix/host-modules/hive-gateway/options.nix
View file

@ -147,8 +147,8 @@ in
config.security.acme.certs."example.com".directory;
```
When using an external CA cert, peer hives can declare this
hive in `services.hyperhive.swarm.peers` without
When using an external CA cert, other hives can declare this
one in `services.hyperhive.swarm.hives` without
`certFingerprint` — the standard CA bundle validates.
Mutual exclusion with `tls.acme.enable` — set one or the other,
@ -207,8 +207,8 @@ in
};
```
After enabling, peer hives can omit `certFingerprint` in
`swarm.peers` — Let's Encrypt certs are CA-trusted
After enabling, this hive's entry in `swarm.hives` can omit
`certFingerprint` — Let's Encrypt certs are CA-trusted
by default.
'';
};

28
nix/host-modules/hive-matrix.nix
View file

@ -395,20 +395,20 @@ in
# all filtering; never run one in here.
networking.firewall.enable = false;
# Peer-hive root CAs (`swarm.peers.<domain>.caCert`) added to THIS
# container's trust bundle so tuwunel validates *federation* TLS
# from a self-signed peer hive (it checks the peer's federation
# cert against its trust bundle). Peer CAs are trusted everywhere
# the hive's own internal CA is — agents get them via the
# meta-flake renderer (`HIVE_PEER_CA_PATHS` → each agent's
# `security.pki.certificateFiles`); this block is the matrix
# container's copy, since the host `security.pki` store doesn't
# cross the container boundary. They are never installed in the
# HOST trust store. Null entries (CA-bundle / fingerprint-pinned
# peers) drop out.
security.pki.certificateFiles = lib.filter (c: c != null) (
lib.mapAttrsToList (_domain: p: p.caCert) config.services.hyperhive.swarm.peers
);
# ⚠️ This container trusts no swarm-internal CA. It used to take
# per-hive root CAs, hand-pinned as nix paths, so tuwunel could
# validate *federation* TLS from a self-signed peer hive; that
# field is gone, and the swarm root that replaces it cannot be
# substituted here. `security.pki.certificateFiles` is read when
# the system is BUILT, and the swarm root is a runtime file
# (`swarm.ca.stateDir`) precisely because its key must never
# reach the store — so there is nothing build-time to name.
#
# Giving the container the swarm root therefore needs a runtime
# mechanism (bind-mount + a bundle assembled at start), which is
# a different shape than this line and is tracked as its own
# issue. Federation with a peer whose cert chains to the swarm
# root does not validate until then.
# tuwunel hard-fails to boot if `/etc/resolv.conf` has no
# `nameserver` line (`Failed to configure DNS resolver ... no

29
nix/host-modules/swarm-ca.nix
View file

@ -1,17 +1,19 @@
# The swarm root CA: the anchor a whole swarm shares, and the issuer of
# each hive's own CA (which is where it gets used — see ./hive-tls.nix).
#
# Why a hierarchy at all: cross-hive trust is hand-pinned today
# (`swarm.peers.<d>.caCert`), so every hive must name every peer — O(n²)
# configuration that a new hive can only join by editing all the others.
# One root makes it O(1): trust the root once and every present *and
# future* peer validates.
# Why a hierarchy at all: cross-hive trust used to be hand-pinned per
# peer, so every hive had to name every other one — O(n²) configuration
# that a new hive could only join by editing all the others. One root
# makes it O(1): trust the root once and every present *and future* peer
# validates. The per-peer CA field is gone precisely because this
# replaced it.
#
# Two provisioning modes, ONE structure — what differs is who puts the
# artifacts on disk, never what the artifacts are:
#
# - autoconfigured (the default while the hive declares no peers): the
# unit below generates the root here on first boot.
# - autoconfigured (`services.hyperhive.enableAllLocalDefaults`, or
# `autoConfigure` set directly): the unit below generates the root
# here on first boot.
# - operator-provided (multi-host): the operator installs the root cert
# — and, on a host that does not hold the root key, the hive CA too —
# into the state dirs, and this unit does nothing. Splitting a
@ -21,10 +23,11 @@
#
# The root KEY is the reason this is a runtime file and not a nix option:
# the store is world-readable and content-addressed, so a key committed
# to a flake is a key *published* to every consumer of that flake. The
# root CERT has the opposite property, and is already distributed at
# build time via `swarm.peers.<d>.caCert`. That asymmetry is what makes
# the layering work at all.
# to a flake is a key *published* to every consumer of that flake.
#
# ⚠️ The root CERT is therefore a runtime file too, which costs something:
# nothing whose trust store is built at build time can name it. See
# docs/swarm/ca.md § "Distributing the root".
{
lib,
config,
@ -83,8 +86,8 @@ in
default = "/var/lib/swarm-ca";
description = ''
Host directory holding the swarm root CA: `root.pem` (the
anchor, safe to distribute — this is what a peer hive is
pointed at via `swarm.peers.<d>.caCert`) and `root-key.pem`
anchor, safe to distribute — copy it to this same path on every
other host in the swarm) and `root-key.pem`
(0600, the one file that must never reach the nix store or
another host). The directory itself is 0700: nothing reads
out of it but the hive CA issuance in `hive-tls.nix`.

70
nix/host-modules/swarm-peers-removed.nix Normal file
View file

@ -0,0 +1,70 @@
# Migration shim for `services.hyperhive.swarm.peers`, replaced by the
# `swarm.hives` directory in ./swarm.nix.
#
# ⚠️ This could not be a `mkRenamedOptionModule`. `hives` is not `peers`
# under a new name, it is **`peers` ∪ {self}**: the set gains a member no
# existing config has ever written down, because a host's own identity
# lived in *other* options entirely. A rename migrates a name and a
# default can re-root a meaning; neither can conjure a new member.
#
# The whole module is self-contained and deletable — one file to remove
# when the deprecation window closes, with nothing else referring to it.
#
# Deliberately a warning rather than a hard failure, with the loudness
# coming from elsewhere: a config that set only `peers` leaves `hives`
# empty, so the self-entry assertion in ./swarm.nix fails the build
# anyway. The warning is what explains it; the assertion is what stops
# it.
{
lib,
config,
...
}:
let
peers = config.services.hyperhive.swarm.peers;
withCaCert = lib.attrNames (lib.filterAttrs (_: p: p ? caCert && p.caCert != null) peers);
in
{
options.services.hyperhive.swarm.peers = lib.mkOption {
type = lib.types.attrsOf lib.types.anything;
default = { };
visible = false;
internal = true;
description = ''
Removed — use `services.hyperhive.swarm.hives` instead, which
describes every hive in the swarm including this one. Kept only so
an existing definition produces a warning that says where to move
it, rather than an "option does not exist" error that says nothing.
'';
};
config.warnings =
lib.optional (peers != { }) ''
services.hyperhive.swarm.peers is removed and ignored. Move these
entries to services.hyperhive.swarm.hives, keyed by hive NAME
rather than domain, and add an entry for this hive itself
(services.hyperhive.hiveName) — `hives` is the swarm's full
directory, identical on every host.
services.hyperhive.swarm.hives.<name> = {
domain = "<the old attrset key>";
# certFingerprint / wireguard* carry over unchanged
};
Still set: ${lib.concatStringsSep ", " (lib.attrNames peers)}
''
++ lib.optional (withCaCert != [ ]) ''
services.hyperhive.swarm.peers.<hive>.caCert is removed and
ignored, and has no replacement in services.hyperhive.swarm.hives.
Trust inside a swarm now derives from the swarm root CA
(services.hyperhive.swarm.ca — see docs/swarm/ca.md): every hive
under it chains to it, so a per-hive CA is dead weight. What this
genuinely drops is trusting a hive whose root this swarm does NOT
own — another swarm's, or one keeping its own CA. certFingerprint
does not cover that: it pins a leaf for hive-c0re's own HTTPS
checks and does not reach Matrix federation.
Still set on: ${lib.concatStringsSep ", " withCaCert}
'';
}

7
nix/host-modules/swarm-wireguard.nix
View file

@ -108,9 +108,12 @@
networking.wireguard.interfaces =
let
wgCfg = config.services.hyperhive.swarm.wireguard;
# `peerHives` is `swarm.hives` minus this hive (../swarm.nix) —
# a mesh that included our own entry would configure a tunnel to
# ourselves.
meshPeers = lib.filterAttrs (
_: p: p.wireguardPublicKey != null && p.wireguardAddress != null
) config.services.hyperhive.swarm.peers;
) config.services.hyperhive.swarm.peerHives;
in
{
wg-hive = {
@ -118,7 +121,7 @@
listenPort = wgCfg.listenPort;
privateKeyFile = wgCfg.privateKeyFile;
peers = lib.mapAttrsToList (
_domain: p:
_name: p:
{
publicKey = p.wireguardPublicKey;
allowedIPs = [ p.wireguardAddress ];

175
nix/host-modules/swarm.nix
View file

@ -1,68 +1,73 @@
# Swarm peering: who the peer hives are. Serialised into hive-c0re's
# environment (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS — see ./hive-c0re)
# and consumed by identity.rs + the dashboard's P33RS tab.
# The swarm's directory: one entry per hive, **including this one**,
# identical on every host in the swarm. `services.hyperhive.hiveName`
# says which entry is us, and `peerHives` below derives the rest.
#
# Declaration only — this module has no `config` block. The mesh that
# uses the `wireguard*` fields below lives in ./swarm-wireguard.nix,
# because bringing up an interface is host networking rather than swarm
# Why a directory rather than a per-host peer list: every field here is
# intrinsic to the hive it describes — none of them says anything about
# the *pair*. A list where every field is intrinsic is a directory each
# host was keeping its own copy of, which is O(n²) duplication that
# deduplicates without loss. It is also a correctness gain: two hosts
# could hold different endpoints for the same third hive and nothing
# detected it. One entry per hive makes that unrepresentable.
#
# Consumed by hive-c0re's environment (HYPERHIVE_PEERS — see
# ./hive-c0re), identity.rs + the dashboard's P33RS tab, and the mesh in
# ./swarm-wireguard.nix. The mesh lives there rather than here because
# bringing up an interface is host networking rather than swarm
# bookkeeping, and a host that runs no hive still needs it.
{
lib,
config,
...
}:
let
cfg = config.services.hyperhive;
swarmCfg = cfg.swarm;
in
{
# Peer hives in the same swarm. Each entry declares a remote hive
# reachable from this host.
options.services.hyperhive.swarm.peers = lib.mkOption {
options.services.hyperhive.swarm.hives = lib.mkOption {
type = lib.types.attrsOf (
lib.types.submodule {
options = {
domain = lib.mkOption {
type = lib.types.str;
example = "lab.example.com";
description = ''
Public DNS domain this hive occupies — used for dashboard
links, peer HTTPS checks and Matrix federation discovery.
Deliberately has no default. It is conventionally
`<name>.<swarm.domain>`, but defaulting to that would let
a typo'd swarm domain produce a name that resolves
somewhere real; an eval failure naming the hive is the
better outcome.
'';
};
certFingerprint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12";
description = ''
Expected TLS certificate fingerprint for this peer's HTTPS
endpoint. Null = trust the system CA bundle (for Let's
Encrypt peers). Set to pin a self-signed cert.
Expected TLS certificate fingerprint for this hive's HTTPS
endpoint. Null = trust the CA bundle — which for a hive
inside the swarm CA hierarchy is the normal case, since
every hive under the swarm root already chains to it.
Set it to pin a leaf that no CA in the bundle vouches for.
Format: the literal `sha256:` followed by exactly 64
hex digits (case-insensitive, no colon separators) — the
SHA-256 digest of the peer's DER-encoded leaf certificate.
SHA-256 digest of the DER-encoded leaf certificate.
Generate with `openssl x509 -noout -fingerprint -sha256`,
then strip the colons and prepend `sha256:`. A malformed
value is ignored with a warning rather than weakening
trust. See docs/swarm/README.md for the full recipe.
Scopes only to hive-c0re's own peer HTTPS checks — it does
NOT help Matrix federation (tuwunel validates against its
container trust bundle). For a self-signed peer whose root
CA you want trusted hive-wide (every agent + Matrix
federation), set `caCert` below.
'';
};
caCert = lib.mkOption {
type = lib.types.nullOr lib.types.path;
default = null;
example = "./peers/edge-ca.pem";
description = ''
Path to this peer hive's root CA certificate (PEM). When
set, the CA is embedded (at build time, into the nix store
— no runtime file on the host) and trusted **everywhere the
hive's own internal CA is**: it rides alongside `hive-ca.pem`
in each agent's `security.pki.certificateFiles` (via the
meta-flake renderer), and is added to the Matrix homeserver
container's trust bundle so tuwunel validates *federation*
TLS from a self-signed peer hive whose cert chains to it.
This is the CA-trust path that `certFingerprint`
(leaf-pinning, c0re-only) can't cover, and is what unblocks
Matrix federation with a self-signed peer hive. Trust stays
inside the hive (agents + the Matrix container), never the
host system trust store. Mutually complementary with
`certFingerprint`; set `caCert` for the federation case. See
docs/swarm/README.md.
NOT help Matrix federation, which validates against the
container's trust bundle. There is no per-hive CA field to
cover that case any more: the swarm root is the trust path
(see ./swarm-ca.nix).
'';
};
@ -71,9 +76,9 @@
default = null;
example = "base64pubkey=";
description = ''
WireGuard public key for this peer host. Required when
WireGuard public key for this hive's host. Required when
`services.hyperhive.swarm.wireguard.enable = true` and
you want this peer reachable over the mesh. Null = TLS-
you want this hive reachable over the mesh. Null = TLS-
only peering (public internet, no mesh tunnel).
'';
};
@ -83,11 +88,13 @@
default = null;
example = "203.0.113.1:51820";
description = ''
WireGuard endpoint for this peer in `host:port` form.
Required when the peer host is behind a firewall and
this host needs to initiate the tunnel. Null = this host
waits for the peer to connect (peer-initiates; peer must
have an endpoint pointing back at this host).
WireGuard endpoint for this hive in `host:port` form.
Null = this hive has no reachable endpoint, so the tunnel
is initiated from the other side.
Reads like a fact about the relationship and is not: it
says whether *this* hive can be dialled, which every other
hive in the swarm needs the same answer to.
'';
};
@ -96,36 +103,82 @@
default = null;
example = "10.100.0.2/32";
description = ''
IP address (with prefix) of the peer host on the
WireGuard mesh. Used as the `allowedIPs` for the peer's
IP address (with prefix) of this hive's host on the
WireGuard mesh. Used as the `allowedIPs` for its
WireGuard config entry and injected into `HYPERHIVE_PEERS`
so hive-c0re can route intra-swarm traffic to the mesh
address rather than the public domain. Required to include
the peer in the WireGuard mesh (peers missing this field
are silently excluded from `wg-hive`).
a hive in the mesh (entries missing this field are
silently excluded from `wg-hive`).
'';
};
};
}
);
default = { };
example = {
"lab.example.com" = {
certFingerprint = "sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12";
pr1ma = {
domain = "pr1ma.example.com";
wireguardAddress = "10.100.0.1/32";
wireguardEndpoint = "203.0.113.1:51820";
};
edge = {
domain = "edge.example.com";
wireguardAddress = "10.100.0.2/32";
};
"edge.corp" = { };
};
description = ''
Peer hives in the same swarm. The attrset key is the peer's DNS
domain — used for dashboard links and Matrix federation discovery.
Null `certFingerprint` trusts the system CA bundle; set it to pin
a self-signed TLS cert. Add `wireguardPublicKey` + `wireguardAddress`
(and optionally `wireguardEndpoint`) to include the peer in the
WireGuard mesh when `swarm.wireguard.enable = true`.
Every hive in this swarm, keyed by `hiveName` — **including this
host's own hive**. The same attrset is meant to be identical on
every host in the swarm, so it can be written once and shared;
`services.hyperhive.hiveName` is what makes a given host read it
as "me and four others" rather than "five peers".
Empty (the default) means this host is not part of a swarm. Once
non-empty it must contain an entry for `hiveName`, which is
asserted — a hive that lists everyone but itself would otherwise
derive its own peer set as *everything* and peer with itself.
'';
};
options.services.hyperhive.swarm.peerHives = lib.mkOption {
type = lib.types.attrsOf (lib.types.attrsOf lib.types.unspecified);
readOnly = true;
internal = true;
description = ''
Read-only: `hives` minus this host's own entry. Derived once here
rather than in each consumer, because "everything that isn't me"
is a filter four different modules were re-implementing and only
one of them has to be wrong for a hive to peer with itself.
'';
};
config = {
services.hyperhive.swarm.peerHives = lib.filterAttrs (name: _: name != cfg.hiveName) swarmCfg.hives;
assertions = [
{
# Guarded on `hiveName != null` so the required-hiveName
# assertion in ./hyperhive.nix is what fires for that case —
# two assertions naming the same missing value is noise.
assertion = swarmCfg.hives == { } || cfg.hiveName == null || swarmCfg.hives ? ${cfg.hiveName};
message = ''
services.hyperhive.swarm.hives has no entry for this hive
(services.hyperhive.hiveName = "${toString cfg.hiveName}").
`hives` describes every hive in the swarm including this one,
so that every host can share one identical attrset. Add:
services.hyperhive.swarm.hives."${toString cfg.hiveName}" = {
domain = "${toString cfg.domain}";
};
Declared hives: ${lib.concatStringsSep ", " (lib.attrNames swarmCfg.hives)}
'';
}
];
};
# `enableRequiredServices` is declared in ./swarm-required-services.nix
# together with the per-service `enable`s it asserts — it is a
# deployment-shape switch rather than swarm bookkeeping, so it lives