refactor(nix): swarm.peers becomes swarm.hives, a directory of every hive
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
This commit is contained in:
atlas
parent
048bdd29a8
commit
433b294099
19 changed files with 484 additions and 293 deletions
|
|
@ -188,7 +188,7 @@ nginx inside the gateway container obtains and auto-renews certs via the ACME HT
|
|||
|
||||
Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails an assertion — pick one external TLS source (or neither, for the self-signed default).
|
||||
|
||||
**Swarm peers**: CA-signed certs are trusted by default — remote hives need no `certFingerprint` in `swarm.peers`.
|
||||
**Swarm peers**: CA-signed certs are trusted by default — this hive's entry in `swarm.hives` needs no `certFingerprint`.
|
||||
|
||||
### Self-signed TLS (default)
|
||||
|
||||
|
|
@ -232,10 +232,10 @@ security.acme.certs."example.com".group = "nginx";
|
|||
|
||||
or make the key world-readable (`0644`) if your threat model allows it. nginx errors out at startup on a key it can't read — the error is explicit in the journal, not a silent failure.
|
||||
|
||||
**Peer hive config**: when using a CA-signed cert, peer hives can declare this hive without `certFingerprint` in `swarm.peers` — the standard CA bundle validates:
|
||||
**Swarm directory entry**: when using a CA-signed cert, this hive's entry needs no `certFingerprint` — the standard CA bundle validates:
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.peers."example.com" = { }; # no certFingerprint needed
|
||||
services.hyperhive.swarm.hives.example = { domain = "example.com"; }; # no certFingerprint needed
|
||||
```
|
||||
|
||||
### Fronting with an external TLS terminator
|
||||
|
|
|
|||
|
|
@ -33,8 +33,8 @@ services.hyperhive.swarm.wireguard = {
|
|||
};
|
||||
```
|
||||
|
||||
The store host is a swarm member like any other: peers declare it, and
|
||||
it declares them, through `services.hyperhive.swarm.peers`. See
|
||||
The store host is a swarm member like any other: it gets an entry in
|
||||
`services.hyperhive.swarm.hives`, the same directory every host holds. See
|
||||
[swarm/](swarm/README.md) for the mesh itself.
|
||||
|
||||
Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
|
||||
|
|
|
|||
|
|
@ -12,8 +12,9 @@ the additional config needed when the swarm spans multiple hosts.
|
|||
containers.
|
||||
- **swarm** — one or more hives whose operators have declared them
|
||||
as peers. Agents can be qualified as `agent@hive-domain`.
|
||||
- **peer hive** — a remote hive declared under
|
||||
`services.hyperhive.swarm.peers` on the local host.
|
||||
- **peer hive** — any hive in `services.hyperhive.swarm.hives` other
|
||||
than this one. Peers are *derived*, not declared: the directory lists
|
||||
every hive including yourself, and `hiveName` says which one you are.
|
||||
|
||||
## Hive identity config
|
||||
|
||||
|
|
@ -71,41 +72,43 @@ One authelia, one matrix, one forge per swarm — which host runs them,
|
|||
and what a hive that runs none of them configures instead:
|
||||
[`services.md`](services.md).
|
||||
|
||||
## Declaring peer hives
|
||||
## The swarm's hive directory
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.peers = {
|
||||
"lab.example.com" = { }; # CA-trusted (Let's Encrypt etc.)
|
||||
"edge.corp" = { certFingerprint = "sha256:…"; }; # self-signed TLS, c0re peer checks only
|
||||
"mesh.internal" = { caCert = ./mesh-ca.pem; }; # self-signed, trusted for matrix federation
|
||||
services.hyperhive.swarm.hives = {
|
||||
pr1ma = { domain = "pr1ma.example.com"; }; # this host, per hiveName
|
||||
lab = { domain = "lab.example.com"; }; # CA-trusted (Let's Encrypt etc.)
|
||||
edge = { domain = "edge.corp"; certFingerprint = "sha256:…"; }; # self-signed leaf, pinned
|
||||
};
|
||||
```
|
||||
|
||||
The attrset key is the peer's DNS domain. Two independent, optional
|
||||
trust knobs — pick by what you need to trust:
|
||||
One attrset describing **every** hive in the swarm, **including this
|
||||
one**, keyed by that hive's `hiveName`. It is meant to be *identical on
|
||||
every host* — write it once, share it, and each host reads it correctly
|
||||
because `services.hyperhive.hiveName` says which entry is itself.
|
||||
|
||||
- **`certFingerprint`** (`"sha256:…"`) — pin the peer's TLS _leaf_
|
||||
fingerprint. Scopes **only** to hive-c0re's own peer HTTPS checks
|
||||
(the P33RS dashboard links + agent peer discovery below). It is
|
||||
**not** consulted by matrix federation — tuwunel validates a peer's
|
||||
federation certificate against the system CA bundle independently
|
||||
(see _Matrix federation_ below), so a fingerprint pin does nothing
|
||||
for a self-signed matrix cert.
|
||||
- **`caCert`** (path to the peer's trust bundle / root CA PEM, i.e. what
|
||||
its `hivectl peer-config` told you to copy) — embeds that CA (at
|
||||
build time, into the nix store — no runtime file on the host) and
|
||||
trusts it **everywhere the hive's own internal CA is**: it rides
|
||||
alongside `hive-ca.pem` in every agent's
|
||||
`security.pki.certificateFiles` (via the meta-flake renderer) **and**
|
||||
in the matrix container's trust bundle, so tuwunel validates the
|
||||
peer's _federation_ TLS when it chains to that CA. Trust stays
|
||||
**inside the hive** (agents + the matrix container), never the host
|
||||
system trust store. **This is the knob that unblocks federation with
|
||||
a self-signed peer hive** — use it instead of `certFingerprint` when
|
||||
you control the peer's CA. (It does not affect hive-c0re's own peer
|
||||
HTTPS checks — those stay on `certFingerprint` / the system bundle.)
|
||||
- **Both omitted** — the stock system CA bundle validates the peer
|
||||
(correct for Let's Encrypt / any publicly-trusted peer).
|
||||
Empty (the default) means this host isn't in a swarm. Once non-empty it
|
||||
**must** contain an entry for `hiveName`; eval fails naming the missing
|
||||
hive. That assertion is load-bearing rather than pedantic — "my peers"
|
||||
is derived as *everything that isn't me*, so a directory that doesn't
|
||||
contain you derives every hive as a peer and you peer with yourself.
|
||||
|
||||
`domain` is required per entry and deliberately undefaulted: it is
|
||||
conventionally `<name>.<swarm.domain>`, but a wrong domain that
|
||||
evaluates cleanly points at a real machine that isn't the one you meant.
|
||||
|
||||
**`certFingerprint`** (`"sha256:…"`, optional) pins that hive's TLS
|
||||
_leaf_. Scopes **only** to hive-c0re's own peer HTTPS checks (the P33RS
|
||||
dashboard links + agent peer discovery below); matrix federation never
|
||||
consults it. Omit it for any hive under the swarm root CA or a public
|
||||
CA — which is the normal case.
|
||||
|
||||
> **There is no per-hive CA field.** Trust inside a swarm comes from the
|
||||
> swarm root ([`ca.md`](ca.md)): every hive chains to it, so one anchor
|
||||
> replaces the O(n²) pinning. What that genuinely drops is trusting a
|
||||
> hive whose root this swarm does *not* own — another swarm's, or one
|
||||
> keeping its own CA. That is a cross-swarm problem and wants a
|
||||
> mechanism designed for it, not a field that happened to work.
|
||||
|
||||
### Fingerprint format
|
||||
|
||||
|
|
@ -163,27 +166,33 @@ environment and forwarded to agent containers.
|
|||
`.well-known/matrix/server` delegation, which the gateway serves).
|
||||
Federation validates the peer's TLS certificate against the matrix
|
||||
**container's** trust bundle — independently of `certFingerprint`,
|
||||
which it never consults. A self-signed gateway certificate therefore
|
||||
won't federate unless the peer's root CA is trusted: set `caCert`
|
||||
above (embeds the peer CA into the matrix container's trust bundle),
|
||||
or give the peers CA-issued certs (ACME). See `docs/matrix.md` for
|
||||
federation firewall + TLS requirements.
|
||||
which it never consults.
|
||||
|
||||
## Bilateral setup
|
||||
⚠️ **That container currently trusts no swarm-internal CA**, so a
|
||||
self-signed gateway certificate does not federate. The swarm root
|
||||
can't simply be listed there: `security.pki.certificateFiles` is
|
||||
read when the system is *built*, and the root is a runtime file (its
|
||||
key must never enter the store), so there is no build-time name for
|
||||
it. Bridging that needs a runtime mechanism and is tracked as its own
|
||||
issue. Until then, federation needs CA-issued certs (ACME). See
|
||||
`docs/matrix.md` for federation firewall + TLS requirements.
|
||||
|
||||
Each hive must declare the other. If hive A lists hive B as a peer,
|
||||
B must also list A for agents on B to see A in their peer list:
|
||||
## One directory, not a bilateral declaration
|
||||
|
||||
Both hives hold the **same** `hives` attrset; neither declares the
|
||||
other. What differs between the two hosts is only `hiveName`:
|
||||
|
||||
```
|
||||
# hive A (pr1ma.example.com)
|
||||
services.hyperhive.swarm.peers."edge.corp" = { };
|
||||
|
||||
# hive B (edge.corp)
|
||||
services.hyperhive.swarm.peers."pr1ma.example.com" = { certFingerprint = "sha256:…"; };
|
||||
# hive A # hive B
|
||||
hiveName = "pr1ma"; hiveName = "edge";
|
||||
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
|
||||
```
|
||||
|
||||
Mixed trust is fine: A trusts B via CA bundle (no fingerprint), B
|
||||
pins A's self-signed cert.
|
||||
That is the point of the shape, and it removes a class of bug rather
|
||||
than saving typing: a per-host peer list let two hosts hold *different*
|
||||
facts about the same third hive — a stale endpoint, a rotated
|
||||
fingerprint — with nothing to detect the disagreement. One entry per
|
||||
hive makes it unrepresentable.
|
||||
|
||||
## WireGuard inter-hive mesh (optional)
|
||||
|
||||
|
|
@ -213,11 +222,22 @@ services.hyperhive = {
|
|||
listenPort = 51820; # optional, default 51820
|
||||
};
|
||||
|
||||
swarm.peers."edge.corp" = {
|
||||
certFingerprint = "sha256:…"; # TLS trust (unchanged)
|
||||
wireguardPublicKey = "base64key="; # peer's wg pubkey
|
||||
wireguardEndpoint = "203.0.113.42:51820"; # peer's public IP:port
|
||||
wireguardAddress = "10.100.0.2/32"; # peer's mesh IP
|
||||
# The same `hives` attrset both hosts hold — mesh fields included,
|
||||
# since "where this hive can be dialled" is a fact about that hive.
|
||||
swarm.hives = {
|
||||
pr1ma = {
|
||||
domain = "pr1ma.example.com";
|
||||
wireguardPublicKey = "base64keyA=";
|
||||
wireguardEndpoint = "198.51.100.1:51820";
|
||||
wireguardAddress = "10.100.0.1/32";
|
||||
};
|
||||
edge = {
|
||||
domain = "edge.corp";
|
||||
certFingerprint = "sha256:…"; # TLS trust (unchanged)
|
||||
wireguardPublicKey = "base64keyB=";
|
||||
wireguardEndpoint = "203.0.113.42:51820";
|
||||
wireguardAddress = "10.100.0.2/32";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
|
|
@ -229,11 +249,7 @@ services.hyperhive = {
|
|||
address = "10.100.0.2/24";
|
||||
};
|
||||
|
||||
swarm.peers."pr1ma.example.com" = {
|
||||
wireguardPublicKey = "base64key="; # hive A's wg pubkey
|
||||
wireguardEndpoint = "198.51.100.1:51820";
|
||||
wireguardAddress = "10.100.0.1/32";
|
||||
};
|
||||
swarm.hives = { /* … identical to hive A's … */ };
|
||||
};
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -126,3 +126,35 @@ named in the message. That is a decision, and it is recorded as one.
|
|||
|
||||
A hive with **no** root configured at all is not affected by any of
|
||||
this: it self-signs exactly as it always has.
|
||||
|
||||
## Distributing the root
|
||||
|
||||
The root **key** is a runtime file for an obvious reason: the nix store
|
||||
is world-readable and content-addressed, so a key committed to a flake
|
||||
is a key *published* to every consumer of that flake.
|
||||
|
||||
The root **certificate** is a runtime file as a *consequence* — it lives
|
||||
beside the key under `swarm.ca.stateDir` — and that has a cost worth
|
||||
naming, because it is not obvious and it bites at a distance:
|
||||
|
||||
> **Nothing whose trust store is assembled at build time can reference
|
||||
> the swarm root.** `security.pki.certificateFiles` is read inside the
|
||||
> derivation; the root does not exist there.
|
||||
|
||||
Two consumers, and only one of them is fine:
|
||||
|
||||
- **Agents are covered.** `hive-tls.nix` folds the root into this hive's
|
||||
`trust-bundle.pem`, hive-c0re receives that path as
|
||||
`HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one file
|
||||
next to every agent's flake. The bundle is the runtime-to-build-time
|
||||
bridge.
|
||||
- **The Matrix container is not.** It has no equivalent bridge, so it
|
||||
trusts no swarm-internal CA and federation with a self-signed peer
|
||||
does not validate. Giving it the root needs a runtime mechanism —
|
||||
bind-mount plus a bundle assembled at unit start, appending to the
|
||||
system bundle rather than replacing it — which is tracked separately.
|
||||
|
||||
To put the root on another host, copy the certificate to the same path
|
||||
there (`scp <stateDir>/root.pem <host>:<stateDir>/root.pem`). One anchor
|
||||
per host, not one per peer: a hive joining later needs no edit on the
|
||||
hives already running, which is the whole point of the hierarchy.
|
||||
|
|
|
|||
|
|
@ -761,7 +761,7 @@ Print the nix to add a peer hive to the mesh
|
|||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<DOMAIN>` — Peer hive's DNS domain (the `swarm.peers` attrset key)
|
||||
* `<DOMAIN>` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
|
||||
|
||||
###### **Options:**
|
||||
|
||||
|
|
|
|||
|
|
@ -607,10 +607,10 @@ prices. Models not covered fall back to hive-c0re's built-in estimate.
|
|||
## P33RS tab
|
||||
|
||||
Peer hives in this swarm. The tab is hidden when the
|
||||
`state.peer_hives` array from `/api/state` is empty (i.e. no
|
||||
`services.hyperhive.swarm.peers` are configured). When at least
|
||||
one peer is present the `hidden` attribute is removed and the tab
|
||||
becomes active.
|
||||
`state.peer_hives` array from `/api/state` is empty — i.e. when
|
||||
`services.hyperhive.swarm.hives` holds no hive other than this one.
|
||||
When at least one peer is present the `hidden` attribute is removed
|
||||
and the tab becomes active.
|
||||
|
||||
**P33R H1V3S** — each peer renders as a card row: a hexagon icon
|
||||
(`⬡`), the peer's DNS domain as the primary name, and the peer
|
||||
|
|
@ -619,15 +619,16 @@ URL opens the peer hive's dashboard in a new tab.
|
|||
|
||||
### Backend wiring
|
||||
|
||||
The host daemon reads `services.hyperhive.swarm.peers` from the
|
||||
nix config (an attrset keyed by peer domain), serialises each
|
||||
entry as `{ name, url }` into `state.peer_hives: Vec<PeerHiveView>`,
|
||||
The host daemon reads `services.hyperhive.swarm.peerHives` from the
|
||||
nix config (the `swarm.hives` directory minus this hive), serialises
|
||||
each entry as `{ name, url }` into `state.peer_hives: Vec<PeerHiveView>`,
|
||||
and includes the field in the `/api/state` snapshot. `tabs.js`
|
||||
reads `state.peer_hives` on every `refreshState` call and calls
|
||||
`swarm.js::renderPeerHives(peers)`, which rebuilds the `#peers-section`
|
||||
div from scratch.
|
||||
|
||||
The `name` field is the peer's DNS domain (the attrset key); `url`
|
||||
The `name` field is the peer's DNS domain (its entry's `domain`, not
|
||||
the attrset key — the key is the hive's name); `url`
|
||||
is `https://{domain}/`. Both are derived from the env var
|
||||
`HYPERHIVE_PEERS` (a JSON array of `{ domain, cert_fingerprint }`
|
||||
objects) that the nix module writes into the c0re container
|
||||
|
|
|
|||
Loading…
Reference in a new issue