Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f798662396 | ||
|
|
b3b1ed19c6 | ||
|
|
ade0bf24c5 | ||
|
|
8d21a47f42 | ||
|
|
6ea43c1151 | ||
|
|
316cef94d1 | ||
|
|
b94382b815 |
14 changed files with 420 additions and 15 deletions
|
|
@ -223,7 +223,7 @@ read them à la carte.
|
|||
bridge network for?"** → [`docs/network.md`](docs/network.md).
|
||||
- **"How do I connect two hives into a swarm? How do I declare peer
|
||||
hives and configure TLS trust?"** →
|
||||
[`docs/swarm.md`](docs/swarm.md).
|
||||
[`docs/swarm/`](docs/swarm/README.md).
|
||||
- **"Where do agent snapshots go? How does the swarm's `btrfs receive`
|
||||
endpoint authenticate a pushing hive, and what does a snapshot
|
||||
actually contain?"** →
|
||||
|
|
|
|||
|
|
@ -71,7 +71,7 @@ declarations.
|
|||
discovery work?** → [`gateway.md`](gateway.md).
|
||||
- **How does DNS resolution work in agent containers? What's the
|
||||
bridge network for?** → [`network.md`](network.md).
|
||||
- **How do I connect two hives into a swarm?** → [`swarm.md`](swarm.md)
|
||||
- **How do I connect two hives into a swarm?** → [`swarm/`](swarm/README.md)
|
||||
(peer hives, TLS trust).
|
||||
- **Where do agent snapshots go? How does the swarm's `btrfs receive`
|
||||
endpoint authenticate a pushing hive?** →
|
||||
|
|
|
|||
|
|
@ -198,7 +198,7 @@ The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host servi
|
|||
|
||||
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated, in-container leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
|
||||
|
||||
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm.md`](swarm.md#swarm-ca) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
|
||||
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/`](swarm/README.md#swarm-ca) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
|
||||
|
||||
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap.
|
||||
|
||||
|
|
|
|||
|
|
@ -180,7 +180,7 @@ at the protocol level from first boot (`allow_federation = true`) but
|
|||
no remote homeserver is trusted until listed here. For a closed
|
||||
single-hive deployment the default empty list is correct — add peer
|
||||
hive domains here when connecting hives into a swarm (see
|
||||
[`docs/swarm.md`](swarm.md)).
|
||||
[`docs/swarm/`](swarm/README.md)).
|
||||
|
||||
**`maxRequestSize`** (default `20_000_000` bytes = 20 MB) — maximum
|
||||
size of a single matrix client request body. Matches the matrix-spec
|
||||
|
|
|
|||
|
|
@ -66,7 +66,7 @@ The flows, end to end:
|
|||
the host forwards + masquerades out its uplink.
|
||||
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
|
||||
and reach each other's gateway/forge across it
|
||||
([`docs/swarm.md`](swarm.md)).
|
||||
([`docs/swarm/`](swarm/README.md)).
|
||||
- **Control plane (no network)** — per-agent broker socket
|
||||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||||
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ services.hyperhive.swarm.wireguard = {
|
|||
|
||||
The store host is a swarm member like any other: peers declare it, and
|
||||
it declares them, through `services.hyperhive.swarm.peers`. See
|
||||
[swarm.md](swarm.md) for the mesh itself.
|
||||
[swarm/](swarm/README.md) for the mesh itself.
|
||||
|
||||
Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
|
||||
`c0re.enable` --- a store host runs no hive and would otherwise get no
|
||||
|
|
|
|||
343
docs/swarm/README.md
Normal file
343
docs/swarm/README.md
Normal file
|
|
@ -0,0 +1,343 @@
|
|||
# Multi-hive swarms
|
||||
|
||||
A **swarm** is a collection of agents that share an identity and
|
||||
coordinate across one or more hives. A single hyperhive instance
|
||||
running on one host is already a swarm (one hive). This doc covers
|
||||
the additional config needed when the swarm spans multiple hosts.
|
||||
|
||||
## Terminology
|
||||
|
||||
- **hive** — a single hyperhive installation on one host. Has its
|
||||
own `services.hyperhive.domain` DNS name and its own set of agent
|
||||
containers.
|
||||
- **swarm** — one or more hives whose operators have declared them
|
||||
as peers. Agents can be qualified as `agent@hive-domain`.
|
||||
- **peer hive** — a remote hive declared under
|
||||
`services.hyperhive.swarm.peers` on the local host.
|
||||
|
||||
## Hive identity config
|
||||
|
||||
```nix
|
||||
services.hyperhive = {
|
||||
swarm.domain = "example.com"; # required — the swarm's DNS domain
|
||||
hiveName = "pr1ma"; # required — this hive's label in it
|
||||
# domain = "pr1ma.example.com"; # derived from the two above
|
||||
swarm.name = "constellat1on"; # shared swarm display name (optional)
|
||||
};
|
||||
```
|
||||
|
||||
`swarm.domain` and `hiveName` are **required** whenever hyperhive is
|
||||
enabled; eval fails with a hint naming each. Neither is defaulted,
|
||||
because a guessed value here is a wrong hostname that evaluates cleanly
|
||||
and deploys — an eval failure asking the operator to write the address
|
||||
down is the cheaper outcome. **Upgrading past this release means setting
|
||||
both once.**
|
||||
|
||||
`domain` is required too, but you no longer have to *write* it: it
|
||||
defaults to `<hiveName>.<swarm.domain>`, since every hive in a swarm
|
||||
occupies its own sub-domain of it. A hive that pins `domain` explicitly
|
||||
keeps exactly the value it has today — that's why this is a default and
|
||||
not a rename: re-rooting where a value *comes from* must not reinterpret
|
||||
the values already deployed.
|
||||
|
||||
`domain` drives `HYPERHIVE_HIVE_DOMAIN` in every container so agents can
|
||||
form qualified labels (`iris@pr1ma.example.com`).
|
||||
|
||||
`swarm.name` is purely display — it surfaces in the dashboard chrome
|
||||
header and per-agent system prompts, and federated hives at different
|
||||
domains can share one. `hiveName` surfaces in the same places but is
|
||||
*not* only display: it is the leftmost label of the hive's domain. That
|
||||
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
|
||||
distinction — one names this hive, the other names the group it belongs
|
||||
to.
|
||||
|
||||
See `docs/conventions.md` § Hive identity for the env-var chain
|
||||
and `qualify()` / `qualified_label()` semantics.
|
||||
|
||||
## Swarm CA
|
||||
|
||||
A hive's internal TLS chains to a **swarm root CA**: the root signs each
|
||||
hive's own CA, and that hive CA signs the gateway leaf. A peer that
|
||||
trusts the root once validates every hive in the swarm, present and
|
||||
future, instead of being pinned to each one by hand.
|
||||
|
||||
Two provisioning modes, one structure — what differs is who puts the
|
||||
artifacts on disk:
|
||||
|
||||
| | swarm root | this hive's CA |
|
||||
| --- | --- | --- |
|
||||
| default | operator-provided | operator-provided, else self-signed as before |
|
||||
| `autoConfigure = true` (all on one host) | generated by `swarm-ca.service` on first boot | issued by `hive-tls-ca.service` under the root |
|
||||
|
||||
`services.hyperhive.swarm.ca.autoConfigure` selects between them, and is
|
||||
**off by default**: a swarm's services and its hives can live on
|
||||
different hosts, and a host cannot tell whether it is the one holding
|
||||
the root, so setting the swarm CA up is an operator action rather than
|
||||
something a host assumes. Turn it on for an all-on-one-host deployment
|
||||
and the hierarchy costs no configuration.
|
||||
|
||||
**A hive given neither artifact keeps the self-signed CA it has always
|
||||
had.** It serves TLS exactly as before and simply isn't part of a
|
||||
swarm's trust hierarchy — the right outcome for a hive nobody has
|
||||
federated yet. Only `autoConfigure` issues a hive sub-CA, because only
|
||||
that case can: signing one needs the root's private key.
|
||||
|
||||
Moving the swarm CA onto its own host is then a matter of moving
|
||||
`services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off —
|
||||
there is no second code path to switch to.
|
||||
|
||||
The root's private key never reaches the nix store: the store is
|
||||
world-readable, so a key committed to a flake is a key published to
|
||||
everyone who builds it. Only certificates are distributed.
|
||||
|
||||
Each hive CA is **name-constrained** (X.509 `nameConstraints`) to that
|
||||
hive's own domain, so a hive CA that leaks can only mint names inside
|
||||
its own subdomain — enforced by every verifier rather than by
|
||||
convention.
|
||||
|
||||
### What to hand a peer
|
||||
|
||||
`hivectl peer-config` prints the `cp` line. The file is
|
||||
`<tls.stateDir>/trust-bundle.pem` — the hive CA plus the swarm root —
|
||||
not `ca.pem`: an intermediate on its own is not something a verifier can
|
||||
build a chain to. On a hive that predates the swarm root the bundle is
|
||||
just that hive's self-signed CA, so the recipe does not change.
|
||||
|
||||
### Adopting the hierarchy on an existing hive
|
||||
|
||||
An existing `ca.pem` is never re-rooted automatically — swapping it
|
||||
would break every consumer that already trusts it, and agents only pick
|
||||
up new trust when their container restarts. To adopt, delete `ca.pem` +
|
||||
`ca-key.pem` under `tls.stateDir` and restart `hive-tls-ca.service`;
|
||||
the CA is re-issued under the root and the leaf re-signed. Until then
|
||||
the hive serves TLS exactly as before and is simply not part of the
|
||||
swarm's trust hierarchy.
|
||||
|
||||
## Running the swarm's shared services
|
||||
|
||||
One authelia, one matrix, one forge per swarm — which host runs them,
|
||||
and what a hive that runs none of them configures instead:
|
||||
[`services.md`](services.md).
|
||||
|
||||
## Declaring peer hives
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.peers = {
|
||||
"lab.example.com" = { }; # CA-trusted (Let's Encrypt etc.)
|
||||
"edge.corp" = { certFingerprint = "sha256:…"; }; # self-signed TLS, c0re peer checks only
|
||||
"mesh.internal" = { caCert = ./mesh-ca.pem; }; # self-signed, trusted for matrix federation
|
||||
};
|
||||
```
|
||||
|
||||
The attrset key is the peer's DNS domain. Two independent, optional
|
||||
trust knobs — pick by what you need to trust:
|
||||
|
||||
- **`certFingerprint`** (`"sha256:…"`) — pin the peer's TLS _leaf_
|
||||
fingerprint. Scopes **only** to hive-c0re's own peer HTTPS checks
|
||||
(the P33RS dashboard links + agent peer discovery below). It is
|
||||
**not** consulted by matrix federation — tuwunel validates a peer's
|
||||
federation certificate against the system CA bundle independently
|
||||
(see _Matrix federation_ below), so a fingerprint pin does nothing
|
||||
for a self-signed matrix cert.
|
||||
- **`caCert`** (path to the peer's trust bundle / root CA PEM, i.e. what
|
||||
its `hivectl peer-config` told you to copy) — embeds that CA (at
|
||||
build time, into the nix store — no runtime file on the host) and
|
||||
trusts it **everywhere the hive's own internal CA is**: it rides
|
||||
alongside `hive-ca.pem` in every agent's
|
||||
`security.pki.certificateFiles` (via the meta-flake renderer) **and**
|
||||
in the matrix container's trust bundle, so tuwunel validates the
|
||||
peer's _federation_ TLS when it chains to that CA. Trust stays
|
||||
**inside the hive** (agents + the matrix container), never the host
|
||||
system trust store. **This is the knob that unblocks federation with
|
||||
a self-signed peer hive** — use it instead of `certFingerprint` when
|
||||
you control the peer's CA. (It does not affect hive-c0re's own peer
|
||||
HTTPS checks — those stay on `certFingerprint` / the system bundle.)
|
||||
- **Both omitted** — the stock system CA bundle validates the peer
|
||||
(correct for Let's Encrypt / any publicly-trusted peer).
|
||||
|
||||
### Fingerprint format
|
||||
|
||||
The value is the string `sha256:` followed by exactly 64 hexadecimal
|
||||
digits — the SHA-256 digest of the peer's DER-encoded TLS leaf
|
||||
certificate. The hex is case-insensitive (upper or lower both parse),
|
||||
carries no colon separators between bytes, and any value not matching
|
||||
this shape is ignored with a warning rather than weakening trust.
|
||||
|
||||
```
|
||||
sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12
|
||||
```
|
||||
|
||||
Generate it from the peer's certificate with openssl. The
|
||||
`-fingerprint -sha256` output is uppercase and colon-separated, so
|
||||
strip the colons, lowercase, and prepend the `sha256:` prefix:
|
||||
|
||||
```sh
|
||||
# from a PEM/CRT file
|
||||
openssl x509 -in peer.crt -noout -fingerprint -sha256 \
|
||||
| sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
|
||||
|
||||
# straight from the live endpoint (port 443)
|
||||
echo | openssl s_client -connect peer.example.com:443 -servername peer.example.com 2>/dev/null \
|
||||
| openssl x509 -noout -fingerprint -sha256 \
|
||||
| sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
|
||||
```
|
||||
|
||||
Pin the leaf certificate, not an intermediate or the CA — the
|
||||
digest must match the exact cert the peer serves on its HTTPS
|
||||
endpoint. When the peer rotates its cert, update the pin to the new
|
||||
fingerprint (or switch the peer to a CA-trusted cert and drop the
|
||||
field).
|
||||
|
||||
The nix module serialises the attrset to a `HYPERHIVE_PEERS` JSON
|
||||
array (`[{ domain, cert_fingerprint }]`) injected into the c0re
|
||||
environment and forwarded to agent containers.
|
||||
|
||||
## What the config does at runtime
|
||||
|
||||
1. **Dashboard P33RS tab** — `parse_peer_hives()` in `dashboard.rs`
|
||||
reads `HYPERHIVE_PEERS` and includes
|
||||
`peer_hives: Vec<{ name, url }>` in `/api/state`. The dashboard
|
||||
shows a P33RS tab (hidden when the list is empty) with a card per
|
||||
peer linking to `https://{domain}/`. See
|
||||
`docs/web-ui/dashboard.md` § P33RS tab.
|
||||
|
||||
2. **Agent identity** — the same `HYPERHIVE_PEERS` env var is
|
||||
forwarded to agent containers by `meta.rs`; agent code can call
|
||||
`identity::peers()` to discover peer hives and address them with
|
||||
qualified names (`agent@domain`).
|
||||
|
||||
3. **Matrix federation** — when `matrix.enable` is on, tuwunel
|
||||
federates with the peer's matrix server (discovered via the peer's
|
||||
`.well-known/matrix/server` delegation, which the gateway serves).
|
||||
Federation validates the peer's TLS certificate against the matrix
|
||||
**container's** trust bundle — independently of `certFingerprint`,
|
||||
which it never consults. A self-signed gateway certificate therefore
|
||||
won't federate unless the peer's root CA is trusted: set `caCert`
|
||||
above (embeds the peer CA into the matrix container's trust bundle),
|
||||
or give the peers CA-issued certs (ACME). See `docs/matrix.md` for
|
||||
federation firewall + TLS requirements.
|
||||
|
||||
## Bilateral setup
|
||||
|
||||
Each hive must declare the other. If hive A lists hive B as a peer,
|
||||
B must also list A for agents on B to see A in their peer list:
|
||||
|
||||
```
|
||||
# hive A (pr1ma.example.com)
|
||||
services.hyperhive.swarm.peers."edge.corp" = { };
|
||||
|
||||
# hive B (edge.corp)
|
||||
services.hyperhive.swarm.peers."pr1ma.example.com" = { certFingerprint = "sha256:…"; };
|
||||
```
|
||||
|
||||
Mixed trust is fine: A trusts B via CA bundle (no fingerprint), B
|
||||
pins A's self-signed cert.
|
||||
|
||||
## WireGuard inter-hive mesh (optional)
|
||||
|
||||
The peer config above uses public HTTPS for all inter-hive traffic.
|
||||
For private deployments — or to reduce latency and TLS overhead on
|
||||
intra-swarm traffic — hive-c0re can configure a host-to-host
|
||||
WireGuard mesh.
|
||||
|
||||
### Generating keys
|
||||
|
||||
On each hive host:
|
||||
|
||||
```bash
|
||||
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
|
||||
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
|
||||
```
|
||||
|
||||
### Config example (two hives)
|
||||
|
||||
```nix
|
||||
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
|
||||
services.hyperhive = {
|
||||
swarm.wireguard = {
|
||||
enable = true;
|
||||
privateKeyFile = "/etc/wireguard/hive.key";
|
||||
address = "10.100.0.1/24";
|
||||
listenPort = 51820; # optional, default 51820
|
||||
};
|
||||
|
||||
swarm.peers."edge.corp" = {
|
||||
certFingerprint = "sha256:…"; # TLS trust (unchanged)
|
||||
wireguardPublicKey = "base64key="; # peer's wg pubkey
|
||||
wireguardEndpoint = "203.0.113.42:51820"; # peer's public IP:port
|
||||
wireguardAddress = "10.100.0.2/32"; # peer's mesh IP
|
||||
};
|
||||
};
|
||||
|
||||
# hive B (edge.corp, mesh IP 10.100.0.2)
|
||||
services.hyperhive = {
|
||||
swarm.wireguard = {
|
||||
enable = true;
|
||||
privateKeyFile = "/etc/wireguard/hive.key";
|
||||
address = "10.100.0.2/24";
|
||||
};
|
||||
|
||||
swarm.peers."pr1ma.example.com" = {
|
||||
wireguardPublicKey = "base64key="; # hive A's wg pubkey
|
||||
wireguardEndpoint = "198.51.100.1:51820";
|
||||
wireguardAddress = "10.100.0.1/32";
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
### What the mesh does
|
||||
|
||||
- `networking.wireguard.interfaces.wg-hive` is configured on the host
|
||||
(not inside agent containers; containers reach peers via the host's
|
||||
routing table).
|
||||
- UDP port 51820 (or `listenPort`) is opened on the host firewall.
|
||||
- `HYPERHIVE_PEERS` gains a `wireguard_address` field for each mesh
|
||||
peer so hive-c0re can reach intra-swarm services without a public
|
||||
DNS round-trip.
|
||||
- `persistentKeepalive = 25` is set by default; override or null to
|
||||
disable (not needed when both sides have public IPs and no NAT).
|
||||
|
||||
### NAT / one-sided endpoints
|
||||
|
||||
If one host is behind NAT and can't accept incoming connections, only
|
||||
that host needs a null `wireguardEndpoint` on the peer config — the
|
||||
other side initiates. With keepalive on, the NAT hole stays open.
|
||||
|
||||
If both hosts are behind NAT, a STUN relay or a third host (exit node)
|
||||
is required. Out of scope for v0.
|
||||
|
||||
## Snapshot store
|
||||
|
||||
One further option lives in this namespace but is documented with the
|
||||
service it points at: `services.hyperhive.swarm.snapshotStore.{address,
|
||||
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
|
||||
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.
|
||||
|
||||
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
|
||||
one store, because the receiver keys destinations by *agent* so a
|
||||
migrating agent keeps one unbroken incremental chain. See
|
||||
[snapshot-store.md](../snapshot-store.md).
|
||||
|
||||
## Swarm controller
|
||||
|
||||
`services.hyperhive.swarm.controller.enable` runs the `swarm-controller`
|
||||
daemon on this host. **Off by default and deliberately not derived from
|
||||
`services.hyperhive.enable`**: a swarm has one controller, so enabling it
|
||||
is a statement about swarm topology, not about whether hyperhive is
|
||||
installed. Every hive runs `hive-c0re` (the agents on that host); one
|
||||
hive additionally runs this (what is true across hives).
|
||||
|
||||
What it serves, why it is a unix socket rather than a port, and the
|
||||
socket-directory constraint that governs where `socketPath` may point:
|
||||
[`swarm-controller/README.md`](../../swarm-controller/README.md).
|
||||
|
||||
## Cross-references
|
||||
|
||||
- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
|
||||
the `swarm.snapshotStore` option that points a hive at it
|
||||
- `docs/conventions.md` § Hive identity — env vars, qualified labels
|
||||
- `docs/matrix.md` — matrix federation, TLS cert auto-generation,
|
||||
firewall posture
|
||||
- `docs/web-ui/dashboard.md` § P33RS tab — dashboard surface
|
||||
- `docs/gateway.md` — nginx vhosts and the `.well-known/matrix/`
|
||||
auto-discovery scheme
|
||||
59
docs/swarm/services.md
Normal file
59
docs/swarm/services.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Swarm-wide services
|
||||
|
||||
Some things exist once per **swarm**, not once per hive: the forge, the
|
||||
matrix homeserver, the SSO provider, the CA. Two options say where the
|
||||
optional ones live, and everything else derives:
|
||||
|
||||
```nix
|
||||
services.hyperhive.enableAllLocalDefaults = true; # everything on this box
|
||||
# or, for a dedicated services host with hives elsewhere:
|
||||
services.hyperhive.swarm.enableRequiredServices = true;
|
||||
```
|
||||
|
||||
`enableAllLocalDefaults` is the all-on-one-box switch: it defaults both
|
||||
`swarm.enableRequiredServices` (the shared services run here) and
|
||||
`swarm.ca.autoConfigure` (the swarm CA is generated here). Each derived
|
||||
toggle can still be set on its own, so "all local except X" needs no
|
||||
further option.
|
||||
|
||||
**Both default to off**, and that is deliberate: a host cannot tell
|
||||
whether it is meant to be the swarm's service host, so this is an
|
||||
operator saying so rather than something inferred. With them off, a hive
|
||||
is a *client* of those services — it configures how to reach them and
|
||||
runs none of them.
|
||||
|
||||
The forge is the exception, and not because it is per-hive: it is
|
||||
swarm-wide but **not optional**, being the canonical store for the meta
|
||||
flake and every agent's config repo, so it deploys with hyperhive itself
|
||||
and has no `enable` to derive from anything.
|
||||
|
||||
### SSO (authelia)
|
||||
|
||||
One authelia per swarm, in a `swarm-authelia` container, at
|
||||
`auth.<hive-domain>`. Operator and agents are both subjects of the same
|
||||
provider, differentiated by roles and claims rather than by mechanism —
|
||||
there is one IdP and one auth path.
|
||||
|
||||
- **`swarm.authelia.enable`** — run the container here. Defaults from
|
||||
`swarm.enableRequiredServices`.
|
||||
- **`swarm.authelia.url`** — where clients are sent to authenticate.
|
||||
Present on **every** hive, defaulting to this host's own instance only
|
||||
when this module is the thing running it; otherwise `null`, and a hive
|
||||
joining someone else's swarm sets it explicitly. Null means "no SSO
|
||||
configured", and consumers say so rather than guessing an address.
|
||||
|
||||
The users database is written by swarm-controller, not by hand: agents
|
||||
are created and destroyed continuously, so the subject set is dynamic.
|
||||
This module only guarantees the file exists and parses, so authelia
|
||||
starts with nobody in it rather than failing to start — a provider with
|
||||
no subjects yet is the correct state before anything has provisioned
|
||||
them. Session and storage keys are generated in the container on first
|
||||
boot and never rotated automatically; replacing one invalidates data
|
||||
already written (sessions, the encrypted store), so that is an operator
|
||||
action.
|
||||
|
||||
Storage is local sqlite and the notifier writes to a file. Both are
|
||||
small-deployment choices, and the scope is the justification: redis
|
||||
buys shared session state across replicas and there is one instance;
|
||||
SMTP exists to mail humans, and provisioning here is programmatic.
|
||||
|
||||
|
|
@ -21,6 +21,7 @@
|
|||
./hive-priv.nix
|
||||
./hive-tls.nix
|
||||
./otel.nix
|
||||
./swarm-authelia.nix
|
||||
./swarm-ca.nix
|
||||
./swarm-controller.nix
|
||||
./swarm-snapshot-store.nix
|
||||
|
|
|
|||
|
|
@ -99,12 +99,19 @@ in
|
|||
options.services.hyperhive.swarm.matrix = {
|
||||
enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
default = config.services.hyperhive.swarm.enableRequiredServices;
|
||||
defaultText = lib.literalExpression "services.hyperhive.swarm.enableRequiredServices";
|
||||
description = ''
|
||||
Run hive-matrix — a private matrix-tuwunel homeserver (in a
|
||||
nixos-container) for hyperhive agents. Off by default while
|
||||
the integration phases in; flip to `true` once the operator
|
||||
has set `services.hyperhive.domain` and is ready to onboard agents.
|
||||
nixos-container) for hyperhive agents.
|
||||
|
||||
Matrix is a swarm-wide service — one homeserver, not one per
|
||||
hive — so this defaults from
|
||||
`services.hyperhive.swarm.enableRequiredServices`, which says
|
||||
the swarm's shared services live on this host. That is off by
|
||||
default, so this is off by default, as before. Set it directly
|
||||
to run the homeserver somewhere other than the host that holds
|
||||
the rest of the swarm's services.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -132,6 +132,32 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# The one switch for "everything runs on this box". Every autoconf
|
||||
# toggle in the tree defaults from it, so an all-local deployment is a
|
||||
# single line rather than one line per service that grew a toggle.
|
||||
options.services.hyperhive.enableAllLocalDefaults = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
example = true;
|
||||
description = ''
|
||||
Run the whole swarm on this host: the swarm-wide services
|
||||
(`services.hyperhive.swarm.enableRequiredServices`) and the swarm
|
||||
CA (`services.hyperhive.swarm.ca.autoConfigure`) all default from
|
||||
this, and anything autoconfigurable added later should too.
|
||||
|
||||
**Off by default, and that is the load-bearing part.** A swarm's
|
||||
services and its hives can live on different hosts, and a host has
|
||||
no way to tell which ones it is meant to be — so this is an
|
||||
operator saying "this is that box", never something inferred.
|
||||
Turn it on for a dev box or a single-hive swarm and get a working
|
||||
deployment with no further configuration; leave it off and every
|
||||
swarm-level artifact is operator-provided.
|
||||
|
||||
Each derived toggle can still be set explicitly to override this
|
||||
one, so "all local except X" needs no new option.
|
||||
'';
|
||||
};
|
||||
|
||||
# Whether this hive runs "ruthless" — with no root/manager agent at
|
||||
# all. Some hives don't want a root agent — see issue tracker
|
||||
# "scope concept: special agents".
|
||||
|
|
|
|||
268
nix/host-modules/swarm-authelia.nix
Normal file
268
nix/host-modules/swarm-authelia.nix
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# The swarm's SSO provider: one authelia for the whole swarm, in a
|
||||
# `swarm-authelia` nixos-container.
|
||||
#
|
||||
# Two halves, and only one of them is conditional:
|
||||
#
|
||||
# - the CLIENT pointer (`url`) exists on every hive, because a hive
|
||||
# that doesn't run authelia still has to know where to send people.
|
||||
# - the CONTAINER only exists where the swarm's shared services live
|
||||
# (`swarm.enableRequiredServices`, itself defaulting from
|
||||
# `enableAllLocalDefaults`). A hive is a client by default.
|
||||
#
|
||||
# Operator and agents are both subjects of the same provider,
|
||||
# differentiated by roles/claims rather than by mechanism — there is one
|
||||
# IdP and one auth path. The users store is therefore written by a
|
||||
# program (swarm-controller), not maintained by hand: agents are created
|
||||
# and destroyed continuously, so the subject set is *dynamic*. That is
|
||||
# also why the file backend is the right one here and not a placeholder
|
||||
# for LDAP: what makes a directory necessary is the size of the subject
|
||||
# set, and this deployment's is bounded by one swarm.
|
||||
#
|
||||
# Per-service integration — putting authelia's `auth_request` in front
|
||||
# of the gateway's existing `auth_basic` locations — is deliberately NOT
|
||||
# here. Standing an SSO provider up is reversible; cutting every
|
||||
# operator-facing vhost over to it is not.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
cfg = config.services.hyperhive.swarm.authelia;
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
swarmCfg = config.services.hyperhive.swarm;
|
||||
hyperhiveDomain = hyperhiveCfg.domain;
|
||||
|
||||
# Upstream's `services.authelia.instances.<name>` derives the unit,
|
||||
# user, group and StateDirectory from the instance name
|
||||
# (`authelia` + `-<name>`). Naming them here rather than repeating the
|
||||
# literal keeps the generator unit below and the module in step.
|
||||
instance = "swarm";
|
||||
unitName = "authelia-${instance}";
|
||||
stateDir = "/var/lib/${unitName}";
|
||||
|
||||
# Total on a null hive domain for the same reason the option defaults
|
||||
# below are: the required-domain assertion in hive-network.nix should
|
||||
# be what an operator sees, not a coercion error from here.
|
||||
cookieDomain = if hyperhiveDomain == null then "invalid" else hyperhiveDomain;
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.swarm.authelia = {
|
||||
enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = swarmCfg.enableRequiredServices;
|
||||
defaultText = lib.literalExpression "services.hyperhive.swarm.enableRequiredServices";
|
||||
example = true;
|
||||
description = ''
|
||||
Run the swarm's authelia in a `swarm-authelia` container on this
|
||||
host. Defaults from
|
||||
`services.hyperhive.swarm.enableRequiredServices` — a swarm has
|
||||
one SSO provider, and this says it lives here.
|
||||
|
||||
With it off, this hive is a *client*: `url` below still points
|
||||
at whoever runs it, and no container is created.
|
||||
'';
|
||||
};
|
||||
|
||||
package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
default = pkgs.authelia;
|
||||
defaultText = lib.literalExpression "pkgs.authelia";
|
||||
description = ''
|
||||
authelia package to run in the container. Defaults to
|
||||
nixpkgs's; override to pin a specific upstream.
|
||||
'';
|
||||
};
|
||||
|
||||
port = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 9091;
|
||||
description = ''
|
||||
TCP port authelia listens on. 9091 is upstream's default and
|
||||
sits outside hyperhive's claimed ranges (dashboard 7000, forge
|
||||
3000, matrix 8008, every agent in 8100..8999 via FNV-1a hash).
|
||||
'';
|
||||
};
|
||||
|
||||
domain = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
# Total on a null hive domain so the required-domain assertion in
|
||||
# hive-network.nix is the thing that fires; see the comment there.
|
||||
default = if hyperhiveDomain == null then "auth.invalid" else "auth.${hyperhiveDomain}";
|
||||
defaultText = lib.literalExpression ''"auth.''${services.hyperhive.domain}"'';
|
||||
example = "login.example.com";
|
||||
description = ''
|
||||
Public hostname for the SSO provider — the sub-domain shape the
|
||||
forge and matrix already use. Doubles as the cookie domain's
|
||||
host, so it must be the name browsers actually visit.
|
||||
'';
|
||||
};
|
||||
|
||||
url = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = if cfg.enable then "https://${cfg.domain}" else null;
|
||||
defaultText = lib.literalExpression ''if enable then "https://''${domain}" else null'';
|
||||
example = "https://auth.example.com";
|
||||
description = ''
|
||||
Base URL clients are sent to for authentication — the half of
|
||||
this module that exists on **every** hive, not just the one
|
||||
running the container.
|
||||
|
||||
Defaults to this host's own instance **only when this module is
|
||||
the thing running it**; in that case the URL is not a guess, it
|
||||
is where this module just put the container. Otherwise `null`,
|
||||
and a hive that federates with a swarm sets it explicitly to
|
||||
wherever the swarm's authelia lives. Null means "no SSO
|
||||
configured" and consumers say so rather than inventing an
|
||||
address — an endpoint baked in as a fallback is one that
|
||||
resolves cleanly and points at the wrong machine.
|
||||
'';
|
||||
};
|
||||
|
||||
usersFile = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "${stateDir}/users.yml";
|
||||
defaultText = lib.literalExpression ''"/var/lib/authelia-swarm/users.yml"'';
|
||||
description = ''
|
||||
Path (inside the container) of authelia's file users database.
|
||||
|
||||
Written by swarm-controller, not by hand: agents come and go
|
||||
continuously, so the subject set is dynamic and belongs to a
|
||||
program. This module only guarantees the file *exists* and is
|
||||
valid YAML at first boot, so authelia starts with no subjects
|
||||
rather than failing to start — a provider with nobody in it yet
|
||||
is the correct state before anything has provisioned users.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf (hyperhiveCfg.enable && cfg.enable) {
|
||||
containers.swarm-authelia = {
|
||||
autoStart = true;
|
||||
ephemeral = false;
|
||||
# Shared host netns, like the forge and matrix containers: the
|
||||
# gateway reaches authelia at 127.0.0.1:<port>.
|
||||
privateNetwork = false;
|
||||
|
||||
config =
|
||||
{ ... }:
|
||||
{
|
||||
system.stateVersion = "26.05";
|
||||
|
||||
# This container shares the host netns, so its own
|
||||
# firewall.service would rewrite the HOST ruleset at every
|
||||
# boot. The host firewall owns all filtering.
|
||||
networking.firewall.enable = false;
|
||||
# Keep the host-copied /etc/resolv.conf intact — resolvconf's
|
||||
# host-tracking would regenerate it to an empty file, since
|
||||
# the host's copy doesn't cross the boundary after start.
|
||||
networking.resolvconf.enable = lib.mkForce false;
|
||||
|
||||
# authelia's own secrets, generated in-container on first
|
||||
# boot. They are jwt/session/storage keys — nothing outside
|
||||
# this container ever reads them, which is what makes
|
||||
# in-container generation right rather than merely easier.
|
||||
# (hive-matrix generates its token host-side only because
|
||||
# hive-c0re has to read that one.)
|
||||
#
|
||||
# Same `User`/`Group`/`StateDirectory` as the authelia unit,
|
||||
# so systemd creates the directory owned by the account that
|
||||
# has to read the files and this unit can write nowhere else.
|
||||
# No chown, no mode juggling: authelia opens these paths
|
||||
# itself, as its own user, under `PrivateUsers=true`.
|
||||
systemd.services."${unitName}-secrets" = {
|
||||
description = "Generate authelia's secrets on first boot";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
before = [ "${unitName}.service" ];
|
||||
requiredBy = [ "${unitName}.service" ];
|
||||
path = [ pkgs.coreutils ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
User = unitName;
|
||||
Group = unitName;
|
||||
StateDirectory = unitName;
|
||||
StateDirectoryMode = "0700";
|
||||
UMask = "0077";
|
||||
SyslogIdentifier = "${unitName}-secrets";
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
# Each is generated once and never rotated here: the
|
||||
# session and storage keys are load-bearing for data
|
||||
# already written (sessions, the encrypted store), so
|
||||
# replacing one is an operator action, not a boot action.
|
||||
for f in jwt session storage-encryption; do
|
||||
p=${lib.escapeShellArg stateDir}/"$f".key
|
||||
if [ ! -s "$p" ]; then
|
||||
head -c 64 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$p"
|
||||
echo "generated $p"
|
||||
fi
|
||||
chmod 0600 "$p"
|
||||
done
|
||||
|
||||
# A users database that exists and parses, with nobody in
|
||||
# it. authelia refuses to start without one, and the
|
||||
# alternative to an empty file is a placeholder account —
|
||||
# which is a credential nobody meant to create.
|
||||
users=${lib.escapeShellArg cfg.usersFile}
|
||||
if [ ! -s "$users" ]; then
|
||||
echo "users: {}" > "$users"
|
||||
echo "seeded empty users database at $users"
|
||||
fi
|
||||
chmod 0600 "$users"
|
||||
'';
|
||||
};
|
||||
|
||||
services.authelia.instances.${instance} = {
|
||||
enable = true;
|
||||
package = cfg.package;
|
||||
|
||||
secrets = {
|
||||
jwtSecretFile = "${stateDir}/jwt.key";
|
||||
sessionSecretFile = "${stateDir}/session.key";
|
||||
storageEncryptionKeyFile = "${stateDir}/storage-encryption.key";
|
||||
};
|
||||
|
||||
# Small-deployment defaults, and the scope is the
|
||||
# justification: one swarm, one authelia, no replicas.
|
||||
# - file users backend, written by swarm-controller
|
||||
# - local sqlite storage: redis buys shared session state
|
||||
# across replicas, and there is one instance
|
||||
# - filesystem notifier: SMTP is for mailing humans, and
|
||||
# provisioning is programmatic; a file is honest about
|
||||
# where those messages go instead of implying a mail path
|
||||
settings = {
|
||||
theme = "dark";
|
||||
server.address = "tcp://127.0.0.1:${toString cfg.port}";
|
||||
log.level = "info";
|
||||
|
||||
authentication_backend.file.path = cfg.usersFile;
|
||||
|
||||
access_control.default_policy = "one_factor";
|
||||
|
||||
# The cookie domain is the hive's domain, NOT authelia's
|
||||
# own host: the session cookie has to be sent to the apps
|
||||
# being protected (`<hive>`, `forge.<hive>`,
|
||||
# `matrix.<hive>`), and a cookie scoped to `auth.<hive>`
|
||||
# reaches none of them. authelia enforces the relationship
|
||||
# from the other side too — `authelia_url` must be a
|
||||
# sub-domain of `domain`, so setting both to the same host
|
||||
# fails validation at startup rather than at first login.
|
||||
session.cookies = [
|
||||
{
|
||||
domain = cookieDomain;
|
||||
authelia_url = "https://${cfg.domain}";
|
||||
}
|
||||
];
|
||||
|
||||
storage.local.path = "${stateDir}/db.sqlite3";
|
||||
notifier.filesystem.filename = "${stateDir}/notification.txt";
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
@ -52,12 +52,19 @@ in
|
|||
options.services.hyperhive.swarm.ca = {
|
||||
autoConfigure = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
default = hyperhiveCfg.enableAllLocalDefaults;
|
||||
defaultText = lib.literalExpression "services.hyperhive.enableAllLocalDefaults";
|
||||
example = true;
|
||||
description = ''
|
||||
Run the whole swarm CA on this one host: generate the swarm
|
||||
root when it is missing, and issue this hive's CA under it.
|
||||
|
||||
Defaults from `services.hyperhive.enableAllLocalDefaults`, the
|
||||
all-on-one-box switch — which is off, so this is off, and the
|
||||
paragraph below still describes what a hive does by default.
|
||||
Set it directly to run the CA on a host that is not otherwise
|
||||
all-local.
|
||||
|
||||
**Off by default, deliberately.** A swarm's services and its
|
||||
hives can live on different hosts, and this host has no way to
|
||||
tell whether it is the one holding the root — so the swarm CA
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@
|
|||
Generate with `openssl x509 -noout -fingerprint -sha256`,
|
||||
then strip the colons and prepend `sha256:`. A malformed
|
||||
value is ignored with a warning rather than weakening
|
||||
trust. See docs/swarm.md for the full recipe.
|
||||
trust. See docs/swarm/README.md for the full recipe.
|
||||
|
||||
Scopes only to hive-c0re's own peer HTTPS checks — it does
|
||||
NOT help Matrix federation (tuwunel validates against its
|
||||
|
|
@ -62,7 +62,7 @@
|
|||
inside the hive (agents + the Matrix container), never the
|
||||
host system trust store. Mutually complementary with
|
||||
`certFingerprint`; set `caCert` for the federation case. See
|
||||
docs/swarm.md.
|
||||
docs/swarm/README.md.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -126,6 +126,37 @@
|
|||
'';
|
||||
};
|
||||
|
||||
# "The swarm-wide services run HERE." A swarm has one forge, one
|
||||
# matrix, one SSO — this says this host is where they live. The
|
||||
# OPTIONAL ones (matrix, authelia) default their own enable from it,
|
||||
# rather than the operator enabling them one at a time; the forge has
|
||||
# no enable to derive because it is not optional.
|
||||
options.services.hyperhive.swarm.enableRequiredServices = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = config.services.hyperhive.enableAllLocalDefaults;
|
||||
defaultText = lib.literalExpression "services.hyperhive.enableAllLocalDefaults";
|
||||
example = true;
|
||||
description = ''
|
||||
Host the swarm's shared services on this hive. The services that
|
||||
exist once per swarm rather than once per hive and are *optional*
|
||||
— the matrix homeserver, the SSO provider — default their
|
||||
`enable` from this, so a swarm's service host is declared in one
|
||||
place.
|
||||
|
||||
The forge is swarm-wide too but has no `enable` to derive: it is
|
||||
the canonical store for the meta flake and every agent's config
|
||||
repo, so it deploys with hyperhive itself and is not optional.
|
||||
|
||||
Defaults from `services.hyperhive.enableAllLocalDefaults` (off),
|
||||
which is the all-on-one-box switch. Set it directly to run the
|
||||
swarm's services on a host that is not otherwise all-local — a
|
||||
dedicated services box with hives elsewhere is exactly that shape.
|
||||
|
||||
With it off, this hive is a *client* of those services: it still
|
||||
configures how to reach them, it just doesn't run them.
|
||||
'';
|
||||
};
|
||||
|
||||
options.services.hyperhive.swarm.snapshotStore = {
|
||||
address = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
|
|
|
|||
Loading…
Reference in a new issue