vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-nats: fetch the callout xkey before the server starts and seal callout traffic

Browse source 
swarm-bao-nats-callout-xkey runs on the queue's host before the
swarm-nats container, under the responder's store leaf: it runs
`swarm-nats-auth ensure-xkey`, reads the public half back from the
store, and writes it where the container's runtime config dir is. On a
change it restarts the server.

nats-server now always starts through the runtime wrapper, in manual
mode too. A privileged ExecStartPre renders auth_callout with the xkey
(and, in auto mode, the two generated nkeys) at every start, and fails
the server when no xkey has been fetched. The auto-mode key unit now
only mints seeds and writes their public halves.

The responder is wanted by nats.service, so a server started late, once
its key arrived, brings the responder up with it.
This commit is contained in:
atlas 2026-10-05 21:25:40 +02:00 • committed by mara
commit 9a241a1df1
4 changed files with 362 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

49
docs/swarm/secrets.md
View file

@ -56,6 +56,7 @@ neither is a renaming of the other.
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-xkey`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
@ -146,10 +147,50 @@ used to pass eval and leave the queue with an auth-callout nobody answers, which
refuses every client rather than degrading — and a refusal reaches the client as
a timeout, so the symptom is every consumer hanging with nothing logged.
One consequence of the generated path worth knowing before you debug it: with
`autoGenerateCallout` set, the queue assembles its config at boot rather than at
build time, so a malformed one surfaces when the container starts instead of
when the system builds. The server names the offending file and refuses to run.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout xkey below exists only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`. Only the
`swarm-nats-auth` role reads it, and it may create the path but never
replace it.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`. Every token
the store issues may read `swarm/public/`; see
[`bao.md`](bao.md#what-every-token-may-read).
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-xkey` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
When the store is down at boot, the fetch unit fails and retries for a day.
The server starts with the key fetched last time, or not at all on a first
boot. The retry that lands the file starts or restarts the server, and the
responder comes up with it.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back. To replace the key, delete both paths, then restart
`swarm-bao-nats-callout-xkey`.
## Hive-level — one of each per hive