199afa41c8
`glue-matrix-bao-token.nix` has read `secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but nothing ever wrote that path. The store was empty in every deployment, so every read degraded to "keep what activation minted" and each hive stayed the origin of a value the swarm has to agree on — two hives never converged. `swarm-secret-publish` is now the producer. It already holds a store identity, already writes under the hive prefix, and already runs per hive in the roster, so the mint is a third loop beside the two OIDC copies rather than a second shape of this unit. Idempotence comes from a record of its own, not from the store: this principal is granted `create`/`update` with no `read`, so it cannot ask whether a hive already has a token. It keeps what it minted under `StateDirectory=` (0700 dir, 0600 file) and mints only when that file is missing or empty; the `put` runs every time, because re-putting the same bytes changes nothing for a reader while a mint whose publish failed must not be left as a token this host holds and no hive can reach. The token never becomes a nix literal and never reaches argv: the mint redirects into a file, and the publish hands bao `value=@<path>` so bao opens it itself — the same handling the OIDC loops use. `hive-matrix.nix`'s activation mint stays as the genuine first-boot fallback. It already fires only when the token file is absent, so it cannot clobber a value the store delivered; `hs_token` has no swarm half and is still minted there for real. Refs #4402
332 lines
33 KiB
Markdown
332 lines
33 KiB
Markdown
# Swarm secrets: what exists, and where each one lives
|
|
|
|
A swarm's credentials are generated in three different places and read in a
|
|
fourth, so "where does this file go" has a different answer per deployment.
|
|
This page is that answer, one row per secret.
|
|
|
|
Two rules run through all of it.
|
|
|
|
**Private key material and access tokens are paths, never values.** Every option
|
|
carrying one takes a file path (`*File`), because a literal written into a nix
|
|
expression ends up in the nix store — world-readable and permanent. No
|
|
option in this tree accepts one inline, and adding one would be a leak rather
|
|
than a convenience.
|
|
|
|
The rule is about what must stay secret, not about credentials generally.
|
|
**Public material is a value**: a certificate, or a public nkey like
|
|
`deploy.nats.calloutUserPublicKey`, is published to every client that connects,
|
|
so the store is a perfectly good place for it.
|
|
|
|
**The generator and the reader typically live in different containers.** They share
|
|
the host's network namespace, which makes them feel co-located, but their
|
|
filesystem roots are separate. That's why delivery is a **host-side copy rather
|
|
than a bind mount**: `nixos-container` refuses to start when a bind source is
|
|
missing, and a secret minted on another container's first boot doesn't exist
|
|
yet. Binding it would make one container wait on a file that waits on a
|
|
container that starts after it.
|
|
|
|
## Topologies, by who places secrets
|
|
|
|
Read every row below against one of these. This is a different cut
|
|
from the [deployment shapes](services.md#deployment-shapes) --- those
|
|
say _where services run_, these say _who is responsible for a secret
|
|
file being there_ --- the two lists don't line up one-to-one, and
|
|
neither is a renaming of the other.
|
|
|
|
| topology | what it means | who places secrets |
|
|
| ------------------ | ---------------------------------------------------------- | --------------------------------------------------------------------------- |
|
|
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — each secret is generated where it's read, or copied by a host unit |
|
|
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
|
|
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
|
|
|
|
## Swarm-level — one of each per swarm
|
|
|
|
| secret | generated by | lives at | hive elsewhere |
|
|
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
|
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
|
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
|
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
|
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
|
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
|
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
|
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
|
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
|
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
|
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
|
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
|
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
|
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
|
|
|
Authelia mints the three keys for itself, in-container, precisely
|
|
because nothing outside that container ever reads them. **That's the test worth
|
|
applying to any secret added here** — and the client secret's plaintext half is
|
|
the one row that fails it, which is the entire reason a delivery step exists.
|
|
|
|
Two telemetry collectors exist, and they land on opposite sides of that test.
|
|
|
|
The **hive's** collector needs no delivery step. It authenticates to the swarm's
|
|
collector as its own hive, and it's a host unit rather than a container, so on
|
|
an all-local swarm it reads authelia's file where it lies and no second copy is
|
|
made. On any other topology it's an ordinary "operator provides the file"
|
|
case — see `services.hyperhive.otel.clientSecretFile`.
|
|
|
|
The **swarm's** collector does need one, and its delivery route is the fourth
|
|
store reader — the same shape as Grafana's, described in full below. In short:
|
|
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
|
out of the store on every host that runs the collector and holds a store
|
|
identity, whether authelia is a network away or in the container next door.
|
|
The client it reads for is registered unconditionally, the same call
|
|
`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a
|
|
bearer-authz client with no audience, and the push audiences supply one even
|
|
for a collector with nothing published to scrape.
|
|
|
|
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
|
|
where an absent OIDC block is a container with no way in at all, a collector
|
|
with nothing to authenticate with still receives every hive's telemetry —
|
|
only its own pushes to the stores go out unauthenticated and get refused
|
|
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
|
`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice
|
|
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
|
|
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
|
|
uses for its non-optional one. The secret can still be delivered by hand, at
|
|
`services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this
|
|
unit never reaches.
|
|
|
|
### Minting the queue's callout nkeys
|
|
|
|
`deploy.nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
|
starts. It's on by default only under `singleHostSwarm` — the one
|
|
topology where the queue, its responder and the operator are the same person. On
|
|
every other topology, mint them yourself:
|
|
|
|
```
|
|
nk -gen user > callout-user.seed # the responder's own identity
|
|
nk -gen account > issuer.seed # signs the user JWTs it hands out
|
|
nk -inkey callout-user.seed -pubout # → calloutUserPublicKey
|
|
nk -inkey issuer.seed -pubout # → calloutIssuerPublicKey
|
|
```
|
|
|
|
Keep both seeds at `0600` and name them in `deploy.nats.calloutUserSeedFile` /
|
|
`deploy.nats.calloutIssuerSeedFile`. Possession of the **issuer** seed is the authority to
|
|
admit anyone to the queue, so it belongs wherever the responder runs and nowhere
|
|
else.
|
|
|
|
A hive that sets neither the public keys nor `autoGenerateCallout` fails at
|
|
eval, naming the option it wants. That's deliberate: a queue that started
|
|
without them would accept `CONNECT {"user":"auth"}` from anyone sharing the
|
|
host's network namespace, and nothing would look wrong until somebody connected.
|
|
|
|
**All four or none** — the seed paths are required too, not just the public
|
|
keys. They're two halves of the same pair: the server verifies with the public
|
|
half, the responder signs with the private one. Supplying only the public keys
|
|
used to pass eval and leave the queue with an auth-callout nobody answers, which
|
|
refuses every client rather than degrading — and a refusal reaches the client as
|
|
a timeout, so the symptom is every consumer hanging with nothing logged.
|
|
|
|
One consequence of the generated path worth knowing before you debug it: with
|
|
`autoGenerateCallout` set, the queue assembles its config at boot rather than at
|
|
build time, so a malformed one surfaces when the container starts instead of
|
|
when the system builds. The server names the offending file and refuses to run.
|
|
|
|
## Hive-level — one of each per hive
|
|
|
|
| secret | generated by | lives at |
|
|
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
|
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
|
|
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
|
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
|
|
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
|
|
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
|
|
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
|
|
|
|
Both delivery units wait for authelia's first boot to mint the secret — a
|
|
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
|
|
produces a service whose login button always fails, which is a symptom many
|
|
layers from its cause.
|
|
|
|
The store's **first reader** is the matrix appservice token, and it's worth
|
|
saying why that one: it's an opaque 32-byte value with no second file and no
|
|
format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so
|
|
starting there would have meant debugging "can a reader authenticate and get
|
|
bytes back" and "is authelia's file format right" at once, with an
|
|
SSO outage as the failure mode.
|
|
|
|
`glue-matrix-bao-token.nix` fetches it, writes the file `hive-matrix.nix`
|
|
already reads, and then runs that module's own renderer to re-stamp the
|
|
appservice registration naming the token — so the homeserver never learns the
|
|
store exists. The re-render isn't housekeeping: the token is half an
|
|
agreement, and a registration carrying the previous value authenticates
|
|
nobody. Every failure path — no such key, sealed store, unreachable store,
|
|
empty value — leaves the locally minted token in place, so a hive with no store
|
|
behaves exactly as it did before.
|
|
|
|
The store path is `swarm/hives/<hive>/matrix/appservice-token`, and the
|
|
**producer is `swarm-secret-publish`** — the same unit that copies authelia's
|
|
OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s
|
|
it there, so the store is the source of truth and every hive converges on the
|
|
value it holds. The hive's own activation mint is still there, but it's a
|
|
first-boot fallback now: it fires only when the token file is absent, and the
|
|
reader overwrites whatever it produced.
|
|
|
|
The mint is **idempotent by keeping its own record**, not by asking the store.
|
|
The publisher's grant is `create`/`update` under `swarm/hives/*` with no
|
|
`read` — write-only on purpose, so a file-copier can't recover every hive's
|
|
credentials — which means it can't check whether a hive already has a token.
|
|
Instead it keeps the value it minted under its `StateDirectory`,
|
|
`/var/lib/swarm-secret-publish/matrix-appservice-token/<hive>` (`0700` dir,
|
|
`0600` file), and mints only when that file is missing or empty. The `put`
|
|
itself runs every time: re-putting the same bytes changes nothing for any
|
|
reader, while a mint whose publish failed must not be left as a token this
|
|
host holds and no hive can reach.
|
|
|
|
Lose the state directory and the next run mints once more and republishes.
|
|
That rotates the token, which readers pick up on their next start — nothing
|
|
that already registered breaks, because the token authenticates the
|
|
appservice rather than living inside any account it created.
|
|
|
|
The path was `…/matrix/registration-token` while the homeserver still took a
|
|
shared registration secret; a value left at the old path is read by nothing.
|
|
|
|
The **second reader** is the agent containers' queue credential:
|
|
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
|
|
the client id it authenticates, because that's how a queue client reads them —
|
|
the secret by path, the id by value. No local fallback exists, and none is
|
|
possible, so absent files mean this hive's agents don't connect, which is what
|
|
a swarm looks like before the publisher on the authelia host has run. The
|
|
reader runs before `hive-c0re.service` and is wanted (not required) by it, so
|
|
an agent container never renders ahead of the credential; an unreachable store
|
|
delays the daemon's start rather than failing it.
|
|
|
|
That credential still has one hop left, because the reader of it lives inside
|
|
an agent container. **It crosses as a systemd credential, not as a bind
|
|
mount**, and the mode decides that: the secret is `root:0600` and a harness
|
|
runs as its own unprivileged agent user, so a bind would deliver a file that
|
|
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
|
|
it needs no read access to either — and hands them to
|
|
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
|
|
Inside, `hive-agent.service` names the same two ids in `LoadCredential=` and
|
|
reads them out of its own `$CREDENTIALS_DIRECTORY`, owned by the agent user
|
|
and by nobody else. `hive-c0re` never reads the bytes at any point: it runs as
|
|
`hive-core`, which is the reason a copy wasn't an option either. When the
|
|
files aren't there the daemon forwards nothing and says so in its journal, and
|
|
the harness logs that it has no queue coordinates — the same absent-and-legal state, twice,
|
|
rather than a container that refuses to start.
|
|
|
|
The **third reader** is Grafana's OIDC client secret, and it's the first one
|
|
that's a _swarm service's_ own credential rather than a hive's. It has exactly
|
|
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
|
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
|
|
authelia's host writes that path whether the reader is a network away or in the
|
|
container next door.
|
|
|
|
That's a deliberate refusal of the shortcut. Copying the plaintext
|
|
straight out of authelia's tree when the two happen to share a host is one
|
|
round trip cheaper and costs a second delivery unit, a second way for the file
|
|
to be wrong, and a gate deciding between them — and the gate is the expensive
|
|
part, because whatever it's wrong about is an outage nobody can read. The store
|
|
exists so a host holds **one** out-of-band secret, its client certificate, and
|
|
reads everything else with it.
|
|
|
|
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
|
|
running Grafana requires `swarm.authelia.url` (its local login form is disabled
|
|
unconditionally, so SSO isn't a feature of some topologies — it's the only way
|
|
in), and it requires this host's `deploy.bao.clientCertFile` /
|
|
`clientKeyFile`. Each refusal names the option to set. Both used to be silent:
|
|
a null URL dropped the OIDC block, a missing leaf produced a warning, and
|
|
either one left a Grafana with no SSO and no password box — nothing failed, so
|
|
nothing said anything. Registration is separate and stays where authelia is
|
|
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
|
so it's declared on the host running authelia, which isn't necessarily the host
|
|
running Grafana.
|
|
|
|
The **fourth reader** is the swarm collector's own OIDC client secret, the
|
|
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
|
runs the collector, and registration lives separately in
|
|
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
|
differs is what an absent credential means: the collector still receives
|
|
telemetry with none, so `swarm-otel.nix` doesn't assert
|
|
`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it
|
|
doesn't render the reading unit without them, the shape
|
|
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
|
|
their own optional reads.
|
|
|
|
A service's secret is one value for the whole swarm rather than one per hive, so
|
|
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
|
whole. That's because nothing in a swarm's configuration records which hive runs
|
|
a given service — placement is a `deploy.*` fact, per-host by definition — so
|
|
there is nothing to scope the grant to. Every hive can therefore read every swarm
|
|
service's client secret; that's the cost of a shape where the service's host has
|
|
no store identity of its own to present, and it's stated in
|
|
`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves
|
|
as it does for the other readers: a store that says "not here" leaves the
|
|
file alone and says so, since there is no such thing as a locally valid OIDC
|
|
client secret to stand in.
|
|
|
|
⚠️ **Service↔store mTLS is its own trust domain.** A credential you must
|
|
already hold to authenticate can't be fetched from the thing it authenticates
|
|
you to, so the store's identity can't come from an authority the store
|
|
distributes — which excludes the hive CA and the swarm CA both, and has nothing
|
|
to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
|
|
mints a CA that signs exactly two things, the store's server certificate and a
|
|
reader's client certificate, and distributes nothing. A deployment with a real
|
|
internal CA deletes that file and names its own paths in
|
|
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
|
|
A hive that reads from a store on **another** machine names the reader's half —
|
|
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand.
|
|
It's the one credential that can't come out of the store, being what opens it;
|
|
everything else a hive needs does.
|
|
|
|
## How a reader reaches the store
|
|
|
|
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
|
|
the host that runs the store that name resolves twice. `/etc/hosts` answers a
|
|
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
|
|
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
|
|
and the network namespace that asked decides which half of it answers.
|
|
|
|
The container's half goes through nginx and **still doesn't terminate TLS**. It
|
|
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
|
|
ClientHello and splices the rest of the connection through byte for byte, so
|
|
openbao completes the handshake itself. A vhost would decrypt here instead, and
|
|
the client certificate — the store's whole authentication — would stop at nginx,
|
|
leaving openbao seeing one client for every hive in the swarm. That's why the
|
|
store has no vhost and gets a passthrough: not an exception to the no-vhost
|
|
rule, the one shape that keeps it.
|
|
|
|
The stream listener binds the bridge IP rather than every address, because
|
|
openbao already holds loopback on that port in the same network namespace and a
|
|
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
|
|
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
|
|
on the bridge firewall and nowhere else.
|
|
|
|
Reaching the port grants nothing by itself: openbao answers nothing without a
|
|
client certificate signed by `deploy.bao.clientCaFile`. The passthrough carries
|
|
whichever certificate the reader presents, unchanged.
|
|
|
|
## The constraint that decides where the root lives
|
|
|
|
A hive CA carries `nameConstraints=permitted;DNS:<hive domain>`, and **a swarm
|
|
service name is a sibling of the hive domain rather than a child** — `forge.<swarm>`
|
|
next to `<hive>.<swarm>`. A hive CA can't issue a certificate for a swarm
|
|
service. Not by policy: by construction, and openssl enforces it.
|
|
|
|
Whatever holds the swarm root is therefore what makes swarm-service certificates
|
|
possible at all. Two things follow:
|
|
|
|
- **The root's private key is a runtime file and must never enter the nix
|
|
store**, so nothing build-time can name it — `security.pki.certificateFiles` is
|
|
read when the system is built, and is the wrong tool here. Trust reaches
|
|
containers through a bind-mounted bundle assembled at boot instead.
|
|
- **On any topology other than all-local, placing that key is an operations
|
|
decision**, not something this module tree makes for you. A hive that hosts no
|
|
swarm services needs only the root's _cert_, to trust what others issue.
|
|
|
|
## Adding a secret
|
|
|
|
State three things, in the row you add above: **who mints it**, **which
|
|
container reads it**, and **what happens when they differ**. If they differ, it
|
|
needs a delivery unit, and the unit copies — it doesn't bind.
|