hyperhive/docs/swarm/secrets.md

Swarm secrets: what exists, and where each one lives

A swarm generates its credentials in three different places and reads them in a fourth, so "where does this file go" has a different answer per deployment. This page is that answer, one row per secret.

Two rules run through all of it.

Private key material and access tokens are paths, never values. Every option carrying one takes a file path (*File), because a literal written into a nix expression ends up in the nix store — world-readable and permanent. No option in this tree accepts one inline, and adding one would be a leak rather than a convenience.

The rule is about what must stay secret, not about credentials generally. Public material is a value: a certificate, or a public nkey like deploy.nats.calloutUserPublicKey, reaches every client that connects, so the store is a perfectly good place for it.

The generator and the reader typically live in different containers. They share the host's network namespace, which makes them feel co-located, but their filesystem roots are separate. That's why delivery is a host-side copy rather than a bind mount: nixos-container refuses to start when a bind source is missing, and a secret minted on another container's first boot doesn't exist yet. Binding it would make one container wait on a file that waits on a container that starts after it.

Topologies, by who places secrets

Read every row below against one of these. This is a different cut from the deployment shapes --- those say where services run, these say who is responsible for a secret file being there --- the two lists don't line up one-to-one, and neither is a renaming of the other.

topology	what it means	who places secrets
all-local	one host runs the swarm's shared services and its own hive	nobody — the reader itself generates each secret, or a host unit copies it
swarm-managed	the swarm's services run on a host with swarmctl	swarmctl writes what it owns; the rest is still generated in place
hive elsewhere	a hive that federates with a swarm it doesn't host	the operator provides the file and names it in config

Swarm-level — one of each per swarm

secret	generated by	lives at	hive elsewhere
swarm root CA cert	swarm-ca.nix first-boot unit, when the operator sets autoConfigure	/var/lib/swarm-ca/root.pem	operator copies the cert in; it's public
swarm root CA key	same unit	/var/lib/swarm-ca/root-key.pem, 0600	stays on whichever host holds it — see the constraint below
swarm-services root CA (cert + key)	swarm-bao.nix's granting unit, inside the store's pki mount	the key never leaves the mount; the cert is published at /var/lib/swarm-bao-services-pki/services-root.pem (0644) and at /var/lib/hive-tls/swarm-services-root.pem on every hive	nothing to copy — every hive receives the cert with the leaf it asks the store for
authelia session, JWT and storage-encryption keys	authelia's first-boot unit, in-container	/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key	generated in place; nothing outside that container reads them
authelia OIDC HMAC key	same unit	/var/lib/authelia-swarm/oidc-hmac.key	same
authelia OIDC issuer key (RSA)	same unit	/var/lib/authelia-swarm/oidc-issuer.key	same — relying parties verify against the public half at /jwks.json
OIDC client secret, plaintext half	authelia crypto hash generate --random	/var/lib/authelia-swarm/oidc-clients/<id>.secret	operator provides the file and names it in whichever option reads it — sso.clientSecretFile for a service, otel.clientSecretFile for the hive's telemetry collector
OIDC client secret, digest half	the same mint	oidc-clients/<id>.digest	authelia's own half; merged at runtime via settingsFiles
the swarm collector's copy of its OIDC secret	swarm-bao-otel-oidc.service reads it out of the swarm secret store, on every host that runs the collector and holds a store identity	/var/lib/swarm-otel-oidc/<id>.secret inside the swarm-otel container	same unit, same path — one route, co-located or not. A collector with no store identity has clientSecretFile == null, its already-supported unauthenticated-push degrade — see below
Grafana's copy of its OIDC secret	swarm-bao-grafana-oidc.service reads it out of the swarm secret store, on every host that runs Grafana	/var/lib/grafana-oidc/<id>.secret inside the swarm-grafana container	same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below
authelia subject store	swarmctl and swarm-authelia-bridge	users.yml — one file, read and written by both	swarmctl, on the host that runs authelia
wireguard private key	the operator — wg genkey	whatever deploy.wireguard.privateKeyFile names	always operator-provided; nothing generates this for you
queue auth-callout nkeys (user seed + account seed)	swarm-nats-callout-keys first-boot unit, when the operator sets deploy.nats.autoGenerateCallout	/var/lib/swarm-nats-callout/{callout-user,issuer}.seed, 0600	operator mints both with nk and names them in deploy.nats.calloutUserSeedFile / deploy.nats.calloutIssuerSeedFile
the secret store's own contents	openbao, on first bao operator init — an operator action, not a unit	inside the swarm-bao container, at its own /var/lib/openbao, kept across rebuilds by ephemeral = false. ⚠️ Not a host path: nixos-container destroy swarm-bao takes the raft data with it, so back up the container's tree, not /var/lib/. Only the store's TLS material (/var/lib/swarm-bao-tls) and its PKCS11 token (/var/lib/swarm-bao-token) are host-level	n/a — there is one store; a hive elsewhere is a client of it and holds none of this
the secret store's unseal material	the HSM/TPM under deploy.bao.seal = "pkcs11"; openbao itself under "shamir"	in the token; or held by whoever ran bao operator init, which is what "shamir" means and why it's stated rather than inferred	n/a — only the host running the store seals anything

Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. That's the test worth applying to any secret added here — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists.

Three telemetry collectors exist, and they don't land on the same side of that test.

The hive's collector needs no delivery step. It authenticates to the swarm's collector as its own hive, and it's a host unit rather than a container, so on an all-local swarm it reads authelia's file where it lies and nothing makes a second copy. On any other topology it's an ordinary "operator provides the file" case — see services.hyperhive.otel.clientSecretFile.

The swarm's collector does need one, and its delivery route is the fourth store reader — the same shape as Grafana's, described in full below. In short: swarm-bao-otel-oidc.service reads swarm/services/<client-id>/oidc/client out of the store on every host that runs the collector and holds a store identity, whether authelia is a network away or in the container next door. The same call glue-grafana-oidc-client.nix makes for Grafana's registers the client it reads for unconditionally: authelia refuses a bearer-authz client with no audience, and the push audiences supply one even for a collector with nothing published to scrape.

⚠️ A collector with no store identity isn't refused. Unlike Grafana, where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — only its own pushes to the stores go out unauthenticated and get refused there. swarm-bao-otel-oidc.service therefore doesn't render at all without deploy.bao.otelOidcClientCertFile / otelOidcClientKeyFile, the same choice glue-matrix-bao-token.nix and glue-queue-agent-credential.nix make for their own optional readers, rather than the hard assertion swarm-grafana.nix uses for its non-optional one. The operator can still deliver the secret by hand, at services.hyperhive.deploy.swarm-otel.clientSecretFile, on a collector this unit never reaches.

The secret store's own collector — the forwarder inside the swarm-bao container — needs a delivery step too, and it takes the same route with one principal of its own: swarm-bao-forwarder-oidc.service reads swarm/services/<client-id>/oidc/client under this host's certificate and lands it in the container's tree, where LoadCredential hands it to the collector. The client id is its own (services.hyperhive.swarm.bao.otel.clientId), registered by glue-swarm-bao-otel-oidc-client.nix wherever authelia runs, and the swarm collector admits it on a receiver of its own — one identity per principal, so this forwarder borrows neither the swarm collector's client nor swarm-controller's.

⚠️ This one has no degraded mode, and the difference is structural. The route it exports over accepts an authenticated request and nothing else, so a forwarder without the secret has nothing to fall back to: the unit fails and retries, and the collector refuses to start until the file arrives rather than exporting into a 401 while reporting healthy. Its ordering differs from the unit above it for the same structural reason — it reads the store inside the container it delivers into, so it runs after that container rather than before it, and only a first boot ever waits.

Minting the queue's callout nkeys

deploy.nats.autoGenerateCallout mints both keypairs on the host before the queue starts. It's on by default only under singleHostSwarm — the one topology where the queue, its responder and the operator are the same person. On every other topology, mint them yourself:

nk -gen user    > callout-user.seed     # the responder's own identity
nk -gen account > issuer.seed           # signs the user JWTs it hands out
nk -inkey callout-user.seed -pubout     # → calloutUserPublicKey
nk -inkey issuer.seed       -pubout     # → calloutIssuerPublicKey

Keep both seeds at 0600 and name them in deploy.nats.calloutUserSeedFile / deploy.nats.calloutIssuerSeedFile. Possession of the issuer seed is the authority to admit anyone to the queue, so it belongs wherever the responder runs and nowhere else.

A hive that sets neither the public keys nor autoGenerateCallout fails at eval, naming the option it wants. That's deliberate: a queue that started without them would accept CONNECT {"user":"auth"} from anyone sharing the host's network namespace, and nothing would look wrong until somebody connected.

All four or none — you must supply the seed paths too, not just the public keys. They're two halves of the same pair: the server verifies with the public half, the responder signs with the private one. Supplying only the public keys used to pass eval and leave the queue with an auth-callout nobody answers, which refuses every client rather than degrading — and a refusal reaches the client as a timeout, so the symptom is every consumer hanging with nothing logged.

One consequence of the generated path worth knowing before you debug it: with autoGenerateCallout set, the queue assembles its config at boot rather than at build time, so a malformed one surfaces when the container starts instead of when the system builds. The server names the offending file and refuses to run.

Hive-level — one of each per hive

secret	generated by	lives at
hive CA cert + key	hive-tls.nix first-boot unit	<deploy.hive-controller.tls.stateDir>/ca.pem, ca-key.pem (0600)
hive leaf certs	hive-tls.nix, signed by the hive CA	<deploy.hive-controller.tls.stateDir>/<name>.pem
matrix appservice token	swarm-secret-publish, published to the store; a host activation script only as a first-boot fallback	/var/lib/hyperhive/matrix-appservice-token (0600)
the forge's copy of its OIDC secret	hive-forge-oidc-secret.service copies it from authelia's tree	/var/lib/forgejo-oidc/<id>.secret inside the forge container
the homeserver's copy of its OIDC secret	hive-matrix-oidc-secret.service, same shape	/var/lib/tuwunel-oidc/<id>.secret, handed to tuwunel through LoadCredential
the agent containers' queue credential	authelia, published to the store by swarm-secret-publish	<deploy.hive-controller.queue.agentCredentialDir>/secret (0600) and /client_id (0644)

Both delivery units wait for authelia's first boot to mint the secret — a bounded wait, 120s — and then fail loudly rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom many layers from its cause.

The store's first reader is the matrix appservice token, and it's worth saying why that one: it's an opaque 32-byte value with no second file and no format. Authelia's OIDC secret needs a .secret and a matching .digest, so starting there would have meant debugging "can a reader authenticate and get bytes back" and "is authelia's file format right" at once, with an SSO outage as the failure mode.

glue-matrix-bao-token.nix fetches it, writes the file hive-matrix.nix already reads, and then runs that module's own renderer to re-stamp the appservice registration naming the token — so the homeserver never learns the store exists. The re-render isn't housekeeping: the token is half an agreement, and a registration carrying the previous value authenticates nobody. Every failure path — no such key, sealed store, unreachable store, empty value — leaves the locally minted token in place, so a hive with no store behaves exactly as it did before.

The store path is swarm/hives/<hive>/matrix/appservice-token, and the producer is swarm-secret-publish — the same unit that copies authelia's OIDC secrets in. It mints one token per hive in the swarm's roster and puts it there, so the store is the source of truth and every hive converges on the value it holds. The hive's own activation mint is still there, but it's a first-boot fallback now: it fires only when the token file is absent, and the reader overwrites whatever it produced.

The mint is idempotent by keeping its own record, not by asking the store. The publisher's grant is create/update under swarm/hives/* with no read — write-only on purpose, so a file-copier can't recover every hive's credentials — which means it can't check whether a hive already has a token. Instead it keeps the value it minted under its StateDirectory, /var/lib/swarm-secret-publish/matrix-appservice-token/<hive> (0700 dir, 0600 file), and mints only when that file is missing or empty. The put itself runs every time: re-putting the same bytes changes nothing for any reader, while a mint whose publish failed must not leave behind a token this host holds and no hive can reach.

Lose the state directory and the next run mints once more and republishes. That rotates the token, which readers pick up on their next start — nothing that already registered breaks, because the token authenticates the appservice rather than living inside any account it created.

The path was …/matrix/registration-token while the homeserver still took a shared registration secret; nothing reads a value left at the old path.

The second reader is the agent containers' queue credential: glue-queue-agent-credential.nix lands it as two files, the client secret and the client id it authenticates, because that's how a queue client reads them — the secret by path, the id by value. No local fallback exists, and none is possible, so absent files mean this hive's agents don't connect, which is what a swarm looks like before the publisher on the authelia host has run. The reader runs before hive-c0re.service, which wants it (doesn't require it), so an agent container never renders ahead of the credential; an unreachable store delays the daemon's start rather than failing it.

That credential still has one hop left, because the reader of it lives inside an agent container. It crosses as a systemd credential, not as a bind mount, and the mode decides that: the secret is root:0600 and a harness runs as its own unprivileged agent user, so a bind would deliver a file that user can't open. hive-c0re stats the two files — the directory is 0755, so it needs no read access to either — and hands them to systemd-nspawn --load-credential through hive-priv, which runs as root. Inside, hive-agent.service names the same two ids in LoadCredential= and reads them out of its own $CREDENTIALS_DIRECTORY, owned by the agent user and by nobody else. hive-c0re never reads the bytes at any point: it runs as hive-core, which is the reason a copy wasn't an option either. When the files aren't there the daemon forwards nothing and says so in its journal, and the harness logs that it has no queue coordinates — the same absent-and-legal state, twice, rather than a container that refuses to start.

The third reader is Grafana's OIDC client secret, and it's the first one that's a swarm service's own credential rather than a hive's. It has exactly one delivery route: swarm-bao-grafana-oidc.service reads swarm/services/<client-id>/oidc/client out of the store, on every host that runs Grafana. Where authelia is doesn't enter into it — the publisher on authelia's host writes that path whether the reader is a network away or in the container next door.

That's a deliberate refusal of the shortcut. Copying the plaintext straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store exists so the only secrets a host holds out of band are certificates, and a host reads everything else with them.

Two things follow, and swarm-grafana.nix asserts both rather than degrading: running Grafana requires swarm.authelia.url (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the only way in), and it requires this host's deploy.bao.grafanaOidcClientCertFile / grafanaOidcClientKeyFile. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is (glue-grafana-oidc-client.nix): a client is a row in authelia's own config, so it's declared on the host running authelia, which isn't necessarily the host running Grafana.

The fourth reader is the swarm collector's own OIDC client secret, the same shape one level down: swarm-bao-otel-oidc.service reads swarm/services/<client-id>/oidc/client out of the store on every host that runs the collector, and registration lives separately in glue-swarm-otel-oidc-client.nix for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so swarm-otel.nix doesn't assert deploy.bao.otelOidcClientCertFile / otelOidcClientKeyFile the way Grafana does — it doesn't render the reading unit without them, the shape glue-matrix-bao-token.nix and glue-queue-agent-credential.nix use for their own optional reads.

The fifth reader is the secret store forwarder's OIDC client secret: swarm-bao-forwarder-oidc.service, on the store's own host, registered by glue-swarm-bao-otel-oidc-client.nix on authelia's. It differs from the fourth in both directions. It renders unconditionally, because the export it authenticates has no unauthenticated mode to degrade into — and it orders itself after container@swarm-bao, because the store it reads runs in the container it delivers into. Nothing circular sits behind that: the identity it logs in with is this host's static swarm-bao-pki leaf, not anything the store mints.

A service's secret is one value for the whole swarm rather than one per hive, so it lives under the services prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a deploy.* fact, per-host by definition — so there is nothing to scope the grant to. A hive's own leaf can therefore read every swarm service's client secret, and that's stated in swarm-secret-client's policy module beside the grant itself.

⚠️ The readers no longer present it. Four units used to log in with deploy.bao.clientCertFile, which is the hive's own leaf, and bao identifies a principal by the subject of the certificate it presents — so four readers behind one leaf were one principal holding the union of their needs. Each now holds a leaf of its own, and a policy naming only the path that unit reads. See per-principal identities below.

Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in.

⚠️ Service↔store mTLS is its own trust domain. You can't fetch a credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the hive CA and the swarm CA both, and has nothing to do with the gateway's HTTPS certificates either way. glue-bao-tls.nix mints a CA that signs a fixed, short list of leaves — the store's server certificate, the hive's own reader certificate, and one per service principal — and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in deploy.bao.serverCertFile / clientCaFile; the store itself has no opinion. A hive that reads from a store on another machine names the reader's half — clientCertFile, clientKeyFile, serverCaFile — and places that leaf by hand, plus one leaf per principal it runs (the table below names the options). Those are the credentials that can't come out of the store, being what opens it; everything else a hive needs does.

Per-principal identities

Bao matches a cert-auth role on the certificate's subject, so a certificate is an identity and sharing one merges the identities. These four units read one path each and each holds a leaf, a role and a policy of its own:

unit	option pair under deploy.bao.	reads
swarm-bao-matrix-token	matrixTokenClientCertFile / matrixTokenClientKeyFile	swarm/hives/<hive>/matrix/appservice-token
swarm-bao-queue-agent	queueAgentClientCertFile / queueAgentClientKeyFile	swarm/hives/<hive>/queue/agent
swarm-bao-grafana-oidc	grafanaOidcClientCertFile / grafanaOidcClientKeyFile	swarm/services/<grafana client id>/oidc/client
swarm-bao-otel-oidc	otelOidcClientCertFile / otelOidcClientKeyFile	swarm/services/<collector client id>/oidc/client

The first two exist per hive, because the path they read carries a hive name and every hive runs its own reader. Their subjects are <deploy.bao.matrixTokenCommonNamePrefix>-<hive> and <deploy.bao.queueAgentCommonNamePrefix>-<hive>; swarm.nix reserves both composed spellings as hive names, so nobody can name a hive into another hive's role. The other two read a path that names a swarm service rather than a hive, so one role each is enough and their subjects are the flat deploy.bao.grafanaOidcCommonName and deploy.bao.otelOidcCommonName.

On a host that mints its own PKI, glue-bao-tls.nix signs all four and defaults all eight options, and there is nothing to do. Elsewhere you issue each leaf from that CA out of band and name it here — one file per principal rather than one file shared by four, which is the whole of what this buys.

Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four readers used to render only where their leaf existed, so a hand-configured remote-store hive that named the hive's own clientCertFile and missed a principal's pair just lost that unit; swarm-grafana.nix was alone in refusing the build. All four now refuse it, each naming its own option pair — see what a missing leaf costs below.

What a missing leaf costs

Each of the four modules asserts its own pair, on the same condition: this host already reads the store — it holds deploy.bao.clientCertFile and clientKeyFile — and the consumer is on, and this principal's pair is null. The refusal names the two options, so an operator acts on the message without opening the nix.

That condition fires on exactly one shape, the hand-configured remote-store hive that named seven of the eight. It stays silent everywhere else, and each half of that matters on its own:

• A host with no store identity at all holds no clientCertFile either, so the first clause is false. That's the supported no-store deployment, and also the one an operator passes through while bringing a hive up — neither should fail to evaluate.
• A host that runs the store gets all eight from glue-bao-tls.nix as defaults, so the third clause is false. Nothing to set, nothing to refuse.
• A consumer that's off — no homeserver, no collector, no Grafana — has no unit to skip, so the second clause is false. Only the queue-credential reader has no toggle of its own to check: every hive runs one for its own agents, so reading the store at all is what asks for it.

The collector keeps a separate degrade underneath this, unchanged: a host with no store identity runs a collector with no OIDC client secret and still receives telemetry. The assertion doesn't touch that shape, because that host has no clientCertFile either.

How a reader reaches the store

Every reader dials the same URL — https://bao.<swarm domain>:<port> — and on the host that runs the store that name resolves twice. /etc/hosts answers a host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a container with the bridge IP, where nginx holds the same port. One BAO_ADDR, and the network namespace that asked decides which half of it answers.

The container's half goes through nginx and still doesn't terminate TLS. It is an nginx stream server with ssl_preread on: it reads the SNI off the ClientHello and splices the rest of the connection through byte for byte, so openbao completes the handshake itself. A vhost would decrypt here instead, and the client certificate — the store's whole authentication — would stop at nginx, leaving openbao seeing one client for every hive in the swarm. That's why the store has no vhost and gets a passthrough: not an exception to the no-vhost rule, the one shape that keeps it.

The stream listener binds the bridge IP rather than every address, because openbao already holds loopback on that port in the same network namespace and a wildcard bind would fail with EADDRINUSE — taking nginx, and with it every hive domain behind the gateway, down. network.exposeHostPorts opens the port on the bridge firewall and nowhere else.

Reaching the port grants nothing by itself: openbao answers nothing without a client certificate signed by deploy.bao.clientCaFile. The passthrough carries whichever certificate the reader presents, unchanged.

The constraint that decides where the root lives

A hive CA carries nameConstraints=permitted;DNS:<hive domain>, and a swarm service name is a sibling of the hive domain rather than a child — forge.<swarm> next to <hive>.<swarm>. A hive CA can't issue a certificate for a swarm service. Not by policy: by construction, and openssl enforces it.

Whatever holds the swarm root is therefore what makes swarm-service certificates possible at all. Two things follow:

• The root's private key is a runtime file and must never enter the nix store, so nothing build-time can name it — the system reads security.pki.certificateFiles' value at build time, which makes it the wrong tool here. Trust reaches containers through a bind-mounted bundle assembled at boot instead.
• On any topology other than all-local, placing that key is an operations decision, not something this module tree makes for you. A hive that hosts no swarm services needs only the root's cert, to trust what others issue.

Adding a secret

State three things, in the row you add above: who mints it, which container reads it, and what happens when they differ. If they differ, it needs a delivery unit, and the unit copies — it doesn't bind.