hyperhive/docs/4-implementation/swarm/credential-delivery.md

How each credential reaches its reader

The swarm-level and hive-level secrets, one by one: which unit mints each, where it lands, and what a missing identity costs. The rules a credential follows are on the handbook's credentials page; the table of every credential is in credentials-reference.md, and the store's own mechanics are in credentials.md.

Swarm-level secrets

The rows are in the reference.

Nothing outside authelia's container reads its keys (session, JWT, storage encryption, OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. That's the test worth applying to any secret added here — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists.

The store still holds those five keys, at swarm/services/swarm-authelia/keys/<name>, so that losing the host doesn't lose them. authelia-swarm-secrets in the container reads each with the secret publisher's identity and writes it to the file authelia loads. A store with no value takes the file's, so the key keeps its value; the unit mints a new key only when there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's database, and a new issuer key invalidates every token issued. A container without the publisher's identity generates the files as before.

The same unit mints the client secrets, straight into the store. It reads swarm/hives/<hive>/oidc/client or swarm/services/<client id>/oidc/client first, and only when the store answers and holds none does it run authelia crypto hash generate, create the leaf and write the digest. Authelia keeps only the digest on disk, in oidc-clients/<id>.digest, and starts from it while the store is down. A store that can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and swarm-authelia-bridge finds its secret once the store answers. A client with a secret in the store but no digest gets the digest of that secret and keeps the secret; authelia crypto hash generate computes the digest, with the password typed into its prompt through a pty, never in a command line. swarm-authelia-bridge fetches its own secret from swarm/services/swarm-authelia-bridge/oidc/client into /run/swarm-authelia-bridge/client-secret before each start, and a store that doesn't answer keeps the bridge down while it retries. A container without the publisher's identity keeps the plaintext in oidc-clients/<id>.secret, as before.

The mint reads an oidc-clients/<id>.secret file an earlier release left once: when the store holds no value for that client, it stores the file's value with -cas=0, so the secret keeps its value. A store that holds a value wins, whatever the file says, and the file stays on disk, unused.

Three telemetry collectors exist, and each one fails that test: each reads its own secret out of the store, under its own identity.

The hive's collector authenticates to the swarm's collector as its own hive, so it reads its secret as the hive: swarm-bao-hive-otel-oidc.service fetches swarm/hives/<hive>/oidc/client under the hive's own leaf (deploy.bao.clientCertFile), on every topology, into /run/hive-otel-oidc on the host's tmpfs. The file never reaches persistent disk and a reboot fetches it again; until the store answers, the collector runs on an empty file and the swarm refuses its exports. A hive with no leaf has no default, and the build stops at the collector's identity assertion unless services.hyperhive.otel.clientSecretFile names a file delivered some other way.

The swarm's collector reads its own under a leaf of its own, the fourth store reader — the same shape as Grafana's, described in full below. In short: swarm-bao-otel-oidc.service reads swarm/services/<client-id>/oidc/client out of the store on every host that runs the collector and holds a store identity, whether authelia is a network away or in the container next door. The same call glue-grafana-oidc-client.nix makes for Grafana's registers the client it reads for unconditionally: authelia refuses a bearer-authz client with no audience, and every host publishes at least forgejo's /metrics as a scrape target, so the audience is never empty.

⚠️ A collector with no store identity isn't refused. Unlike Grafana, where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — its pushes to the stores go out unauthenticated and get refused there, and its scrape jobs for the published targets read a credential file nothing loads, so they can't get a token. swarm-bao-otel-oidc.service therefore doesn't render at all without deploy.bao.otelOidcClientCertFile / otelOidcClientKeyFile, rather than the hard assertion swarm-grafana.nix uses for its non-optional one. The operator can still deliver the secret by hand, at services.hyperhive.deploy.swarm-otel.clientSecretFile, on a collector this unit never reaches.

The secret store's own collector — the forwarder inside the swarm-bao container — needs a delivery step too, and it takes the same route with one principal of its own: swarm-bao-forwarder-oidc.service reads swarm/services/<client-id>/oidc/client under a leaf of its own and writes it to /run/swarm-bao-forwarder-oidc on the host's tmpfs, which the container bind-mounts read-only and where LoadCredential hands it to the collector. The client id is its own (services.hyperhive.swarm.bao.otel.clientId), registered by glue-swarm-bao-otel-oidc-client.nix wherever authelia runs, and the swarm collector admits it on a receiver of its own — one identity per principal, so this forwarder borrows neither the swarm collector's client nor swarm-controller's.

⚠️ This one has no degraded mode, and the difference is structural. The route it exports over accepts an authenticated request and nothing else, so a forwarder without the secret has nothing to fall back to: the unit fails and retries, and the collector refuses to start until the file arrives, after every boot, rather than exporting into a 401 while reporting healthy. Its ordering differs from the unit above it for the same structural reason — it reads the store inside the container it delivers into, so it runs after that container rather than before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.

Hive-level secrets

The rows are in the reference.

The forge and the homeserver read their copies out of the store, the way Grafana does below: the forge under the forge publisher's leaf, the homeserver under matrix-ctl's. Each fails loudly and retries until the mint in authelia's container has stored the secret, rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom many layers from its cause.

The store's first reader is Grafana's OIDC client secret, a swarm service's own credential rather than a hive's. It has exactly one delivery route: swarm-bao-grafana-oidc.service reads swarm/services/<client-id>/oidc/client out of the store, on every host that runs Grafana. Where authelia is doesn't enter into it — the mint in authelia's container writes that path whether the reader is a network away or in the container next door.

That's a deliberate refusal of the shortcut. Copying the plaintext straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store exists so the only secrets a host holds out of band are certificates, and a host reads everything else with them.

Two things follow, and swarm-grafana.nix asserts both rather than degrading: running Grafana requires swarm.authelia.url (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the only way into the UI), and it requires this host's deploy.bao.grafanaOidcClientCertFile / grafanaOidcClientKeyFile. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is (glue-grafana-oidc-client.nix): a client is a row in authelia's own config, so it's declared on the host running authelia, which isn't necessarily the host running Grafana. The swarm controller, the queue, the homeserver and the forge register the same way (glue-controller-oidc-client.nix, glue-nats-oidc-client.nix, glue-matrix-oidc-client.nix, glue-forge-oidc-client.nix), and the queue's responder, the homeserver and the forge read their secrets from the store too: the responder itself under its own leaf, swarm-bao-matrix-oidc.service under matrix-ctl's, swarm-bao-forge-oidc.service under the forge publisher's. Each fails, naming the option pair, when that leaf is unset — neither the responder nor the homeserver starts without its secret, and the forge registers no login source without its.

Grafana's admin password rides the same leaf and the same store. Where Grafana and the store run on different hosts, deploy the store's host first: swarm-bao-grafana-oidc-policy there writes the grant that lets Grafana create the password, and until it has, the store refuses swarm-bao-grafana-admin on the Grafana host, which retries, while Grafana stays down because its config reads that file.

The second reader is the swarm collector's own OIDC client secret, the same shape one level down: swarm-bao-otel-oidc.service reads swarm/services/<client-id>/oidc/client out of the store on every host that runs the collector, and registration lives separately in glue-swarm-otel-oidc-client.nix for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so swarm-otel.nix doesn't assert deploy.bao.otelOidcClientCertFile / otelOidcClientKeyFile the way Grafana does — it doesn't render the reading unit without them.

The third reader is the secret store forwarder's OIDC client secret: swarm-bao-forwarder-oidc.service, on the store's own host, registered by glue-swarm-bao-otel-oidc-client.nix on authelia's. It differs from the second in both directions. It renders unconditionally, because the export it authenticates has no unauthenticated mode to degrade into — and it orders itself after container@swarm-bao, because the store it reads runs in the container it delivers into. Nothing circular sits behind that: the identity it logs in with is the static forwarder-oidc.pem leaf swarm-bao-pki signs, not anything the store mints. With no leaf it has no fallback, so swarm-bao.nix refuses the build and names deploy.bao.forwarderOidcClientCertFile / forwarderOidcClientKeyFile.

A service's secret is one value for the whole swarm rather than one per hive, so it lives under the services prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a deploy.* fact, per-host by definition — so there is nothing to scope the grant to. A hive's own leaf can therefore read every swarm service's client secret, and that's stated in swarm-secret-client's policy module beside the grant itself.

⚠️ The readers no longer present it. Five units used to log in with deploy.bao.clientCertFile, which is the hive's own leaf, and bao identifies a principal by the subject of the certificate it presents — so five readers behind one leaf were one principal holding the union of their needs. Each now holds a leaf of its own, and a policy naming only the path that unit reads. See per-principal identities.

Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in.

⚠️ Service↔store mTLS is its own trust domain. You can't fetch a credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the store's pki root, and has nothing to do with the gateway's HTTPS certificates either way. glue-bao-tls.nix mints a CA that signs a fixed, short list of leaves — the store's server certificate, the hive's own reader certificate, and one per service principal — and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in deploy.bao.serverCertFile / clientCaFile; the store itself has no opinion. A hive that reads from a store on another machine names the reader's half — clientCertFile, clientKeyFile, serverCaFile — and places that leaf by hand, plus one leaf per principal it runs (the table in the reference names the options). Those are the credentials that can't come out of the store, being what opens it; everything else a hive needs does.

Agents are the one principal whose leaf the store issues. Its pki-agents mount (deploy.bao.agentPkiMountPath) holds an agent CA generated inside the store, and swarm-controller, already logged in under its own host leaf, asks that mount's one role for a hive-agent-<agent> client certificate at agent creation. Host roles never pin that CA and agent roles pin only it, so an agent's certificate opens that agent's own swarm/agents/<agent>/* and nothing else: read on the values there and list on their names, and create (never update) on one leaf, backend-env, which the agent seeds from its environment file.

All agents share one policy, hive-agent, which the agent's cert-auth role attaches and no other. It states the rule with {{identity.entity.name}} where an agent's name would go, so the rule is one document and a change to it is one write. swarm-controller writes it at agent creation and at its own start.

For that the controller names each agent's identity entity after the agent: at each certificate mint or renewal, before it writes the agent's cert-auth role, and for every agent at its own start. It reads the cert mount's accessor and creates the entity <agent> with the alias hive-agent-<agent>, or renames the entity a first login created under a generated name. At a mint or renewal this is a precondition: when the store refuses a call, or the alias sits on any other entity, the mint fails and the role stays as it was, so a live agent keeps working until an operator resolves it. The start pass only logs the same failures, because it writes no role. The controller needs the identity/ and sys/auth/cert grants in swarm-bao.nix's controllerPolicyText for that.

A cert-auth role that predates the shared policy may still name a policy hive-agent-<agent>. The controller no longer writes or attaches these. The objects stay in the store and keep granting what they did, to a token that already carries them and to a role not yet rewritten. A live agent's role is rewritten at its next certificate renewal.

A hive's cert-auth role hive-<hive> attaches one policy, hive-hives, shared by all hives. A hive's certificate carries its bare name as the common name, which is the login's alias name, so hives need no entity: the policy states the hive's own path as swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*, where <accessor> is the cert mount's accessor. The controller reads the accessor at its own start, writes the policy, and rewrites the roles to attach it. No hive name can spell hive-hives: it would be a hive called hives, and reserved-hive-fragments.nix refuses the substring hive. When the controller can't read the accessor or write the policy it logs a warning and leaves the roles untouched, so a role that names a hive-<hive> policy keeps it.

What a missing leaf costs

Each of the four modules asserts its own pair, on the same condition: this host already reads the store — it holds deploy.bao.clientCertFile and clientKeyFile — and the consumer is on, and this principal's pair is null. The refusal names the two options, so an operator acts on the message without opening the nix.

That condition fires on exactly one shape, the hand-configured remote-store hive that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of that matters on its own:

• A host with no store identity at all holds no clientCertFile either, so the first clause is false. That's the supported no-store deployment, and also the one an operator passes through while bringing a hive up — neither should fail to evaluate.
• A host that runs the store gets every pair from glue-bao-tls.nix as defaults, so the third clause is false. Nothing to set, nothing to refuse.
• A consumer that's off — no collector, no Grafana — has no unit to skip, so the second clause is false.

The collector keeps a separate degrade underneath this, unchanged: a host with no store identity runs a collector with no OIDC client secret and still receives telemetry. The assertion doesn't touch that shape, because that host has no clientCertFile either.

The secret store browser UI

OpenBao's built-in web UI is at https://bao-ui.<swarm domain>/ (swarm.bao.ui.domain), for members of authelia's admins group only. Log in with the OIDC tab, or with a bao token under Other. The gateway vhost checks the session with authelia and proxies to an nginx inside the store's container on 127.0.0.1:<deploy.bao.uiProxyPort>. That nginx forwards /ui/ and /v1/ to a second openbao listener on 127.0.0.1:<deploy.bao.uiPort>, answers 403 on the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on everything else. ⚠️ That listener asks for no client certificate, because a browser has none to present, so on this one door a bao token is the whole credential. Anything on the store's host that can dial loopback, and any admins session through the vhost, needs only a token to use the API. Unseal from the host's bao CLI; the UI's unseal form gets a 403. On a self-signed gateway where the UI is the only swarm name its host fronts, that host's nginx doesn't wait for the store: it serves the hive leaf, or on a host that runs no hive a throwaway self-signed certificate, on the UI's name (a browser warning) until the unsealed store issues the services leaf, so the stream passthrough readers use on that host comes up with the store sealed.