# Swarm secrets: what exists, and where each one lives A swarm generates its credentials in three different places and reads them in a fourth, so "where does this file go" has a different answer per deployment. This page is that answer, one row per secret. Two rules run through all of it. **Private key material and access tokens are paths, never values.** Every option carrying one takes a file path (`*File`), because a literal written into a nix expression ends up in the nix store — world-readable and permanent. No option in this tree accepts one inline, and adding one would be a leak rather than a convenience. The rule is about what must stay secret, not about credentials generally. **Public material is a value**: a certificate, or a public nkey like the queue callout's user key, reaches every client that connects, so the store is a perfectly good place for it. **The generator and the reader typically live in different containers.** They share the host's network namespace, which makes them feel co-located, but their filesystem roots are separate. That's why delivery is a **host-side copy rather than a bind mount**: `nixos-container` refuses to start when a bind source is missing, and a secret minted on another container's first boot doesn't exist yet. Binding it would make one container wait on a file that waits on a container that starts after it. ## Topologies, by who places secrets Read every row below against one of these. This is a different cut from the [deployment shapes](services.md#deployment-shapes) --- those say _where services run_, these say _who is responsible for a secret file being there_ --- the two lists don't line up one-to-one, and neither is a renaming of the other. | topology | what it means | who places secrets | | ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- | | **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it | | **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place | | **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config | ## Swarm-level — one of each per swarm | secret | generated by | lives at | hive elsewhere | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | | the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | | the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | | the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | | the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | | the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. **That's the test worth applying to any secret added here** — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists. Three telemetry collectors exist, and each one fails that test: each reads its own secret out of the store, under its own identity. The **hive's** collector authenticates to the swarm's collector as its own hive, so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches `swarm/hives//oidc/client` under the hive's own leaf (`deploy.bao.clientCertFile`), on every topology. A hive with no leaf has no default, and the build stops at the collector's identity assertion unless `services.hyperhive.otel.clientSecretFile` names a file delivered some other way. The **swarm's** collector reads its own under a leaf of its own, the fourth store reader — the same shape as Grafana's, described in full below. In short: `swarm-bao-otel-oidc.service` reads `swarm/services//oidc/client` out of the store on every host that runs the collector and holds a store identity, whether authelia is a network away or in the container next door. The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers the client it reads for unconditionally: authelia refuses a bearer-authz client with no audience, and every host publishes at least forgejo's `/metrics` as a scrape target, so the audience is never empty. ⚠️ **A collector with no store identity isn't refused.** Unlike Grafana, where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — its pushes to the stores go out unauthenticated and get refused there, and its scrape jobs for the published targets read a credential file nothing loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without `deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice `glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this unit never reaches. The **secret store's own** collector — the forwarder inside the `swarm-bao` container — needs a delivery step too, and it takes the same route with one principal of its own: `swarm-bao-forwarder-oidc.service` reads `swarm/services//oidc/client` under a leaf of its own and writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the collector. The client id is its own (`services.hyperhive.swarm.bao.otel.clientId`), registered by `glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm collector admits it on a receiver of its own — one identity per principal, so this forwarder borrows neither the swarm collector's client nor `swarm-controller`'s. ⚠️ **This one has no degraded mode, and the difference is structural.** The route it exports over accepts an authenticated request and nothing else, so a forwarder without the secret has nothing to fall back to: the unit fails and retries, and the collector refuses to start until the file arrives, after every boot, rather than exporting into a 401 while reporting healthy. Its ordering differs from the unit above it for the same structural reason — it reads the store inside the container it delivers into, so it runs **after** that container rather than before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers. ### The queue's callout nkeys The responder holds two nkeys besides the callout xkey below. It signs every user JWT it issues with the **issuer** key, an account key, so possession of the issuer seed is the authority to admit anyone to the queue. It connects to the queue as the **user** key, the one identity the server exempts from callout. You configure nothing about either: both live in the swarm secret store beside the xkey, and come into being on the queue's first boot. - **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`, **public half** at `secret/swarm/public/swarm-nats/callout-issuer`. - **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`, **public half** at `secret/swarm/public/swarm-nats/callout-user`. The unit that creates the xkey creates these too, under the same rules: only the `swarm-nats-auth` role reads or creates them, nothing replaces a stored seed, and the policy of `swarm-secret-publisher` denies it both seed paths. The server renders both public halves into its `auth_callout` block as it starts, and doesn't start without them. A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or `/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what `swarm-nats-auth ensure-callout-keys` stores for that key when the store holds none, so the key keeps its public half. Once a read of the store returns a seed, the verb removes the file. When the store already holds a different seed, the verb keeps the store's seed, logs a warning naming the file, then removes it. A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`, `calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`, under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the option and where the keys live now. To replace either key, delete both of its paths the way the xkey's section below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`. One consequence worth knowing before you debug it: the queue assembles its config each time the server starts rather than at build time, in every mode, because the callout keys exist only at runtime. A malformed config surfaces when the server starts instead of when the system builds. The server names the offending file and refuses to run. ### The queue's callout xkey The server seals every auth-callout request to a curve key, the callout xkey, and the responder seals its reply back, so nothing else in the callout account or on the wire can read the request or the user JWT minted for it. You configure nothing about the key: it lives in the swarm secret store and comes into being on the queue's first boot. - **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`. - **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`. Only the `swarm-nats-auth` role reads either path, and it may create each but never replace it. The policy of `swarm-secret-publisher`, which writes the rest of `swarm/services/`, denies it the seed's path. On the queue's host, `swarm-bao-nats-callout-xkey` runs before the `swarm-nats` container, under the responder's store leaf: 1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no seed, and otherwise only reads it. It needs no queue, so on a first boot the key exists before the server that needs it. 2. It reads the public half back from the store into `/var/lib/nats-callout/callout-xkey.pub` inside the container. 3. Each time `nats-server` starts, it renders that key into `auth_callout.xkey`. With no file, the server doesn't start. 4. The responder starts after the server and reads the seed from the store. A responder that can't read it doesn't start, rather than answering requests it can't open. The store has to be reachable each time the responder starts. When it's down, the fetch unit fails and retries for a day. The server starts with the key fetched last time, or not at all on a first boot. A responder that starts while the store is down exits and retries every 5 seconds, and until one start reads the seed the queue refuses every client, OIDC ones included. The retry that lands the file starts or restarts the server, and the responder comes up with it. The responder's store leaf is `deploy.nats.authBaoClientCertFile` / `authBaoClientKeyFile`. On the store's host both default to the leaf `glue-bao-tls.nix` mints there. A queue host without the store gets no default, even when it holds the hive's own reader `deploy.bao.clientCertFile`: issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and name it in both options. Until you do, evaluation fails naming both. Where the queue and the store run on different hosts, deploy the store's host first. `swarm-bao-nats-auth-policy` there writes the role's grants on the keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of the seeds' paths; where the queue runs on that host too, the fetch waits for both. A queue host deployed before it can't read or create the keys, so its fetch fails, and the queue refuses every client until you deploy the store's host: the server starts only on public keys fetched before, and the responder can't load its seeds. Seed files the import is waiting for stay where they are. If that takes longer than the fetch's day of retries, run `systemctl restart swarm-bao-nats-callout-xkey` on the queue's host. Two starts that create the key at once can't both win: each write is a create that fails once the path holds anything, and the start that lost reads the winner's key back. To replace the key, remove both paths with their metadata and every version, then restart the fetch unit on the queue's host. Its restart of `nats.service` restarts the responder too, which then loads the new seed: ```sh bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey systemctl restart swarm-bao-nats-callout-xkey ``` `bao kv delete` isn't enough. It deletes the latest version but keeps the path's metadata, so the store still counts the path as existing and refuses the create that makes a new key. No principal hyperhive configures may delete either path's metadata, so run the two deletes with a token that may, such as the root token. ## Hive-level — one of each per hive | secret | generated by | lives at | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `/.pem` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | | the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `/secret` (`0600`) and `/client_id` (`0644`) | The forge and the homeserver read their copies out of the store, the way Grafana does below: the forge under the forge publisher's leaf, the homeserver under matrix-ctl's. Each **fails loudly** and retries until the publisher on authelia's host has written the secret, rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom many layers from its cause. The store's **first reader** is the agent containers' queue credential: `glue-queue-agent-credential.nix` lands it as two files, the client secret and the client id it authenticates, because that's how a queue client reads them — the secret by path, the id by value. No local fallback exists, and none is possible, so absent files mean an agent built before `swarm-logs` read its own queue secret has no working `swarm-logs`, which is what a swarm looks like before the publisher on the authelia host has run. The reader runs before `hive-c0re.service`, which wants it (doesn't require it), so an agent container never renders ahead of the credential; an unreachable store delays the daemon's start rather than failing it. That credential still has one hop left, because the reader of it lives inside an agent container. **It crosses as a systemd credential, not as a bind mount**, and the mode decides that: the secret is `root:0600` and a harness runs as its own unprivileged agent user, so a bind would deliver a file that user can't open. `hive-c0re` stats the two files — the directory is `0755`, so it needs no read access to either — and hands them to `systemd-nspawn --load-credential` through `hive-priv`, which runs as root. Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by nobody else, where a `swarm-logs` built before it read the agent's own queue secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. `hive-c0re` never reads the bytes at any point: it runs as `hive-core`, which is the reason a copy wasn't an option either. When the files aren't there the daemon forwards nothing and says so in its journal, and the container starts without them. The **second reader** is Grafana's OIDC client secret, and it's the first one that's a _swarm service's_ own credential rather than a hive's. It has exactly one delivery route: `swarm-bao-grafana-oidc.service` reads `swarm/services//oidc/client` out of the store, on **every** host that runs Grafana. Where authelia is doesn't enter into it — the publisher on authelia's host writes that path whether the reader is a network away or in the container next door. That's a deliberate refusal of the shortcut. Copying the plaintext straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store exists so the only secrets a host holds out of band are **certificates**, and a host reads everything else with them. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is (`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config, so it's declared on the host running authelia, which isn't necessarily the host running Grafana. The swarm controller, the queue, the homeserver and the forge register the same way (`glue-controller-oidc-client.nix`, `glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`, `glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the forge read their secrets from the store too: the responder itself under its own leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's, `swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming the option pair, when that leaf is unset — neither the responder nor the homeserver starts without its secret, and the forge registers no login source without its. Grafana's admin password rides the same leaf and the same store. Where Grafana and the store run on different hosts, deploy the store's host first: `swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create the password, and until it has, the store refuses `swarm-bao-grafana-admin` on the Grafana host, which retries, while Grafana stays down because its config reads that file. The **third reader** is the swarm collector's own OIDC client secret, the same shape one level down: `swarm-bao-otel-oidc.service` reads `swarm/services//oidc/client` out of the store on every host that runs the collector, and registration lives separately in `glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so `swarm-otel.nix` doesn't assert `deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana does — it doesn't render the reading unit without them, the shape `glue-queue-agent-credential.nix` uses for its own read. The **fourth reader** is the secret store forwarder's OIDC client secret: `swarm-bao-forwarder-oidc.service`, on the store's own host, registered by `glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the third in both directions. It renders unconditionally, because the export it authenticates has no unauthenticated mode to degrade into — and it orders itself **after** `container@swarm-bao`, because the store it reads runs in the container it delivers into. Nothing circular sits behind that: the identity it logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix` refuses the build and names `deploy.bao.forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile`. A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a `deploy.*` fact, per-host by definition — so there is nothing to scope the grant to. A hive's **own** leaf can therefore read every swarm service's client secret, and that's stated in `swarm-secret-client`'s `policy` module beside the grant itself. ⚠️ **The readers no longer present it.** Five units used to log in with `deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a principal by the subject of the certificate it presents — so five readers behind one leaf were one principal holding the union of their needs. Each now holds a leaf of its own, and a policy naming only the path that unit reads. See [per-principal identities](#per-principal-identities) below. Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. ⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the store's `pki` root, and has nothing to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix` mints a CA that signs a fixed, short list of leaves — the store's server certificate, the hive's own reader certificate, and one per service principal — and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — `clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, plus one leaf per principal it runs (the table below names the options). Those are the credentials that can't come out of the store, being what opens it; everything else a hive needs does. Agents are the one principal whose leaf the store issues. Its `pki-agents` mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the store, and `swarm-controller`, already logged in under its own host leaf, asks that mount's one role for a `hive-agent-` client certificate at agent creation. Host roles never pin that CA and agent roles pin only it, so an agent's certificate opens that agent's own `swarm/agents//*` and nothing else: `read` on the values there and `list` on their names. `swarm-controller` writes that policy at agent creation and rewrites every agent's at its own start, so a change to it reaches existing agents with the next controller restart. ### Per-principal identities Bao matches a cert-auth role on the certificate's subject, so a certificate is an identity and sharing one merges the identities. Each of these ten leaves has a role and a policy of its own, and reaches one path, except the review classifier's, the CI runner's, the builder's, the CI run poller's, the claude login minter's, the forge host's and Grafana's, which reach more than one: | unit | option pair under `deploy.bao.` | reads, or writes | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | | `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services//oidc/client` and `swarm/services//oidc/client`; creates and reads `swarm/services//admin/password` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services//oidc/client` | | `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services//oidc/client` | | `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services//oidc/client` | | `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services//oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services//oidc/client` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services//oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` and `swarm/services//oidc/client` | The first exists **per hive**, because the path it reads carries a hive name and every hive runs its own reader. Its subject is `-`; `swarm.nix` reserves that composed spelling as a hive name, so nobody can name a hive into another hive's role. The other nine reach a path that names a swarm service rather than a hive, so one role each is enough and their subjects are the flat `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, `deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`, `deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`, `deploy.bao.ciMetricsCommonName`, `deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`. The last two are the writers, and the last is the one leaf two units share: the forge's host holds one store identity for the token it publishes and the OIDC client secret it reads. On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from the store's host and name it here: a service principal's from the top of `/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from `/var/lib/swarm-bao-pki/hives//` ([bao.md § TLS](bao.md#tls) lists that directory). That's one file per principal rather than one file shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix` refuses the build wherever the runner runs without its pair, as `swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix` for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start naming both options, and the controller runs without forge access and the forge without a login source until an operator places the leaf. On a host without the store, where the forwarder never runs, forgetting one of the other nine isn't a quiet degrade. A hand-configured remote-store hive that names the hive's own `clientCertFile` and misses a principal's pair has the build refused, by the module of the reader it missed, naming that pair — see [what a missing leaf costs](#what-a-missing-leaf-costs) below. ### What a missing leaf costs Each of the three modules asserts its own pair, on the same condition: **this host already reads the store** — it holds `deploy.bao.clientCertFile` and `clientKeyFile` — **and the consumer is on, and this principal's pair is null**. The refusal names the two options, so an operator acts on the message without opening the nix. That condition fires on exactly one shape, the hand-configured remote-store hive that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of that matters on its own: - **A host with no store identity at all** holds no `clientCertFile` either, so the first clause is false. That's the supported no-store deployment, and also the one an operator passes through while bringing a hive up — neither should fail to evaluate. - **A host that runs the store** gets every pair from `glue-bao-tls.nix` as defaults, so the third clause is false. Nothing to set, nothing to refuse. - **A consumer that's off** — no collector, no Grafana — has no unit to skip, so the second clause is false. Only the queue-credential reader has no toggle of its own to check: every hive runs one for its own agents, so reading the store at all is what asks for it. The collector keeps a separate degrade underneath this, unchanged: a host with no store identity runs a collector with no OIDC client secret and still receives telemetry. The assertion doesn't touch that shape, because that host has no `clientCertFile` either. ## How a reader reaches the store Every reader dials the same URL — `https://bao.:` — and on the host that runs the store that name resolves twice. `/etc/hosts` answers a host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`, and the network namespace that asked decides which half of it answers. The container's half goes through nginx and **still doesn't terminate TLS**. It is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the ClientHello and splices the rest of the connection through byte for byte, so openbao completes the handshake itself. A vhost would decrypt here instead, and the client certificate — the store's whole authentication — would stop at nginx, leaving openbao seeing one client for every hive in the swarm. That's why the store has no vhost and gets a passthrough: not an exception to the no-vhost rule, the one shape that keeps it. The stream listener binds the bridge IP rather than every address, because openbao already holds loopback on that port in the same network namespace and a wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every hive domain behind the gateway, down. `network.exposeHostPorts` opens the port on the bridge firewall and nowhere else. Reaching the port grants nothing by itself: openbao answers nothing without a client certificate signed by `deploy.bao.clientCaFile` or by the store's own agent CA. The listener reads both from `listener-client-ca.pem`, which no cert-auth role pins. The passthrough carries whichever certificate the reader presents, unchanged. ## The browser UI OpenBao's built-in web UI is at `https://bao-ui./` (`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost checks the session with authelia and proxies to an nginx inside the store's container on `127.0.0.1:`. That nginx forwards `/ui/` and `/v1/` to a second openbao listener on `127.0.0.1:`, answers 403 on the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on everything else. ⚠️ That listener asks for **no client certificate**, because a browser has none to present, so on this one door a bao token is the whole credential. Anything on the store's host that can dial loopback, and any `admins` session through the vhost, needs only a token to use the API. Unseal from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed gateway where the UI is the only swarm name its host fronts, that host's nginx doesn't wait for the store: it serves the hive leaf, or on a host that runs no hive a throwaway self-signed certificate, on the UI's name (a browser warning) until the unsealed store issues the services leaf, so the stream passthrough readers use on that host comes up with the store sealed. ### OIDC login The OIDC tab logs in through authelia as the client `swarm-bao-ui` (`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under `swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and `create` and `update` on `secret/data/swarm/services/review-classifier/anthropic-api-key` and `secret/data/swarm/services/review-classifier/acp-provider`. That shows the key tree and each key's versions and timestamps, and no value: the store refuses every `secret/data/` read. The two writes are the review classifier's keys, which the operator fills in by hand. Anything more needs a token. | piece | written by | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs | | its secret at `swarm/services/swarm-bao-ui/oidc/client` | `swarm-secret-publish`, like every other service's | | the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token | | `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config | The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token step. A store set up before the mount existed needs the [one-time granter step](../getting-started/setup.md) again: it re-writes `bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own policy, which covers `auth/oidc/`. Until then `swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs it each time the step succeeds, so OIDC login needs no further restart. ## The constraint that decides where the root lives The store issues a hive's gateway leaf through a role whose `allowed_domains` is that hive's domain and `*.` that domain, and **a swarm service name is a sibling of the hive domain rather than a child** — `forge.` next to `.`. That role can't issue a certificate for a swarm service; the store refuses it. The store's `swarm-services` role issues them instead, from the root inside its `pki` mount. That root's private key never leaves the mount, and its certificate is the only anchor the swarm has. Each hive fetches the certificate from the store at boot, so nothing build-time can name it — the system reads `security.pki.certificateFiles`' value at build time, which makes it the wrong tool here. Trust reaches containers through a bind-mounted bundle assembled at boot instead. ## Adding a secret State three things, in the row you add above: **who mints it**, **which container reads it**, and **what happens when they differ**. If they differ, it needs a delivery unit, and the unit copies — it doesn't bind.