hyperhive/docs/2-handbook/swarm/credentials.md

# Credentials

The swarm's credential store is bao. This page is the overview of how the swarm
treats credentials; [`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md)
lists each one: who mints it, who reads it, and how it renews, with the notes for
operating them, and [`credentials.md`](../../4-implementation/swarm/credentials.md)
holds the store's mechanics.

**Store secrets stay off disk.** A process pulls a value from the store at
runtime and holds it in memory only.

**The mTLS client certificate is the only credential on disk.** Each
identity (an agent, a hive, a swarm-level service) has one. It's what logs a
principal in to the store, so you place a hive's or service's certificate by
hand. The nix option for it takes a path to the certificate, never its bytes,
because a literal in a nix expression lands in the world-readable nix store.
A hive hands each agent its certificate; the agent pulls everything else
itself.

**Each credential has a minter, a reader and a renewal story.** Who mints it, who
reads it, and whether it renews without an operator are what the
[reference table](../../4-implementation/swarm/credentials-reference.md) gives for every credential, with the
operating notes that go with them.

## Where a secret lives

A swarm generates its credentials in three different places and reads them
in a fourth, so "where does this file go" has a different answer per deployment.
This page is that answer, one row per secret.

Two rules run through all of it.

**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.

The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
the queue callout's user key, reaches every client that connects,
so the store is a perfectly good place for it.

**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.

### Topologies, by who places secrets

Read every row of the [reference tables](../../4-implementation/swarm/credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.

| topology           | what it means                                              | who places secrets                                                         |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local**      | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed**  | the swarm's services run on a host with `swarmctl`         | `swarmctl` writes what it owns; the rest is still generated in place       |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host         | the operator provides the file and names it in config                      |

## Swarm-level secrets

The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#swarm-level--one-of-each-per-swarm).

Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
test worth applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.

The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
each with the secret publisher's identity and writes it to the file authelia loads. A store
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
database, and a new issuer key invalidates every token issued. A container without the
publisher's identity generates the files as before.

The same unit mints the client secrets, straight into the store. It reads
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from
`swarm/services/swarm-authelia-bridge/oidc/client` into
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
answer keeps the bridge down while it retries. A container without the publisher's
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.

The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.

Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.

The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.

The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.

⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.

The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.

⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.

## Hive-level secrets

The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#hive-level--one-of-each-per-hive).

The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
mint in authelia's container has stored the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.

The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the mint in
authelia's container writes that path whether the reader is a network away or in the
container next door.

That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.

Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.

Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.

The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.

The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.

A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.

⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](../../4-implementation/swarm/credentials.md#per-principal-identities).

Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.

⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](../../4-implementation/swarm/credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.

Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.

All agents share one policy, `hive-agent`, which the agent's cert-auth role
attaches and no other. It states the rule with `{{identity.entity.name}}`
where an agent's name would go, so the rule is one document and a change to it
is one write. `swarm-controller` writes it at agent creation and at its own
start.

For that the controller names each agent's identity entity after the agent: at
each certificate mint or renewal, before it writes the agent's cert-auth role,
and for every agent at its own start. It reads the cert mount's accessor and
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
entity a first login created under a generated name. At a mint or renewal this is
a precondition: when the store refuses a call, or the alias sits on any other
entity, the mint fails and the role stays as it was, so a live agent keeps
working until an operator resolves it. The start pass only logs the same
failures, because it writes no role. The controller needs the `identity/` and
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.

A cert-auth role that predates the shared policy may still name a policy
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
objects stay in the store and keep granting what they did, to a token that
already carries them and to a role not yet rewritten. A live agent's role is
rewritten at its next certificate renewal.

A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and rewrites the roles to attach it. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.

### What a missing leaf costs

Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.

That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:

- **A host with no store identity at all** holds no `clientCertFile` either, so
  the first clause is false. That's the supported no-store deployment, and also
  the one an operator passes through while bringing a hive up — neither should
  fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
  defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
  unit to skip, so the second clause is false.

The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.

## The browser UI

OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.

## Adding a secret

State three things, in the row you add to the [reference](../../4-implementation/swarm/credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.