vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: put the credentials prose on the overview and the tables on the reference

Browse source 
The folded-in secrets.md text sat after the reference's table, so the
reference was half prose. The overview now holds all the prose (where a
secret lives, topologies, the swarm- and hive-level notes, how a reader
reaches the store, the browser UI, adding a secret) and the reference holds
only the tables, each pointing at the section that explains it. No text
dropped; links that asked for the explanation now go to the overview, and
the ones that asked for a list or a row go to the reference.
This commit is contained in:
iris 2026-10-26 18:05:38 +01:00 • committed by mara
commit b53365e371
18 changed files with 618 additions and 593 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/README.md
View file

@ -78,9 +78,10 @@ declarations.
[`networking/snapshot-store.md`](networking/snapshot-store.md). [`networking/snapshot-store.md`](networking/snapshot-store.md).
- **Who mints each credential, who reads it, and how does it rotate — and - **Who mints each credential, who reads it, and how does it rotate — and
where's that shape headed?** → [`swarm/credentials.md`](swarm/credentials.md) where's that shape headed?** → [`swarm/credentials.md`](swarm/credentials.md)
(the overview) and [`swarm/credentials-reference.md`](swarm/credentials-reference.md) (the rules, where each secret goes and why) and
(each credential, current state and target state, and where each file lives [`swarm/credentials-reference.md`](swarm/credentials-reference.md) (the
today). tables: each credential, current state and target state, and where each
file lives today).
- **How does the secret store come up, who writes its grants, how's it - **How does the secret store come up, who writes its grants, how's it
sealed?** → [`swarm/bao.md`](swarm/bao.md). sealed?** → [`swarm/bao.md`](swarm/bao.md).
- **How does an agent get a prompt at set times, and why doesn't it fire - **How does an agent get a prompt at set times, and why doesn't it fire

2
docs/getting-started/setup.md
View file

@ -9,7 +9,7 @@ Every command runs as **root on the host**.
> **Not all-local?** Each step says which host it runs on. A split swarm > **Not all-local?** Each step says which host it runs on. A split swarm
> also has credentials you can't generate where they're read — each > also has credentials you can't generate where they're read — each
> host's mTLS identity at the secret store. Read [`swarm/credentials-reference.md`](../swarm/credentials-reference.md) first; it > host's mTLS identity at the secret store. Read [`swarm/credentials.md`](../swarm/credentials.md) first; it
> says which files you place, and where. > says which files you place, and where.
## 1 · Secret store ## 1 · Secret store

2
docs/scheduler/observability.md
View file

@ -148,7 +148,7 @@ run lands a changed value it restarts the collector. A timer runs it again
every ten minutes, so a secret rotated later reaches the collector within every ten minutes, so a secret rotated later reaches the collector within
about ten minutes. On a hive that about ten minutes. On a hive that
runs the store the leaf is already there; elsewhere it's the credential you runs the store the leaf is already there; elsewhere it's the credential you
place — see [secret placement](../swarm/credentials-reference.md). place — see [secret placement](../swarm/credentials.md).
To deliver the secret some other way, name the file. The fetch unit isn't To deliver the secret some other way, name the file. The fetch unit isn't
rendered, and the collector waits up to 120s for the file to appear: rendered, and the collector waits up to 120s for the file to appear:

2
docs/swarm/README.md
View file

@ -169,7 +169,7 @@ it's stored in the queue, not in the daemon.
An agent connects to the queue as itself, with a credential `swarm-controller` An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials-reference.md`](credentials-reference.md)). The harness reads it from secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is: holds it in memory only. The one thing to configure is where the queue is:

2
docs/swarm/bao.md
View file

@ -114,7 +114,7 @@ Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about certificates come from a real internal CA; the store has no opinion about
which. A host that does **not** deploy the store names the reader's three which. A host that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see itself, plus a pair for every principal it runs — see
[per-principal identities](credentials-reference.md#per-principal-identities) for the list. [per-principal identities](credentials-reference.md#store-identities-one-per-principal) for the list.
They're the credentials the store can't hand you, being what opens it, so the They're the credentials the store can't hand you, being what opens it, so the
store's host signs them and the operator copies them to the host that needs store's host signs them and the operator copies them to the host that needs
them and names them in config. The service principals' leaves are at the top of them and names them in config. The service principals' leaves are at the top of

552
docs/swarm/credentials-reference.md
View file

@ -4,8 +4,8 @@ Every credential in the swarm's secret store, in one table: who mints it, who
reads it, and whether it renews on its own. The overview and the rules a reads it, and whether it renews on its own. The overview and the rules a
credential follows are in [`credentials.md`](credentials.md); this page is credential follows are in [`credentials.md`](credentials.md); this page is
the detail, so it changes whenever someone adds a credential or moves its minting. the detail, so it changes whenever someone adds a credential or moves its minting.
The second half, [where each secret lives](#where-each-secret-lives), maps the The second half, [where each secret lives on disk](#where-each-secret-lives-on-disk), maps the
files that exist on disk today. files that exist today.
In the table, **automatic re-mint** is something replacing the stored value In the table, **automatic re-mint** is something replacing the stored value
without an operator. **Automatic re-pull** is the reader picking up a replaced without an operator. **Automatic re-pull** is the reader picking up a replaced
@ -84,60 +84,25 @@ can't compare file times it logs `cannot compare the mtime of <path> … —
leaving <unit> as it is` at `err`, and the service keeps the old value until leaving <unit> as it is` at `err`, and the service keeps the old value until
the next timer tick. the next timer tick.
## Where each secret lives ## Where each secret lives on disk
A swarm generates its credentials in three different places and reads them One row per secret, by the place that owns it: who generates it, where the file
in a fourth, so "where does this file go" has a different answer per deployment. or store path is, and what an operator of a hive elsewhere places. How the
This page is that answer, one row per secret. pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.md).
Two rules run through all of it. ### Swarm-level — one of each per swarm
**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.
The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
the queue callout's user key, reaches every client that connects,
so the store is a perfectly good place for it.
**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.
## Topologies, by who places secrets
Read every row below against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store ([how](credentials.md#hive-level-secrets)) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | | the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
@ -162,184 +127,7 @@ neither is a renaming of the other.
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | | the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | | the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
Authelia mints the three keys for itself, in-container, precisely ### Hive-level — one of each per hive
because nothing outside that container ever reads them. **That's the test worth
applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
### The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Hive-level — one of each per hive
| secret | generated by | lives at | | secret | generated by | lives at |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
@ -349,154 +137,7 @@ such as the root token.
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token | | this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
The forge and the homeserver read their copies out of the store, the way ### Store identities, one per principal
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
publisher in authelia's container has written the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the table below names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names.
`swarm-controller` writes that policy at agent creation and rewrites every
agent's at its own start, so a change to it reaches existing agents with the
next controller restart.
The agent's cert-auth role attaches a second policy, `hive-agent`, shared by
all agents. It states the same rule with `{{identity.entity.name}}` where the
per-agent policy has the agent's name, so the rule is one document. For that
the controller names each agent's identity entity after the agent: at agent
creation, and for every agent at its own start, it reads the cert mount's
accessor and creates the entity `<agent>` with the alias `hive-agent-<agent>`,
or renames the entity a first login created under a generated name. When the alias
sits on any other entity, the controller logs an error and changes nothing. It needs
the `identity/` and `sys/auth/cert` grants in `swarm-bao.nix`'s
`controllerPolicyText` for that. Without them it logs a warning and the agent
keeps working through its own policy, which stays attached until every agent
has its entity. An agent's role gains `hive-agent` at its next certificate
mint.
A hive's cert-auth role `hive-<hive>` likewise attaches a second policy,
`hive-hives`, shared by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and attaches it when it rewrites the roles. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and the role
attaches the hive's own policy alone, which stays attached.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these ten leaves has identity and sharing one merges the identities. Each of these ten leaves has
@ -517,165 +158,6 @@ login minter's, the forge host's and Grafana's, which reach more than one:
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
Each reaches a path that names a swarm service rather than a hive, so one role The option names are `deploy.bao.<pair>`; the common names, what a missing leaf
each is enough and their subjects are the flat costs and how to place a leaf by hand are in
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, [per-principal identities](credentials.md#per-principal-identities).
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.claudeLoginCommonName` and
`deploy.bao.forgePublisherCommonName`.
The last two are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token. The other roles keep service tokens, because some of
their readers hold a handle for longer than five minutes.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
### OIDC login
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key` and
`secret/data/swarm/services/review-classifier/acp-provider`. That shows the key
tree and each key's versions and timestamps, and no value: the store refuses
every `secret/data/` read. The two writes are the review classifier's keys,
which the operator fills in by hand. Anything more needs a token.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | `swarm-secret-publish`, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.
## Adding a secret
State three things, in the row you add above: **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.

546
docs/swarm/credentials.md
View file

@ -5,8 +5,8 @@
The swarm's credential store is bao. This page is the overview of how the swarm The swarm's credential store is bao. This page is the overview of how the swarm
treats credentials; [`credentials-reference.md`](credentials-reference.md) lists treats credentials; [`credentials-reference.md`](credentials-reference.md) lists
each one: who mints it, who reads it, and how it renews, with the notes for each one: who mints it, who reads it, and how it renews, with the notes for
operating them. It describes the target, not what's on disk today; the reference also operating them. It describes the target, not what's on disk today; the second half of the
maps [the files that exist right now](credentials-reference.md#where-each-secret-lives). reference maps [the files that exist right now](credentials-reference.md#where-each-secret-lives-on-disk).
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
@ -25,3 +25,545 @@ itself.
reads it, and whether it renews without an operator are what the reads it, and whether it renews without an operator are what the
[reference table](credentials-reference.md) gives for every credential, with the [reference table](credentials-reference.md) gives for every credential, with the
operating notes that go with them. operating notes that go with them.
## Where a secret lives
A swarm generates its credentials in three different places and reads them
in a fourth, so "where does this file go" has a different answer per deployment.
This page is that answer, one row per secret.
Two rules run through all of it.
**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.
The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
the queue callout's user key, reaches every client that connects,
so the store is a perfectly good place for it.
**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.
### Topologies, by who places secrets
Read every row of the [reference tables](credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level secrets
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
Authelia mints the three keys for itself, in-container, precisely
because nothing outside that container ever reads them. **That's the test worth
applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
### The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Hive-level secrets
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
publisher in authelia's container has written the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names.
`swarm-controller` writes that policy at agent creation and rewrites every
agent's at its own start, so a change to it reaches existing agents with the
next controller restart.
The agent's cert-auth role attaches a second policy, `hive-agent`, shared by
all agents. It states the same rule with `{{identity.entity.name}}` where the
per-agent policy has the agent's name, so the rule is one document. For that
the controller names each agent's identity entity after the agent: at agent
creation, and for every agent at its own start, it reads the cert mount's
accessor and creates the entity `<agent>` with the alias `hive-agent-<agent>`,
or renames the entity a first login created under a generated name. When the alias
sits on any other entity, the controller logs an error and changes nothing. It needs
the `identity/` and `sys/auth/cert` grants in `swarm-bao.nix`'s
`controllerPolicyText` for that. Without them it logs a warning and the agent
keeps working through its own policy, which stays attached until every agent
has its entity. An agent's role gains `hive-agent` at its next certificate
mint.
A hive's cert-auth role `hive-<hive>` likewise attaches a second policy,
`hive-hives`, shared by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and attaches it when it rewrites the roles. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and the role
attaches the hive's own policy alone, which stays attached.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. The leaves, their option names
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.claudeLoginCommonName` and
`deploy.bao.forgePublisherCommonName`.
The last two are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token. The other roles keep service tokens, because some of
their readers hold a handle for longer than five minutes.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
### OIDC login
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key` and
`secret/data/swarm/services/review-classifier/acp-provider`. That shows the key
tree and each key's versions and timestamps, and no value: the store refuses
every `secret/data/` read. The two writes are the review classifier's keys,
which the operator fills in by hand. Anything more needs a token.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | `swarm-secret-publish`, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.
## Adding a secret
State three things, in the row you add to the [reference](credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.

8
docs/swarm/services.md
View file

@ -146,13 +146,13 @@ any other session gets `deny`.
The container also runs `swarm-secret-publish`, which copies the client The container also runs `swarm-secret-publish`, which copies the client
secrets authelia mints into the swarm secret store under the secret secrets authelia mints into the swarm secret store under the secret
publisher's own store leaf, handed in as a system credential; see publisher's own store leaf, handed in as a system credential; see
[`credentials-reference.md`](credentials-reference.md). Authelia doesn't wait for it, so the IdP comes up [`credentials.md`](credentials.md). Authelia doesn't wait for it, so the IdP comes up
while the store is down, and the publisher retries every 30 seconds. The while the store is down, and the publisher retries every 30 seconds. The
container's journal forwarder reads its client secret back from the store container's journal forwarder reads its client secret back from the store
under the same leaf, and is off where the container has none. under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`credentials-reference.md`](credentials-reference.md) for where authelia relying-party flow, and [`credentials.md`](credentials.md) for where authelia
generates and reads each of its keys. generates and reads each of its keys.
### Metrics (VictoriaMetrics + Grafana) ### Metrics (VictoriaMetrics + Grafana)
@ -193,7 +193,7 @@ authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every (`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`credentials-reference.md`](credentials-reference.md). Grafana start — see [`credentials.md`](credentials.md).
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
@ -311,7 +311,7 @@ second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation authelia session. Because one bearer opens both read APIs, a rotation
(see [`credentials-reference.md`](credentials-reference.md)) restarts Grafana once, and the log store (see [`credentials.md`](credentials.md)) restarts Grafana once, and the log store
accepts the new value within a minute. accepts the new value within a minute.
The collector also scrapes this store's own metrics, at The collector also scrapes this store's own metrics, at

2
docs/swarm/sso.md
View file

@ -314,7 +314,7 @@ gateway refuses its token too. See [`swarmctl/README.md`](../../swarmctl/README.
in the store (or, as root on the Grafana host, from in the store (or, as root on the Grafana host, from
`/run/swarm-grafana-secrets/admin-password`) `/run/swarm-grafana-secrets/admin-password`)
and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana) and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana)
and [`credentials-reference.md`](credentials-reference.md). SSO logins are the everyday way in, which and [`credentials.md`](credentials.md). SSO logins are the everyday way in, which
is why `swarm.grafana.oidc.role` defaults to `Admin`. is why `swarm.grafana.oidc.role` defaults to `Admin`.
The three secrets Grafana reads live on the host's tmpfs, so after a reboot The three secrets Grafana reads live on the host's tmpfs, so after a reboot
Grafana stays down (no login, no dashboards) until the secret store Grafana stays down (no login, no dashboards) until the secret store

2
docs/swarm/ui.md
View file

@ -817,7 +817,7 @@ always serves the client at `/` on that vhost, so the entry is present
whenever the swarm names a gateway host. whenever the swarm names a gateway host.
The **Bao** entry opens the secret store's web UI, which admits members of The **Bao** entry opens the secret store's web UI, which admits members of
authelia's `admins` group only; see [`credentials-reference.md`](credentials-reference.md#the-browser-ui). authelia's `admins` group only; see [`credentials.md`](credentials.md#the-browser-ui).
<details><summary>Adding a swarm service name: the two wiring sites</summary> <details><summary>Adding a swarm service name: the two wiring sites</summary>

2
nix/host-modules/nats-callout-removed.nix
View file

@ -20,7 +20,7 @@ let
The queue's callout issuer and user keys live in the swarm secret store: The queue's callout issuer and user keys live in the swarm secret store:
swarm-bao-nats-callout-xkey creates them there, and the server renders swarm-bao-nats-callout-xkey creates them there, and the server renders
their public halves from it. Remove this definition. See their public halves from it. Remove this definition. See
docs/swarm/credentials-reference.md. docs/swarm/credentials.md.
''; '';
# The seed-file options name a key an operator may want to keep. # The seed-file options name a key an operator may want to keep.
seedFile = name: '' seedFile = name: ''

2
nix/host-modules/otel.nix
View file

@ -895,7 +895,7 @@ in
the default is the file swarm-bao-hive-otel-oidc.service fetches the default is the file swarm-bao-hive-otel-oidc.service fetches
under it. On a hive that runs the store both are set already; under it. On a hive that runs the store both are set already;
elsewhere place the hive's leaf and name it there — see elsewhere place the hive's leaf and name it there — see
docs/swarm/credentials-reference.md. docs/swarm/credentials.md.
''; '';
} }
{ {

2
nix/host-modules/swarm-bao.nix
View file

@ -2,7 +2,7 @@
# `swarm-bao` nixos-container. # `swarm-bao` nixos-container.
# #
# Today every credential in ./swarm-*.nix is minted where it is read or copied # Today every credential in ./swarm-*.nix is minted where it is read or copied
# there by a delivery unit (docs/swarm/credentials-reference.md), which ties each secret's # there by a delivery unit (docs/swarm/credentials.md), which ties each secret's
# lifetime to its container's. # lifetime to its container's.
# #
# ⚠️ It authenticates hive clients with a CLIENT CERTIFICATE, not with the # ⚠️ It authenticates hive clients with a CLIENT CERTIFICATE, not with the

4
nix/host-modules/swarm-grafana.nix
View file

@ -592,7 +592,7 @@ in
On a hive that runs the store, glue-bao-tls.nix supplies both as On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/credentials-reference.md. from that CA out of band and named here — see docs/swarm/credentials.md.
''; '';
} }
{ {
@ -718,7 +718,7 @@ in
# with it, so skipping the store when the producer is local would buy a # with it, so skipping the store when the producer is local would buy a
# round trip and cost a second delivery path, a second way for the file # round trip and cost a second delivery path, a second way for the file
# to be wrong, and a gate whose false arm is a Grafana with no login. # to be wrong, and a gate whose false arm is a Grafana with no login.
# Recorded in docs/swarm/credentials-reference.md. # Recorded in docs/swarm/credentials.md.
# #
# It is a store reader, and this is what the store's readers do: a cert # It is a store reader, and this is what the store's readers do: a cert
# login that fails LOUDLY, since every state it fails on is one a retry # login that fails LOUDLY, since every state it fails on is one a retry

2
nix/host-modules/swarm-nats.nix
View file

@ -360,7 +360,7 @@ in
# the host running it decides is below — which responder build it runs and # the host running it decides is below — which responder build it runs and
# the store identities its units present. `enable` already lives in # the store identities its units present. `enable` already lives in
# ./deploy.nix, which also carries the renames. The callout's keys are no # ./deploy.nix, which also carries the renames. The callout's keys are no
# option at all: they live in the secret store (docs/swarm/credentials-reference.md). # option at all: they live in the secret store (docs/swarm/credentials.md).
options.services.hyperhive.deploy.nats = { options.services.hyperhive.deploy.nats = {
authPackage = lib.mkOption { authPackage = lib.mkOption {
type = lib.types.package; type = lib.types.package;

4
nix/host-modules/swarm-otel.nix
View file

@ -545,7 +545,7 @@ in
# tree: that is reachable only where authelia shares this host's network # tree: that is reachable only where authelia shares this host's network
# namespace, and breaks the moment authelia is on another host. The store # namespace, and breaks the moment authelia is on another host. The store
# exists so a host holds ONE out-of-band secret — its client certificate — # exists so a host holds ONE out-of-band secret — its client certificate —
# and reads everything else with it. Recorded in docs/swarm/credentials-reference.md. # and reads everything else with it. Recorded in docs/swarm/credentials.md.
# #
# Shaped after ./swarm-grafana.nix's store reader: a cert login that fails # Shaped after ./swarm-grafana.nix's store reader: a cert login that fails
# LOUDLY, since every state it fails on is one a retry fixes, then a read # LOUDLY, since every state it fails on is one a retry fixes, then a read
@ -801,7 +801,7 @@ in
On a hive that runs the store, glue-bao-tls.nix supplies both as On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/credentials-reference.md. from that CA out of band and named here — see docs/swarm/credentials.md.
''; '';
} }
# ⚠️ There is deliberately no assertion that this collector has # ⚠️ There is deliberately no assertion that this collector has

2
swarm-nats-auth/README.md
View file

@ -48,7 +48,7 @@ start. The `ensure-callout-keys` verb creates each pair when the store holds
none, taking an issuer or user seed from the import file it is given when that none, taking an issuer or user seed from the import file it is given when that
file exists, and removes the file once the store holds a seed. It runs from file exists, and removes the file once the store holds a seed. It runs from
its own unit before the server, because the server needs the public halves to its own unit before the server, because the server needs the public halves to
start and the responder needs the server. `docs/swarm/credentials-reference.md` has the start and the responder needs the server. `docs/swarm/credentials.md` has the
paths and the boot order. paths and the boot order.
## Rules the code follows ## Rules the code follows

2
swarm-secret-client/README.md
View file

@ -7,7 +7,7 @@ field its bytes are in, and how this deployment's environment becomes a
logged-in client. logged-in client.
The surrounding picture — which secret is minted where, and why delivery is a The surrounding picture — which secret is minted where, and why delivery is a
copy rather than a bind mount — is [`docs/swarm/credentials-reference.md`](../docs/swarm/credentials-reference.md). copy rather than a bind mount — is [`docs/swarm/credentials.md`](../docs/swarm/credentials.md).
## Why a crate and not a module per binary ## Why a crate and not a module per binary