mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: the credentials page keeps the approach and moves the mechanics out

Browse source 
credentials.md drops the target-shape framing and keeps the rules, where a
secret lives, the swarm- and hive-level secrets and how to add one. The queue
responder's keys, per-principal identities, how a reader reaches the store, the
store UI's login and where the root lives move to
4-implementation/swarm/credentials.md. The per-secret reference moves to
4-implementation/swarm/credentials-reference.md. Links and comments follow.
This commit is contained in:
iris 2026-11-10 22:26:10 +01:00
commit ead667da42
13 changed files with 350 additions and 346 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/2-handbook/external-accounts/github.md
View file

@ -50,7 +50,7 @@ when the token changed. It needs a store identity
(`services.hyperhive.agent.bao.addr`); an agent without one gets no token.
The unit never deletes the file: a `github-token` already in place stays
when the store holds none or doesn't answer. Where the token lives and who
reads it: [credentials-reference.md](../swarm/credentials-reference.md).
reads it: [credentials-reference.md](../../4-implementation/swarm/credentials-reference.md).
## Security

2
docs/2-handbook/swarm/README.md
View file

@ -52,7 +52,7 @@ grafana, the controller, authelia, …) →
→ [`sso.md`](sso.md)
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
every credential the swarm holds, who mints it, who reads it, how it renews
and where it lives → [`credentials-reference.md`](credentials-reference.md) ·
and where it lives → [`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md) ·
how the store comes up, who writes its grants and how it unseals →
[`bao.md`](../../4-implementation/swarm/bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to

2
docs/2-handbook/swarm/archive.md
View file

@ -187,4 +187,4 @@ on disk. The archive host renders its users file (password hashes only) into
`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the
S3 key from the store when a process starts and passes it to rclone in the
environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the
[credentials reference](credentials-reference.md).
[credentials reference](../../4-implementation/swarm/credentials-reference.md).

299
docs/2-handbook/swarm/credentials.md
View file

@ -1,14 +1,10 @@
# Credentials: the target shape
<!-- vale write-good.Passive = NO -->
# Credentials
The swarm's credential store is bao. This page is the overview of how the swarm
treats credentials; [`credentials-reference.md`](credentials-reference.md) lists
each one: who mints it, who reads it, and how it renews, with the notes for
operating them. It describes the target, not what's on disk today; the second half of the
reference maps [the files that exist right now](credentials-reference.md#where-each-secret-lives-on-disk).
<!-- vale write-good.Passive = YES -->
treats credentials; [`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md)
lists each one: who mints it, who reads it, and how it renews, with the notes for
operating them, and [`credentials.md`](../../4-implementation/swarm/credentials.md)
holds the store's mechanics.
**Store secrets stay off disk.** A process pulls a value from the store at
runtime and holds it in memory only.
@ -23,7 +19,7 @@ itself.
**Each credential has a minter, a reader and a renewal story.** Who mints it, who
reads it, and whether it renews without an operator are what the
[reference table](credentials-reference.md) gives for every credential, with the
[reference table](../../4-implementation/swarm/credentials-reference.md) gives for every credential, with the
operating notes that go with them.
## Where a secret lives
@ -55,7 +51,7 @@ container that starts after it.
### Topologies, by who places secrets
Read every row of the [reference tables](credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
Read every row of the [reference tables](../../4-implementation/swarm/credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
@ -69,7 +65,7 @@ neither is a renaming of the other.
## Swarm-level secrets
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#swarm-level--one-of-each-per-swarm).
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
@ -157,126 +153,9 @@ unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
### The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
### The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Hive-level secrets
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
@ -365,7 +244,7 @@ every swarm service's client secret, and that's stated in
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
[per-principal identities](../../4-implementation/swarm/credentials.md#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
@ -384,7 +263,7 @@ internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
plus one leaf per principal it runs (the [table in the reference](../../4-implementation/swarm/credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
@ -431,75 +310,6 @@ spell `hive-hives`: it would be a hive called `hives`, and
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. The leaves, their option names
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`, `deploy.bao.archiveCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](../../4-implementation/swarm/bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token.
The other roles issue service tokens that live an hour (`token_ttl=1h` and
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
agent and hive roles the controller writes through
`SecretStore::write_cert_role`, which takes the same value from
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
expires, and the mount default of 32 days leaves one lease per login for a
month; at the 17 thousand logins a day this swarm made, that came to half a
million leases. A reader logs in per operation and holds the token for seconds.
The longest hold is the controller's forge work, which stops after two minutes.
A client that keeps a `SecretStore` handle longer than an hour must log in
again, because nothing renews the token. A role the controller writes takes the
TTL at its next write: a hive's at the controller's start, an agent's at its
next certificate mint or renewal. At its start the controller also reads every
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
differs from the TTL, writes those two fields and no others. The store keeps the
fields a write leaves out, so the role's certificate, allowed names and
policies stay. A role already at the TTL isn't written. The start logs one line
with `rewritten`, `already_current` and `failed` counts, and a role that fails
gets a warning that names it, then the pass moves on. Tokens issued before
expire on their own; the controller revokes nothing. At that rate an hour keeps
about 700 tokens live.
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
it, and a deployed store never rewrites it, so the longer TTL would not reach
it; the granter logs in and finishes within seconds.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
@ -526,35 +336,6 @@ store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
@ -576,64 +357,8 @@ hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
### OIDC login
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
`url`, `cache` and `public-key`). That shows the key tree and each key's
versions and timestamps, and no value: the store refuses every `secret/data/`
read. The writes are the leaves the operator fills in by hand: the review
classifier's keys, and the cache's tokens and facts, which configure an external
cache or replace the in-swarm cache's own. The cache's host units create those
leaves only while the store holds none, so a saved value survives their next run
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
Anything more needs a token.
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
`services` > `attic`, pick the leaf and choose **Create new version**. The
form starts empty because the login can't read the current value: type the
key `value` and the new value, and save. A new version replaces the whole
secret, so `value` is the only key to enter. `bao kv put` with the same login
works too.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../../1-getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.
## Adding a secret
State three things, in the row you add to the [reference](credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
State three things, in the row you add to the [reference](../../4-implementation/swarm/credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.

2
docs/2-handbook/swarm/ui.md
View file

@ -968,7 +968,7 @@ GitHub units never delete a file, so a token already fetched into `<state>`
stays there.
Where each credential lives and who reads it:
[`credentials-reference.md`](credentials-reference.md).
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md).
## Enabling

2
docs/4-implementation/networking/network.md
View file

@ -246,7 +246,7 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
minute. Nothing rotates a credential: to replace one, delete its leaf in the
store and the next pass mints a new one. HTTP Basic crosses the bridge in
clear between an agent and the proxy. See
[the credentials reference](../../2-handbook/swarm/credentials-reference.md).
[the credentials reference](../../4-implementation/swarm/credentials-reference.md).
A request without a valid credential gets a 407, once the destination
checks below have passed it.

4
docs/4-implementation/swarm/bao.md
View file

@ -3,7 +3,7 @@
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
credential an agent or service needs comes from it under that
principal's own certificate identity. What lives in it and who reads what:
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md). This page covers the store itself — how it
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md). This page covers the store itself — how it
comes up, who may write its grants, and how it's sealed and reached.
The operator steps (init, the one-time granter bootstrap) are in
@ -119,7 +119,7 @@ Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about
which. A host that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see
[per-principal identities](../../2-handbook/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
[per-principal identities](../../4-implementation/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
They're the credentials the store can't hand you, being what opens it, so the
store's host signs them and the operator copies them to the host that needs
them and names them in config. The service principals' leaves are at the top of

2
docs/4-implementation/swarm/builder.md
View file

@ -668,7 +668,7 @@ status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
status with its own forge token. Details per path:
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md).
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md).
The builder exports build counts, durations and queue wait; see
[Swarm builder metrics](../scheduler/observability.md#swarm-builder-metrics).

2
docs/4-implementation/swarm/ca.md
View file

@ -24,7 +24,7 @@ IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and `read` on its own `attic/public-key`, `attic/pull-token` and
`squid-selfcheck`
(see [the cache facts for hive-priv](../../2-handbook/swarm/credentials-reference.md) and the
(see [the cache facts for hive-priv](../../4-implementation/swarm/credentials-reference.md) and the
squid self-check password), and nothing
else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every

196
docs/4-implementation/swarm/credentials-reference.md Normal file
View file

@ -0,0 +1,196 @@
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/`, and before each deploy to the hive | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
# Credentials: the per-secret reference
Every credential in the swarm's secret store, in one table: who mints it, who
reads it, and whether it renews on its own. The overview and the rules a
credential follows are in [`credentials.md`](../../2-handbook/swarm/credentials.md); this page is
the detail, so it changes whenever someone adds a credential or moves its minting.
The second half, [where each secret lives on disk](#where-each-secret-lives-on-disk), maps the
files that exist today.
In the table, **automatic re-mint** is something replacing the stored value
without an operator. **Automatic re-pull** is the reader picking up a replaced
value without a restart. ✅ means it happens today, ❌ means it doesn't, and
the rest of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice render` in `hive-matrix`, when the store answers and holds none; before that, the tokens an earlier version kept in the container, stored unchanged | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` render the new pair and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read once; a store down at start is re-read in the background until it answers |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation, and by the renewal pass when none is stored or its cert-auth role is missing | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity; minted when none is stored or its cert-auth role is missing, for agents not `Destroyed` | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/observability` | `swarm-controller`, when an operator sets, clears or creates an agent's permissions with `logs` or `metrics` in the old or new list, and when it destroys the agent; the leaf stays absent for an agent that never held either | the agent (read-only, its own prefix); `swarm-logs-auth` and `swarm-metrics-agent-auth`, which deny when it's absent or unreadable | ✅ the controller rewrites it with the permission, and every five minutes makes it equal the record's | ✅ the stores honour a changed grant for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/gitlab/<label>` | `swarm-controller`, when you link a GitLab account through the API | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes; a deleted account leaves the agent's `glab` configuration on the next run |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/backend-env` | `hive-agent-backend-env`, in the agent's container, from the agent's `backendEnvironmentFile` when the leaf holds nothing (field `env`, the file's exact text); or `swarm-controller`, when you set the API-key environment in the swarm UI | the agent | ❌ nothing replaces it without you; a save in the swarm UI overwrites it, including text an agent wrote into an empty leaf; the agent's file stays as it was | ❌ the file in `/run` refreshes every 2 minutes, and the harness and the subagent daemon load it at their next start |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/agents/<agent>/proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds |
| `swarm/hives/<hive>/squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute, and while they can't reach the store they keep accepting the value they last read, for an hour at most; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the agent's deploy is held until a push lands | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, to copy it and to check narinfos | ❌ write a new token | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored and again whenever the cache reports another key (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ✅ on the cache's host, the unit's next run replaces any other value, a hand edit included, with the cache's key; with an external cache, replace it yourself | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes, and before each deploy to the hive; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/hives/<hive>/archive/<agent>` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run |
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes |
| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->
## Operating notes
The name `main` belongs to the swarm's own homeserver account, which the
controller mints, so the controller rejects any account you link under it.
`swarm-controller` checks every forge token itself: it looks at every agent
that has a store identity at start and every five minutes. For any whose
stored token is missing or stale it creates the forge user if there isn't one,
then mints the token. To check one agent now:
```sh
swarmctl agent mint-forge-token <agent>
```
An agent without a store identity gets no swarm token and keeps using the
`forge-token` file in its state dir, if it has one; re-create it with
`swarmctl agent create` to give it one. The swarm leaves an agent it already
declares as it is. For an agent with no declaration, such as one only its hive
knows about, `agent create` declares it `paused`; set it `up` from its card in
the swarm UI.
⚠️ **Re-create existing agents before deploying a hive-side change that makes a
container require a credential it may not have.** A container whose store
identity is absent doesn't start, so create the identities first, setting any
newly paused agent `up` as above.
**Revoking an agent's queue credential.** Declaring an agent `destroyed` —
`PUT /api/hives/{hive}/agents/{agent}/state` — deletes
`swarm/agents/<agent>/queue` and every version it held. An open queue
connection closes within an hour. An agent recreated under the same name gets a
fresh secret. It leaves the agent's mTLS leaf alone. If the store refuses the
delete, `swarm-controller` logs an `error` naming the agent and the teardown
continues; declare the agent destroyed again to re-run the revocation.
A failed store login retries every 30 seconds for a day. When a later attempt
lands a changed value, the service that loaded it restarts. If the fetch unit
can't compare file times it logs `cannot compare the mtime of <path> … —
leaving <unit> as it is` at `err`, and the service keeps the old value until
the next timer tick.
## Where each secret lives on disk
One row per secret, by the place that owns it: who generates it, where the file
or store path is, and what an operator of a hive elsewhere places. How the
pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-handbook/swarm/credentials.md).
### Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store is down, refuses the login, or fails the read of a leaf (a refusal, a timeout, an error), the unit runs on the files, logs the store's error, and writes nothing to the store; the unit seeds or mints only a leaf the store reports as absent. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting |
| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](../../2-handbook/swarm/credentials.md#hive-level-secrets)) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](../../2-handbook/swarm/credentials.md#swarm-level-secrets) |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](../../2-handbook/swarm/credentials.md#hive-level-secrets) |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services/<grafana client id>/secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk, unused once the store holds the key. A store that's down or doesn't grant the leaf fails the unit, and Grafana doesn't start | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it |
| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 until the store answers: the controller re-reads it in the background, at most five minutes apart, and registers the forge hooks once it has the secret |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once. An attempt that fails or runs past 30 seconds retries after 5 seconds, then at growing intervals up to 5 minutes | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](../../2-handbook/swarm/claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the fetch fails fast while the store is down and retries with a delay that grows to 5 minutes; the attempt that succeeds starts the collector, which isn't part of the container's boot, and it resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. `ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`, public), the host's own CA bundle followed by the swarm's root from `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem`, so that only nix trusts the root. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these leaves has
a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the cache's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, `swarm-bao-grafana-secret-key`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` and `swarm/services/<grafana client id>/secret-key` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services/<runner forwarder client id>/oidc/client` |
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`, reads and deletes its metadata; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
| `swarm-archive-users`, `swarm-archive-prune` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The option names are `deploy.bao.<pair>`; the common names, what a missing leaf
costs and how to place a leaf by hand are in
[per-principal identities](credentials.md#per-principal-identities).

278
docs/4-implementation/swarm/credentials.md Normal file
View file

@ -0,0 +1,278 @@
# Credentials: mechanics
The store's identities, the queue responder's keys, how a reader reaches the
store, the store UI's login and where the root lives. The overview and the rules
a credential follows are in [credentials](../../2-handbook/swarm/credentials.md);
each credential's minter, reader and renewal are in
[the reference](credentials-reference.md).
## The queue's callout nkeys
The responder holds two nkeys besides the callout xkey below. It signs every
user JWT it issues with the **issuer** key, an account key, so possession of
the issuer seed is the authority to admit anyone to the queue. It connects to
the queue as the **user** key, the one identity the server exempts from
callout. You configure nothing about either: both live in the swarm secret
store beside the xkey, and come into being on the queue's first boot.
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
The unit that creates the xkey creates these too, under the same rules: only
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
The server renders both public halves into its `auth_callout` block as it
starts, and doesn't start without them.
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
none, so the key keeps its public half. Once a read of the store returns a
seed, the verb removes the file. When the store already holds a different
seed, the verb keeps the store's seed, logs a warning naming the file, then
removes it.
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
option and where the keys live now.
To replace either key, delete both of its paths the way the xkey's section
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
One consequence worth knowing before you debug it: the queue assembles its
config each time the server starts rather than at build time, in every mode,
because the callout keys exist only at runtime. A malformed config
surfaces when the server starts instead of when the system builds. The server
names the offending file and refuses to run.
## The queue's callout xkey
The server seals every auth-callout request to a curve key, the callout
xkey, and the responder seals its reply back, so nothing else in the callout
account or on the wire can read the request or the user JWT minted for it.
You configure nothing about the key: it lives in the swarm secret store and
comes into being on the queue's first boot.
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
Only the `swarm-nats-auth` role reads either path, and it may create each but
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
of `swarm/services/`, denies it the seed's path.
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
`swarm-nats` container, under the responder's store leaf:
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
seed, and otherwise only reads it. It needs no queue, so on a first boot
the key exists before the server that needs it.
2. It reads the public half back from the store into
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
3. Each time `nats-server` starts, it renders that key into
`auth_callout.xkey`. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store.
A responder that can't read it doesn't start, rather than answering
requests it can't open.
The store has to be reachable each time the responder starts. When it's down,
the fetch unit fails and retries for a day. The server starts with the key
fetched last time, or not at all on a first boot. A responder that starts
while the store is down exits and retries every 5 seconds, and until one
start reads the seed the queue refuses every client, OIDC ones included. The
retry that lands the file starts or restarts the server, and the responder
comes up with it.
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
`authBaoClientKeyFile`. On the store's host both default to the leaf
`glue-bao-tls.nix` mints there. A queue host without the store gets no
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
name it in both options. Until you do, evaluation fails naming both.
Where the queue and the store run on different hosts, deploy the store's host
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
the seeds' paths; where the queue runs on that host too, the fetch waits for
both. A queue host deployed before it can't read or create the keys, so its
fetch fails, and the queue refuses every client until you deploy the store's
host: the server starts only on public keys fetched before, and the responder
can't load its seeds. Seed files the import is waiting for stay where they
are. If that takes longer than the fetch's day of retries, run
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
Two starts that create the key at once can't both win: each write is a
create that fails once the path holds anything, and the start that lost reads
the winner's key back.
To replace the key, remove both paths with their metadata and every version,
then restart the fetch unit on the queue's host. Its restart of `nats.service`
restarts the responder too, which then loads the new seed:
```sh
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey
```
`bao kv delete` isn't enough. It deletes the latest version but keeps the
path's metadata, so the store still counts the path as existing and refuses
the create that makes a new key. No principal hyperhive configures may
delete either path's metadata, so run the two deletes with a token that may,
such as the root token.
## Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. The leaves, their option names
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`, `deploy.bao.archiveCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](../../4-implementation/swarm/bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](../../2-handbook/swarm/credentials.md#what-a-missing-leaf-costs) below.
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
responders log in on every check, and the store keeps no record of a batch token,
so unauthenticated requests don't grow its token storage. Nothing can renew
or revoke a batch token.
The other roles issue service tokens that live an hour (`token_ttl=1h` and
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
agent and hive roles the controller writes through
`SecretStore::write_cert_role`, which takes the same value from
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
expires, and the mount default of 32 days leaves one lease per login for a
month; at the 17 thousand logins a day this swarm made, that came to half a
million leases. A reader logs in per operation and holds the token for seconds.
The longest hold is the controller's forge work, which stops after two minutes.
A client that keeps a `SecretStore` handle longer than an hour must log in
again, because nothing renews the token. A role the controller writes takes the
TTL at its next write: a hive's at the controller's start, an agent's at its
next certificate mint or renewal. At its start the controller also reads every
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
differs from the TTL, writes those two fields and no others. The store keeps the
fields a write leaves out, so the role's certificate, allowed names and
policies stay. A role already at the TTL isn't written. The start logs one line
with `rewritten`, `already_current` and `failed` counts, and a role that fails
gets a warning that names it, then the pass moves on. Tokens issued before
expire on their own; the controller revokes nothing. At that rate an hour keeps
about 700 tokens live.
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
it, and a deployed store never rewrites it, so the longer TTL would not reach
it; the granter logs in and finishes within seconds.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
agent CA. The listener reads both from `listener-client-ca.pem`, which no
cert-auth role pins. The passthrough carries
whichever certificate the reader presents, unchanged.
## OIDC login to the store UI
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
`create` and `update` on
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
`url`, `cache` and `public-key`). That shows the key tree and each key's
versions and timestamps, and no value: the store refuses every `secret/data/`
read. The writes are the leaves the operator fills in by hand: the review
classifier's keys, and the cache's tokens and facts, which configure an external
cache or replace the in-swarm cache's own. The cache's host units create those
leaves only while the store holds none, so a saved value survives their next run
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
Anything more needs a token.
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
`services` > `attic`, pick the leaf and choose **Create new version**. The
form starts empty because the login can't read the current value: type the
key `value` and the new value, and save. A new version replaces the whole
secret, so `value` is the only key to enter. `bao kv put` with the same login
works too.
| piece | written by |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
step. A store set up before the mount existed needs the
[one-time granter step](../../1-getting-started/setup.md) again: it re-writes
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
policy, which covers `auth/oidc/`. Until then
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. That root's private key never leaves the mount, and its
certificate is the only anchor the swarm has. Each hive fetches the certificate
from the store at boot, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the wrong
tool here. Host containers get a bind-mounted bundle assembled at boot; an
agent fetches the root itself with its own store certificate.

2
docs/4-implementation/swarm/sso.md
View file

@ -6,7 +6,7 @@ differ, and why machine callers must fail closed. The operator's view is
## What secrets exist, and where each one lives
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md) tabulates every secret in the swarm, with its
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md) tabulates every secret in the swarm, with its
generator and its path, in one place — including authelia's own keys
(session, JWT, storage-encryption, OIDC HMAC, OIDC issuer) and the two
halves of each client secret. That page's two rules — a secret is always

13
docs/README.md
View file

@ -49,12 +49,13 @@ The services that exist once per swarm. Start at
`services.hyperhive.agent.matrixAccounts`.
- **What's `/knowledge`? How does the swarm-wide knowledge repo sync,
and how to contribute a document?** → [`2-handbook/swarm/knowledge.md`](2-handbook/swarm/knowledge.md).
- **Who mints each credential, who reads it, and how does it rotate — and
where's that shape headed?** → [`2-handbook/swarm/credentials.md`](2-handbook/swarm/credentials.md)
(the rules, where each secret goes and why) and
[`2-handbook/swarm/credentials-reference.md`](2-handbook/swarm/credentials-reference.md) (the
tables: each credential, current state and target state, and where each
file lives today).
- **Who mints each credential, who reads it, and how does it rotate?** →
[`2-handbook/swarm/credentials.md`](2-handbook/swarm/credentials.md) (the
rules, where each secret goes and why, adding a secret) and
[`4-implementation/swarm/credentials-reference.md`](4-implementation/swarm/credentials-reference.md) (the
tables: each credential and where each file lives). The store's identities,
keys and login:
[`4-implementation/swarm/credentials.md`](4-implementation/swarm/credentials.md).
- **How does an agent get a prompt at set times, and why doesn't it fire
twice after downtime?** → [`2-handbook/swarm/schedules.md`](2-handbook/swarm/schedules.md).
- **How to create an agent from a ready-made set of permissions and