vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

docs/swarm/credentials.md: cut the page down to the target shape only

Browse source 
Drop the current-state table and the prose describing how the swarm
currently breaks the target (unrotated secrets persisted to disk outside
the store, no agent store identity, the appservice token's second
minter). Scope the path-never-value rule to the mTLS client certificate
— the one credential a nix option can carry, now that everything else
goes through bao. Trim the migration checklist to unconditional rules
for any credential-touching change, since there's no current-state table
left for the old existing-functionality carve-outs to reference.

Refs #4445
This commit is contained in:
atlas 2026-09-16 22:41:22 +02:00
commit 940e11ecda
1 changed files with 62 additions and 151 deletions
Download patch file Download diff file Expand all files Collapse all files

201
docs/swarm/credentials.md
View file

@ -1,159 +1,70 @@
# Credentials: current state, target state, and the rule between the two # Credentials: the target shape
Where [`secrets.md`](secrets.md) is a map of every file a swarm holds, this The swarm's credential store is bao. This page describes the shape every
page is the **direction of travel**: what the credential shape is today, what credential is meant to have — who mints it, who reads it, and how it renews
it should become, and the test a change has to pass to count as movement — not what's on disk today. [`secrets.md`](secrets.md) remains the map of
toward it rather than away. the files that exist right now; this page replaces it, and `secrets.md` gets
deleted, once the swarm's credential path matches what's described below.
It's the thing a discussion should point at. **When the migration
below is complete, this page supersedes the old credential docs** — `secrets.md`
gets deleted and this file moves into its place. Until then the two coexist and
neither is wrong: that one says where a file is, this one says whether it should
be a file at all.
Two conventions frame everything below.
**Where a nix option does carry a secret, it takes a path, never a value.** A
literal in a nix expression lands in the nix store — world-readable and
permanent. The target doesn't relax that rule; it removes almost every option
the rule applies to, because in the target shape there is almost nothing on
disk for an option to point at.
**Public material is a value.** A certificate or a public nkey is handed to **Public material is a value.** A certificate or a public nkey is handed to
every client that connects, so the store is a fine place for it. Nothing below every client that connects, so the store is a fine place for it. Nothing
is about those. below is about those.
## Current state
Four credential families live in the swarm's secret store, plus the mTLS
identity that opens it. That fifth row isn't a stored secret — it's the
credential that can't be, being what authenticates the reader to the store.
| store path | kind | minter | writer | reader | renewal | persisted outside the store |
| -------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/<account>` | one agent's matrix account credential | `swarm-controller`, via a caller-supplied bearer token or its own `m.login.password` (`swarm-controller/src/matrix_account.rs:14-21`) | same unit, `put_matrix_account`'s `store.write` call (`swarm-controller/src/matrix_account.rs:189-197`) | `hive-c0re`'s credential worker, under the **hive's** identity (`hive-c0re/src/workers/credential.rs:33-64`) | `NONE` | **yes** — `hive-priv` writes `<agent-state>/matrix-token`, `0600`, chowned to the agent (`hive-priv/src/main.rs:535-556`) |
| `swarm/hives/<hive>/matrix/appservice-token` | the appservice token authenticating one hive's homeserver | `swarm-secret-publisher.nix` itself, `head -c 32 /dev/urandom` on the authelia host (`nix/host-modules/swarm-secret-publisher.nix:271-278`) | same unit (`nix/host-modules/swarm-secret-publisher.nix:288-289`) | `swarm-bao-matrix-token.service` on the hive (`nix/host-modules/glue-matrix-bao-token.nix:159-167`) | `NONE` | **yes, twice** — the publisher's own state copy (`nix/host-modules/swarm-secret-publisher.nix:185`) and the hive's token file, `0600` (`nix/host-modules/glue-matrix-bao-token.nix:180-181`) |
| `swarm/hives/<hive>/queue/agent` | the OIDC client id + secret agent containers present to the swarm queue | authelia, on first boot, into its own client-secret dir | `swarm-secret-publish.service` (`nix/host-modules/swarm-secret-publisher.nix:222-225`) | `swarm-bao-queue-agent.service` on the hive (`nix/host-modules/glue-queue-agent-credential.nix:187-197`) | `NONE` | **yes** — `secret` `0600` and `client_id` `0644` under the hive's agent-credential dir (`nix/host-modules/glue-queue-agent-credential.nix:218-224`) |
| `swarm/services/<clientId>/oidc/client` | a swarm service's own OIDC client secret, one per service | authelia, on first boot | `swarm-secret-publish.service` (`nix/host-modules/swarm-secret-publisher.nix:245-248`) | whichever host runs that service — `swarm-bao-grafana-oidc.service` (`nix/host-modules/swarm-grafana.nix:565-684`), the collector's twin (`nix/host-modules/swarm-otel.nix:685-767`) | `NONE` | **yes** — Grafana's `0400` host file (`nix/host-modules/swarm-grafana.nix:678-682`), the collector's `0400` file handed in by `LoadCredential` (`nix/host-modules/swarm-otel.nix:764-768`) |
| _(not stored)_ the store's own mTLS leaves | the certificate a host presents to reach the store at all | `swarm-bao-pki.service`, self-signed, 10-year (`nix/host-modules/glue-bao-tls.nix:114-149`) | n/a | every client reads its own leaf off disk via `BAO_CLIENT_CERT` (`swarm-secret-client/src/client.rs:12-14`) | `NONE` — signed only when absent (`nix/host-modules/glue-bao-tls.nix:128-149`) | **always, by construction** — it has no other form |
Four things in that table are uncomfortable, and they're the reason this page
exists rather than a note in the margin of the old one.
**Every stored secret is persisted as a plaintext file outside the store.** All
four, without exception. The appservice token is persisted twice: once in the
publisher's own state directory on the authelia host, and again in the token
file on the hive that reads it.
**Nothing rotates.** Every renewal cell reads `NONE`. No rotation path exists
for any of the four, and none for the mTLS leaves either — those are signed once
if absent and carry ten years of validity, and deleting the CA to reissue
invalidates every client certificate already trusting it.
**No agent container has a store identity at all.** Not one
`BAO_ADDR` or `BAO_CLIENT_CERT` exists anywhere under `nix/agent-modules/`. An agent
never talks to the store; the hive host reads on the agent's behalf and writes
a file into the container. That's the exact inverse of the target below.
**The appservice token has a second, uncoordinated minter.** `hive-matrix.nix`
keeps a local first-boot mint as a fallback for an unreachable store. Two
minters exist for one logical credential, on different hosts, with no agreement
between them — they can diverge, and the only thing that reconciles them is the
reader overwriting the local value on its next successful start.
One half of the target is already real: all four families are minted once and
read by an identity that logs in with a certificate, with no ambient trust
anywhere in the path. The cleanest examples are `swarm-bao-queue-agent.service`
and `swarm-bao-grafana-oidc.service`. What each of them then does is write the
fetched bytes to disk.
Two per-agent credential files — the forge token and the github token
(`hive-priv/src/main.rs`'s `write_forge_token`/`write_github_token`) — are deliberately absent from the table.
They're operator-supplied and never pass through the store at all, so they're
not credentials this page governs. Their absence isn't an oversight.
## Target state
⚠️ **This section describes the shape the swarm is meant to have — not what
exists today, and not a claim of movement toward it.** Read the current-state
table above for what exists. Nothing here is partly done.
**No secret the store holds is ever written to disk.** That's the invariant, **No secret the store holds is ever written to disk.** That's the invariant,
and everything else in this section follows from it. A value pulled from the and everything else in this page follows from it. A value pulled from the
store — bao — lives in the memory of the process that asked for it and nowhere store — bao — lives in the memory of the process that asked for it and
else: not in a state directory, not in a bind-mounted file, not in a systemd nowhere else: not in a state directory, not in a bind-mounted file, not in a
credential, not in a rendered config, not for a moment before a unit deletes systemd credential, not in a rendered config, not for a moment before a unit
it. The target isn't a shorter list of secret files. It's the store, plus deletes it. The target isn't a shorter list of secret files. It's the store,
exactly one file. plus exactly one file.
**That one file is the mTLS client certificate, and it's the only credential on **That one file is the mTLS client certificate, and it's the only credential
disk.** It has to be a file, and the reason is the whole asymmetry: the on disk.** It has to be a file, and the reason is the whole asymmetry: the
certificate is what authenticates a principal to the store, so it's the one certificate is what authenticates a principal to the store, so it's the one
credential that can't be fetched from the store. Something has to exist on disk credential that can't be fetched from the store. Something has to exist on
before the first request, or there's nothing to make the request with. Every disk before the first request, or there's nothing to make the request with.
host has one, and its only power is to ask the store for the rest. Today only Every host has one; its only power is to ask the store for the rest.
the host running the store mints its own leaves automatically — for itself, and `swarm-bao.nix:529-531` states the rule for the nix option that carries it:
pre-emptively for a controller and a publisher that might be co-located. For this is _"the credential an operator places by hand"_, and _"a path, never a
every other host, `swarm-bao.nix:529-531` says it outright: this is _"the value."_ A literal in a nix expression lands in the nix store —
credential an operator places by hand."_ world-readable and permanent — so that option takes a path to the
certificate on disk, never the certificate's bytes.
**The hive hands an agent an identity, never a secret.** An agent container is **The hive hands an agent an identity, never a secret.** An agent container
passed an mTLS certificate from its hive and from then on authenticates to the is passed an mTLS certificate from its hive and from then on authenticates
store under its own name, pulling what it needs when it needs it. No process to the store under its own name, pulling what it needs when it needs it. No
reads a secret on another principal's behalf: the principal that needs a value process reads a secret on another principal's behalf: the principal that
is the principal that authenticates for it. needs a value is the principal that authenticates for it.
Two per-agent credential files — the forge token and the github token — sit
outside this page: they're operator-supplied and never pass through the
store, so the table below doesn't govern them.
**Per secret, the target specifies minter, reader, and renewal strategy.** **Per secret, the target specifies minter, reader, and renewal strategy.**
Those three are the contract, and in this shape the **reader is a process Those three are the contract, and the reader is a process pulling a store
pulling a store path at runtime** — not a path on disk, and not a unit whose path at runtime — not a path on disk, and not a unit whose job is to turn a
job is to turn a store value into a file. store value into a file. A renewal cell may never read `NONE`: state the
strategy for every credential, including the mTLS leaf.
| store path | minter | reader — pulls at runtime, holds in memory | renewal | | store path | minter | reader — pulls at runtime, holds in memory | renewal |
| -------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | --- | --- | --- | --- |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | **must be stated** — no strategy is chosen for this credential, and `NONE` isn't one | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated |
| `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | **must be stated** — as above | | `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated |
| `swarm/hives/<hive>/queue/agent` | authelia | the agent container presenting the OIDC client to the swarm queue, under its own certificate | **must be stated** — as above | | `swarm/hives/<hive>/queue/agent` | authelia | the agent container presenting the OIDC client to the swarm queue, under its own certificate | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | **must be stated** — as above | | `swarm/services/<clientId>/oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated |
| _(not in the store)_ the mTLS client leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — **the sole exception above**, because it's what makes every other row's pull possible | **must be stated** — as above | | _(not in the store)_ the mTLS client leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the sole exception above, because it's what makes every other row's pull possible | must be stated |
A renewal cell reading `NONE` is allowed only in the current-state table above, ## Touching a credential
as an honest record of where the swarm is. It isn't a strategy, and a new
secret doesn't get to write it.
The appservice token's second, uncoordinated minter isn't a row here for a A pull request that touches a credential can't:
reason: in the target each credential has exactly one minter, so a fallback
mint on the reading host is a thing the target deletes, not a thing it
describes.
## Progressive enhancement - add a minter outside the swarm's existing mint path
- persist a store-provided secret to disk — a state directory, a bind mount,
The rule has two halves, and the second is the one that makes the first a rendered config
survivable. - add a credential whose renewal strategy is `NONE` — state the strategy,
even if it's "operator reissues and restarts the reader"
**New functionality matches the target shape immediately.** No legacy clause - read a secret on another principal's behalf and hand it over — the
covers code that doesn't exist yet. principal that needs the value authenticates for it
- give a host or container an out-of-band credential that isn't the store
**Existing functionality moves step by step** — each individual step moving mTLS leaf — one out-of-band credential per principal is the whole point of
toward the target shape, never sideways and never away. A step doesn't have to the store
finish the migration. It has to not undo it.
That's checkable, so check it. Apply these to a pull request that touches a
credential:
| question | a `yes` means |
| ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Does it add a minter outside the swarm's existing mint path? | **block** for new functionality. For existing functionality, only if it removes one too. |
| Does it persist a store-provided secret to disk — a state directory, a bind mount, a rendered config? | **block** for new functionality. For existing functionality, only if the change removes a persisted copy elsewhere and nets out fewer. |
| Does it add a credential whose renewal strategy is `NONE`? | **block** for new functionality. State the strategy, even if it's "operator reissues and restarts the reader." |
| Does it read a secret on another principal's behalf and hand it over? | **block** for new functionality. The principal that needs the value authenticates for it. |
| Does it give a host or container an out-of-band credential that's not the store mTLS leaf? | **block**, both. One out-of-band credential per principal is the whole point of the store. |
| Does it change the minter, reader, or renewal of an existing credential? | The current-state table above is **stale until updated in the same change.** |
For existing functionality the questions aren't pass/fail — they're a
direction check. A change that persists one fewer copy, or moves one reader from
a host-written file to its own store read, is a step and lands. A change that
leaves the count the same while moving the file somewhere tidier isn't a step;
it's churn with a migration story attached.
When the last `yes` in the current-state table's final column becomes a `no`,
this page replaces `secrets.md` and the current-state section goes with it.