Agent creation at swarm level is event-driven and nothing sweeps for
agents missing a credential, so an agent created before a credential
joined the mint never receives one -- nothing comes back around to it.
Without a way to re-run the mint by hand, the only route to giving an
existing agent its queue credential would be to delete and recreate the
agent.
POST /api/agents/{name}/identity enqueues the same MintAgentIdentity
node POST /api/agents declares, rather than writing inline: a second
code path that mints an identity is a second place for the four strings
that have to agree to disagree. swarmctl agent mint-identity is the
operator end, the same POST-and-print-the-node-id shape agent create
already has.
--hive is required on both ends. Neither the CLI nor the controller
keeps a roster of which agent runs where, and the credentials this mints
name a hive, so a default would be a guess that hands an agent subjects
on a hive it does not run on.
Documents the backfill as a runbook step, and fills in the renewal cell
the credential matrix requires for the new row.
13 KiB
Credentials: the target shape
The swarm's credential store is bao. This page describes the shape every
credential is meant to have — who mints it, who reads it, and how it renews
— not what's on disk today. secrets.md remains the map of
the files that exist right now; this page replaces it, and secrets.md gets
deleted, once the swarm's credential path matches what's described below.
Public material is a value. The store hands a certificate or a public nkey to every client that connects, so it's a fine place for that material. Nothing below is about those.
No secret the store holds is ever written to disk. That's the invariant, and everything else in this page follows from it. A value pulled from the store — bao — lives in the memory of the process that asked for it and nowhere else: not in a state directory, not in a bind-mounted file, not in a systemd credential, not in a rendered config, not for a moment before a unit deletes it. The target isn't a shorter list of secret files. It's the store, plus one file per identity.
Those files are mTLS client certificates, one per identity, and they're
the only credential on disk. Each has to be a file, and the reason is the
whole asymmetry: the certificate is what authenticates a principal to the
store, so it's the one credential nothing can fetch from the store.
Something has to exist on disk before the first request, or there's nothing
to make the request with. Every identity — an agent, a hive, a swarm-level
service — needs one; a host running several holds several, and its only
power is to ask the store for the rest.
swarm-bao.nix:529-533 states the rule for the nix option that carries it:
this is "the credential an operator places by hand", and "a path, never a
value." A literal in a nix expression lands in the nix store —
world-readable and permanent — so that option takes a path to the
certificate on disk, never the certificate's bytes.
The hive hands an agent an identity, never a secret. Its hive passes an agent container an mTLS certificate, and from then on the agent authenticates to the store under its own name, pulling what it needs when it needs it. No process reads a secret on another principal's behalf: the principal that needs a value is the principal that authenticates for it.
Two per-agent credential files — the forge token and the github token — sit outside this page: they're operator-supplied and never pass through the store, so the table below doesn't govern them.
Per secret, the target specifies minter, reader, and renewal strategy.
Those three are the contract, and the reader is a process pulling a store
path at runtime — not a path on disk, and not a unit whose job is to turn a
store value into a file. A renewal cell may never read NONE: state the
strategy for every credential, including the mTLS leaf.
| store path | minter | reader — pulls at runtime, holds in memory | renewal |
|---|---|---|---|
swarm/agents/<agent>/matrix/<account> |
swarm-controller |
the agent container itself, under the certificate its hive passed in | must be stated |
swarm/agents/<agent>/bao-mtls |
swarm-controller, at agent creation |
hive-c0re, under the hive's own certificate, when it writes the agent's container config |
must be stated |
swarm/agents/<agent>/queue |
swarm-controller, at agent creation |
the agent container itself, under its own certificate — the identity it presents to the swarm queue, naming that one agent rather than its hive | none: the secret is fixed for the life of the agent and is revoked by deleting the path. A rotation mechanism is tracked as separate work, because rotating this credential needs a reconnect path — a queue client holding a revoked secret does not find out until it reconnects |
swarm/hives/<hive>/matrix/appservice-token |
one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated |
swarm/hives/<hive>/matrix/sender-token |
swarm-matrix-ctl, in the hive-matrix container |
swarm-matrix-ctl itself, under its own certificate, before it decides whether to mint, and hive-c0re's stored_sender_token(), under the hive's own certificate |
must be stated |
swarm/hives/<hive>/queue/agent |
authelia | the agent container presenting the OIDC client to the swarm queue, under its own certificate | must be stated |
swarm/services/<clientId>/oidc/client |
authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated |
| (not in the store) a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated |
An agent's mTLS leaf is in the store; a hive's isn't, and the difference
isn't an inconsistency. The rule the exception protects is that nothing
can fetch from the store the credential it would need in order to fetch. A
hive's leaf is that credential, so it can only come off disk. The hive,
which already holds one, reads an agent's — so publishing it costs nothing
and buys the property this page asks for: the swarm mints it, the hive only
carries it, and no hive ever needs the capability to mint an identity.
swarm-controller proves the leaf it publishes before the creation job
reports success, by logging in with it and reading the row back.
Backfilling an agent that predates a credential. Agent creation at swarm
level is purely event-driven — swarm-controller mints an agent's store
identity on the job graph POST /api/agents inserts, and nothing sweeps for
agents that already exist. So an agent created before a credential joined that
mint never receives one, and nothing will ever come back around to it. Re-run
the mint for one agent with:
swarmctl agent mint-identity <agent> --hive <hive>
--hive is required: neither the CLI nor the controller keeps a roster of
which agent runs where, and the credentials this mints name a hive. The queue
secret half is idempotent — an agent that already has one keeps exactly the
value it has, so running this against an already-migrated agent does not drop
its queue connection. The certificate half is not: the agent gets a fresh leaf
and picks it up on its next boot.
⚠️ Run this for every existing agent before deploying a hive-side change that makes a container require a credential it may not have. A container whose credential is absent does not start — that is deliberate, and it is what makes the backfill a step rather than a suggestion.
Who reads that row, and what happens to it. hive-c0re reads it every
time it writes an agent's container configuration
(lifecycle::agent_identity), stages the certificate and its key 0600
outside every bind-mounted tree, and passes both to the container as systemd
credentials — the same mechanism, and for the same mode reason, as the
per-hive queue secret. A bind mount would hand the agent's unprivileged user
a file it lacks the rights to open; the container manager reads a credential
as root and re-exposes it under the consuming unit's own user.
Inside the container, hive-agent-bao-identity.service logs in with that
certificate and reads this row back before reporting success, so an agent
locked out of its own identity says so at boot rather than at whichever pull
needed the store first. The unit exists whenever
services.hyperhive.agent.bao.addr has a value, which the hive's meta flake
sets from its own store address — the same all-or-nothing gate the per-hive
queue credential beside it uses, and the reason the delivery above never lands
in a container with nothing to read it. It fails loudly where the hive-side
readers degrade quietly, which is deliberate: a missing queue secret means a
swarm whose publisher has yet to run, while a refused certificate means an
agent that believes it reaches the store and never does.
Progressive enhancement
New functionality has to match this shape immediately — no PR introducing a credential gets a pass on any of the rules below. A PR can move existing functionality step by step, as long as each individual step moves toward the target shape; a step that doesn't isn't allowed just because it's existing.
A pull request that touches a credential can't:
- add a minter outside the swarm's existing mint path
- persist a store-provided secret to disk — a state directory, a bind mount, a rendered config
- add a credential whose renewal strategy is
NONE— state the strategy, even if it's "operator reissues and restarts the reader" - read a secret on another principal's behalf and hand it over — the principal that needs the value authenticates for it
- give a host or container an out-of-band credential that isn't the store mTLS leaf — one out-of-band credential per principal is the whole point of the store
While the swarm's credential path is still moving to this shape, a PR that moves a secret into bao may leave its renewal strategy unresolved, provided it opens a follow-up issue to settle renewal. That's a migration-era allowance, not a standing exception to the renewal-strategy rule above.