hyperhive/docs/swarm/credentials.md

Credentials: the target shape

The swarm's credential store is bao. This page describes the shape every credential is meant to have — who mints it, who reads it, and how it renews — not what's on disk today. secrets.md remains the map of the files that exist right now; this page replaces it, and secrets.md gets deleted, once the swarm's credential path matches what's described below.

Public material is a value. The store hands a certificate or a public nkey to every client that connects, so it's a fine place for that material. Nothing below is about those.

No secret the store holds is ever written to disk. That's the invariant, and everything else in this page follows from it. A value pulled from the store — bao — lives in the memory of the process that asked for it and nowhere else: not in a state directory, not in a bind-mounted file, not in a systemd credential, not in a rendered config, not for a moment before a unit deletes it. The target isn't a shorter list of secret files. It's the store, plus one file per identity.

Those files are mTLS client certificates, one per identity, and they're the only credential on disk. Each has to be a file, and the reason is the whole asymmetry: the certificate is what authenticates a principal to the store, so it's the one credential nothing can fetch from the store. Something has to exist on disk before the first request, or there's nothing to make the request with. Every identity — an agent, a hive, a swarm-level service — needs one; a host running several holds several, and its only power is to ask the store for the rest. swarm-bao.nix:529-533 states the rule for the nix option that carries it: this is "the credential an operator places by hand", and "a path, never a value." A literal in a nix expression lands in the nix store — world-readable and permanent — so that option takes a path to the certificate on disk, never the certificate's bytes.

The hive hands an agent an identity, never a secret. Its hive passes an agent container an mTLS certificate, and from then on the agent authenticates to the store under its own name, pulling what it needs when it needs it. No process reads a secret on another principal's behalf: the principal that needs a value is the principal that authenticates for it.

Per secret, the target specifies minter, reader, and renewal strategy. Those three are the contract, and the reader is a process pulling a store path at runtime — not a path on disk, and not a unit whose job is to turn a store value into a file. A renewal cell may never read NONE: state the strategy for every credential, including the mTLS leaf.

Renewal is two columns. Automatic re-mint is something replacing the stored value without an operator. Automatic re-pull is the reader picking up a replaced value without a restart. ✅ or ❌ says which exist today, and the rest of the cell says how.

store path	minter	reader — pulls at runtime, holds in memory	automatic re-mint	automatic re-pull
swarm/agents/<agent>/matrix/main	swarm-controller, with the swarm's appservice token, at agent creation and in a five-minute pass	the agent container itself, under the certificate its hive passed in	✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's	✅ hive-matrix-daemon exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again
swarm/agents/<agent>/matrix/<account>	swarm-controller	the agent container itself, under the certificate its hive passed in	must be stated	must be stated
swarm/controller/swarm-controller/matrix/appservice-token	swarm-matrix-ctl, inside the hive-matrix container, once	swarm-controller, under its own certificate	❌ swarm-matrix-ctl mints it once; the container keeps its copy and republishes it when the store's differs	✅ the controller reads it on every five-minute matrix pass
swarm/controller/swarm-controller/oidc/client	authelia, at its first boot; swarm-secret-publish copies it in	swarm-controller, under its own certificate, at start and on refusal	❌ authelia mints it once. A re-mint is republished by swarm-secret-publish's path unit	✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value
swarm/controller/swarm-controller/forge/token	the forge, inside the hive-forge container, once; swarm-bao-forge-publisher on the forge's host copies it in under its own certificate, with a write-only grant	swarm-controller, under its own certificate, at start	❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file	❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart
swarm/agents/<agent>/bao-mtls	the store's agent PKI mount (deploy.bao.agentPkiMountPath), which generates the key, at swarm-controller's request at agent creation	hive-c0re, under the hive's own certificate, when it writes the agent's container config; then hive-agent-bao-identity in the agent container, under the leaf it holds	✅ swarm-controller's five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself)	✅ hive-agent-bao-identity re-reads it every 10 minutes and swaps a new leaf into /run/hive-agent-bao-identity/, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid
swarm/agents/<agent>/queue	swarm-controller, at agent creation	hive-agent in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; swarm-logs in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; swarm-nats-auth and swarm-logs-auth, each under its own certificate, to check a presented secret	✅ swarm-controller's five-minute pass re-mints a live agent's secret once it's 45 days old by minted_at on the stored object; a secret with no minted_at gets one stamped, value unchanged. The pass skips agents declared Destroyed — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one	✅ hive-agent reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. swarm-logs reads it on every query and once more after a 401. swarm-logs-auth keeps granting a secret it verified for deploy.victorialogs.authVerdictTtl (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long
swarm/agents/<agent>/forge-token	swarm-controller, at agent creation and in a pass every 5 minutes over every agent with a store identity	the agent container itself, under its own certificate, fetched to /run/hive-agent-forge-token/token	✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes)	✅ the agent re-fetches on a 10-minute timer
swarm/agents/<agent>/forge/<label>	swarm-controller, when an operator links an external forge account in the swarm UI	hive-agent-forge-accounts in the agent container, under the agent's own certificate, into <state>/forge-<label>-token and forge-<label>.json	❌ an operator's token; replaced only by deleting it and linking the label again	✅ the agent re-fetches on a 2-minute timer
swarm/agents/<agent>/github-token	swarm-controller, when an operator links a GitHub account in the swarm UI	hive-agent-github-token in the agent container, under the agent's own certificate, into <state>/github-token	❌ an operator's token; replaced only by deleting it and linking it again	✅ the agent re-fetches on a 2-minute timer, and hive-github-notify re-reads the file on every poll
swarm/hives/<hive>/matrix/sender-token	swarm-controller, with the swarm's appservice token, for every hive in its directory in a five-minute pass	swarm-controller under its own certificate, before it decides whether to mint, and hive-c0re's stored_sender_token(), under the hive's own certificate	✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's	✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs
swarm/hives/<hive>/forge/token	swarm-controller, with the forge's admin API, for every hive in its directory in a five-minute pass	swarm-controller under its own certificate, before it decides whether to mint, and hive-c0re's forge::reader_token(), under the hive's own certificate	✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes)	✅ hive-c0re reads the store on every forge read
swarm/hives/<hive>/queue/agent	authelia	swarm-bao-queue-agent on the hive's host, under its own per-hive certificate; no agent's policy reaches it	must be stated	must be stated
swarm/hives/<hive>/oidc/client	authelia, at its first boot; swarm-secret-publish copies it in	hive-c0re, under the hive's own certificate, when it first connects to the queue	❌ authelia mints it once. A re-mint is republished by swarm-secret-publish's path unit	✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it
swarm/services/<clientId>/oidc/client	authelia	that service's own reader, under its own certificate — the queue responder's under swarm-nats-auth's, the homeserver's under swarm-matrix-ctl's, the forge's under swarm-forge-publisher's, the review classifier's under swarm-review-classifier's; the bao UI's by the store's granter	must be stated	must be stated
swarm/services/ci-runner/registration-token	swarm-controller, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only	swarm-bao-ci-runner on the runner's host, under the runner's own certificate, into a file only swarm-ci-runner-register in the runner's container loads; no other principal's policy reads it	❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes	✅ swarm-bao-ci-runner reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed
swarm/services/review-classifier/anthropic-api-key	the operator, by hand in the store's UI, field api_key	swarm-review-classifier in its container, under its own certificate, at the start of every run; held for that run and handed to claude's environment only; no other principal's policy reads it	❌ an operator's key; replaced by writing a new one	✅ read on every run, so a written or removed key takes effect on the next run
swarm/services/review-classifier/forge/token	swarm-controller, with the forge's admin API, for its review-classifier account, at start and in a five-minute pass; the secret publisher's policy denies it	swarm-controller under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it	✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes)	✅ the controller reads the store on every review it posts
(not in the store) a hive's mTLS leaf	the store's own PKI, or an operator placing it by hand	its own client, off disk — the exception above, because it's what makes every other row's pull possible	must be stated	must be stated

Two rows share the matrix/ prefix and have different minters, on purpose. An agent's main account is on the swarm's own homeserver, and swarm-controller creates it with the swarm's appservice token. Every other account under that prefix is somewhere else entirely, and an operator hands the controller a credential for it. The operator-facing route refuses the name main for exactly this reason: two writers, one name, and that refusal is what keeps them apart.

The swarm's appservice token sits under controller/, the one kind no hive's policy reads. Its sender is the homeserver's admin. Under agents/, hives/ or services/ every hive could read it. matrix-ctl may write and read that one leaf; the controller may only read it.

An agent's mTLS leaf is in the store; a hive's isn't, and the difference isn't an inconsistency. The rule the exception protects is that nothing can fetch from the store the credential it would need in order to fetch. A hive's leaf is that credential, so it can only come off disk. The hive, which already holds one, reads an agent's — so publishing it costs nothing and buys the property this page asks for: the swarm mints it, the hive only carries it, and no hive ever needs the capability to mint an identity. swarm-controller proves the leaf it publishes before the creation job reports success, by logging in with it and reading the row back.

swarm-controller checks every forge token itself: it looks at every agent that has a store identity at start and every five minutes. For any whose stored token is missing or stale it creates the forge user if there isn't one, then mints the token. To check one agent now:

swarmctl agent mint-forge-token <agent>

An agent without a store identity gets no swarm token and keeps using the forge-token file in its state dir, if it has one; re-create it with swarmctl agent create to give it one. The swarm leaves an agent it already declares as it is. For an agent with no declaration, such as one only its hive knows about, agent create declares it paused; set it up from its card in the swarm UI.

⚠️ Re-create existing agents before deploying a hive-side change that makes a container require a credential it may not have. A container whose store identity is absent doesn't start (nix/agent-modules/bao.nix exits 1), so create the identities first, setting any newly paused agent up as above.

Revoking an agent's queue credential. Declaring an agent destroyed — PUT /api/hives/{hive}/agents/{agent}/state — deletes swarm/agents/<agent>/queue and every version it ever held, right after the declaration lands. The credential is a bearer secret, so a soft delete would leave the same value readable at an older version number; the controller removes the path's metadata, which takes the versions with it.

An agent recreated under the same name draws a fresh secret, because the mint keeps an existing value only when it finds one at that path (agent_identity::mint_and_verify, step 3) and the revocation left nothing to find.

Two things this deliberately doesn't do. It doesn't touch the agent's mTLS leaf, its ACL document or its cert-auth role — the mint rewrites all three on every run, so a re-created agent gets new ones regardless. And it doesn't fail the destroy: the declaration is already published by the time the revocation runs, so a store that refuses the delete gets a swarm-controller log line at error naming the agent, and the teardown continues. Declare the agent destroyed again to re-run the revocation.

Who reads that row, and what happens to it. hive-c0re reads it every time it writes an agent's container configuration (lifecycle::agent_identity), stages the certificate and its key 0600 outside every bind-mounted tree, and passes both to the container as systemd credentials — the same mechanism, and for the same mode reason, as the per-hive queue secret. A bind mount would hand the agent's unprivileged user a file it lacks the rights to open; the container manager reads a credential as root and re-exposes it under the consuming unit's own user.

Inside the container, hive-agent-bao-identity.service logs in with that certificate and reads this row back before reporting success, so an agent locked out of its own identity says so at boot rather than at whichever pull needed the store first. It keeps a live copy of the pair in /run/hive-agent-bao-identity/, which every unit in the container that logs in to the store presents, and re-runs every 10 minutes: a leaf the controller re-issued replaces that copy once a login with it succeeds. The unit exists whenever services.hyperhive.agent.bao.addr has a value, which the hive's meta flake sets from its own store address — the same all-or-nothing gate the per-hive queue credential beside it uses, and the reason the delivery above never lands in a container with nothing to read it. It fails loudly where the hive-side readers treat a missing value as a normal state, which is deliberate: a missing queue secret means a swarm whose publisher has yet to run, while a refused certificate means an agent that believes it reaches the store and never does.

Both kinds of unit retry a failed login every 30 seconds for a day (nix/host-modules/lib/store-retry.nix). A unit ordered after one waits for a single attempt, not for the retries: an attempt that fails ends that start, so the dependent starts with whatever is already on disk. When a later attempt lands a changed value, the homeserver, Grafana and collector readers restart the service that loaded it (nix/host-modules/lib/refresh-consumer.nix). When the fetch unit can't read the credential file's mtime or the service's start time, it logs cannot compare the mtime of <path> … — leaving <unit> as it is at err in its own journal, and the service keeps running on the old value until the next timer tick.

Progressive enhancement

New functionality has to match this shape immediately — no PR introducing a credential gets a pass on any of the rules below. A PR can move existing functionality step by step, as long as each individual step moves toward the target shape; a step that doesn't isn't allowed just because it's existing.

A pull request that touches a credential can't:

• add a minter outside the swarm's existing mint path
• persist a store-provided secret to disk — a state directory, a bind mount, a rendered config
• add a credential whose renewal strategy is NONE — state the strategy, even if it's "operator reissues and restarts the reader"
• read a secret on another principal's behalf and hand it over — the principal that needs the value authenticates for it
• give a host or container an out-of-band credential that isn't the store mTLS leaf — one out-of-band credential per principal is the whole point of the store

While the swarm's credential path is still moving to this shape, a PR that moves a secret into bao may leave its renewal strategy unresolved, provided it opens a follow-up issue to settle renewal. That's a migration-era allowance, not a standing exception to the renewal-strategy rule above.