hyperhive/docs/4-implementation/swarm/credentials.md

Credentials: mechanics

The store's identities, the queue responder's keys, how a reader reaches the store, the store UI's login and where the root lives. The overview and the rules a credential follows are in credentials; each credential's minter, reader and renewal are in the reference.

The queue's callout nkeys

The responder holds two nkeys besides the callout xkey below. It signs every user JWT it issues with the issuer key, an account key, so possession of the issuer seed is the authority to admit anyone to the queue. It connects to the queue as the user key, the one identity the server exempts from callout. You configure nothing about either: both live in the swarm secret store beside the xkey, and come into being on the queue's first boot.

• Issuer seed at secret/swarm/services/swarm-nats/callout/issuer-seed, public half at secret/swarm/public/swarm-nats/callout-issuer.
• User seed at secret/swarm/services/swarm-nats/callout/user-seed, public half at secret/swarm/public/swarm-nats/callout-user.

The unit that creates the xkey creates these too, under the same rules: only the swarm-nats-auth role reads or creates them, nothing replaces a stored seed, and the policy of swarm-secret-publisher denies it both seed paths. The server renders both public halves into its auth_callout block as it starts, and doesn't start without them.

A seed file at /var/lib/swarm-nats-callout/issuer.seed or /var/lib/swarm-nats-callout/callout-user.seed on the queue's host is what swarm-nats-auth ensure-callout-keys stores for that key when the store holds none, so the key keeps its public half. Once a read of the store returns a seed, the verb removes the file. When the store already holds a different seed, the verb keeps the store's seed, logs a warning naming the file, then removes it.

A hive that sets autoGenerateCallout, calloutUserPublicKey, calloutIssuerPublicKey, calloutUserSeedFile or calloutIssuerSeedFile, under deploy.nats or swarm.nats, fails at eval with a message naming the option and where the keys live now.

To replace either key, delete both of its paths the way the xkey's section below shows for the xkey, then restart swarm-bao-nats-callout-xkey.

One consequence worth knowing before you debug it: the queue assembles its config each time the server starts rather than at build time, in every mode, because the callout keys exist only at runtime. A malformed config surfaces when the server starts instead of when the system builds. The server names the offending file and refuses to run.

The queue's callout xkey

The server seals every auth-callout request to a curve key, the callout xkey, and the responder seals its reply back, so nothing else in the callout account or on the wire can read the request or the user JWT minted for it. You configure nothing about the key: it lives in the swarm secret store and comes into being on the queue's first boot.

• Seed at secret/swarm/services/swarm-nats/callout/xkey-seed.
• Public half at secret/swarm/public/swarm-nats/callout-xkey.

Only the swarm-nats-auth role reads either path, and it may create each but never replace it. The policy of swarm-secret-publisher, which writes the rest of swarm/services/, denies it the seed's path.

On the queue's host, swarm-bao-nats-callout-xkey runs before the swarm-nats container, under the responder's store leaf:

1. swarm-nats-auth ensure-callout-keys creates the pair when the store holds no seed, and otherwise only reads it. It needs no queue, so on a first boot the key exists before the server that needs it.
2. It reads the public half back from the store into /var/lib/nats-callout/callout-xkey.pub inside the container.
3. Each time nats-server starts, it renders that key into auth_callout.xkey. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store. A responder that can't read it doesn't start, rather than answering requests it can't open.

The store has to be reachable each time the responder starts. When it's down, the fetch unit fails and retries for a day. The server starts with the key fetched last time, or not at all on a first boot. A responder that starts while the store is down exits and retries every 5 seconds, and until one start reads the seed the queue refuses every client, OIDC ones included. The retry that lands the file starts or restarts the server, and the responder comes up with it.

The responder's store leaf is deploy.nats.authBaoClientCertFile / authBaoClientKeyFile. On the store's host both default to the leaf glue-bao-tls.nix mints there. A queue host without the store gets no default, even when it holds the hive's own reader deploy.bao.clientCertFile: issue a leaf whose subject is deploy.bao.natsAuthCommonName, place it, and name it in both options. Until you do, evaluation fails naming both.

Where the queue and the store run on different hosts, deploy the store's host first. swarm-bao-nats-auth-policy there writes the role's grants on the keys' paths, and swarm-bao-secret-publisher-policy the publisher's denial of the seeds' paths; where the queue runs on that host too, the fetch waits for both. A queue host deployed before it can't read or create the keys, so its fetch fails, and the queue refuses every client until you deploy the store's host: the server starts only on public keys fetched before, and the responder can't load its seeds. Seed files the import is waiting for stay where they are. If that takes longer than the fetch's day of retries, run systemctl restart swarm-bao-nats-callout-xkey on the queue's host.

Two starts that create the key at once can't both win: each write is a create that fails once the path holds anything, and the start that lost reads the winner's key back.

To replace the key, remove both paths with their metadata and every version, then restart the fetch unit on the queue's host. Its restart of nats.service restarts the responder too, which then loads the new seed:

bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey

bao kv delete isn't enough. It deletes the latest version but keeps the path's metadata, so the store still counts the path as existing and refuses the create that makes a new key. No principal hyperhive configures may delete either path's metadata, so run the two deletes with a token that may, such as the root token.

Per-principal identities

Bao matches a cert-auth role on the certificate's subject, so a certificate is an identity and sharing one merges the identities. The leaves, their option names and what each reaches are in the reference.

Each reaches a path that names a swarm service rather than a hive, so one role each is enough and their subjects are the flat deploy.bao.grafanaOidcCommonName, deploy.bao.otelOidcCommonName, deploy.bao.forwarderOidcCommonName, deploy.bao.ciRunnerCommonName, deploy.bao.reviewClassifierCommonName, deploy.bao.builderCommonName, deploy.bao.ciMetricsCommonName, deploy.bao.atticCommonName, deploy.bao.archiveCommonName, deploy.bao.claudeLoginCommonName and deploy.bao.forgePublisherCommonName. The last three are the writers, and the last is the one leaf two units share: the forge's host holds one store identity for the token it publishes and the OIDC client secret it reads.

On a host that mints its own PKI, glue-bao-tls.nix signs all ten and defaults all twenty options, and there is nothing to do. Elsewhere you copy each leaf from the store's host and name it here: a service principal's from the top of /var/lib/swarm-bao-pki/, another hive's leaves from /var/lib/swarm-bao-pki/hives/<hive>/ (bao.md § TLS lists that directory). That's one file per principal rather than one file shared by ten, which is the whole of what this buys. swarm-ci-runner.nix refuses the build wherever the runner runs without its pair, as swarm-review-classifier.nix does for the classifier, swarm-ci-metrics.nix for the CI run poller and swarm-grafana.nix for Grafana. The forge's pair refuses nothing: without it forgejo-swarm-controller-account and swarm-bao-forge-oidc render, fail at start naming both options, and the controller runs without forge access and the forge without a login source until an operator places the leaf.

On a host without the store, where the forwarder never runs, forgetting one of the others isn't a quiet degrade. A hand-configured remote-store hive that names the hive's own clientCertFile and misses a principal's pair has the build refused, by the module of the reader it missed, naming that pair — see what a missing leaf costs below.

The roles of swarm-nats-auth, swarm-logs-auth and swarm-metrics-auth issue batch tokens with a five-minute TTL (token_type=batch, token_ttl=5m). The responders log in on every check, and the store keeps no record of a batch token, so unauthenticated requests don't grow its token storage. Nothing can renew or revoke a batch token.

The other roles issue service tokens that live an hour (token_ttl=1h and token_max_ttl=1h). That covers the roles swarm-bao.nix writes, and the agent and hive roles the controller writes through SecretStore::write_cert_role, which takes the same value from CERT_ROLE_TOKEN_TTL. The store keeps a lease for every token until it expires, and the mount default of 32 days leaves one lease per login for a month; at the 17 thousand logins a day this swarm made, that came to half a million leases. A reader logs in per operation and holds the token for seconds. The longest hold is the controller's forge work, which stops after two minutes. A client that keeps a SecretStore handle longer than an hour must log in again, because nothing renews the token. A role the controller writes takes the TTL at its next write: a hive's at the controller's start, an agent's at its next certificate mint or renewal. At its start the controller also reads every existing hive-agent-* role and, for one whose token_ttl or token_max_ttl differs from the TTL, writes those two fields and no others. The store keeps the fields a write leaves out, so the role's certificate, allowed names and policies stay. A role already at the TTL isn't written. The start logs one line with rewritten, already_current and failed counts, and a role that fails gets a warning that names it, then the pass moves on. Tokens issued before expire on their own; the controller revokes nothing. At that rate an hour keeps about 700 tokens live.

The granter's role keeps 15 minutes. Only the one-time bootstrap token writes it, and a deployed store never rewrites it, so the longer TTL would not reach it; the granter logs in and finishes within seconds.

How a reader reaches the store

Every reader dials the same URL — https://bao.<swarm domain>:<port> — and on the host that runs the store that name resolves twice. /etc/hosts answers a host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a container with the bridge IP, where nginx holds the same port. One BAO_ADDR, and the network namespace that asked decides which half of it answers.

The container's half goes through nginx and still doesn't terminate TLS. It is an nginx stream server with ssl_preread on: it reads the SNI off the ClientHello and splices the rest of the connection through byte for byte, so openbao completes the handshake itself. A vhost would decrypt here instead, and the client certificate — the store's whole authentication — would stop at nginx, leaving openbao seeing one client for every hive in the swarm. That's why the store has no vhost and gets a passthrough: not an exception to the no-vhost rule, the one shape that keeps it.

The stream listener binds the bridge IP rather than every address, because openbao already holds loopback on that port in the same network namespace and a wildcard bind would fail with EADDRINUSE — taking nginx, and with it every hive domain behind the gateway, down. network.exposeHostPorts opens the port on the bridge firewall and nowhere else.

Reaching the port grants nothing by itself: openbao answers nothing without a client certificate signed by deploy.bao.clientCaFile or by the store's own agent CA. The listener reads both from listener-client-ca.pem, which no cert-auth role pins. The passthrough carries whichever certificate the reader presents, unchanged.

OIDC login to the store UI

The OIDC tab logs in through authelia as the client swarm-bao-ui (swarm.bao.ui.oidc.clientId). An admins member gets a token under swarm-operator-viewer: list and read on secret/metadata/*, and create and update on secret/data/swarm/services/review-classifier/anthropic-api-key, secret/data/swarm/services/review-classifier/acp-provider and the cache's five leaves under secret/data/swarm/services/attic/ (push-token, pull-token, url, cache and public-key). That shows the key tree and each key's versions and timestamps, and no value: the store refuses every secret/data/ read. The writes are the leaves the operator fills in by hand: the review classifier's keys, and the cache's tokens and facts, which configure an external cache or replace the in-swarm cache's own. The cache's host units create those leaves only while the store holds none, so a saved value survives their next run and every deploy. The cache's signing key, attic/jwt-secret, isn't writable here. Anything more needs a token.

To replace one of the cache's leaves in the UI, open secret > swarm > services > attic, pick the leaf and choose Create new version. The form starts empty because the login can't read the current value: type the key value and the new value, and save. A new version replaces the whole secret, so value is the only key to enter. bao kv put with the same login works too.

piece	written by
authelia client, with redirect swarm.bao.ui.oidc.redirectUri	glue-bao-ui-oidc-client.nix, wherever authelia runs
its secret at swarm/services/swarm-bao-ui/oidc/client	authelia's mint, like every other service's
the oidc auth mount	swarm-bao-granter-role, with the bootstrap token
swarm-operator-viewer policy, the mount's config and role swarm-operator-viewer	swarm-bao-operator-viewer-policy, as the granter, which reads the secret above into the config

The granter holds no sys/auth, so enabling the mount stays a bootstrap-token step. A store set up before the mount existed needs the one-time granter step again: it re-writes bao-bootstrap, which covers sys/auth/oidc, and the granter's own policy, which covers auth/oidc/. Until then swarm-bao-operator-viewer-policy writes the viewer policy, logs the step, and exits 0, and the UI offers token login only. swarm-bao-granter-role re-runs it each time the step succeeds, so OIDC login needs no further restart.

The constraint that decides where the root lives

The store issues a hive's gateway leaf through a role whose allowed_domains is that hive's domain and *. that domain, and a swarm service name is a sibling of the hive domain rather than a child — forge.<swarm> next to <hive>.<swarm>. That role can't issue a certificate for a swarm service; the store refuses it.

The store's swarm-services role issues them instead, from the root inside its pki mount. That root's private key never leaves the mount, and its certificate is the only anchor the swarm has. Each hive fetches the certificate from the store at boot, so nothing build-time can name it — the system reads security.pki.certificateFiles' value at build time, which makes it the wrong tool here. Host containers get a bind-mounted bundle assembled at boot; an agent fetches the root itself with its own store certificate.