hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.
`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.
swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.
The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.
agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.
Part of #5017 (slice Q).
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):
- `forge::enable` tries the store once at start. On failure it logs once and
re-reads in the background with backoff (30s doubling to a 5-minute cap,
no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
poll and the webhook registration start from that moment, through
`start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
"no forge" before and work after.
A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.
Refs #4998
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.
Now the forge's host publishes it and the controller reads it:
- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
waits for the in-container token, logs in with its own forge-publisher leaf
and writes it to swarm/controller/swarm-controller/forge/token. It runs on
every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
policy create/update on that one path, written by
swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
at start under its own certificate with the OIDC secret's retry, and runs
without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
removed, and a tmpfiles rule deletes the old host copy on the forge host.
The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.
Refs #4998
The controller's OIDC client secret (client `swarm-controller`, used for
the queue connection, the auth-bridge bearer and the OTLP push) came from
an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`,
handed in by `LoadCredential=`.
Now `swarm-secret-publish`, which already copies authelia's minted OIDC
secrets into the store, also publishes this one, to
`swarm/controller/swarm-controller/oidc/client`. That path sits under
`controller/`, which no hive's policy reads. The controller reads it once
at start with its existing store certificate and holds it in memory, as
`swarm_queue_client::ClientSecret::Value`. If the store is down, it
retries for about a minute and then fails the start, so `Restart=` tries
again.
Policy delta: the controller gets `read` on that leaf, and the publisher
gets `create`/`update` on that leaf.
Removed: the `queue.clientSecretFile` option (both spellings, now removed
options with a message), its singleHostSwarm default, the credential and
placeholder, and the path watcher plus its restart oneshot. A controller
without a store identity is now an eval error, because it has no other
way to get the secret.
