hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.
Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.
The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.
Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.
Refs #5015
The hive policy rendered by policy::render granted every hive cert read
on secret/data/swarm/services/*, which holds each swarm service's OIDC
client secret. No reader of that tree authenticates with a hive cert:
grafana, the otel collector and the store forwarder each read with
their own leaf, and the bao UI's secret is read by the store's granter.
render() now emits two stanzas: swarm/agents/* and the hive's own
swarm/hives/<hive>/*. Kind::Service had no other caller and is removed.
Docs and comments that described the grant as a decision are rewritten
to the current state.
Closes#4960
A stored queue secret with no `minted_at` counted as due, and no secret
minted before the renewal pass has one, so the first pass after deploy
would re-mint every agent's secret. Every reconnect before that agent's
next restart would then be refused.
Such a secret is now stamped instead: `minted_at = now` is written beside
the unchanged `value`, and its 45-day clock starts there. Only a secret
whose recorded mint time is at least 45 days old gets a new value.
The decision is `secret_step` (Keep / Backfill / Remint), and the pass
reports an unstamped secret as `Observed::Unstamped`. Backfill and
re-mint log different lines.
A five-minute pass over every agent some hive's wanted state declares as
anything but destroyed queues, per agent:
- `MintAgentIdentity` (the node agent creation uses) when the stored
certificate at swarm/agents/<agent>/bao-mtls is past half its validity,
read from its own notBefore/notAfter: day 45 of the role's 90;
- the new `RenewAgentQueueCredential` node when the queue secret at
swarm/agents/<agent>/queue is 45 days old or has no mint time. The node
re-decides, writes a fresh value with `minted_at`, reads it back, and logs
the agent and the old age.
When both are due the secret node runs after_any the certificate node,
because mint_and_verify compares the queue secret it read with the one it
reads back. A credential that is not stored is never created here.
`queue::AgentCredential` gains an optional `minted_at` (unix seconds);
agent creation now sets it. Stored objects without it decode unchanged and
count as due, so every existing queue secret is re-minted on the first pass.
Both replacements reach the agent at its next start. The old certificate
stays valid until it expires; the old queue secret does not, so a queue
reconnect before that restart is denied.
Adds x509-cert 0.2 (with der_derive and flagset) to read the validity.
docs/swarm/credentials.md: the renewal column splits into automatic re-mint
and automatic re-pull, filled from the code as it stands.
The controller's OIDC client secret (client `swarm-controller`, used for
the queue connection, the auth-bridge bearer and the OTLP push) came from
an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`,
handed in by `LoadCredential=`.
Now `swarm-secret-publish`, which already copies authelia's minted OIDC
secrets into the store, also publishes this one, to
`swarm/controller/swarm-controller/oidc/client`. That path sits under
`controller/`, which no hive's policy reads. The controller reads it once
at start with its existing store certificate and holds it in memory, as
`swarm_queue_client::ClientSecret::Value`. If the store is down, it
retries for about a minute and then fails the start, so `Restart=` tries
again.
Policy delta: the controller gets `read` on that leaf, and the publisher
gets `create`/`update` on that leaf.
Removed: the `queue.clientSecretFile` option (both spellings, now removed
options with a message), its singleHostSwarm default, the credential and
placeholder, and the path watcher plus its restart oneshot. A controller
without a store identity is now an eval error, because it has no other
way to get the secret.
`swarm_queue_client::agent_token::format_agent_token` / `parse_agent_token`
are the spelling an agent presents its own queue secret in,
`swarm-agent.<agent>.<secret>`, and the one the auth-callout responder
reads back. The prefix is what separates it from an OIDC access token,
which may itself contain `.`. Parsing distinguishes "not an agent token"
(no prefix) from "a malformed one"; the error names the problem and never
the value. The module is store-free, so the agent formats its token
without linking the secret-store client.
`swarm_secret_client::queue::AgentCredential` loses `hive`: an agent's
identity is not tied to a hive, and nothing reads the field. Objects
already in the store carry it and still decode, since unknown fields are
ignored; a test parses one. The controller stops writing it.
With the credential no longer naming a hive, and the agent's policy
naming none since #4762, nothing in the mint consumes one. `hive` goes
from `mint_and_verify`, from the `MintAgentIdentity` node, and from
`POST /api/agents/{name}/identity`, which now takes no body and no longer
checks a hive against the roster; a caller that still sends one is not
refused, the body is ignored. `swarmctl agent mint-identity` loses
`--hive`, so passing it is now a usage error.
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped secret today, so at the auth callout one agent is
indistinguishable from its co-hived neighbours and no subject can be
scoped to one of them.
Mint a secret per agent instead, at swarm level, into
secret/swarm/agents/<agent>/queue -- inside the stanza every agent's ACL
document already grants, so no policy changes and no existing agent's
document is rewritten. It is written by the same node that already mints
the agent's certificate, and read back under the agent's own token
before that node reports success.
The secret is not derived from the agent's mTLS identity: the two
credentials answer different questions and coupling their lifetimes
would mean renewing either implied renewing the other. Nothing here
rotates a queue secret -- a re-run keeps the existing value and only
corrects the principal it names, because this function is re-run
deliberately against agents that are already connected. Revoking one
means deleting the path.
Nothing reads the new credential yet; this is the minting half.
Pinning a decision rather than changing behaviour, because I was one
edit away from reversing it and left no reason on the field.
The sibling `matrix::Credential::homeserver` is an `Option`, which reads
like the house style to copy. It is not: that field is optional because
it was added to objects already in the store, and KV2 keeps those
versions forever. This path has never been written, so there is nothing
to stay compatible with.
Making it optional would also defeat the field. The id rides with the
secret so a reader never has to spell `hive-<name>-agent` itself, and
the only thing a reader holding `None` can do is exactly that. An object
without an id is not a usable credential, so failing to decode is the
behaviour we want.
Refs #3853
The agreement half of delivering the agent queue principal's client secret
through the store. No producer yet, so nothing writes this path — the unit
that does lands in the same PR, with the write grant it needs.
queue.rs is the sibling matrix.rs prescribes for a second kind of secret
rather than another field on a shared struct. Keyed per HIVE, not per agent:
the queue identity is minted once per hive at deploy time and says which hive
an agent belongs to, never which agent.
The client id rides with the secret for matrix.rs's stated reason — a
credential has to be reconstructable from the store alone, and deriving
`hive-<name>-agent` on the reading side is the split spelling the authelia
module warns denies every agent as a timeout.
policy.rs's render() takes the hive name now and emits a second, narrow
stanza for that hive's own path. The agent stanza is untouched: an agent's
path does not name its hive, so narrowing it still needs the enumeration
docs/trust-boundary/security.md rejects. A hive path does name its principal,
so scoping it costs nothing and drifts nowhere.
every_hive_gets_a_byte_identical_document is replaced rather than deleted.
Its surviving half is that the text is a function of the deploy-time name
alone, so a re-emission cannot drift; the new arms are that one hive's
document cannot reach another's path, and that a name which could close the
stanza is refused — live again now that a name reaches the document text.
Refs #3853
