When nats-server has an auth_callout xkey it seals each request to that
key and names its own public xkey in the Nats-Server-Xkey header. The
responder now loads the callout xkey seed from the secret store before
it connects, opens a sealed request with it, and seals its reply (grant
or deny) back to the header's key. A request with no header is answered
in plaintext, as before.
A new `ensure-xkey` verb creates the pair when the store holds no seed:
the seed at swarm/services/swarm-nats/callout/xkey-seed, the public half
at swarm/public/swarm-nats/callout-xkey. Each write is a create-only
(cas=0) write, and a run whose create fails reads the stored value back,
so two concurrent runs end up with one key. The responder itself only
reads the seed and refuses to start without it.
swarm-secret-client gains SecretStore::create, the cas=0 write.
hive-matrix-daemon now learns which external matrix accounts it has from
the swarm secret store, under the agent's own certificate, and the hive
push chain for matrix is gone.
The daemon lists swarm/agents/<agent>/matrix/ (the `list` its policy
grants on its own metadata subtree), reads each account's homeserver
from its credential, and brings the accounts up with their tokens from
the store. Every two minutes it lists again and exits with 75 when the
set of linked accounts changed; the unit restarts on 75 without counting
a failure. A listed name whose credential reads as absent is skipped and
logged once. At start it removes the matrix-token-<a> /
matrix-account-<a>.json pairs a hive delivered (a sidecar marks a pair
as delivered; a declared tokenFile keeps its token).
Removed: CredentialNotice and the $SWARM.credential.* subject and NATS
grant, the controller's publish and its queue precondition on the PUT
route, hive-c0re's credential subscription arm and workers/credential.rs,
priv_client::write_agent_matrix_token, hive-priv's WriteAgentMatrixToken
and its helpers, and the daemon's state-dir account discovery.
Kept: WriteAgentGithubToken and the external-forge path
(WriteAgentExtraForgeAccount, extra_forges.rs) are untouched, and a
declared matrixAccounts tokenFile is still read when the store has no
token for that account.
Refs #4348
Pins the three properties the revocation rests on and cannot check
against a store: that only `Destroyed` revokes (a revocation on
`Offline` or `Paused` would give an agent that stops and never
restarts), that a 404 is absence while a 403 stays a failure, and that
the delete addresses `secret/metadata/` -- the path that takes every
version, which is the string the grant has to match.
docs/swarm/credentials.md gains the revocation section and its table
cell stops describing the deletion as something an operator does by
hand.
A per-agent queue credential is minted at agent creation and nothing has
ever removed it. An agent declared destroyed loses its container and
keeps its credential: a bearer secret recovered from a snapshot or a
stale capture still authenticates as that agent, so the set of usable
credentials only grows.
Delete the path the mint published, on the one transition that ends an
agent's life. It mirrors step 3 of `mint_and_verify` and no other step:
the leaf, the ACL document and the cert role are what a hive uses to
collect an agent's secrets and are re-minted on every run of the mint.
Every version, not the newest. The mint rewrites the path when the
principal it names needs correcting, so KV v2's plain delete would leave
the identical secret readable at ?version=N. That is a separately-ACL'd
path, hence the second stanza in the controller's grant -- `delete` on
metadata discloses nothing, and `update` on the data path already lets
this principal destroy any agent credential's usability.
The destroy is not blocked by a failed revocation: the declaration is
already published and refusing the call would leave an operator with an
agent they cannot tear down. The failure is logged at error instead,
naming the agent, since a silent orphan is the fault being removed.
An agent's store identity was signed in swarm-controller's memory by a CA
a controller-host unit generated on disk, and the listener never trusted
that CA. Agent leaves now come from the store itself: a `pki-agents` PKI
mount whose root openbao generates internally, so the agent CA's key
never exists outside the store.
- swarm-bao-agent-pki (new, store host, as the bao granter): enables and
tunes the mount, generates the root once (guarded on an empty issuer
list, no replace branch), upserts the `swarm-agent` role (client
certificates named `hive-agent-*` only, 90 days), caches the CA at
/var/lib/swarm-bao-tls/agent-ca.pem and composes the listener bundle.
- The listener's tls_client_ca_file is a new listener-client-ca.pem
(client-ca.pem, then the agent CA). Host cert-auth roles still pin
client-ca.pem, so an agent leaf satisfies no host role. swarm-bao-certs
composes the same bundle before openbao starts.
- openbao reads tls_client_ca_file only at start, so when the bundle
changed after openbao started, swarm-bao-agent-pki restarts
openbao.service in the container; under `seal = "shamir"` it prints
the step instead. Once swarm-bao-certs has a cached CA, later boots
start openbao with it and do not restart.
- The controller policy gains exactly `update` on
pki-agents/issue/swarm-agent. mint_and_verify now asks that role for
the leaf (the store generates the key), writes the agent's cert-auth
role pinning the issuing CA bao returned, and writes the agent's
policy as render_agent alone: the hive-shared queue credential stanza
is gone.
- deploy.bao.agentPkiRoleName (must start `swarm-`, asserted with the
other pki role names); swarm-controller gets
SWARM_CONTROLLER_AGENT_PKI_MOUNT/_ROLE from the deploy.bao options.
Deleted: swarm-controller-agent-ca and its options (agentCaFile,
agentCaKeyFile), env, LoadCredential entries and assertion;
agent_identity's Authority, rcgen signing and validity window; the
rcgen and time dependencies of swarm-controller (rcgen leaves the
workspace); policy::render_agent_with_queue and its tests. The CN-prefix
assertion policy.rs said was owed is not: agent and host roles pin
different CAs.
Migration is re-creating each agent after deploy; that overwrites the
stale role and policy.
Closes#4756
A MintAgentForgeToken node mints a fixed-name swarm-agent token with the
admin API, keeps it when the stored value's last eight and the normalised
scopes match the forge's list, and otherwise deletes and re-creates it.
The token is stored at swarm/agents/<agent>/forge-token. Agent creation
inserts the node, and a pass at start and every five minutes inserts it
for every agent holding a store identity whose token is missing or stale.
Refs #3782
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped secret today, so at the auth callout one agent is
indistinguishable from its co-hived neighbours and no subject can be
scoped to one of them.
Mint a secret per agent instead, at swarm level, into
secret/swarm/agents/<agent>/queue -- inside the stanza every agent's ACL
document already grants, so no policy changes and no existing agent's
document is rewritten. It is written by the same node that already mints
the agent's certificate, and read back under the agent's own token
before that node reports success.
The secret is not derived from the agent's mTLS identity: the two
credentials answer different questions and coupling their lifetimes
would mean renewing either implied renewing the other. Nothing here
rotates a queue secret -- a re-run keeps the existing value and only
corrects the principal it names, because this function is re-run
deliberately against agents that are already connected. Revoking one
means deleting the path.
Nothing reads the new credential yet; this is the minting half.
`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.
The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.
The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.
`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.
The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.
Refs #4137
The agreement half of delivering the agent queue principal's client secret
through the store. No producer yet, so nothing writes this path — the unit
that does lands in the same PR, with the write grant it needs.
queue.rs is the sibling matrix.rs prescribes for a second kind of secret
rather than another field on a shared struct. Keyed per HIVE, not per agent:
the queue identity is minted once per hive at deploy time and says which hive
an agent belongs to, never which agent.
The client id rides with the secret for matrix.rs's stated reason — a
credential has to be reconstructable from the store alone, and deriving
`hive-<name>-agent` on the reading side is the split spelling the authelia
module warns denies every agent as a timeout.
policy.rs's render() takes the hive name now and emits a second, narrow
stanza for that hive's own path. The agent stanza is untouched: an agent's
path does not name its hive, so narrowing it still needs the enumeration
docs/trust-boundary/security.md rejects. A hive path does name its principal,
so scoping it costs nothing and drifts nowhere.
every_hive_gets_a_byte_identical_document is replaced rather than deleted.
Its surviving half is that the text is a function of the deploy-time name
alone, so a re-emission cannot drift; the new arms are that one hive's
document cannot reach another's path, and that a name which could close the
stanza is refused — live again now that a name reaches the document text.
Refs #3853
`write_policy` called `vaultrs::sys::policy::set`, which targets
`sys/policy/<name>` — the deprecated alias the store gates as a path of
its own. Every grant in this tree names `sys/policies/acl/hive-*`, so a
correctly-scoped controller was refused with a 403 and no hive read
policy has ever been written, on any deployment. The doc comment three
lines above the call already named the modern path; the code is what
moves to meet it.
vaultrs has no endpoint for that path (`grep policies/acl` over 0.8.0:
zero hits, against 8 for `sys/policy`), so this defines one over its own
endpoint machinery — which keeps the client's token header and `/v1`
prefix middleware rather than re-deriving them.
The alternative was to widen the grant to cover the legacy path. This
way needs no policy change at all: the deployed grant already permits
the write, so no store is re-bootstrapped and nothing is entrenched on
an alias upstream has deprecated.
Two tests pin the path and the body shape. The reason this survived
deployment is that nothing in the tree ever asserted either one.
Closes#4177.
A hive holds an mTLS pair and a policy naming what it may read, and still
cannot log in: nothing creates the role that maps its certificate to that
policy. The one pre-shared credential in the system therefore buys no
access.
Minting happens here rather than in nix, which was the first plan. Nix
mints from the store's own container, and that path is gated on the
bootstrap token -- so onboarding a hive later would mean placing the one
genuinely pre-shared secret again. Doing it from the controller costs a
public certificate authority as an input and makes the bootstrap token
one-time.
A startup pass, not a hook: the hive list is loaded once and a config
change means a redeploy, so the roles are as static as the list. Only the
policy is derived from something that moves.
The subject is the hive's name because glue-bao-tls.nix mints a hive's
client leaf with its name as the CN, and cert auth matches on that.
Per-hive failures are logged and skipped, matching the queue, bridge and
forge connects above it: a controller whose store is unreachable still
serves everything else, and the next start retries.
Not covered by a test: ensure_hive_roles is IO from end to end, and the
seam that would make it assertable is the one the read-grant sink already
has. Said here rather than implied by a green suite.
The renderer beside this produces a hive's read policy; nothing could put
one in the store. `read`/`write` are KV v2 verbs and a policy is not a
secret, so this is a second endpoint rather than another path.
A whole-document write, because the store has no merge verb: the caller
renders from the current agent set, so a stanza missing from the render
is a grant that is meant to be gone.
mara, reviewing the previous commit: "the field is specific to matrix, why
add it to the general struct". She is right, and the answer is that there was
no general struct — `Credential` had one consumer, the crate's only path
builder was `matrix_account`, and `value` is pinned by
`glue-matrix-bao-token.nix`, a matrix unit. It was matrix's throughout,
wearing a general name; adding `homeserver` is what made that visible.
`client` now moves whatever type a caller names and decodes nothing itself.
That is forwarding rather than machinery: `vaultrs::kv2::read`/`set` are
already generic over the payload.
The matrix agreement moves to its own module holding both halves — where a
credential lives (`account_path`, was `path::matrix_account`) and what the
object at that path holds. `path` keeps only what every path obeys, so a
second kind of swarm secret becomes a module beside `matrix` rather than
another optional field on a struct it shares. argus raised the same collision
from the other direction on #4092: two mutually-exclusive `Option`s modelling
one concept is the failure mode this forecloses.
`checked_segment` stays public in `path`: hive-priv builds an on-disk path
from the same names and must accept the same charset.
Behaviour is unchanged. The compatibility properties move with the struct —
`Option` is what lets a pre-`homeserver` stored object decode, and
`skip_serializing_if` is what keeps a token-only object free of
`"homeserver":null` for that nix reader.
Refs #3726
A delivered matrix account needs two things: the token and the homeserver it
belongs to. Only the token was stored, so the homeserver had to ride on the
queue notice — and a notice is not persistence. Re-delivering a credential
(agent moved, hive re-provisioned, token rotated) has to reconstruct it from
somewhere, and there is nowhere; keeping it separately at swarm level would be
a second store for one logical object, free to drift from the first.
So `Credential` grows a `homeserver` field and `read`/`write` carry the whole
object rather than a bare string.
`value` keeps its name. `nix/host-modules/glue-matrix-bao-token.nix` reads the
store with `bao kv get -field=value` and is the only nix reader of it, checked
rather than assumed — so this had to be an addition, never a rename.
Two compatibility properties, both of which fail silently if broken:
KV2 keeps every prior version, so objects written before this field existed
are still decoded by this type. What tolerates their absence is the field
being `Option` — a bare `String` would not fail as a migration, every stored
credential would become unreadable at once. The new test pins that, with a
presence control so the arm is about absence being tolerated rather than the
field being ignored.
`skip_serializing_if` keeps a token-only credential serialising to exactly the
bytes the previous version wrote, with no `homeserver` key rather than a null,
which is what that nix reader would otherwise trip over. The existing test
pinning `{"value":"t"}` proves it and became the control for free.
Mutation testing earned its place here: `#[serde(default)]` was in the first
draft and its comment claimed it was what made old objects decode. Dropping it
changed nothing — serde already decodes a missing field to `None` for an
optional type — so the attribute was redundant and the comment was wrong about
its own mechanism. Both removed rather than left to mislead the next reader.
The delivery half needed no change: `write_agent_matrix_token` already took a
homeserver and already wrote the `matrix-account-<name>.json` sidecar beside
the token. `deliver` simply stops passing `None`. A credential stored without
one still works exactly as before — no sidecar, and the account needs a
configured entry.
Refs #3726
mara ruled (a) on #3726: a thin workspace crate over `vaultrs` rather than
keeping bao access in nix and having each end trigger units. The HTTP is the
SDK's job; what this crate owns is the things the controller and a hive must
say *identically*, and which have no other home because neither end is senior
to the other.
Three such agreements:
`path::matrix_account` builds where a credential lives. It is fallible rather
than a `format!`, because both names reach it from elsewhere -- the agent name
from the topology, the account name from an agent's own config -- and a `/` or
`..` in either does not produce a malformed path, it produces a valid path to
a *different agent's* secret. The charset mirrors the KV bucket-name rule.
`Credential`'s `value` field is not a free choice: glue-matrix-bao-token.nix
reads the store with `bao kv get -field=value`, so the name is load-bearing
for a consumer no Rust test can reach. A test pins the serialised shape.
`client::Settings` reads BAO_ADDR / BAO_CLIENT_CERT / BAO_CLIENT_KEY /
BAO_CACERT explicitly instead of letting vaultrs fall through to its own
defaults, which look for VAULT_ADDR / VAULT_CLIENT_CERT / VAULT_CLIENT_KEY.
Every unit in this tree sets the BAO_ spellings, so the defaults would yield a
client with no identity at all -- surfacing as a TLS handshake failure, which
names neither the missing variable nor the reason.
The env read is split from the connect so every misconfiguration arm is
testable without a reachable store and without touching process-global env.
Dependency impact, measured against the lock at forge/main rather than assumed:
native-tls 0 -> 0, openssl-sys 0 -> 0, one reqwest (0.13.4) which vaultrs
shares, and 10 new crates that are all derive/proc-macro helpers.
Refs #3726
