The responder reads its OIDC client secret at
`swarm/services/<client-id>/oidc/client` with the store login it already
opens for the callout keys, and holds it in memory. `--client-secret-file`
is removed.
When introspection answers non-2xx, the responder re-reads the secret from
the store once and, if it changed, retries the introspection once. Both fit
inside the request's 1.5s introspection budget, under the server's 2s
callout timeout: the budget is now one deadline for the whole request. A
re-read that does not finish by then keeps running in the background, the
request is denied, and the next request introspects with what it read. One
re-read runs at a time; a refusal while one runs waits on it.
In swarm-nats.nix the `swarm-bao-nats-auth-oidc` fetch unit, the flag and
the `oidc-client.secret` LoadCredential entry are gone, along with the
unit's policy ordering in glue-bao-readers-policy-order.nix. A tmpfiles `r`
rule removes the copy a container still holds at
/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret;
nothing reads it.
No policy change: the `swarm-nats-auth` role already reads this path
(`natsAuthReaders` in swarm-bao.nix), which is the role the removed fetch
unit logged in as.
Deploy order: none, and no operator step. The store's grant already exists,
so the queue host can be deployed before, after or without the store's host
and authelia's. Rolling back re-creates the fetch unit and the file; the
store leaf is untouched.
The responder loads the issuer and callout-user seeds from the secret
store under its own role, beside the xkey, and no longer takes
--issuer-seed-file / --user-seed-file.
`ensure-xkey` becomes `ensure-callout-keys`: it ensures the xkey as before,
and for each nkey creates the seed create-only when the store holds none,
from --import-{issuer,user}-seed-file when that file is present, so a key
kept on disk keeps its public half. Once a read of the store returns a seed,
the import file is removed; when that seed differs from the file the
store's is kept and a warning names the file. The public halves are
published create-only at swarm/public/swarm-nats/callout-{issuer,user}.
The KeyStore trait and its test double move to `keystore` so both key
modules share them.
Every nats-server this module starts renders `auth_callout.xkey` or does
not start, and the responder serves only that server, in the same
container and generation. A request without `Nats-Server-Xkey` therefore
comes from no supported server, so the plaintext framing is gone: such a
request is logged at warn and gets no reply, and the server that sent it
times out the connect as a denial. A signed denial would have had to go
out unsealed, which is the path this removes.
Also scopes two claims to what is true: the xkey's public half is read by
the swarm-nats-auth policy only, and the metadata-delete gap in the
replace-key procedure is about the two xkey paths, not all of
`secret/metadata/`.
The responder reads the callout xkey before it connects, so a responder
with no secret store no longer starts, and its "deny every agent token,
serve OIDC" mode could not run. That mode is gone: `agent_token::Store`
is built from the settings the xkey load already read, and `authorize`
takes a source rather than an `Option` of one.
On the nix side, `deploy.nats.authBaoClientCertFile` and
`authBaoClientKeyFile` are now asserted at eval when the queue is
enabled. The units' no-identity branches, the copy unit's empty-file
fallback and its "denies agent tokens" message are removed; an absent
leaf file fails `swarm-nats-auth-secrets` like an absent seed does.
When nats-server has an auth_callout xkey it seals each request to that
key and names its own public xkey in the Nats-Server-Xkey header. The
responder now loads the callout xkey seed from the secret store before
it connects, opens a sealed request with it, and seals its reply (grant
or deny) back to the header's key. A request with no header is answered
in plaintext, as before.
A new `ensure-xkey` verb creates the pair when the store holds no seed:
the seed at swarm/services/swarm-nats/callout/xkey-seed, the public half
at swarm/public/swarm-nats/callout-xkey. Each write is a create-only
(cas=0) write, and a run whose create fails reads the stored value back,
so two concurrent runs end up with one key. The responder itself only
reads the seed and refuses to start without it.
swarm-secret-client gains SecretStore::create, the cas=0 write.
Docs (review on #4948): the harness's restart settings are not useful to
a reader, so the `Restart`/`RestartSec` bullet in docs/turn-loop and the
"30 seconds later, no start limit" wording in docs/swarm are removed.
What stays is the behaviour: the harness exits and comes back until the
queue accepts it.
Grant: with the harness's shared-client fallback gone, nothing presents
the hive's shared agent client (`hive-<hive>-agent`) to the queue;
swarm-logs uses that OIDC client over HTTP only. swarm-nats-auth now
refuses it outright instead of granting the configured
`$SWARM.term.{hive}.>` / `$SWARM.agent-state.{hive}.>` publish and its
`_INBOX.client.<id>.>` subscribe, so an agent holding it can no longer
publish as a sibling. The agent arm still runs before the hive arm, so
that id never falls through to its hive's grant.
Removed with it: the `--agent-publish-subject` flag and its two uses in
swarm-nats.nix, `Policy`'s agent subject list, and
swarm-queue-client's `client_inbox_prefix`. The module-eval now asserts
the flag is absent; the policy tests assert the refusal and that the
hive itself is still admitted.
swarm-controller keeps every scheduled prompt (id, name, agents, body,
anchor, interval, paused) as one document under one key of a new
`schedule-definitions` KV bucket, so each change is a single
compare-and-set and a prompt exists for all of its agents or none. It
writes each agent's key of a new `agent-schedules` bucket as the prompts
naming that agent, without names or other agents, after every change and
on a one-minute reconcile pass. It is the only writer of both, behind
GET/POST /api/schedules and PATCH/DELETE /api/schedules/{id}.
`swarmctl schedule add|list|edit|pause|resume|rm` drives those routes over
the controller socket.
An agent's own queue credential is granted one more subject, the direct
get of its own key in `agent-schedules`. A KV watch is not granted: its
consumer create subject carries no key, so it would cover every agent's.
hive-agent's new schedule ticker reads that key every 10 seconds and fires
due slots as the same `schedule:<id>` todo hive-c0re's worker pushes. Fire
times are anchor + k*interval; the last fired slot per schedule id is kept
in the state db, so a body edit or list refresh never moves or refires a
slot, downtime fires the latest missed slot once, a restart refires
nothing, a one-shot fires once, and paused slots are skipped.
Refs #4914
swarm-controller now creates `term-sub-<agent>` for every agent a hive is
declared to run, at start and every minute after, with the config
`swarm_queue_client::subagent_term::open_or_create` spells (subjects
`$SWARM.term.<agent>.sub.>`, max_age 24h). An existing stream is opened as
it is, as the controller does for its other streams and buckets, under the
`$JS.API.STREAM.CREATE.*` grant it already holds.
The agent no longer creates the stream: its token is granted publish on
`$SWARM.term.<agent>.sub.>` and no `$JS.API.STREAM.CREATE|INFO` subject,
and the subagent daemon only publishes. A `CREATE` carries the stream's
config in its payload, which no subject grant narrows, so the agent could
otherwise pick the stream's subjects and limits.
An agent's subagent daemon publishes each subagent's output as terminal
rows on `$SWARM.term.<agent>.sub.<subagent>`, as the agent, into a
per-agent stream it creates itself; swarm-controller lists an agent's
subagents from that stream's subjects and relays one subagent's rows as
SSE; the swarm UI lists them under the agent's terminal preview and
reuses AgentTermPreview, full-screen tab included, with no input.
- swarm-nats.nix: the agent token may also publish
`$SWARM.term.{agent}.sub.>` and `$JS.API.STREAM.CREATE|INFO` on
`term-sub-{agent}`, and nothing else of JetStream. A module-eval arm
pins the agent-token grant as an exact list.
- mcp.nix: hive-subagent-daemon loads the agent's store identity
(`hive-agent-bao-cert/-key/-server-ca`, the ones hive-agent loads)
whenever the agent has a store, not only on the opencode preset. The
agent's own queue secret lives in the store, so this is the credential
the harness connects with.
- hive-subagent-mcp: `swarm_term` reads the agent's queue secret under
that identity, connects with the agent token, opens or creates
`term-sub-<agent>` (max_age 24h), and publishes classified rows from
the sink every subagent line already passes through. The sink only
queues (bounded, drop-and-count); a missing store, refused credential,
failed stream create or failed publish is a log line.
- The stream-json classifier (`stream_enrich`) and the `TermMsg` row
types plus `fit` move from the hive-agent binary into hive-sh4re, so
the subagent daemon publishes the rows AgentTermPreview already
renders. hive-agent keeps its LiveEvent classifier on top.
- swarm-controller: `GET /api/agents/{name}/subagents` and
`GET /api/agents/{name}/subagents/{subagent}/term/stream`.
- docs/swarm: what the UI shows and what the queue carries.
Closes#4827
The auth callout grants an agent that presents its own queue credential
one more subject, `$KV.agent-icons.<agent>`: its own key in the
agent-icons bucket and no other. The hive's shared agent client is
granted none of the bucket, since every agent on a hive presents it.
hive-agent writes `/etc/hyperhive/icon.svg`, the file its `GET /icon`
serves, to that key once per start, as a JetStream publish straight to
the subject (what `kv::Store::put` sends, minus the bucket lookup), so
the one subject is the whole grant. No icon deletes the key. A failed
write, including one that arrives before the bucket exists, is retried
with backoff until acked. An agent connected with the hive's shared
client publishes nothing.
swarm-controller creates the bucket as soon as its queue connection is
up, instead of on the first icon read, so an agent's write does not
wait for someone to look.
Measured against a local nats-server with a user allowed publish on
`$KV.agent-icons.atlas` only: the write to its own key is stored and
readable, a write to `$KV.agent-icons.argus` is refused (the ack times
out), the DEL marker makes the key read as absent, and a write before
the bucket exists fails with "no responders".
An `auth_token` spelled `swarm-agent.<agent>.<secret>` is no longer sent
to introspection. The responder reads `swarm/agents/<agent>/queue` with
an identity of its own, checks that the stored object names the same
agent, compares the secret in constant time, and grants the subjects
`--agent-token-publish-subject` lists with `{agent}` expanded. Every
other outcome denies: a malformed token, no store identity, nothing
stored, a failed or slow lookup, a different secret. A token without
the prefix takes the OIDC path unchanged.
The journal's `auth request` line names such a caller `agent:<agent>`;
the hive-shared credential keeps `hive-<h>-agent`.
The new principal: a `swarm-nats-auth` cert-auth role and policy with
read on `secret/data/swarm/agents/+/queue` alone, a leaf signed by the
store's PKI glue, and `glue-nats-auth-bao-identity.nix` pairing the two.
The copy unit delivers the identity into the queue's container, and an
absent leaf is delivered empty so the responder still starts and only
agent tokens are refused.
The policy and role are written by `swarm-bao-nats-auth-policy`, logged in
as the bao granter: both names fall under its `swarm-*` globs, so the
deploy writes them with no operator step. module-eval counts it among the
granting units, so every generic granting-unit case covers it.
The secret compare uses `subtle`, already in the lock file through the
TLS stack; no workspace crate offered one directly.
rustls is built with both `ring` (async-nats's `ring` feature) and
`aws-lc-rs` (reqwest's `rustls` feature), so it cannot pick a
process-level default by itself. Since the queue started requiring TLS
(1d261b3f), async-nats builds its config with `ClientConfig::builder()`,
which panics without an installed default. The panic kills the async-nats
connector task, and every queue client (swarm-controller, hive-c0re, all
hive-agents) has sat in `Pending` since the 2026-09-25 23:04Z deploy.
Add `swarm_queue_client::install_crypto_provider()`, which installs
aws-lc-rs and ignores the "already installed" error. It is called first in
`main` of every binary that links async-nats: hive-agent, hive-c0re,
swarm-controller, swarm-nats-auth. `connect()` also calls it, so a new
binary that dials through this crate is covered without remembering to.
aws-lc-rs because reqwest already falls back to it when no default is
installed, so HTTPS in these processes keeps its current provider. The
other rustls users in the tree reach it only through reqwest, which never
panics here.
Closes#4738
`--agent-client-suffix` takes a value that opens with `-`, and clap reads
one space-separated as a short-flag cluster. `swarm-nats.nix` passes it that
way, so the responder aborted at startup on `unexpected argument '-a'`.
A responder that never subscribes is not a degraded queue. `auth_callout` is
fail-closed by design, so the server denies every client, and each denial
arrives as `authorization violation` on the client — naming nothing, on a
process that is itself healthy. Two unrelated daemons reported it at once and
neither was at fault.
The option's own `default_value` is `-agent`, so the binary could not accept
the value it ships. Fixing the module's call site instead would leave that
true for anyone running it by hand.
The three tests parse argument vectors rather than assert on a struct: one is
the literal vector the unit builds, one is the control that an ordinary value
goes through the same flag, and one pins the default against the module.
`agent-<hive>` reads as "the agent named <hive>" — which is the one thing that
identity does not carry, since it is minted per hive. It becomes
`hive-<hive>-agent`: the hive's own id, extended.
The rename is not a string swap. `hive-foo-agent` satisfies the hive parse too
(it strips to a hive named `foo-agent`), so the responder's agent rule now runs
BEFORE its hive rule — most specific wins. Hive-first would have handed every
agent its hive's grant, including writing that hive's status key, with nothing
to report it: the client authenticates and is merely able to do more than it
should.
`Policy::new`'s overlap check goes with the prefix it was written for. The
invariant the suffix form needs instead is that the suffix is non-empty: an
empty one makes `strip_suffix` succeed on every hive id, so the two principals
become one string and whichever arm runs first answers for both.
The suffix form also introduces a collision the prefix form did not have: a hive
genuinely named `foo-agent` mints `hive-foo-agent`, which is hive `foo`'s agent
id. The responder cannot see it — it has no roster, deliberately — so
`swarm-authelia.nix` asserts at eval that no hive name ends with the suffix. The
existing duplicate-id assertion does not cover this: it fires only when both
`foo` and `foo-agent` are on the roster, and with `foo-agent` alone there is no
duplicate, just a hive quietly receiving its agents' grant.
A test written by analogy with `the_prefix_alone_names_no_hive` failed, correctly
— `hive--agent` is a hive named `-agent` under the hive parse, which this module
cannot rule out. It now asserts only the part this module owns: no empty hive
name is ever expanded into a subject.
Agents have authelia *users*; they had no machine identity at all, so an
agent could not authenticate to the swarm queue as anything. This mints
one `agent-<hive>` OIDC client per hive beside the existing
`hive-<hive>` one, teaches the auth-callout responder an agent arm, and
opens the queue's client port on the bridge so a container can reach it.
One client per HIVE, not per agent: agents are created at runtime, and a
per-agent client would make creating one a config change plus an
authelia reload. The cost is that agents on a hive are indistinguishable
to the broker, which is deliberate and tracked separately.
The agent grant is deny-by-default twice over. An agent id matches no
hive rule, so it gets a hive's status-key grant from neither; and with
no agent subject configured the responder returns no grant at all rather
than an empty publish list, which would be a denial wearing a grant's
shape. What an agent may publish is a deployment's decision, taken
through `--agent-publish-subject` the same way `--hive-publish-subject`
already works.
`Policy::new` now refuses two prefixes where one contains the other. The
arms are tried in order, so that overlap does not error at match time -
it silently hands one principal the other's grant.
Not shipped here, and neither is reachable without it: no subject is
configured for agents anywhere in nix, and nothing yet delivers
`agent-<hive>.secret` into an agent container. Both belong to the stream
that will be the first consumer.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
queue that is the ports it answers on, the client id it is registered
under, and the responder package. Whether this host mints its own
auth-callout keypairs and where the seeds sit are decisions of the
machine running it, so autoGenerateCallout, both seed files and both
public keys move to `deploy.nats.*`.
The two PUBLIC keys are the non-obvious half, so the reasoning is here
rather than in a comment nobody re-reads. A public key looks swarm-wide,
and docs/swarm/secrets.md says the user key "is published to every
client that connects" — which is true and does not make it swarm
config: peers RECEIVE it over the wire at connect time, they never
configure it. What decides the placement is that its seed is
unambiguously host-side, and splitting a keypair across two namespaces
is worse than either placement.
local-defaults.nix set `nats.autoGenerateCallout` from INSIDE
`config.services.hyperhive.swarm = { ... }`, so the bare prefix meant
`swarm.nats` and no full-path grep could see it. It moves out of that
block into a `deploy` statement rather than being rewritten in place.
swarm-controller.nix bound `natsCfg` and never used it — one mention,
no bare pass, while its sibling bindings have 5 and 10. Pre-existing
dead code, found by this slice's own alias sweep, removed with it.
Four assertion messages and five doc lines named moved options by a
short form (`nats.calloutUserSeedFile`) that is ambiguous now that both
`swarm.nats` and `deploy.nats` exist; one opened with
"services.hyperhive.swarm.nats has callout public keys", which the split
makes false. Prose that names a VALUE rather than a path — the `nk`
pipeline's `# -> calloutUserPublicKey` — is left bare on purpose.
module-eval configures a hive through all five OLD paths and asserts the
responder's delivery unit exists and interpolates the seed path.
`autoGenerateCallout` is deliberately FALSE there: that is what makes
the seed options decide `responderConfigured`, so the assertion is about
the seeds rather than about the auto-mint branch.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
The auth-callout responder learned whether a token was valid, never whose
it was, so every admitted client got the same unscoped grant. Scoping a
grant to one hive's subjects needs an identity to scope it to.
`is_active` becomes `identify_caller` and returns the identity rather
than a boolean: `Ok(Some(client_id))` admits as that client, `Ok(None)`
denies. `active: true` with no `client_id` is a denial, and returning an
identity is what makes that unrepresentable instead of remembered --
there is no admitted-but-unscoped value to construct, so there is no
branch a later edit can forget to handle.
The module deliberately ignores `sub`/`scope`/`exp` on the grounds that
modelling a field implies checking it. That still holds: `active` remains
the whole admission decision, made in one place. `client_id` answers a
different question -- as whom -- and is used downstream of an admission
that has already happened, not as a second gate. The module docs say so,
next to the paragraph that would otherwise argue for deleting the field.
An inactive token has no identity even when the body names one, and an
empty `client_id` counts as absent: it would become a blank component in
a subject the grant is scoped to, which is not a narrower permission but
a different one. The two credential parameters are renamed `own_*` --
this responder's introspection credential is not the caller's identity,
and the function now takes one and returns the other.
The grant itself is still unscoped; narrowing it is the next slice.
Per mara on the PR: all deps go into workspace level so versions stay
consistent. async-nats, data-encoding, nkeys and nats-jwt are new
[workspace.dependencies] entries; sha2 and futures-util now come from
there.
The crate had asked for sha2 0.10 while the workspace standard is 0.11,
and for the futures facade where the workspace carries futures-util. Both
resolved without adding a Cargo.lock entry - each was already in the tree
via something else - so the drift was invisible in the lock and would only
have surfaced as two versions of a hashing crate in one binary.
The responder answered granted=true and the server still refused every
client:
Error non operator mode account "AUTH": attempted to use issuer_account
nats_jwt::Token::new_user always sets issuer_account, which is an
operator-mode field naming the account when a signing key rather than the
account identity key signed the token. In server-config mode - what this
module renders - its mere presence is fatal, and the account is named by
the claims' aud instead.
nats-jwt can express neither aud nor the omission, so the user JWT is now
hand-built by the same signer as the response wrapper, and nats-jwt moves
to dev-dependencies as the encoder's test oracle.
Every unit test passed throughout: they assert fields that must be
present, and the defect was a field that must be absent.
Slice 1 shipped the NATS container with an auth_callout block and no
responder, which is the fail-closed state: the server answers
auth_required and admits nobody. This crate is what lets it say yes.
Connects as the callout-exempt user by nkey (never by name - the server
refuses to start if that entry carries a username), subscribes to
$SYS.REQ.USER.AUTH, validates the presented bearer token against
authelia's introspection endpoint, and replies with a signed NATS user
JWT. A denial is a signed response carrying an error, never silence: a
server that hears nothing cannot tell a refusing responder from a dead
one, so staying quiet would turn every rejection into a timeout and hide
an outage inside what looks like ordinary denials.
Everything that is not an explicit active:true denies - network error,
timeout, non-2xx, unparseable body, no token at all. Those are exactly
the conditions under which an attacker would most like this to fall
open. The introspection budget is held under the server's own 2s
auth_callout timeout by a test, since the two numbers live in different
languages in different files.
nats-jwt mints the user JWT. It cannot mint the authorization_response
wrapper - its claim enum is closed and its claims carry no aud, which
the response needs so a reply cannot be replayed at another server in
the cluster - so that half is hand-written, and a test builds a user
token both ways and requires the bytes to match. That is the only
honest basis for trusting the hand-written path on the shape the crate
does not model.
async-nats is taken with default-features off: the default set carries
jetstream, kv, object-store, websockets and service, none of which a
callout responder speaks.
