swarm-controller now creates `term-main-<agent>` for every live agent, in
the same one-minute pass and with the same config style as
`term-sub-<agent>` (#4897): subjects `$SWARM.term.<agent>` only, max_age
24h, max_bytes 64 MiB, discard Old, max_message_size from the queue's
max_payload. An existing stream is opened as it is. The agent keeps its
plain publish on `$SWARM.term.<agent>` and gets no JetStream subject.
`GET /api/agents/{name}/term/stream` creates an ephemeral push consumer on
that stream starting 100 rows back, so a swarm-ui terminal opens with
recent history and then follows new rows, in order, from one
subscription. Each SSE event carries the row's stream sequence as its id,
and a reconnect's Last-Event-ID resumes after it. Without the stream the
route falls back to the live core subscription it used before.
The stream is `term-main-<agent>`, not `term-<agent>`: an agent may be
named `sub-x`, whose `term-<agent>` would be agent `x`'s subagent stream.
Its one literal subject cannot match `$SWARM.term.<agent>.sub.>`, which
needs more tokens, so the two streams never overlap.
The reader gains `$JS.API.CONSUMER.CREATE.*.*.$SWARM.term.*`, the
extended create form whose last tokens are the consumer's filter. Checked
against nats-server 2.14.4, the trailing `*` also admits a literal `*` or
`>` filter, so a consumer under it reaches any terminal row, main or
subagent; those are rows the reader already subscribes to and reads with
`STREAM.MSG.GET.*`.
Closes#4826
swarm-controller keeps each agent's inference settings in a new
`agent-inference` KV bucket, one key per agent, and is its only writer
(`GET`/`PUT`/`DELETE /api/agents/{name}/inference`; `swarmctl agent
inference get|set|clear`). Every field is optional. Each agent reads its
own key every 10 s through a direct-get grant on that key alone, and a
set `model` or `effort` replaces the Bus value, so the next turn runs on
it and the badge follows.
The swarm value is applied without persisting it: `Bus::set_model` and
`set_effort` are split into a persisting setter and a non-persisting
`apply_*`, and the poller uses the latter. Otherwise the swarm value
would overwrite `hyperhive-model`/`hyperhive-effort`, and clearing it, or
a restart, would no longer find the agent's own value. An unset field
changes nothing; a field cleared after being applied goes back to the
local value (`harness_state::local_model`/`local_effort`, the same
resolution `Bus::new` uses), read again at the clear.
`subagent_models` is validated, stored and shown by `get`; no agent
applies it yet.
Refs #5034
- swarm-controller webhook: regenerate the secret only when the file is
absent. Any other read error is returned with the path; the caller
already logs "webhook secret unavailable" and the endpoint answers 503.
Overwriting meant every registered hook kept the old secret and every
delivery failed HMAC, with nothing re-registering them.
- swarm-queue-client wanted: open_read_only/watch return Result; only
STREAM_NOT_FOUND on the bucket maps to None. hive-c0re's wanted pull
fails the DAG node with the cause instead of logging "nothing has been
declared"; the watch warns with the error chain.
- swarm-nats-auth: warn when the OIDC secret re-read task dies.
- swarm-controller forge: listing a repo's config PRs failing is warn,
not debug.
Closes#5056
The reader's subscribe allow-list spelled `$SWARM.term.*`,
`$SWARM.term.*.sub.*` and `$SWARM.agent-state.*` as literals, copies of
private prefixes in hive-agent and swarm-controller. A rename on either side
left the policy behind and the stream stopped with only a connection-level
server error.
swarm-queue-client now owns `term_subject`, `agent_state_subject` and their
`*_wildcard` forms, and `subagent_term::subject_wildcard`; hive-agent's
publishers, swarm-controller's relays and the policy all build from them.
Subjects are byte-identical.
A hive's queue credential and the swarm controller's could subscribe to
every subject in the swarm account, so a hive could read every other
hive's deploy requests and every agent's terminal, state, input and
request replies.
Their grants now carry a subscribe allow-list:
- a hive `hive-<h>`: `$SWARM.knowledge`, `$SWARM.deploy.<h>` and its
reply inbox `_INBOX.client.hive-<h>.>`
- the reader (swarm-controller): `$SWARM.term.*`, `$SWARM.term.*.sub.*`,
`$SWARM.agent-state.*` and its reply inbox
`_INBOX.client.<client-id>.>`
`swarm_queue_client::connect`, which hive-c0re and swarm-controller both
use, now connects with `_INBOX.client.<client-id>` as its inbox prefix,
the same shape agents got for their own credential. In async-nats 0.50
every request reply and every ordered consumer (the hive's wanted-state
watch, the controller's KV key listings) delivers under that prefix, so
the one inbox grant covers them all.
A hive or controller still on a build without the prefix has every
JetStream reply refused once the responder runs this policy, so the
clients need to be rebuilt before, or with, swarm-nats-auth.
Closes#4945
When the store re-read failed or timed out, introspect fell through to the
unchanged-secret branch and logged "the stored OIDC client secret is the one
the IdP refused" after the re-read's own failure line. The re-read now says
whether it read the store, and a failed one denies without the second line.
The verdict is unchanged.
The re-read of the responder's own OIDC client secret had no timeout:
neither swarm-secret-client nor vaultrs nor reqwest sets one by default,
and while a re-read runs every later refusal joins it rather than
starting another. A store connection whose peer stays alive but silent
therefore held off rotation pickup indefinitely. The spawned re-read
now gives up after 10 s, logs that it did (without the secret), keeps
the secret in use, and frees the slot for the next refusal's re-read.
The link from a non-2xx introspection (identify_caller's
`Answer::Refused`) to that re-read had no test: every existing test
handed `OidcSecret::introspect` an `Answer` directly. A new test runs
the real `identify_caller` against a loopback IdP that answers 401 to
the old secret and admits the new one, so mapping a non-2xx to
anything but `Refused` turns it red. axum is a dev-dependency for that
stub, as in swarm-controller's forge tests.
The responder reads its OIDC client secret at
`swarm/services/<client-id>/oidc/client` with the store login it already
opens for the callout keys, and holds it in memory. `--client-secret-file`
is removed.
When introspection answers non-2xx, the responder re-reads the secret from
the store once and, if it changed, retries the introspection once. Both fit
inside the request's 1.5s introspection budget, under the server's 2s
callout timeout: the budget is now one deadline for the whole request. A
re-read that does not finish by then keeps running in the background, the
request is denied, and the next request introspects with what it read. One
re-read runs at a time; a refusal while one runs waits on it.
In swarm-nats.nix the `swarm-bao-nats-auth-oidc` fetch unit, the flag and
the `oidc-client.secret` LoadCredential entry are gone, along with the
unit's policy ordering in glue-bao-readers-policy-order.nix. A tmpfiles `r`
rule removes the copy a container still holds at
/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret;
nothing reads it.
No policy change: the `swarm-nats-auth` role already reads this path
(`natsAuthReaders` in swarm-bao.nix), which is the role the removed fetch
unit logged in as.
Deploy order: none, and no operator step. The store's grant already exists,
so the queue host can be deployed before, after or without the store's host
and authelia's. Rolling back re-creates the fetch unit and the file; the
store leaf is untouched.
The responder loads the issuer and callout-user seeds from the secret
store under its own role, beside the xkey, and no longer takes
--issuer-seed-file / --user-seed-file.
`ensure-xkey` becomes `ensure-callout-keys`: it ensures the xkey as before,
and for each nkey creates the seed create-only when the store holds none,
from --import-{issuer,user}-seed-file when that file is present, so a key
kept on disk keeps its public half. Once a read of the store returns a seed,
the import file is removed; when that seed differs from the file the
store's is kept and a warning names the file. The public halves are
published create-only at swarm/public/swarm-nats/callout-{issuer,user}.
The KeyStore trait and its test double move to `keystore` so both key
modules share them.
Every nats-server this module starts renders `auth_callout.xkey` or does
not start, and the responder serves only that server, in the same
container and generation. A request without `Nats-Server-Xkey` therefore
comes from no supported server, so the plaintext framing is gone: such a
request is logged at warn and gets no reply, and the server that sent it
times out the connect as a denial. A signed denial would have had to go
out unsealed, which is the path this removes.
Also scopes two claims to what is true: the xkey's public half is read by
the swarm-nats-auth policy only, and the metadata-delete gap in the
replace-key procedure is about the two xkey paths, not all of
`secret/metadata/`.
Writing bao's `default` policy needed a new stanza in the granter's own
policy, which only the bootstrap token can upload, so deploying it took
a manual root-token step. Nothing needed it: the one reader of the
callout xkey's public half is `swarm-bao-nats-callout-xkey`, which logs
in as the `swarm-nats-auth` role, and that role's policy already names
the path. `swarm-bao-nats-auth-policy` writes that policy as the granter
on every deploy, under the granter's existing `sys/policies/acl/swarm-*`
grant.
Removed: the `swarm-bao-public-read-policy` unit, its marker block, the
granter's `sys/policies/acl/default` stanza, the granter-role restart of
that unit, the bao.md section and the module-eval cases pinning them.
The responder reads the callout xkey before it connects, so a responder
with no secret store no longer starts, and its "deny every agent token,
serve OIDC" mode could not run. That mode is gone: `agent_token::Store`
is built from the settings the xkey load already read, and `authorize`
takes a source rather than an `Option` of one.
On the nix side, `deploy.nats.authBaoClientCertFile` and
`authBaoClientKeyFile` are now asserted at eval when the queue is
enabled. The units' no-identity branches, the copy unit's empty-file
fallback and its "denies agent tokens" message are removed; an absent
leaf file fails `swarm-nats-auth-secrets` like an absent seed does.
When nats-server has an auth_callout xkey it seals each request to that
key and names its own public xkey in the Nats-Server-Xkey header. The
responder now loads the callout xkey seed from the secret store before
it connects, opens a sealed request with it, and seals its reply (grant
or deny) back to the header's key. A request with no header is answered
in plaintext, as before.
A new `ensure-xkey` verb creates the pair when the store holds no seed:
the seed at swarm/services/swarm-nats/callout/xkey-seed, the public half
at swarm/public/swarm-nats/callout-xkey. Each write is a create-only
(cas=0) write, and a run whose create fails reads the stored value back,
so two concurrent runs end up with one key. The responder itself only
reads the seed and refuses to start without it.
swarm-secret-client gains SecretStore::create, the cas=0 write.
Docs (review on #4948): the harness's restart settings are not useful to
a reader, so the `Restart`/`RestartSec` bullet in docs/turn-loop and the
"30 seconds later, no start limit" wording in docs/swarm are removed.
What stays is the behaviour: the harness exits and comes back until the
queue accepts it.
Grant: with the harness's shared-client fallback gone, nothing presents
the hive's shared agent client (`hive-<hive>-agent`) to the queue;
swarm-logs uses that OIDC client over HTTP only. swarm-nats-auth now
refuses it outright instead of granting the configured
`$SWARM.term.{hive}.>` / `$SWARM.agent-state.{hive}.>` publish and its
`_INBOX.client.<id>.>` subscribe, so an agent holding it can no longer
publish as a sibling. The agent arm still runs before the hive arm, so
that id never falls through to its hive's grant.
Removed with it: the `--agent-publish-subject` flag and its two uses in
swarm-nats.nix, `Policy`'s agent subject list, and
swarm-queue-client's `client_inbox_prefix`. The module-eval now asserts
the flag is absent; the policy tests assert the refusal and that the
hive itself is still admitted.
GET /api/agents/{name}/subagents reads each subagent's newest row with
get_last_raw_message_by_subject, which publishes
$JS.API.STREAM.MSG.GET.term-sub-<agent>. The controller's NATS grant had
no STREAM.MSG.GET, so every lookup was refused and waited the 10s request
timeout, one subagent after another; six subagents passed nginx's 60s.
Grant `$JS.API.STREAM.MSG.GET.*`. A wildcard matches a whole subject
token only, so a `term-sub-*` spelling would name one stream literally;
`*` is the narrowest grant that covers the per-agent streams, the same
width as the controller's STREAM.INFO.* grant.
The lookups now run concurrently (16 at a time) with a 2s timeout each.
A lookup that fails or times out lists its subagent with a null
last_row_unix, and one warn per request names how many failed and the
first error.
An agent's queue credential could subscribe to every subject in the swarm
account, so any agent could read every other agent's terminal, turn state,
operator input and request replies. Agents are the least trusted clients
on the queue (mara's ruling on #4901), so their grants now carry a
subscribe allow-list:
- an agent's own credential (the harness and its subagent daemon):
`_INBOX.agent.<agent>.>` and `$SWARM.input.<agent>`
- the hive's shared agent client: `_INBOX.client.<client-id>.>`
Hive and swarm-controller grants are unchanged (no `sub` claim, so still
every subject).
The clients connect with the matching inbox prefix, so the replies they
rely on still arrive: the icon write's JetStream ack and the schedule
DIRECT.GET. `connect_with_token` takes the agent name and sets the prefix
itself; the hive-shared fallback uses the new
`connect_with_inbox_prefix`.
The harness no longer subscribes to its input subject when it fell back
to the hive's shared client: that identity names no agent, so it cannot
be granted one agent's input without all of them. It logs that and
skips, the same as the schedule reader on that path.
Refs #4901
swarm-controller keeps every scheduled prompt (id, name, agents, body,
anchor, interval, paused) as one document under one key of a new
`schedule-definitions` KV bucket, so each change is a single
compare-and-set and a prompt exists for all of its agents or none. It
writes each agent's key of a new `agent-schedules` bucket as the prompts
naming that agent, without names or other agents, after every change and
on a one-minute reconcile pass. It is the only writer of both, behind
GET/POST /api/schedules and PATCH/DELETE /api/schedules/{id}.
`swarmctl schedule add|list|edit|pause|resume|rm` drives those routes over
the controller socket.
An agent's own queue credential is granted one more subject, the direct
get of its own key in `agent-schedules`. A KV watch is not granted: its
consumer create subject carries no key, so it would cover every agent's.
hive-agent's new schedule ticker reads that key every 10 seconds and fires
due slots as the same `schedule:<id>` todo hive-c0re's worker pushes. Fire
times are anchor + k*interval; the last fired slot per schedule id is kept
in the state db, so a body edit or list refresh never moves or refires a
slot, downtime fires the latest missed slot once, a restart refires
nothing, a one-shot fires once, and paused slots are skipped.
Refs #4914
The agent view in the swarm UI gets a message box. The text reaches the
agent's harness straight from swarm-controller over the swarm queue and
becomes an inbox turn from the operator's authelia username. hive-c0re is
not in the path.
- gateway: `/api/` forwards authelia's `Remote-User`, replacing any the
client sent.
- swarm-controller: `POST /api/agents/{name}/input` takes that header as
the sender (401 without it) and sends `{from, body}` as a NATS request
on `$SWARM.input.<agent>`. No subscriber is a 503.
- swarm-nats-auth: the controller is granted publish on `$SWARM.input.*`;
no hive or agent grant names the family, so the harness can trust
`from`.
- hive-agent: a task subscribes to the agent's own input subject and
pushes each message into the serve loop's existing reminder channel,
renamed `local_msg_*` since it now carries both.
The harness does not reply to the request: no agent grant covers a reply
subject. The controller therefore answers 202 once the request times out
(2s) and 503 on the server's no-responders status.
Closes#4828
swarm-controller now creates `term-sub-<agent>` for every agent a hive is
declared to run, at start and every minute after, with the config
`swarm_queue_client::subagent_term::open_or_create` spells (subjects
`$SWARM.term.<agent>.sub.>`, max_age 24h). An existing stream is opened as
it is, as the controller does for its other streams and buckets, under the
`$JS.API.STREAM.CREATE.*` grant it already holds.
The agent no longer creates the stream: its token is granted publish on
`$SWARM.term.<agent>.sub.>` and no `$JS.API.STREAM.CREATE|INFO` subject,
and the subagent daemon only publishes. A `CREATE` carries the stream's
config in its payload, which no subject grant narrows, so the agent could
otherwise pick the stream's subjects and limits.
An agent's subagent daemon publishes each subagent's output as terminal
rows on `$SWARM.term.<agent>.sub.<subagent>`, as the agent, into a
per-agent stream it creates itself; swarm-controller lists an agent's
subagents from that stream's subjects and relays one subagent's rows as
SSE; the swarm UI lists them under the agent's terminal preview and
reuses AgentTermPreview, full-screen tab included, with no input.
- swarm-nats.nix: the agent token may also publish
`$SWARM.term.{agent}.sub.>` and `$JS.API.STREAM.CREATE|INFO` on
`term-sub-{agent}`, and nothing else of JetStream. A module-eval arm
pins the agent-token grant as an exact list.
- mcp.nix: hive-subagent-daemon loads the agent's store identity
(`hive-agent-bao-cert/-key/-server-ca`, the ones hive-agent loads)
whenever the agent has a store, not only on the opencode preset. The
agent's own queue secret lives in the store, so this is the credential
the harness connects with.
- hive-subagent-mcp: `swarm_term` reads the agent's queue secret under
that identity, connects with the agent token, opens or creates
`term-sub-<agent>` (max_age 24h), and publishes classified rows from
the sink every subagent line already passes through. The sink only
queues (bounded, drop-and-count); a missing store, refused credential,
failed stream create or failed publish is a log line.
- The stream-json classifier (`stream_enrich`) and the `TermMsg` row
types plus `fit` move from the hive-agent binary into hive-sh4re, so
the subagent daemon publishes the rows AgentTermPreview already
renders. hive-agent keeps its LiveEvent classifier on top.
- swarm-controller: `GET /api/agents/{name}/subagents` and
`GET /api/agents/{name}/subagents/{subagent}/term/stream`.
- docs/swarm: what the UI shows and what the queue carries.
Closes#4827
hive-matrix-daemon now learns which external matrix accounts it has from
the swarm secret store, under the agent's own certificate, and the hive
push chain for matrix is gone.
The daemon lists swarm/agents/<agent>/matrix/ (the `list` its policy
grants on its own metadata subtree), reads each account's homeserver
from its credential, and brings the accounts up with their tokens from
the store. Every two minutes it lists again and exits with 75 when the
set of linked accounts changed; the unit restarts on 75 without counting
a failure. A listed name whose credential reads as absent is skipped and
logged once. At start it removes the matrix-token-<a> /
matrix-account-<a>.json pairs a hive delivered (a sidecar marks a pair
as delivered; a declared tokenFile keeps its token).
Removed: CredentialNotice and the $SWARM.credential.* subject and NATS
grant, the controller's publish and its queue precondition on the PUT
route, hive-c0re's credential subscription arm and workers/credential.rs,
priv_client::write_agent_matrix_token, hive-priv's WriteAgentMatrixToken
and its helpers, and the daemon's state-dir account discovery.
Kept: WriteAgentGithubToken and the external-forge path
(WriteAgentExtraForgeAccount, extra_forges.rs) are untouched, and a
declared matrixAccounts tokenFile is still read when the store has no
token for that account.
Refs #4348
A five-minute pass over every agent some hive's wanted state declares as
anything but destroyed queues, per agent:
- `MintAgentIdentity` (the node agent creation uses) when the stored
certificate at swarm/agents/<agent>/bao-mtls is past half its validity,
read from its own notBefore/notAfter: day 45 of the role's 90;
- the new `RenewAgentQueueCredential` node when the queue secret at
swarm/agents/<agent>/queue is 45 days old or has no mint time. The node
re-decides, writes a fresh value with `minted_at`, reads it back, and logs
the agent and the old age.
When both are due the secret node runs after_any the certificate node,
because mint_and_verify compares the queue secret it read with the one it
reads back. A credential that is not stored is never created here.
`queue::AgentCredential` gains an optional `minted_at` (unix seconds);
agent creation now sets it. Stored objects without it decode unchanged and
count as due, so every existing queue secret is re-minted on the first pass.
Both replacements reach the agent at its next start. The old certificate
stays valid until it expires; the old queue secret does not, so a queue
reconnect before that restart is denied.
Adds x509-cert 0.2 (with der_derive and flagset) to read the validity.
docs/swarm/credentials.md: the renewal column splits into automatic re-mint
and automatic re-pull, filled from the code as it stands.
The auth callout grants an agent that presents its own queue credential
one more subject, `$KV.agent-icons.<agent>`: its own key in the
agent-icons bucket and no other. The hive's shared agent client is
granted none of the bucket, since every agent on a hive presents it.
hive-agent writes `/etc/hyperhive/icon.svg`, the file its `GET /icon`
serves, to that key once per start, as a JetStream publish straight to
the subject (what `kv::Store::put` sends, minus the bucket lookup), so
the one subject is the whole grant. No icon deletes the key. A failed
write, including one that arrives before the bucket exists, is retried
with backoff until acked. An agent connected with the hive's shared
client publishes nothing.
swarm-controller creates the bucket as soon as its queue connection is
up, instead of on the first icon read, so an agent's write does not
wait for someone to look.
Measured against a local nats-server with a user allowed publish on
`$KV.agent-icons.atlas` only: the write to its own key is stored and
readable, a write to `$KV.agent-icons.argus` is refused (the ack times
out), the DEL marker makes the key read as absent, and a write before
the bucket exists fails with "no responders".
An `auth_token` spelled `swarm-agent.<agent>.<secret>` is no longer sent
to introspection. The responder reads `swarm/agents/<agent>/queue` with
an identity of its own, checks that the stored object names the same
agent, compares the secret in constant time, and grants the subjects
`--agent-token-publish-subject` lists with `{agent}` expanded. Every
other outcome denies: a malformed token, no store identity, nothing
stored, a failed or slow lookup, a different secret. A token without
the prefix takes the OIDC path unchanged.
The journal's `auth request` line names such a caller `agent:<agent>`;
the hive-shared credential keeps `hive-<h>-agent`.
The new principal: a `swarm-nats-auth` cert-auth role and policy with
read on `secret/data/swarm/agents/+/queue` alone, a leaf signed by the
store's PKI glue, and `glue-nats-auth-bao-identity.nix` pairing the two.
The copy unit delivers the identity into the queue's container, and an
absent leaf is delivered empty so the responder still starts and only
agent tokens are refused.
The policy and role are written by `swarm-bao-nats-auth-policy`, logged in
as the bao granter: both names fall under its `swarm-*` globs, so the
deploy writes them with no operator step. module-eval counts it among the
granting units, so every generic granting-unit case covers it.
The secret compare uses `subtle`, already in the lock file through the
TLS stack; no workspace crate offered one directly.
rustls is built with both `ring` (async-nats's `ring` feature) and
`aws-lc-rs` (reqwest's `rustls` feature), so it cannot pick a
process-level default by itself. Since the queue started requiring TLS
(1d261b3f), async-nats builds its config with `ClientConfig::builder()`,
which panics without an installed default. The panic kills the async-nats
connector task, and every queue client (swarm-controller, hive-c0re, all
hive-agents) has sat in `Pending` since the 2026-09-25 23:04Z deploy.
Add `swarm_queue_client::install_crypto_provider()`, which installs
aws-lc-rs and ignores the "already installed" error. It is called first in
`main` of every binary that links async-nats: hive-agent, hive-c0re,
swarm-controller, swarm-nats-auth. `connect()` also calls it, so a new
binary that dials through this crate is covered without remembering to.
aws-lc-rs because reqwest already falls back to it when no default is
installed, so HTTPS in these processes keeps its current provider. The
other rustls users in the tree reach it only through reqwest, which never
panics here.
Closes#4738
`--agent-client-suffix` takes a value that opens with `-`, and clap reads
one space-separated as a short-flag cluster. `swarm-nats.nix` passes it that
way, so the responder aborted at startup on `unexpected argument '-a'`.
A responder that never subscribes is not a degraded queue. `auth_callout` is
fail-closed by design, so the server denies every client, and each denial
arrives as `authorization violation` on the client — naming nothing, on a
process that is itself healthy. Two unrelated daemons reported it at once and
neither was at fault.
The option's own `default_value` is `-agent`, so the binary could not accept
the value it ships. Fixing the module's call site instead would leave that
true for anyone running it by hand.
The three tests parse argument vectors rather than assert on a struct: one is
the literal vector the unit builds, one is the control that an ordinary value
goes through the same flag, and one pins the default against the module.
`agent-<hive>` reads as "the agent named <hive>" — which is the one thing that
identity does not carry, since it is minted per hive. It becomes
`hive-<hive>-agent`: the hive's own id, extended.
The rename is not a string swap. `hive-foo-agent` satisfies the hive parse too
(it strips to a hive named `foo-agent`), so the responder's agent rule now runs
BEFORE its hive rule — most specific wins. Hive-first would have handed every
agent its hive's grant, including writing that hive's status key, with nothing
to report it: the client authenticates and is merely able to do more than it
should.
`Policy::new`'s overlap check goes with the prefix it was written for. The
invariant the suffix form needs instead is that the suffix is non-empty: an
empty one makes `strip_suffix` succeed on every hive id, so the two principals
become one string and whichever arm runs first answers for both.
The suffix form also introduces a collision the prefix form did not have: a hive
genuinely named `foo-agent` mints `hive-foo-agent`, which is hive `foo`'s agent
id. The responder cannot see it — it has no roster, deliberately — so
`swarm-authelia.nix` asserts at eval that no hive name ends with the suffix. The
existing duplicate-id assertion does not cover this: it fires only when both
`foo` and `foo-agent` are on the roster, and with `foo-agent` alone there is no
duplicate, just a hive quietly receiving its agents' grant.
A test written by analogy with `the_prefix_alone_names_no_hive` failed, correctly
— `hive--agent` is a hive named `-agent` under the hive parse, which this module
cannot rule out. It now asserts only the part this module owns: no empty hive
name is ever expanded into a subject.
Agents have authelia *users*; they had no machine identity at all, so an
agent could not authenticate to the swarm queue as anything. This mints
one `agent-<hive>` OIDC client per hive beside the existing
`hive-<hive>` one, teaches the auth-callout responder an agent arm, and
opens the queue's client port on the bridge so a container can reach it.
One client per HIVE, not per agent: agents are created at runtime, and a
per-agent client would make creating one a config change plus an
authelia reload. The cost is that agents on a hive are indistinguishable
to the broker, which is deliberate and tracked separately.
The agent grant is deny-by-default twice over. An agent id matches no
hive rule, so it gets a hive's status-key grant from neither; and with
no agent subject configured the responder returns no grant at all rather
than an empty publish list, which would be a denial wearing a grant's
shape. What an agent may publish is a deployment's decision, taken
through `--agent-publish-subject` the same way `--hive-publish-subject`
already works.
`Policy::new` now refuses two prefixes where one contains the other. The
arms are tried in order, so that overlap does not error at match time -
it silently hands one principal the other's grant.
Not shipped here, and neither is reachable without it: no subject is
configured for agents anywhere in nix, and nothing yet delivers
`agent-<hive>.secret` into an agent container. Both belong to the stream
that will be the first consumer.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
queue that is the ports it answers on, the client id it is registered
under, and the responder package. Whether this host mints its own
auth-callout keypairs and where the seeds sit are decisions of the
machine running it, so autoGenerateCallout, both seed files and both
public keys move to `deploy.nats.*`.
The two PUBLIC keys are the non-obvious half, so the reasoning is here
rather than in a comment nobody re-reads. A public key looks swarm-wide,
and docs/swarm/secrets.md says the user key "is published to every
client that connects" — which is true and does not make it swarm
config: peers RECEIVE it over the wire at connect time, they never
configure it. What decides the placement is that its seed is
unambiguously host-side, and splitting a keypair across two namespaces
is worse than either placement.
local-defaults.nix set `nats.autoGenerateCallout` from INSIDE
`config.services.hyperhive.swarm = { ... }`, so the bare prefix meant
`swarm.nats` and no full-path grep could see it. It moves out of that
block into a `deploy` statement rather than being rewritten in place.
swarm-controller.nix bound `natsCfg` and never used it — one mention,
no bare pass, while its sibling bindings have 5 and 10. Pre-existing
dead code, found by this slice's own alias sweep, removed with it.
Four assertion messages and five doc lines named moved options by a
short form (`nats.calloutUserSeedFile`) that is ambiguous now that both
`swarm.nats` and `deploy.nats` exist; one opened with
"services.hyperhive.swarm.nats has callout public keys", which the split
makes false. Prose that names a VALUE rather than a path — the `nk`
pipeline's `# -> calloutUserPublicKey` — is left bare on purpose.
module-eval configures a hive through all five OLD paths and asserts the
responder's delivery unit exists and interpolates the seed path.
`autoGenerateCallout` is deliberately FALSE there: that is what makes
the seed options decide `responderConfigured`, so the assertion is about
the seeds rather than about the auto-mint branch.
A hive's grants for its own `KV_hive-wanted-<hive>` were `STREAM.INFO` +
`DIRECT.GET`, which cover the boot-time read and nothing after it. The
convergence path now opens a KV watch on that bucket, and a watch is a
consumer, so the broker denies it — and the client's `watch()` ends in
`.ok()`, so the denial becomes `None` and the path silently never fires.
The comment four lines above the grant list already argues for this: the
per-hive bucket split exists "so that a watch can be granted without
widening the read". The design was taken and the grant was never written.
Both subject forms, matching the hive-status and agent-status blocks that
grant both for the same documented reason — an ephemeral consumer's
subject carries no name, and `>` never matches zero tokens.
Refs #4006.
mara on #4015: "not merging code without callers", and on the same PR
"see issue, we decided what the first thing should be". #3726 decided it:
the controller writes a token to the store and tells the hive; the hive
reads it back and writes /agents/<agent>/state/matrix-token-<account> at
0600, where matrix.nix's existing systemd.paths glob re-fires the daemon.
So this is the hive half of that, and the library's first caller.
The notice names a credential and never carries one, and deploy_subject's
own doc is why: the auth-callout responder scopes publish and leaves sub
unrestricted, so a hive that wanted another's messages could subscribe to
them. A secret in that payload would be readable swarm-wide. The value is
read from the store under the reading hive's own certificate, where the
store's policy is what actually scopes it.
Two boundaries guard the two addresses, and they are not the same check.
`path::matrix_account` guards the address in the store. `Ident` guards the
address on disk -- `agent_state_dir` takes one, so an unvalidated name off
the queue cannot reach a directory. I had written the first and assumed it
covered both; the compiler refused the `&str` and was right. `token_path`
now takes the newtype so a call site cannot forget.
The write is atomic because the path-watcher fires on the file appearing:
written in place it would be visible while partial, and the daemon would
read a truncated credential exactly once, which is the hardest possible
failure to reproduce. The temp name is dot-prefixed so it cannot match the
`matrix-token*` glob on its way past.
The publish grant is here because without it the failure is invisible.
policy.rs already says why for its siblings: a refused publish reaches the
client as a timeout, so the symptom is a hive that never receives a
credential with nothing in either log naming a permission. Two tests: the
controller may publish, a hive may not -- its own subject included. A
forged notice leaks nothing, but it would make a hive fetch and overwrite
a token file for a name the forger chose.
Refs #3726
A hive reads its own declaration today and that scopes cleanly: DIRECT.GET
carries the key in the subject, so the grant can name it. A *watch* cannot be
scoped that way — a consumer's filter travels in the request payload, so
$JS.API.CONSUMER.CREATE.<stream> grants the whole stream. With every hive in
one bucket, letting a hive watch its own declaration would let it read every
other hive's.
One bucket per hive (hive-wanted-<hive>) makes the stream a hive may hold
exactly as wide as what it is allowed to see, which is what #4006's live-watch
needs. That watch is a separate change; this only moves the boundary.
mara's calls, both on #4006: one stream per hive rather than teaching the auth
responder a hive roster, and a wildcard for the controller — "its okay if
swarm controller can theoretically override hive". A bucket name is a single
subject token with no prefix matching, so no wildcard narrower than * covers N
per-hive buckets; the controller's grant is account-wide by consequence, and
documented as chosen rather than left to look accidental.
The reader arm of #4005's key-layout guard asserted the opposite of that
ruling, so it is replaced rather than deleted: the hive arm survives as
no_hive_may_write_another_role_s_agent_status (with a positive control), and
the_readers_grant_is_deliberately_account_wide pins the decision and names the
ruling, so the width reads as chosen to whoever finds it next.
Two pre-existing negative assertions were silently defanged by the rename --
they matched hive-wanted.beta and $KV.hive-wanted.alpha, strings nothing
produces any more, and kept passing. Both now match current names.
swarm-controller resolves the store per hive per call instead of caching one in
a OnceCell: there is no single handle that serves N buckets, and declarations
change on operator action rather than per tick.
A KV entry publishes to $KV.<bucket>.<key> and NATS wildcards match whole
.-delimited tokens, so a /-joined {hive}/{agent} key is a single token: the
only expressible write grants are one exact subject per agent (needs a roster
in the auth responder, which Policy::hive_name argues against) or a bucket-wide
wildcard that lets any hive overwrite any other hive's agents.
Joining with a dot puts the hive in its own token, so hive_subjects can grant
$KV.agent-status.<hive>.* — every agent of one hive and nothing else, the same
containment hive-status already has.
The grant lands with the bucket-open pair (STREAM.INFO + STREAM.CREATE):
open_or_create resolves the bucket before it writes, so alone the publish
subject is unreachable and the sweep fails one step later instead.
Verified the client accepts a dotted key rather than assuming it: async-nats
0.50.0 VALID_KEY_RE is \A[-/_=.a-zA-Z0-9]+\z and is_valid_key rejects only
empty / leading / trailing dot; the subject is prefix + key verbatim.
The swarm-controller had no grant for the per-agent status bucket, so
resolving it failed and the dashboard answered 503.
Grants the reader the same five subjects the hive-status bucket already
has, for the same measured reasons -- including BOTH `CONSUMER.CREATE`
forms. `keys()` builds an ephemeral consumer whose create subject carries
no consumer name, and `>` matches one or more tokens, never zero, so the
`.>` form alone does not cover it; that omission surfaces as a client
timeout and an operator-visible 503, which is the shape reported here.
Deliberately grants NO `$KV.agent-status.*` subject, to anyone. That is
the write side, and writing is what picks a key layout -- whether agent
status is published by the agent or forwarded by its hive, and under
which key, is still open. A write grant here would answer that by
implication. Reading needs none of it: a KV read is a DIRECT.GET, and the
`.>` form is bucket-wide rather than per-key, so nothing added here
encodes a layout. A test asserts that, for the reader and for a hive,
with a control proving the same matcher fires on a bucket that does have
write grants.
The stream name is derived from the crate constant, like the wanted-state
bucket beside it, so writer and reader cannot disagree about it.
Pure `nix fmt` output from the commit before this one — no hand edits.
203 files: 52 md, 42 tsx, 32 js, 32 css, 21 ts, 13 html, 8 json, 3 mjs.
Reproduce with `nix develop -c nix fmt` on the parent commit; the result
should be byte-identical to this tree.
None of the 13 `.prettierignore` entries appears here — verified by
intersecting the changed-file list against the ignore file, with a
control proving the intersection finds a match when one exists.
The bucket the previous commit adds had no grants, so the controller
could not create or write it and no hive could read its own key.
Measured against nats-server 2.14.4 rather than extended by analogy,
because the shapes are not symmetric:
- controller: `STREAM.INFO` + `STREAM.CREATE` on `KV_hive-wanted` and
`$KV.hive-wanted.>`. With only today's status grants, `get_key_value`
timed out and the server named the two missing stream subjects.
- hive: `STREAM.INFO` plus **one** direct-get subject carrying its own
key. A KV read is a publish — `store.get` is a request — and the
direct-get subject embeds the key, so the read scopes per hive. By
analogy with `reader_subjects` this would have been `.>`, handing
every hive every hive's wanted set.
Both refusals were verified to fire, not assumed: as `alpha`,
`get beta` was refused naming
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.beta`, and
`put alpha` was refused naming `$KV.hive-wanted.alpha`.
`a_reader_may_list_and_fetch_but_not_write` asserted a reader holds no
`$KV.` subject at all, which held only while status was the sole
bucket. Narrowed to the status bucket — the invariant it defends is
that the controller cannot forge a hive's own report, and the wanted
bucket runs the other way.
Refs #3124
`hive_subjects` granted `$JS.API.STREAM.INFO`/`CREATE` on `hive-notices`
but never the per-hive subject a publish actually targets, so every
`notices::publish` was refused with a permissions violation naming
`hive-notices.<hive>`.
The gap survived because `notices_subjects`'s own doc comment asserted
`extra_hive_subjects` covered it "since that one *is* per-hive". Nothing
in `nix/` has ever passed `--hive-publish-subject` (`git grep` → 0 hits;
control: `--reader-client` is passed at `nix/host-modules/swarm-nats.nix:711`),
so the publish had been refused for as long as the feature existed.
Subject comes from `swarm_queue_client::notices::subject`, the same
constant the publisher formats from, rather than a literal here.
Refs #3859
Per mara on the PR: *"split by hive. its not a security thing, just so
hives dont get messages they dont care about."*
She agreed with the finding and still wanted the split, which is the
part worth recording. I measured that a per-hive subject gives no
confidentiality — `sub` is unrestricted, so a hive that wanted another's
messages could subscribe to them — and concluded it bought nothing.
"Nothing" is a claim over every axis and I had checked one. The axis I
never priced: every hive in the swarm being woken by every other hive's
deploys.
So `deploy_subject(hive)` replaces the single literal, and the payload
drops `hive` to carry only the agent — the subject names the hive, and
two places stating one fact are free to disagree. The hive subscribes to
its own subject and no longer filters.
The grant is a wildcard rather than a subject per hive because the
responder has no roster: it cannot enumerate hives, and a grant that had
to track one would be a second place to get the list wrong — the same
argument `hive_name`'s doc makes about admission.
The negative test gets stronger rather than merely adapted. Splitting
the family makes "another hive's subject" and "its own" separate strings
for the first time, so it now asserts a hive reaches neither, nor the
wildcard.
The subject and its payload live in `swarm-queue-client` for the reason
the knowledge event's already does: three crates have to agree on the
string, and the one that agrees hardest — the auth-callout responder,
which decides whether the publish is permitted at all — speaks neither
`jetstream` nor `kv`.
Swarm-wide rather than a `$SWARM.deploy.<hive>` family. That family
would look like isolation and provide none: this responder scopes
publish only, leaving `sub` unrestricted, so a hive could subscribe to
another's subject as easily as to its own. Until `sub` is scoped the
split costs a wider grant and buys nothing, so the addressing goes in
the payload and each hive filters on its own name.
Unlike the knowledge event the message is addressed, so it carries a
payload — a trigger, never the config. The hive already tracks the
agent's config repo; desired state on the wire would make this a second
source of truth for something git owns, and a hive that missed a
message would be wrong rather than late.
Both test arms mirrored from the knowledge event. The negative one
matters more here: a forged knowledge event makes a hive re-read a
repo, a forged deploy event makes it rebuild and restart a named agent.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The controller could not emit an event at all: a reader's grant is
`reader_subjects()`, which is `$JS.API.*` only, so a publish to any event
subject would be refused — and a NATS refusal reaches the client as a
timeout, so the visible symptom would have been a hive that never hears
about a change, with nothing in any log naming a permission.
Adds `swarm_queue_client::events`, following `status::BUCKET`: three
crates must agree on these strings (the controller publishes, a hive
subscribes, the callout responder decides whether the publish is
permitted), and a literal repeated across crates is an agreement nothing
checks. The responder speaks neither jetstream nor kv, so the module is
unconditional and carries no NATS types, exactly as the bucket name is.
The grant takes the wildcard form from the same function the publisher
calls, so the two cannot drift; a separate wildcard constant would have
re-created the disagreement this module exists to prevent.
Tests pin that a reader gets the subject and that a hive does NOT — a
hive able to publish here could tell a neighbour the knowledge repo
changed when it had not, which is an unauthenticated write into someone
else's control path. That one asserts on the subject root rather than a
rendered subject, so a future event leaf fails it too instead of passing
because the test only knew about `knowledge`.
Both assertions mutation-tested: removing the grant fails the reader
test, granting a hive the subject fails the denial test, each on its own
assertion line, and the unmutated tree is green.
`store.keys()` creates an ephemeral ordered consumer, whose create subject
ends at the stream name. `>` matches one or more tokens and never zero, so
the `.>` form alone never covered it: the server refused
`$JS.API.CONSUMER.CREATE.KV_hive-status`, the refusal reached the client as a
timeout, and the operator saw a 503 on the hive status page.
The test asserted only the `.>` form, which reads as covering the bare one, so
the suite stayed green while every list timed out in production. It now names
the bare subject separately and first.
The module doc had grown a 37-line preamble carrying three separate
arguments, which the comment-block lint refuses. Splitting it is the
better fix than raising the limit: the reader who is about to widen a
subject list meets the reason not to at the list, not seven screens up.
Also corrects one claim I had no measurement for. The CREATE note said
creating an existing stream with a different config "is an error rather
than a rewrite" — asserted, not observed. What is observed is narrower
and enough: a hive holding this grant leaves the stream config untouched
and never publishes $JS.API.STREAM.UPDATE at all.
A grant carrying every bucket-specific subject and neither of these
cannot create the bucket at all: the client times out on `$JS.API.INFO`
long before it reaches a subject that was granted, and a NATS denial
reaches the client as a hang rather than an error.
Both were named by the server's own log, not reasoned about.
`$JS.API.INFO` is the account-level JetStream info every client requests
on connect; `$JS.API.STREAM.NAMES` is how a client finds the stream
backing a bucket. The latter lets a client enumerate stream names in the
account, which in an account holding one bucket discloses a name both
ends already share.
Every earlier measurement missed them, because each either granted
`$JS.API.>` wholesale or ran against a bucket the setup had already
created while unscoped. A minimum established against an existing bucket
is not the minimum for making one, so the leave-one-out that trimmed the
reader's set could not have found this — every candidate it tried was
tried in a world where the bucket existed.
Found by running the shipping gate against the real binary. No unit test
could have: the failure is a timeout inside a real server's permission
check.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
