hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.
`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.
swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.
The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.
agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.
Part of #5017 (slice Q).
An agent's CPU and memory caps, set from the swarm. Keyed by the agent
alone, in a new `agent-limits` KV bucket that swarm-controller alone
writes, the same shape as `agent-inference` (#5034): mara on #4268,
"agents are not tied to a hive. dont tie the limits to the hive."
PUT {"cpu_quota": "400%", "memory_max": "8G"} every field optional,
replaces the stored value
GET the stored caps, {} when there are none
DELETE 204
PUT refuses an agent no hive's wanted state declares (as the inference
route does), unknown fields (422), zero and anything systemd does not take,
before the queue is asked, and stores memory in the spelling systemd takes
(8gb -> 8G). The hive hands these values to systemd verbatim.
The routes sit under /api/, behind the swarm UI vhost's authelia
subrequest like every sibling; the core-toggle module-eval case pins that
gate.
The bucket's reader on the hive comes in the next commit.
Contributes to #3865.
swarm-queue-client gains review_classifier: the try-out subject, the
request and reply that travel on it, and the deadlines both ends share.
The controller's reader grant may publish the subject; a client named
with --review-classifier-client may subscribe to it and its inbox, and
answers under NATS allow_responses (resp in the user JWT) within the
run deadline. Every other client id is still denied.
swarm-controller now creates `term-main-<agent>` for every live agent, in
the same one-minute pass and with the same config style as
`term-sub-<agent>` (#4897): subjects `$SWARM.term.<agent>` only, max_age
24h, max_bytes 64 MiB, discard Old, max_message_size from the queue's
max_payload. An existing stream is opened as it is. The agent keeps its
plain publish on `$SWARM.term.<agent>` and gets no JetStream subject.
`GET /api/agents/{name}/term/stream` creates an ephemeral push consumer on
that stream starting 100 rows back, so a swarm-ui terminal opens with
recent history and then follows new rows, in order, from one
subscription. Each SSE event carries the row's stream sequence as its id,
and a reconnect's Last-Event-ID resumes after it. Without the stream the
route falls back to the live core subscription it used before.
The stream is `term-main-<agent>`, not `term-<agent>`: an agent may be
named `sub-x`, whose `term-<agent>` would be agent `x`'s subagent stream.
Its one literal subject cannot match `$SWARM.term.<agent>.sub.>`, which
needs more tokens, so the two streams never overlap.
The reader gains `$JS.API.CONSUMER.CREATE.*.*.$SWARM.term.*`, the
extended create form whose last tokens are the consumer's filter. Checked
against nats-server 2.14.4, the trailing `*` also admits a literal `*` or
`>` filter, so a consumer under it reaches any terminal row, main or
subagent; those are rows the reader already subscribes to and reads with
`STREAM.MSG.GET.*`.
Closes#4826
swarm-controller keeps each agent's inference settings in a new
`agent-inference` KV bucket, one key per agent, and is its only writer
(`GET`/`PUT`/`DELETE /api/agents/{name}/inference`; `swarmctl agent
inference get|set|clear`). Every field is optional. Each agent reads its
own key every 10 s through a direct-get grant on that key alone, and a
set `model` or `effort` replaces the Bus value, so the next turn runs on
it and the badge follows.
The swarm value is applied without persisting it: `Bus::set_model` and
`set_effort` are split into a persisting setter and a non-persisting
`apply_*`, and the poller uses the latter. Otherwise the swarm value
would overwrite `hyperhive-model`/`hyperhive-effort`, and clearing it, or
a restart, would no longer find the agent's own value. An unset field
changes nothing; a field cleared after being applied goes back to the
local value (`harness_state::local_model`/`local_effort`, the same
resolution `Bus::new` uses), read again at the clear.
`subagent_models` is validated, stored and shown by `get`; no agent
applies it yet.
Refs #5034
swarm-controller keeps every scheduled prompt (id, name, agents, body,
anchor, interval, paused) as one document under one key of a new
`schedule-definitions` KV bucket, so each change is a single
compare-and-set and a prompt exists for all of its agents or none. It
writes each agent's key of a new `agent-schedules` bucket as the prompts
naming that agent, without names or other agents, after every change and
on a one-minute reconcile pass. It is the only writer of both, behind
GET/POST /api/schedules and PATCH/DELETE /api/schedules/{id}.
`swarmctl schedule add|list|edit|pause|resume|rm` drives those routes over
the controller socket.
An agent's own queue credential is granted one more subject, the direct
get of its own key in `agent-schedules`. A KV watch is not granted: its
consumer create subject carries no key, so it would cover every agent's.
hive-agent's new schedule ticker reads that key every 10 seconds and fires
due slots as the same `schedule:<id>` todo hive-c0re's worker pushes. Fire
times are anchor + k*interval; the last fired slot per schedule id is kept
in the state db, so a body edit or list refresh never moves or refires a
slot, downtime fires the latest missed slot once, a restart refires
nothing, a one-shot fires once, and paused slots are skipped.
Refs #4914
swarm-controller now creates `term-sub-<agent>` for every agent a hive is
declared to run, at start and every minute after, with the config
`swarm_queue_client::subagent_term::open_or_create` spells (subjects
`$SWARM.term.<agent>.sub.>`, max_age 24h). An existing stream is opened as
it is, as the controller does for its other streams and buckets, under the
`$JS.API.STREAM.CREATE.*` grant it already holds.
The agent no longer creates the stream: its token is granted publish on
`$SWARM.term.<agent>.sub.>` and no `$JS.API.STREAM.CREATE|INFO` subject,
and the subagent daemon only publishes. A `CREATE` carries the stream's
config in its payload, which no subject grant narrows, so the agent could
otherwise pick the stream's subjects and limits.
An agent's subagent daemon publishes each subagent's output as terminal
rows on `$SWARM.term.<agent>.sub.<subagent>`, as the agent, into a
per-agent stream it creates itself; swarm-controller lists an agent's
subagents from that stream's subjects and relays one subagent's rows as
SSE; the swarm UI lists them under the agent's terminal preview and
reuses AgentTermPreview, full-screen tab included, with no input.
- swarm-nats.nix: the agent token may also publish
`$SWARM.term.{agent}.sub.>` and `$JS.API.STREAM.CREATE|INFO` on
`term-sub-{agent}`, and nothing else of JetStream. A module-eval arm
pins the agent-token grant as an exact list.
- mcp.nix: hive-subagent-daemon loads the agent's store identity
(`hive-agent-bao-cert/-key/-server-ca`, the ones hive-agent loads)
whenever the agent has a store, not only on the opencode preset. The
agent's own queue secret lives in the store, so this is the credential
the harness connects with.
- hive-subagent-mcp: `swarm_term` reads the agent's queue secret under
that identity, connects with the agent token, opens or creates
`term-sub-<agent>` (max_age 24h), and publishes classified rows from
the sink every subagent line already passes through. The sink only
queues (bounded, drop-and-count); a missing store, refused credential,
failed stream create or failed publish is a log line.
- The stream-json classifier (`stream_enrich`) and the `TermMsg` row
types plus `fit` move from the hive-agent binary into hive-sh4re, so
the subagent daemon publishes the rows AgentTermPreview already
renders. hive-agent keeps its LiveEvent classifier on top.
- swarm-controller: `GET /api/agents/{name}/subagents` and
`GET /api/agents/{name}/subagents/{subagent}/term/stream`.
- docs/swarm: what the UI shows and what the queue carries.
Closes#4827
rustls is built with both `ring` (async-nats's `ring` feature) and
`aws-lc-rs` (reqwest's `rustls` feature), so it cannot pick a
process-level default by itself. Since the queue started requiring TLS
(1d261b3f), async-nats builds its config with `ClientConfig::builder()`,
which panics without an installed default. The panic kills the async-nats
connector task, and every queue client (swarm-controller, hive-c0re, all
hive-agents) has sat in `Pending` since the 2026-09-25 23:04Z deploy.
Add `swarm_queue_client::install_crypto_provider()`, which installs
aws-lc-rs and ignores the "already installed" error. It is called first in
`main` of every binary that links async-nats: hive-agent, hive-c0re,
swarm-controller, swarm-nats-auth. `connect()` also calls it, so a new
binary that dials through this crate is covered without remembering to.
aws-lc-rs because reqwest already falls back to it when no default is
installed, so HTTPS in these processes keeps its current provider. The
other rustls users in the tree reach it only through reqwest, which never
panics here.
Closes#4738
The bucket has two ends in two crates: a hive writes its own key, the
controller reads every key. `swarm-controller` declared the name as a
private const with a doc comment arguing that "reader and writer must
name the same bucket" — an argument the writer, in another crate, could
not obey.
The name is the mild half. Both ends do get-or-create, because either may
come up first on a fresh swarm and neither can assume the other has run.
Two `Config`s that drift means whichever end created the bucket wins and
the other's `get_key_value` succeeds against a bucket it did not ask for:
no error, no log, just a retention policy nobody chose. Sharing the
constructor gives that race one outcome.
Behind a default-off `kv` feature, so the crate's other consumer — the
auth-callout responder, which speaks the connect and nothing else — still
pulls neither `jetstream` nor `kv`. That was the actual reason the
feature was excluded when this crate was extracted; the flag preserves
it. The surface is deliberately narrow: one bucket's name and creation
config, not a general KV facade.
Operator ruling: libs should not use anyhow. The queue connect was moved
here verbatim from swarm-controller, which is a binary, so it arrived
still wearing a binary's error handling — the move changed what the code
is without changing how it reports.
Callers get variants they can match on, split by what an operator does
about them: a half-configured environment is a deployment bug, a refused
token is an identity-provider config problem, an unreachable queue is a
network one. The binaries that consume this keep anyhow and `?` converts,
so nothing downstream is more verbose for it. Same split hive-claude uses.
One thing anyhow was doing unpaid: the auth callback hands async-nats a
plain string, and a Display that stops at the top message drops the cause
— the half that says why the mint failed. `chain()` walks the source
chain, which is what `{:#}` was doing before.
A hive publishing its own status needs the same connect the controller
already has - mint an authelia token, present it at CONNECT for the
callout responder, let async-nats re-run the callback per attempt. Only
the use differs: the controller reads, a hive writes.
Copying it would put credential handling in two places, and a
token-refresh fix would then have to be found twice. That is the same
reasoning that already put hive-sock-client in its own crate rather than
in each daemon that speaks to a unix socket.
`from_env` takes a prefix rather than hardcoding SWARM_CONTROLLER_*: the
variables belong to the consuming unit, since a NixOS module sets them
alongside its other options. What is shared is the RULE - all four
together or none at all - not the spelling. The half-set case gains a
test, because it is the case the rule exists for and it previously had
none.
No jetstream/kv feature on the crate: it ends at a connected client, and
what a consumer does with it should be visible in that consumer's own
Cargo.toml.
Behaviour-preserving, and proven that way rather than by inspection: the
full behavioural gate (real nats-server, credential rotation, mutation)
is 20/0 unchanged, and the controller's own tests still pass.
