A subagent's terminal opened directly showed nothing until the subagent
published again: the relay was a core subscription, though its rows sit
in term-sub-<agent> for 24 hours. It now goes through the agent route's
replay-then-tail, so it opens with the subagent's newest 100 rows and
resumes after Last-Event-ID.
- term_stream.rs: the replay takes a stream name and a filter subject
instead of an agent, and `replay_or_live` (replay, else the live core
subscription) is the one fn both routes call. When the filter is one of
several subjects in the stream, a fresh reader's consumer starts at the
oldest row and drops all but the newest REPLAY_ROWS it counts as
pending, since the stream's newest sequences may belong to sibling
subagents. A single-subject stream (term-main-<agent>) picks its start
with replay_start exactly as before. Tests for both subagent cases.
- subagent_term.rs: the relay calls replay_or_live on the subagent's
subject in term-sub-<agent> and reads Last-Event-ID; module doc and
OpenAPI text no longer say "live only".
- main.rs: the handler test passes the new headers argument.
- swarm-queue-client agent_term.rs: replay_consumer_config takes the
filter subject instead of the agent.
- swarm-queue-client subagent_term.rs: consumer_create_grant, the
extended CONSUMER.CREATE form with a subagent subject as filter.
- swarm-nats-auth policy.rs: the controller holds that grant; the agent
grant's `$SWARM.term.*` filter is two tokens short of a subagent
subject, and the test that refused it now admits it.
- docs/swarm/ui.md, README.md: a subagent terminal opens with its newest
rows.
Fixes#5527
POST /api/agents takes an archetype the stored list holds, records it in
the new agent-archetype bucket, and queues no CreateRepo, AddRepoMember or
InitAgentConfigRepo for it. A destroy clears the key. Hives may read the
bucket.
Refs #5386
The agent read the operator-profiles bucket with a keys() listing, an
ordered consumer whose grant (STREAM.INFO, CONSUMER.CREATE) and flow-control
reply had never run against a permissioned queue. The controller now writes
the whole list as one JSON array under one key, only when it changed, and
the agent reads it with one DIRECT.GET the way agent_inference::read_own
reads its key. The agent grant shrinks to that one subject.
swarm-controller reads each member of authelia's operator group's forge
profile every five minutes and keeps one key per operator in the new
operator-profiles bucket. Every agent may list and read that bucket, and
hive-agent renders it into the system prompt at harness start, falling
back to today's prompt when the bucket is empty or doesn't answer within
five seconds.
Fixes#5383
A flake declares agent archetypes in its `hyperhiveArchetypes` output,
`<name> = { module; meta; }`. The hyperhive flake ships `reviewer`
(default permissions without `execution`).
swarm-builder answers `$SWARM.builder.archetypes` with the `meta` of
every archetype its own hyperhive source declares. swarm-controller asks
once the queue connects and every 10 minutes, checks each archetype as
`POST /api/agents` checks its settings, and stores the result in the
`agent-archetypes` KV bucket when it changed. `GET /api/archetypes`
serves it, and the create form's archetype select prefills permissions,
resource caps and inference from it.
The agent created is an ordinary agent with its own config repo; the
archetype's module is not applied yet.
Refs #5386
mara on #5170: "dont split caps and perms. its all perms. / dont include
the now empty tool groups / nothing set -> default / also add perms to
agent creation screen".
- hive-sh4re: one `Permission` enum (messaging, meta, inbox, execution,
web_tools, manage_root_agent) replaces `ToolGroup` and `Capability`;
`lifecycle`/`forge` are gone. `Permission::default_for` is the default
for an agent the swarm sets nothing for.
- The record is `{ "permissions": [...] }`. Missing key, unset field or
no successful read yet = default; `[]` = none; a failed read after a
good one keeps the last value.
- hive-agent reads its own key every 10s and builds each turn's --tools,
--allowedTools and MCP config (rewritten before every turn) from it;
the subagent daemon reads it on each start/continue. Agents get a
read-only grant on their own key.
- hive-c0re restarts an agent only when its manage_root_agent changes.
The import now merges meta/capabilities.json and meta/tool-groups.json
into one list per agent and deletes both; the "import pending, don't
read" decision is `ImportOutcome::read_allowed`, table-tested.
- swarm-controller: `GET /api/permissions` (names + default) and an
optional `permissions` on `POST /api/agents`, stored before the job is
queued. swarm-ui's create-agent form gets a permissions field.
- Deleted: HIVE_TOOL_GROUPS and its forward to the subagent daemon,
tool_groups.rs, ruth's tool-group seed, the permissions rebuild job
(PermPayload/WritePermFile/perm_change), the dashboard permissions tab
and its stale-entries panel.
Part of #5170.
An agent's capabilities move from the hive's meta repo
(`meta/capabilities.json`, injected as `HIVE_CAPABILITIES`) into the
`capabilities` field of its key in a new `agent-permissions` KV bucket,
which swarm-controller alone writes.
- swarm-queue-client: `agent_permissions` (bucket, key, `AgentPermissions`,
direct-get read, the import subject and its wire types).
- swarm-controller: creates the bucket at start; `GET|PUT|DELETE
/api/agents/{name}/permissions` (deny_unknown_fields and unknown
capability ⇒ 422, duplicate ⇒ 400, agent must be declared by a hive);
answers each hive's one-shot import, writing a key only where there is
none and only for agents that hive's wanted state declares.
- swarmctl: `agent permissions get|set|clear`.
- swarm-nats-auth: a hive reads every key of the bucket and sends its
import on its own subject; the controller subscribes to the imports and
may answer them (`responses`).
- hive-c0re: reads each of its agents' keys every 30s into
/var/lib/hyperhive/swarm-agent-permissions.json and restarts an agent,
gracefully, when its capabilities change. Unset is the compiled-in
default (ruth: manage_root_agent; everyone else: none). A failed read
keeps the stored value. On first boot it seeds that file from
meta/capabilities.json, sends the grants to the controller, and deletes
the file once the controller answers.
Deleted: the capabilities.json reader/writer, `HIVE_CAPABILITIES` in the
rendered meta flake, ruth's capability seed, the capabilities half of the
permissions rebuild job (`PermPayload::Capabilities`/`Combined`), the
dashboard's capabilities table and `capabilities_changed` event, and the
now-unused `Capability::description`.
Part of #5170.
A new swarm service, `swarm-claude-login`, in its own container on the host
that sets `deploy.claude-login.enable` (defaulted on by `allSwarmServices`
where the host has the minter's store leaf, like the review classifier).
- swarm-ui: an agent card showing "login needed" gets a "log in" button.
Its dialog starts the login, shows the URL claude printed, and takes the
pasted code.
- swarm-controller: `POST/DELETE /api/agents/{name}/claude-login` and
`POST /api/agents/{name}/claude-login/code`, behind the swarm UI's
authelia-gated `/api/`, relay the steps to the minter as requests on
`$SWARM.claude-login.request`.
- swarm-claude-login: runs `claude auth login` over piped stdio with HOME
and CLAUDE_CONFIG_DIR in a per-login scratch dir, scrapes the first
https URL, writes the code to stdin, and stores the `.credentials.json`
at `swarm/agents/<agent>/claude-credentials` with a KV v2 write, a new
version over any earlier one. The scratch dir goes on every exit path; a
login waits 15 minutes for its code.
- hive-agent: while parked in `wait_for_login`, reads that leaf every 30 s
under the agent's own cert and installs it only when its KV version is
newer than the one recorded in `<harness>/claude-credentials-version`.
An equal or older version is never put back, because claude rotates the
refresh token and the stored copy is stale after the first refresh.
- swarm-secret-client: `claude_login` paths and `SecretStore::read_versioned`.
- Credentials: a `claude-login` store leaf minted by glue-bao-tls, a
`swarm-claude-login` policy (create/update on
`swarm/agents/+/claude-credentials`, read on its own queue client
secret), an authelia machine client, and a `--claude-login-client`
grant in the queue's auth-callout responder. The unit runs as a static
system user.
The hive's own web-UI login is unchanged.
Refs #5268
A separate $SWARM.control.<agent> subject carries commands, apart from
the text subject. swarm-controller exposes POST /api/agents/{name}/cancel
as a NATS request; the harness answers whether a turn was actually
stopped, through the same cancel path as the dashboard button. The agent
credential gets subscribe on its control subject and allow_responses to
answer; only the controller may publish there.
Refs #3803
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
The grant test covered the bare consumer-create subject only, so a grant
of `$JS.API.CONSUMER.CREATE.KV_agent-limits.>` would have passed it.
It now also checks a named consumer, bare and with a key filter.
Raised by the #4268 verify pass.
Every 30 seconds `workers::swarm_limits` reads the `agent-limits` key of
each agent this hive has and stores the result in
swarm-resource-limits.json, where the drop-in, the heap ceiling and the
dashboard read it. A cap therefore follows its agent to whichever hive
runs it.
Each value goes through the hive's own validate_cpu_quota /
validate_memory_max before it is stored, since hive-priv renders it
verbatim into the container's drop-in, and one bad value refuses the whole
set (argus on #4268, carried over from the wanted-state consumer this
replaces). A failed read, or a bucket that does not answer, leaves the
stored caps as they are; three failures in a row raise a banner.
swarm-nats-auth grants every hive `$JS.API.DIRECT.GET.KV_agent-limits.>`:
read-only, and every agent's key, not just the hive's own. Scoping it to
the hive's agents would need a roster in the auth responder, which
policy.rs argues against. mara chose this on #4268 (c95293); narrowing it
is a possible low-priority follow-up, not planned. A test pins that the
grant reads every key, writes none, and reaches no other bucket.
Contributes to #3865.
swarm-queue-client gains review_classifier: the try-out subject, the
request and reply that travel on it, and the deadlines both ends share.
The controller's reader grant may publish the subject; a client named
with --review-classifier-client may subscribe to it and its inbox, and
answers under NATS allow_responses (resp in the user JWT) within the
run deadline. Every other client id is still denied.
swarm-controller now creates `term-main-<agent>` for every live agent, in
the same one-minute pass and with the same config style as
`term-sub-<agent>` (#4897): subjects `$SWARM.term.<agent>` only, max_age
24h, max_bytes 64 MiB, discard Old, max_message_size from the queue's
max_payload. An existing stream is opened as it is. The agent keeps its
plain publish on `$SWARM.term.<agent>` and gets no JetStream subject.
`GET /api/agents/{name}/term/stream` creates an ephemeral push consumer on
that stream starting 100 rows back, so a swarm-ui terminal opens with
recent history and then follows new rows, in order, from one
subscription. Each SSE event carries the row's stream sequence as its id,
and a reconnect's Last-Event-ID resumes after it. Without the stream the
route falls back to the live core subscription it used before.
The stream is `term-main-<agent>`, not `term-<agent>`: an agent may be
named `sub-x`, whose `term-<agent>` would be agent `x`'s subagent stream.
Its one literal subject cannot match `$SWARM.term.<agent>.sub.>`, which
needs more tokens, so the two streams never overlap.
The reader gains `$JS.API.CONSUMER.CREATE.*.*.$SWARM.term.*`, the
extended create form whose last tokens are the consumer's filter. Checked
against nats-server 2.14.4, the trailing `*` also admits a literal `*` or
`>` filter, so a consumer under it reaches any terminal row, main or
subagent; those are rows the reader already subscribes to and reads with
`STREAM.MSG.GET.*`.
Closes#4826
swarm-controller keeps each agent's inference settings in a new
`agent-inference` KV bucket, one key per agent, and is its only writer
(`GET`/`PUT`/`DELETE /api/agents/{name}/inference`; `swarmctl agent
inference get|set|clear`). Every field is optional. Each agent reads its
own key every 10 s through a direct-get grant on that key alone, and a
set `model` or `effort` replaces the Bus value, so the next turn runs on
it and the badge follows.
The swarm value is applied without persisting it: `Bus::set_model` and
`set_effort` are split into a persisting setter and a non-persisting
`apply_*`, and the poller uses the latter. Otherwise the swarm value
would overwrite `hyperhive-model`/`hyperhive-effort`, and clearing it, or
a restart, would no longer find the agent's own value. An unset field
changes nothing; a field cleared after being applied goes back to the
local value (`harness_state::local_model`/`local_effort`, the same
resolution `Bus::new` uses), read again at the clear.
`subagent_models` is validated, stored and shown by `get`; no agent
applies it yet.
Refs #5034
The reader's subscribe allow-list spelled `$SWARM.term.*`,
`$SWARM.term.*.sub.*` and `$SWARM.agent-state.*` as literals, copies of
private prefixes in hive-agent and swarm-controller. A rename on either side
left the policy behind and the stream stopped with only a connection-level
server error.
swarm-queue-client now owns `term_subject`, `agent_state_subject` and their
`*_wildcard` forms, and `subagent_term::subject_wildcard`; hive-agent's
publishers, swarm-controller's relays and the policy all build from them.
Subjects are byte-identical.
A hive's queue credential and the swarm controller's could subscribe to
every subject in the swarm account, so a hive could read every other
hive's deploy requests and every agent's terminal, state, input and
request replies.
Their grants now carry a subscribe allow-list:
- a hive `hive-<h>`: `$SWARM.knowledge`, `$SWARM.deploy.<h>` and its
reply inbox `_INBOX.client.hive-<h>.>`
- the reader (swarm-controller): `$SWARM.term.*`, `$SWARM.term.*.sub.*`,
`$SWARM.agent-state.*` and its reply inbox
`_INBOX.client.<client-id>.>`
`swarm_queue_client::connect`, which hive-c0re and swarm-controller both
use, now connects with `_INBOX.client.<client-id>` as its inbox prefix,
the same shape agents got for their own credential. In async-nats 0.50
every request reply and every ordered consumer (the hive's wanted-state
watch, the controller's KV key listings) delivers under that prefix, so
the one inbox grant covers them all.
A hive or controller still on a build without the prefix has every
JetStream reply refused once the responder runs this policy, so the
clients need to be rebuilt before, or with, swarm-nats-auth.
Closes#4945
Docs (review on #4948): the harness's restart settings are not useful to
a reader, so the `Restart`/`RestartSec` bullet in docs/turn-loop and the
"30 seconds later, no start limit" wording in docs/swarm are removed.
What stays is the behaviour: the harness exits and comes back until the
queue accepts it.
Grant: with the harness's shared-client fallback gone, nothing presents
the hive's shared agent client (`hive-<hive>-agent`) to the queue;
swarm-logs uses that OIDC client over HTTP only. swarm-nats-auth now
refuses it outright instead of granting the configured
`$SWARM.term.{hive}.>` / `$SWARM.agent-state.{hive}.>` publish and its
`_INBOX.client.<id>.>` subscribe, so an agent holding it can no longer
publish as a sibling. The agent arm still runs before the hive arm, so
that id never falls through to its hive's grant.
Removed with it: the `--agent-publish-subject` flag and its two uses in
swarm-nats.nix, `Policy`'s agent subject list, and
swarm-queue-client's `client_inbox_prefix`. The module-eval now asserts
the flag is absent; the policy tests assert the refusal and that the
hive itself is still admitted.
GET /api/agents/{name}/subagents reads each subagent's newest row with
get_last_raw_message_by_subject, which publishes
$JS.API.STREAM.MSG.GET.term-sub-<agent>. The controller's NATS grant had
no STREAM.MSG.GET, so every lookup was refused and waited the 10s request
timeout, one subagent after another; six subagents passed nginx's 60s.
Grant `$JS.API.STREAM.MSG.GET.*`. A wildcard matches a whole subject
token only, so a `term-sub-*` spelling would name one stream literally;
`*` is the narrowest grant that covers the per-agent streams, the same
width as the controller's STREAM.INFO.* grant.
The lookups now run concurrently (16 at a time) with a 2s timeout each.
A lookup that fails or times out lists its subagent with a null
last_row_unix, and one warn per request names how many failed and the
first error.
An agent's queue credential could subscribe to every subject in the swarm
account, so any agent could read every other agent's terminal, turn state,
operator input and request replies. Agents are the least trusted clients
on the queue (mara's ruling on #4901), so their grants now carry a
subscribe allow-list:
- an agent's own credential (the harness and its subagent daemon):
`_INBOX.agent.<agent>.>` and `$SWARM.input.<agent>`
- the hive's shared agent client: `_INBOX.client.<client-id>.>`
Hive and swarm-controller grants are unchanged (no `sub` claim, so still
every subject).
The clients connect with the matching inbox prefix, so the replies they
rely on still arrive: the icon write's JetStream ack and the schedule
DIRECT.GET. `connect_with_token` takes the agent name and sets the prefix
itself; the hive-shared fallback uses the new
`connect_with_inbox_prefix`.
The harness no longer subscribes to its input subject when it fell back
to the hive's shared client: that identity names no agent, so it cannot
be granted one agent's input without all of them. It logs that and
skips, the same as the schedule reader on that path.
Refs #4901
swarm-controller keeps every scheduled prompt (id, name, agents, body,
anchor, interval, paused) as one document under one key of a new
`schedule-definitions` KV bucket, so each change is a single
compare-and-set and a prompt exists for all of its agents or none. It
writes each agent's key of a new `agent-schedules` bucket as the prompts
naming that agent, without names or other agents, after every change and
on a one-minute reconcile pass. It is the only writer of both, behind
GET/POST /api/schedules and PATCH/DELETE /api/schedules/{id}.
`swarmctl schedule add|list|edit|pause|resume|rm` drives those routes over
the controller socket.
An agent's own queue credential is granted one more subject, the direct
get of its own key in `agent-schedules`. A KV watch is not granted: its
consumer create subject carries no key, so it would cover every agent's.
hive-agent's new schedule ticker reads that key every 10 seconds and fires
due slots as the same `schedule:<id>` todo hive-c0re's worker pushes. Fire
times are anchor + k*interval; the last fired slot per schedule id is kept
in the state db, so a body edit or list refresh never moves or refires a
slot, downtime fires the latest missed slot once, a restart refires
nothing, a one-shot fires once, and paused slots are skipped.
Refs #4914
The agent view in the swarm UI gets a message box. The text reaches the
agent's harness straight from swarm-controller over the swarm queue and
becomes an inbox turn from the operator's authelia username. hive-c0re is
not in the path.
- gateway: `/api/` forwards authelia's `Remote-User`, replacing any the
client sent.
- swarm-controller: `POST /api/agents/{name}/input` takes that header as
the sender (401 without it) and sends `{from, body}` as a NATS request
on `$SWARM.input.<agent>`. No subscriber is a 503.
- swarm-nats-auth: the controller is granted publish on `$SWARM.input.*`;
no hive or agent grant names the family, so the harness can trust
`from`.
- hive-agent: a task subscribes to the agent's own input subject and
pushes each message into the serve loop's existing reminder channel,
renamed `local_msg_*` since it now carries both.
The harness does not reply to the request: no agent grant covers a reply
subject. The controller therefore answers 202 once the request times out
(2s) and 503 on the server's no-responders status.
Closes#4828
swarm-controller now creates `term-sub-<agent>` for every agent a hive is
declared to run, at start and every minute after, with the config
`swarm_queue_client::subagent_term::open_or_create` spells (subjects
`$SWARM.term.<agent>.sub.>`, max_age 24h). An existing stream is opened as
it is, as the controller does for its other streams and buckets, under the
`$JS.API.STREAM.CREATE.*` grant it already holds.
The agent no longer creates the stream: its token is granted publish on
`$SWARM.term.<agent>.sub.>` and no `$JS.API.STREAM.CREATE|INFO` subject,
and the subagent daemon only publishes. A `CREATE` carries the stream's
config in its payload, which no subject grant narrows, so the agent could
otherwise pick the stream's subjects and limits.
An agent's subagent daemon publishes each subagent's output as terminal
rows on `$SWARM.term.<agent>.sub.<subagent>`, as the agent, into a
per-agent stream it creates itself; swarm-controller lists an agent's
subagents from that stream's subjects and relays one subagent's rows as
SSE; the swarm UI lists them under the agent's terminal preview and
reuses AgentTermPreview, full-screen tab included, with no input.
- swarm-nats.nix: the agent token may also publish
`$SWARM.term.{agent}.sub.>` and `$JS.API.STREAM.CREATE|INFO` on
`term-sub-{agent}`, and nothing else of JetStream. A module-eval arm
pins the agent-token grant as an exact list.
- mcp.nix: hive-subagent-daemon loads the agent's store identity
(`hive-agent-bao-cert/-key/-server-ca`, the ones hive-agent loads)
whenever the agent has a store, not only on the opencode preset. The
agent's own queue secret lives in the store, so this is the credential
the harness connects with.
- hive-subagent-mcp: `swarm_term` reads the agent's queue secret under
that identity, connects with the agent token, opens or creates
`term-sub-<agent>` (max_age 24h), and publishes classified rows from
the sink every subagent line already passes through. The sink only
queues (bounded, drop-and-count); a missing store, refused credential,
failed stream create or failed publish is a log line.
- The stream-json classifier (`stream_enrich`) and the `TermMsg` row
types plus `fit` move from the hive-agent binary into hive-sh4re, so
the subagent daemon publishes the rows AgentTermPreview already
renders. hive-agent keeps its LiveEvent classifier on top.
- swarm-controller: `GET /api/agents/{name}/subagents` and
`GET /api/agents/{name}/subagents/{subagent}/term/stream`.
- docs/swarm: what the UI shows and what the queue carries.
Closes#4827
hive-matrix-daemon now learns which external matrix accounts it has from
the swarm secret store, under the agent's own certificate, and the hive
push chain for matrix is gone.
The daemon lists swarm/agents/<agent>/matrix/ (the `list` its policy
grants on its own metadata subtree), reads each account's homeserver
from its credential, and brings the accounts up with their tokens from
the store. Every two minutes it lists again and exits with 75 when the
set of linked accounts changed; the unit restarts on 75 without counting
a failure. A listed name whose credential reads as absent is skipped and
logged once. At start it removes the matrix-token-<a> /
matrix-account-<a>.json pairs a hive delivered (a sidecar marks a pair
as delivered; a declared tokenFile keeps its token).
Removed: CredentialNotice and the $SWARM.credential.* subject and NATS
grant, the controller's publish and its queue precondition on the PUT
route, hive-c0re's credential subscription arm and workers/credential.rs,
priv_client::write_agent_matrix_token, hive-priv's WriteAgentMatrixToken
and its helpers, and the daemon's state-dir account discovery.
Kept: WriteAgentGithubToken and the external-forge path
(WriteAgentExtraForgeAccount, extra_forges.rs) are untouched, and a
declared matrixAccounts tokenFile is still read when the store has no
token for that account.
Refs #4348
The auth callout grants an agent that presents its own queue credential
one more subject, `$KV.agent-icons.<agent>`: its own key in the
agent-icons bucket and no other. The hive's shared agent client is
granted none of the bucket, since every agent on a hive presents it.
hive-agent writes `/etc/hyperhive/icon.svg`, the file its `GET /icon`
serves, to that key once per start, as a JetStream publish straight to
the subject (what `kv::Store::put` sends, minus the bucket lookup), so
the one subject is the whole grant. No icon deletes the key. A failed
write, including one that arrives before the bucket exists, is retried
with backoff until acked. An agent connected with the hive's shared
client publishes nothing.
swarm-controller creates the bucket as soon as its queue connection is
up, instead of on the first icon read, so an agent's write does not
wait for someone to look.
Measured against a local nats-server with a user allowed publish on
`$KV.agent-icons.atlas` only: the write to its own key is stored and
readable, a write to `$KV.agent-icons.argus` is refused (the ack times
out), the DEL marker makes the key read as absent, and a write before
the bucket exists fails with "no responders".
An `auth_token` spelled `swarm-agent.<agent>.<secret>` is no longer sent
to introspection. The responder reads `swarm/agents/<agent>/queue` with
an identity of its own, checks that the stored object names the same
agent, compares the secret in constant time, and grants the subjects
`--agent-token-publish-subject` lists with `{agent}` expanded. Every
other outcome denies: a malformed token, no store identity, nothing
stored, a failed or slow lookup, a different secret. A token without
the prefix takes the OIDC path unchanged.
The journal's `auth request` line names such a caller `agent:<agent>`;
the hive-shared credential keeps `hive-<h>-agent`.
The new principal: a `swarm-nats-auth` cert-auth role and policy with
read on `secret/data/swarm/agents/+/queue` alone, a leaf signed by the
store's PKI glue, and `glue-nats-auth-bao-identity.nix` pairing the two.
The copy unit delivers the identity into the queue's container, and an
absent leaf is delivered empty so the responder still starts and only
agent tokens are refused.
The policy and role are written by `swarm-bao-nats-auth-policy`, logged in
as the bao granter: both names fall under its `swarm-*` globs, so the
deploy writes them with no operator step. module-eval counts it among the
granting units, so every generic granting-unit case covers it.
The secret compare uses `subtle`, already in the lock file through the
TLS stack; no workspace crate offered one directly.
`agent-<hive>` reads as "the agent named <hive>" — which is the one thing that
identity does not carry, since it is minted per hive. It becomes
`hive-<hive>-agent`: the hive's own id, extended.
The rename is not a string swap. `hive-foo-agent` satisfies the hive parse too
(it strips to a hive named `foo-agent`), so the responder's agent rule now runs
BEFORE its hive rule — most specific wins. Hive-first would have handed every
agent its hive's grant, including writing that hive's status key, with nothing
to report it: the client authenticates and is merely able to do more than it
should.
`Policy::new`'s overlap check goes with the prefix it was written for. The
invariant the suffix form needs instead is that the suffix is non-empty: an
empty one makes `strip_suffix` succeed on every hive id, so the two principals
become one string and whichever arm runs first answers for both.
The suffix form also introduces a collision the prefix form did not have: a hive
genuinely named `foo-agent` mints `hive-foo-agent`, which is hive `foo`'s agent
id. The responder cannot see it — it has no roster, deliberately — so
`swarm-authelia.nix` asserts at eval that no hive name ends with the suffix. The
existing duplicate-id assertion does not cover this: it fires only when both
`foo` and `foo-agent` are on the roster, and with `foo-agent` alone there is no
duplicate, just a hive quietly receiving its agents' grant.
A test written by analogy with `the_prefix_alone_names_no_hive` failed, correctly
— `hive--agent` is a hive named `-agent` under the hive parse, which this module
cannot rule out. It now asserts only the part this module owns: no empty hive
name is ever expanded into a subject.
Agents have authelia *users*; they had no machine identity at all, so an
agent could not authenticate to the swarm queue as anything. This mints
one `agent-<hive>` OIDC client per hive beside the existing
`hive-<hive>` one, teaches the auth-callout responder an agent arm, and
opens the queue's client port on the bridge so a container can reach it.
One client per HIVE, not per agent: agents are created at runtime, and a
per-agent client would make creating one a config change plus an
authelia reload. The cost is that agents on a hive are indistinguishable
to the broker, which is deliberate and tracked separately.
The agent grant is deny-by-default twice over. An agent id matches no
hive rule, so it gets a hive's status-key grant from neither; and with
no agent subject configured the responder returns no grant at all rather
than an empty publish list, which would be a denial wearing a grant's
shape. What an agent may publish is a deployment's decision, taken
through `--agent-publish-subject` the same way `--hive-publish-subject`
already works.
`Policy::new` now refuses two prefixes where one contains the other. The
arms are tried in order, so that overlap does not error at match time -
it silently hands one principal the other's grant.
Not shipped here, and neither is reachable without it: no subject is
configured for agents anywhere in nix, and nothing yet delivers
`agent-<hive>.secret` into an agent container. Both belong to the stream
that will be the first consumer.
A hive's grants for its own `KV_hive-wanted-<hive>` were `STREAM.INFO` +
`DIRECT.GET`, which cover the boot-time read and nothing after it. The
convergence path now opens a KV watch on that bucket, and a watch is a
consumer, so the broker denies it — and the client's `watch()` ends in
`.ok()`, so the denial becomes `None` and the path silently never fires.
The comment four lines above the grant list already argues for this: the
per-hive bucket split exists "so that a watch can be granted without
widening the read". The design was taken and the grant was never written.
Both subject forms, matching the hive-status and agent-status blocks that
grant both for the same documented reason — an ephemeral consumer's
subject carries no name, and `>` never matches zero tokens.
Refs #4006.
mara on #4015: "not merging code without callers", and on the same PR
"see issue, we decided what the first thing should be". #3726 decided it:
the controller writes a token to the store and tells the hive; the hive
reads it back and writes /agents/<agent>/state/matrix-token-<account> at
0600, where matrix.nix's existing systemd.paths glob re-fires the daemon.
So this is the hive half of that, and the library's first caller.
The notice names a credential and never carries one, and deploy_subject's
own doc is why: the auth-callout responder scopes publish and leaves sub
unrestricted, so a hive that wanted another's messages could subscribe to
them. A secret in that payload would be readable swarm-wide. The value is
read from the store under the reading hive's own certificate, where the
store's policy is what actually scopes it.
Two boundaries guard the two addresses, and they are not the same check.
`path::matrix_account` guards the address in the store. `Ident` guards the
address on disk -- `agent_state_dir` takes one, so an unvalidated name off
the queue cannot reach a directory. I had written the first and assumed it
covered both; the compiler refused the `&str` and was right. `token_path`
now takes the newtype so a call site cannot forget.
The write is atomic because the path-watcher fires on the file appearing:
written in place it would be visible while partial, and the daemon would
read a truncated credential exactly once, which is the hardest possible
failure to reproduce. The temp name is dot-prefixed so it cannot match the
`matrix-token*` glob on its way past.
The publish grant is here because without it the failure is invisible.
policy.rs already says why for its siblings: a refused publish reaches the
client as a timeout, so the symptom is a hive that never receives a
credential with nothing in either log naming a permission. Two tests: the
controller may publish, a hive may not -- its own subject included. A
forged notice leaks nothing, but it would make a hive fetch and overwrite
a token file for a name the forger chose.
Refs #3726
A hive reads its own declaration today and that scopes cleanly: DIRECT.GET
carries the key in the subject, so the grant can name it. A *watch* cannot be
scoped that way — a consumer's filter travels in the request payload, so
$JS.API.CONSUMER.CREATE.<stream> grants the whole stream. With every hive in
one bucket, letting a hive watch its own declaration would let it read every
other hive's.
One bucket per hive (hive-wanted-<hive>) makes the stream a hive may hold
exactly as wide as what it is allowed to see, which is what #4006's live-watch
needs. That watch is a separate change; this only moves the boundary.
mara's calls, both on #4006: one stream per hive rather than teaching the auth
responder a hive roster, and a wildcard for the controller — "its okay if
swarm controller can theoretically override hive". A bucket name is a single
subject token with no prefix matching, so no wildcard narrower than * covers N
per-hive buckets; the controller's grant is account-wide by consequence, and
documented as chosen rather than left to look accidental.
The reader arm of #4005's key-layout guard asserted the opposite of that
ruling, so it is replaced rather than deleted: the hive arm survives as
no_hive_may_write_another_role_s_agent_status (with a positive control), and
the_readers_grant_is_deliberately_account_wide pins the decision and names the
ruling, so the width reads as chosen to whoever finds it next.
Two pre-existing negative assertions were silently defanged by the rename --
they matched hive-wanted.beta and $KV.hive-wanted.alpha, strings nothing
produces any more, and kept passing. Both now match current names.
swarm-controller resolves the store per hive per call instead of caching one in
a OnceCell: there is no single handle that serves N buckets, and declarations
change on operator action rather than per tick.
A KV entry publishes to $KV.<bucket>.<key> and NATS wildcards match whole
.-delimited tokens, so a /-joined {hive}/{agent} key is a single token: the
only expressible write grants are one exact subject per agent (needs a roster
in the auth responder, which Policy::hive_name argues against) or a bucket-wide
wildcard that lets any hive overwrite any other hive's agents.
Joining with a dot puts the hive in its own token, so hive_subjects can grant
$KV.agent-status.<hive>.* — every agent of one hive and nothing else, the same
containment hive-status already has.
The grant lands with the bucket-open pair (STREAM.INFO + STREAM.CREATE):
open_or_create resolves the bucket before it writes, so alone the publish
subject is unreachable and the sweep fails one step later instead.
Verified the client accepts a dotted key rather than assuming it: async-nats
0.50.0 VALID_KEY_RE is \A[-/_=.a-zA-Z0-9]+\z and is_valid_key rejects only
empty / leading / trailing dot; the subject is prefix + key verbatim.
The swarm-controller had no grant for the per-agent status bucket, so
resolving it failed and the dashboard answered 503.
Grants the reader the same five subjects the hive-status bucket already
has, for the same measured reasons -- including BOTH `CONSUMER.CREATE`
forms. `keys()` builds an ephemeral consumer whose create subject carries
no consumer name, and `>` matches one or more tokens, never zero, so the
`.>` form alone does not cover it; that omission surfaces as a client
timeout and an operator-visible 503, which is the shape reported here.
Deliberately grants NO `$KV.agent-status.*` subject, to anyone. That is
the write side, and writing is what picks a key layout -- whether agent
status is published by the agent or forwarded by its hive, and under
which key, is still open. A write grant here would answer that by
implication. Reading needs none of it: a KV read is a DIRECT.GET, and the
`.>` form is bucket-wide rather than per-key, so nothing added here
encodes a layout. A test asserts that, for the reader and for a hive,
with a control proving the same matcher fires on a bucket that does have
write grants.
The stream name is derived from the crate constant, like the wanted-state
bucket beside it, so writer and reader cannot disagree about it.
The bucket the previous commit adds had no grants, so the controller
could not create or write it and no hive could read its own key.
Measured against nats-server 2.14.4 rather than extended by analogy,
because the shapes are not symmetric:
- controller: `STREAM.INFO` + `STREAM.CREATE` on `KV_hive-wanted` and
`$KV.hive-wanted.>`. With only today's status grants, `get_key_value`
timed out and the server named the two missing stream subjects.
- hive: `STREAM.INFO` plus **one** direct-get subject carrying its own
key. A KV read is a publish — `store.get` is a request — and the
direct-get subject embeds the key, so the read scopes per hive. By
analogy with `reader_subjects` this would have been `.>`, handing
every hive every hive's wanted set.
Both refusals were verified to fire, not assumed: as `alpha`,
`get beta` was refused naming
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.beta`, and
`put alpha` was refused naming `$KV.hive-wanted.alpha`.
`a_reader_may_list_and_fetch_but_not_write` asserted a reader holds no
`$KV.` subject at all, which held only while status was the sole
bucket. Narrowed to the status bucket — the invariant it defends is
that the controller cannot forge a hive's own report, and the wanted
bucket runs the other way.
Refs #3124
`hive_subjects` granted `$JS.API.STREAM.INFO`/`CREATE` on `hive-notices`
but never the per-hive subject a publish actually targets, so every
`notices::publish` was refused with a permissions violation naming
`hive-notices.<hive>`.
The gap survived because `notices_subjects`'s own doc comment asserted
`extra_hive_subjects` covered it "since that one *is* per-hive". Nothing
in `nix/` has ever passed `--hive-publish-subject` (`git grep` → 0 hits;
control: `--reader-client` is passed at `nix/host-modules/swarm-nats.nix:711`),
so the publish had been refused for as long as the feature existed.
Subject comes from `swarm_queue_client::notices::subject`, the same
constant the publisher formats from, rather than a literal here.
Refs #3859
Per mara on the PR: *"split by hive. its not a security thing, just so
hives dont get messages they dont care about."*
She agreed with the finding and still wanted the split, which is the
part worth recording. I measured that a per-hive subject gives no
confidentiality — `sub` is unrestricted, so a hive that wanted another's
messages could subscribe to them — and concluded it bought nothing.
"Nothing" is a claim over every axis and I had checked one. The axis I
never priced: every hive in the swarm being woken by every other hive's
deploys.
So `deploy_subject(hive)` replaces the single literal, and the payload
drops `hive` to carry only the agent — the subject names the hive, and
two places stating one fact are free to disagree. The hive subscribes to
its own subject and no longer filters.
The grant is a wildcard rather than a subject per hive because the
responder has no roster: it cannot enumerate hives, and a grant that had
to track one would be a second place to get the list wrong — the same
argument `hive_name`'s doc makes about admission.
The negative test gets stronger rather than merely adapted. Splitting
the family makes "another hive's subject" and "its own" separate strings
for the first time, so it now asserts a hive reaches neither, nor the
wildcard.
The subject and its payload live in `swarm-queue-client` for the reason
the knowledge event's already does: three crates have to agree on the
string, and the one that agrees hardest — the auth-callout responder,
which decides whether the publish is permitted at all — speaks neither
`jetstream` nor `kv`.
Swarm-wide rather than a `$SWARM.deploy.<hive>` family. That family
would look like isolation and provide none: this responder scopes
publish only, leaving `sub` unrestricted, so a hive could subscribe to
another's subject as easily as to its own. Until `sub` is scoped the
split costs a wider grant and buys nothing, so the addressing goes in
the payload and each hive filters on its own name.
Unlike the knowledge event the message is addressed, so it carries a
payload — a trigger, never the config. The hive already tracks the
agent's config repo; desired state on the wire would make this a second
source of truth for something git owns, and a hive that missed a
message would be wrong rather than late.
Both test arms mirrored from the knowledge event. The negative one
matters more here: a forged knowledge event makes a hive re-read a
repo, a forged deploy event makes it rebuild and restart a named agent.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The controller could not emit an event at all: a reader's grant is
`reader_subjects()`, which is `$JS.API.*` only, so a publish to any event
subject would be refused — and a NATS refusal reaches the client as a
timeout, so the visible symptom would have been a hive that never hears
about a change, with nothing in any log naming a permission.
Adds `swarm_queue_client::events`, following `status::BUCKET`: three
crates must agree on these strings (the controller publishes, a hive
subscribes, the callout responder decides whether the publish is
permitted), and a literal repeated across crates is an agreement nothing
checks. The responder speaks neither jetstream nor kv, so the module is
unconditional and carries no NATS types, exactly as the bucket name is.
The grant takes the wildcard form from the same function the publisher
calls, so the two cannot drift; a separate wildcard constant would have
re-created the disagreement this module exists to prevent.
Tests pin that a reader gets the subject and that a hive does NOT — a
hive able to publish here could tell a neighbour the knowledge repo
changed when it had not, which is an unauthenticated write into someone
else's control path. That one asserts on the subject root rather than a
rendered subject, so a future event leaf fails it too instead of passing
because the test only knew about `knowledge`.
Both assertions mutation-tested: removing the grant fails the reader
test, granting a hive the subject fails the denial test, each on its own
assertion line, and the unmutated tree is green.
`store.keys()` creates an ephemeral ordered consumer, whose create subject
ends at the stream name. `>` matches one or more tokens and never zero, so
the `.>` form alone never covered it: the server refused
`$JS.API.CONSUMER.CREATE.KV_hive-status`, the refusal reached the client as a
timeout, and the operator saw a 503 on the hive status page.
The test asserted only the `.>` form, which reads as covering the bare one, so
the suite stayed green while every list timed out in production. It now names
the bare subject separately and first.
The module doc had grown a 37-line preamble carrying three separate
arguments, which the comment-block lint refuses. Splitting it is the
better fix than raising the limit: the reader who is about to widen a
subject list meets the reason not to at the list, not seven screens up.
Also corrects one claim I had no measurement for. The CREATE note said
creating an existing stream with a different config "is an error rather
than a rewrite" — asserted, not observed. What is observed is narrower
and enough: a hive holding this grant leaves the stream config untouched
and never publishes $JS.API.STREAM.UPDATE at all.
A grant carrying every bucket-specific subject and neither of these
cannot create the bucket at all: the client times out on `$JS.API.INFO`
long before it reaches a subject that was granted, and a NATS denial
reaches the client as a hang rather than an error.
Both were named by the server's own log, not reasoned about.
`$JS.API.INFO` is the account-level JetStream info every client requests
on connect; `$JS.API.STREAM.NAMES` is how a client finds the stream
backing a bucket. The latter lets a client enumerate stream names in the
account, which in an account holding one bucket discloses a name both
ends already share.
Every earlier measurement missed them, because each either granted
`$JS.API.>` wholesale or ran against a bucket the setup had already
created while unscoped. A minimum established against an existing bucket
is not the minimum for making one, so the leave-one-out that trimmed the
reader's set could not have found this — every candidate it tried was
tried in a world where the bucket existed.
Found by running the shipping gate against the real binary. No unit test
could have: the failure is a timeout inside a real server's permission
check.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
