per #5471, verifiers stop depending on the queue client. the module is
store-free and needs only thiserror, so swarm-agent-verify gains one
dependency and loses its swarm-queue-client edge. consumers import it
from swarm_agent_verify; swarm-logs and swarm-metrics, whose only use of
swarm-queue-client was this module, drop that dependency, as does
swarm-logs-auth.
hh-swarm-queue-connect-failed matched "swarm queue connect failed; this
agent publishes nothing upward", which nothing has logged since the
harness started retrying its first connect and exiting (235bc481b). The
rule now matches the error the harness logs when every attempt has
failed, CONNECT_EXHAUSTED in swarm_queue.rs, and its comment and summary
describe the retry-then-exit behaviour.
A failed MintAgentIdentity node logged nothing: the error reached only
the job graph. mint_identity now warns with the agent and the error, and
hh-agent-identity-mint-failed fires on that line.
Refs #4651
A new swarm service, `swarm-claude-login`, in its own container on the host
that sets `deploy.claude-login.enable` (defaulted on by `allSwarmServices`
where the host has the minter's store leaf, like the review classifier).
- swarm-ui: an agent card showing "login needed" gets a "log in" button.
Its dialog starts the login, shows the URL claude printed, and takes the
pasted code.
- swarm-controller: `POST/DELETE /api/agents/{name}/claude-login` and
`POST /api/agents/{name}/claude-login/code`, behind the swarm UI's
authelia-gated `/api/`, relay the steps to the minter as requests on
`$SWARM.claude-login.request`.
- swarm-claude-login: runs `claude auth login` over piped stdio with HOME
and CLAUDE_CONFIG_DIR in a per-login scratch dir, scrapes the first
https URL, writes the code to stdin, and stores the `.credentials.json`
at `swarm/agents/<agent>/claude-credentials` with a KV v2 write, a new
version over any earlier one. The scratch dir goes on every exit path; a
login waits 15 minutes for its code.
- hive-agent: while parked in `wait_for_login`, reads that leaf every 30 s
under the agent's own cert and installs it only when its KV version is
newer than the one recorded in `<harness>/claude-credentials-version`.
An equal or older version is never put back, because claude rotates the
refresh token and the stored copy is stale after the first refresh.
- swarm-secret-client: `claude_login` paths and `SecretStore::read_versioned`.
- Credentials: a `claude-login` store leaf minted by glue-bao-tls, a
`swarm-claude-login` policy (create/update on
`swarm/agents/+/claude-credentials`, read on its own queue client
secret), an authelia machine client, and a `--claude-login-client`
grant in the queue's auth-callout responder. The unit runs as a static
system user.
The hive's own web-UI login is unchanged.
Refs #5268
The docs, option descriptions and comments that named the swarm root, the
swarm CA or the hive CA as an existing trust anchor now name the secret
store's pki root or the trust bundle that holds it:
- docs/swarm/README.md, secrets.md, bao.md
- deploy.bao.clientCaFile's and gateway.verifiedProxyTo's descriptions
- comments in hive-c0re/src/meta.rs, hive-agent/src/swarm_queue.rs,
swarm-logs/src/auth.rs, swarm-controller/src/vcs_metrics.rs,
hive-forge, swarm-authelia, swarm-ci-runner, glue-bao-tls, swarm-bao,
swarm-grafana, swarm-otel, otel, hive-c0re, lib/hive-ca-trust.nix,
hive-forge/Cargo.toml and the flake-update workflow
Nothing behaves differently: every edit is a nix, Rust or TOML comment, an
option description, or a test assertion message. The Rust edits change the
workspace source, which this PR already rebuilds for hivectl.
Refs #4588
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.
Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.
The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.
The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.
With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.
Closes#4909
An agent's queue credential could subscribe to every subject in the swarm
account, so any agent could read every other agent's terminal, turn state,
operator input and request replies. Agents are the least trusted clients
on the queue (mara's ruling on #4901), so their grants now carry a
subscribe allow-list:
- an agent's own credential (the harness and its subagent daemon):
`_INBOX.agent.<agent>.>` and `$SWARM.input.<agent>`
- the hive's shared agent client: `_INBOX.client.<client-id>.>`
Hive and swarm-controller grants are unchanged (no `sub` claim, so still
every subject).
The clients connect with the matching inbox prefix, so the replies they
rely on still arrive: the icon write's JetStream ack and the schedule
DIRECT.GET. `connect_with_token` takes the agent name and sets the prefix
itself; the hive-shared fallback uses the new
`connect_with_inbox_prefix`.
The harness no longer subscribes to its input subject when it fell back
to the hive's shared client: that identity names no agent, so it cannot
be granted one agent's input without all of them. It logs that and
skips, the same as the schedule reader on that path.
Refs #4901
The harness now reads swarm/agents/<agent>/queue from the store itself,
under the agent's own store certificate, and holds it in memory only.
It reads once before the first connect and again on every reconnect
attempt (async-nats `ConnectOptions::with_auth_callback`), so an agent
whose secret was re-minted reconnects with the new value instead of
being refused until the container restarts.
hive-agent-queue-credential.service, the /run file it wrote, and
HIVE_AGENT_QUEUE_AGENT_SECRET_FILE are gone; queue-identity.nix now
hands hive-agent.service the store address, its certificate paths and
the agent name.
A failed or empty read before the first connect still falls back to
the hive's shared client. Each read is bounded by a 10s timeout, and
retries wait out the existing reconnect backoff (500ms doubling, capped
at 60s).
Closes#4783
The controller's OIDC client secret (client `swarm-controller`, used for
the queue connection, the auth-bridge bearer and the OTLP push) came from
an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`,
handed in by `LoadCredential=`.
Now `swarm-secret-publish`, which already copies authelia's minted OIDC
secrets into the store, also publishes this one, to
`swarm/controller/swarm-controller/oidc/client`. That path sits under
`controller/`, which no hive's policy reads. The controller reads it once
at start with its existing store certificate and holds it in memory, as
`swarm_queue_client::ClientSecret::Value`. If the store is down, it
retries for about a minute and then fails the start, so `Restart=` tries
again.
Policy delta: the controller gets `read` on that leaf, and the publisher
gets `create`/`update` on that leaf.
Removed: the `queue.clientSecretFile` option (both spellings, now removed
options with a message), its singleHostSwarm default, the credential and
placeholder, and the path watcher plus its restart oneshot. A controller
without a store identity is now an eval error, because it has no other
way to get the secret.
When the per-agent secret `queue-identity.nix` fetched is present, the
harness connects with `swarm-agent.<agent>.<secret>` as a static token
and publishes on `$SWARM.term.<agent>` and `$SWARM.agent-state.<agent>`.
When it is absent, or that first connect fails for any reason, a refusal
from a responder that does not verify agent tokens included, it connects
with the hive's shared OIDC client and publishes on the hive-scoped
subjects as before. Which one it took is logged once per connect.
`swarm_queue_client::connect_with_token` is the static-token connect: no
retry on the initial attempt, so the caller sees the refusal and can
fall back. Reconnects share the existing backoff, now a named function.
Closes#4630
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped OIDC client, so at the auth callout one agent is
indistinguishable from its co-hived neighbours. The commit before this
one mints a secret per agent at swarm level into
secret/swarm/agents/<agent>/queue; nothing read it.
Read it here, and read it from the container itself. A hive courier in
the path would be the hive vouching for which agent this is, which is
the property a per-agent credential exists to remove -- so the agent
logs in to the store with the certificate hive-agent-bao-identity
already proves it can log in with, and reads its own path. The store
certificate is for reaching the store and nothing else: what the new
unit writes to /run is the secret it read back, and nothing hands a
BAO_CLIENT_* path to anything queue-shaped.
The read needs no policy change. render_agent grants read on
secret/data/swarm/agents/<agent>/*, which covers this path and the
bao-mtls one beside it alike -- which is also why this unit degrades
where the identity check fails. A refusal this unit sees and that check
did not cannot be a policy that drifted; it is an object not yet minted,
the ordinary state of every agent created before its swarm knew to mint
one.
The harness resolves the path and reports which credential this agent
can present. It does not yet present it: the auth-callout responder
still verifies only the hive-scoped token, and an agent offering a
credential nothing on the other end reads back would simply be refused.
Teaching swarm-nats-auth to read the same path is the next slice.
The harness has had its queue coordinates since the credential reached
the container, but nothing used them. This offers each terminal row
upward on `$SWARM.term.<hive>.<agent>`, so a swarm-level terminal can
render an agent without reaching into the hive that hosts it.
It publishes the same `TermMsg` the web UI is handed rather than a
second model of the same events, so a new tool or a reclassified event
changes both surfaces together. It subscribes to the event bus rather
than to the SSE handler: the handler classifies per connected browser,
so hanging this off it would mean an agent nobody is watching publishes
nothing. That also means its own long-lived `ClassifyCtx`, since a
publisher restarting its correlation state would lose the `tool_use` →
name mapping a `tool_result` needs to render.
The hive in the subject is derived from the queue client id, not from
the harness's hive display name. Those come from different sources with
no rule tying them together, and the responder builds its grant from the
client id — so deriving it from the display name yields a publish the
broker refuses, reaching an operator as a terminal that is merely empty.
The prefix and suffix that bracket the hive are the responder's flags,
which the agent is not told; it restates their defaults, and the symptom
of a deployment retuning one without changing this is every publish
refused rather than a wrong subject accepted.
Oversize rows degrade in the publisher. Exceeding `max_payload` is not a
truncation: the server refuses the message and closes the connection, so
an oversize publish costs the row, the connection, and the rows racing
behind it through the reconnect. The body is the only unbounded field —
summaries are already trimmed at classification — so it is the field
spent, and the row keeps its icon, level, summary and coalesce key. A
row that does not fit even then is logged and dropped rather than sent.
The limit is read off the connection, so `8388608` stays spelled once in
the queue's own module; size is measured by serializing, because JSON
escaping separates character count from wire length by an unbounded
factor on exactly the rows already near the limit.
Best-effort throughout: no queue, an unparseable client id and a failed
connect each disable the publisher with one log line, and a failed
publish loses its row and nothing else. The turn loop and the web UI
never block on the queue.
Refs #3805
The swarm always has exactly one queue; a hive can only lack its
address. Reworded every prose site this PR added that stated or
implied the opposite, to name what is actually absent (coordinates,
credential, or address) instead of the queue itself.
Refs #3805
hive-c0re stats the two files `swarm-bao-queue-agent` lands and forwards
them into every agent container as systemd credentials, and the harness
resolves a `QueueConfig` out of them at boot. Nothing connects yet.
A credential and not a bind mount, and the mode is what forces it: the
secret is root:0600 and the harness runs as the unprivileged agent user,
so a bind would deliver a file that user cannot open. nspawn's
`--load-credential` is read by the container manager as root and
re-exposed under the consuming unit's own `User=`. hive-c0re never reads
the bytes either way, which is just as well — it runs as `hive-core`.
Absent files stay legal and become visible rather than silent: the
publisher lives on the authelia host and mints on its first boot, so
"nothing at that path" is the ordinary early state of a swarm. c0re
forwards nothing and logs why; the harness logs that it has no queue.
The client id comes out of the delivered file rather than being rebuilt
from `hiveName` in nix, which is the agreement `swarm-secret-client`
states. `QueueConfig::from_env` wants it as a value, so the harness reads
the file itself — assigning the variable instead would need
`std::env::set_var` in a process that has already spawned threads.
Refs #3805
