vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-agent: the swarm queue connection is required; retry, then exit

Browse source 
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.

Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.

The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.

The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.

With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.

Closes #4909
This commit is contained in:
atlas 2026-10-04 19:05:48 +02:00 • committed by mara
commit 235bc481b3
16 changed files with 392 additions and 964 deletions
Download patch file Download diff file Expand all files Collapse all files

84
docs/swarm/README.md
View file

@ -131,14 +131,15 @@ it's stored in the queue, not in the daemon.
### Giving the agents the queue too
An agent authenticates as its own client, not as its hive, so it needs its
own coordinates. Two of them are options on the hive; the other two arrive
with the credential itself and aren't configurable.
An agent connects to the queue as itself, with a credential `swarm-controller`
mints for it when it creates the agent (`swarm/agents/<agent>/queue` in the
secret store, see [`credentials.md`](credentials.md)). The harness reads it from
the store under the agent's own certificate before each connection attempt and
holds it in memory only. The one thing to configure is where the queue is:
| option | what to set it to |
| ------------------------------------------- | -------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` — the same one the hive uses |
| option | what to set it to |
| ------------------------------------------- | ------------------------------------------------------------- |
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
On every hive, `agentNatsUrl` defaults to the same
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. On the queue's
@ -147,26 +148,33 @@ firewall opens the port; on any other hive it resolves through the host's DNS,
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
own network namespace, so `127.0.0.1` reaches the agent.
The harness sees four variables, and treats them as all-or-none:
`HIVE_AGENT_NATS_URL` and `HIVE_AGENT_OIDC_TOKEN_ENDPOINT` from the two options
above, plus `HIVE_AGENT_OIDC_CLIENT_SECRET_FILE` and
`HIVE_AGENT_OIDC_CLIENT_ID_FILE`, which point into the unit's own credentials
directory. The last two come from the delivered credential rather than from
config — see [`secrets.md`](secrets.md#hive-level--one-of-each-per-hive) for
how it gets there. A hive lacking the queue's address for its
agents sets none of the four and each agent logs that it has none; a half-set
environment logs an error and the harness keeps serving.
**An agent can't run without the connection.** Without it the swarm can neither
show the agent nor send it anything, so a harness without one stops:
- A harness started without the queue's address (`HIVE_AGENT_NATS_URL`) or the
store's coordinates (`BAO_ADDR`, `HIVE_AGENT_NAME` and the agent's
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits.
Either way the `hive-agent` unit fails, and systemd starts it again 30 seconds
later, with no start limit, until the queue accepts it. Once connected, the
client reconnects by itself after a dropped connection, reading the credential
from the store again for each attempt.
The hive's shared queue client is still delivered into the harness unit's
credentials directory, and `swarm.statusPublish.tokenEndpoint` still reaches
the container, but only for `swarm-logs`: the harness never connects with
them.
<details><summary>What an agent publishes over the queue</summary>
What an agent does with that connection is publish its terminal. Every row its
own web UI renders also goes to `$SWARM.term.<agent>`, one subject per agent, so
a swarm-level terminal can follow one agent without subscribing to the swarm's
whole traffic. An agent connected with its own queue credential gets that
subject. An agent without one, or whose own credential the queue
refused, connects with its hive's shared client and publishes to
`$SWARM.term.<hive>.<agent>` instead, the `<hive>` being the one that client id
names. The swarm controller relays both. Publishing only: an
whole traffic. The queue grants that subject to the agent's own credential,
and the swarm controller relays it. Publishing only: an
agent talks about itself here and reads nothing. Rows aren't retained — a
subscriber that wasn't listening missed them, the same as on the agent's own
live stream.
@ -192,11 +200,11 @@ sending and leaves a marker in its place; the summary, level and icon still
arrive. The harness logs and skips a row that's too large even without its body.
The second thing an agent publishes is its **turn-state header**, on
`$SWARM.agent-state.<agent>` (or `$SWARM.agent-state.<hive>.<agent>`) — same shape of subject, same grant
mechanics, same lack of retention. It carries what a header bar wants: what the
turn loop is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC
stamp), which model (`model` and the resolved id the last turn actually ran on),
the context budget and the last turn's context and cost token blocks, and
`$SWARM.agent-state.<agent>` — same shape of subject, same grant mechanics,
same lack of retention. It carries what a header bar wants: what the turn loop
is doing (`turn_state`, plus `turn_state_since` as an ISO 8601 UTC stamp),
which model (`model` and the resolved id the last turn actually ran on), the
context budget and the last turn's context and cost token blocks, and
`agent_state`.
`agent_state` reuses the swarm's own wanted-state vocabulary
@ -224,24 +232,24 @@ The payload passes through opaquely — the controller never parses a header.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
`<agent>`, with no hive in it, so the swarm can show the icon of an agent that's
stopped or has moved hives. Only an agent connected with its own queue
credential publishes it: the queue grants that credential
`$KV.agent-icons.<agent>` and no other key, and grants the hive's shared client
none of the bucket. The harness writes once per start, because a config change
reaches an agent by restarting its container. An agent with no icon deletes its
key. Swarm-side, `GET /api/agents/<name>/icon` serves the stored bytes, and 404
means the agent has no icon. swarm-ui's agent cards load it as an `<img>` and show the
dimmed hyperhive mark for an agent without one, as the hive dashboard does.
stopped or has moved hives. The queue grants the agent's own credential
`$KV.agent-icons.<agent>` and no other key. The harness writes once per start,
because a config change reaches an agent by restarting its container. An agent
with no icon deletes its key. Swarm-side, `GET /api/agents/<name>/icon` serves
the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards
load it as an `<img>` and show the dimmed hyperhive mark for an agent without
one, as the hive dashboard does.
The queue also limits what an agent may **subscribe** to, because agents are the
least trusted clients on the queue. An agent's own queue credential, which its
subagent daemon also presents, may subscribe to its reply inbox
`_INBOX.agent.<agent>.>` and its operator-input subject `$SWARM.input.<agent>`,
and nothing else. The hive's shared agent client may subscribe to its reply
inbox `_INBOX.client.<client-id>.>` only. Each connects with its prefix as its
inbox prefix, so the replies to the icon write and the schedule read arrive.
Neither can read another agent's terminal, state, input or replies. Hives and
the swarm controller may still subscribe to any subject in the account.
inbox `_INBOX.client.<client-id>.>` only. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to
the icon write and the schedule read arrive. Neither credential can read
another agent's terminal, state, input or replies. Hives and the swarm
controller may still subscribe to any subject in the account.
</details>