hyperhive/docs/4-implementation/swarm/controller.md

Swarm controller: status, queue and forge objects

How the swarm controller takes in hive status, what a hive and an agent need to reach the queue, how it keeps an agent's AGENTS.md, and the forge objects and webhooks it owns. The operator's overview is in the swarm page.

Per-hive status (GET /api/hives/status)

One row per hive in swarm.hives, saying when it last reported and what it said. Hives publish upward through the swarm queue; the controller never reaches down to collect, so a hive that can't reach the swarm still knows its own state — you just can't see it from here.

A hive publishes only once it holds all three status-publish coordinates below. A hive without them reads never_reported — it's not broken, it just has nothing to say upward.

freshness	what to do about it
fresh	nothing — reported within staleAfterSeconds
stale	the hive stopped reporting. Its last payload is still shown, so check age_seconds and the payload for what it managed to say
never_reported	this hive has never reported at all — normally a deployment that hasn't happened, not an outage
unknown	something is publishing under a name that's not in swarm.hives — a typo in the roster, or a hive removed from the roster but still running

The snapshot is the hive's /health/ready document plus capabilities, a list of names for what the hive's release can do. Two exist: tolerant-permissions, and no-config-checkout, which turning an archetype agent into a custom one needs. A hive on an older release lists none.

The snapshot also carries egress_deny_applied, the revision of the egress deny list the hive last rendered into its proxy (0 when the swarm has none). A hive on an older release, or one that hasn't read the list yet, leaves it out; see the egress proxy.

Every row also carries last_seen_unix and age_seconds if you want to apply your own threshold. The timestamp is the one the queue recorded on arrival, not one the hive put in its own payload. A roster hive's row carries dashboard, the host its dashboard answers on; the swarm UI's hive list links there.

Set services.hyperhive.swarm.controller.staleAfterSeconds (default 120) above the rate hives publish at, or everything reads stale between reports. Hives publish once a minute, so the default tolerates one missed report and flags two. It takes effect on the next request; nothing has to re-publish.

Making a hive report

Two options on the hive, plus its store identity. They sit in two namespaces, because the queue's URL is a fact about this machine and the token endpoint is the swarm's single address:

option	what to set it to
deploy.hive-controller.statusPublish.natsUrl	tls://<swarm.nats.domain>:<swarm.nats.port>
swarm.statusPublish.tokenEndpoint	the swarm IdP's /api/oidc/token

Both default to the swarm's own addresses on every hive, so there is nothing to set for them. The client secret isn't an option: hive-c0re reads it from the swarm's secret store, at swarm/hives/<hiveName>/oidc/client, under the hive's own store leaf (deploy.bao.clientCertFile). Authelia's mint, in its container, puts it there. A hive without a store leaf doesn't publish; setting natsUrl to null turns publishing off too.

Until the secret is in the store, hive-c0re shows a swarm_queue_config warning and tries again, waiting 5 seconds after the first failure and doubling up to 5 minutes. It starts publishing as soon as a read succeeds, with no restart.

The queue URL is the same string on every hive. The queue accepts TLS only, with a certificate for swarm.nats.domain (default nats.<swarm.domain>) and no other name or address, so a URL with an IP address or nats:// fails. The queue's host resolves the name itself. A multi-host swarm needs one upstream DNS record, nats.<swarm.domain> pointing at the queue host's mesh address, the same contract as bao.<swarm.domain>. The queue's port is open on wg-hive when that host is on the mesh.

The queue's host asks the secret store for that certificate under a store leaf of its own, deploy.nats.baoClientCertFile / baoClientKeyFile. On the store's host both default to the leaf glue-bao-tls.nix mints there. A queue host without the store gets no default, even when it holds the hive's own reader deploy.bao.clientCertFile: issue a leaf whose subject is deploy.bao.natsCommonName, place it, and name it in both options. Until you do, swarm-bao-nats-tls fails naming both whenever the queue's certificate is missing or due for renewal.

The queue host's gateway also serves the queue's metrics, at https://<swarm.nats.domain>/metrics, so a swarm collector on any host scrapes them. Only the collector's authelia client gets through; any other session gets deny, and every other path on that name is 403. With the gateway's default self-signed TLS, that vhost serves the queue's own certificate, so a host whose only swarm service is the queue needs no swarm-services certificate.

The identity isn't a choice — a hive authenticates as hive-<hiveName> and publishes under hiveName, the same name that keys swarm.hives.

If a hive stops reporting, its /health/ready and the hyperhive.hive.warning metric are the place to look: a failure to publish raises a hive warning after three consecutive misses. It stays warn rather than crit on purpose — a hive that can't reach the queue isn't itself unhealthy, so it doesn't start calling itself degraded for being unable to say it's fine.

The endpoint answers 503 when this host has no swarm queue configured, or has one and can't read it — deliberately not an empty list, which would look like a silent swarm rather than a controller that can't see. The body says which. Status survives a controller restart: it's stored in the queue, not in the daemon.

Giving the agents the queue too

An agent connects to the queue as itself, with a credential swarm-controller mints for it when it creates the agent (swarm/agents/<agent>/queue in the secret store, see credentials.md). The harness reads it from the store under the agent's own certificate before each connection attempt and holds it in memory only. The one thing to configure is where the queue is:

option	what to set it to
deploy.hive-controller.queue.agentNatsUrl	where the queue listens, as an agent container reaches it

It reaches each container at start, as HIVE_AGENT_NATS_URL in /etc/hyperhive-hive-facts, so no agent build carries it. On every hive, agentNatsUrl defaults to the same tls://<swarm.nats.domain>:<swarm.nats.port> as the hive's own. The option takes one <scheme>://<host>:<port> (a hostname or IPv4 address), not an IPv6 literal or a list: evaluation refuses anything else. On the queue's host the name resolves inside a container to the bridge address, where the firewall opens the port; on any other hive it resolves through the host's DNS, like the store's name. ⚠️ Never a loopback address here: an agent has its own network namespace, so 127.0.0.1 reaches the agent.

An agent can't run without the connection. Without it the swarm can neither show the agent nor send it anything, so a harness without one stops:

• A harness started without the queue's address (HIVE_AGENT_NATS_URL) or the store's coordinates (BAO_ADDR, HIVE_AGENT_NAME and the agent's certificate) exits at boot with an error naming what's missing.
• A harness whose first connect fails tries again after 5 seconds, doubling the wait each time, and logs a warning for each attempt that fails. After the seventh, about five minutes in, it logs an error and exits. The error reads no swarm queue connection; exiting, since this agent cannot run without one, and the Grafana alert rule hh-swarm-queue-connect-failed fires on it.

Either way the harness exits and comes back until the queue accepts it. Once connected, the client reconnects by itself after a dropped connection, reading the credential from the store again for each attempt.

The hive's shared queue client is still delivered into the harness unit's credentials directory, and swarm.statusPublish.tokenEndpoint still reaches the container, but only for a swarm-logs built before it read the agent's own queue secret: the harness never connects with them, and a current swarm-logs presents that secret to the log store instead.

What an agent publishes over the queue

What an agent does with that connection is publish its terminal. Every row its own web UI renders also goes to $SWARM.term.<agent>, one subject per agent, so a swarm-level terminal can follow one agent without subscribing to the swarm's whole traffic. The queue grants that subject to the agent's own credential, and the swarm controller relays it. Publishing only: an agent talks about itself here and reads nothing. The queue keeps the rows in the stream term-main-<agent> for 24 hours or 64 MiB, the same limits and the same creator as the subagent stream below, so a swarm terminal that attaches later replays the newest of them. The two streams hold disjoint subjects.

An agent's subagents publish their terminals too, from the agent's subagent daemon: each subagent's rows go to $SWARM.term.<agent>.sub.<subagent>, classified the same way. The queue grants that family to the agent's own queue credential alone, so the daemon reads it from the store under the agent's store identity, exactly as the harness does, and publishes nothing without it. The queue keeps these rows in the stream term-sub-<agent> for 24 hours or 64 MiB, whichever comes first, dropping the oldest rows once either limit kicks in so a publish never fails because of it, and the swarm lists an agent's subagents from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for every agent a hive's wanted state names, within a minute of the agent appearing there. The agent's grant is publish on its own $SWARM.term.<agent>.sub.> and no JetStream subject, so the stream's subjects and limits are the controller's and never the agent's. Subagents publish output only and read nothing.

The queue would refuse a row too large for its max_payload outright and take the connection down with it, so the harness drops such a row's body before sending and leaves a marker in its place; the summary, level and icon still arrive. The harness logs and skips a row that's too large even without its body.

The second thing an agent publishes is its turn-state header, on $SWARM.agent-state.<agent> — same shape of subject, same grant mechanics, same lack of retention. It carries what a header bar wants: what the turn loop is doing (turn_state, plus turn_state_since as an ISO 8601 UTC stamp), which model and effort (model, effort, where each comes from as model_source and effort_source, and the resolved id the last turn actually ran on), the context budget and the last turn's context and cost token blocks, agent_state, and needs_login: true while the agent's claude has no session and its turns wait for a login. The harness sets it from its own login state, so it reaches the swarm without going through the hive.

A header also carries paused_by_hive while the harness's own pause is in force: the reason it wrote into the pause marker when it paused its turn loop. A pause by an operator (hivectl agent <name> pause, the dashboard) leaves the marker empty and reports nothing here, and so does a harness that predates the field. The controller reacts by declaring the agent paused, but only when the swarm declares it up at that moment. Without that, the hive resumes any paused agent the swarm declares up on its next convergence. It acts once per pause: the entry stays until a header without the field arrives. Setting the agent up again resumes it, and the hive clears the marker. If the swarm is on a controller that predates the field, the hive's pause lasts until its next convergence, as before.

agent_state reuses the swarm's own wanted-state vocabulary (up/offline/paused/destroyed) so a reader can compare what an agent is against what the swarm declared it should be without translating between two spellings. ⚠️ From inside the container only two of those four are sayable: the harness reports up, or paused when the pause marker is present. offline and destroyed are hive-c0re's observations — a stopped agent publishes nothing and a destroyed one doesn't exist — so a view that needs the full four-state picture takes them from the agent-status bucket and uses this subject to sharpen the rest.

Headers go out on transition: the harness rebuilds the header whenever its event bus moves and publishes when the result differs from what it last sent. That's the whole point of the subject — the agent-status bucket already republishes once a minute, which is far too slow for "is this agent thinking right now." A core subject keeps nothing for a late subscriber, so the harness also re-sends the current header once a minute: a subscriber attaching mid-idle sees it within that minute. The swarm's stream sends a new client the latest header it holds first, when that's under 150 seconds old. A renderer opens with the bucket's snapshot and lets this stream refine it.

Swarm-side, GET /api/agents/<name>/state/stream relays that subject as SSE, resolving the agent's hive at request time exactly as the terminal stream does. The payload passes through opaquely. Separately, the controller keeps each agent's latest model, effort and their sources for GET /api/agents/<name>/inference (inference.md), and each agent's latest needs_login for the needs_login field of GET /api/agents/status. That field is null until a header carrying the flag arrives, so for up to a minute after the controller starts, and for an agent whose harness doesn't send it. The row's paused_by_hive comes from the same headers and is null in the same cases.

The third thing an agent publishes is its icon, the same SVG its own GET /icon serves. It goes into the agent-icons KV bucket under the key <agent>, with no hive in it, so the swarm can show the icon of an agent that's stopped or has moved hives. The queue grants the agent's own credential $KV.agent-icons.<agent> and no other key of that bucket. The harness writes once per start, because a config change reaches an agent by restarting its container. An agent with no icon deletes its key. Swarm-side, GET /api/agents/<name>/icon serves the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards load it as an <img> and show the dimmed hyperhive mark for an agent without one, as the hive dashboard does.

The fourth is its AGENTS.md, which goes into the agents-md bucket under the key <agent>, writable by the agent's own credential alone ($KV.agents-md.<agent>). The swarm reads that copy and edits the file through the agent; An agent's AGENTS.md has the details.

The queue also limits what an agent may subscribe to, because agents are the least trusted clients on the queue. An agent's own queue credential, which its subagent daemon also presents, may subscribe to its reply inbox _INBOX.agent.<agent>.>, its operator-input subject $SWARM.input.<agent> and its command subject $SWARM.control.<agent>, and nothing else. Besides the subjects it publishes to, the credential may answer the one request it was just sent, which is how a command gets its reply. The harness and the subagent daemon connect with _INBOX.agent.<agent> as their inbox prefix, so the replies to the icon write and the schedule read arrive. An agent can't read another agent's terminal, state, input, commands or replies.

Hives and the swarm controller get a subscribe allow-list too, each with its own reply inbox _INBOX.client.<client-id>.>, which swarm_queue_client::connect sets as their inbox prefix. A KV watch or key listing delivers there as well. A hive may also subscribe to $SWARM.knowledge and its own $SWARM.deploy.<hive>, so it can't read another hive's deploys or any agent's terminal. The controller may also subscribe to $SWARM.term.<agent>, $SWARM.term.<agent>.sub.<subagent> and $SWARM.agent-state.<agent>, the streams it relays to swarm-ui. A hive or controller that connects with the library's default _INBOX.<nuid> prefix gets no replies, so every request it makes times out.

An agent's AGENTS.md

The file in the agent's state dir, state/AGENTS.md, is the only copy that counts, and only the agent writes it: itself, or its harness applying an edit from the swarm. No hive is in either path.

Reading. The harness publishes the file's text and its revision, the git blob id that git hash-object prints for it, at start, within ten seconds of any change, and straight after an edit it applied. An agent with no AGENTS.md publishes "". GET /api/agents/<name>/agents-md answers {content, revision} from that copy, so it works while the agent isn't running. A 404 means the agent hasn't published one: its harness hasn't run since it began to.

Initial text. POST /api/agents and swarmctl agent create --agents-md <file> take an optional agents_md, the text the file starts with, up to 64 KiB; a longer text is a 400. The controller stores it under the key <agent> of the agents-md-initial bucket, which only it writes, before it queues the creation. At start the harness reads its own key, with one direct get on $JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>, and writes the text to AGENTS.md only when no file exists there. It does this before the first mirror publish and the first turn. An existing file is never replaced, so a restart or a re-create keeps the agent's edits; a later change to the file never reaches the key. Destroying the agent deletes the key. When the harness can't read the key it starts without the text and logs a warning. The text holds no secret, so nothing goes through the secret store.

Editing. PUT /api/agents/<name>/agents-md takes {base_revision, diff}. The diff is a unified diff that names a/AGENTS.md and b/AGENTS.md, made against the text at base_revision. swarm-controller sends it as a request on the agent's command subject $SWARM.control.<name>, as it sends a cancel. The harness refuses it when the file isn't at base_revision. Otherwise it runs git apply --check and then git apply on a copy of the file, and writes the result over the file only when both pass:

Status	Meaning
200	applied; the body is {revision}, the file's new revision
409	the file isn't at base_revision any more; nothing applied, and revision in the body is the one it's at
422	git apply refused the diff, or it touches a file other than AGENTS.md; detail carries git's message, and the file stays as it was
503	the agent isn't running or isn't connected to the queue
504	the harness didn't answer within ten seconds, so whether the edit applied is unknown
500	the harness couldn't read or write the file or run git, or the queue failed the request

The harness refuses a diff that sets any file mode other than 100644, so an edit can't turn AGENTS.md into a symlink. PUT is operator-only, behind Remote-User like cancel: whoever can call it can rewrite the agent's instructions, so don't widen who holds it without revisiting this.

An edit needs the agent running, and a read of a stopped agent shows the text it last published. A harness older than this route doesn't know the command and drops it unanswered, which reads as a 504. swarmctl agent agents-md get and apply call the two routes.

Swarm-wide forge objects

The controller also keeps the forge objects that are one per swarm, not one per hive. It ensures them at start and every five minutes after (swarm-controller/src/forge/objects.rs):

• the orgs agent-configs, internal and agents, plus each mirror's owner org;
• the empty operators merge-gate team in agents and agent-configs;
• the main merge gate on every agent-configs repo: merge and approval whitelists = the operators team, and no user. The controller leaves a repo with no main rule alone;
• the pull-mirrors from deploy.forgejo.mirrors on the controller's host (with the actions/checkout and actions/upload-artifact ones deploy.forgejo.actionsFromSelf adds, and hyperhive/hyperhive from deploy.forgejo.hyperhiveUpstream);
• internal/docs (private) and internal/knowledge (public, with a seed README.md while empty);
• the agent-configs org avatar (deploy.swarm-controller.configOrgAvatarPng).

A pass that can't finish logs a warn line per object plus swarm forge objects: pass incomplete in journalctl -u swarm-controller, and retries on the next tick. While the controller is down the objects stay as they are.

The controller gives each agent read on internal/docs. A GrantDocsRead node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the store's hive-agent-* roster (forge/docs_grant.rs, at start and every five minutes) adds any agent that isn't one, so an agent whose create-time grant failed gets it within five minutes. The pass only adds: a roster it can't read, a forge read that fails, or an internal/docs that doesn't exist (checked once per pass, logged at warn) changes nothing. The grant doesn't depend on the hive the agent runs on, or on a forge container there.

Swarm-wide forge webhooks

At startup the controller registers its Forgejo hooks pointing at itself, all under https://<swarm.domain>/webhook/forge/:

• a push hook on internal/knowledge
• a pull_request hook on the agent-configs org
• an instance-wide push system hook, for commit metrics
• an instance-wide pull_request_review_request system hook, which starts the review classifier
• instance-wide issues + issue_assign and pull_request + pull_request_assign + pull_request_sync + pull_request_review_approved + pull_request_review_rejected + pull_request_review_comment system hooks, and an instance-wide workflow_run_failure + workflow_run_cancelled system hook, whose deliveries it keeps in the SWARM_EVENTS stream for the notification router

The instance-wide push hook also starts a check of the open PRs targeting the pushed branch (the default branch, or a branch the controller has seen a PR target), which publishes a pull_request_mergeability event for each to the same stream: merge conflicts as a source.

The controller interprets a delivery and sends hives a specific message — the knowledge repo changed, deploy agent foo at rev abc123 — rather than forwarding forge payloads for each hive to re-derive. Approval happens once, at the swarm level: a hive receives a decision, not an event to adjudicate. The issues and pull_request deliveries are the exception: no hive gets them, and the notification router, not the controller, decides what they mean.

A config-pr delivery reporting a PR merged into main queues a deploy of its merge_commit_sha on the one hive whose wanted state places the agent; with no such hive, or more than one, the controller deploys nothing. See config-changes.md § Config changes. A merge whose closed delivery never arrives deploys nothing either; the operator recovers by redelivering that delivery from the forge hook page, which re-enters the same handler and re-queues the deploy.

internal/knowledge is on that path. The controller's is the only hook on it (knowledge.md covers clearing a leftover). A webhook has exactly one target URL, so a second registration would take delivery away from the first rather than add a recipient.

The agent-configs org is on it too. The controller's hook is the only one that acts on config PRs. No hive serves /webhook/config-pr any more and none deletes it: a hook of that name left on the org by an older release stays registered and collects failed deliveries until an operator deletes it by hand. Removing the controller's hook stops forge-UI merges from deploying until its next start recreates it.

Nothing to configure. The controller registers the hooks only when this host also serves the swarm UI vhost — that's what publishes the endpoint, and a hook the forge can't reach would collect failed deliveries while looking healthy. The controller generates the HMAC secret on first start and keeps it (see docs/4-implementation/agent-state/persistence.md).

To check it's working, push to internal/knowledge and look for webhook: verified delivery in journalctl -u swarm-controller. A refused delivery logs webhook: refused delivery with the reason.

The controller registers the hooks within five minutes of having both the forge token and the secret. It keeps trying, at most five minutes apart, for as long as the forge or the secret store is down. A restart isn't needed to recover.