hyperhive/swarm-controller/README.md

# swarm-controller

The **swarm-level** daemon. Where `hive-c0re` owns the agents on one host, this
owns what is true _across_ hives — so a swarm runs one of them and most hives
leave it off.

Opt-in per host via `services.hyperhive.deploy.swarm-controller.enable`, which is
deliberately **not** derived from
`services.hyperhive.deploy.hive-controller.enable`: turning it on is a statement
about swarm topology, not about whether this host runs a hive.

## What it does

The swarm's control plane. The swarm UI and `swarmctl` are its clients.

- **Hive directory** — serves `swarm.hives` (`GET /api/hives`) and what each
  hive last published about itself (`GET /api/hives/status`).
- **Agent roster and wanted state** — every agent the swarm knows
  (`GET /api/agents`, `/api/agents/status`), and the state it declares for
  each one on its hive (`up`/`offline`/`paused`/`destroyed`,
  `PUT /api/hives/{hive}/agents/{agent}/state`). An agent that has never
  reported takes its hive and declared state in `/api/agents/status` from the
  one hive whose declaration places it.
- **Job graph** — a `hive-jobq` scheduler, served at `GET /api/jobq/graph`.
  Every provisioning step below runs as a node in it.
- **Agent creation** — `POST /api/agents` queues the SSO identity (through
  `swarm-authelia-bridge`), forge user, config repo, store identity, forge
  token and matrix account, declares the agent `paused`, then sends its hive
  a deploy message.
- **Agent credentials** — at start and every five minutes it re-checks every
  agent's forge token and matrix account, and renews store certificates and
  queue secrets as they age.
- **Linked external accounts** — an operator-supplied matrix, forge or
  GitHub account for one agent, stored in the swarm secret store
  (`PUT /api/hives/{hive}/agents/{agent}/matrix-accounts/{account}`,
  `.../forge-accounts/{label}`, `.../github-account`); distinct from the agent's own swarm-minted
  accounts above. `GET .../linked-accounts` lists them, plus the agent's own
  `main` matrix account, by kind, name and host, never with a credential.
  Each of the three also takes a `DELETE`; matrix's also takes
  `?revoke=true`, which logs the stored token out at its homeserver first and
  leaves the account in place if that fails.
- **Config PR status** — each agent's open config-repo PR, cached from forge
  webhooks (`GET /api/config-prs`, `/api/agents/{name}/config-pr`).
- **Swarm-wide forge objects and webhooks** →
  [`docs/swarm/README.md`](../docs/swarm/README.md#swarm-wide-forge-objects).
- **Scheduled prompts** — named prompts for one or more agents, kept in
  the `schedule-definitions` KV bucket and written into each agent's key of
  `agent-schedules`, both of which this daemon alone writes
  (`GET`/`POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`); each
  agent fires them itself → [`docs/swarm/schedules.md`](../docs/swarm/schedules.md).
- **Inference settings** — each agent's model, effort, subagent models and
  cap on concurrently running subagents, one key per agent in the
  `agent-inference` KV bucket, which this daemon
  alone writes (`GET`/`PUT`/`DELETE /api/agents/{name}/inference`); each
  agent applies them itself. `GET` also answers the model and effort the
  agent last reported in its turn-state header, kept in memory →
  [`docs/swarm/inference.md`](../docs/swarm/inference.md).
- **Resource caps** — each agent's CPU and memory cap, one key per agent in
  the `agent-limits` KV bucket, which this daemon alone writes
  (`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the
  agent reads it, and each cap replaces the hive's own answer for that field →
  [`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits).
- **Agent instructions** — each agent's `AGENTS.md` as its harness last
  published it into the `agents-md` KV bucket
  (`GET /api/agents/{name}/agents-md`), and edits to it as git diffs, which
  this daemon hands to the agent's harness to apply
  (`PUT /api/agents/{name}/agents-md`). `POST /api/agents` can set the text the
  file starts with (`agents_md`), stored in the `agents-md-initial` bucket for
  the harness to write when no file exists →
  [`docs/swarm/README.md`](../docs/swarm/README.md#an-agents-agentsmd).
- **Review classifier** — try-outs from the swarm UI and reviews on review
  requests, and the newest 100 finished runs, one document in the
  `review-classifier-runs` KV bucket, which this daemon alone writes
  (`POST /api/review-classifier/try-out`, `GET /api/review-classifier/try-out/{id}`,
  `GET /api/review-classifier/rules`, `GET /api/review-classifier/runs`), and
  the try-out page's branch for a PR, pushed as the classifier's forge account
  (`POST /api/review-classifier/open-pr`) →
  [`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md).
- **Review classifier settings** — the classifier's runtime, model, effort and how
  many requested reviews run at once, one document in the
  `review-classifier-settings` KV bucket, which this daemon alone reads and
  writes (`GET`/`PUT /api/review-classifier/settings`) →
  [`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings).
- **Operator profiles** — each operator's name and the pronouns on their
  forge profile, for every member of authelia's operator group, as one list
  under the one key of the `operator-profiles` KV bucket. This daemon alone
  writes it, at start and every five minutes, and only when the list changed.
  A forge read that fails keeps that operator's stored entry, and a name that
  leaves the group drops out of the list. Every agent reads the key into its
  system prompt →
  [`docs/turn-loop/claude-invocation.md`](../docs/turn-loop/claude-invocation.md).
- **Builder runs** — the swarm builder's newest 100 finished builds of agent
  config PRs, with `nix`'s tail for a failed one, one document in the
  `builder-runs` KV bucket, which this daemon alone writes
  (`GET /api/builder/runs`, `GET /api/builder/runs/{id}`, the link on each
  `swarm-builder` commit status) →
  [`docs/swarm/builder.md`](../docs/swarm/builder.md#build-records).
- **Relays** — each agent's terminal and turn-state header as SSE, agent
  icons, the cross-repo issue report, and the UI's quick links.
- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read
  at request time with a Viewer service-account token from the secret store
  → [`docs/swarm/secrets.md`](../docs/swarm/secrets.md).

It reads its configuration once at startup, from the environment the nix module
sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its
state directory.
The job graph lives in memory; hive status and wanted state live in the swarm
queue, so both survive a restart.

## Why a unix socket, not a port

The hive-gateway's nginx is the only intended client (`swarmctl` runs as root).
A listener that is never bound to an address cannot be
reached from off-host by mistake.

The socket path is `services.hyperhive.deploy.swarm-controller.socketPath`, default
`/run/swarm-controller/controller.sock`, exported to the process as
`SWARM_CONTROLLER_SOCKET`.

## ⚠️ The socket's directory is its access control

The socket is `0666`. It has to be: nginx runs as a different user and
`connect(2)` needs write. This matches how `hive-c0re` publishes the per-agent
sockets, and rests on the same argument — _"the bind source dir is per-agent on
host so blast radius is unchanged."_

What keeps that safe is that the directory is `0750`, owned by the
`swarm-controller-clients` group, and holds **one** socket. nginx is in that
group through its unit's `SupplementaryGroups`; root bypasses mode bits, so
`swarmctl` needs no membership. Every other local user gets `EACCES` at the
directory. The group is not `swarm-controller`'s own, which also owns the
state directory. So:

> **Never point `socketPath` at a directory that carries anything else.**
> `/run/hyperhive` above all — it holds `host.sock`, the host **admin** socket.
> Pointing nginx at that directory to reach this socket would put the admin
> socket within its reach too.

nginx is a host service, so nothing narrows what it can reach except the
directory itself — that is the whole of the access control. The unit sets the
group with a root `ExecStartPre` (`chgrp`), because systemd chowns the runtime
directory to the unit's own group on every start. A unit test pins
the default path so a tidying edit fails instead of reviewing cleanly.

`RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are
a **pair**: preserving the directory without the unlink means `bind` fails with
`EADDRINUSE` after a restart.

## Packaging

Built by the workspace derivation and extracted as its own package
(`nix build .#swarm-controller`). Deliberately **not** in `nix/packages`'
`daemonBins` — that list is the core stack and drives the bundle
`services.hyperhive.c0re.package` points at, so folding this in would put a
swarm-scoped service into every hive's closure.