49015eb553
The agent read the operator-profiles bucket with a keys() listing, an ordered consumer whose grant (STREAM.INFO, CONSUMER.CREATE) and flow-control reply had never run against a permissioned queue. The controller now writes the whole list as one JSON array under one key, only when it changed, and the agent reads it with one DIRECT.GET the way agent_inference::read_own reads its key. The agent grant shrinks to that one subject.
132 lines
7.4 KiB
Markdown
132 lines
7.4 KiB
Markdown
# swarm-controller
|
|
|
|
The **swarm-level** daemon. Where `hive-c0re` owns the agents on one host, this
|
|
owns what is true _across_ hives — so a swarm runs one of them and most hives
|
|
leave it off.
|
|
|
|
Opt-in per host via `services.hyperhive.deploy.swarm-controller.enable`, which is
|
|
deliberately **not** derived from
|
|
`services.hyperhive.deploy.hive-controller.enable`: turning it on is a statement
|
|
about swarm topology, not about whether this host runs a hive.
|
|
|
|
## What it does
|
|
|
|
The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|
|
|
- **Hive directory** — serves `swarm.hives` (`GET /api/hives`) and what each
|
|
hive last published about itself (`GET /api/hives/status`).
|
|
- **Agent roster and wanted state** — every agent the swarm knows
|
|
(`GET /api/agents`, `/api/agents/status`), and the state it declares for
|
|
each one on its hive (`up`/`offline`/`paused`/`destroyed`,
|
|
`PUT /api/hives/{hive}/agents/{agent}/state`).
|
|
- **Job graph** — a `hive-jobq` scheduler, served at `GET /api/jobq/graph`.
|
|
Every provisioning step below runs as a node in it.
|
|
- **Agent creation** — `POST /api/agents` queues the SSO identity (through
|
|
`swarm-authelia-bridge`), forge user, config repo, store identity, forge
|
|
token and matrix account, declares the agent `paused`, then sends its hive
|
|
a deploy message.
|
|
- **Agent credentials** — at start and every five minutes it re-checks every
|
|
agent's forge token and matrix account, and renews store certificates and
|
|
queue secrets as they age.
|
|
- **Linked external accounts** — an operator-supplied matrix, forge or
|
|
GitHub account for one agent, stored in the swarm secret store
|
|
(`PUT /api/hives/{hive}/agents/{agent}/matrix-accounts/{account}`,
|
|
`.../forge-accounts/{label}`, `.../github-account`); distinct from the agent's own swarm-minted
|
|
accounts above. `GET .../linked-accounts` lists them, plus the agent's own
|
|
`main` matrix account, by kind, name and host, never with a credential.
|
|
Each of the three also takes a `DELETE`; matrix's also takes
|
|
`?revoke=true`, which logs the stored token out at its homeserver first and
|
|
leaves the account in place if that fails.
|
|
- **Config PR status** — each agent's open config-repo PR, cached from forge
|
|
webhooks (`GET /api/config-prs`, `/api/agents/{name}/config-pr`).
|
|
- **Swarm-wide forge objects and webhooks** →
|
|
[`docs/swarm/README.md`](../docs/swarm/README.md#swarm-wide-forge-objects).
|
|
- **Scheduled prompts** — named prompts for one or more agents, kept in
|
|
the `schedule-definitions` KV bucket and written into each agent's key of
|
|
`agent-schedules`, both of which this daemon alone writes
|
|
(`GET`/`POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`); each
|
|
agent fires them itself → [`docs/swarm/schedules.md`](../docs/swarm/schedules.md).
|
|
- **Inference settings** — each agent's model, effort, subagent models and
|
|
cap on concurrently running subagents, one key per agent in the
|
|
`agent-inference` KV bucket, which this daemon
|
|
alone writes (`GET`/`PUT`/`DELETE /api/agents/{name}/inference`); each
|
|
agent applies them itself. `GET` also answers the model and effort the
|
|
agent last reported in its turn-state header, kept in memory →
|
|
[`docs/swarm/inference.md`](../docs/swarm/inference.md).
|
|
- **Resource caps** — each agent's CPU and memory cap, one key per agent in
|
|
the `agent-limits` KV bucket, which this daemon alone writes
|
|
(`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the
|
|
agent reads it, and each cap replaces the hive's own answer for that field →
|
|
[`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits).
|
|
- **Review classifier** — try-outs from the swarm UI and reviews on review
|
|
requests, and the newest 100 finished runs, one document in the
|
|
`review-classifier-runs` KV bucket, which this daemon alone writes
|
|
(`POST /api/review-classifier/try-out`, `GET /api/review-classifier/try-out/{id}`,
|
|
`GET /api/review-classifier/rules`, `GET /api/review-classifier/runs`), and
|
|
the try-out page's branch for a PR, pushed as the classifier's forge account
|
|
(`POST /api/review-classifier/open-pr`) →
|
|
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md).
|
|
- **Review classifier settings** — the classifier's runtime, model, effort and how
|
|
many requested reviews run at once, one document in the
|
|
`review-classifier-settings` KV bucket, which this daemon alone reads and
|
|
writes (`GET`/`PUT /api/review-classifier/settings`) →
|
|
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings).
|
|
- **Operator profiles** — each operator's name and the pronouns on their
|
|
forge profile, for every member of authelia's operator group, as one list
|
|
under the one key of the `operator-profiles` KV bucket. This daemon alone
|
|
writes it, at start and every five minutes, and only when the list changed.
|
|
A forge read that fails keeps that operator's stored entry, and a name that
|
|
leaves the group drops out of the list. Every agent reads the key into its
|
|
system prompt →
|
|
[`docs/turn-loop/claude-invocation.md`](../docs/turn-loop/claude-invocation.md).
|
|
- **Relays** — each agent's terminal and turn-state header as SSE, agent
|
|
icons, the cross-repo issue report, and the UI's quick links.
|
|
- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read
|
|
at request time with a Viewer service-account token from the secret store
|
|
→ [`docs/swarm/secrets.md`](../docs/swarm/secrets.md).
|
|
|
|
It reads its configuration once at startup, from the environment the nix module
|
|
sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its
|
|
state directory.
|
|
The job graph lives in memory; hive status and wanted state live in the swarm
|
|
queue, so both survive a restart.
|
|
|
|
## Why a unix socket, not a port
|
|
|
|
The hive-gateway's nginx is the only intended client and reaches the socket
|
|
through a bind-mount. A listener that is never bound to an address cannot be
|
|
reached from off-host by mistake.
|
|
|
|
The socket path is `services.hyperhive.deploy.swarm-controller.socketPath`, default
|
|
`/run/swarm-controller/controller.sock`, exported to the process as
|
|
`SWARM_CONTROLLER_SOCKET`.
|
|
|
|
## ⚠️ The socket's directory is its access control
|
|
|
|
The socket is `0666`. It has to be: nginx runs as a different user and
|
|
`connect(2)` needs write. This matches how `hive-c0re` publishes the per-agent
|
|
sockets, and rests on the same argument — _"the bind source dir is per-agent on
|
|
host so blast radius is unchanged."_
|
|
|
|
What keeps that safe is that the directory holds **one** socket. So:
|
|
|
|
> **Never point `socketPath` at a directory that carries anything else.**
|
|
> `/run/hyperhive` above all — it holds `host.sock`, the host **admin** socket.
|
|
> Pointing nginx at that directory to reach this socket would put the admin
|
|
> socket within its reach too.
|
|
|
|
nginx is a host service, so nothing narrows what it can reach except the
|
|
directory itself — that is the whole of the access control. A unit test pins
|
|
the default path so a tidying edit fails instead of reviewing cleanly.
|
|
|
|
`RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are
|
|
a **pair**: preserving the directory without the unlink means `bind` fails with
|
|
`EADDRINUSE` after a restart.
|
|
|
|
## Packaging
|
|
|
|
Built by the workspace derivation and extracted as its own package
|
|
(`nix build .#swarm-controller`). Deliberately **not** in `nix/packages`'
|
|
`daemonBins` — that list is the core stack and drives the bundle
|
|
`services.hyperhive.c0re.package` points at, so folding this in would put a
|
|
swarm-scoped service into every hive's closure.
|