2fc0166427
Every run is recorded when it ends: a try-out when its reply settles it, a requested review once its review is posted or the post fails, and a request turned away by a full queue once its comment is posted. A duplicate request that starts nothing is not recorded. The newest 100 are one JSON array under `recent` in a new controller-owned KV bucket, `review-classifier-runs`, written with compare-and-set as `schedule-definitions` is (five attempts). A failed write is logged; the review is already posted by then. Each recorded run is also one info event, "review classifier run finished", with the record's fields as tracing fields. The controller logs natively to the journal, so they reach VictoriaLogs as F_<NAME>. GET /api/review-classifier/runs returns the list, newest first, gated by the gateway like the try-out GET. post_classifier_review now returns the posted review's html_url, which the record keeps as review_url. Closes #5153, part of #5096.
113 lines
6.2 KiB
Markdown
113 lines
6.2 KiB
Markdown
# swarm-controller
|
|
|
|
The **swarm-level** daemon. Where `hive-c0re` owns the agents on one host, this
|
|
owns what is true _across_ hives — so a swarm runs one of them and most hives
|
|
leave it off.
|
|
|
|
Opt-in per host via `services.hyperhive.deploy.swarm-controller.enable`, which is
|
|
deliberately **not** derived from
|
|
`services.hyperhive.deploy.hive-controller.enable`: turning it on is a statement
|
|
about swarm topology, not about whether this host runs a hive.
|
|
|
|
## What it does
|
|
|
|
The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|
|
|
- **Hive directory** — serves `swarm.hives` (`GET /api/hives`) and what each
|
|
hive last published about itself (`GET /api/hives/status`).
|
|
- **Agent roster and wanted state** — every agent the swarm knows
|
|
(`GET /api/agents`, `/api/agents/status`), and the state it declares for
|
|
each one on its hive (`up`/`offline`/`paused`/`destroyed`,
|
|
`PUT /api/hives/{hive}/agents/{agent}/state`).
|
|
- **Job graph** — a `hive-jobq` scheduler, served at `GET /api/jobq/graph`.
|
|
Every provisioning step below runs as a node in it.
|
|
- **Agent creation** — `POST /api/agents` queues the SSO identity (through
|
|
`swarm-authelia-bridge`), forge user, config repo, store identity, forge
|
|
token and matrix account, declares the agent `paused`, then sends its hive
|
|
a deploy message.
|
|
- **Agent credentials** — at start and every five minutes it re-checks every
|
|
agent's forge token and matrix account, and renews store certificates and
|
|
queue secrets as they age.
|
|
- **Linked external accounts** — an operator-supplied matrix, forge or
|
|
GitHub account for one agent, stored in the swarm secret store
|
|
(`PUT /api/hives/{hive}/agents/{agent}/matrix-accounts/{account}`,
|
|
`.../forge-accounts/{label}`, `.../github-account`); distinct from the agent's own swarm-minted
|
|
accounts above. `GET .../linked-accounts` lists them, plus the agent's own
|
|
`main` matrix account, by kind, name and host, never with a credential.
|
|
Each of the three also takes a `DELETE`; matrix's also takes
|
|
`?revoke=true`, which logs the stored token out at its homeserver first and
|
|
leaves the account in place if that fails.
|
|
- **Config PR status** — each agent's open config-repo PR, cached from forge
|
|
webhooks (`GET /api/config-prs`, `/api/agents/{name}/config-pr`).
|
|
- **Swarm-wide forge objects and webhooks** →
|
|
[`docs/swarm/README.md`](../docs/swarm/README.md#swarm-wide-forge-objects).
|
|
- **Scheduled prompts** — named prompts for one or more agents, kept in
|
|
the `schedule-definitions` KV bucket and written into each agent's key of
|
|
`agent-schedules`, both of which this daemon alone writes
|
|
(`GET`/`POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`); each
|
|
agent fires them itself → [`docs/swarm/schedules.md`](../docs/swarm/schedules.md).
|
|
- **Inference settings** — each agent's model, effort and subagent models,
|
|
one key per agent in the `agent-inference` KV bucket, which this daemon
|
|
alone writes (`GET`/`PUT`/`DELETE /api/agents/{name}/inference`); each
|
|
agent applies them itself. `GET` also answers the model and effort the
|
|
agent last reported in its turn-state header, kept in memory →
|
|
[`docs/swarm/inference.md`](../docs/swarm/inference.md).
|
|
- **Resource caps** — each agent's CPU and memory cap, one key per agent in
|
|
the `agent-limits` KV bucket, which this daemon alone writes
|
|
(`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the
|
|
agent reads it, and each cap replaces the hive's own answer for that field →
|
|
[`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits).
|
|
- **Review classifier** — try-outs from the swarm UI and reviews on review
|
|
requests, and the newest 100 finished runs, one document in the
|
|
`review-classifier-runs` KV bucket, which this daemon alone writes
|
|
(`POST /api/review-classifier/try-out`, `GET /api/review-classifier/try-out/{id}`,
|
|
`GET /api/review-classifier/rules`, `GET /api/review-classifier/runs`) →
|
|
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md).
|
|
- **Relays** — each agent's terminal and turn-state header as SSE, agent
|
|
icons, the cross-repo issue report, and the UI's quick links.
|
|
|
|
It reads its configuration once at startup, from the environment the nix module
|
|
sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its
|
|
state directory.
|
|
The job graph lives in memory; hive status and wanted state live in the swarm
|
|
queue, so both survive a restart.
|
|
|
|
## Why a unix socket, not a port
|
|
|
|
The hive-gateway's nginx is the only intended client and reaches the socket
|
|
through a bind-mount. A listener that is never bound to an address cannot be
|
|
reached from off-host by mistake.
|
|
|
|
The socket path is `services.hyperhive.deploy.swarm-controller.socketPath`, default
|
|
`/run/swarm-controller/controller.sock`, exported to the process as
|
|
`SWARM_CONTROLLER_SOCKET`.
|
|
|
|
## ⚠️ The socket's directory is its access control
|
|
|
|
The socket is `0666`. It has to be: nginx runs as a different user and
|
|
`connect(2)` needs write. This matches how `hive-c0re` publishes the per-agent
|
|
sockets, and rests on the same argument — _"the bind source dir is per-agent on
|
|
host so blast radius is unchanged."_
|
|
|
|
What keeps that safe is that the directory holds **one** socket. So:
|
|
|
|
> **Never point `socketPath` at a directory that carries anything else.**
|
|
> `/run/hyperhive` above all — it holds `host.sock`, the host **admin** socket.
|
|
> Pointing nginx at that directory to reach this socket would put the admin
|
|
> socket within its reach too.
|
|
|
|
nginx is a host service, so nothing narrows what it can reach except the
|
|
directory itself — that is the whole of the access control. A unit test pins
|
|
the default path so a tidying edit fails instead of reviewing cleanly.
|
|
|
|
`RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are
|
|
a **pair**: preserving the directory without the unlink means `bind` fails with
|
|
`EADDRINUSE` after a restart.
|
|
|
|
## Packaging
|
|
|
|
Built by the workspace derivation and extracted as its own package
|
|
(`nix build .#swarm-controller`). Deliberately **not** in `nix/packages`'
|
|
`daemonBins` — that list is the core stack and drives the bundle
|
|
`services.hyperhive.c0re.package` points at, so folding this in would put a
|
|
swarm-scoped service into every hive's closure.
|