7d09776742
swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in to Grafana's API over its socket as the built-in admin (the password swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller` service account that is an enabled Viewer. It mints a token for that account and writes it to bao at swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A ten-minute timer runs it again. The unit keeps a stored token Grafana accepts. If Grafana refuses the stored token with 401, or bao holds none, the unit deletes every token of the account and mints a new one. Grafana's grant gains create/read/update on that leaf, and the controller's gains read. swarm-controller serves GET /api/alerts from Grafana's /api/prometheus/grafana/api/v1/alerts. It reads the token from bao, keeps it in memory, and re-reads bao once on a 401. The HTTP client has 5s connect and 15s request timeouts. Any failure is a 503 that names its cause. swarm-ui gets an /alerts page listing them, built the same way as the hives page. This is option (c) on #5129, the operator's choice.
124 lines
6.9 KiB
Markdown
124 lines
6.9 KiB
Markdown
# swarm-controller
|
|
|
|
The **swarm-level** daemon. Where `hive-c0re` owns the agents on one host, this
|
|
owns what is true _across_ hives — so a swarm runs one of them and most hives
|
|
leave it off.
|
|
|
|
Opt-in per host via `services.hyperhive.deploy.swarm-controller.enable`, which is
|
|
deliberately **not** derived from
|
|
`services.hyperhive.deploy.hive-controller.enable`: turning it on is a statement
|
|
about swarm topology, not about whether this host runs a hive.
|
|
|
|
## What it does
|
|
|
|
The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|
|
|
- **Hive directory** — serves `swarm.hives` (`GET /api/hives`) and what each
|
|
hive last published about itself (`GET /api/hives/status`).
|
|
- **Agent roster and wanted state** — every agent the swarm knows
|
|
(`GET /api/agents`, `/api/agents/status`), and the state it declares for
|
|
each one on its hive (`up`/`offline`/`paused`/`destroyed`,
|
|
`PUT /api/hives/{hive}/agents/{agent}/state`).
|
|
- **Job graph** — a `hive-jobq` scheduler, served at `GET /api/jobq/graph`.
|
|
Every provisioning step below runs as a node in it.
|
|
- **Agent creation** — `POST /api/agents` queues the SSO identity (through
|
|
`swarm-authelia-bridge`), forge user, config repo, store identity, forge
|
|
token and matrix account, declares the agent `paused`, then sends its hive
|
|
a deploy message.
|
|
- **Agent credentials** — at start and every five minutes it re-checks every
|
|
agent's forge token and matrix account, and renews store certificates and
|
|
queue secrets as they age.
|
|
- **Linked external accounts** — an operator-supplied matrix, forge or
|
|
GitHub account for one agent, stored in the swarm secret store
|
|
(`PUT /api/hives/{hive}/agents/{agent}/matrix-accounts/{account}`,
|
|
`.../forge-accounts/{label}`, `.../github-account`); distinct from the agent's own swarm-minted
|
|
accounts above. `GET .../linked-accounts` lists them, plus the agent's own
|
|
`main` matrix account, by kind, name and host, never with a credential.
|
|
Each of the three also takes a `DELETE`; matrix's also takes
|
|
`?revoke=true`, which logs the stored token out at its homeserver first and
|
|
leaves the account in place if that fails.
|
|
- **Config PR status** — each agent's open config-repo PR, cached from forge
|
|
webhooks (`GET /api/config-prs`, `/api/agents/{name}/config-pr`).
|
|
- **Swarm-wide forge objects and webhooks** →
|
|
[`docs/swarm/README.md`](../docs/swarm/README.md#swarm-wide-forge-objects).
|
|
- **Scheduled prompts** — named prompts for one or more agents, kept in
|
|
the `schedule-definitions` KV bucket and written into each agent's key of
|
|
`agent-schedules`, both of which this daemon alone writes
|
|
(`GET`/`POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`); each
|
|
agent fires them itself → [`docs/swarm/schedules.md`](../docs/swarm/schedules.md).
|
|
- **Inference settings** — each agent's model, effort, subagent models and
|
|
cap on concurrently running subagents, one key per agent in the
|
|
`agent-inference` KV bucket, which this daemon
|
|
alone writes (`GET`/`PUT`/`DELETE /api/agents/{name}/inference`); each
|
|
agent applies them itself. `GET` also answers the model and effort the
|
|
agent last reported in its turn-state header, kept in memory →
|
|
[`docs/swarm/inference.md`](../docs/swarm/inference.md).
|
|
- **Resource caps** — each agent's CPU and memory cap, one key per agent in
|
|
the `agent-limits` KV bucket, which this daemon alone writes
|
|
(`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the
|
|
agent reads it, and each cap replaces the hive's own answer for that field →
|
|
[`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits).
|
|
- **Review classifier** — try-outs from the swarm UI and reviews on review
|
|
requests, and the newest 100 finished runs, one document in the
|
|
`review-classifier-runs` KV bucket, which this daemon alone writes
|
|
(`POST /api/review-classifier/try-out`, `GET /api/review-classifier/try-out/{id}`,
|
|
`GET /api/review-classifier/rules`, `GET /api/review-classifier/runs`), and
|
|
the try-out page's branch for a PR, pushed as the classifier's forge account
|
|
(`POST /api/review-classifier/open-pr`) →
|
|
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md).
|
|
- **Review classifier settings** — the classifier's runtime, model, effort and how
|
|
many requested reviews run at once, one document in the
|
|
`review-classifier-settings` KV bucket, which this daemon alone reads and
|
|
writes (`GET`/`PUT /api/review-classifier/settings`) →
|
|
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings).
|
|
- **Relays** — each agent's terminal and turn-state header as SSE, agent
|
|
icons, the cross-repo issue report, and the UI's quick links.
|
|
- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read
|
|
at request time with a Viewer service-account token from the secret store
|
|
→ [`docs/swarm/secrets.md`](../docs/swarm/secrets.md).
|
|
|
|
It reads its configuration once at startup, from the environment the nix module
|
|
sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its
|
|
state directory.
|
|
The job graph lives in memory; hive status and wanted state live in the swarm
|
|
queue, so both survive a restart.
|
|
|
|
## Why a unix socket, not a port
|
|
|
|
The hive-gateway's nginx is the only intended client and reaches the socket
|
|
through a bind-mount. A listener that is never bound to an address cannot be
|
|
reached from off-host by mistake.
|
|
|
|
The socket path is `services.hyperhive.deploy.swarm-controller.socketPath`, default
|
|
`/run/swarm-controller/controller.sock`, exported to the process as
|
|
`SWARM_CONTROLLER_SOCKET`.
|
|
|
|
## ⚠️ The socket's directory is its access control
|
|
|
|
The socket is `0666`. It has to be: nginx runs as a different user and
|
|
`connect(2)` needs write. This matches how `hive-c0re` publishes the per-agent
|
|
sockets, and rests on the same argument — _"the bind source dir is per-agent on
|
|
host so blast radius is unchanged."_
|
|
|
|
What keeps that safe is that the directory holds **one** socket. So:
|
|
|
|
> **Never point `socketPath` at a directory that carries anything else.**
|
|
> `/run/hyperhive` above all — it holds `host.sock`, the host **admin** socket.
|
|
> Pointing nginx at that directory to reach this socket would put the admin
|
|
> socket within its reach too.
|
|
|
|
nginx is a host service, so nothing narrows what it can reach except the
|
|
directory itself — that is the whole of the access control. A unit test pins
|
|
the default path so a tidying edit fails instead of reviewing cleanly.
|
|
|
|
`RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are
|
|
a **pair**: preserving the directory without the unlink means `bind` fails with
|
|
`EADDRINUSE` after a restart.
|
|
|
|
## Packaging
|
|
|
|
Built by the workspace derivation and extracted as its own package
|
|
(`nix build .#swarm-controller`). Deliberately **not** in `nix/packages`'
|
|
`daemonBins` — that list is the core stack and drives the bundle
|
|
`services.hyperhive.c0re.package` points at, so folding this in would put a
|
|
swarm-scoped service into every hive's closure.
|