302 lines
19 KiB
Markdown
302 lines
19 KiB
Markdown
# Swarm-wide services
|
|
|
|
Some things exist once per **swarm** rather than once per hive: the forge,
|
|
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
|
|
log stack. This page says which host runs them, what a hive that runs none of
|
|
them configures instead, and what each service is for. The all-local quick
|
|
start sets everything with one line →
|
|
[README](../../../README.md#quick-start-an-all-local-swarm).
|
|
|
|
Two options say where they live, and everything else derives:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
|
|
# or, for a dedicated services host with hives elsewhere:
|
|
services.hyperhive.deploy.allSwarmServices = true;
|
|
```
|
|
|
|
**`deploy.allSwarmServices` is what "the swarm's shared services run
|
|
here" means: every once-per-swarm service takes its `enable` from it.** The
|
|
service sections below don't repeat that. A service that needs the secret
|
|
store comes on only where the host has the service's store identity, so a
|
|
services host with the store elsewhere gets it once it holds a copy of that
|
|
identity; set the service's own `enable` to override either way.
|
|
|
|
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
|
|
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`),
|
|
the host's `/etc/hosts` entries for the names it serves
|
|
(`gateway.localHostsEntry`) and where the secret store's bootstrap token goes
|
|
(`deploy.bao.bootstrapTokenFile`). You can still set each derived toggle on
|
|
its own, and that setting wins.
|
|
|
|
**Both default to off**, because a host can't tell whether it's meant to be
|
|
the swarm's service host. With them off, a hive is a _client_ of those
|
|
services: it configures how to reach them and runs none of them.
|
|
|
|
That includes the forge (`deploy.forgejo.enable`). Every swarm needs one,
|
|
being the canonical store for the meta flake and every agent's config repo,
|
|
so **exactly one host must turn it on**: `singleHostSwarm`,
|
|
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand.
|
|
|
|
A hive can't start or stop any of these. `hivectl stop`, `start` and
|
|
`restart` act on the hive's agents and its gateway only, even on the host that
|
|
runs the swarm's services. The forge and matrix containers are systemd units
|
|
on that host (`container@hive-forge.service`,
|
|
`container@hive-matrix.service`), and you manage them there with `systemctl`.
|
|
|
|
A hive that runs none of these reaches each one by name, the forge at
|
|
`forge.<swarm-domain>` for example. Only the host running a service answers its
|
|
name from its own resolver, so on a swarm spread over more than one host, your
|
|
DNS has to resolve those names to that host.
|
|
|
|
<details><summary>Moving an existing hive's forge to the swarm's</summary>
|
|
|
|
A hive that stops running the forge keeps the old container's state at
|
|
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
|
|
forge: push anything worth keeping there by hand.
|
|
|
|
</details>
|
|
|
|
## Deployment shapes
|
|
|
|
- **All-local.** Everything on one machine: `singleHostSwarm = true`, plus
|
|
`deploy.hive-controller.enable = true` for a hive to run agents on. After the
|
|
first switch, the steps in [`setup.md`](../../1-getting-started/setup.md)
|
|
remain.
|
|
- **Services on the swarm controller host.** Set `deploy.allSwarmServices` and
|
|
`deploy.swarm-controller.enable` there, with hives elsewhere. The controller
|
|
doesn't derive from `allSwarmServices`.
|
|
- **Fully spread out.** One container, VM or machine per service.
|
|
- **Semi-spread.** The services split over a few machines by weight, hives on
|
|
their own, a VPN between them, and most vhosts answering on the VPN only. A
|
|
worked example with its NixOS files:
|
|
[`semi-spread-deploy.md`](semi-spread-deploy.md).
|
|
|
|
These are a set, not a ladder with a correct top, and in-between shapes work:
|
|
you can set each derived toggle on its own, which makes "all local except X" a
|
|
configuration rather than an unsupported edge case. Co-location is normal, and
|
|
where one service has something to say about sharing a host, that belongs in
|
|
the service's own doc.
|
|
|
|
## Services
|
|
|
|
An operator who isn't spreading the swarm over more than one host can skip most
|
|
of this page: `singleHostSwarm` turns on the shared services below, with two
|
|
exceptions. The **CI runner** stays off until you set `deploy.ci-runner.enable`,
|
|
and the **builder** and **binary cache** come on only where the host has their
|
|
store identities. The builder and the CI runner are the two that cost real
|
|
resources (they build and run pull-request code), so they're the two worth a
|
|
look before you switch them on.
|
|
|
|
| Service | What it does | Option | Details |
|
|
| ------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------ |
|
|
| Forge | Holds the meta flake and every agent's config repo. | `deploy.forgejo.enable` | [forge.md](forge.md) |
|
|
| Matrix | The chat homeserver agents and the operator talk over. | `deploy.matrix.enable` | [matrix.md](matrix.md) |
|
|
| Secret store | Keeps the swarm's credentials. | `deploy.bao.enable` | [credentials.md](credentials.md) |
|
|
| Queue | Carries hive status and swarm events. | `deploy.nats.enable` | [Deployment shapes](#deployment-shapes) |
|
|
| SSO | One login for the operator and the agents. | `deploy.authelia.enable` | [SSO](#sso-authelia) |
|
|
| Metrics | Stores telemetry and shows it in Grafana. | `deploy.victoriametrics.enable`, `deploy.grafana.enable` | [Metrics](#metrics-victoriametrics--grafana) |
|
|
| Logs | Stores every journal for search in Grafana. | `deploy.victorialogs.enable` | [Logs](#logs-victorialogs) |
|
|
| Telemetry collector | Routes metrics and logs to the stores and an optional upstream. | `deploy.swarm-otel.enable` | [Telemetry collector](#telemetry-collector-otel) |
|
|
| CI runner | Runs Forgejo Actions for pull requests. Heavy. | `deploy.ci-runner.enable` | [CI runner](#ci-runner) |
|
|
| Builder | Evaluates and builds agent config PRs. Heavy. | `deploy.builder.enable` | [Builder](#builder) |
|
|
| Binary cache | Serves built closures to the hives. | `deploy.attic.enable` | [Binary cache](#binary-cache) |
|
|
| Backup archive | Keeps every agent's state backups, one repository each. | `deploy.archive.enable` | [Backup archive](archive.md) |
|
|
| CI metrics | Records CI run times in the metrics store. | `deploy.ci-metrics.enable` | [CI metrics](#ci-metrics) |
|
|
| Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) |
|
|
| Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) |
|
|
| Notification router | Turns forge events into notifications for agents. | `deploy.notify-router.enable` | [Notification router](#notification-router) |
|
|
| Speech to text | Turns voice memos into text for agents. | `deploy.stt.enable` | [Speech to text](#speech-to-text) |
|
|
|
|
### SSO (authelia)
|
|
|
|
One authelia per swarm, in a `swarm-authelia` container at
|
|
`auth.<swarm-domain>`. The operator and the agents are both subjects of the
|
|
same provider, told apart by roles and claims. Set `deploy.authelia.enable` to
|
|
run it here. Every hive sets `swarm.authelia.url`, the same value on all of
|
|
them: `https://<swarm.authelia.domain>` by default, or the other name when you
|
|
join a swarm whose IdP lives elsewhere. Its keys live in the secret store, and
|
|
nothing rotates them automatically: replacing one invalidates the sessions and
|
|
the encrypted store, so that's an action you take on purpose.
|
|
|
|
Bootstrapping the first user and the login flow: [`sso.md`](sso.md). The
|
|
container, its sockets, secrets and metrics:
|
|
[Host setup](../../4-implementation/swarm/sso.md#host-setup).
|
|
|
|
### Metrics (VictoriaMetrics + Grafana)
|
|
|
|
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it,
|
|
at `metrics.<swarm-domain>` and `grafana.<swarm-domain>`. They sit in two
|
|
containers so you can restart or break Grafana without losing the time-series
|
|
database. Enabling either enables both; to run one alone, set it directly with
|
|
`deploy.victoriametrics.enable` or `deploy.grafana.enable`.
|
|
|
|
Log in to Grafana with your swarm SSO account. Grafana provisions its
|
|
dashboards from the repo, so edits made in the UI aren't kept. Each service has
|
|
its own board; open the folder to see them.
|
|
|
|
⚠️ **The metrics store grows for as long as the swarm runs.** Options to touch:
|
|
|
|
| Option | When you'd touch it |
|
|
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you know how fast the swarm fills a disk. A too-short value discards history you can't get back. |
|
|
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower it to `Viewer` or `Editor` if some operators shouldn't reconfigure Grafana. |
|
|
|
|
The dashboard list, the "No data" cases, and how the gateway guards the store:
|
|
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#metrics-victoriametrics--grafana).
|
|
|
|
### Logs (VictoriaLogs)
|
|
|
|
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
|
|
behind the same SSO as everything else: the swarm's service containers, the
|
|
hive's daemons and the harness units in every agent container.
|
|
|
|
To read them, open Grafana, pick **Explore** and choose the `VictoriaLogs`
|
|
datasource. Agents query it with `swarm-logs`, and need the `logs` permission.
|
|
|
|
| Option | When you'd touch it |
|
|
| ------------------------------------- | ----------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's. Raise it to answer questions about last quarter. |
|
|
|
|
The other options, who may query, and the gateway rules:
|
|
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#logs-victorialogs).
|
|
|
|
### Telemetry collector (OTEL)
|
|
|
|
The swarm's collector receives from every hive's own collector and decides where
|
|
telemetry goes. It always writes the metrics and log stores, and exports to
|
|
`deploy.swarm-otel.upstream.endpoint` as well when you set one. It holds the
|
|
upstream credential, so no hive and no agent needs one. Every hive reaches it
|
|
by its gateway name, `otel.<swarm-domain>`, so a hive that doesn't run the
|
|
swarm's services needs nothing set.
|
|
|
|
How the collector connects, its per-service routes and the homeserver's
|
|
stand-in metrics:
|
|
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#telemetry-collector-otel).
|
|
|
|
### CI runner
|
|
|
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
|
runner executes pull-request code, so running one is a decision of its own. It
|
|
needs neither the forge nor a hive on its host, and registers itself with the
|
|
forge. Its **ci runner** board in Grafana shows its jobs and capacity, and the
|
|
forge's **CI runs** tab shows the run times of every run. Registration, the
|
|
metrics it exports and the threat model:
|
|
[Host setup](../../4-implementation/scheduler/ci.md#host-setup) and the rest of
|
|
[`ci.md`](../../4-implementation/scheduler/ci.md).
|
|
|
|
### Builder
|
|
|
|
The swarm builder, in a `swarm-builder` container on the host that sets
|
|
`deploy.builder.enable`; set it to `false` to opt out. It evaluates and builds
|
|
unreviewed agent config PRs, and the result shows as a `swarm-builder` commit
|
|
status on the PR. A red status blocks nothing: the merge still deploys. It also
|
|
builds each agent before the swarm sends its deploy, and a failed build holds
|
|
that deploy back. The jobs show on the swarm UI's Jobs page. It builds through
|
|
its host's nix daemon, so it costs real CPU and disk.
|
|
[Host setup](../../4-implementation/swarm/builder.md#host-setup), and what it
|
|
builds and why: [`builder.md`](../../4-implementation/swarm/builder.md).
|
|
|
|
### Binary cache
|
|
|
|
The swarm's attic server, in a `swarm-attic` container on the host that sets
|
|
`deploy.attic.enable`. The builder pushes green builds to it, and the hives pull
|
|
closures from it at `swarm.attic.domain` (default `attic.<swarm-domain>`). For
|
|
an external cache, override `swarm.attic.{domain,cache}` and don't set
|
|
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
|
|
`allSwarmServices`. Its keys and tokens create themselves in the secret store.
|
|
How it works:
|
|
[Binary cache host setup](../../4-implementation/swarm/builder.md#binary-cache-host-setup)
|
|
and [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache).
|
|
|
|
### CI metrics
|
|
|
|
The CI run poller, a `swarm-ci-metrics` timer unit on the host that sets
|
|
`deploy.ci-metrics.enable`; set it to `false` to opt out. Every five minutes it
|
|
pushes the run time and queue wait of each newly finished forge Actions run to
|
|
the metrics store. What it reads and pushes:
|
|
[`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md).
|
|
|
|
### Review classifier
|
|
|
|
The review classifier's try-out service, in a `swarm-review-classifier`
|
|
container on the host that sets `deploy.review-classifier.enable`. You set the
|
|
key for the runtime the swarm settings choose in the swarm UI: the **classifier
|
|
settings** card on `/jobs` takes a key for that runtime and shows whether one
|
|
exists and when it changed, never the key. "Set" means a key exists, not that it
|
|
works, and every run spends money on that key; until the chosen one exists,
|
|
every run fails with `no API key…` or `no ACP provider key…`. The service reads
|
|
the key on every run, so writing or removing it takes effect on the next run.
|
|
The store paths for the two keys, and what a run is and costs:
|
|
[Host setup](../../4-implementation/swarm/review-classifier.md#host-setup) and
|
|
[`review-classifier.md`](../../4-implementation/swarm/review-classifier.md).
|
|
|
|
### Claude login minter
|
|
|
|
The claude login minter, in a `swarm-claude-login` container on the host that
|
|
sets `deploy.claude-login.enable`. Run one: a login in progress lives in the
|
|
service's memory between its start and its code. An operator logs an agent in
|
|
from the **log in** button on its card in the swarm UI, and the agent's
|
|
harness installs the new login. The flow and the version rule:
|
|
[`claude-login.md`](claude-login.md). The service's host setup:
|
|
[Host setup](../../4-implementation/swarm/claude-login.md#host-setup).
|
|
|
|
### Notification router
|
|
|
|
The notification router, in a `swarm-notify-router` container on the host that
|
|
sets `deploy.notify-router.enable`. Run one: it handles each issue's events in
|
|
order. It turns forge events (assigning an issue or pull request, closing it, a
|
|
failed CI run on a pull request) into todos for the agents they concern, and an
|
|
agent that's down gets its todos when it's back. The flows and the delivery
|
|
path:
|
|
[`notify-router.md`](../../4-implementation/swarm/notify-router.md), and
|
|
[Host setup](../../4-implementation/swarm/notify-router.md#host-setup).
|
|
|
|
### Speech to text
|
|
|
|
The speech-to-text service, in a `swarm-stt` container on the host that sets
|
|
`deploy.stt.enable`. Run it where the inference hardware is: it transcribes
|
|
with whisper.cpp. An agent that receives a voice memo sends the audio to it
|
|
over the queue and gets the text back, and a memo sent to several agents costs
|
|
one run. `deploy.stt.settings.modelSize` picks the model (`small` by default,
|
|
or `base`). The audio comes from rooms with outside people, so the service runs
|
|
under a tight sandbox, but its network isn't restricted, which the design page
|
|
spells out. The settings, limits and reasons:
|
|
[`stt.md`](../../4-implementation/swarm/stt.md) and
|
|
[Host setup](../../4-implementation/swarm/stt.md#host-setup).
|
|
|
|
### Health endpoints
|
|
|
|
The queue daemons (`swarm-stt`, `swarm-review-classifier`,
|
|
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`),
|
|
`swarm-nats-auth` and the three auth responders (`swarm-logs-auth`,
|
|
`swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two probe endpoints.
|
|
They answer 200 or 503 and send no body.
|
|
|
|
- `GET /healthz` is 200 while the daemon's main loop is running. The loop
|
|
beats as it takes messages, so a loop that stops polling turns the answer to 503
|
|
after 30 seconds (5 minutes for `swarm-notify-router`, whose loop handles a
|
|
batch of events between beats). Before the loop's first beat the answer is
|
|
200: waiting on the secret store at startup doesn't count as stuck. A
|
|
daemon that only serves requests (the three auth responders) beats from a task
|
|
on its runtime, so `/healthz` shows the runtime still runs tasks, not that
|
|
its accept loop is healthy.
|
|
- `GET /readyz` is 200 while the daemon has what it needs to work: its queue
|
|
connection works (`swarm-stt`, `swarm-review-classifier`,
|
|
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`,
|
|
`swarm-nats-auth`), or it has bound its socket (the three auth responders).
|
|
|
|
Nothing listens by default. Set `SWARM_STT_HEALTH_ADDR`,
|
|
`SWARM_REVIEW_CLASSIFIER_HEALTH_ADDR`, `SWARM_NOTIFY_ROUTER_HEALTH_ADDR`,
|
|
`SWARM_BUILDER_HEALTH_ADDR`, `SWARM_CLAUDE_LOGIN_HEALTH_ADDR` or
|
|
`SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` to a `host:port` to bind the listener. For
|
|
`swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth`, pass
|
|
`--health-addr`, or set `SWARM_NATS_AUTH_HEALTH_ADDR`,
|
|
`SWARM_LOGS_AUTH_HEALTH_ADDR` or `SWARM_METRICS_AUTH_HEALTH_ADDR`; the flag
|
|
wins. No NixOS module sets one. The listener stops on the same SIGTERM as the
|
|
daemon.
|
|
|
|
`swarm-controller` serves `GET /readyz` on its existing socket, next to
|
|
`/health`. It answers 200 while the queue connection works and the secret store
|
|
accepts the controller's login, and 503 otherwise.
|