814edd1d67
Each service section lost its container, credential and metrics detail to a Host setup section on its own implementation page, unchanged apart from links. The four agentAuth option rows that sat above the page title moved into the telemetry page's metrics option table.
267 lines
17 KiB
Markdown
267 lines
17 KiB
Markdown
# Swarm-wide services
|
|
|
|
Some things exist once per **swarm** rather than once per hive: the forge,
|
|
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
|
|
log stack. This page says which host runs them, what a hive that runs none of
|
|
them configures instead, and what each service is for. The all-local quick
|
|
start sets everything with one line →
|
|
[README](../../../README.md#quick-start-an-all-local-swarm).
|
|
|
|
Two options say where they live, and everything else derives:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
|
|
# or, for a dedicated services host with hives elsewhere:
|
|
services.hyperhive.deploy.allSwarmServices = true;
|
|
```
|
|
|
|
**`deploy.allSwarmServices` is what "the swarm's shared services run
|
|
here" means: every once-per-swarm service takes its `enable` from it.** The
|
|
service sections below don't repeat that. A service that needs the secret
|
|
store comes on only where the host has the service's store identity, so a
|
|
services host with the store elsewhere gets it once it holds a copy of that
|
|
identity; set the service's own `enable` to override either way.
|
|
|
|
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
|
|
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`),
|
|
the host's `/etc/hosts` entries for the names it serves
|
|
(`gateway.localHostsEntry`) and where the secret store's bootstrap token goes
|
|
(`deploy.bao.bootstrapTokenFile`). You can still set each derived toggle on
|
|
its own, and that setting wins.
|
|
|
|
**Both default to off**, because a host can't tell whether it's meant to be
|
|
the swarm's service host. With them off, a hive is a _client_ of those
|
|
services: it configures how to reach them and runs none of them.
|
|
|
|
That includes the forge (`deploy.forgejo.enable`). Every swarm needs one,
|
|
being the canonical store for the meta flake and every agent's config repo,
|
|
so **exactly one host must turn it on**: `singleHostSwarm`,
|
|
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand.
|
|
|
|
A hive can't start or stop any of these. `hivectl stop`, `start` and
|
|
`restart` act on the hive's agents and its gateway only, even on the host that
|
|
runs the swarm's services. The forge and matrix containers are systemd units
|
|
on that host (`container@hive-forge.service`,
|
|
`container@hive-matrix.service`), and you manage them there with `systemctl`.
|
|
|
|
A hive that runs none of these reaches each one by name, the forge at
|
|
`forge.<swarm-domain>` for example. Only the host running a service answers its
|
|
name from its own resolver, so on a swarm spread over more than one host, your
|
|
DNS has to resolve those names to that host.
|
|
|
|
<details><summary>Moving an existing hive's forge to the swarm's</summary>
|
|
|
|
A hive that stops running the forge keeps the old container's state at
|
|
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
|
|
forge: push anything worth keeping there by hand.
|
|
|
|
</details>
|
|
|
|
## Deployment shapes
|
|
|
|
- **All-local.** Everything on one machine: `singleHostSwarm = true`, plus
|
|
`deploy.hive-controller.enable = true` for a hive to run agents on. After the
|
|
first switch, the steps in [`setup.md`](../../1-getting-started/setup.md)
|
|
remain.
|
|
- **Services on the swarm controller host.** Set `deploy.allSwarmServices` and
|
|
`deploy.swarm-controller.enable` there, with hives elsewhere. The controller
|
|
doesn't derive from `allSwarmServices`.
|
|
- **Fully spread out.** One container, VM or machine per service.
|
|
- **Semi-spread.** The services split over a few machines by weight, hives on
|
|
their own, a VPN between them, and most vhosts answering on the VPN only. A
|
|
worked example with its NixOS files:
|
|
[`semi-spread-deploy.md`](semi-spread-deploy.md).
|
|
|
|
These are a set, not a ladder with a correct top, and in-between shapes work:
|
|
you can set each derived toggle on its own, which makes "all local except X" a
|
|
configuration rather than an unsupported edge case. Co-location is normal, and
|
|
where one service has something to say about sharing a host, that belongs in
|
|
the service's own doc.
|
|
|
|
## Services
|
|
|
|
An operator who isn't spreading the swarm over more than one host can skip most
|
|
of this page: `singleHostSwarm` turns on the shared services below, with two
|
|
exceptions. The **CI runner** stays off until you set `deploy.ci-runner.enable`,
|
|
and the **builder** and **binary cache** come on only where the host has their
|
|
store identities. The builder and the CI runner are the two that cost real
|
|
resources (they build and run pull-request code), so they're the two worth a
|
|
look before you switch them on.
|
|
|
|
| Service | What it does | Option | Details |
|
|
| ------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------ |
|
|
| Forge | Holds the meta flake and every agent's config repo. | `deploy.forgejo.enable` | [forge.md](forge.md) |
|
|
| Matrix | The chat homeserver agents and the operator talk over. | `deploy.matrix.enable` | [matrix.md](matrix.md) |
|
|
| Secret store | Keeps the swarm's credentials. | `deploy.bao.enable` | [credentials.md](credentials.md) |
|
|
| Queue | Carries hive status and swarm events. | `deploy.nats.enable` | [Deployment shapes](#deployment-shapes) |
|
|
| SSO | One login for the operator and the agents. | `deploy.authelia.enable` | [SSO](#sso-authelia) |
|
|
| Metrics | Stores telemetry and shows it in Grafana. | `deploy.victoriametrics.enable`, `deploy.grafana.enable` | [Metrics](#metrics-victoriametrics--grafana) |
|
|
| Logs | Stores every journal for search in Grafana. | `deploy.victorialogs.enable` | [Logs](#logs-victorialogs) |
|
|
| Telemetry collector | Routes metrics and logs to the stores and an optional upstream. | `deploy.swarm-otel.enable` | [Telemetry collector](#telemetry-collector-otel) |
|
|
| CI runner | Runs Forgejo Actions for pull requests. Heavy. | `deploy.ci-runner.enable` | [CI runner](#ci-runner) |
|
|
| Builder | Evaluates and builds agent config PRs. Heavy. | `deploy.builder.enable` | [Builder](#builder) |
|
|
| Binary cache | Serves built closures to the hives. | `deploy.attic.enable` | [Binary cache](#binary-cache) |
|
|
| Backup archive | Keeps every agent's state backups, one repository each. | `deploy.archive.enable` | [Backup archive](archive.md) |
|
|
| CI metrics | Records CI run times in the metrics store. | `deploy.ci-metrics.enable` | [CI metrics](#ci-metrics) |
|
|
| Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) |
|
|
| Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) |
|
|
| Notification router | Turns forge events into notifications for agents. | `deploy.notify-router.enable` | [Notification router](#notification-router) |
|
|
| Speech to text | Turns voice memos into text for agents. | `deploy.stt.enable` | [Speech to text](#speech-to-text) |
|
|
|
|
### SSO (authelia)
|
|
|
|
One authelia per swarm, in a `swarm-authelia` container at
|
|
`auth.<swarm-domain>`. The operator and the agents are both subjects of the
|
|
same provider, told apart by roles and claims. Set `deploy.authelia.enable` to
|
|
run it here. Every hive sets `swarm.authelia.url`, the same value on all of
|
|
them: `https://<swarm.authelia.domain>` by default, or the other name when you
|
|
join a swarm whose IdP lives elsewhere. Its keys live in the secret store, and
|
|
nothing rotates them automatically: replacing one invalidates the sessions and
|
|
the encrypted store, so that's an action you take on purpose.
|
|
|
|
Bootstrapping the first user and the login flow: [`sso.md`](sso.md). The
|
|
container, its sockets, secrets and metrics:
|
|
[Host setup](../../4-implementation/swarm/sso.md#host-setup).
|
|
|
|
### Metrics (VictoriaMetrics + Grafana)
|
|
|
|
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it,
|
|
at `metrics.<swarm-domain>` and `grafana.<swarm-domain>`. They sit in two
|
|
containers so you can restart or break Grafana without losing the time-series
|
|
database. Enabling either enables both; to run one alone, set it directly with
|
|
`deploy.victoriametrics.enable` or `deploy.grafana.enable`.
|
|
|
|
Log in to Grafana with your swarm SSO account. Grafana provisions its
|
|
dashboards from the repo, so edits made in the UI aren't kept. Each service has
|
|
its own board; open the folder to see them.
|
|
|
|
⚠️ **The metrics store grows for as long as the swarm runs.** Options to touch:
|
|
|
|
| Option | When you'd touch it |
|
|
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you know how fast the swarm fills a disk. A too-short value discards history you can't get back. |
|
|
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower it to `Viewer` or `Editor` if some operators shouldn't reconfigure Grafana. |
|
|
|
|
The dashboard list, the "No data" cases, and how the gateway guards the store:
|
|
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#metrics-victoriametrics--grafana).
|
|
|
|
### Logs (VictoriaLogs)
|
|
|
|
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
|
|
behind the same SSO as everything else: the swarm's service containers, the
|
|
hive's daemons and the harness units in every agent container.
|
|
|
|
To read them, open Grafana, pick **Explore** and choose the `VictoriaLogs`
|
|
datasource. Agents query it with `swarm-logs`, and need the `logs` permission.
|
|
|
|
| Option | When you'd touch it |
|
|
| ------------------------------------- | ----------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's. Raise it to answer questions about last quarter. |
|
|
|
|
The other options, who may query, and the gateway rules:
|
|
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#logs-victorialogs).
|
|
|
|
### Telemetry collector (OTEL)
|
|
|
|
The swarm's collector receives from every hive's own collector and decides where
|
|
telemetry goes. It always writes the metrics and log stores, and exports to
|
|
`deploy.swarm-otel.upstream.endpoint` as well when you set one. It holds the
|
|
upstream credential, so no hive and no agent needs one. Every hive reaches it
|
|
by its gateway name, `otel.<swarm-domain>`, so a hive that doesn't run the
|
|
swarm's services needs nothing set.
|
|
|
|
How the collector connects, its per-service routes and the homeserver's
|
|
stand-in metrics:
|
|
[`telemetry.md`](../../4-implementation/swarm/telemetry.md#telemetry-collector-otel).
|
|
|
|
### CI runner
|
|
|
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
|
runner executes pull-request code, so running one is a decision of its own. It
|
|
needs neither the forge nor a hive on its host, and registers itself with the
|
|
forge. Its **ci runner** board in Grafana shows its jobs and capacity, and the
|
|
forge's **CI runs** tab shows the run times of every run. Registration, the
|
|
metrics it exports and the threat model:
|
|
[Host setup](../../4-implementation/scheduler/ci.md#host-setup) and the rest of
|
|
[`ci.md`](../../4-implementation/scheduler/ci.md).
|
|
|
|
### Builder
|
|
|
|
The swarm builder, in a `swarm-builder` container on the host that sets
|
|
`deploy.builder.enable`; set it to `false` to opt out. It evaluates and builds
|
|
unreviewed agent config PRs, and the result shows as a `swarm-builder` commit
|
|
status on the PR. A red status blocks nothing: the merge still deploys. It also
|
|
builds each agent before the swarm sends its deploy, and a failed build holds
|
|
that deploy back. The jobs show on the swarm UI's Jobs page. It builds through
|
|
its host's nix daemon, so it costs real CPU and disk.
|
|
[Host setup](../../4-implementation/swarm/builder.md#host-setup), and what it
|
|
builds and why: [`builder.md`](../../4-implementation/swarm/builder.md).
|
|
|
|
### Binary cache
|
|
|
|
The swarm's attic server, in a `swarm-attic` container on the host that sets
|
|
`deploy.attic.enable`. The builder pushes green builds to it, and the hives pull
|
|
closures from it at `swarm.attic.domain` (default `attic.<swarm-domain>`). For
|
|
an external cache, override `swarm.attic.{domain,cache}` and don't set
|
|
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
|
|
`allSwarmServices`. Its keys and tokens create themselves in the secret store.
|
|
How it works:
|
|
[Binary cache host setup](../../4-implementation/swarm/builder.md#binary-cache-host-setup)
|
|
and [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache).
|
|
|
|
### CI metrics
|
|
|
|
The CI run poller, a `swarm-ci-metrics` timer unit on the host that sets
|
|
`deploy.ci-metrics.enable`; set it to `false` to opt out. Every five minutes it
|
|
pushes the run time and queue wait of each newly finished forge Actions run to
|
|
the metrics store. What it reads and pushes:
|
|
[`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md).
|
|
|
|
### Review classifier
|
|
|
|
The review classifier's try-out service, in a `swarm-review-classifier`
|
|
container on the host that sets `deploy.review-classifier.enable`. You set the
|
|
key for the runtime the swarm settings choose in the swarm UI: the **classifier
|
|
settings** card on `/jobs` takes a key for that runtime and shows whether one
|
|
exists and when it changed, never the key. "Set" means a key exists, not that it
|
|
works, and every run spends money on that key; until the chosen one exists,
|
|
every run fails with `no API key…` or `no ACP provider key…`. The service reads
|
|
the key on every run, so writing or removing it takes effect on the next run.
|
|
The store paths for the two keys, and what a run is and costs:
|
|
[Host setup](../../4-implementation/swarm/review-classifier.md#host-setup) and
|
|
[`review-classifier.md`](../../4-implementation/swarm/review-classifier.md).
|
|
|
|
### Claude login minter
|
|
|
|
The claude login minter, in a `swarm-claude-login` container on the host that
|
|
sets `deploy.claude-login.enable`. Run one: a login in progress lives in the
|
|
service's memory between its start and its code. An operator logs an agent in
|
|
from the **log in** button on its card in the swarm UI, and the agent's
|
|
harness installs the new login. The flow and the version rule:
|
|
[`claude-login.md`](claude-login.md). The service's host setup:
|
|
[Host setup](../../4-implementation/swarm/claude-login.md#host-setup).
|
|
|
|
### Notification router
|
|
|
|
The notification router, in a `swarm-notify-router` container on the host that
|
|
sets `deploy.notify-router.enable`. Run one: it handles each issue's events in
|
|
order. It turns forge events (assigning an issue or pull request, closing it, a
|
|
failed CI run on a pull request) into todos for the agents they concern, and an
|
|
agent that's down gets its todos when it's back. The flows and the delivery
|
|
path:
|
|
[`notify-router.md`](../../4-implementation/swarm/notify-router.md), and
|
|
[Host setup](../../4-implementation/swarm/notify-router.md#host-setup).
|
|
|
|
### Speech to text
|
|
|
|
The speech-to-text service, in a `swarm-stt` container on the host that sets
|
|
`deploy.stt.enable`. Run it where the inference hardware is: it transcribes
|
|
with whisper.cpp. An agent that receives a voice memo sends the audio to it
|
|
over the queue and gets the text back, and a memo sent to several agents costs
|
|
one run. `deploy.stt.settings.modelSize` picks the model (`small` by default,
|
|
or `base`). The audio comes from rooms with outside people, so the service runs
|
|
under a tight sandbox, but its network isn't restricted, which the design page
|
|
spells out. The settings, limits and reasons:
|
|
[`stt.md`](../../4-implementation/swarm/stt.md) and
|
|
[Host setup](../../4-implementation/swarm/stt.md#host-setup).
|