mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: swarm/services.md is one blurb per service, the host detail moves to the service's implementation page

Browse source 
Each service section lost its container, credential and metrics detail to a
Host setup section on its own implementation page, unchanged apart from links.
The four agentAuth option rows that sat above the page title moved into the
telemetry page's metrics option table.
This commit is contained in:
iris 2026-11-14 12:39:40 +01:00 • committed by mara
commit 814edd1d67
10 changed files with 392 additions and 308 deletions
Download patch file Download diff file Expand all files Collapse all files

414
docs/2-handbook/swarm/services.md
View file

@ -1,15 +1,11 @@
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. |
| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |
# Swarm-wide services # Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge, Some things exist once per **swarm** rather than once per hive: the forge,
the matrix homeserver, SSO, the secret store, the queue, and the metrics and the matrix homeserver, SSO, the secret store, the queue, and the metrics and
log stack. This page says which host runs them and what a hive that runs none log stack. This page says which host runs them, what a hive that runs none of
of them configures instead. The all-local quick start sets everything with one them configures instead, and what each service is for. The all-local quick
line → [README](../../../README.md#quick-start-an-all-local-swarm). start sets everything with one line →
[README](../../../README.md#quick-start-an-all-local-swarm).
Two options say where they live, and everything else derives: Two options say where they live, and everything else derives:
@ -20,42 +16,38 @@ services.hyperhive.deploy.allSwarmServices = true;
``` ```
**`deploy.allSwarmServices` is what "the swarm's shared services run **`deploy.allSwarmServices` is what "the swarm's shared services run
here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service here" means: every once-per-swarm service takes its `enable` from it.** The
sections below don't repeat it, so a service that stops deriving is a service sections below don't repeat that. A service that needs the secret
visible difference rather than one more paragraph saying the same thing. store comes on only where the host has the service's store identity, so a
services host with the store elsewhere gets it once it holds a copy of that
identity; set the service's own `enable` to override either way.
`singleHostSwarm` is the all-on-one-box mode above it. It defaults `singleHostSwarm` is the all-on-one-box mode above it. It defaults
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the `deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`),
host's `/etc/hosts` entries for the names it serves the host's `/etc/hosts` entries for the names it serves
(`gateway.localHostsEntry`) and where the secret store's bootstrap (`gateway.localHostsEntry`) and where the secret store's bootstrap token goes
token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived (`deploy.bao.bootstrapTokenFile`). You can still set each derived toggle on
toggle on its own, and that setting wins. its own, and that setting wins.
**Both default to off**, and that's deliberate: a host can't tell **Both default to off**, because a host can't tell whether it's meant to be
whether it's meant to be the swarm's service host, so this is an the swarm's service host. With them off, a hive is a _client_ of those
operator saying so rather than something inferred. With them off, a hive services: it configures how to reach them and runs none of them.
is a _client_ of those services — it configures how to reach them and
runs none of them.
That includes the forge (`deploy.forgejo.enable`). Every swarm needs That includes the forge (`deploy.forgejo.enable`). Every swarm needs one,
one, being the canonical store for the meta flake and every agent's being the canonical store for the meta flake and every agent's config repo,
config repo, so **exactly one host must turn it on**: `singleHostSwarm`, so **exactly one host must turn it on**: `singleHostSwarm`,
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A `allSwarmServices`, or `deploy.forgejo.enable = true` set by hand.
host that enables its services one by one without any of those runs no
forge.
A hive can't start or stop any of these. `hivectl stop`, `start` and A hive can't start or stop any of these. `hivectl stop`, `start` and
`restart` act on the hive's agents and its gateway only, even on the `restart` act on the hive's agents and its gateway only, even on the host that
host that runs the swarm's services. The forge and matrix containers are runs the swarm's services. The forge and matrix containers are systemd units
systemd units on that host (`container@hive-forge.service`, on that host (`container@hive-forge.service`,
`container@hive-matrix.service`), and the operator manages them there `container@hive-matrix.service`), and you manage them there with `systemctl`.
with `systemctl`.
A hive that runs none of these reaches each one by name — the forge at A hive that runs none of these reaches each one by name, the forge at
`forge.<swarm-domain>`, for example. Only the host running a service `forge.<swarm-domain>` for example. Only the host running a service answers its
answers its name from its own resolver, so on a swarm spread over name from its own resolver, so on a swarm spread over more than one host, your
more than one host, the operator's DNS has to resolve those names to that DNS has to resolve those names to that host.
host.
<details><summary>Moving an existing hive's forge to the swarm's</summary> <details><summary>Moving an existing hive's forge to the swarm's</summary>
@ -67,46 +59,32 @@ forge: push anything worth keeping there by hand.
## Deployment shapes ## Deployment shapes
Those two options are what makes the difference between deployments, so - **All-local.** Everything on one machine: `singleHostSwarm = true`, plus
the shapes worth naming are the ones they produce: `deploy.hive-controller.enable = true` for a hive to run agents on. After the
first switch, the steps in [`setup.md`](../../1-getting-started/setup.md)
- **All-local.** Everything on one machine: remain.
`singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for - **Services on the swarm controller host.** Set `deploy.allSwarmServices` and
a hive to run agents on. After the first switch, the steps in `deploy.swarm-controller.enable` there, with hives elsewhere. The controller
[`setup.md`](../../1-getting-started/setup.md) remain. doesn't derive from `allSwarmServices`.
- **Services on the swarm controller host.** - **Fully spread out.** One container, VM or machine per service.
Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there,
with hives elsewhere. The controller doesn't derive from
`allSwarmServices`.
- **Fully spread out.** One container / VM / machine per service,
somewhere.
- **Semi-spread.** The services split over a few machines by weight, hives on - **Semi-spread.** The services split over a few machines by weight, hives on
their own, a VPN between them, and most vhosts answering on the VPN only. their own, a VPN between them, and most vhosts answering on the VPN only. A
A worked example with its NixOS files: worked example with its NixOS files:
[`semi-spread-deploy.md`](semi-spread-deploy.md). [`semi-spread-deploy.md`](semi-spread-deploy.md).
**These are a set, not a ladder with a correct top, and hyperhive These are a set, not a ladder with a correct top, and in-between shapes work:
supports in-between shapes.** You can set each derived toggle on its own (see you can set each derived toggle on its own, which makes "all local except X" a
above), which is what makes "all local except X" a configuration rather configuration rather than an unsupported edge case. Co-location is normal, and
than an unsupported edge case. Nothing in hyperhive prescribes a where one service has something to say about sharing a host, that belongs in
deployment model, so a doc that treats one shape as the real one and the service's own doc.
the others as compromises is wrong about the system rather than
opinionated.
The practical consequence is that **co-location is normal**: in the
first two shapes services share a host by design. Where a specific
service has something to say about sharing a host --- extra
requirements, or a cost worth weighing --- that belongs in the doc for
that service, not here.
## Services ## Services
An operator who isn't spreading the swarm over more than one host can skip most of An operator who isn't spreading the swarm over more than one host can skip most
this page: `singleHostSwarm` turns on the shared services below, with two of this page: `singleHostSwarm` turns on the shared services below, with two
exceptions. The **CI runner** stays off until you set `deploy.ci-runner.enable`, exceptions. The **CI runner** stays off until you set `deploy.ci-runner.enable`,
and the **builder** and **binary cache** come on only where the host has their and the **builder** and **binary cache** come on only where the host has their
store identities. The table says what each one does and where its option and store identities. The builder and the CI runner are the two that cost real
its section are. The builder and the CI runner are the two that cost real
resources (they build and run pull-request code), so they're the two worth a resources (they build and run pull-request code), so they're the two worth a
look before you switch them on. look before you switch them on.
@ -128,64 +106,22 @@ look before you switch them on.
| Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) | | Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) |
| Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) | | Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) |
| Notification router | Turns forge events into notifications for agents. | `deploy.notify-router.enable` | [Notification router](#notification-router) | | Notification router | Turns forge events into notifications for agents. | `deploy.notify-router.enable` | [Notification router](#notification-router) |
| Speech to text | Turns voice memos into text for agents. | `deploy.stt.enable` | [Speech to text](#notification-router) | | Speech to text | Turns voice memos into text for agents. | `deploy.stt.enable` | [Speech to text](#speech-to-text) |
### SSO (authelia) ### SSO (authelia)
One authelia per swarm, in a `swarm-authelia` container, at One authelia per swarm, in a `swarm-authelia` container at
`auth.<swarm-domain>`. Operator and agents are both subjects of the same `auth.<swarm-domain>`. The operator and the agents are both subjects of the
provider, differentiated by roles and claims rather than by mechanism — same provider, told apart by roles and claims. Set `deploy.authelia.enable` to
there is one IdP and one auth path. run it here. Every hive sets `swarm.authelia.url`, the same value on all of
them: `https://<swarm.authelia.domain>` by default, or the other name when you
join a swarm whose IdP lives elsewhere. Its keys live in the secret store, and
nothing rotates them automatically: replacing one invalidates the sessions and
the encrypted store, so that's an action you take on purpose.
The container sits on the hive bridge in a network namespace of its own and Bootstrapping the first user and the login flow: [`sso.md`](sso.md). The
dials the store, the collector and its own public name through it. It binds container, its sockets, secrets and metrics:
no TCP port: the login server, the metrics listener and the bridge each [Host setup](../../4-implementation/swarm/sso.md#host-setup).
listen on a unix socket in `/run/swarm-authelia`, a directory only the
gateway's nginx and the container's processes can open.
- **`deploy.authelia`** — run the container here.
- **`swarm.authelia.url`** — where clients go to authenticate. Present
on **every** hive and the same value on all of them:
`https://<swarm.authelia.domain>`, whether or not this host runs the
container. Set it explicitly when joining a swarm whose IdP is under
another name.
Agent subjects come from swarm-controller's agent-creation job, written
into the users database by `swarm-authelia-bridge`; human ones come from
`swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account).
The bridge listens on a unix socket on the authelia host, and the gateway
publishes it as
`https://<swarm.authelia.domain>/bridge/`, which is where
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
It answers only swarm-controller's OIDC client
(`swarm.controller.queueClientId`) and refuses every other token `403`.
On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one
invalidates data already written (sessions, the encrypted store), so
that's an operator action.
Storage is local sqlite and the notifier writes to a file. Both are
small-deployment choices, and the scope is the justification: redis
buys shared session state across replicas and there is one instance;
SMTP exists to mail humans, and provisioning here is programmatic.
The swarm collector scrapes authelia's own metrics at
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`.
The container's `authelia-swarm-secrets` unit creates each client secret directly in
the swarm secret store, under the secret publisher's own store leaf handed in as
a system credential, and writes only the digest to disk; see
[`credentials.md`](credentials.md). Authelia starts from its digests, so the IdP
comes up while the store is down, and a unit that needs the store retries every
30 seconds. The container's journal forwarder reads its client secret back from
the store under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`credentials.md`](credentials.md) for where authelia
generates and reads each of its keys.
### Metrics (VictoriaMetrics + Grafana) ### Metrics (VictoriaMetrics + Grafana)
@ -242,204 +178,90 @@ stand-in metrics:
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
runner executes pull-request code, so running one is a decision of its own. runner executes pull-request code, so running one is a decision of its own. It
needs neither the forge nor a hive on its host, and registers itself with the
It needs neither the forge nor a hive on its host. It reaches the forge at forge. Its **ci runner** board in Grafana shows its jobs and capacity, and the
`https://<swarm.forge.domain>`, and registers with the token swarm-controller forge's **CI runs** tab shows the run times of every run. Registration, the
publishes to the secret store, which it reads under its own `ci-runner` leaf. metrics it exports and the threat model:
On a host without the store, that leaf is the one file to copy. Registration, [Host setup](../../4-implementation/scheduler/ci.md#host-setup) and the rest of
the container and the threat model are in [`ci.md`](../../4-implementation/scheduler/ci.md).
[`4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md).
The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in
the container's own network namespace. The container's journal forwarder
scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and
pushes the series to the swarm collector under the job `gitea-runner`, so
nothing listens on the bridge. Series, all prefixed `gitea_runner_`:
`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`,
`info` (labels `version` and `name`), `job_total{status}`,
`job_duration_seconds`, `poll_fetch_total{result}`,
`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`,
`report_state_*` and `client_errors_total`. A pull-request job runs in the same
namespace and can read the endpoint, which carries counts, durations, the
runner's name and its version, and no secrets.
The **ci runner** board in Grafana shows these series. Its **Runner** filter
(the series' `service.instance.id`, the scrape address) narrows every panel,
and uptime, job slots and slot use show one series per runner, so a runner
registered by hand appears beside the swarm's once its metrics reach the
store under its own instance id. The forge's **CI runs** tab is the other
view: run times and queue wait of every run, whichever runner ran it.
### Builder ### Builder
The swarm builder, in a `swarm-builder` container on the host that sets The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host `deploy.builder.enable`; set it to `false` to opt out. It evaluates and builds
has the builder's store identity, as for the review classifier below: the host unreviewed agent config PRs, and the result shows as a `swarm-builder` commit
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It status on the PR. A red status blocks nothing: the merge still deploys. It also
evaluates unreviewed config PRs and builds them through its host's nix daemon, builds each agent before the swarm sends its deploy, and a failed build holds
from a container with its own network namespace. that deploy back. The jobs show on the swarm UI's Jobs page. It builds through
its host's nix daemon, so it costs real CPU and disk.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push, [Host setup](../../4-implementation/swarm/builder.md#host-setup), and what it
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a builds and why: [`builder.md`](../../4-implementation/swarm/builder.md).
newer push replaces its PR's job while that job is still waiting. The builder
builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job;
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). Those jobs share the one-build slot with
the PR checks.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](../../4-implementation/swarm/builder.md).
### Binary cache ### Binary cache
The swarm's attic server, in a `swarm-attic` container on the host that sets The swarm's attic server, in a `swarm-attic` container on the host that sets
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has `deploy.attic.enable`. The builder pushes green builds to it, and the hives pull
the cache's store identity, as for the builder: the host that runs the store has closures from it at `swarm.attic.domain` (default `attic.<swarm-domain>`). For
it. For an external cache, override `swarm.attic.{domain,cache}` and don't set an external cache, override `swarm.attic.{domain,cache}` and don't set
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under `deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
`allSwarmServices`. `allSwarmServices`. Its keys and tokens create themselves in the secret store.
How it works:
The container sits on the hive bridge in a network namespace of its own, and atticd [Binary cache host setup](../../4-implementation/swarm/builder.md#binary-cache-host-setup)
listens on that namespace's loopback only; the gateway reaches it over a unix socket. and [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache).
The gateway serves it at `swarm.attic.domain` (default
`attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
its own tokens. Storage is local disk in the container's state.
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
create its signing key, the cache and the push and pull tokens in the secret
store, each only while absent. On a host without the store, that leaf is the one
file to copy. The builder pushes green builds to it. How it works, and the
external alternative, are in [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache).
The container forwards atticd's journal to the swarm collector like the other
service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host
units' lines reach the log store too: through the hive collector's journal
allow-list on a hive host, or through the swarm collector's whole-journal
receiver where the host runs that. A host with neither keeps them in its own
journal.
### CI metrics ### CI metrics
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host The CI run poller, a `swarm-ci-metrics` timer unit on the host that sets
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container. `deploy.ci-metrics.enable`; set it to `false` to opt out. Every five minutes it
`allSwarmServices` defaults it on wherever the host has the poller's store pushes the run time and queue wait of each newly finished forge Actions run to
identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to the metrics store. What it reads and pushes:
opt out. [`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md).
Each pass pushes the run time and queue wait of every newly finished forge
Actions run to the metrics store, through its own route on the swarm collector;
the first pass pushes every run since the first. It reads the forge with the
read-only token of its own `ci-metrics` forge account, under its own
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
copy. What it reads and pushes are in [`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md).
### Review classifier ### Review classifier
The review classifier's try-out service, in a `swarm-review-classifier` The review classifier's try-out service, in a `swarm-review-classifier`
container on the host that sets `deploy.review-classifier.enable`. container on the host that sets `deploy.review-classifier.enable`. You set the
`allSwarmServices` defaults it on wherever the host has the classifier's store key for the runtime the swarm settings choose in the swarm UI: the **classifier
identity: the host that runs the store has it, and a services host with the settings** card on `/jobs` takes a key for that runtime and shows whether one
store elsewhere gets the classifier only once it holds a copy of the pair (see exists and when it changed, never the key. "Set" means a key exists, not that it
below). Set `deploy.review-classifier.enable` directly to override either way. works, and every run spends money on that key; until the chosen one exists,
The operator sets the key of the runtime the swarm settings choose in the
swarm UI: the **classifier settings** card on `/jobs` takes a key for the
chosen runtime and shows whether one exists and when it changed, never the
key. "Set" means a key exists, not that it works: a run with a bad key
still fails. You can also write the same two keys in the secret store's UI under the
OIDC login, field `api_key`:
- the API key runtime's Anthropic key at
`secret/swarm/services/review-classifier/anthropic-api-key`
- the ACP runtime's provider key at
`secret/swarm/services/review-classifier/acp-provider`
Nothing writes them for you: until the chosen one exists the service runs and
every run fails with `no API key…` or `no ACP provider key…`. The service reads every run fails with `no API key…` or `no ACP provider key…`. The service reads
the key on every run, so writing or removing it takes effect on the next run, the key on every run, so writing or removing it takes effect on the next run.
with no restart. Every run spends money on that key. The store paths for the two keys, and what a run is and costs:
[Host setup](../../4-implementation/swarm/review-classifier.md#host-setup) and
It needs neither the forge nor a hive on its host. It reaches the queue and the [`review-classifier.md`](../../4-implementation/swarm/review-classifier.md).
secret store by name, under its own `review-classifier` leaf. On a host
without the store, that leaf is the one file to copy, as for the CI runner.
The swarm UI's try-out drives it through swarm-controller; what a run is
and what it costs are in [`review-classifier.md`](../../4-implementation/swarm/review-classifier.md).
### Claude login minter ### Claude login minter
The claude login minter, in a `swarm-claude-login` container on the host that The claude login minter, in a `swarm-claude-login` container on the host that
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever sets `deploy.claude-login.enable`. Run one: a login in progress lives in the
the host has the minter's store identity, on the review classifier's terms. service's memory between its start and its code. An operator logs an agent in
Run one: a login in progress lives in the service's memory between its start from the **log in** button on its card in the swarm UI, and the agent's
and its code. harness installs the new login. The flow and the version rule:
[`claude-login.md`](claude-login.md). The service's host setup:
An operator logs an agent in from the **log in** button on its card in the [Host setup](../../4-implementation/swarm/claude-login.md#host-setup).
swarm UI. swarm-controller relays the start and the pasted code over the queue,
the minter runs `claude auth login` and writes the credentials to the agent's
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
version newer than the one it installed last.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `claude-login` leaf, and reaches claude's
login endpoints over the network. On a host without the store, that leaf is
the one file to copy, as for the CI runner. The flow and the version rule are
in [`claude-login.md`](claude-login.md).
### Notification router ### Notification router
The notification router, in a `swarm-notify-router` container on the host The notification router, in a `swarm-notify-router` container on the host that
that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on sets `deploy.notify-router.enable`. Run one: it handles each issue's events in
wherever the host has the router's store identity, on the review classifier's order. It turns forge events (assigning an issue or pull request, closing it, a
terms. Run one: it handles each issue's events in order. failed CI run on a pull request) into todos for the agents they concern, and an
agent that's down gets its todos when it's back. The flows and the delivery
swarm-controller keeps every `issues` and `pull_request` delivery, and every path:
failed or cancelled CI run, from its instance-wide hooks in the `SWARM_EVENTS` [`notify-router.md`](../../4-implementation/swarm/notify-router.md), and
stream. The router reads that [Host setup](../../4-implementation/swarm/notify-router.md#host-setup).
stream and queues todos for the agents an event concerns: an agent assigned to
an issue or pull request gets a todo, one taken off it gets that todo rewritten
to say so, and closing it clears them. The author of a pull request gets a todo
when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes
it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos
from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's
back.
It needs neither the forge nor a hive on its host, and holds no forge token.
It reaches the queue and the secret store by name, under its own
`notify-router` leaf. On a host without the store, that leaf is the one file to
copy, as for the CI runner. The flows and the delivery path are in
[`notify-router.md`](../../4-implementation/swarm/notify-router.md).
### Speech to text ### Speech to text
The speech-to-text service, in a `swarm-stt` container on the host that sets The speech-to-text service, in a `swarm-stt` container on the host that sets
`deploy.stt.enable`. `allSwarmServices` defaults it on wherever the host has the `deploy.stt.enable`. Run it where the inference hardware is: it transcribes
service's store identity, on the router's terms. Run it where the inference with whisper.cpp. An agent that receives a voice memo sends the audio to it
hardware is: it transcribes with whisper.cpp. over the queue and gets the text back, and a memo sent to several agents costs
one run. `deploy.stt.settings.modelSize` picks the model (`small` by default,
An agent that receives a voice memo sends the audio to the service over the or `base`). The audio comes from rooms with outside people, so the service runs
queue, on its own subject, and gets the text back. The service keeps each under a tight sandbox, but its network isn't restricted, which the design page
transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs spells out. The settings, limits and reasons:
one whisper run. `deploy.stt.settings.modelSize` picks the [`stt.md`](../../4-implementation/swarm/stt.md) and
model (`small` by default, or `base`), and `modelFile` takes any other ggml [Host setup](../../4-implementation/swarm/stt.md#host-setup).
model. The cache time, the limits on file size, audio length and run time, and
how many requests it holds at once are settings next to it.
The audio comes from rooms with outside people, so the service runs as its own
unprivileged user under a tight systemd sandbox, with a memory ceiling and
a CPU quota. Its network isn't restricted: the decoder runs in the same
network namespace as the service and can read its credentials, so a decoder
exploit could read them and send them anywhere the container can reach. Closing
that's for the strict-egress work on service containers, or for running the
decode with no network and no credentials. The file's first bytes must look like an audio container
before anything decodes it. Its leaf and its queue client work as the router's
do. The design, the settings and the reasons are in
[`stt.md`](../../4-implementation/swarm/stt.md).

31
docs/4-implementation/scheduler/ci.md
View file

@ -389,3 +389,34 @@ builder — disk-pressure collection at least runs while nothing is mid-copy.
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token. - `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
- `.forgejo/workflows/ci.yml`: workflow definition. - `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers. - `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers.
## Host setup
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
runner executes pull-request code, so running one is a decision of its own.
It needs neither the forge nor a hive on its host. It reaches the forge at
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
publishes to the secret store, which it reads under its own `ci-runner` leaf.
On a host without the store, that leaf is the one file to copy.
The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in
the container's own network namespace. The container's journal forwarder
scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and
pushes the series to the swarm collector under the job `gitea-runner`, so
nothing listens on the bridge. Series, all prefixed `gitea_runner_`:
`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`,
`info` (labels `version` and `name`), `job_total{status}`,
`job_duration_seconds`, `poll_fetch_total{result}`,
`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`,
`report_state_*` and `client_errors_total`. A pull-request job runs in the same
namespace and can read the endpoint, which carries counts, durations, the
runner's name and its version, and no secrets.
The **ci runner** board in Grafana shows these series. Its **Runner** filter
(the series' `service.instance.id`, the scrape address) narrows every panel,
and uptime, job slots and slot use show one series per runner, so a runner
registered by hand appears beside the swarm's once its metrics reach the
store under its own instance id. The forge's **CI runs** tab is the other
view: run times and queue wait of every run, whichever runner ran it.

56
docs/4-implementation/swarm/builder.md
View file

@ -681,3 +681,59 @@ is pure and runs in the container. Builds go to the host's nix daemon as an
untrusted client: a config can set no daemon option, and its derivations build untrusted client: a config can set no daemon option, and its derivations build
in the host's sandbox. A fixed-output derivation still has the network, as it in the host's sandbox. A fixed-output derivation still has the network, as it
does for any build on that host. does for any build on that host.
## Host setup
The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
has the builder's store identity, as for the other swarm services: the host
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It
evaluates unreviewed config PRs and builds them through its host's nix daemon,
from a container with its own network namespace.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
newer push replaces its PR's job while that job is still waiting. The builder
builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job;
[deploy gate](#deploy-gate)). Those jobs share the one-build slot with
the PR checks.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the other swarm services.
## Binary cache host setup
The swarm's attic server, in a `swarm-attic` container on the host that sets
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
the cache's store identity, as for the other swarm services: the host that runs the store has
it. For an external cache, override `swarm.attic.{domain,cache}` and don't set
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
`allSwarmServices`.
The container sits on the hive bridge in a network namespace of its own, and atticd
listens on that namespace's loopback only; the gateway reaches it over a unix socket.
The gateway serves it at `swarm.attic.domain` (default
`attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
its own tokens. Storage is local disk in the container's state.
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
create its signing key, the cache and the push and pull tokens in the secret
store, each only while absent. On a host without the store, that leaf is the one
file to copy. The builder pushes green builds to it. How it works, and the
external alternative, are in [Binary cache](#binary-cache).
The container forwards atticd's journal to the swarm collector like the other
service containers (see [Telemetry collector](telemetry.md#telemetry-collector-otel)). The two host
units' lines reach the log store too: through the hive collector's journal
allow-list on a hive host, or through the swarm collector's whole-journal
receiver where the host runs that. A host with neither keeps them in its own
journal.

15
docs/4-implementation/swarm/ci-metrics.md
View file

@ -101,3 +101,18 @@ variable:
- finished runs per day, by outcome; - finished runs per day, by outcome;
- a table per PR over the time range: mean and longest successful run time, - a table per PR over the time range: mean and longest successful run time,
runs, failed runs. runs, failed runs.
## Host setup
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
`allSwarmServices` defaults it on wherever the host has the poller's store
identity, as for the other swarm services. Set `deploy.ci-metrics.enable = false` to
opt out.
Each pass pushes the run time and queue wait of every newly finished forge
Actions run to the metrics store, through its own route on the swarm collector;
the first pass pushes every run since the first. It reads the forge with the
read-only token of its own `ci-metrics` forge account, under its own
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
copy.

20
docs/4-implementation/swarm/claude-login.md
View file

@ -118,3 +118,23 @@ gauges (`bao.client_cert.*`) come with them.
`invalid` is a request that didn't decode or named no agent. A `start` spawns `invalid` is a request that didn't decode or named no agent. A `start` spawns
`claude auth login` and a `code` stores the credentials, so those two are the `claude auth login` and a `code` stores the credentials, so those two are the
slow steps. `expired` counts logins stopped because their code never came. slow steps. `expired` counts logins stopped because their code never came.
## Host setup
The claude login minter, in a `swarm-claude-login` container on the host that
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever
the host has the minter's store identity, like the other swarm services.
Run one: a login in progress lives in the service's memory between its start
and its code.
An operator logs an agent in from the **log in** button on its card in the
swarm UI. swarm-controller relays the start and the pasted code over the queue,
the minter runs `claude auth login` and writes the credentials to the agent's
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
version newer than the one it installed last.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `claude-login` leaf, and reaches claude's
login endpoints over the network. On a host without the store, that leaf is
the one file to copy, as for the other swarm services. The flow and the version rule are
in [`claude-login.md`](../../2-handbook/swarm/claude-login.md).

23
docs/4-implementation/swarm/notify-router.md
View file

@ -433,3 +433,26 @@ no flow matched. `retried` is one the queue or the store failed on: it comes bac
and counts again. A growing `consumer.pending` means the router is falling and counts again. A growing `consumer.pending` means the router is falling
behind. The router reads the consumer counts at the start of each pull, at most behind. The router reads the consumer counts at the start of each pull, at most
30 seconds apart. 30 seconds apart.
## Host setup
The notification router, in a `swarm-notify-router` container on the host
that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on
wherever the host has the router's store identity, on the review classifier's
terms. Run one: it handles each issue's events in order.
swarm-controller keeps every `issues` and `pull_request` delivery, and every
failed or cancelled CI run, from its instance-wide hooks in the `SWARM_EVENTS`
stream. The router reads that
stream and queues todos for the agents an event concerns: an agent assigned to
an issue or pull request gets a todo, one taken off it gets that todo rewritten
to say so, and closing it clears them. The author of a pull request gets a todo
when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes
it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos
from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's
back.
It needs neither the forge nor a hive on its host, and holds no forge token.
It reaches the queue and the secret store by name, under its own
`notify-router` leaf. On a host without the store, that leaf is the one file to
copy, as for the other swarm services.

35
docs/4-implementation/swarm/review-classifier.md
View file

@ -194,7 +194,7 @@ own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
there would offer the model tools again. there would offer the model tools again.
The service is `swarm-review-classifier`, and its container is the one The service is `swarm-review-classifier`, and its container is the one
[`services.md`](../../2-handbook/swarm/services.md#review-classifier) describes. It holds no forge [Host setup](#host-setup) describes. It holds no forge
credential: `swarm-controller` reads the PR and sends it along. credential: `swarm-controller` reads the PR and sends it along.
### Swarm-level settings ### Swarm-level settings
@ -315,7 +315,7 @@ deleted, still shows as set, because the controller can't read a value to check.
The classifier reads the leaf on every run, so a new key applies from the next The classifier reads the leaf on every run, so a new key applies from the next
run. The operator can still write the leaves in the secret store's UI run. The operator can still write the leaves in the secret store's UI
([`services.md`](../../2-handbook/swarm/services.md#review-classifier)). ([Host setup](#host-setup)).
The per-agent equivalent is [`inference.md`](inference.md), where the The per-agent equivalent is [`inference.md`](inference.md), where the
controller writes model and effort to a queue KV bucket controller writes model and effort to a queue KV bucket
@ -765,3 +765,34 @@ These get settled, and filled in here, as the work reaches them:
`secret/swarm/services/review-classifier/forge/token`. `secret/swarm/services/review-classifier/forge/token`.
- **Reviewer handoff.** How a later reviewer, human or agent, learns the - **Reviewer handoff.** How a later reviewer, human or agent, learns the
classifier passed a PR before it starts its own review. classifier passed a PR before it starts its own review.
## Host setup
The review classifier's try-out service, in a `swarm-review-classifier`
container on the host that sets `deploy.review-classifier.enable`.
`allSwarmServices` defaults it on wherever the host has the classifier's store
identity: the host that runs the store has it, and a services host with the
store elsewhere gets the classifier only once it holds a copy of the pair (see
below). Set `deploy.review-classifier.enable` directly to override either way.
The operator sets the key of the runtime the swarm settings choose in the
swarm UI: the **classifier settings** card on `/jobs` takes a key for the
chosen runtime and shows whether one exists and when it changed, never the
key. "Set" means a key exists, not that it works: a run with a bad key
still fails. You can also write the same two keys in the secret store's UI under the
OIDC login, field `api_key`:
- the API key runtime's Anthropic key at
`secret/swarm/services/review-classifier/anthropic-api-key`
- the ACP runtime's provider key at
`secret/swarm/services/review-classifier/acp-provider`
Nothing writes them for you: until the chosen one exists the service runs and
every run fails with `no API key…` or `no ACP provider key…`. The service reads
the key on every run, so writing or removing it takes effect on the next run,
with no restart. Every run spends money on that key.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `review-classifier` leaf. On a host
without the store, that leaf is the one file to copy, as for the other swarm services.
The swarm UI's try-out drives it through swarm-controller.

57
docs/4-implementation/swarm/sso.md
View file

@ -184,3 +184,60 @@ upstream and compare three requests, not one:
All three matter. A change that silently deleted the browser page would All three matter. A change that silently deleted the browser page would
pass a deny-only check, and one that quietly stopped denying would pass a pass a deny-only check, and one that quietly stopped denying would pass a
page-only check. The person who introduced the split verified it this way. page-only check. The person who introduced the split verified it this way.
## Host setup
One authelia per swarm, in a `swarm-authelia` container, at
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
provider, differentiated by roles and claims rather than by mechanism —
there is one IdP and one auth path.
The container sits on the hive bridge in a network namespace of its own and
dials the store, the collector and its own public name through it. It binds
no TCP port: the login server, the metrics listener and the bridge each
listen on a unix socket in `/run/swarm-authelia`, a directory only the
gateway's nginx and the container's processes can open.
- **`deploy.authelia`** — run the container here.
- **`swarm.authelia.url`** — where clients go to authenticate. Present
on **every** hive and the same value on all of them:
`https://<swarm.authelia.domain>`, whether or not this host runs the
container. Set it explicitly when joining a swarm whose IdP is under
another name.
Agent subjects come from swarm-controller's agent-creation job, written
into the users database by `swarm-authelia-bridge`; human ones come from
`swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account).
The bridge listens on a unix socket on the authelia host, and the gateway
publishes it as
`https://<swarm.authelia.domain>/bridge/`, which is where
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
It answers only swarm-controller's OIDC client
(`swarm.controller.queueClientId`) and refuses every other token `403`.
On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one
invalidates data already written (sessions, the encrypted store), so
that's an operator action.
Storage is local sqlite and the notifier writes to a file. Both are
small-deployment choices, and the scope is the justification: redis
buys shared session state across replicas and there is one instance;
SMTP exists to mail humans, and provisioning here is programmatic.
The swarm collector scrapes authelia's own metrics at
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`.
The container's `authelia-swarm-secrets` unit creates each client secret directly in
the swarm secret store, under the secret publisher's own store leaf handed in as
a system credential, and writes only the digest to disk; see
[`credentials.md`](../../2-handbook/swarm/credentials.md). Authelia starts from its digests, so the IdP
comes up while the store is down, and a unit that needs the store retries every
30 seconds. The container's journal forwarder reads its client secret back from
the store under the same leaf, and is off where the container has none.
See [`sso.md`](../../2-handbook/swarm/sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`credentials.md`](../../2-handbook/swarm/credentials.md) for where authelia
generates and reads each of its keys.

25
docs/4-implementation/swarm/stt.md
View file

@ -81,3 +81,28 @@ set them.
Two things the daemon doesn't do yet: sweep scratch directories a crash left Two things the daemon doesn't do yet: sweep scratch directories a crash left
behind, and set whisper's thread count. The issue 6383 discussion has the behind, and set whisper's thread count. The issue 6383 discussion has the
choices behind the defaults. choices behind the defaults.
## Host setup
The speech-to-text service, in a `swarm-stt` container on the host that sets
`deploy.stt.enable`. `allSwarmServices` defaults it on wherever the host has the
service's store identity, like the other swarm services. Run it where the inference
hardware is: it transcribes with whisper.cpp.
An agent that receives a voice memo sends the audio to the service over the
queue, on its own subject, and gets the text back. The service keeps each
transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs
one whisper run. `deploy.stt.settings.modelSize` picks the
model (`small` by default, or `base`), and `modelFile` takes any other ggml
model. The cache time, the limits on file size, audio length and run time, and
how many requests it holds at once are settings next to it.
The audio comes from rooms with outside people, so the service runs as its own
unprivileged user under a tight systemd sandbox, with a memory ceiling and
a CPU quota. Its network isn't restricted: the decoder runs in the same
network namespace as the service and can read its credentials, so a decoder
exploit could read them and send them anywhere the container can reach. Closing
that's for the strict-egress work on service containers, or for running the
decode with no network and no credentials. The file's first bytes must look like an audio container
before anything decodes it. Its leaf and its queue client work as the notification router's
do.

16
docs/4-implementation/swarm/telemetry.md
View file

@ -31,12 +31,16 @@ services.hyperhive.deploy = {
⚠️ **This starts a database that grows for as long as the swarm runs.** ⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default. See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it | | Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. | | `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. | | `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. | | `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. | | `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. |
| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->