diff --git a/docs/2-handbook/swarm/services.md b/docs/2-handbook/swarm/services.md index 84f229275..e9a4679c2 100644 --- a/docs/2-handbook/swarm/services.md +++ b/docs/2-handbook/swarm/services.md @@ -1,15 +1,11 @@ -| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. | -| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. | -| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. | -| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. | - # Swarm-wide services Some things exist once per **swarm** rather than once per hive: the forge, the matrix homeserver, SSO, the secret store, the queue, and the metrics and -log stack. This page says which host runs them and what a hive that runs none -of them configures instead. The all-local quick start sets everything with one -line → [README](../../../README.md#quick-start-an-all-local-swarm). +log stack. This page says which host runs them, what a hive that runs none of +them configures instead, and what each service is for. The all-local quick +start sets everything with one line → +[README](../../../README.md#quick-start-an-all-local-swarm). Two options say where they live, and everything else derives: @@ -20,42 +16,38 @@ services.hyperhive.deploy.allSwarmServices = true; ``` **`deploy.allSwarmServices` is what "the swarm's shared services run -here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service -sections below don't repeat it, so a service that stops deriving is a -visible difference rather than one more paragraph saying the same thing. +here" means: every once-per-swarm service takes its `enable` from it.** The +service sections below don't repeat that. A service that needs the secret +store comes on only where the host has the service's store identity, so a +services host with the store elsewhere gets it once it holds a copy of that +identity; set the service's own `enable` to override either way. `singleHostSwarm` is the all-on-one-box mode above it. It defaults -`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the -host's `/etc/hosts` entries for the names it serves -(`gateway.localHostsEntry`) and where the secret store's bootstrap -token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived -toggle on its own, and that setting wins. +`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), +the host's `/etc/hosts` entries for the names it serves +(`gateway.localHostsEntry`) and where the secret store's bootstrap token goes +(`deploy.bao.bootstrapTokenFile`). You can still set each derived toggle on +its own, and that setting wins. -**Both default to off**, and that's deliberate: a host can't tell -whether it's meant to be the swarm's service host, so this is an -operator saying so rather than something inferred. With them off, a hive -is a _client_ of those services — it configures how to reach them and -runs none of them. +**Both default to off**, because a host can't tell whether it's meant to be +the swarm's service host. With them off, a hive is a _client_ of those +services: it configures how to reach them and runs none of them. -That includes the forge (`deploy.forgejo.enable`). Every swarm needs -one, being the canonical store for the meta flake and every agent's -config repo, so **exactly one host must turn it on**: `singleHostSwarm`, -`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A -host that enables its services one by one without any of those runs no -forge. +That includes the forge (`deploy.forgejo.enable`). Every swarm needs one, +being the canonical store for the meta flake and every agent's config repo, +so **exactly one host must turn it on**: `singleHostSwarm`, +`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A hive can't start or stop any of these. `hivectl stop`, `start` and -`restart` act on the hive's agents and its gateway only, even on the -host that runs the swarm's services. The forge and matrix containers are -systemd units on that host (`container@hive-forge.service`, -`container@hive-matrix.service`), and the operator manages them there -with `systemctl`. +`restart` act on the hive's agents and its gateway only, even on the host that +runs the swarm's services. The forge and matrix containers are systemd units +on that host (`container@hive-forge.service`, +`container@hive-matrix.service`), and you manage them there with `systemctl`. -A hive that runs none of these reaches each one by name — the forge at -`forge.`, for example. Only the host running a service -answers its name from its own resolver, so on a swarm spread over -more than one host, the operator's DNS has to resolve those names to that -host. +A hive that runs none of these reaches each one by name, the forge at +`forge.` for example. Only the host running a service answers its +name from its own resolver, so on a swarm spread over more than one host, your +DNS has to resolve those names to that host.
Moving an existing hive's forge to the swarm's @@ -67,46 +59,32 @@ forge: push anything worth keeping there by hand. ## Deployment shapes -Those two options are what makes the difference between deployments, so -the shapes worth naming are the ones they produce: - -- **All-local.** Everything on one machine: - `singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for - a hive to run agents on. After the first switch, the steps in - [`setup.md`](../../1-getting-started/setup.md) remain. -- **Services on the swarm controller host.** - Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there, - with hives elsewhere. The controller doesn't derive from - `allSwarmServices`. -- **Fully spread out.** One container / VM / machine per service, - somewhere. +- **All-local.** Everything on one machine: `singleHostSwarm = true`, plus + `deploy.hive-controller.enable = true` for a hive to run agents on. After the + first switch, the steps in [`setup.md`](../../1-getting-started/setup.md) + remain. +- **Services on the swarm controller host.** Set `deploy.allSwarmServices` and + `deploy.swarm-controller.enable` there, with hives elsewhere. The controller + doesn't derive from `allSwarmServices`. +- **Fully spread out.** One container, VM or machine per service. - **Semi-spread.** The services split over a few machines by weight, hives on - their own, a VPN between them, and most vhosts answering on the VPN only. - A worked example with its NixOS files: + their own, a VPN between them, and most vhosts answering on the VPN only. A + worked example with its NixOS files: [`semi-spread-deploy.md`](semi-spread-deploy.md). -**These are a set, not a ladder with a correct top, and hyperhive -supports in-between shapes.** You can set each derived toggle on its own (see -above), which is what makes "all local except X" a configuration rather -than an unsupported edge case. Nothing in hyperhive prescribes a -deployment model, so a doc that treats one shape as the real one and -the others as compromises is wrong about the system rather than -opinionated. - -The practical consequence is that **co-location is normal**: in the -first two shapes services share a host by design. Where a specific -service has something to say about sharing a host --- extra -requirements, or a cost worth weighing --- that belongs in the doc for -that service, not here. +These are a set, not a ladder with a correct top, and in-between shapes work: +you can set each derived toggle on its own, which makes "all local except X" a +configuration rather than an unsupported edge case. Co-location is normal, and +where one service has something to say about sharing a host, that belongs in +the service's own doc. ## Services -An operator who isn't spreading the swarm over more than one host can skip most of -this page: `singleHostSwarm` turns on the shared services below, with two +An operator who isn't spreading the swarm over more than one host can skip most +of this page: `singleHostSwarm` turns on the shared services below, with two exceptions. The **CI runner** stays off until you set `deploy.ci-runner.enable`, and the **builder** and **binary cache** come on only where the host has their -store identities. The table says what each one does and where its option and -its section are. The builder and the CI runner are the two that cost real +store identities. The builder and the CI runner are the two that cost real resources (they build and run pull-request code), so they're the two worth a look before you switch them on. @@ -128,64 +106,22 @@ look before you switch them on. | Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) | | Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) | | Notification router | Turns forge events into notifications for agents. | `deploy.notify-router.enable` | [Notification router](#notification-router) | -| Speech to text | Turns voice memos into text for agents. | `deploy.stt.enable` | [Speech to text](#notification-router) | +| Speech to text | Turns voice memos into text for agents. | `deploy.stt.enable` | [Speech to text](#speech-to-text) | ### SSO (authelia) -One authelia per swarm, in a `swarm-authelia` container, at -`auth.`. Operator and agents are both subjects of the same -provider, differentiated by roles and claims rather than by mechanism — -there is one IdP and one auth path. +One authelia per swarm, in a `swarm-authelia` container at +`auth.`. The operator and the agents are both subjects of the +same provider, told apart by roles and claims. Set `deploy.authelia.enable` to +run it here. Every hive sets `swarm.authelia.url`, the same value on all of +them: `https://` by default, or the other name when you +join a swarm whose IdP lives elsewhere. Its keys live in the secret store, and +nothing rotates them automatically: replacing one invalidates the sessions and +the encrypted store, so that's an action you take on purpose. -The container sits on the hive bridge in a network namespace of its own and -dials the store, the collector and its own public name through it. It binds -no TCP port: the login server, the metrics listener and the bridge each -listen on a unix socket in `/run/swarm-authelia`, a directory only the -gateway's nginx and the container's processes can open. - -- **`deploy.authelia`** — run the container here. -- **`swarm.authelia.url`** — where clients go to authenticate. Present - on **every** hive and the same value on all of them: - `https://`, whether or not this host runs the - container. Set it explicitly when joining a swarm whose IdP is under - another name. - -Agent subjects come from swarm-controller's agent-creation job, written -into the users database by `swarm-authelia-bridge`; human ones come from -`swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account). - -The bridge listens on a unix socket on the authelia host, and the gateway -publishes it as -`https:///bridge/`, which is where -`deploy.swarm-controller.authBridgeUrl` points by default on every host. -It answers only swarm-controller's OIDC client -(`swarm.controller.queueClientId`) and refuses every other token `403`. - -On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one -invalidates data already written (sessions, the encrypted store), so -that's an operator action. - -Storage is local sqlite and the notifier writes to a file. Both are -small-deployment choices, and the scope is the justification: redis -buys shared session state across replicas and there is one instance; -SMTP exists to mail humans, and provisioning here is programmatic. - -The swarm collector scrapes authelia's own metrics at -`https:///metrics`, whichever host runs the -container. The gateway admits only the collector's OAuth2 client there; -any other session gets `deny`. - -The container's `authelia-swarm-secrets` unit creates each client secret directly in -the swarm secret store, under the secret publisher's own store leaf handed in as -a system credential, and writes only the digest to disk; see -[`credentials.md`](credentials.md). Authelia starts from its digests, so the IdP -comes up while the store is down, and a unit that needs the store retries every -30 seconds. The container's journal forwarder reads its client secret back from -the store under the same leaf, and is off where the container has none. - -See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC -relying-party flow, and [`credentials.md`](credentials.md) for where authelia -generates and reads each of its keys. +Bootstrapping the first user and the login flow: [`sso.md`](sso.md). The +container, its sockets, secrets and metrics: +[Host setup](../../4-implementation/swarm/sso.md#host-setup). ### Metrics (VictoriaMetrics + Grafana) @@ -242,204 +178,90 @@ stand-in metrics: One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a -runner executes pull-request code, so running one is a decision of its own. - -It needs neither the forge nor a hive on its host. It reaches the forge at -`https://`, and registers with the token swarm-controller -publishes to the secret store, which it reads under its own `ci-runner` leaf. -On a host without the store, that leaf is the one file to copy. Registration, -the container and the threat model are in -[`4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md). - -The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in -the container's own network namespace. The container's journal forwarder -scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and -pushes the series to the swarm collector under the job `gitea-runner`, so -nothing listens on the bridge. Series, all prefixed `gitea_runner_`: -`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`, -`info` (labels `version` and `name`), `job_total{status}`, -`job_duration_seconds`, `poll_fetch_total{result}`, -`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`, -`report_state_*` and `client_errors_total`. A pull-request job runs in the same -namespace and can read the endpoint, which carries counts, durations, the -runner's name and its version, and no secrets. - -The **ci runner** board in Grafana shows these series. Its **Runner** filter -(the series' `service.instance.id`, the scrape address) narrows every panel, -and uptime, job slots and slot use show one series per runner, so a runner -registered by hand appears beside the swarm's once its metrics reach the -store under its own instance id. The forge's **CI runs** tab is the other -view: run times and queue wait of every run, whichever runner ran it. +runner executes pull-request code, so running one is a decision of its own. It +needs neither the forge nor a hive on its host, and registers itself with the +forge. Its **ci runner** board in Grafana shows its jobs and capacity, and the +forge's **CI runs** tab shows the run times of every run. Registration, the +metrics it exports and the threat model: +[Host setup](../../4-implementation/scheduler/ci.md#host-setup) and the rest of +[`ci.md`](../../4-implementation/scheduler/ci.md). ### Builder The swarm builder, in a `swarm-builder` container on the host that sets -`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host -has the builder's store identity, as for the review classifier below: the host -that runs the store has it. Set `deploy.builder.enable = false` to opt out. It -evaluates unreviewed config PRs and builds them through its host's nix daemon, -from a container with its own network namespace. - -When an `agent-configs/` PR opens, reopens or gets a new push, -swarm-controller queues a `build_agent` job. Jobs run one at a time, and a -newer push replaces its PR's job while that job is still waiting. The builder -builds that config at the PR's head on the agent base its own host runs, -through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder` -commit status on the head. The job shows on the swarm UI's Jobs page. A red -status blocks nothing: the merge still deploys. - -Separately, the builder builds each agent from `internal/meta` before the swarm -sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job; -[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). Those jobs share the one-build slot with -the PR checks. - -It needs neither the forge nor a hive on its host. It reaches the queue, the -secret store and the forge by name, under its own `builder` leaf, and fetches -the private config repo with the read-only token of its own `builder` forge -account. On a host without the store, that leaf is the one file to copy, as -for the CI runner. What it builds and why are in [`builder.md`](../../4-implementation/swarm/builder.md). +`deploy.builder.enable`; set it to `false` to opt out. It evaluates and builds +unreviewed agent config PRs, and the result shows as a `swarm-builder` commit +status on the PR. A red status blocks nothing: the merge still deploys. It also +builds each agent before the swarm sends its deploy, and a failed build holds +that deploy back. The jobs show on the swarm UI's Jobs page. It builds through +its host's nix daemon, so it costs real CPU and disk. +[Host setup](../../4-implementation/swarm/builder.md#host-setup), and what it +builds and why: [`builder.md`](../../4-implementation/swarm/builder.md). ### Binary cache The swarm's attic server, in a `swarm-attic` container on the host that sets -`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has -the cache's store identity, as for the builder: the host that runs the store has -it. For an external cache, override `swarm.attic.{domain,cache}` and don't set +`deploy.attic.enable`. The builder pushes green builds to it, and the hives pull +closures from it at `swarm.attic.domain` (default `attic.`). For +an external cache, override `swarm.attic.{domain,cache}` and don't set `deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under -`allSwarmServices`. - -The container sits on the hive bridge in a network namespace of its own, and atticd -listens on that namespace's loopback only; the gateway reaches it over a unix socket. -The gateway serves it at `swarm.attic.domain` (default -`attic.`, on every host) over TLS, with no login in front, because attic checks -its own tokens. Storage is local disk in the container's state. - -It needs no manual secret. Two host units, under the cache's own `attic` leaf, -create its signing key, the cache and the push and pull tokens in the secret -store, each only while absent. On a host without the store, that leaf is the one -file to copy. The builder pushes green builds to it. How it works, and the -external alternative, are in [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache). - -The container forwards atticd's journal to the swarm collector like the other -service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host -units' lines reach the log store too: through the hive collector's journal -allow-list on a hive host, or through the swarm collector's whole-journal -receiver where the host runs that. A host with neither keeps them in its own -journal. +`allSwarmServices`. Its keys and tokens create themselves in the secret store. +How it works: +[Binary cache host setup](../../4-implementation/swarm/builder.md#binary-cache-host-setup) +and [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache). ### CI metrics -The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host -that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container. -`allSwarmServices` defaults it on wherever the host has the poller's store -identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to -opt out. - -Each pass pushes the run time and queue wait of every newly finished forge -Actions run to the metrics store, through its own route on the swarm collector; -the first pass pushes every run since the first. It reads the forge with the -read-only token of its own `ci-metrics` forge account, under its own -`ci-metrics` leaf. On a host without the store, that leaf is the one file to -copy. What it reads and pushes are in [`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md). +The CI run poller, a `swarm-ci-metrics` timer unit on the host that sets +`deploy.ci-metrics.enable`; set it to `false` to opt out. Every five minutes it +pushes the run time and queue wait of each newly finished forge Actions run to +the metrics store. What it reads and pushes: +[`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md). ### Review classifier The review classifier's try-out service, in a `swarm-review-classifier` -container on the host that sets `deploy.review-classifier.enable`. -`allSwarmServices` defaults it on wherever the host has the classifier's store -identity: the host that runs the store has it, and a services host with the -store elsewhere gets the classifier only once it holds a copy of the pair (see -below). Set `deploy.review-classifier.enable` directly to override either way. - -The operator sets the key of the runtime the swarm settings choose in the -swarm UI: the **classifier settings** card on `/jobs` takes a key for the -chosen runtime and shows whether one exists and when it changed, never the -key. "Set" means a key exists, not that it works: a run with a bad key -still fails. You can also write the same two keys in the secret store's UI under the -OIDC login, field `api_key`: - -- the API key runtime's Anthropic key at - `secret/swarm/services/review-classifier/anthropic-api-key` -- the ACP runtime's provider key at - `secret/swarm/services/review-classifier/acp-provider` - -Nothing writes them for you: until the chosen one exists the service runs and +container on the host that sets `deploy.review-classifier.enable`. You set the +key for the runtime the swarm settings choose in the swarm UI: the **classifier +settings** card on `/jobs` takes a key for that runtime and shows whether one +exists and when it changed, never the key. "Set" means a key exists, not that it +works, and every run spends money on that key; until the chosen one exists, every run fails with `no API key…` or `no ACP provider key…`. The service reads -the key on every run, so writing or removing it takes effect on the next run, -with no restart. Every run spends money on that key. - -It needs neither the forge nor a hive on its host. It reaches the queue and the -secret store by name, under its own `review-classifier` leaf. On a host -without the store, that leaf is the one file to copy, as for the CI runner. -The swarm UI's try-out drives it through swarm-controller; what a run is -and what it costs are in [`review-classifier.md`](../../4-implementation/swarm/review-classifier.md). +the key on every run, so writing or removing it takes effect on the next run. +The store paths for the two keys, and what a run is and costs: +[Host setup](../../4-implementation/swarm/review-classifier.md#host-setup) and +[`review-classifier.md`](../../4-implementation/swarm/review-classifier.md). ### Claude login minter The claude login minter, in a `swarm-claude-login` container on the host that -sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever -the host has the minter's store identity, on the review classifier's terms. -Run one: a login in progress lives in the service's memory between its start -and its code. - -An operator logs an agent in from the **log in** button on its card in the -swarm UI. swarm-controller relays the start and the pasted code over the queue, -the minter runs `claude auth login` and writes the credentials to the agent's -`swarm/agents//claude-credentials`, and the agent's harness installs a -version newer than the one it installed last. - -It needs neither the forge nor a hive on its host. It reaches the queue and the -secret store by name, under its own `claude-login` leaf, and reaches claude's -login endpoints over the network. On a host without the store, that leaf is -the one file to copy, as for the CI runner. The flow and the version rule are -in [`claude-login.md`](claude-login.md). +sets `deploy.claude-login.enable`. Run one: a login in progress lives in the +service's memory between its start and its code. An operator logs an agent in +from the **log in** button on its card in the swarm UI, and the agent's +harness installs the new login. The flow and the version rule: +[`claude-login.md`](claude-login.md). The service's host setup: +[Host setup](../../4-implementation/swarm/claude-login.md#host-setup). ### Notification router -The notification router, in a `swarm-notify-router` container on the host -that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on -wherever the host has the router's store identity, on the review classifier's -terms. Run one: it handles each issue's events in order. - -swarm-controller keeps every `issues` and `pull_request` delivery, and every -failed or cancelled CI run, from its instance-wide hooks in the `SWARM_EVENTS` -stream. The router reads that -stream and queues todos for the agents an event concerns: an agent assigned to -an issue or pull request gets a todo, one taken off it gets that todo rewritten -to say so, and closing it clears them. The author of a pull request gets a todo -when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes -it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos -from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's -back. - -It needs neither the forge nor a hive on its host, and holds no forge token. -It reaches the queue and the secret store by name, under its own -`notify-router` leaf. On a host without the store, that leaf is the one file to -copy, as for the CI runner. The flows and the delivery path are in -[`notify-router.md`](../../4-implementation/swarm/notify-router.md). +The notification router, in a `swarm-notify-router` container on the host that +sets `deploy.notify-router.enable`. Run one: it handles each issue's events in +order. It turns forge events (assigning an issue or pull request, closing it, a +failed CI run on a pull request) into todos for the agents they concern, and an +agent that's down gets its todos when it's back. The flows and the delivery +path: +[`notify-router.md`](../../4-implementation/swarm/notify-router.md), and +[Host setup](../../4-implementation/swarm/notify-router.md#host-setup). ### Speech to text The speech-to-text service, in a `swarm-stt` container on the host that sets -`deploy.stt.enable`. `allSwarmServices` defaults it on wherever the host has the -service's store identity, on the router's terms. Run it where the inference -hardware is: it transcribes with whisper.cpp. - -An agent that receives a voice memo sends the audio to the service over the -queue, on its own subject, and gets the text back. The service keeps each -transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs -one whisper run. `deploy.stt.settings.modelSize` picks the -model (`small` by default, or `base`), and `modelFile` takes any other ggml -model. The cache time, the limits on file size, audio length and run time, and -how many requests it holds at once are settings next to it. - -The audio comes from rooms with outside people, so the service runs as its own -unprivileged user under a tight systemd sandbox, with a memory ceiling and -a CPU quota. Its network isn't restricted: the decoder runs in the same -network namespace as the service and can read its credentials, so a decoder -exploit could read them and send them anywhere the container can reach. Closing -that's for the strict-egress work on service containers, or for running the -decode with no network and no credentials. The file's first bytes must look like an audio container -before anything decodes it. Its leaf and its queue client work as the router's -do. The design, the settings and the reasons are in -[`stt.md`](../../4-implementation/swarm/stt.md). +`deploy.stt.enable`. Run it where the inference hardware is: it transcribes +with whisper.cpp. An agent that receives a voice memo sends the audio to it +over the queue and gets the text back, and a memo sent to several agents costs +one run. `deploy.stt.settings.modelSize` picks the model (`small` by default, +or `base`). The audio comes from rooms with outside people, so the service runs +under a tight sandbox, but its network isn't restricted, which the design page +spells out. The settings, limits and reasons: +[`stt.md`](../../4-implementation/swarm/stt.md) and +[Host setup](../../4-implementation/swarm/stt.md#host-setup). diff --git a/docs/4-implementation/scheduler/ci.md b/docs/4-implementation/scheduler/ci.md index ae39445cc..4b0c40748 100644 --- a/docs/4-implementation/scheduler/ci.md +++ b/docs/4-implementation/scheduler/ci.md @@ -389,3 +389,34 @@ builder — disk-pressure collection at least runs while nothing is mid-copy. - `swarm-controller/src/forge/runner_token.rs`: publishes the registration token. - `.forgejo/workflows/ci.yml`: workflow definition. - `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers. + +## Host setup + +One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that +sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a +runner executes pull-request code, so running one is a decision of its own. + +It needs neither the forge nor a hive on its host. It reaches the forge at +`https://`, and registers with the token swarm-controller +publishes to the secret store, which it reads under its own `ci-runner` leaf. +On a host without the store, that leaf is the one file to copy. + +The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in +the container's own network namespace. The container's journal forwarder +scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and +pushes the series to the swarm collector under the job `gitea-runner`, so +nothing listens on the bridge. Series, all prefixed `gitea_runner_`: +`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`, +`info` (labels `version` and `name`), `job_total{status}`, +`job_duration_seconds`, `poll_fetch_total{result}`, +`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`, +`report_state_*` and `client_errors_total`. A pull-request job runs in the same +namespace and can read the endpoint, which carries counts, durations, the +runner's name and its version, and no secrets. + +The **ci runner** board in Grafana shows these series. Its **Runner** filter +(the series' `service.instance.id`, the scrape address) narrows every panel, +and uptime, job slots and slot use show one series per runner, so a runner +registered by hand appears beside the swarm's once its metrics reach the +store under its own instance id. The forge's **CI runs** tab is the other +view: run times and queue wait of every run, whichever runner ran it. diff --git a/docs/4-implementation/swarm/builder.md b/docs/4-implementation/swarm/builder.md index 4ad90a5cc..57596c812 100644 --- a/docs/4-implementation/swarm/builder.md +++ b/docs/4-implementation/swarm/builder.md @@ -681,3 +681,59 @@ is pure and runs in the container. Builds go to the host's nix daemon as an untrusted client: a config can set no daemon option, and its derivations build in the host's sandbox. A fixed-output derivation still has the network, as it does for any build on that host. + +## Host setup + +The swarm builder, in a `swarm-builder` container on the host that sets +`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host +has the builder's store identity, as for the other swarm services: the host +that runs the store has it. Set `deploy.builder.enable = false` to opt out. It +evaluates unreviewed config PRs and builds them through its host's nix daemon, +from a container with its own network namespace. + +When an `agent-configs/` PR opens, reopens or gets a new push, +swarm-controller queues a `build_agent` job. Jobs run one at a time, and a +newer push replaces its PR's job while that job is still waiting. The builder +builds that config at the PR's head on the agent base its own host runs, +through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder` +commit status on the head. The job shows on the swarm UI's Jobs page. A red +status blocks nothing: the merge still deploys. + +Separately, the builder builds each agent from `internal/meta` before the swarm +sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job; +[deploy gate](#deploy-gate)). Those jobs share the one-build slot with +the PR checks. + +It needs neither the forge nor a hive on its host. It reaches the queue, the +secret store and the forge by name, under its own `builder` leaf, and fetches +the private config repo with the read-only token of its own `builder` forge +account. On a host without the store, that leaf is the one file to copy, as +for the other swarm services. + +## Binary cache host setup + +The swarm's attic server, in a `swarm-attic` container on the host that sets +`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has +the cache's store identity, as for the other swarm services: the host that runs the store has +it. For an external cache, override `swarm.attic.{domain,cache}` and don't set +`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under +`allSwarmServices`. + +The container sits on the hive bridge in a network namespace of its own, and atticd +listens on that namespace's loopback only; the gateway reaches it over a unix socket. +The gateway serves it at `swarm.attic.domain` (default +`attic.`, on every host) over TLS, with no login in front, because attic checks +its own tokens. Storage is local disk in the container's state. + +It needs no manual secret. Two host units, under the cache's own `attic` leaf, +create its signing key, the cache and the push and pull tokens in the secret +store, each only while absent. On a host without the store, that leaf is the one +file to copy. The builder pushes green builds to it. How it works, and the +external alternative, are in [Binary cache](#binary-cache). + +The container forwards atticd's journal to the swarm collector like the other +service containers (see [Telemetry collector](telemetry.md#telemetry-collector-otel)). The two host +units' lines reach the log store too: through the hive collector's journal +allow-list on a hive host, or through the swarm collector's whole-journal +receiver where the host runs that. A host with neither keeps them in its own +journal. diff --git a/docs/4-implementation/swarm/ci-metrics.md b/docs/4-implementation/swarm/ci-metrics.md index 3a13abf50..6d2f41220 100644 --- a/docs/4-implementation/swarm/ci-metrics.md +++ b/docs/4-implementation/swarm/ci-metrics.md @@ -101,3 +101,18 @@ variable: - finished runs per day, by outcome; - a table per PR over the time range: mean and longest successful run time, runs, failed runs. + +## Host setup + +The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host +that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container. +`allSwarmServices` defaults it on wherever the host has the poller's store +identity, as for the other swarm services. Set `deploy.ci-metrics.enable = false` to +opt out. + +Each pass pushes the run time and queue wait of every newly finished forge +Actions run to the metrics store, through its own route on the swarm collector; +the first pass pushes every run since the first. It reads the forge with the +read-only token of its own `ci-metrics` forge account, under its own +`ci-metrics` leaf. On a host without the store, that leaf is the one file to +copy. diff --git a/docs/4-implementation/swarm/claude-login.md b/docs/4-implementation/swarm/claude-login.md index 841345ce4..0e4062619 100644 --- a/docs/4-implementation/swarm/claude-login.md +++ b/docs/4-implementation/swarm/claude-login.md @@ -118,3 +118,23 @@ gauges (`bao.client_cert.*`) come with them. `invalid` is a request that didn't decode or named no agent. A `start` spawns `claude auth login` and a `code` stores the credentials, so those two are the slow steps. `expired` counts logins stopped because their code never came. + +## Host setup + +The claude login minter, in a `swarm-claude-login` container on the host that +sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever +the host has the minter's store identity, like the other swarm services. +Run one: a login in progress lives in the service's memory between its start +and its code. + +An operator logs an agent in from the **log in** button on its card in the +swarm UI. swarm-controller relays the start and the pasted code over the queue, +the minter runs `claude auth login` and writes the credentials to the agent's +`swarm/agents//claude-credentials`, and the agent's harness installs a +version newer than the one it installed last. + +It needs neither the forge nor a hive on its host. It reaches the queue and the +secret store by name, under its own `claude-login` leaf, and reaches claude's +login endpoints over the network. On a host without the store, that leaf is +the one file to copy, as for the other swarm services. The flow and the version rule are +in [`claude-login.md`](../../2-handbook/swarm/claude-login.md). diff --git a/docs/4-implementation/swarm/notify-router.md b/docs/4-implementation/swarm/notify-router.md index 7e330f446..a2675abb1 100644 --- a/docs/4-implementation/swarm/notify-router.md +++ b/docs/4-implementation/swarm/notify-router.md @@ -433,3 +433,26 @@ no flow matched. `retried` is one the queue or the store failed on: it comes bac and counts again. A growing `consumer.pending` means the router is falling behind. The router reads the consumer counts at the start of each pull, at most 30 seconds apart. + +## Host setup + +The notification router, in a `swarm-notify-router` container on the host +that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on +wherever the host has the router's store identity, on the review classifier's +terms. Run one: it handles each issue's events in order. + +swarm-controller keeps every `issues` and `pull_request` delivery, and every +failed or cancelled CI run, from its instance-wide hooks in the `SWARM_EVENTS` +stream. The router reads that +stream and queues todos for the agents an event concerns: an agent assigned to +an issue or pull request gets a todo, one taken off it gets that todo rewritten +to say so, and closing it clears them. The author of a pull request gets a todo +when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes +it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos +from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's +back. + +It needs neither the forge nor a hive on its host, and holds no forge token. +It reaches the queue and the secret store by name, under its own +`notify-router` leaf. On a host without the store, that leaf is the one file to +copy, as for the other swarm services. diff --git a/docs/4-implementation/swarm/review-classifier.md b/docs/4-implementation/swarm/review-classifier.md index 95a526f45..974d39699 100644 --- a/docs/4-implementation/swarm/review-classifier.md +++ b/docs/4-implementation/swarm/review-classifier.md @@ -194,7 +194,7 @@ own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one there would offer the model tools again. The service is `swarm-review-classifier`, and its container is the one -[`services.md`](../../2-handbook/swarm/services.md#review-classifier) describes. It holds no forge +[Host setup](#host-setup) describes. It holds no forge credential: `swarm-controller` reads the PR and sends it along. ### Swarm-level settings @@ -315,7 +315,7 @@ deleted, still shows as set, because the controller can't read a value to check. The classifier reads the leaf on every run, so a new key applies from the next run. The operator can still write the leaves in the secret store's UI -([`services.md`](../../2-handbook/swarm/services.md#review-classifier)). +([Host setup](#host-setup)). The per-agent equivalent is [`inference.md`](inference.md), where the controller writes model and effort to a queue KV bucket @@ -765,3 +765,34 @@ These get settled, and filled in here, as the work reaches them: `secret/swarm/services/review-classifier/forge/token`. - **Reviewer handoff.** How a later reviewer, human or agent, learns the classifier passed a PR before it starts its own review. + +## Host setup + +The review classifier's try-out service, in a `swarm-review-classifier` +container on the host that sets `deploy.review-classifier.enable`. +`allSwarmServices` defaults it on wherever the host has the classifier's store +identity: the host that runs the store has it, and a services host with the +store elsewhere gets the classifier only once it holds a copy of the pair (see +below). Set `deploy.review-classifier.enable` directly to override either way. + +The operator sets the key of the runtime the swarm settings choose in the +swarm UI: the **classifier settings** card on `/jobs` takes a key for the +chosen runtime and shows whether one exists and when it changed, never the +key. "Set" means a key exists, not that it works: a run with a bad key +still fails. You can also write the same two keys in the secret store's UI under the +OIDC login, field `api_key`: + +- the API key runtime's Anthropic key at + `secret/swarm/services/review-classifier/anthropic-api-key` +- the ACP runtime's provider key at + `secret/swarm/services/review-classifier/acp-provider` + +Nothing writes them for you: until the chosen one exists the service runs and +every run fails with `no API key…` or `no ACP provider key…`. The service reads +the key on every run, so writing or removing it takes effect on the next run, +with no restart. Every run spends money on that key. + +It needs neither the forge nor a hive on its host. It reaches the queue and the +secret store by name, under its own `review-classifier` leaf. On a host +without the store, that leaf is the one file to copy, as for the other swarm services. +The swarm UI's try-out drives it through swarm-controller. diff --git a/docs/4-implementation/swarm/sso.md b/docs/4-implementation/swarm/sso.md index 0b98bc1a3..aeae0b9d2 100644 --- a/docs/4-implementation/swarm/sso.md +++ b/docs/4-implementation/swarm/sso.md @@ -184,3 +184,60 @@ upstream and compare three requests, not one: All three matter. A change that silently deleted the browser page would pass a deny-only check, and one that quietly stopped denying would pass a page-only check. The person who introduced the split verified it this way. + +## Host setup + +One authelia per swarm, in a `swarm-authelia` container, at +`auth.`. Operator and agents are both subjects of the same +provider, differentiated by roles and claims rather than by mechanism — +there is one IdP and one auth path. + +The container sits on the hive bridge in a network namespace of its own and +dials the store, the collector and its own public name through it. It binds +no TCP port: the login server, the metrics listener and the bridge each +listen on a unix socket in `/run/swarm-authelia`, a directory only the +gateway's nginx and the container's processes can open. + +- **`deploy.authelia`** — run the container here. +- **`swarm.authelia.url`** — where clients go to authenticate. Present + on **every** hive and the same value on all of them: + `https://`, whether or not this host runs the + container. Set it explicitly when joining a swarm whose IdP is under + another name. + +Agent subjects come from swarm-controller's agent-creation job, written +into the users database by `swarm-authelia-bridge`; human ones come from +`swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account). + +The bridge listens on a unix socket on the authelia host, and the gateway +publishes it as +`https:///bridge/`, which is where +`deploy.swarm-controller.authBridgeUrl` points by default on every host. +It answers only swarm-controller's OIDC client +(`swarm.controller.queueClientId`) and refuses every other token `403`. + +On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one +invalidates data already written (sessions, the encrypted store), so +that's an operator action. + +Storage is local sqlite and the notifier writes to a file. Both are +small-deployment choices, and the scope is the justification: redis +buys shared session state across replicas and there is one instance; +SMTP exists to mail humans, and provisioning here is programmatic. + +The swarm collector scrapes authelia's own metrics at +`https:///metrics`, whichever host runs the +container. The gateway admits only the collector's OAuth2 client there; +any other session gets `deny`. + +The container's `authelia-swarm-secrets` unit creates each client secret directly in +the swarm secret store, under the secret publisher's own store leaf handed in as +a system credential, and writes only the digest to disk; see +[`credentials.md`](../../2-handbook/swarm/credentials.md). Authelia starts from its digests, so the IdP +comes up while the store is down, and a unit that needs the store retries every +30 seconds. The container's journal forwarder reads its client secret back from +the store under the same leaf, and is off where the container has none. + +See [`sso.md`](../../2-handbook/swarm/sso.md) for bootstrapping the first user and the OIDC +relying-party flow, and [`credentials.md`](../../2-handbook/swarm/credentials.md) for where authelia +generates and reads each of its keys. diff --git a/docs/4-implementation/swarm/stt.md b/docs/4-implementation/swarm/stt.md index 05445e567..2998179e6 100644 --- a/docs/4-implementation/swarm/stt.md +++ b/docs/4-implementation/swarm/stt.md @@ -81,3 +81,28 @@ set them. Two things the daemon doesn't do yet: sweep scratch directories a crash left behind, and set whisper's thread count. The issue 6383 discussion has the choices behind the defaults. + +## Host setup + +The speech-to-text service, in a `swarm-stt` container on the host that sets +`deploy.stt.enable`. `allSwarmServices` defaults it on wherever the host has the +service's store identity, like the other swarm services. Run it where the inference +hardware is: it transcribes with whisper.cpp. + +An agent that receives a voice memo sends the audio to the service over the +queue, on its own subject, and gets the text back. The service keeps each +transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs +one whisper run. `deploy.stt.settings.modelSize` picks the +model (`small` by default, or `base`), and `modelFile` takes any other ggml +model. The cache time, the limits on file size, audio length and run time, and +how many requests it holds at once are settings next to it. + +The audio comes from rooms with outside people, so the service runs as its own +unprivileged user under a tight systemd sandbox, with a memory ceiling and +a CPU quota. Its network isn't restricted: the decoder runs in the same +network namespace as the service and can read its credentials, so a decoder +exploit could read them and send them anywhere the container can reach. Closing +that's for the strict-egress work on service containers, or for running the +decode with no network and no credentials. The file's first bytes must look like an audio container +before anything decodes it. Its leaf and its queue client work as the notification router's +do. diff --git a/docs/4-implementation/swarm/telemetry.md b/docs/4-implementation/swarm/telemetry.md index 2012f97fd..02ac26562 100644 --- a/docs/4-implementation/swarm/telemetry.md +++ b/docs/4-implementation/swarm/telemetry.md @@ -31,12 +31,16 @@ services.hyperhive.deploy = { ⚠️ **This starts a database that grows for as long as the swarm runs.** See `retentionPeriod` below before leaving it at its default. -| Option | When you'd touch it | -| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. | -| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. | -| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://`), so Grafana and the store can run on different hosts. | -| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://`), so Grafana and the store can run on different hosts. | +| Option | When you'd touch it | +| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. | +| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. | +| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://`), so Grafana and the store can run on different hosts. | +| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://`), so Grafana and the store can run on different hosts. | +| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. | +| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. | +| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. | +| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |