| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. | | `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. | | `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. | | `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. | # Swarm-wide services Some things exist once per **swarm** rather than once per hive: the forge, the matrix homeserver, SSO, the secret store, the queue, and the metrics and log stack. This page says which host runs them and what a hive that runs none of them configures instead. The all-local quick start sets everything with one line → [README](../../../README.md#quick-start-an-all-local-swarm). Two options say where they live, and everything else derives: ```nix services.hyperhive.deploy.singleHostSwarm = true; # everything on this box # or, for a dedicated services host with hives elsewhere: services.hyperhive.deploy.allSwarmServices = true; ``` **`deploy.allSwarmServices` is what "the swarm's shared services run here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service sections below don't repeat it, so a service that stops deriving is a visible difference rather than one more paragraph saying the same thing. `singleHostSwarm` is the all-on-one-box mode above it. It defaults `deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the host's `/etc/hosts` entries for the names it serves (`gateway.localHostsEntry`) and where the secret store's bootstrap token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived toggle on its own, and that setting wins. **Both default to off**, and that's deliberate: a host can't tell whether it's meant to be the swarm's service host, so this is an operator saying so rather than something inferred. With them off, a hive is a _client_ of those services — it configures how to reach them and runs none of them. That includes the forge (`deploy.forgejo.enable`). Every swarm needs one, being the canonical store for the meta flake and every agent's config repo, so **exactly one host must turn it on**: `singleHostSwarm`, `allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A host that enables its services one by one without any of those runs no forge. A hive can't start or stop any of these. `hivectl stop`, `start` and `restart` act on the hive's agents and its gateway only, even on the host that runs the swarm's services. The forge and matrix containers are systemd units on that host (`container@hive-forge.service`, `container@hive-matrix.service`), and the operator manages them there with `systemctl`. A hive that runs none of these reaches each one by name — the forge at `forge.`, for example. Only the host running a service answers its name from its own resolver, so on a swarm spread over more than one host, the operator's DNS has to resolve those names to that host.
Moving an existing hive's forge to the swarm's A hive that stops running the forge keeps the old container's state at `/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's forge: push anything worth keeping there by hand.
## Deployment shapes Those two options are what makes the difference between deployments, so the shapes worth naming are the ones they produce: - **All-local.** Everything on one machine: `singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for a hive to run agents on. After the first switch, the steps in [`setup.md`](../../1-getting-started/setup.md) remain. - **Services on the swarm controller host.** Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there, with hives elsewhere. The controller doesn't derive from `allSwarmServices`. - **Fully spread out.** One container / VM / machine per service, somewhere. - **Semi-spread.** The services split over a few machines by weight, hives on their own, a VPN between them, and most vhosts answering on the VPN only. A worked example with its NixOS files: [`semi-spread-deploy.md`](semi-spread-deploy.md). **These are a set, not a ladder with a correct top, and hyperhive supports in-between shapes.** You can set each derived toggle on its own (see above), which is what makes "all local except X" a configuration rather than an unsupported edge case. Nothing in hyperhive prescribes a deployment model, so a doc that treats one shape as the real one and the others as compromises is wrong about the system rather than opinionated. The practical consequence is that **co-location is normal**: in the first two shapes services share a host by design. Where a specific service has something to say about sharing a host --- extra requirements, or a cost worth weighing --- that belongs in the doc for that service, not here. ## Services An operator who isn't spreading the swarm over more than one host can skip most of this page: `singleHostSwarm` turns on the shared services below, with two exceptions. The **CI runner** stays off until you set `deploy.ci-runner.enable`, and the **builder** and **binary cache** come on only where the host has their store identities. The table says what each one does and where its option and its section are. The builder and the CI runner are the two that cost real resources (they build and run pull-request code), so they're the two worth a look before you switch them on. | Service | What it does | Option | Details | | ------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------ | | Forge | Holds the meta flake and every agent's config repo. | `deploy.forgejo.enable` | [forge.md](forge.md) | | Matrix | The chat homeserver agents and the operator talk over. | `deploy.matrix.enable` | [matrix.md](matrix.md) | | Secret store | Keeps the swarm's credentials. | `deploy.bao.enable` | [credentials.md](credentials.md) | | Queue | Carries hive status and swarm events. | `deploy.nats.enable` | [Deployment shapes](#deployment-shapes) | | SSO | One login for the operator and the agents. | `deploy.authelia.enable` | [SSO](#sso-authelia) | | Metrics | Stores telemetry and shows it in Grafana. | `deploy.victoriametrics.enable`, `deploy.grafana.enable` | [Metrics](#metrics-victoriametrics--grafana) | | Logs | Stores every journal for search in Grafana. | `deploy.victorialogs.enable` | [Logs](#logs-victorialogs) | | Telemetry collector | Routes metrics and logs to the stores and an optional upstream. | `deploy.swarm-otel.enable` | [Telemetry collector](#telemetry-collector-otel) | | CI runner | Runs Forgejo Actions for pull requests. Heavy. | `deploy.ci-runner.enable` | [CI runner](#ci-runner) | | Builder | Evaluates and builds agent config PRs. Heavy. | `deploy.builder.enable` | [Builder](#builder) | | Binary cache | Serves built closures to the hives. | `deploy.attic.enable` | [Binary cache](#binary-cache) | | Backup archive | Keeps every agent's state backups, one repository each. | `deploy.archive.enable` | [Backup archive](archive.md) | | CI metrics | Records CI run times in the metrics store. | `deploy.ci-metrics.enable` | [CI metrics](#ci-metrics) | | Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) | | Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) | | Notification router | Turns forge events into notifications for agents. | `deploy.notify-router.enable` | [Notification router](#notification-router) | ### SSO (authelia) One authelia per swarm, in a `swarm-authelia` container, at `auth.`. Operator and agents are both subjects of the same provider, differentiated by roles and claims rather than by mechanism — there is one IdP and one auth path. The container sits on the hive bridge in a network namespace of its own and dials the store, the collector and its own public name through it. It binds no TCP port: the login server, the metrics listener and the bridge each listen on a unix socket in `/run/swarm-authelia`, a directory only the gateway's nginx and the container's processes can open. - **`deploy.authelia`** — run the container here. - **`swarm.authelia.url`** — where clients go to authenticate. Present on **every** hive and the same value on all of them: `https://`, whether or not this host runs the container. Set it explicitly when joining a swarm whose IdP is under another name. Agent subjects come from swarm-controller's agent-creation job, written into the users database by `swarm-authelia-bridge`; human ones come from `swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account). The bridge listens on a unix socket on the authelia host, and the gateway publishes it as `https:///bridge/`, which is where `deploy.swarm-controller.authBridgeUrl` points by default on every host. It answers only swarm-controller's OIDC client (`swarm.controller.queueClientId`) and refuses every other token `403`. On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one invalidates data already written (sessions, the encrypted store), so that's an operator action. Storage is local sqlite and the notifier writes to a file. Both are small-deployment choices, and the scope is the justification: redis buys shared session state across replicas and there is one instance; SMTP exists to mail humans, and provisioning here is programmatic. The swarm collector scrapes authelia's own metrics at `https:///metrics`, whichever host runs the container. The gateway admits only the collector's OAuth2 client there; any other session gets `deny`. The container's `authelia-swarm-secrets` unit creates each client secret directly in the swarm secret store, under the secret publisher's own store leaf handed in as a system credential, and writes only the digest to disk; see [`credentials.md`](credentials.md). Authelia starts from its digests, so the IdP comes up while the store is down, and a unit that needs the store retries every 30 seconds. The container's journal forwarder reads its client secret back from the store under the same leaf, and is off where the container has none. See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC relying-party flow, and [`credentials.md`](credentials.md) for where authelia generates and reads each of its keys. ### Metrics (VictoriaMetrics + Grafana) The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it, at `metrics.` and `grafana.`. They sit in two containers so you can restart or break Grafana without losing the time-series database. Enabling either enables both; to run one alone, set it directly with `deploy.victoriametrics.enable` or `deploy.grafana.enable`. Log in to Grafana with your swarm SSO account. Grafana provisions its dashboards from the repo, so edits made in the UI aren't kept. Each service has its own board; open the folder to see them. ⚠️ **The metrics store grows for as long as the swarm runs.** Options to touch: | Option | When you'd touch it | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you know how fast the swarm fills a disk. A too-short value discards history you can't get back. | | `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower it to `Viewer` or `Editor` if some operators shouldn't reconfigure Grafana. | The dashboard list, the "No data" cases, and how the gateway guards the store: [`telemetry.md`](../../4-implementation/swarm/telemetry.md#metrics-victoriametrics--grafana). ### Logs (VictoriaLogs) Each hive ships its journals to one VictoriaLogs at `logs.`, behind the same SSO as everything else: the swarm's service containers, the hive's daemons and the harness units in every agent container. To read them, open Grafana, pick **Explore** and choose the `VictoriaLogs` datasource. Agents query it with `swarm-logs`, and need the `logs` permission. | Option | When you'd touch it | | ------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's. Raise it to answer questions about last quarter. | The other options, who may query, and the gateway rules: [`telemetry.md`](../../4-implementation/swarm/telemetry.md#logs-victorialogs). ### Telemetry collector (OTEL) The swarm's collector receives from every hive's own collector and decides where telemetry goes. It always writes the metrics and log stores, and exports to `deploy.swarm-otel.upstream.endpoint` as well when you set one. It holds the upstream credential, so no hive and no agent needs one. Every hive reaches it by its gateway name, `otel.`, so a hive that doesn't run the swarm's services needs nothing set. How the collector connects, its per-service routes and the homeserver's stand-in metrics: [`telemetry.md`](../../4-implementation/swarm/telemetry.md#telemetry-collector-otel). ### CI runner One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a runner executes pull-request code, so running one is a decision of its own. It needs neither the forge nor a hive on its host. It reaches the forge at `https://`, and registers with the token swarm-controller publishes to the secret store, which it reads under its own `ci-runner` leaf. On a host without the store, that leaf is the one file to copy. Registration, the container and the threat model are in [`4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md). The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in the container's own network namespace. The container's journal forwarder scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and pushes the series to the swarm collector under the job `gitea-runner`, so nothing listens on the bridge. Series, all prefixed `gitea_runner_`: `job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`, `info` (labels `version` and `name`), `job_total{status}`, `job_duration_seconds`, `poll_fetch_total{result}`, `poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`, `report_state_*` and `client_errors_total`. A pull-request job runs in the same namespace and can read the endpoint, which carries counts, durations, the runner's name and its version, and no secrets. The **ci runner** board in Grafana shows these series. Its **Runner** filter (the series' `service.instance.id`, the scrape address) narrows every panel, and uptime, job slots and slot use show one series per runner, so a runner registered by hand appears beside the swarm's once its metrics reach the store under its own instance id. The forge's **CI runs** tab is the other view: run times and queue wait of every run, whichever runner ran it. ### Builder The swarm builder, in a `swarm-builder` container on the host that sets `deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host has the builder's store identity, as for the review classifier below: the host that runs the store has it. Set `deploy.builder.enable = false` to opt out. It evaluates unreviewed config PRs and builds them through its host's nix daemon, from a container with its own network namespace. When an `agent-configs/` PR opens, reopens or gets a new push, swarm-controller queues a `build_agent` job. Jobs run one at a time, and a newer push replaces its PR's job while that job is still waiting. The builder builds that config at the PR's head on the agent base its own host runs, through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder` commit status on the head. The job shows on the swarm UI's Jobs page. A red status blocks nothing: the merge still deploys. Separately, the builder builds each agent from `internal/meta` before the swarm sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job; [deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). Those jobs share the one-build slot with the PR checks. It needs neither the forge nor a hive on its host. It reaches the queue, the secret store and the forge by name, under its own `builder` leaf, and fetches the private config repo with the read-only token of its own `builder` forge account. On a host without the store, that leaf is the one file to copy, as for the CI runner. What it builds and why are in [`builder.md`](../../4-implementation/swarm/builder.md). ### Binary cache The swarm's attic server, in a `swarm-attic` container on the host that sets `deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has the cache's store identity, as for the builder: the host that runs the store has it. For an external cache, override `swarm.attic.{domain,cache}` and don't set `deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under `allSwarmServices`. The container sits on the hive bridge in a network namespace of its own, and atticd listens on that namespace's loopback only; the gateway reaches it over a unix socket. The gateway serves it at `swarm.attic.domain` (default `attic.`, on every host) over TLS, with no login in front, because attic checks its own tokens. Storage is local disk in the container's state. It needs no manual secret. Two host units, under the cache's own `attic` leaf, create its signing key, the cache and the push and pull tokens in the secret store, each only while absent. On a host without the store, that leaf is the one file to copy. The builder pushes green builds to it. How it works, and the external alternative, are in [`builder.md`](../../4-implementation/swarm/builder.md#binary-cache). The container forwards atticd's journal to the swarm collector like the other service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host units' lines reach the log store too: through the hive collector's journal allow-list on a hive host, or through the swarm collector's whole-journal receiver where the host runs that. A host with neither keeps them in its own journal. ### CI metrics The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container. `allSwarmServices` defaults it on wherever the host has the poller's store identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to opt out. Each pass pushes the run time and queue wait of every newly finished forge Actions run to the metrics store, through its own route on the swarm collector; the first pass pushes every run since the first. It reads the forge with the read-only token of its own `ci-metrics` forge account, under its own `ci-metrics` leaf. On a host without the store, that leaf is the one file to copy. What it reads and pushes are in [`ci-metrics.md`](../../4-implementation/swarm/ci-metrics.md). ### Review classifier The review classifier's try-out service, in a `swarm-review-classifier` container on the host that sets `deploy.review-classifier.enable`. `allSwarmServices` defaults it on wherever the host has the classifier's store identity: the host that runs the store has it, and a services host with the store elsewhere gets the classifier only once it holds a copy of the pair (see below). Set `deploy.review-classifier.enable` directly to override either way. The operator writes the key of the runtime the swarm settings choose by hand, in the secret store's UI under the OIDC login, field `api_key`: - the API key runtime's Anthropic key at `secret/swarm/services/review-classifier/anthropic-api-key` - the ACP runtime's provider key at `secret/swarm/services/review-classifier/acp-provider` Nothing writes them for you: until the chosen one exists the service runs and every run fails with `no API key…` or `no ACP provider key…`. The service reads the key on every run, so writing or removing it takes effect on the next run, with no restart. Every run spends money on that key. It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own `review-classifier` leaf. On a host without the store, that leaf is the one file to copy, as for the CI runner. The swarm UI's try-out drives it through swarm-controller; what a run is and what it costs are in [`review-classifier.md`](../../4-implementation/swarm/review-classifier.md). ### Claude login minter The claude login minter, in a `swarm-claude-login` container on the host that sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever the host has the minter's store identity, on the review classifier's terms. Run one: a login in progress lives in the service's memory between its start and its code. An operator logs an agent in from the **log in** button on its card in the swarm UI. swarm-controller relays the start and the pasted code over the queue, the minter runs `claude auth login` and writes the credentials to the agent's `swarm/agents//claude-credentials`, and the agent's harness installs a version newer than the one it installed last. It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own `claude-login` leaf, and reaches claude's login endpoints over the network. On a host without the store, that leaf is the one file to copy, as for the CI runner. The flow and the version rule are in [`claude-login.md`](claude-login.md). ### Notification router The notification router, in a `swarm-notify-router` container on the host that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on wherever the host has the router's store identity, on the review classifier's terms. Run one: it handles each issue's events in order. swarm-controller keeps every `issues` and `pull_request` delivery, and every failed or cancelled CI run, from its instance-wide hooks in the `SWARM_EVENTS` stream. The router reads that stream and queues todos for the agents an event concerns: an agent assigned to an issue or pull request gets a todo, one taken off it gets that todo rewritten to say so, and closing it clears them. The author of a pull request gets a todo when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's back. It needs neither the forge nor a hive on its host, and holds no forge token. It reaches the queue and the secret store by name, under its own `notify-router` leaf. On a host without the store, that leaf is the one file to copy, as for the CI runner. The flows and the delivery path are in [`notify-router.md`](../../4-implementation/swarm/notify-router.md).