# Swarm-wide services Some things exist once per **swarm** rather than once per hive: the forge, the matrix homeserver, SSO, the secret store, the queue, and the metrics and log stack. This page says which host runs them and what a hive that runs none of them configures instead. The all-local quick start sets everything with one line → [README](../../README.md#quick-start-an-all-local-swarm). Two options say where they live, and everything else derives: ```nix services.hyperhive.deploy.singleHostSwarm = true; # everything on this box # or, for a dedicated services host with hives elsewhere: services.hyperhive.deploy.allSwarmServices = true; ``` **`deploy.allSwarmServices` is what "the swarm's shared services run here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service sections below don't repeat it, so a service that stops deriving is a visible difference rather than one more paragraph saying the same thing. `singleHostSwarm` is the all-on-one-box mode above it. It defaults `deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the host's `/etc/hosts` entries for the names it serves (`gateway.localHostsEntry`) and where the secret store's bootstrap token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived toggle on its own, and that setting wins. **Both default to off**, and that's deliberate: a host can't tell whether it's meant to be the swarm's service host, so this is an operator saying so rather than something inferred. With them off, a hive is a _client_ of those services — it configures how to reach them and runs none of them. That includes the forge (`deploy.forgejo.enable`). Every swarm needs one, being the canonical store for the meta flake and every agent's config repo, so **exactly one host must turn it on**: `singleHostSwarm`, `allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A host that enables its services one by one without any of those runs no forge. A hive that runs none of these reaches each one by name — the forge at `forge.`, for example. Only the host running a service answers its name from its own resolver, so on a swarm spread over more than one host, the operator's DNS has to resolve those names to that host.
Moving an existing hive's forge to the swarm's A hive that stops running the forge keeps the old container's state at `/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's forge: push anything worth keeping there by hand. Its `/var/lib/hyperhive/forge-core-token` came from that old forge and fails against the swarm's one.
## Deployment shapes Those two options are what makes the difference between deployments, so the shapes worth naming are the ones they produce: - **All-local.** Everything on one machine: `singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for a hive to run agents on. After the first switch, the steps in [`setup.md`](../getting-started/setup.md) remain. - **Services on the swarm controller host.** Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there, with hives elsewhere. The controller doesn't derive from `allSwarmServices`. - **Fully spread out.** One container / VM / machine per service, somewhere. **These are a set, not a ladder with a correct top, and hyperhive supports in-between shapes.** You can set each derived toggle on its own (see above), which is what makes "all local except X" a configuration rather than an unsupported edge case. Nothing in hyperhive prescribes a deployment model, so a doc that treats one shape as the real one and the others as compromises is wrong about the system rather than opinionated. The practical consequence is that **co-location is normal**: in the first two shapes services share a host by design. Where a specific service has something to say about sharing a host --- extra requirements, or a cost worth weighing --- that belongs in the doc for that service, not here. ## Services ### SSO (authelia) One authelia per swarm, in a `swarm-authelia` container, at `auth.`. Operator and agents are both subjects of the same provider, differentiated by roles and claims rather than by mechanism — there is one IdP and one auth path. - **`deploy.authelia`** — run the container here. - **`swarm.authelia.url`** — where clients go to authenticate. Present on **every** hive and the same value on all of them: `https://`, whether or not this host runs the container. Set it explicitly when joining a swarm whose IdP is under another name. Agent subjects come from swarm-controller's agent-creation job, written into the users database by `swarm-authelia-bridge`; human ones come from `swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account). The bridge listens on loopback on the authelia host, and the gateway publishes it as `https:///bridge/`, which is where `deploy.swarm-controller.authBridgeUrl` points by default on every host. It answers only swarm-controller's OIDC client (`swarm.controller.queueClientId`) and refuses every other token `403`. On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia generates session and storage keys in the container on first boot and never rotates them automatically; replacing one invalidates data already written (sessions, the encrypted store), so that's an operator action. Storage is local sqlite and the notifier writes to a file. Both are small-deployment choices, and the scope is the justification: redis buys shared session state across replicas and there is one instance; SMTP exists to mail humans, and provisioning here is programmatic. The swarm collector scrapes authelia's own metrics at `https:///metrics`, whichever host runs the container. The gateway admits only the collector's OAuth2 client there; any other session gets `deny`. See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC relying-party flow, and [`secrets.md`](secrets.md) for where authelia generates and reads each of its keys. ### Metrics (VictoriaMetrics + Grafana) The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it, in two containers at `metrics.` and `grafana.`. Two containers rather than one so you can restart or break Grafana without taking the time-series database with it. They derive together: a store with no UI is unreadable and a UI with no store is empty. To run one without the other, set it directly: ```nix services.hyperhive.deploy.victoriametrics.enable = true; services.hyperhive.deploy.grafana.enable = false; ``` ⚠️ **This starts a database that grows for as long as the swarm runs.** See `retentionPeriod` below before leaving it at its default. | Option | When you'd touch it | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. | | `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. | | `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://`), so Grafana and the store can run on different hosts. | | `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://`), so Grafana and the store can run on different hosts. | **Logging in.** Grafana is behind swarm SSO, so the accounts are the authelia ones, and the local login form is always off. Grafana's API still accepts HTTP basic auth for its built-in `admin` account, so that account's password is a random one minted into the swarm secret store (`swarm/services//admin/password`) and applied on every Grafana start — see [`secrets.md`](secrets.md). **Dashboards.** Grafana provisions these from `nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't kept. - `agents`: per-agent resource usage and Claude Code activity. - `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue. - `claude-usage`: Claude use by model, effort, token type and source. - `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge. - `forge`: what the forge holds, and the forge process's CPU, memory and descriptors. - `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections. - `collector`: what the OTEL collectors accept, refuse, export and queue, with queue depth against capacity. - `logstore`, `metricstore`: health of the log store and the metrics store. A new panel can read "No data" until its source exists: the push and commit counters on `controller` appear after the first push webhook, and the refused and failure series on `collector` appear once something fails. **Where the data comes from.** The swarm's OTEL collector, below. Neither container is reachable except through the gateway: each listens on a unix socket only the gateway's nginx can open, and VictoriaMetrics runs in a network namespace of its own with no interface but loopback. VictoriaMetrics authenticates nothing itself; the `metrics.` vhost does it: - **The read API** (the Prometheus query, series, label and metadata routes): Grafana's datasource or an operator. Grafana sends a bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret store, and `swarm-metrics-auth` on the store's host checks it against the same store path. A request without a bearer needs an operator's authelia session (group `admins`); without one it gets a 401, not a login redirect. - **vmui**: an operator's authelia session, with a redirect to the login. The bare domain redirects to vmui. - **The collector's OTLP push, and its scrape of the store's own `/metrics`**: only the swarm collector's authelia client. Authelia refuses any other session or client there, operators included. - **Every other path**, the store's write and admin API included: 403. `swarm-metrics-auth` reads the store with a leaf of its own, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`. Where the secret store runs on the same host, these default to the leaf it mints. On any other host you place a leaf whose CN is `deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until you set both. It reads the stored bearer at most once a minute, so a replaced bearer starts working up to a minute later. The collector scrapes Grafana's own metrics at `https:///metrics`, so a collector on a host without Grafana still gets them. The gateway admits only the collector's client to that path; any other session gets `deny`. The collector scrapes the store's own metrics the same way, at `https:///metrics`, so a collector on a host that doesn't run the store still gets them. On this name the gateway admits only the collector's client to `/metrics` and the push route; any other session gets `deny`. ### Logs (VictoriaLogs) Each hive ships its journals to one VictoriaLogs at `logs.`, behind the same SSO as everything else: the swarm's own service containers, the hive's daemons and infra containers, and the harness units inside every agent container. The collector below is what writes to it. **Reading them.** Open Grafana, pick **Explore**, and choose the `VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs Drilldown_ app is deliberately not installed: it only supports Loki, and no setting here changes that, so Explore is the log browser for this swarm. | Option | When you'd touch it | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. | | `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. | | `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. | | `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. | | `swarm.victorialogs.domain` | Only to rename it. | | `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. | It binds loopback and takes no credential of its own: the gateway vhost is the only way in, and the collector is the only intended writer. **Who may query.** `/select/logsql/` takes one of three credentials. An agent presents its `swarm-agent..` bearer, checked by `swarm-logs-auth`. Grafana's datasource presents the same bearer as its metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log store's host checks it against the same store path; the gateway tells it from the others by its shape, 64 hex digits. Anything else needs an operator's authelia session. Because one bearer opens both read APIs, a rotation (see [`secrets.md`](secrets.md)) restarts Grafana once, and the log store accepts the new value within a minute. The collector also scrapes this store's own metrics, at `https:///metrics`, so a collector on a host without the store still gets them. The gateway admits only the collector's client to that path; any other session gets `deny`. ### Telemetry collector (OTEL) The swarm's collector receives from every hive's own collector and is the only process that decides where telemetry goes: it always writes the stores above, and exports to `otel.endpoint` as well when that's set. It holds the upstream credential, which is why no hive and no agent needs one. It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to `4319` rather than OTLP's usual `4318`, which the hive tier uses — swarm containers share the host's network namespace, so two collectors on one port is a coin toss at runtime rather than an error at build time. Every hive's own collector reaches this one by its gateway name, `swarm.otel.domain` (default `otel.`) — the same by-domain-through-the-gateway shape every other swarm service uses, not a loopback URL an operator has to redirect. Nothing needs setting on a hive that doesn't run the swarm's services; the name resolves through the gateway either way. | Option | When you'd touch it | | ------------------- | --------------------------------------------------------------------------------------- | | `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. | | `swarm.otel.port` | Only if something else on the services host already claims `4319`. | The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login` and `swarm-review-classifier` containers each run a collector of their own that forwards the container's journal here, on a route and receiver per service (`https:////`), authenticated as that service's own OIDC client. Their records carry `service.name` (`bao`, `nats`, `builder`, `claude-login`, `review-classifier`) and no `hive` label. The forwarder of all but the store's is `nix/container-modules/swarm-journal-forwarder.nix`. **Both store exporters are unconditional**, and `deploy.victoriametrics.enable` doesn't gate them: that option says this host _runs_ the store, while the swarm has one either way, reached by its swarm name through the gateway. Agent-side configuration, and what a hive's own collector does, are in [`../scheduler/observability.md`](../scheduler/observability.md). ### CI runner One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a runner executes pull-request code, so running one is a decision of its own. It needs neither the forge nor a hive on its host. It reaches the forge at `https://`, and registers with the token swarm-controller publishes to the secret store, which it reads under its own `ci-runner` leaf. On a host without the store, that leaf is the one file to copy. Registration, the container and the threat model are in [`../scheduler/ci.md`](../scheduler/ci.md). ### Builder The swarm builder, in a `swarm-builder` container on the host that sets `deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host has the builder's store identity, as for the review classifier below: the host that runs the store has it. Set `deploy.builder.enable = false` to opt out. It evaluates unreviewed config PRs and builds them through its host's nix daemon, from a container with its own network namespace. When an `agent-configs/` PR opens, reopens or gets a new push, swarm-controller queues a `build_agent` job. Jobs run one at a time, and a newer push replaces its PR's job while that job is still waiting. The builder builds that config at the PR's head on the agent base its own host runs, through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder` commit status on the head. The job shows on the swarm UI's Jobs page. A red status blocks nothing: the merge still deploys. It needs neither the forge nor a hive on its host. It reaches the queue, the secret store and the forge by name, under its own `builder` leaf, and fetches the private config repo with the read-only token of its own `builder` forge account. On a host without the store, that leaf is the one file to copy, as for the CI runner. What it builds and why are in [`builder.md`](builder.md). ### Review classifier The review classifier's try-out service, in a `swarm-review-classifier` container on the host that sets `deploy.review-classifier.enable`. `allSwarmServices` defaults it on wherever the host has the classifier's store identity: the host that runs the store has it, and a services host with the store elsewhere gets the classifier only once it holds a copy of the pair (see below). Set `deploy.review-classifier.enable` directly to override either way. The operator writes the key of the runtime the swarm settings choose by hand, in the secret store's UI under the OIDC login, field `api_key`: - the API key runtime's Anthropic key at `secret/swarm/services/review-classifier/anthropic-api-key` - the ACP runtime's provider key at `secret/swarm/services/review-classifier/acp-provider` Nothing writes them for you: until the chosen one exists the service runs and every run fails with `no API key…` or `no ACP provider key…`. The service reads the key on every run, so writing or removing it takes effect on the next run, with no restart. Every run spends money on that key. It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own `review-classifier` leaf. On a host without the store, that leaf is the one file to copy, as for the CI runner. The swarm UI's try-out drives it through swarm-controller; what a run is and what it costs are in [`review-classifier.md`](review-classifier.md). ### Claude login minter The claude login minter, in a `swarm-claude-login` container on the host that sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever the host has the minter's store identity, on the review classifier's terms. Run one: a login in progress lives in the service's memory between its start and its code. An operator logs an agent in from the **log in** button on its card in the swarm UI. swarm-controller relays the start and the pasted code over the queue, the minter runs `claude auth login` and writes the credentials to the agent's `swarm/agents//claude-credentials`, and the agent's harness installs a version newer than the one it installed last. It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own `claude-login` leaf, and reaches claude's login endpoints over the network. On a host without the store, that leaf is the one file to copy, as for the CI runner. The flow and the version rule are in [`claude-login.md`](claude-login.md).