hyperhive/docs/2-handbook/swarm/services.md

| deploy.victoriametrics.agentAuthRate | Default 50r/s per client address, for agent reads checked by swarm-metrics-agent-auth; over it and agentAuthBurst, the gateway answers 403. | | deploy.victoriametrics.agentAuthBurst | Default 500 requests above agentAuthRate before the 403s start. | | deploy.victoriametrics.agentAuthVerdictTtl | Default 60 seconds that swarm-metrics-agent-auth keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked metrics permission does too. | | deploy.victoriametrics.agentAuthPackage | The swarm-logs-auth build that runs as swarm-metrics-agent-auth. Set on its own: deploy.victorialogs.authPackage doesn't reach it. |

Swarm-wide services

Some things exist once per swarm rather than once per hive: the forge, the matrix homeserver, SSO, the secret store, the queue, and the metrics and log stack. This page says which host runs them and what a hive that runs none of them configures instead. The all-local quick start sets everything with one line → README.

Two options say where they live, and everything else derives:

services.hyperhive.deploy.singleHostSwarm = true;   # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.deploy.allSwarmServices = true;

deploy.allSwarmServices is what "the swarm's shared services run here" means: every once-per-swarm service takes its enable from it. That's the whole rule, stated once — the per-service sections below don't repeat it, so a service that stops deriving is a visible difference rather than one more paragraph saying the same thing.

singleHostSwarm is the all-on-one-box mode above it. It defaults deploy.allSwarmServices, the swarm controller (deploy.swarm-controller.enable), the host's /etc/hosts entries for the names it serves (gateway.localHostsEntry) and where the secret store's bootstrap token goes (deploy.bao.bootstrapTokenFile). You can still set each derived toggle on its own, and that setting wins.

Both default to off, and that's deliberate: a host can't tell whether it's meant to be the swarm's service host, so this is an operator saying so rather than something inferred. With them off, a hive is a client of those services — it configures how to reach them and runs none of them.

That includes the forge (deploy.forgejo.enable). Every swarm needs one, being the canonical store for the meta flake and every agent's config repo, so exactly one host must turn it on: singleHostSwarm, allSwarmServices, or deploy.forgejo.enable = true set by hand. A host that enables its services one by one without any of those runs no forge.

A hive can't start or stop any of these. hivectl stop, start and restart act on the hive's agents and its gateway only, even on the host that runs the swarm's services. The forge and matrix containers are systemd units on that host (container@hive-forge.service, container@hive-matrix.service), and the operator manages them there with systemctl.

A hive that runs none of these reaches each one by name — the forge at forge.<swarm-domain>, for example. Only the host running a service answers its name from its own resolver, so on a swarm spread over more than one host, the operator's DNS has to resolve those names to that host.

Moving an existing hive's forge to the swarm's

A hive that stops running the forge keeps the old container's state at /var/lib/nixos-containers/hive-forge/. Nothing moves it to the swarm's forge: push anything worth keeping there by hand.

Deployment shapes

Those two options are what makes the difference between deployments, so the shapes worth naming are the ones they produce:

• All-local. Everything on one machine: singleHostSwarm = true, plus deploy.hive-controller.enable = true for a hive to run agents on. After the first switch, the steps in setup.md remain.
• Services on the swarm controller host. Set deploy.allSwarmServices and deploy.swarm-controller.enable there, with hives elsewhere. The controller doesn't derive from allSwarmServices.
• Fully spread out. One container / VM / machine per service, somewhere.
• Semi-spread. The services split over a few machines by weight, hives on their own, a VPN between them, and most vhosts answering on the VPN only. A worked example with its NixOS files: semi-spread-deploy.md.

These are a set, not a ladder with a correct top, and hyperhive supports in-between shapes. You can set each derived toggle on its own (see above), which is what makes "all local except X" a configuration rather than an unsupported edge case. Nothing in hyperhive prescribes a deployment model, so a doc that treats one shape as the real one and the others as compromises is wrong about the system rather than opinionated.

The practical consequence is that co-location is normal: in the first two shapes services share a host by design. Where a specific service has something to say about sharing a host --- extra requirements, or a cost worth weighing --- that belongs in the doc for that service, not here.

Services

An operator who isn't spreading the swarm over more than one host can skip most of this page: singleHostSwarm turns on the shared services below, with two exceptions. The CI runner stays off until you set deploy.ci-runner.enable, and the builder and binary cache come on only where the host has their store identities. The table says what each one does and where its option and its section are. The builder and the CI runner are the two that cost real resources (they build and run pull-request code), so they're the two worth a look before you switch them on.

Service	What it does	Option	Details
Forge	Holds the meta flake and every agent's config repo.	deploy.forgejo.enable	forge.md
Matrix	The chat homeserver agents and the operator talk over.	deploy.matrix.enable	matrix.md
Secret store	Keeps the swarm's credentials.	deploy.bao.enable	credentials.md
Queue	Carries hive status and swarm events.	deploy.nats.enable	Deployment shapes
SSO	One login for the operator and the agents.	deploy.authelia.enable	SSO
Metrics	Stores telemetry and shows it in Grafana.	deploy.victoriametrics.enable, deploy.grafana.enable	Metrics
Logs	Stores every journal for search in Grafana.	deploy.victorialogs.enable	Logs
Telemetry collector	Routes metrics and logs to the stores and an optional upstream.	deploy.swarm-otel.enable	Telemetry collector
CI runner	Runs Forgejo Actions for pull requests. Heavy.	deploy.ci-runner.enable	CI runner
Builder	Evaluates and builds agent config PRs. Heavy.	deploy.builder.enable	Builder
Binary cache	Serves built closures to the hives.	deploy.attic.enable	Binary cache
Backup archive	Keeps every agent's state backups, one repository each.	deploy.archive.enable	Backup archive
CI metrics	Records CI run times in the metrics store.	deploy.ci-metrics.enable	CI metrics
Review classifier	Checks PRs against a repo's review rules.	deploy.review-classifier.enable	Review classifier
Claude login minter	Runs the log in button on an agent's card.	deploy.claude-login.enable	Claude login minter
Notification router	Turns forge events into notifications for agents.	deploy.notify-router.enable	Notification router

SSO (authelia)

One authelia per swarm, in a swarm-authelia container, at auth.<swarm-domain>. Operator and agents are both subjects of the same provider, differentiated by roles and claims rather than by mechanism — there is one IdP and one auth path.

The container sits on the hive bridge in a network namespace of its own and dials the store, the collector and its own public name through it. It binds no TCP port: the login server, the metrics listener and the bridge each listen on a unix socket in /run/swarm-authelia, a directory only the gateway's nginx and the container's processes can open.

• deploy.authelia — run the container here.
• swarm.authelia.url — where clients go to authenticate. Present on every hive and the same value on all of them: https://<swarm.authelia.domain>, whether or not this host runs the container. Set it explicitly when joining a swarm whose IdP is under another name.

Agent subjects come from swarm-controller's agent-creation job, written into the users database by swarm-authelia-bridge; human ones come from swarmctl user add → setup.md § 2.

The bridge listens on a unix socket on the authelia host, and the gateway publishes it as https://<swarm.authelia.domain>/bridge/, which is where deploy.swarm-controller.authBridgeUrl points by default on every host. It answers only swarm-controller's OIDC client (swarm.controller.queueClientId) and refuses every other token 403.

On first boot this module seeds the users database with a placeholder subject marked disabled, so authelia has a non-empty store to start against before any real account exists (swarm-authelia.nix:873). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one invalidates data already written (sessions, the encrypted store), so that's an operator action.

Storage is local sqlite and the notifier writes to a file. Both are small-deployment choices, and the scope is the justification: redis buys shared session state across replicas and there is one instance; SMTP exists to mail humans, and provisioning here is programmatic.

The swarm collector scrapes authelia's own metrics at https://<swarm.authelia.domain>/metrics, whichever host runs the container. The gateway admits only the collector's OAuth2 client there; any other session gets deny.

The container's authelia-swarm-secrets unit creates each client secret directly in the swarm secret store, under the secret publisher's own store leaf handed in as a system credential, and writes only the digest to disk; see credentials.md. Authelia starts from its digests, so the IdP comes up while the store is down, and a unit that needs the store retries every 30 seconds. The container's journal forwarder reads its client secret back from the store under the same leaf, and is off where the container has none.

See sso.md for bootstrapping the first user and the OIDC relying-party flow, and credentials.md for where authelia generates and reads each of its keys.

Metrics (VictoriaMetrics + Grafana)

The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it, at metrics.<swarm-domain> and grafana.<swarm-domain>. They sit in two containers so you can restart or break Grafana without losing the time-series database. Enabling either enables both; to run one alone, set it directly with deploy.victoriametrics.enable or deploy.grafana.enable.

Log in to Grafana with your swarm SSO account. Grafana provisions its dashboards from the repo, so edits made in the UI aren't kept. Each service has its own board; open the folder to see them.

⚠️ The metrics store grows for as long as the swarm runs. Options to touch:

Option	When you'd touch it
deploy.victoriametrics.retentionPeriod	Default 5y. Lower it once you know how fast the swarm fills a disk. A too-short value discards history you can't get back.
swarm.grafana.oidc.role	Default Admin for everyone who logs in. Lower it to Viewer or Editor if some operators shouldn't reconfigure Grafana.

The dashboard list, the "No data" cases, and how the gateway guards the store: telemetry.md.

Logs (VictoriaLogs)

Each hive ships its journals to one VictoriaLogs at logs.<swarm-domain>, behind the same SSO as everything else: the swarm's service containers, the hive's daemons and the harness units in every agent container.

To read them, open Grafana, pick Explore and choose the VictoriaLogs datasource. Agents query it with swarm-logs, and need the logs permission.

Option	When you'd touch it
deploy.victorialogs.retentionPeriod	Default 30d, far shorter than the metrics store's. Raise it to answer questions about last quarter.

The other options, who may query, and the gateway rules: telemetry.md.

Telemetry collector (OTEL)

The swarm's collector receives from every hive's own collector and decides where telemetry goes. It always writes the metrics and log stores, and exports to deploy.swarm-otel.upstream.endpoint as well when you set one. It holds the upstream credential, so no hive and no agent needs one. Every hive reaches it by its gateway name, otel.<swarm-domain>, so a hive that doesn't run the swarm's services needs nothing set.

How the collector connects, its per-service routes and the homeserver's stand-in metrics: telemetry.md.

CI runner

One Forgejo Actions runner, in a swarm-ci-runner container on the host that sets deploy.ci-runner.enable. It doesn't derive from allSwarmServices: a runner executes pull-request code, so running one is a decision of its own.

It needs neither the forge nor a hive on its host. It reaches the forge at https://<swarm.forge.domain>, and registers with the token swarm-controller publishes to the secret store, which it reads under its own ci-runner leaf. On a host without the store, that leaf is the one file to copy. Registration, the container and the threat model are in 4-implementation/scheduler/ci.md.

The runner's built-in Prometheus endpoint is on, bound to 127.0.0.1:9101 in the container's own network namespace. The container's journal forwarder scrapes it (services.hyperhive.swarmJournalForwarder.extraScrapeTargets) and pushes the series to the swarm collector under the job gitea-runner, so nothing listens on the bridge. Series, all prefixed gitea_runner_: job_running, capacity, job_capacity_utilization_ratio, uptime_seconds, info (labels version and name), job_total{status}, job_duration_seconds, poll_fetch_total{result}, poll_fetch_duration_seconds, poll_backoff_seconds, report_log_*, report_state_* and client_errors_total. A pull-request job runs in the same namespace and can read the endpoint, which carries counts, durations, the runner's name and its version, and no secrets.

The ci runner board in Grafana shows these series. Its Runner filter (the series' service.instance.id, the scrape address) narrows every panel, and uptime, job slots and slot use show one series per runner, so a runner registered by hand appears beside the swarm's once its metrics reach the store under its own instance id. The forge's CI runs tab is the other view: run times and queue wait of every run, whichever runner ran it.

Builder

The swarm builder, in a swarm-builder container on the host that sets deploy.builder.enable. allSwarmServices defaults it on wherever the host has the builder's store identity, as for the review classifier below: the host that runs the store has it. Set deploy.builder.enable = false to opt out. It evaluates unreviewed config PRs and builds them through its host's nix daemon, from a container with its own network namespace.

When an agent-configs/<agent> PR opens, reopens or gets a new push, swarm-controller queues a build_agent job. Jobs run one at a time, and a newer push replaces its PR's job while that job is still waiting. The builder builds that config at the PR's head on the agent base its own host runs, through the host's nix daemon, and swarm-controller writes the result as a swarm-builder commit status on the head. The job shows on the swarm UI's Jobs page. A red status blocks nothing: the merge still deploys.

Separately, the builder builds each agent from internal/meta before the swarm sends its deploy, and a failed or unpushed build holds that deploy back (build_meta job; deploy gate). Those jobs share the one-build slot with the PR checks.

It needs neither the forge nor a hive on its host. It reaches the queue, the secret store and the forge by name, under its own builder leaf, and fetches the private config repo with the read-only token of its own builder forge account. On a host without the store, that leaf is the one file to copy, as for the CI runner. What it builds and why are in builder.md.

Binary cache

The swarm's attic server, in a swarm-attic container on the host that sets deploy.attic.enable. allSwarmServices defaults it on wherever the host has the cache's store identity, as for the builder: the host that runs the store has it. For an external cache, override swarm.attic.{domain,cache} and don't set deploy.attic.enable anywhere, or set it to lib.mkForce false under allSwarmServices.

The container sits on the hive bridge in a network namespace of its own, and atticd listens on that namespace's loopback only; the gateway reaches it over a unix socket. The gateway serves it at swarm.attic.domain (default attic.<swarm domain>, on every host) over TLS, with no login in front, because attic checks its own tokens. Storage is local disk in the container's state.

It needs no manual secret. Two host units, under the cache's own attic leaf, create its signing key, the cache and the push and pull tokens in the secret store, each only while absent. On a host without the store, that leaf is the one file to copy. The builder pushes green builds to it. How it works, and the external alternative, are in builder.md.

The container forwards atticd's journal to the swarm collector like the other service containers (see Telemetry collector). The two host units' lines reach the log store too: through the hive collector's journal allow-list on a hive host, or through the swarm collector's whole-journal receiver where the host runs that. A host with neither keeps them in its own journal.

CI metrics

The CI run poller, a swarm-ci-metrics oneshot on a 5-minute timer on the host that sets deploy.ci-metrics.enable. It runs on the host, not in a container. allSwarmServices defaults it on wherever the host has the poller's store identity, as for the builder above. Set deploy.ci-metrics.enable = false to opt out.

Each pass pushes the run time and queue wait of every newly finished forge Actions run to the metrics store, through its own route on the swarm collector; the first pass pushes every run since the first. It reads the forge with the read-only token of its own ci-metrics forge account, under its own ci-metrics leaf. On a host without the store, that leaf is the one file to copy. What it reads and pushes are in ci-metrics.md.

Review classifier

The review classifier's try-out service, in a swarm-review-classifier container on the host that sets deploy.review-classifier.enable. allSwarmServices defaults it on wherever the host has the classifier's store identity: the host that runs the store has it, and a services host with the store elsewhere gets the classifier only once it holds a copy of the pair (see below). Set deploy.review-classifier.enable directly to override either way.

The operator writes the key of the runtime the swarm settings choose by hand, in the secret store's UI under the OIDC login, field api_key:

• the API key runtime's Anthropic key at secret/swarm/services/review-classifier/anthropic-api-key
• the ACP runtime's provider key at secret/swarm/services/review-classifier/acp-provider

Nothing writes them for you: until the chosen one exists the service runs and every run fails with no API key… or no ACP provider key…. The service reads the key on every run, so writing or removing it takes effect on the next run, with no restart. Every run spends money on that key.

It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own review-classifier leaf. On a host without the store, that leaf is the one file to copy, as for the CI runner. The swarm UI's try-out drives it through swarm-controller; what a run is and what it costs are in review-classifier.md.

Claude login minter

The claude login minter, in a swarm-claude-login container on the host that sets deploy.claude-login.enable. allSwarmServices defaults it on wherever the host has the minter's store identity, on the review classifier's terms. Run one: a login in progress lives in the service's memory between its start and its code.

An operator logs an agent in from the log in button on its card in the swarm UI. swarm-controller relays the start and the pasted code over the queue, the minter runs claude auth login and writes the credentials to the agent's swarm/agents/<agent>/claude-credentials, and the agent's harness installs a version newer than the one it installed last.

It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own claude-login leaf, and reaches claude's login endpoints over the network. On a host without the store, that leaf is the one file to copy, as for the CI runner. The flow and the version rule are in claude-login.md.

Notification router

The notification router, in a swarm-notify-router container on the host that sets deploy.notify-router.enable. allSwarmServices defaults it on wherever the host has the router's store identity, on the review classifier's terms. Run one: it handles each issue's events in order.

swarm-controller keeps every issues and pull_request delivery, and every failed or cancelled CI run, from its instance-wide hooks in the SWARM_EVENTS stream. The router reads that stream and queues todos for the agents an event concerns: an agent assigned to an issue or pull request gets a todo, one taken off it gets that todo rewritten to say so, and closing it clears them. The author of a pull request gets a todo when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos from the ROUTER_TODOS stream, so an agent that's down gets them when it's back.

It needs neither the forge nor a hive on its host, and holds no forge token. It reaches the queue and the secret store by name, under its own notify-router leaf. On a host without the store, that leaf is the one file to copy, as for the CI runner. The flows and the delivery path are in notify-router.md.