hyperhive/docs/swarm/services.md

Swarm-wide services

Some things exist once per swarm rather than once per hive: the forge, the matrix homeserver, SSO, the secret store, the queue, and the metrics and log stack. This page says which host runs them and what a hive that runs none of them configures instead. The all-local quick start sets everything with one line → README.

Two options say where they live, and everything else derives:

services.hyperhive.deploy.singleHostSwarm = true;   # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.deploy.allSwarmServices = true;

deploy.allSwarmServices is what "the swarm's shared services run here" means: every once-per-swarm service takes its enable from it. That's the whole rule, stated once — the per-service sections below don't repeat it, so a service that stops deriving is a visible difference rather than one more paragraph saying the same thing.

singleHostSwarm is the all-on-one-box mode above it. It defaults deploy.allSwarmServices, the swarm controller (deploy.swarm-controller.enable), the host's /etc/hosts entries for the names it serves (gateway.localHostsEntry) and where the secret store's bootstrap token goes (deploy.bao.bootstrapTokenFile). You can still set each derived toggle on its own, and that setting wins.

Both default to off, and that's deliberate: a host can't tell whether it's meant to be the swarm's service host, so this is an operator saying so rather than something inferred. With them off, a hive is a client of those services — it configures how to reach them and runs none of them.

That includes the forge (deploy.forgejo.enable). Every swarm needs one, being the canonical store for the meta flake and every agent's config repo, so exactly one host must turn it on: singleHostSwarm, allSwarmServices, or deploy.forgejo.enable = true set by hand. A host that enables its services one by one without any of those runs no forge.

A hive that runs none of these reaches each one by name — the forge at forge.<swarm-domain>, for example. Only the host running a service answers its name from its own resolver, so on a swarm spread over more than one host, the operator's DNS has to resolve those names to that host.

Moving an existing hive's forge to the swarm's

A hive that stops running the forge keeps the old container's state at /var/lib/nixos-containers/hive-forge/. Nothing moves it to the swarm's forge: push anything worth keeping there by hand. Its /var/lib/hyperhive/forge-core-token came from that old forge and fails against the swarm's one.

Deployment shapes

Those two options are what makes the difference between deployments, so the shapes worth naming are the ones they produce:

• All-local. Everything on one machine: singleHostSwarm = true, plus deploy.hive-controller.enable = true for a hive to run agents on. After the first switch, the steps in setup.md remain.
• Services on the swarm controller host. Set deploy.allSwarmServices and deploy.swarm-controller.enable there, with hives elsewhere. The controller doesn't derive from allSwarmServices.
• Fully spread out. One container / VM / machine per service, somewhere.

These are a set, not a ladder with a correct top, and hyperhive supports in-between shapes. You can set each derived toggle on its own (see above), which is what makes "all local except X" a configuration rather than an unsupported edge case. Nothing in hyperhive prescribes a deployment model, so a doc that treats one shape as the real one and the others as compromises is wrong about the system rather than opinionated.

The practical consequence is that co-location is normal: in the first two shapes services share a host by design. Where a specific service has something to say about sharing a host --- extra requirements, or a cost worth weighing --- that belongs in the doc for that service, not here.

Services

SSO (authelia)

One authelia per swarm, in a swarm-authelia container, at auth.<swarm-domain>. Operator and agents are both subjects of the same provider, differentiated by roles and claims rather than by mechanism — there is one IdP and one auth path.

• deploy.authelia — run the container here.
• swarm.authelia.url — where clients go to authenticate. Present on every hive and the same value on all of them: https://<swarm.authelia.domain>, whether or not this host runs the container. Set it explicitly when joining a swarm whose IdP is under another name.

Agent subjects come from swarm-controller's agent-creation job, written into the users database by swarm-authelia-bridge; human ones come from swarmctl user add → setup.md § 2.

The bridge listens on loopback on the authelia host, and the gateway publishes it as https://<swarm.authelia.domain>/bridge/, which is where deploy.swarm-controller.authBridgeUrl points by default on every host. It answers only swarm-controller's OIDC client (swarm.controller.queueClientId) and refuses every other token 403.

On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (swarm-authelia.nix:873). Authelia generates session and storage keys in the container on first boot and never rotates them automatically; replacing one invalidates data already written (sessions, the encrypted store), so that's an operator action.

Storage is local sqlite and the notifier writes to a file. Both are small-deployment choices, and the scope is the justification: redis buys shared session state across replicas and there is one instance; SMTP exists to mail humans, and provisioning here is programmatic.

The swarm collector scrapes authelia's own metrics at https://<swarm.authelia.domain>/metrics, whichever host runs the container. The gateway admits only the collector's OAuth2 client there; any other session gets deny.

See sso.md for bootstrapping the first user and the OIDC relying-party flow, and secrets.md for where authelia generates and reads each of its keys.

Metrics (VictoriaMetrics + Grafana)

The swarm's telemetry lands in one VictoriaMetrics, and one Grafana reads it, in two containers at metrics.<swarm-domain> and grafana.<swarm-domain>. Two containers rather than one so you can restart or break Grafana without taking the time-series database with it.

They derive together: a store with no UI is unreadable and a UI with no store is empty. To run one without the other, set it directly:

services.hyperhive.deploy.victoriametrics.enable = true;
services.hyperhive.deploy.grafana.enable = false;

⚠️ This starts a database that grows for as long as the swarm runs. See retentionPeriod below before leaving it at its default.

Option	When you'd touch it
deploy.victoriametrics.retentionPeriod	Default 5y. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back.
swarm.grafana.oidc.role	Default Admin for everyone who logs in. Lower to Viewer/Editor if the swarm grows operators who shouldn't be able to reconfigure Grafana.
deploy.grafana.datasourceUrl	Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (https://<swarm.victoriametrics.domain>), so Grafana and the store can run on different hosts.

Logging in. Grafana is behind swarm SSO, so the accounts are the authelia ones, and the local login form is always off. Grafana's API still accepts HTTP basic auth for its built-in admin account, so that account's password is a random one minted into the swarm secret store (swarm/services/<grafana client id>/admin/password) and applied on every Grafana start — see secrets.md.

Where the data comes from. The swarm's OTEL collector, below.

Neither container is reachable except through the gateway: both bind loopback, and VictoriaMetrics authenticates nothing itself. The metrics.<swarm-domain> vhost forwards only the store's read API (the Prometheus query, series, label and metadata routes, plus the vmui web UI) without a credential. It forwards the collector's OTLP push, and the collector's scrape of the store's own /metrics, only with an authelia token, and refuses every other path, the store's write and admin API included, with 403. The bare domain redirects to vmui.

Logs (VictoriaLogs)

Each hive ships its journals to one VictoriaLogs at logs.<swarm-domain>, behind the same SSO as everything else: the swarm's own service containers, the hive's daemons and infra containers, and the harness units inside every agent container. The collector below is what writes to it.

Reading them. Open Grafana, pick Explore, and choose the VictoriaLogs datasource — it's provisioned for you. Grafana's Logs Drilldown app is deliberately not installed: it only supports Loki, and no setting here changes that, so Explore is the log browser for this swarm.

Option	When you'd touch it
deploy.victorialogs.retentionPeriod	Default 30d, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter.
deploy.victorialogs.authRate	Default 50r/s per client address, for agent queries checked by swarm-logs-auth; over it and authBurst, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together.
deploy.victorialogs.authBurst	Default 500 requests above authRate before the 403s start.
deploy.victorialogs.authVerdictTtl	Default 60 seconds that swarm-logs-auth keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long.
swarm.victorialogs.domain	Only to rename it.
swarm.victorialogs.port	Only if something else on the services host already claims 9428.

Like the metrics store, it binds loopback and takes no credential of its own: the gateway vhost is the only way in, and the collector is the only intended writer.

Telemetry collector (OTEL)

The swarm's collector receives from every hive's own collector and is the only process that decides where telemetry goes: it always writes the stores above, and exports to otel.endpoint as well when that's set. It holds the upstream credential, which is why no hive and no agent needs one.

It runs in a swarm-otel container. Its swarm.otel.port defaults to 4319 rather than OTLP's usual 4318, which the hive tier uses — swarm containers share the host's network namespace, so two collectors on one port is a coin toss at runtime rather than an error at build time.

Every hive's own collector reaches this one by its gateway name, swarm.otel.domain (default otel.<swarm domain>) — the same by-domain-through-the-gateway shape every other swarm service uses, not a loopback URL an operator has to redirect. Nothing needs setting on a hive that doesn't run the swarm's services; the name resolves through the gateway either way.

Option	When you'd touch it
swarm.otel.domain	Only to rename it — the default already resolves correctly for every hive in the swarm.
swarm.otel.port	Only if something else on the services host already claims 4319.

Both store exporters are unconditional, and deploy.victoriametrics.enable doesn't gate them: that option says this host runs the store, while the swarm has one either way, reached by its swarm name through the gateway.

Agent-side configuration, and what a hive's own collector does, are in ../scheduler/observability.md.

CI runner

One Forgejo Actions runner, in a swarm-ci-runner container on the host that sets deploy.ci-runner.enable. It doesn't derive from allSwarmServices: a runner executes pull-request code, so running one is a decision of its own.

It needs neither the forge nor a hive on its host. It reaches the forge at https://<swarm.forge.domain>, and registers with the token swarm-controller publishes to the secret store, which it reads under its own ci-runner leaf. On a host without the store, that leaf is the one file to copy. Registration, the container and the threat model are in ../scheduler/ci.md.

Builder

The swarm builder, in a swarm-builder container on the host that sets deploy.builder.enable. It doesn't derive from allSwarmServices: it evaluates and builds unreviewed config PRs through its host's nix daemon, so running one is a decision of its own, as for the CI runner.

When an agent-configs/<agent> PR opens, reopens or gets a new push, swarm-controller queues a build_agent job. Jobs run one at a time, and a newer push replaces its PR's job while that job is still waiting. The builder builds that config at the PR's head on the agent base its own host runs, through the host's nix daemon, and swarm-controller writes the result as a swarm-builder commit status on the head. The job shows on the swarm UI's Jobs page. A red status blocks nothing: the merge still deploys.

It needs neither the forge nor a hive on its host. It reaches the queue, the secret store and the forge by name, under its own builder leaf, and fetches the private config repo with the read-only token of its own builder forge account. On a host without the store, that leaf is the one file to copy, as for the CI runner. What it builds and why are in builder.md.

Review classifier

The review classifier's try-out service, in a swarm-review-classifier container on the host that sets deploy.review-classifier.enable. allSwarmServices defaults it on wherever the host has the classifier's store identity: the host that runs the store has it, and a services host with the store elsewhere gets the classifier only once it holds a copy of the pair (see below). Set deploy.review-classifier.enable directly to override either way.

The operator writes the key of the runtime the swarm settings choose by hand, in the secret store's UI under the OIDC login, field api_key:

• the API key runtime's Anthropic key at secret/swarm/services/review-classifier/anthropic-api-key
• the ACP runtime's provider key at secret/swarm/services/review-classifier/acp-provider

Nothing writes them for you: until the chosen one exists the service runs and every run fails with no API key… or no ACP provider key…. The service reads the key on every run, so writing or removing it takes effect on the next run, with no restart. Every run spends money on that key.

It needs neither the forge nor a hive on its host. It reaches the queue and the secret store by name, under its own review-classifier leaf. On a host without the store, that leaf is the one file to copy, as for the CI runner. The swarm UI's try-out page drives it through swarm-controller; what a run is and what it costs are in review-classifier.md.