Watch
0
0
Fork
You've already forked hyperhive
0
hyperhive/docs/swarm/services.md
2026-10-21 22:10:23 +02:00

475 lines
28 KiB
Markdown

| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. |
| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |
# Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge,
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
log stack. This page says which host runs them and what a hive that runs none
of them configures instead. The all-local quick start sets everything with one
line → [README](../../README.md#quick-start-an-all-local-swarm).
Two options say where they live, and everything else derives:
```nix
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
# or, for a dedicated services host with hives elsewhere:
services.hyperhive.deploy.allSwarmServices = true;
```
**`deploy.allSwarmServices` is what "the swarm's shared services run
here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service
sections below don't repeat it, so a service that stops deriving is a
visible difference rather than one more paragraph saying the same thing.
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the
host's `/etc/hosts` entries for the names it serves
(`gateway.localHostsEntry`) and where the secret store's bootstrap
token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived
toggle on its own, and that setting wins.
**Both default to off**, and that's deliberate: a host can't tell
whether it's meant to be the swarm's service host, so this is an
operator saying so rather than something inferred. With them off, a hive
is a _client_ of those services — it configures how to reach them and
runs none of them.
That includes the forge (`deploy.forgejo.enable`). Every swarm needs
one, being the canonical store for the meta flake and every agent's
config repo, so **exactly one host must turn it on**: `singleHostSwarm`,
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A
host that enables its services one by one without any of those runs no
forge.
A hive can't start or stop any of these. `hivectl stop`, `start` and
`restart` act on the hive's agents and its gateway only, even on the
host that runs the swarm's services. The forge and matrix containers are
systemd units on that host (`container@hive-forge.service`,
`container@hive-matrix.service`), and the operator manages them there
with `systemctl`.
A hive that runs none of these reaches each one by name — the forge at
`forge.<swarm-domain>`, for example. Only the host running a service
answers its name from its own resolver, so on a swarm spread over
more than one host, the operator's DNS has to resolve those names to that
host.
<details><summary>Moving an existing hive's forge to the swarm's</summary>
A hive that stops running the forge keeps the old container's state at
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
forge: push anything worth keeping there by hand. Its
`/var/lib/hyperhive/forge-core-token` came from that old forge and
fails against the swarm's one.
</details>
## Deployment shapes
Those two options are what makes the difference between deployments, so
the shapes worth naming are the ones they produce:
- **All-local.** Everything on one machine:
`singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for
a hive to run agents on. After the first switch, the steps in
[`setup.md`](../getting-started/setup.md) remain.
- **Services on the swarm controller host.**
Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there,
with hives elsewhere. The controller doesn't derive from
`allSwarmServices`.
- **Fully spread out.** One container / VM / machine per service,
somewhere.
**These are a set, not a ladder with a correct top, and hyperhive
supports in-between shapes.** You can set each derived toggle on its own (see
above), which is what makes "all local except X" a configuration rather
than an unsupported edge case. Nothing in hyperhive prescribes a
deployment model, so a doc that treats one shape as the real one and
the others as compromises is wrong about the system rather than
opinionated.
The practical consequence is that **co-location is normal**: in the
first two shapes services share a host by design. Where a specific
service has something to say about sharing a host --- extra
requirements, or a cost worth weighing --- that belongs in the doc for
that service, not here.
## Services
### SSO (authelia)
One authelia per swarm, in a `swarm-authelia` container, at
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
provider, differentiated by roles and claims rather than by mechanism —
there is one IdP and one auth path.
The container sits on the hive bridge in a network namespace of its own and
dials the store, the collector and its own public name through it. It binds
no TCP port: the login server, the metrics listener and the bridge each
listen on a unix socket in `/run/swarm-authelia`, a directory only the
gateway's nginx and the container's processes can open.
- **`deploy.authelia`** — run the container here.
- **`swarm.authelia.url`** — where clients go to authenticate. Present
on **every** hive and the same value on all of them:
`https://<swarm.authelia.domain>`, whether or not this host runs the
container. Set it explicitly when joining a swarm whose IdP is under
another name.
Agent subjects come from swarm-controller's agent-creation job, written
into the users database by `swarm-authelia-bridge`; human ones come from
`swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account).
The bridge listens on a unix socket on the authelia host, and the gateway
publishes it as
`https://<swarm.authelia.domain>/bridge/`, which is where
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
It answers only swarm-controller's OIDC client
(`swarm.controller.queueClientId`) and refuses every other token `403`.
On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia generates session and storage keys in the container on
first boot and never rotates them automatically; replacing one
invalidates data already written (sessions, the encrypted store), so
that's an operator action.
Storage is local sqlite and the notifier writes to a file. Both are
small-deployment choices, and the scope is the justification: redis
buys shared session state across replicas and there is one instance;
SMTP exists to mail humans, and provisioning here is programmatic.
The swarm collector scrapes authelia's own metrics at
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`.
The container also runs `swarm-secret-publish`, which copies the client
secrets authelia mints into the swarm secret store under the secret
publisher's own store leaf, handed in as a system credential; see
[`secrets.md`](secrets.md). Authelia doesn't wait for it, so the IdP comes up
while the store is down, and the publisher retries every 30 seconds. The
container's journal forwarder reads its client secret back from the store
under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`secrets.md`](secrets.md) for where authelia
generates and reads each of its keys.
### Metrics (VictoriaMetrics + Grafana)
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
reads it, in two containers at `metrics.<swarm-domain>` and
`grafana.<swarm-domain>`. Two containers rather than one so you can
restart or break Grafana without taking the time-series database with it.
They derive together: a store with no UI is unreadable and a UI with no
store is empty. To run one without the other, set it directly:
```nix
services.hyperhive.deploy.victoriametrics.enable = true;
services.hyperhive.deploy.grafana.enable = false;
```
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
<!-- vale write-good.Passive = NO -->
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones, and the local login form is always off. Grafana's API still
accepts HTTP basic auth for its built-in `admin` account, so that account's
password is a random one minted into the swarm secret store
(`swarm/services/<grafana client id>/admin/password`) and applied on every
Grafana start — see [`secrets.md`](secrets.md).
<!-- vale write-good.Passive = YES -->
**Dashboards.** Grafana provisions these from
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
- `forge`: what the forge holds, and the forge process's CPU, memory and descriptors.
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
- `collector`: what the OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
- `logstore`, `metricstore`: health of the log store and the metrics store.
A new panel can read "No data" until its source exists: the push and commit
counters on `controller` appear after the first push webhook, and the refused
and failure series on `collector` appear once something fails.
**Where the data comes from.** The swarm's OTEL collector, below.
The `controller` panels come from the controller itself. It exports OTLP
through the gateway to the swarm's collector from whatever host it runs on,
whether or not that host runs the collector.
Neither container is reachable except through the gateway: each listens
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
runs in a network namespace of its own with no interface but loopback.
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it:
- **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource, an agent, or an operator. Grafana sends a
bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
store, and `swarm-metrics-auth` on the store's host checks it against the
same store path. An agent's `swarm-metrics` sends its
`swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. `swarm-metrics-auth` reads the stored bearer at most once a
minute, so a replaced bearer starts working up to a minute later. After a
failed or timed-out read it denies for three seconds without trying again.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to
that path; any other session gets `deny`.
The collector scrapes the store's own metrics the same way, at
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
that doesn't run the store still gets them. On this name the gateway admits
only the collector's client to `/metrics` and the push route; any other
session gets `deny`.
### Logs (VictoriaLogs)
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
behind the same SSO as everything else: the swarm's own service containers,
the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
| Option | When you'd touch it |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. |
| `deploy.victorialogs.grafanaAuthPackage` | The `swarm-metrics-auth` build that runs as `swarm-logs-grafana-auth`. Set on its own: `deploy.victoriametrics.authPackage` doesn't reach it. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`secrets.md`](secrets.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. On this name the gateway admits only the
collector's client to `/metrics` and the ingest route
`/insert/opentelemetry/v1/logs`; any other session gets `deny` on both. The
read side stays open to any authenticated caller.
### Telemetry collector (OTEL)
The swarm's collector receives from every hive's own collector and is the
only process that decides where telemetry goes: it always writes the stores
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
always get both. It holds
the upstream credential, which is why no hive and no agent needs one.
It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to
`4319` rather than OTLP's usual `4318`, which the hive tier uses — swarm
containers share the host's network namespace, so two collectors on one
port is a coin toss at runtime rather than an error at build time.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a
loopback URL an operator has to redirect. Nothing needs setting on a hive
that doesn't run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter and classifier reuse the
service's own client; the store, CI runner, Grafana, log store, metrics store,
authelia, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
The swarm collector's own container, `swarm-otel`, has no forwarder: its
collector reads the container's journal itself (receiver `journald/self`) and
exports it to the log store like the host's journal, stamped `service.name`
`swarm-otel` and with no `hive` label. Unlike the host's journal, it goes to the
log store only, never the upstream: the collector writes an upstream export failure to this
journal, and shipping it upstream would fail again. A `filter` processor also
drops the collector's own failure lines that name the store exporter
(`otlphttp/victorialogs`), which can't reach the store anyway; failures of
every other exporter still arrive. The container mounts the host's journal
read-only at `/run/host-journal`, not over its own
`/var/log/journal`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in
[`../scheduler/observability.md`](../scheduler/observability.md).
### CI runner
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
runner executes pull-request code, so running one is a decision of its own.
It needs neither the forge nor a hive on its host. It reaches the forge at
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
publishes to the secret store, which it reads under its own `ci-runner` leaf.
On a host without the store, that leaf is the one file to copy. Registration,
the container and the threat model are in
[`../scheduler/ci.md`](../scheduler/ci.md).
### Builder
The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
has the builder's store identity, as for the review classifier below: the host
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It
evaluates unreviewed config PRs and builds them through its host's nix daemon,
from a container with its own network namespace.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
newer push replaces its PR's job while that job is still waiting. The builder
builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
### CI metrics
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
`allSwarmServices` defaults it on wherever the host has the poller's store
identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to
opt out.
Each pass pushes the run time and queue wait of every newly finished forge
Actions run to the metrics store, through its own route on the swarm collector;
the first pass pushes every run since the first. It reads the forge with the
read-only token of its own `ci-metrics` forge account, under its own
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
copy. What it reads and pushes are in [`ci-metrics.md`](ci-metrics.md).
### Review classifier
The review classifier's try-out service, in a `swarm-review-classifier`
container on the host that sets `deploy.review-classifier.enable`.
`allSwarmServices` defaults it on wherever the host has the classifier's store
identity: the host that runs the store has it, and a services host with the
store elsewhere gets the classifier only once it holds a copy of the pair (see
below). Set `deploy.review-classifier.enable` directly to override either way.
The operator writes the key of the runtime the swarm settings choose by hand,
in the secret store's UI under the OIDC login, field `api_key`:
- the API key runtime's Anthropic key at
`secret/swarm/services/review-classifier/anthropic-api-key`
- the ACP runtime's provider key at
`secret/swarm/services/review-classifier/acp-provider`
Nothing writes them for you: until the chosen one exists the service runs and
every run fails with `no API key…` or `no ACP provider key…`. The service reads
the key on every run, so writing or removing it takes effect on the next run,
with no restart. Every run spends money on that key.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `review-classifier` leaf. On a host
without the store, that leaf is the one file to copy, as for the CI runner.
The swarm UI's try-out drives it through swarm-controller; what a run is
and what it costs are in [`review-classifier.md`](review-classifier.md).
### Claude login minter
The claude login minter, in a `swarm-claude-login` container on the host that
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever
the host has the minter's store identity, on the review classifier's terms.
Run one: a login in progress lives in the service's memory between its start
and its code.
An operator logs an agent in from the **log in** button on its card in the
swarm UI. swarm-controller relays the start and the pasted code over the queue,
the minter runs `claude auth login` and writes the credentials to the agent's
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
version newer than the one it installed last.
It needs neither the forge nor a hive on its host. It reaches the queue and the
secret store by name, under its own `claude-login` leaf, and reaches claude's
login endpoints over the network. On a host without the store, that leaf is
the one file to copy, as for the CI runner. The flow and the version rule are
in [`claude-login.md`](claude-login.md).