368 lines
21 KiB
Markdown
368 lines
21 KiB
Markdown
# Swarm-wide services
|
|
|
|
Some things exist once per **swarm** rather than once per hive: the forge,
|
|
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
|
|
log stack. This page says which host runs them and what a hive that runs none
|
|
of them configures instead. The all-local quick start sets everything with one
|
|
line → [README](../../README.md#quick-start-an-all-local-swarm).
|
|
|
|
Two options say where they live, and everything else derives:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
|
|
# or, for a dedicated services host with hives elsewhere:
|
|
services.hyperhive.deploy.allSwarmServices = true;
|
|
```
|
|
|
|
**`deploy.allSwarmServices` is what "the swarm's shared services run
|
|
here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service
|
|
sections below don't repeat it, so a service that stops deriving is a
|
|
visible difference rather than one more paragraph saying the same thing.
|
|
|
|
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
|
|
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the
|
|
host's `/etc/hosts` entries for the names it serves
|
|
(`gateway.localHostsEntry`) and where the secret store's bootstrap
|
|
token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived
|
|
toggle on its own, and that setting wins.
|
|
|
|
**Both default to off**, and that's deliberate: a host can't tell
|
|
whether it's meant to be the swarm's service host, so this is an
|
|
operator saying so rather than something inferred. With them off, a hive
|
|
is a _client_ of those services — it configures how to reach them and
|
|
runs none of them.
|
|
|
|
That includes the forge (`deploy.forgejo.enable`). Every swarm needs
|
|
one, being the canonical store for the meta flake and every agent's
|
|
config repo, so **exactly one host must turn it on**: `singleHostSwarm`,
|
|
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A
|
|
host that enables its services one by one without any of those runs no
|
|
forge.
|
|
|
|
A hive that runs none of these reaches each one by name — the forge at
|
|
`forge.<swarm-domain>`, for example. Only the host running a service
|
|
answers its name from its own resolver, so on a swarm spread over
|
|
more than one host, the operator's DNS has to resolve those names to that
|
|
host.
|
|
|
|
<details><summary>Moving an existing hive's forge to the swarm's</summary>
|
|
|
|
A hive that stops running the forge keeps the old container's state at
|
|
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
|
|
forge: push anything worth keeping there by hand. Its
|
|
`/var/lib/hyperhive/forge-core-token` came from that old forge and
|
|
fails against the swarm's one.
|
|
|
|
</details>
|
|
|
|
## Deployment shapes
|
|
|
|
Those two options are what makes the difference between deployments, so
|
|
the shapes worth naming are the ones they produce:
|
|
|
|
- **All-local.** Everything on one machine:
|
|
`singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for
|
|
a hive to run agents on. After the first switch, the steps in
|
|
[`setup.md`](../getting-started/setup.md) remain.
|
|
- **Services on the swarm controller host.**
|
|
Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there,
|
|
with hives elsewhere. The controller doesn't derive from
|
|
`allSwarmServices`.
|
|
- **Fully spread out.** One container / VM / machine per service,
|
|
somewhere.
|
|
|
|
**These are a set, not a ladder with a correct top, and hyperhive
|
|
supports in-between shapes.** You can set each derived toggle on its own (see
|
|
above), which is what makes "all local except X" a configuration rather
|
|
than an unsupported edge case. Nothing in hyperhive prescribes a
|
|
deployment model, so a doc that treats one shape as the real one and
|
|
the others as compromises is wrong about the system rather than
|
|
opinionated.
|
|
|
|
The practical consequence is that **co-location is normal**: in the
|
|
first two shapes services share a host by design. Where a specific
|
|
service has something to say about sharing a host --- extra
|
|
requirements, or a cost worth weighing --- that belongs in the doc for
|
|
that service, not here.
|
|
|
|
## Services
|
|
|
|
### SSO (authelia)
|
|
|
|
One authelia per swarm, in a `swarm-authelia` container, at
|
|
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
|
|
provider, differentiated by roles and claims rather than by mechanism —
|
|
there is one IdP and one auth path.
|
|
|
|
- **`deploy.authelia`** — run the container here.
|
|
- **`swarm.authelia.url`** — where clients go to authenticate. Present
|
|
on **every** hive and the same value on all of them:
|
|
`https://<swarm.authelia.domain>`, whether or not this host runs the
|
|
container. Set it explicitly when joining a swarm whose IdP is under
|
|
another name.
|
|
|
|
Agent subjects come from swarm-controller's agent-creation job, written
|
|
into the users database by `swarm-authelia-bridge`; human ones come from
|
|
`swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account).
|
|
|
|
The bridge listens on loopback on the authelia host, and the gateway
|
|
publishes it as
|
|
`https://<swarm.authelia.domain>/bridge/`, which is where
|
|
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
|
|
It answers only swarm-controller's OIDC client
|
|
(`swarm.controller.queueClientId`) and refuses every other token `403`.
|
|
|
|
On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia generates session and storage keys in the container on
|
|
first boot and never rotates them automatically; replacing one
|
|
invalidates data already written (sessions, the encrypted store), so
|
|
that's an operator action.
|
|
|
|
Storage is local sqlite and the notifier writes to a file. Both are
|
|
small-deployment choices, and the scope is the justification: redis
|
|
buys shared session state across replicas and there is one instance;
|
|
SMTP exists to mail humans, and provisioning here is programmatic.
|
|
|
|
The swarm collector scrapes authelia's own metrics at
|
|
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
|
|
container. The gateway admits only the collector's OAuth2 client there;
|
|
any other session gets `deny`.
|
|
|
|
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
|
|
relying-party flow, and [`secrets.md`](secrets.md) for where authelia
|
|
generates and reads each of its keys.
|
|
|
|
### Metrics (VictoriaMetrics + Grafana)
|
|
|
|
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
|
|
reads it, in two containers at `metrics.<swarm-domain>` and
|
|
`grafana.<swarm-domain>`. Two containers rather than one so you can
|
|
restart or break Grafana without taking the time-series database with it.
|
|
|
|
They derive together: a store with no UI is unreadable and a UI with no
|
|
store is empty. To run one without the other, set it directly:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.victoriametrics.enable = true;
|
|
services.hyperhive.deploy.grafana.enable = false;
|
|
```
|
|
|
|
⚠️ **This starts a database that grows for as long as the swarm runs.**
|
|
See `retentionPeriod` below before leaving it at its default.
|
|
|
|
| Option | When you'd touch it |
|
|
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
|
|
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
|
|
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
|
|
authelia ones, and the local login form is always off. Grafana's API still
|
|
accepts HTTP basic auth for its built-in `admin` account, so that account's
|
|
password is a random one minted into the swarm secret store
|
|
(`swarm/services/<grafana client id>/admin/password`) and applied on every
|
|
Grafana start — see [`secrets.md`](secrets.md).
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
**Dashboards.** Grafana provisions these from
|
|
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
|
|
kept.
|
|
|
|
- `agents`: per-agent resource usage and Claude Code activity.
|
|
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
|
|
- `claude-usage`: Claude use by model, effort, token type and source.
|
|
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
|
|
- `forge` and `forgejo`: what the forge holds, and the forge process's CPU, memory and descriptors.
|
|
- `collector`: what the OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
|
|
- `logstore`, `metricstore`: health of the log store and the metrics store.
|
|
|
|
A new panel can read "No data" until its source exists: the push and commit
|
|
counters on `controller` appear after the first push webhook, and the refused
|
|
and failure series on `collector` appear once something fails.
|
|
|
|
**Where the data comes from.** The swarm's OTEL collector, below.
|
|
|
|
Neither container is reachable except through the gateway: each listens
|
|
on a unix socket only the gateway's nginx can open, and VictoriaMetrics
|
|
runs in a network namespace of its own with no interface but loopback.
|
|
VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
|
|
vhost does it:
|
|
|
|
- **The read API** (the Prometheus query, series, label and metadata
|
|
routes): Grafana's datasource or an operator. Grafana sends a bearer that
|
|
`swarm-bao-grafana-metrics-bearer` creates in the swarm secret store, and
|
|
`swarm-metrics-auth` on the store's host checks it against the same store
|
|
path. A request without a bearer needs an operator's authelia session
|
|
(group `admins`); without one it gets a 401, not a login redirect.
|
|
- **vmui**: an operator's authelia session, with a redirect to the login.
|
|
The bare domain redirects to vmui.
|
|
- **The collector's OTLP push, and its scrape of the store's own
|
|
`/metrics`**: only the swarm collector's authelia client. Authelia refuses
|
|
any other session or client there, operators included.
|
|
- **Every other path**, the store's write and admin API included: 403.
|
|
|
|
`swarm-metrics-auth` reads the store with a leaf of its own,
|
|
`deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
|
|
Where the secret store runs on the same host, these default to the leaf it
|
|
mints. On any other host you place a leaf whose CN is
|
|
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
|
|
you set both. It reads the stored bearer at most once a minute, so a
|
|
replaced bearer starts working up to a minute later.
|
|
|
|
The collector scrapes Grafana's own metrics at
|
|
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without
|
|
Grafana still gets them. The gateway admits only the collector's client to
|
|
that path; any other session gets `deny`.
|
|
|
|
The collector scrapes the store's own metrics the same way, at
|
|
`https://<swarm.victoriametrics.domain>/metrics`, so a collector on a host
|
|
that doesn't run the store still gets them. On this name the gateway admits
|
|
only the collector's client to `/metrics` and the push route; any other
|
|
session gets `deny`.
|
|
|
|
### Logs (VictoriaLogs)
|
|
|
|
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
|
|
behind the same SSO as everything else: the swarm's own service containers,
|
|
the hive's daemons and infra containers, and the harness units inside every
|
|
agent container. The collector below is what writes to it.
|
|
|
|
**Reading them.** Open Grafana, pick **Explore**, and choose the
|
|
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
|
|
Drilldown_ app is deliberately not installed: it only supports Loki, and
|
|
no setting here changes that, so Explore is the log browser for this
|
|
swarm.
|
|
|
|
| Option | When you'd touch it |
|
|
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
|
|
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
|
|
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
|
|
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. |
|
|
| `swarm.victorialogs.domain` | Only to rename it. |
|
|
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
|
|
|
|
It binds loopback and takes no credential of its own: the gateway vhost is
|
|
the only way in, and the collector is the only intended writer.
|
|
|
|
The collector also scrapes this store's own metrics, at
|
|
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
|
|
without the store still gets them. The gateway admits only the collector's
|
|
client to that path; any other session gets `deny`.
|
|
|
|
### Telemetry collector (OTEL)
|
|
|
|
The swarm's collector receives from every hive's own collector and is the
|
|
only process that decides where telemetry goes: it always writes the stores
|
|
above, and exports to `otel.endpoint` as well when that's set. It holds
|
|
the upstream credential, which is why no hive and no agent needs one.
|
|
|
|
It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to
|
|
`4319` rather than OTLP's usual `4318`, which the hive tier uses — swarm
|
|
containers share the host's network namespace, so two collectors on one
|
|
port is a coin toss at runtime rather than an error at build time.
|
|
|
|
Every hive's own collector reaches this one by its gateway name,
|
|
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
|
|
by-domain-through-the-gateway shape every other swarm service uses, not a
|
|
loopback URL an operator has to redirect. Nothing needs setting on a hive
|
|
that doesn't run the swarm's services; the name resolves through the
|
|
gateway either way.
|
|
|
|
| Option | When you'd touch it |
|
|
| ------------------- | --------------------------------------------------------------------------------------- |
|
|
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
|
|
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
|
|
|
|
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
|
|
doesn't gate them: that option says this host _runs_ the store, while the swarm
|
|
has one either way, reached by its swarm name through the gateway.
|
|
|
|
Agent-side configuration, and what a hive's own collector does, are in
|
|
[`../scheduler/observability.md`](../scheduler/observability.md).
|
|
|
|
### CI runner
|
|
|
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
|
runner executes pull-request code, so running one is a decision of its own.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the forge at
|
|
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
|
|
publishes to the secret store, which it reads under its own `ci-runner` leaf.
|
|
On a host without the store, that leaf is the one file to copy. Registration,
|
|
the container and the threat model are in
|
|
[`../scheduler/ci.md`](../scheduler/ci.md).
|
|
|
|
### Builder
|
|
|
|
The swarm builder, in a `swarm-builder` container on the host that sets
|
|
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
|
|
has the builder's store identity, as for the review classifier below: the host
|
|
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It
|
|
evaluates unreviewed config PRs and builds them through its host's nix daemon,
|
|
from a container with its own network namespace.
|
|
|
|
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
|
|
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
|
|
newer push replaces its PR's job while that job is still waiting. The builder
|
|
builds that config at the PR's head on the agent base its own host runs,
|
|
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
|
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
|
status blocks nothing: the merge still deploys.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the queue, the
|
|
secret store and the forge by name, under its own `builder` leaf, and fetches
|
|
the private config repo with the read-only token of its own `builder` forge
|
|
account. On a host without the store, that leaf is the one file to copy, as
|
|
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
|
|
|
### Review classifier
|
|
|
|
The review classifier's try-out service, in a `swarm-review-classifier`
|
|
container on the host that sets `deploy.review-classifier.enable`.
|
|
`allSwarmServices` defaults it on wherever the host has the classifier's store
|
|
identity: the host that runs the store has it, and a services host with the
|
|
store elsewhere gets the classifier only once it holds a copy of the pair (see
|
|
below). Set `deploy.review-classifier.enable` directly to override either way.
|
|
|
|
The operator writes the key of the runtime the swarm settings choose by hand,
|
|
in the secret store's UI under the OIDC login, field `api_key`:
|
|
|
|
- the API key runtime's Anthropic key at
|
|
`secret/swarm/services/review-classifier/anthropic-api-key`
|
|
- the ACP runtime's provider key at
|
|
`secret/swarm/services/review-classifier/acp-provider`
|
|
|
|
Nothing writes them for you: until the chosen one exists the service runs and
|
|
every run fails with `no API key…` or `no ACP provider key…`. The service reads
|
|
the key on every run, so writing or removing it takes effect on the next run,
|
|
with no restart. Every run spends money on that key.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the queue and the
|
|
secret store by name, under its own `review-classifier` leaf. On a host
|
|
without the store, that leaf is the one file to copy, as for the CI runner.
|
|
The swarm UI's try-out drives it through swarm-controller; what a run is
|
|
and what it costs are in [`review-classifier.md`](review-classifier.md).
|
|
|
|
### Claude login minter
|
|
|
|
The claude login minter, in a `swarm-claude-login` container on the host that
|
|
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever
|
|
the host has the minter's store identity, on the review classifier's terms.
|
|
Run one: a login in progress lives in the service's memory between its start
|
|
and its code.
|
|
|
|
An operator logs an agent in from the **log in** button on its card in the
|
|
swarm UI. swarm-controller relays the start and the pasted code over the queue,
|
|
the minter runs `claude auth login` and writes the credentials to the agent's
|
|
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
|
|
version newer than the one it installed last.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the queue and the
|
|
secret store by name, under its own `claude-login` leaf, and reaches claude's
|
|
login endpoints over the network. On a host without the store, that leaf is
|
|
the one file to copy, as for the CI runner. The flow and the version rule are
|
|
in [`claude-login.md`](claude-login.md).
|