cafa120d77
argus's review on #5177: the git credential helper answered every `get`, so a config PR's own flake inputs could have git send the builder's forge token to any host. The helper is now configured under `credential.<forge-url>.helper` and itself answers only an https request whose host is the forge's; a test runs it as git does and checks that other hosts, http and non-get operations get nothing. Builds are bounded: every `build_agent` node needs the one `builder` jobq resource, so one runs at a time and the rest wait pending, and a newer head cancels its PR's pending build (a running one finishes and writes on its own commit). The builder also holds a one-permit semaphore around each build. `builder` and `review-classifier`, the forge logins swarm-controller owns, join nix/reserved-names.nix, with a test tying the list to both constants. The builder is now opt-in like the CI runner: `allSwarmServices` no longer turns it on, since it runs unreviewed config PRs through the host's nix daemon. mara had not answered atlas's question on the PR.
286 lines
17 KiB
Markdown
286 lines
17 KiB
Markdown
# Swarm-wide services
|
|
|
|
Some things exist once per **swarm** rather than once per hive: the forge,
|
|
the matrix homeserver, SSO, the secret store, the queue, and the metrics and
|
|
log stack. This page says which host runs them and what a hive that runs none
|
|
of them configures instead. The all-local quick start sets everything with one
|
|
line → [README](../../README.md#quick-start-an-all-local-swarm).
|
|
|
|
Two options say where they live, and everything else derives:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.singleHostSwarm = true; # everything on this box
|
|
# or, for a dedicated services host with hives elsewhere:
|
|
services.hyperhive.deploy.allSwarmServices = true;
|
|
```
|
|
|
|
**`deploy.allSwarmServices` is what "the swarm's shared services run
|
|
here" means: every once-per-swarm service takes its `enable` from it.** That's the whole rule, stated once — the per-service
|
|
sections below don't repeat it, so a service that stops deriving is a
|
|
visible difference rather than one more paragraph saying the same thing.
|
|
|
|
`singleHostSwarm` is the all-on-one-box mode above it. It defaults
|
|
`deploy.allSwarmServices`, the swarm controller (`deploy.swarm-controller.enable`), the
|
|
host's `/etc/hosts` entries for the names it serves
|
|
(`gateway.localHostsEntry`) and where the secret store's bootstrap
|
|
token goes (`deploy.bao.bootstrapTokenFile`). You can still set each derived
|
|
toggle on its own, and that setting wins.
|
|
|
|
**Both default to off**, and that's deliberate: a host can't tell
|
|
whether it's meant to be the swarm's service host, so this is an
|
|
operator saying so rather than something inferred. With them off, a hive
|
|
is a _client_ of those services — it configures how to reach them and
|
|
runs none of them.
|
|
|
|
That includes the forge (`deploy.forgejo.enable`). Every swarm needs
|
|
one, being the canonical store for the meta flake and every agent's
|
|
config repo, so **exactly one host must turn it on**: `singleHostSwarm`,
|
|
`allSwarmServices`, or `deploy.forgejo.enable = true` set by hand. A
|
|
host that enables its services one by one without any of those runs no
|
|
forge.
|
|
|
|
A hive that runs none of these reaches each one by name — the forge at
|
|
`forge.<swarm-domain>`, for example. Only the host running a service
|
|
answers its name from its own resolver, so on a swarm spread over
|
|
more than one host, the operator's DNS has to resolve those names to that
|
|
host.
|
|
|
|
<details><summary>Moving an existing hive's forge to the swarm's</summary>
|
|
|
|
A hive that stops running the forge keeps the old container's state at
|
|
`/var/lib/nixos-containers/hive-forge/`. Nothing moves it to the swarm's
|
|
forge: push anything worth keeping there by hand. Its
|
|
`/var/lib/hyperhive/forge-core-token` came from that old forge and
|
|
fails against the swarm's one.
|
|
|
|
</details>
|
|
|
|
## Deployment shapes
|
|
|
|
Those two options are what makes the difference between deployments, so
|
|
the shapes worth naming are the ones they produce:
|
|
|
|
- **All-local.** Everything on one machine:
|
|
`singleHostSwarm = true`, plus `deploy.hive-controller.enable = true` for
|
|
a hive to run agents on. After the first switch, the steps in
|
|
[`setup.md`](../getting-started/setup.md) remain.
|
|
- **Services on the swarm controller host.**
|
|
Set `deploy.allSwarmServices` and `deploy.swarm-controller.enable` there,
|
|
with hives elsewhere. The controller doesn't derive from
|
|
`allSwarmServices`.
|
|
- **Fully spread out.** One container / VM / machine per service,
|
|
somewhere.
|
|
|
|
**These are a set, not a ladder with a correct top, and hyperhive
|
|
supports in-between shapes.** You can set each derived toggle on its own (see
|
|
above), which is what makes "all local except X" a configuration rather
|
|
than an unsupported edge case. Nothing in hyperhive prescribes a
|
|
deployment model, so a doc that treats one shape as the real one and
|
|
the others as compromises is wrong about the system rather than
|
|
opinionated.
|
|
|
|
The practical consequence is that **co-location is normal**: in the
|
|
first two shapes services share a host by design. Where a specific
|
|
service has something to say about sharing a host --- extra
|
|
requirements, or a cost worth weighing --- that belongs in the doc for
|
|
that service, not here.
|
|
|
|
## Services
|
|
|
|
### SSO (authelia)
|
|
|
|
One authelia per swarm, in a `swarm-authelia` container, at
|
|
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
|
|
provider, differentiated by roles and claims rather than by mechanism —
|
|
there is one IdP and one auth path.
|
|
|
|
- **`deploy.authelia`** — run the container here.
|
|
- **`swarm.authelia.url`** — where clients go to authenticate. Present
|
|
on **every** hive and the same value on all of them:
|
|
`https://<swarm.authelia.domain>`, whether or not this host runs the
|
|
container. Set it explicitly when joining a swarm whose IdP is under
|
|
another name.
|
|
|
|
Agent subjects come from swarm-controller's agent-creation job, written
|
|
into the users database by `swarm-authelia-bridge`; human ones come from
|
|
`swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account).
|
|
|
|
The bridge listens on loopback on the authelia host, and the gateway
|
|
publishes it as
|
|
`https://<swarm.authelia.domain>/bridge/`, which is where
|
|
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
|
|
It answers only swarm-controller's OIDC client
|
|
(`swarm.controller.queueClientId`) and refuses every other token `403`.
|
|
|
|
On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia generates session and storage keys in the container on
|
|
first boot and never rotates them automatically; replacing one
|
|
invalidates data already written (sessions, the encrypted store), so
|
|
that's an operator action.
|
|
|
|
Storage is local sqlite and the notifier writes to a file. Both are
|
|
small-deployment choices, and the scope is the justification: redis
|
|
buys shared session state across replicas and there is one instance;
|
|
SMTP exists to mail humans, and provisioning here is programmatic.
|
|
|
|
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
|
|
relying-party flow, and [`secrets.md`](secrets.md) for where authelia
|
|
generates and reads each of its keys.
|
|
|
|
### Metrics (VictoriaMetrics + Grafana)
|
|
|
|
The swarm's telemetry lands in one VictoriaMetrics, and one Grafana
|
|
reads it, in two containers at `metrics.<swarm-domain>` and
|
|
`grafana.<swarm-domain>`. Two containers rather than one so you can
|
|
restart or break Grafana without taking the time-series database with it.
|
|
|
|
They derive together: a store with no UI is unreadable and a UI with no
|
|
store is empty. To run one without the other, set it directly:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.victoriametrics.enable = true;
|
|
services.hyperhive.deploy.grafana.enable = false;
|
|
```
|
|
|
|
⚠️ **This starts a database that grows for as long as the swarm runs.**
|
|
See `retentionPeriod` below before leaving it at its default.
|
|
|
|
| Option | When you'd touch it |
|
|
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
|
|
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
|
|
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
|
|
authelia ones, and the local login form is always off. Grafana's API still
|
|
accepts HTTP basic auth for its built-in `admin` account, so that account's
|
|
password is a random one minted into the swarm secret store
|
|
(`swarm/services/<grafana client id>/admin/password`) and applied on every
|
|
Grafana start — see [`secrets.md`](secrets.md).
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
**Where the data comes from.** The swarm's OTEL collector, below.
|
|
|
|
Neither container is reachable except through the gateway: both bind
|
|
loopback, and VictoriaMetrics authenticates nothing itself. The
|
|
`metrics.<swarm-domain>` vhost forwards only the store's read API (the
|
|
Prometheus query, series, label and metadata routes, plus the vmui web UI)
|
|
without a credential. It forwards the collector's OTLP push only with an
|
|
authelia token, and refuses every other path, the store's write and admin
|
|
API included, with 403. The bare domain redirects to vmui.
|
|
|
|
### Logs (VictoriaLogs)
|
|
|
|
Each hive ships its journals to one VictoriaLogs at `logs.<swarm-domain>`,
|
|
behind the same SSO as everything else: the swarm's own service containers,
|
|
the hive's daemons and infra containers, and the harness units inside every
|
|
agent container. The collector below is what writes to it.
|
|
|
|
**Reading them.** Open Grafana, pick **Explore**, and choose the
|
|
`VictoriaLogs` datasource — it's provisioned for you. Grafana's _Logs
|
|
Drilldown_ app is deliberately not installed: it only supports Loki, and
|
|
no setting here changes that, so Explore is the log browser for this
|
|
swarm.
|
|
|
|
| Option | When you'd touch it |
|
|
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are typically read within days of ingestion. Raise it if you need to answer questions about last quarter. |
|
|
| `deploy.victorialogs.authRate` | Default `50r/s` per client address, for agent queries checked by `swarm-logs-auth`; over it and `authBurst`, the gateway answers 403. Agents on one host share the address; the default covers about a hundred cold-starting together. |
|
|
| `deploy.victorialogs.authBurst` | Default `500` requests above `authRate` before the 403s start. |
|
|
| `deploy.victorialogs.authVerdictTtl` | Default `60` seconds that `swarm-logs-auth` keeps granting an agent secret it verified, without asking the secret store. A replaced or deleted secret keeps working at the log store for up to that long. |
|
|
| `swarm.victorialogs.domain` | Only to rename it. |
|
|
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
|
|
|
|
Like the metrics store, it binds loopback and takes no credential of its
|
|
own: the gateway vhost is the only way in, and the collector is the only
|
|
intended writer.
|
|
|
|
### Telemetry collector (OTEL)
|
|
|
|
The swarm's collector receives from every hive's own collector and is the
|
|
only process that decides where telemetry goes: it always writes the stores
|
|
above, and exports to `otel.endpoint` as well when that's set. It holds
|
|
the upstream credential, which is why no hive and no agent needs one.
|
|
|
|
It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to
|
|
`4319` rather than OTLP's usual `4318`, which the hive tier uses — swarm
|
|
containers share the host's network namespace, so two collectors on one
|
|
port is a coin toss at runtime rather than an error at build time.
|
|
|
|
Every hive's own collector reaches this one by its gateway name,
|
|
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
|
|
by-domain-through-the-gateway shape every other swarm service uses, not a
|
|
loopback URL an operator has to redirect. Nothing needs setting on a hive
|
|
that doesn't run the swarm's services; the name resolves through the
|
|
gateway either way.
|
|
|
|
| Option | When you'd touch it |
|
|
| ------------------- | --------------------------------------------------------------------------------------- |
|
|
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
|
|
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
|
|
|
|
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
|
|
doesn't gate them: that option says this host _runs_ the store, while the swarm
|
|
has one either way, reached by its swarm name through the gateway.
|
|
|
|
Agent-side configuration, and what a hive's own collector does, are in
|
|
[`../scheduler/observability.md`](../scheduler/observability.md).
|
|
|
|
### CI runner
|
|
|
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
|
runner executes pull-request code, so running one is a decision of its own.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the forge at
|
|
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
|
|
publishes to the secret store, which it reads under its own `ci-runner` leaf.
|
|
On a host without the store, that leaf is the one file to copy. Registration,
|
|
the container and the threat model are in
|
|
[`../scheduler/ci.md`](../scheduler/ci.md).
|
|
|
|
### Builder
|
|
|
|
The swarm builder, in a `swarm-builder` container on the host that sets
|
|
`deploy.builder.enable`. It doesn't derive from `allSwarmServices`: it
|
|
evaluates and builds unreviewed config PRs through its host's nix daemon, so
|
|
running one is a decision of its own, as for the CI runner.
|
|
|
|
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
|
|
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
|
|
newer push replaces its PR's job while that job is still waiting. The builder
|
|
builds that config at the PR's head on the agent base its own host runs,
|
|
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
|
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
|
status blocks nothing: the merge still deploys.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the queue, the
|
|
secret store and the forge by name, under its own `builder` leaf, and fetches
|
|
the private config repo with the read-only token of its own `builder` forge
|
|
account. On a host without the store, that leaf is the one file to copy, as
|
|
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
|
|
|
### Review classifier
|
|
|
|
The review classifier's try-out service, in a `swarm-review-classifier`
|
|
container on the host that sets `deploy.review-classifier.enable`.
|
|
`allSwarmServices` defaults it on wherever the host has the classifier's store
|
|
identity: the host that runs the store has it, and a services host with the
|
|
store elsewhere gets the classifier only once it holds a copy of the pair (see
|
|
below). Set `deploy.review-classifier.enable` directly to override either way.
|
|
|
|
The operator writes the Anthropic key by hand, in the secret store's UI under
|
|
the OIDC login, at
|
|
`secret/swarm/services/review-classifier/anthropic-api-key`, field `api_key`.
|
|
Nothing writes it for you: until it exists the service runs and every run fails
|
|
with `no API key…`. The service reads the key on every run, so writing or
|
|
removing it takes effect on the next run, with no restart. Every run spends
|
|
money on that key.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the queue and the
|
|
secret store by name, under its own `review-classifier` leaf. On a host
|
|
without the store, that leaf is the one file to copy, as for the CI runner.
|
|
The swarm UI's try-out page drives it through swarm-controller; what a run is
|
|
and what it costs are in [`review-classifier.md`](review-classifier.md).
|