272 lines
17 KiB
Markdown
272 lines
17 KiB
Markdown
# Swarm builder
|
|
|
|
The swarm builder checks that an agent config PR builds before an operator
|
|
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
|
push, the builder builds the PR's head on the agent base, and the result shows as a
|
|
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
|
swarm UI's Jobs page.
|
|
|
|
The builder also builds each agent from `internal/meta` before the swarm sends
|
|
its deploy, and a failed build holds the deploy back: see
|
|
[Deploy gate](#deploy-gate).
|
|
|
|
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|
|
|
## What happens on a config PR
|
|
|
|
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
|
swarm-controller already registers on `agent-configs`. An `opened`,
|
|
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
|
node carrying the agent, the head commit and the PR number. An edit, a
|
|
label or a close builds nothing. Every `build_agent` node needs the one
|
|
`builder` slot, so builds run one at a time and the rest wait pending. A
|
|
newer head cancels its PR's pending build; a build already running
|
|
finishes and writes its status on its own, older commit.
|
|
2. The node sets the status to `pending`, then sends the build as a request on
|
|
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
|
reply. The builder runs one build at a time and refuses a request that has
|
|
waited 30 seconds for its turn, such as one behind a build a controller
|
|
restart orphaned: a build started later could still run when the 61
|
|
minutes end. Reading the forge token before a build has 20 seconds, and
|
|
the rest of the minute covers the reply's trip back.
|
|
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
|
with every input overridden:
|
|
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
|
deployed with (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`). A hive's meta
|
|
flake pins `hyperhive` by forge URL and rev instead.
|
|
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
|
A build has one hour.
|
|
4. The reply is the system closure's store path, or the last lines `nix`
|
|
printed. The node writes the status on the head commit:
|
|
|
|
| Outcome | Status | Description | Node |
|
|
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
|
| built | `success` | the store path | done |
|
|
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
|
| the builder busy for 30 seconds | `failure` | why | failed |
|
|
| no builder listening, or no reply in time | `error` | why | failed |
|
|
|
|
The controller records every build that gets a final status, and the
|
|
status links to that record: see [Build records](#build-records). The
|
|
full tail is also in the failed node's error on the Jobs page and in the
|
|
builder's journal line for the build, as the field `F_ERROR_TAIL`:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
|
|
```
|
|
|
|
The builder holds the last 64 KiB of each of `nix`'s output streams while the
|
|
build runs, and the reply is the last 30 lines of that (at most 4000
|
|
bytes), so a long build log never grows its memory.
|
|
|
|
## Build records
|
|
|
|
`swarm-controller` records each build when it writes the final status:
|
|
built, failed, or no build ran or came back. The newest 100 are one JSON
|
|
array under the key `recent` in the `builder-runs` KV bucket on the swarm
|
|
queue, which only the controller reads or writes. It creates the bucket on
|
|
first use and adds each build with a compare-and-set write, so the records
|
|
survive a controller restart. Recording the 101st drops the oldest. A record
|
|
the controller can't decode stays in the bucket, out of the list, and counts
|
|
toward the 100.
|
|
|
|
Each record has:
|
|
|
|
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>`
|
|
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`
|
|
- `started_unix` and `finished_unix`
|
|
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
|
|
or `error`
|
|
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
|
|
`failed`, and `detail` (why) for `error`
|
|
|
|
`GET /api/builder/runs` lists them, newest first, and
|
|
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
|
|
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
|
|
when the controller has no swarm queue or can't open the bucket, 500 when it
|
|
can't read it.
|
|
|
|
The final status links to `https://<swarm domain>/builds?run=<id>`, the swarm
|
|
UI's builds page, so selecting **Details** on a red `swarm-builder` check opens
|
|
that build's card, with `nix`'s tail, in the browser. The raw record stays at
|
|
`GET /api/builder/runs/<id>`. A status has no link when the record wasn't
|
|
stored (the controller logs the failed write) or when the controller doesn't
|
|
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
|
|
status has no link.
|
|
|
|
## What the check is, and isn't
|
|
|
|
- It composes the config the way a hive's meta flake does
|
|
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
|
what a hive adds: its CA bundle, `claude-code` path, ports and
|
|
OTEL settings, and the `follows` that point the config's own inputs at the
|
|
hive's. A green check can therefore still fail on a hive, and the reverse.
|
|
- The agent base is the builder's host's, not the hive's. Hives on another
|
|
hyperhive or nixpkgs pin build something else.
|
|
- The builder pushes nothing. The closure stays in the builder
|
|
host's store until it's collected, so on a host that also runs the hive, the
|
|
hive's own build can reuse whatever matches.
|
|
- The PR check gates nothing. A merge deploys whatever the status says. The
|
|
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
|
|
- A lost webhook delivery isn't retried: the head isn't built until the next
|
|
push. The jobs graph is in memory, so a controller restart drops a build in
|
|
flight and leaves its status `pending`.
|
|
|
|
## Archetype evaluation
|
|
|
|
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
|
|
`meta` of every agent archetype its own hyperhive source declares, with no
|
|
forge token, and answers with the JSON. An evaluation doesn't wait for a
|
|
build in progress and has five minutes; the controller waits six. The builder
|
|
logs each evaluation's time as `elapsed_ms`, and a timeout's error carries the last
|
|
lines nix printed. What the controller does with the answer: [`archetypes.md`](archetypes.md).
|
|
|
|
## Meta-flake lock
|
|
|
|
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
|
|
rendered `flake.nix`. The builder writes it into a scratch directory, runs
|
|
`nix flake lock` there with the same forge credential helper its builds use,
|
|
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
|
|
lock doesn't wait for a build in progress and has ten minutes. The builder
|
|
commits nothing.
|
|
|
|
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
|
|
with the `hive-meta-flake` renderer, all over `https`.
|
|
Each custom agent that a hive places pins to its `agent-configs/<agent>`
|
|
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
|
|
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
|
|
running rev (`services.hyperhive.c0re.hyperhiveRev`, from `self.rev`):
|
|
|
|
- the first commit pins hyperhive's forge `main`;
|
|
- later, the pin moves to the controller's rev only when the forge's compare
|
|
API shows that rev is a strict descendant of the committed pin: commits
|
|
ahead of it, and none the pin has that it lacks. The compare endpoint counts
|
|
histories with no common ancestor as zero commits ahead, so the check needs
|
|
both directions;
|
|
- an older controller, a rev on another line, a controller built from a dirty
|
|
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
|
|
`main` moves nothing.
|
|
|
|
The controller commits `flake.nix` and the returned `flake.lock` to
|
|
`internal/meta` in one commit, and gives each hive's forge account read on the
|
|
repo. It re-renders when it starts, and again every ten minutes while the latest
|
|
update hasn't succeeded (a tick adds nothing while one is pending or running,
|
|
or after a success), before it deploys a merged config PR, and
|
|
before the first deploy of a new agent. Those deploys run only after the
|
|
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
|
|
builds only from `internal/meta`, so after a failed update the scheduler skips
|
|
the deploy node, and the graph shows the update as failed and the deploy as
|
|
skipped. Each ten-minute retry of a failed update declares a fresh build and
|
|
deploy behind it for every agent whose latest deploy the failed update skipped,
|
|
so the run that succeeds sends them (see
|
|
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
|
equal to the committed `flake.nix` sends no lock.
|
|
|
|
The controller sends a deploy to every agent a hive places whenever the pin in
|
|
`internal/meta` differs from the pin it last sent, so they rebuild on the new
|
|
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
|
|
which only the controller reads or writes, and a `RecordFanout` job writes the
|
|
key once every deploy of the fan-out has finished or the deploy gate has skipped
|
|
it. The check runs after each update and once when the controller starts.
|
|
|
|
- The first commit sends a deploy, because the key holds no pin yet.
|
|
- A controller that starts with no key, as after the upgrade that added it,
|
|
sends one round. A deploy is idempotent per pin, so the round is harmless.
|
|
- A controller that dies after the commit, or a deploy that fails, leaves the
|
|
key behind. The next update or start sends the deploys again.
|
|
- A deploy the gate skipped counts as sent. A failed build holds only its own
|
|
agent back, and the key still moves, so one broken agent doesn't resend the
|
|
healthy ones on every retry. The timer declares a build and deploy again for
|
|
a deploy skipped because its build failed.
|
|
- A restart that finds the key equal to the pin sends nothing.
|
|
- When the bucket won't open, or its key won't read, the controller logs a
|
|
warning and sends deploys only when its own update moved the pin. That case
|
|
writes no key.
|
|
|
|
## Deploy gate
|
|
|
|
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
|
|
the flake a hive builds from. Four paths send deploys, and each declares the same
|
|
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
|
|
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
|
|
and needs the one `builder` slot, so it queues behind PR checks. The deploy
|
|
runs only when the build is green.
|
|
|
|
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
|
|
most paths declare it before the update that commits that head. It sends the
|
|
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
|
|
minutes. The builder takes the same one-build slot as PR checks, and
|
|
refuses a request that waited 30 seconds for it. It runs `nix build` of
|
|
`nixosConfigurations.<agent>.config.system.build.toplevel` from
|
|
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
|
|
the committed `flake.lock` decides every input. Like the hive's build, it uses
|
|
that attribute, so a green build here means the flake evaluates and builds for
|
|
the builder's platform. The hive doesn't reuse the result yet: it still builds
|
|
locally, from whatever head it reads at deploy time, which can be newer than
|
|
the one the swarm built.
|
|
|
|
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
|
|
Jobs page as a failed `build_meta` node, and the timer builds it again:
|
|
|
|
| What happened | `build_meta` node | Deploy |
|
|
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
|
|
| built | done, closure recorded | runs |
|
|
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
|
|
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
|
|
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
|
|
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
|
|
|
|
On every ten-minute round the timer declares a build and deploy again for each
|
|
agent whose latest build failed, without a meta update when the latest update
|
|
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
|
|
still records the pin when the gate skips a deploy, so one failing agent doesn't
|
|
resend the healthy ones. A controller that updates before its builder host does
|
|
is the same case: the old builder has no subscriber for the new subject, so the
|
|
controller gets no responders at once, and the deploys go out after the builder
|
|
updates. Each round adds a build and deploy node per failing agent.
|
|
|
|
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
|
|
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
|
|
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
|
|
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
|
|
not updating.
|
|
|
|
On a built reply the controller writes the agent's key in the `agent-closures`
|
|
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
|
|
platform and the time. Only the controller writes the bucket, and the builder
|
|
holds no grant on it. A failed build writes nothing and leaves the previous key.
|
|
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
|
|
store.
|
|
|
|
The controller gives the `builder` forge account read on `internal/meta` when it
|
|
updates the flake, and again before each build, since the repo is private. A
|
|
grant the forge refuses fails the build with no verdict.
|
|
|
|
## Credentials
|
|
|
|
| What | Where | Who reads it |
|
|
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
|
|
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
|
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
|
|
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
|
|
|
The token belongs to the `builder` forge account, which swarm-controller
|
|
creates, puts in the `hives` team of `agent-configs` (read on every config
|
|
repo, and on `internal/meta` through the meta update) and mints a
|
|
`read:repository` token for, in a five-minute pass. `git`
|
|
gets the token through a credential helper reading the child's environment:
|
|
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
|
status. The same `git` fetches a config's own inputs, so the builder registers
|
|
the helper for the forge's URL only, and the helper answers only an `https`
|
|
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
|
|
status with its own forge token. Details per path:
|
|
[`credentials-reference.md`](credentials-reference.md).
|
|
|
|
## Isolation
|
|
|
|
The container has its own network namespace on the bridge, as the CI runner
|
|
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
|
is pure and runs in the container. Builds go to the host's nix daemon as an
|
|
untrusted client: a config can set no daemon option, and its derivations build
|
|
in the host's sandbox. A fixed-output derivation still has the network, as it
|
|
does for any build on that host.
|