bbc6174361
on deploy, one agent whose agent-configs repo has no commit on main is skipped and logged instead of failing the whole internal/meta render, so the other agents' pin moves go through. the controller reads each config head from main, the branch hive-c0re compares against, not the repo's default branch.
184 lines
10 KiB
Markdown
184 lines
10 KiB
Markdown
# Swarm builder
|
|
|
|
The swarm builder checks that an agent config PR builds before an operator
|
|
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
|
push, the builder builds the PR's head on the agent base, and the result shows as a
|
|
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
|
swarm UI's Jobs page.
|
|
|
|
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|
|
|
## What happens on a config PR
|
|
|
|
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
|
swarm-controller already registers on `agent-configs`. An `opened`,
|
|
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
|
node carrying the agent, the head commit and the PR number. An edit, a
|
|
label or a close builds nothing. Every `build_agent` node needs the one
|
|
`builder` slot, so builds run one at a time and the rest wait pending. A
|
|
newer head cancels its PR's pending build; a build already running
|
|
finishes and writes its status on its own, older commit.
|
|
2. The node sets the status to `pending`, then sends the build as a request on
|
|
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
|
reply. The builder runs one build at a time and refuses a request that has
|
|
waited 30 seconds for its turn, such as one behind a build a controller
|
|
restart orphaned: a build started later could still run when the 61
|
|
minutes end. Reading the forge token before a build has 20 seconds, and
|
|
the rest of the minute covers the reply's trip back.
|
|
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
|
with every input overridden:
|
|
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
|
deployed with (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`). A hive's meta
|
|
flake pins `hyperhive` by forge URL and rev instead.
|
|
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
|
A build has one hour.
|
|
4. The reply is the system closure's store path, or the last lines `nix`
|
|
printed. The node writes the status on the head commit:
|
|
|
|
| Outcome | Status | Description | Node |
|
|
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
|
| built | `success` | the store path | done |
|
|
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
|
| the builder busy for 30 seconds | `failure` | why | failed |
|
|
| no builder listening, or no reply in time | `error` | why | failed |
|
|
|
|
The controller records every build that gets a final status, and the
|
|
status links to that record: see [Build records](#build-records). The
|
|
full tail is also in the failed node's error on the Jobs page and in the
|
|
builder's journal line for the build, as the field `F_ERROR_TAIL`:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
|
|
```
|
|
|
|
The builder holds the last 64 KiB of each of `nix`'s output streams while the
|
|
build runs, and the reply is the last 30 lines of that (at most 4000
|
|
bytes), so a long build log never grows its memory.
|
|
|
|
## Build records
|
|
|
|
`swarm-controller` records each build when it writes the final status:
|
|
built, failed, or no build ran or came back. The newest 100 are one JSON
|
|
array under the key `recent` in the `builder-runs` KV bucket on the swarm
|
|
queue, which only the controller reads or writes. It creates the bucket on
|
|
first use and adds each build with a compare-and-set write, so the records
|
|
survive a controller restart. Recording the 101st drops the oldest. A record
|
|
the controller can't decode stays in the bucket, out of the list, and counts
|
|
toward the 100.
|
|
|
|
Each record has:
|
|
|
|
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>`
|
|
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`
|
|
- `started_unix` and `finished_unix`
|
|
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
|
|
or `error`
|
|
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
|
|
`failed`, and `detail` (why) for `error`
|
|
|
|
`GET /api/builder/runs` lists them, newest first, and
|
|
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
|
|
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
|
|
when the controller has no swarm queue or can't open the bucket, 500 when it
|
|
can't read it.
|
|
|
|
The final status links to `https://<swarm domain>/builds?run=<id>`, the swarm
|
|
UI's builds page, so selecting **Details** on a red `swarm-builder` check opens
|
|
that build's card, with `nix`'s tail, in the browser. The raw record stays at
|
|
`GET /api/builder/runs/<id>`. A status has no link when the record wasn't
|
|
stored (the controller logs the failed write) or when the controller doesn't
|
|
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
|
|
status has no link.
|
|
|
|
## What the check is, and isn't
|
|
|
|
- It composes the config the way a hive's meta flake does
|
|
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
|
what a hive adds: its CA bundle, `claude-code` path, ports and
|
|
OTEL settings, and the `follows` that point the config's own inputs at the
|
|
hive's. A green check can therefore still fail on a hive, and the reverse.
|
|
- The agent base is the builder's host's, not the hive's. Hives on another
|
|
hyperhive or nixpkgs pin build something else.
|
|
- The builder pushes nothing. The closure stays in the builder
|
|
host's store until it's collected, so on a host that also runs the hive, the
|
|
hive's own build can reuse whatever matches.
|
|
- It doesn't gate anything. A merge deploys whatever the status says.
|
|
- A lost webhook delivery isn't retried: the head isn't built until the next
|
|
push. The jobs graph is in memory, so a controller restart drops a build in
|
|
flight and leaves its status `pending`.
|
|
|
|
## Archetype evaluation
|
|
|
|
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
|
|
`meta` of every agent archetype its own hyperhive source declares, with no
|
|
forge token, and answers with the JSON. An evaluation doesn't wait for a
|
|
build in progress and has two minutes. What the controller does with the
|
|
answer: [`archetypes.md`](archetypes.md).
|
|
|
|
## Meta-flake lock
|
|
|
|
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
|
|
rendered `flake.nix`. The builder writes it into a scratch directory, runs
|
|
`nix flake lock` there with the same forge credential helper its builds use,
|
|
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
|
|
lock doesn't wait for a build in progress and has ten minutes. The builder
|
|
commits nothing.
|
|
|
|
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
|
|
with the same renderer hive-c0re uses (`hive-meta-flake`), all over `https`.
|
|
Each custom agent that a hive places pins to its `agent-configs/<agent>`
|
|
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
|
|
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
|
|
running rev (`services.hyperhive.c0re.hyperhiveRev`, from `self.rev`):
|
|
|
|
- the first commit pins hyperhive's forge `main`;
|
|
- later, the pin moves to the controller's rev only when the forge's compare
|
|
API shows that rev is a strict descendant of the committed pin: commits
|
|
ahead of it, and none the pin has that it lacks. The compare endpoint counts
|
|
histories with no common ancestor as zero commits ahead, so the check needs
|
|
both directions;
|
|
- an older controller, a rev on another line, a controller built from a dirty
|
|
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
|
|
`main` moves nothing.
|
|
|
|
The controller commits `flake.nix` and the returned `flake.lock` to
|
|
`internal/meta` in one commit, and gives each hive's forge account read on the
|
|
repo. It re-renders when it starts, before it deploys a merged config PR, and
|
|
before the first deploy of a new agent. Those deploys run whether or not the
|
|
update succeeded: a hive that builds from `internal/meta` does so only while it
|
|
pins the agent's current config (see
|
|
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
|
equal to the committed `flake.nix` sends no lock.
|
|
|
|
When a commit moves the pin from one rev to another, the controller sends a
|
|
deploy to every agent a hive places, so they rebuild on the new hyperhive. A
|
|
restart that moves nothing deploys nothing, and the first commit deploys
|
|
nothing.
|
|
|
|
## Credentials
|
|
|
|
| What | Where | Who reads it |
|
|
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
|
|
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
|
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
|
|
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
|
|
|
The token belongs to the `builder` forge account, which swarm-controller
|
|
creates, puts in the `hives` team of `agent-configs` (read on every config
|
|
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
|
|
gets the token through a credential helper reading the child's environment:
|
|
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
|
status. The same `git` fetches a config's own inputs, so the builder registers
|
|
the helper for the forge's URL only, and the helper answers only an `https`
|
|
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
|
|
status with its own forge token. Details per path:
|
|
[`credentials.md`](credentials.md).
|
|
|
|
## Isolation
|
|
|
|
The container has its own network namespace on the bridge, as the CI runner
|
|
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
|
is pure and runs in the container. Builds go to the host's nix daemon as an
|
|
untrusted client: a config can set no daemon option, and its derivations build
|
|
in the host's sandbox. A fixed-output derivation still has the network, as it
|
|
does for any build on that host.
|