# Swarm builder The swarm builder checks that an agent config PR builds before an operator merges it. Each time an `agent-configs/` PR opens, reopens or gets a new push, the builder builds the PR's head on the agent base, and the result shows as a `swarm-builder` commit status on the PR and as a `build_agent` node on the swarm UI's Jobs page. Where it runs and how it's enabled: [`services.md`](services.md#builder). ## What happens on a config PR 1. The forge delivers the `pull_request` event on the `config-pr` webhook swarm-controller already registers on `agent-configs`. An `opened`, `reopened` or `synchronized` action on an open PR queues one `build_agent` node carrying the agent, the head commit and the PR number. An edit, a label or a close builds nothing. Every `build_agent` node needs the one `builder` slot, so builds run one at a time and the rest wait pending. A newer head cancels its PR's pending build; a build already running finishes and writes its status on its own, older commit. 2. The node sets the status to `pending`, then sends the build as a request on the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the reply. The builder runs one build at a time and refuses a request that has waited 30 seconds for its turn, such as one behind a build a controller restart orphaned: a build started later could still run when the 61 minutes end. Reading the forge token before a build has 20 seconds, and the rest of the minute covers the reply's trip back. 3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output with every input overridden: - `hyperhive` and `nixpkgs`: the sources the builder's own host was deployed with (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`). A hive's meta flake pins `hyperhive` by forge URL and rev instead. - `agent`: `git+https:///agent-configs/?ref=refs/pull//head&rev=`. A build has one hour. 4. The reply is the system closure's store path, or the last lines `nix` printed. The node writes the status on the head commit: | Outcome | Status | Description | Node | | ----------------------------------------- | --------- | --------------------------- | ------ | | built | `success` | the store path | done | | `nix` failed or timed out | `failure` | the last line `nix` printed | failed | | the builder busy for 30 seconds | `failure` | why | failed | | no builder listening, or no reply in time | `error` | why | failed | The controller records every build that gets a final status, and the status links to that record: see [Build records](#build-records). The full tail is also in the failed node's error on the Jobs page and in the builder's journal line for the build, as the field `F_ERROR_TAIL`: ```console $ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json ``` The builder holds the last 64 KiB of each of `nix`'s output streams while the build runs, and the reply is the last 30 lines of that (at most 4000 bytes), so a long build log never grows its memory. ## Build records `swarm-controller` records each build when it writes the final status: built, failed, or no build ran or came back. The newest 100 are one JSON array under the key `recent` in the `builder-runs` KV bucket on the swarm queue, which only the controller reads or writes. It creates the bucket on first use and adds each build with a compare-and-set write, so the records survive a controller restart. Recording the 101st drops the oldest. A record the controller can't decode stays in the bucket, out of the list, and counts toward the 100. Each record has: - `id`: `--` - `repo` (`agent-configs/`), `pr` and `head_sha` - `started_unix` and `finished_unix` - `outcome`: `built`, `failed` or `error`, the status's `success`, `failure` or `error` - `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for `failed`, and `detail` (why) for `error` `GET /api/builder/runs` lists them, newest first, and `GET /api/builder/runs/` returns one, or 404 when it isn't among the newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503 when the controller has no swarm queue or can't open the bucket, 500 when it can't read it. The final status links to `https:///builds?run=`, the swarm UI's builds page, so selecting **Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record stays at `GET /api/builder/runs/`. A status has no link when the record wasn't stored (the controller logs the failed write) or when the controller doesn't serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending` status has no link. ## What the check is, and isn't - It composes the config the way a hive's meta flake does (`agent-base.extendModules` with the config's `nixosModules.default`), minus what a hive adds: its CA bundle, `claude-code` path, ports and OTEL settings, and the `follows` that point the config's own inputs at the hive's. A green check can therefore still fail on a hive, and the reverse. - The agent base is the builder's host's, not the hive's. Hives on another hyperhive or nixpkgs pin build something else. - The builder pushes nothing. The closure stays in the builder host's store until it's collected, so on a host that also runs the hive, the hive's own build can reuse whatever matches. - It doesn't gate anything. A merge deploys whatever the status says. - A lost webhook delivery isn't retried: the head isn't built until the next push. The jobs graph is in memory, so a controller restart drops a build in flight and leaves its status `pending`. ## Archetype evaluation The builder also answers `$SWARM.builder.archetypes`: it evaluates the `meta` of every agent archetype its own hyperhive source declares, with no forge token, and answers with the JSON. An evaluation doesn't wait for a build in progress and has two minutes. What the controller does with the answer: [`archetypes.md`](archetypes.md). ## Meta-flake lock The builder also answers `$SWARM.builder.lock-meta`. The request carries a rendered `flake.nix`. The builder writes it into a scratch directory, runs `nix flake lock` there with the same forge credential helper its builds use, answers with the `flake.lock` that `nix` wrote, and removes the directory. A lock doesn't wait for a build in progress and has ten minutes. The builder commits nothing. swarm-controller sends the request. It renders `internal/meta`'s `flake.nix` with the same renderer hive-c0re uses (`hive-meta-flake`), all over `https`. Each custom agent that a hive places pins to its `agent-configs/` `main`; the controller omits and logs one whose repo has no commit on `main`, so it doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own running rev (`services.hyperhive.c0re.hyperhiveRev`, from `self.rev`): - the first commit pins hyperhive's forge `main`; - later, the pin moves to the controller's rev only when the forge's compare API shows that rev is a strict descendant of the committed pin: commits ahead of it, and none the pin has that it lacks. The compare endpoint counts histories with no common ancestor as zero commits ahead, so the check needs both directions; - an older controller, a rev on another line, a controller built from a dirty tree or a forge that can't answer leaves the pin. A merge to hyperhive's `main` moves nothing. The controller commits `flake.nix` and the returned `flake.lock` to `internal/meta` in one commit, and gives each hive's forge account read on the repo. It re-renders when it starts, before it deploys a merged config PR, and before the first deploy of a new agent. Those deploys run whether or not the update succeeded: a hive that builds from `internal/meta` does so only while it pins the agent's current config (see [Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render equal to the committed `flake.nix` sends no lock. When a commit moves the pin from one rev to another, the controller sends a deploy to every agent a hive places, so they rebuild on the new hyperhive. A restart that moves nothing deploys nothing, and the first commit deploys nothing. ## Credentials | What | Where | Who reads it | | ----------------------- | ------------------------------------------------- | -------------------------------------------------- | | store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential | | queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it | | forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build | The token belongs to the `builder` forge account, which swarm-controller creates, puts in the `hives` team of `agent-configs` (read on every config repo) and mints a `read:repository` token for, in a five-minute pass. `git` gets the token through a credential helper reading the child's environment: it's in no argument, URL or lock, so no line `nix` prints can carry it into a status. The same `git` fetches a config's own inputs, so the builder registers the helper for the forge's URL only, and the helper answers only an `https` request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the status with its own forge token. Details per path: [`credentials.md`](credentials.md). ## Isolation The container has its own network namespace on the bridge, as the CI runner does, so a config's evaluation can't reach a host-loopback service. Evaluation is pure and runs in the container. Builds go to the host's nix daemon as an untrusted client: a config can set no daemon option, and its derivations build in the host's sandbox. A fixed-output derivation still has the network, as it does for any build on that host.