e873195cf6
nix_build::run read nix's stdout and stderr with read_to_end, so a long --print-build-logs run held its whole log in memory until nix exited, though the reply uses only the last 30 lines (4000 bytes) of stderr and the last non-empty line of stdout. Both streams now go through a Tail that holds at most 128 KiB and keeps the last 64 KiB. Output under 64 KiB reaches tail() whole, so those replies are unchanged.
90 lines
5.3 KiB
Markdown
90 lines
5.3 KiB
Markdown
# Swarm builder
|
|
|
|
The swarm builder checks that an agent config PR builds before an operator
|
|
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
|
push, the builder builds the PR's head on the agent base, and the result shows as a
|
|
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
|
swarm UI's Jobs page.
|
|
|
|
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|
|
|
## What happens on a config PR
|
|
|
|
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
|
swarm-controller already registers on `agent-configs`. An `opened`,
|
|
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
|
node carrying the agent, the head commit and the PR number. An edit, a
|
|
label or a close builds nothing. Every `build_agent` node needs the one
|
|
`builder` slot, so builds run one at a time and the rest wait pending. A
|
|
newer head cancels its PR's pending build; a build already running
|
|
finishes and writes its status on its own, older commit.
|
|
2. The node sets the status to `pending`, then sends the build as a request on
|
|
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
|
reply.
|
|
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
|
with every input overridden:
|
|
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
|
deployed with, the same two a hive on that host renders into its meta
|
|
flake (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`).
|
|
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
|
A build has one hour.
|
|
4. The reply is the system closure's store path, or the last lines `nix`
|
|
printed. The node writes the status on the head commit:
|
|
|
|
| Outcome | Status | Description | Node |
|
|
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
|
| built | `success` | the store path | done |
|
|
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
|
| no builder listening, or no reply in time | `error` | why | failed |
|
|
|
|
The full tail is in the failed node's error on the Jobs page and in the
|
|
builder's journal line for the build.
|
|
|
|
The builder holds the last 64 KiB of each of `nix`'s output streams while the
|
|
build runs, and the reply is the last 30 lines of that (at most 4000
|
|
bytes), so a long build log never grows its memory.
|
|
|
|
## What the check is, and isn't
|
|
|
|
- It composes the config the way a hive's meta flake does
|
|
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
|
what a hive adds: its CA bundle, docs source, `claude-code` path, ports and
|
|
OTEL settings, and the `follows` that point the config's own inputs at the
|
|
hive's. A green check can therefore still fail on a hive, and the reverse.
|
|
- The agent base is the builder's host's, not the hive's. Hives on another
|
|
hyperhive or nixpkgs pin build something else.
|
|
- The builder pushes nothing and records nothing. The closure stays in the builder
|
|
host's store until it's collected, so on a host that also runs the hive, the
|
|
hive's own build can reuse whatever matches.
|
|
- It doesn't gate anything. A merge deploys whatever the status says.
|
|
- A lost webhook delivery isn't retried: the head isn't built until the next
|
|
push. The jobs graph is in memory, so a controller restart drops a build in
|
|
flight and leaves its status `pending`.
|
|
|
|
## Credentials
|
|
|
|
| What | Where | Who reads it |
|
|
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
|
|
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
|
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
|
|
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
|
|
|
The token belongs to the `builder` forge account, which swarm-controller
|
|
creates, puts in the `hives` team of `agent-configs` (read on every config
|
|
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
|
|
gets the token through a credential helper reading the child's environment:
|
|
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
|
status. The same `git` fetches a config's own inputs, so the builder registers
|
|
the helper for the forge's URL only, and the helper answers only an `https`
|
|
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
|
|
status with its own forge token. Details per path:
|
|
[`credentials.md`](credentials.md).
|
|
|
|
## Isolation
|
|
|
|
The container has its own network namespace on the bridge, as the CI runner
|
|
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
|
is pure and runs in the container. Builds go to the host's nix daemon as an
|
|
untrusted client: a config can set no daemon option, and its derivations build
|
|
in the host's sandbox. A fixed-output derivation still has the network, as it
|
|
does for any build on that host.
|