e30f5849d5
The builder pushes its build metrics (builds by kind and outcome, build duration, queue depth and wait, reply publish failures) to its forwarder's OTLP receiver, which relays them to the swarm collector. It also registers the standard store client-certificate gauges (bao.client_cert.*, cert=builder) on the same exporter. They are the builder's liveness signal: exported every minute whether or not a build runs. The cert-metrics table in docs/scheduler/observability.md gains the builder row.
70 lines
3.4 KiB
Markdown
70 lines
3.4 KiB
Markdown
# swarm-builder
|
|
|
|
The swarm builder's build service, in the `swarm-builder` container on the
|
|
host that sets `deploy.builder.enable`. The design is
|
|
[`docs/swarm/builder.md`](../docs/swarm/builder.md).
|
|
|
|
swarm-controller sends one request per config PR head: an agent, a commit and
|
|
the PR. This service builds that agent config on the agent base this host was
|
|
deployed with, through the host's nix daemon, and answers with the system
|
|
closure's store path or the tail of what `nix` printed. The controller turns
|
|
the answer into a `swarm-builder` commit status on the PR's head.
|
|
|
|
## What a build is
|
|
|
|
- `nix build` of `nix/builder/check-flake`'s `check` output, with its three
|
|
inputs overridden: `hyperhive` and `nixpkgs` to this host's own sources,
|
|
and `agent` to `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<rev>`.
|
|
- The evaluation runs here, pure; the builds run in the host daemon's sandbox
|
|
as an untrusted client, so a config can neither set daemon options nor read
|
|
this service's environment.
|
|
- Nothing is pushed and nothing is recorded: the closure stays in the host's
|
|
store until it is collected. (A meta build can push, see below.)
|
|
|
|
## Meta builds
|
|
|
|
The service also builds one agent from `internal/meta` on request: an agent and
|
|
a commit of the repo. It runs `nix build` of
|
|
`nixosConfigurations.<agent>.config.system.build.toplevel` from that commit,
|
|
with every input at the flake's own lock, and answers with the store path and
|
|
the platform it builds for. The controller records the answer; the service
|
|
writes nothing. It takes the same one-build slot as the PR checks. See
|
|
[`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate).
|
|
|
|
Where `services.hyperhive.swarm.attic.{domain,cache}` are not null (they default
|
|
to the swarm's own cache), it first pushes a green build's closure to that cache with `attic push`, and the answer says
|
|
whether that worked. A failed push leaves the build green. See
|
|
[`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache).
|
|
|
|
## Metrics
|
|
|
|
The service exports build counts, durations, queue depth and wait, and reply
|
|
publish failures through its container's forwarder to the swarm collector. The
|
|
names and attributes are in
|
|
[`docs/scheduler/observability.md`](../docs/scheduler/observability.md#swarm-builder-metrics).
|
|
|
|
## Credentials
|
|
|
|
The container holds one credential: its own store certificate, which nspawn
|
|
hands in at start without a copy on disk. With it the service reads its
|
|
queue client secret at start and again when authelia refuses it, and its
|
|
read-only forge token at every build, and the cache push token at every push.
|
|
`git` gets the token through a credential helper reading the child's
|
|
environment, never in a URL or an argument, and the helper answers only an
|
|
`https` request for the forge's host. Its store role reads exactly
|
|
those three paths.
|
|
|
|
## Archetypes
|
|
|
|
The service also answers archetype evaluations: `nix eval` of the `meta` of
|
|
every `hyperhiveArchetypes` entry in this host's hyperhive source, with
|
|
nixpkgs pointed at this host's. It fetches nothing and uses no token, and it
|
|
doesn't wait for a running build. See
|
|
[`docs/swarm/archetypes.md`](../docs/swarm/archetypes.md).
|
|
|
|
## Meta-flake lock
|
|
|
|
The service also locks the swarm meta flake on request: `nix flake lock` of
|
|
the `flake.nix` it was sent, in a scratch directory, with the build's forge
|
|
credential. It answers with the lock and commits nothing. See
|
|
[`docs/swarm/builder.md`](../docs/swarm/builder.md#meta-flake-lock).
|