vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

gate each deploy on a swarm build of the agent from internal/meta

Browse source
This commit is contained in:
atlas 2026-10-26 16:40:06 +01:00
commit 5e64048e53
1 changed files with 1570 additions and 178 deletions
Download patch file Download diff file Expand all files Collapse all files

78
docs/swarm/builder.md
View file

@ -6,6 +6,10 @@ push, the builder builds the PR's head on the agent base, and the result shows a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a red build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](services.md#builder).
## What happens on a config PR
@ -102,7 +106,8 @@ status has no link.
- The builder pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches.
- It doesn't gate anything. A merge deploys whatever the status says.
- The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
@ -148,11 +153,12 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run only after the
update succeeded: a hive builds only from `internal/meta`, so after a failed
update the scheduler skips the deploy node, and the graph shows the update as
failed and the deploy as skipped. Each ten-minute retry of a failed update
declares a fresh deploy behind it for every agent whose latest deploy the failed update
skipped, so the run that succeeds sends them (see
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
builds only from `internal/meta`, so after a failed update the scheduler skips
the deploy node, and the graph shows the update as failed and the deploy as
skipped. Each ten-minute retry of a failed update declares a fresh build and
deploy behind it for every agent whose latest deploy the failed update skipped,
so the run that succeeds sends them (see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.
@ -160,19 +166,72 @@ The controller sends a deploy to every agent a hive places whenever the pin in
`internal/meta` differs from the pin it last sent, so they rebuild on the new
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
which only the controller reads or writes, and a `RecordFanout` job writes the
key after every deploy of the fan-out succeeds. The check runs after each update
and once when the controller starts.
key once every deploy of the fan-out has finished or the deploy gate has skipped
it. The check runs after each update and once when the controller starts.
- The first commit sends a deploy, because the key holds no pin yet.
- A controller that starts with no key, as after the upgrade that added it,
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A red build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because the build gave no verdict.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
writes no key.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake a hive builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. Like the hive's build, it uses
that attribute, so a green build here means the flake evaluates and builds for
the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built.
The node ends one of three ways:
| Builder's answer | Node | Deploy | Timer |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------- | --------------------------------- |
| built | done, closure recorded | runs | |
| `nix` failed or timed out (a red build) | failed, with `nix`'s last lines | skipped | declares nothing |
| no builder listening, no reply in time, or the builder couldn't start `nix` (busy for 30 seconds, no forge token) | failed, `not built, will be retried` | skipped | declares a build and deploy again |
A red build is a verdict on the config: a config merge queues its own build and
deploy. A build that gave no verdict isn't, so the ten-minute timer declares
it again, without a meta update when the latest update succeeded, until the
builder answers. That covers a controller that updates before its builder
host does: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each retry adds a build and deploy node per affected agent while the
builder stays away.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A red build writes nothing and leaves the previous key.
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, since the repo is private.
## Credentials
| What | Where | Who reads it |
@ -183,7 +242,8 @@ and once when the controller starts.
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
repo, and on `internal/meta` through the meta update) and mints a
`read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers