vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

build a failed meta build once more, and treat a timeout or missing grant as no verdict

Browse source
This commit is contained in:
atlas 2026-10-26 17:53:39 +01:00
commit 99d2ed5b9e
1 changed files with 621 additions and 111 deletions
Download patch file Download diff file Expand all files Collapse all files

40
docs/swarm/builder.md
View file

@ -194,7 +194,7 @@ runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `.builder.build-meta` and waits up to 61
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
@ -205,22 +205,35 @@ the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built.
The node ends one of three ways:
The node ends one of three ways, and the error text decides what the ten-minute
timer does:
| Builder's answer | Node | Deploy | Timer |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------- | --------------------------------- |
| built | done, closure recorded | runs | |
| `nix` failed or timed out (a red build) | failed, with `nix`'s last lines | skipped | declares nothing |
| no builder listening, no reply in time, or the builder couldn't start `nix` (busy for 30 seconds, no forge token) | failed, `not built, will be retried` | skipped | declares a build and deploy again |
| Builder's answer | Node | Deploy | Timer |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- | ------- | ----------------------------------- |
| built | done, closure recorded | runs | |
| `nix` exited with an error, first time | failed, `build failed, will be built once more: …` and `nix`'s last lines | skipped | builds again, as the second attempt |
| `nix` exited with an error, second time (red) | failed, with `nix`'s last lines | skipped | declares nothing |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, or the builder never ran `nix` (busy for 30 seconds, no forge token, `nix` couldn't start, `nix` hit its one-hour deadline) | failed, `not built, will be retried: …` | skipped | builds again, same attempt |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `not built, will be retried: …` | skipped | builds again, same attempt |
A red build is a verdict on the config: a config merge queues its own build and
deploy. A build that gave no verdict isn't, so the ten-minute timer declares
it again, without a meta update when the latest update succeeded, until the
builder answers. That covers a controller that updates before its builder
Only `nix` exiting with an error twice is a verdict on the config. The first
error may be a fetch hiccup inside `nix`, so the controller builds once more
without reading `nix`'s output. A red build holds the deploy until a config
merge queues its own build and deploy, and the pin fan-out still records the
pin, so one red agent doesn't resend the healthy ones. Anything else gave no
verdict, so the timer declares a build and deploy again every ten minutes,
without a meta update when the latest update succeeded, and the attempt count
stays where it was. That covers a controller that updates before its builder
host does: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each retry adds a build and deploy node per affected agent while the
builder stays away.
builder stays away. The build node carries the attempt count as `attempt`.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `red` or `retrying`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a red config shows in Grafana before anyone notices an agent
not updating.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
@ -230,7 +243,8 @@ Nothing reads the bucket yet. The closure stays unrooted in the builder host's
store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, since the repo is private.
updates the flake, and again before each build, since the repo is private. A
grant the forge refuses counts as no verdict.
## Credentials