vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

retry every failed meta build each round, a timeout included, and drop the attempt count

Browse source
This commit is contained in:
atlas 2026-10-26 18:39:09 +01:00
commit 59456c3c58
1 changed files with 251 additions and 435 deletions
Download patch file Download diff file Expand all files Collapse all files

50
docs/swarm/builder.md
View file

@ -7,7 +7,7 @@ push, the builder builds the PR's head on the agent base, and the result shows a
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a red build holds the deploy back: see
its deploy, and a failed build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](services.md#builder).
@ -174,10 +174,10 @@ it. The check runs after each update and once when the controller starts.
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A red build holds only its own
- A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because the build gave no verdict.
a deploy skipped because its build failed.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
@ -205,46 +205,42 @@ the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built.
The node ends one of three ways, and the error text decides what the ten-minute
timer does:
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
| Builder's answer | Node | Deploy | Timer |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- | ------- | ----------------------------------- |
| built | done, closure recorded | runs | |
| `nix` exited with an error, first time | failed, `build failed, will be built once more: …` and `nix`'s last lines | skipped | builds again, as the second attempt |
| `nix` exited with an error, second time (red) | failed, with `nix`'s last lines | skipped | declares nothing |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, or the builder never ran `nix` (busy for 30 seconds, no forge token, `nix` couldn't start, `nix` hit its one-hour deadline) | failed, `not built, will be retried: …` | skipped | builds again, same attempt |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `not built, will be retried: …` | skipped | builds again, same attempt |
| What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs |
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
Only `nix` exiting with an error twice is a verdict on the config. The first
error may be a fetch hiccup inside `nix`, so the controller builds once more
without reading `nix`'s output. A red build holds the deploy until a config
merge queues its own build and deploy, and the pin fan-out still records the
pin, so one red agent doesn't resend the healthy ones. Anything else gave no
verdict, so the timer declares a build and deploy again every ten minutes,
without a meta update when the latest update succeeded, and the attempt count
stays where it was. That covers a controller that updates before its builder
host does: the old builder has no subscriber for the new subject, so the
On every ten-minute round the timer declares a build and deploy again for each
agent whose latest build failed, without a meta update when the latest update
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
still records the pin when the gate skips a deploy, so one failing agent doesn't
resend the healthy ones. A controller that updates before its builder host does
is the same case: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each retry adds a build and deploy node per affected agent while the
builder stays away. The build node carries the attempt count as `attempt`.
updates. Each round adds a build and deploy node per failing agent.
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
build failed, with the label `reason` set to `red` or `retrying`. The Grafana
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
a dead builder or a red config shows in Grafana before anyone notices an agent
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
not updating.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A red build writes nothing and leaves the previous key.
holds no grant on it. A failed build writes nothing and leaves the previous key.
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
grant the forge refuses counts as no verdict.
grant the forge refuses fails the build with no verdict.
## Credentials