swarm-builder: refuse a build that waited too long for the slot
A request that waits behind another build, e.g. one a controller restart orphaned, could start a full build after most of the controller's 61-minute reply deadline was gone. The builder now waits at most REPLY_DEADLINE - BUILD_DEADLINE - 30 s for the slot and answers Failed, without building, past that. The 30 s covers the forge token read, now bounded to 20 s, and the reply's trip back. Fixes #5437
This commit is contained in:
parent
6630a51022
commit
bf848ba65f
4 changed files with 180 additions and 22 deletions
|
|
@ -20,7 +20,11 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|||
finishes and writes its status on its own, older commit.
|
||||
2. The node sets the status to `pending`, then sends the build as a request on
|
||||
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
||||
reply.
|
||||
reply. The builder runs one build at a time and refuses a request that has
|
||||
waited 30 seconds for its turn, such as one behind a build a controller
|
||||
restart orphaned: a build started later could still run when the 61
|
||||
minutes end. Reading the forge token before a build has 20 seconds, and
|
||||
the rest of the minute covers the reply's trip back.
|
||||
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
||||
with every input overridden:
|
||||
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
||||
|
|
@ -39,6 +43,7 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|||
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
||||
| built | `success` | the store path | done |
|
||||
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
||||
| the builder busy for 30 seconds | `failure` | why | failed |
|
||||
| no builder listening, or no reply in time | `error` | why | failed |
|
||||
|
||||
The full tail is in the failed node's error on the Jobs page and in the
|
||||
|
|
|
|||
Loading…
Reference in a new issue