queue the build of a config head that moved without a webhook build
This commit is contained in:
parent
a0c8892b99
commit
b7d8c0dfc0
2 changed files with 689 additions and 28 deletions
|
|
@ -141,7 +141,7 @@ agent. The controller queues
|
|||
its update of the repo before the deploy request for a merged config commit
|
||||
and for a new agent, and sends the deploy only when that update succeeded and
|
||||
the swarm builder built the agent from the updated repo (the
|
||||
[deploy gate](../swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again.
|
||||
[deploy gate](../swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
|
||||
|
||||
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
|
||||
agent's record in `agent-closures` declares, whichever `internal/meta` commit
|
||||
|
|
|
|||
|
|
@ -121,7 +121,7 @@ status has no link.
|
|||
can push to [a binary cache](#binary-cache).
|
||||
- The PR check gates nothing. A merge deploys whatever the status says. The
|
||||
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
|
||||
- A lost webhook delivery isn't retried: the head isn't built until the next
|
||||
- A lost webhook delivery of a PR push isn't retried: the head isn't built until the next
|
||||
push. The jobs graph is in memory, so a controller restart drops a build in
|
||||
flight and leaves its status `pending`.
|
||||
|
||||
|
|
@ -236,6 +236,19 @@ is the same case: the old builder has no subscriber for the new subject, so the
|
|||
controller gets no responders at once, and the deploys go out after the builder
|
||||
updates. Each round adds a build and deploy node per failing agent.
|
||||
|
||||
A merged config PR whose webhook delivery never arrives would leave the agent
|
||||
on its old config. Every five minutes, after the config-PR scan, the controller
|
||||
reads `main` of each `agent-configs/<agent>` repo that has a commit and that
|
||||
exactly one hive places. It compares that head with the config commit pinned for
|
||||
the agent in the `flake.nix` of the `internal/meta` commit its `agent-closures`
|
||||
key names, since the pin in `internal/meta`'s own head also moves for other
|
||||
agents' merges. When they differ, it declares the same update, build and deploy
|
||||
a merge declares, and logs at info `config drift: …`. It leaves alone an agent
|
||||
with no key, a key whose flake pins no config commit for the agent, and a head
|
||||
the graph already holds a deploy for, in any state: the ten-minute rounds above
|
||||
build a head whose build failed again, not this pass. The graph is
|
||||
in memory, so a controller restart tries each such head once more.
|
||||
|
||||
When the build belongs to a merged config PR and `nix` exited with an error or hit
|
||||
its deadline, the controller also comments once on that PR, with its forge
|
||||
account: the agent, the `internal/meta` commit of the first red build, `nix`'s last lines (at most 4000
|
||||
|
|
|
|||
Loading…
Reference in a new issue