job_queue: grow the rebuild subgraph from DeployApply (#2664)

The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.

Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.

The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.

`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.

The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.

Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
This commit is contained in:
atlas 2026-07-25 23:24:48 +02:00 committed by mara
commit 3429a8c5a6
10 changed files with 388 additions and 257 deletions

View file

@ -123,6 +123,50 @@ pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec<NodeSpe
]
}
/// The rebuild subgraph a [`NodeKind::DeployApply`] grows into its own DAG once
/// the merge has landed and `prepare_deploy` has staged the lock, plus the
/// [`NodeKind::FinalizeDeploy`] that closes the window behind it.
///
/// `relock = false` is the whole reason this composes: `prepare_deploy` already
/// relocked and staged `flake.lock`, so the appended `MetaSync` must do the dir
/// prep + `sync_agents` *without* re-locking over it.
///
/// `FinalizeDeploy` waits on **two** siblings, which together reproduce the gate
/// the old fused node had around its inline `rebuild_no_meta` call:
/// - `AfterOk` `Prebuild` — a parent's state is its roll-up, so this is `Done`
/// only once `StopForUpdate` → `Swap` → `PostSwap` all are (a failed *or*
/// cancelled child rolls the parent up `Failed`). That's the old
/// `build_result`.
/// - `AfterOk` `Reconcile` — the old call passed `deferred_start = false` on
/// purpose: the container had to come back up *before* the deploy was
/// finalized. `Reconcile` alone would not do, being `AfterAny` — it reaches
/// `Done` even after a failed `Swap`.
///
/// Appended, not submitted: the roots below become children of the emitting
/// `DeployApply` (see [`super::JobQueue::append_subgraph`]), which puts them
/// inside the `DeployWindow`'s subtree — so the `MetaWindow` this subgraph's
/// `MetaSync` and `FinalizeDeploy` declare is re-entered from the ancestor
/// already holding it rather than deadlocking against it.
pub(crate) fn deploy_rebuild_nodes(agent: &str) -> Vec<NodeSpec> {
let mut nodes = rebuild_nodes(agent, false, 0);
nodes.push(node(
NodeKind::FinalizeDeploy {
agent: agent.to_owned(),
},
vec![
Dep {
on: 1,
when: DepWhen::AfterOk,
},
Dep {
on: 5,
when: DepWhen::AfterOk,
},
],
));
nodes
}
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
/// noops when already down; the tail `Reconcile` auto-noops the start
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
@ -149,7 +193,9 @@ pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> Dag
/// nothing, so a failure here cancel-cascades its siblings with the forge and
/// the applied repo exactly as they were.
/// - `DeployApply` (2, child, `AfterOk` `MergeVerify`): the irreversible half —
/// ff-merge, `prepare_deploy`, rebuild, `finalize_deploy`.
/// ff-merge + `prepare_deploy`. It doesn't rebuild inline; it grows
/// [`deploy_rebuild_nodes`] into this DAG as its own children, so the build
/// and the closing `FinalizeDeploy` are real nodes under the same window.
/// - `DeployTail` (3, child, `AfterAny` `DeployApply`): the compensation +
/// bookkeeping tail — rollback when a merge landed unfinalized, forge tag
/// mirror, PR failure comment (see [`NodeKind::DeployTail`]).