refactor(#2756): replace the DAG terminal hook with real tail nodes

The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.

`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.

So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.

Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.

Load-bearing details:

- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
  subtree, and returns `bool`. Without this a cancelled approval DAG
  would dangle its approval forever — the hazard `tests.rs` already
  named. The spared tail's deps are `Cancelled`, which satisfies its weak
  edge, so the scheduler claims it and it resolves the row as cancelled.
  `hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
  to settle afterwards so "a weak-edge terminal node observing the
  cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
  deliberately a no-op when a tail was spared (a non-terminal child parks
  the container back in `Finishing`), so power ops still settle
  synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
  cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
  group root that rolled up `Failed` from a child carries no error of its
  own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
  own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
  agent, and that is also what lets one tail close a multi-agent DAG.

Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
This commit is contained in:
atlas 2026-07-27 14:25:03 +02:00 committed by mara
commit e8e6998ac5
11 changed files with 521 additions and 324 deletions

View file

@ -48,6 +48,39 @@ fn claim_one(q: &JobQueue) -> Claim {
claims.pop().expect("one claim")
}
/// Claim an approval DAG's `ResolveApproval` tail, assert which approval it
/// carries and what it will report to that row, then complete it. Replaces the
/// old `terminal_summary()` assertions: the outcome is no longer a struct handed
/// to a hook, it's what this node reads off its own deps.
fn settle_approval_tail(q: &JobQueue, dag_id: u64, approval_id: i64, expect: State) {
let tail = claim_one(q);
assert!(
matches!(tail.kind, NodeKind::ResolveApproval { approval_id: got } if got == approval_id),
"expected the ResolveApproval tail for #{approval_id}, got {:?}",
tail.kind
);
assert_eq!(
tail.deps_state(),
expect,
"outcome the tail reports to approval #{approval_id}"
);
q.complete_node(dag_id, tail.node_id, Ok(()));
}
/// The `EmitRebuilt` counterpart of [`settle_approval_tail`] — claim a rebuild /
/// perm-change DAG's tail, assert the `Rebuilt` event it will emit, complete it.
fn settle_rebuild_tail(q: &JobQueue, dag_id: u64, agent: &str, expect: State) {
let tail = claim_one(q);
assert_eq!(tail.kind.as_str(), "emit_rebuilt");
assert_eq!(tail.agent, agent, "`Rebuilt` is emitted per agent");
assert_eq!(
tail.deps_state(),
expect,
"ok-ness of the emitted `Rebuilt`"
);
q.complete_node(dag_id, tail.node_id, Ok(()));
}
fn state_of(q: &JobQueue, dag_id: u64) -> State {
// A fully-`Done` DAG drops out of the snapshot (its nodes are all
// excluded) — absence is the completion signal, so map it to `Done`.
@ -196,6 +229,7 @@ fn rebuild_chain_claims_in_dep_order() {
);
q.complete_node(id, c.node_id, Ok(()));
}
settle_rebuild_tail(&q, id, "agent-a", State::Done);
assert_eq!(state_of(&q, id), State::Done);
}
@ -315,8 +349,8 @@ fn lease_serializes_two_lifecycle_dags_for_same_agent() {
assert_eq!(second.kind.as_str(), "reconcile");
q.complete_node(restart, second.node_id, Ok(()));
// Restart's work is terminal → its lease releases, so stop's now-unblocked
// Reconcile becomes ready (restart's inline hook fired off the returned
// summary — no terminal-hook node).
// Reconcile becomes ready (a power op has no tail node, so nothing of
// restart's remains claimable).
let third = claim_one(&q);
assert_eq!(third.dag_id, stop);
assert_eq!(third.kind.as_str(), "reconcile");
@ -369,8 +403,8 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
.clone();
q.complete_node(stop, reconcile.node_id, Ok(()));
// stop's Reconcile done → its lease frees, so rebuild's StopForUpdate
// unblocks. (stop's DAG rolls up terminal; its inline hook fires off the
// returned summary — no terminal-hook node in the claim set.)
// unblocks. (stop's DAG rolls up terminal; a power op has no tail node, so
// nothing of stop's is left in the claim set.)
let after = q.claim_ready();
let sfu = after
.iter()
@ -591,7 +625,6 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
// the emitter and its LOCAL 0-based deps are rebased onto the DAG.
let q = JobQueue::new(4);
let spec = DagSpec {
hook: None,
source: Source::AutoUpdate,
reason: "sweep".to_owned(),
approval_id: None,
@ -703,6 +736,73 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
);
}
// ---- dep outcomes on the claim ----
/// Every claimed node reports how each node it depends on finished, and those
/// deps are always already terminal — that is what a node's edges being
/// satisfied *means*. A tail node reads its `deps` instead of going back to the
/// world to find out how the work below it went.
#[test]
fn claim_carries_terminal_dep_outcomes() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
let mut saw_a_dep = false;
loop {
let mut claims = q.claim_ready();
let Some(claim) = claims.pop() else { break };
assert!(claims.is_empty(), "one build slot ⇒ one claim at a time");
for dep in &claim.deps {
saw_a_dep = true;
assert!(
dep.state.is_terminal(),
"{} was claimed with a non-terminal dep ({:?}) — a node's edges \
being satisfied is exactly the claim that its deps have finished",
claim.kind.as_str(),
dep.state
);
assert_eq!(
dep.state,
State::Done,
"on the happy path every dep of {} finished Done",
claim.kind.as_str()
);
assert_eq!(dep.error, None, "a Done dep carries no error");
}
q.complete_node(id, claim.node_id, Ok(()));
}
assert!(saw_a_dep, "the rebuild DAG has at least one dependent node");
assert_eq!(state_of(&q, id), State::Done);
}
/// The failure direction, which is the whole point of carrying outcomes at all:
/// `Reconcile` hangs off `Prebuild` with `AfterAny`, so a failed prebuild
/// cancel-cascades `StopForUpdate`/`Swap`/`PostSwap` and `Reconcile` still runs
/// — and its claim hands it the failure, including the reason, rather than
/// leaving the executor to go and re-derive it from the world.
#[test]
fn claim_dep_outcome_reports_a_failed_dep_with_its_error() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
let meta_sync = claim_one(&q);
q.complete_node(id, meta_sync.node_id, Ok(()));
let prebuild = claim_one(&q);
assert_eq!(prebuild.kind.as_str(), "prebuild");
q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned()));
let reconcile = claim_one(&q);
assert_eq!(reconcile.kind.as_str(), "reconcile");
assert_eq!(
reconcile
.deps
.iter()
.map(|d| (d.state, d.error.as_deref()))
.collect::<Vec<_>>(),
vec![(State::Failed, Some("nix build exploded"))],
"reconcile's claim carries the failed prebuild and its reason"
);
assert_eq!(reconcile.deps_state(), State::Failed);
assert_eq!(reconcile.deps_error(), Some("nix build exploded"));
}
// ---- failure: cancel-downstream + AfterAny ----
#[test]
@ -811,6 +911,7 @@ fn swap_ok_runs_post_swap_before_reconcile() {
let reconcile = claim_one(&q);
assert_eq!(reconcile.kind.as_str(), "reconcile");
q.complete_node(id, reconcile.node_id, Ok(()));
settle_rebuild_tail(&q, id, "agent-a", State::Done);
assert_eq!(state_of(&q, id), State::Done);
}
@ -832,10 +933,17 @@ fn failed_reconcile_marks_dag_failed() {
fn cancel_clears_queued_dag() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
// Cancel returns the terminal summary (state `Cancelled`) — the inline hook
// fires off it at the caller; there's no terminal-hook node to claim.
let terminal = q.cancel(id).expect("cancelled");
assert_eq!(terminal.state, State::Cancelled);
assert!(q.cancel(id), "fully-queued dag cancels");
// Every work node is `Cancelled`, but the tail is spared so it can still
// report the cancellation — so it is the one thing left to claim.
let tail = claim_one(&q);
assert_eq!(tail.kind.as_str(), "emit_rebuilt");
assert_eq!(
tail.deps_state(),
State::Cancelled,
"the spared tail sees its deps cancelled, so it emits no `Rebuilt`"
);
q.complete_node(id, tail.node_id, Ok(()));
assert_eq!(state_of(&q, id), State::Cancelled);
assert!(q.claim_ready().is_empty());
}
@ -845,23 +953,24 @@ fn cancel_refuses_running_dag() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
let _ = claim_one(&q);
assert!(q.cancel(id).is_none());
assert!(!q.cancel(id));
assert_eq!(state_of(&q, id), State::Running);
}
/// A cancelled power op must fire **no** compensating hook — not even one that
/// A cancelled power op must run **no** compensating node — not even one that
/// carries a `SetWanted` head.
///
/// `cancel` refuses unless every work node is still `Pending`
/// (`cancel_refuses_running_dag`) and a cancel *cascade* rolls up `Failed`
/// rather than `Cancelled`, so a `Cancelled` DAG provably never executed a
/// node: its `SetWanted` never ran and the agent's intent still reads whatever
/// Now structural rather than a property of a hook enum: a power op emits no
/// tail node at all, so once its work nodes cancel there is simply nothing left
/// to claim. `cancel` also refuses unless every work node is still `Pending`
/// (`cancel_refuses_running_dag`), so a `Cancelled` DAG provably never executed
/// a node: its `SetWanted` never ran and the agent's intent still reads whatever
/// the operator last set. A "revert" instead writes the agent's *observed*
/// state, which for a down-but-`wanted = Up` agent (crashed, or caught
/// mid-bounce) flips the intent to `Offline` and leaves it
/// deliberately-stopped as far as reconcile and crash-watch are concerned.
#[test]
fn cancelled_power_op_fires_no_hook() {
fn cancelled_power_op_runs_no_compensating_node() {
for graceful in [false, true] {
for running in [false, true] {
let targets = vec![("agent-a".to_owned(), running)];
@ -896,12 +1005,12 @@ fn cancelled_power_op_fires_no_hook() {
);
let q = JobQueue::new(1);
let id = submit(&q, spec);
let summary = q.cancel(id).expect("cancelled while queued");
assert_eq!(summary.state, State::Cancelled);
assert_eq!(
summary.hook, None,
assert!(q.cancel(id), "cancelled while queued");
assert_eq!(state_of(&q, id), State::Cancelled);
assert!(
q.claim_ready().is_empty(),
"cancelled {name} (graceful={graceful}, running={running}) must \
fire no hook no node of it ever ran"
leave nothing to run a power op emits no tail node"
);
}
}
@ -920,13 +1029,10 @@ fn dag_settles_terminal_and_releases_lease_after_work() {
q.complete_node(id, stop.node_id, Ok(()));
let rec = claim_one(&q);
assert_eq!(rec.kind.as_str(), "reconcile");
// Completing the last work node rolls the container up terminal and returns
// the summary the inline hook consumes — there is no terminal-hook node.
let summary = q
.complete_node(id, rec.node_id, Ok(()))
.expect("terminal summary");
assert_eq!(summary.state, State::Done);
assert!(q.claim_ready().is_empty(), "no terminal-hook node to claim");
// Completing the last work node rolls the container up terminal. A power op
// has no tail node, so nothing is left to claim.
q.complete_node(id, rec.node_id, Ok(()));
assert!(q.claim_ready().is_empty(), "no tail node to claim");
assert_eq!(state_of(&q, id), State::Done);
// Lease released when the work chain settled: a new DAG for the agent claims
// immediately.
@ -938,31 +1044,44 @@ fn dag_settles_terminal_and_releases_lease_after_work() {
assert_eq!(c.dag_id, next);
}
/// A DAG cancelled while fully queued must still surface a terminal
/// roll-up for the scheduler's hooks — otherwise a queued approval
/// DAG cancelled by the operator would dangle its approval forever.
/// A DAG cancelled while fully queued must still **run its tail**, or a queued
/// approval DAG cancelled by the operator would dangle its approval forever.
///
/// This is the load-bearing case for sparing tails in [`JobQueue::cancel`]: the
/// work nodes all cancel, but `ResolveApproval` is weak-edged, so a `Cancelled`
/// dep satisfies its edge and it becomes claimable instead of being cancelled
/// along with everything else. It reads `Cancelled` off its own deps and resolves
/// the approval as "cancelled before completion".
#[test]
fn cancelled_dag_finalizes_with_terminal_rollup() {
fn cancelled_dag_still_runs_its_approval_tail() {
let q = JobQueue::new(1);
let id = submit(
&q,
templates::approval_deploy("agent-a", 7, "approval #7".to_owned()),
);
// Cancel rolls the DAG up terminal and returns its summary — the inline hook
// (approval resolution) runs off it at the caller. Cancelled + approval id 7.
let summary = q.cancel(id).expect("cancelled");
assert_eq!(summary.state, State::Cancelled);
assert_eq!(summary.approval_id, Some(7));
// The cancelled DAG's summary stays available (until history-trimmed) and
// unrelated later activity doesn't disturb it.
assert!(q.cancel(id), "fully-queued dag cancels");
let tail = claim_one(&q);
assert_eq!(tail.dag_id, id);
assert_eq!(tail.kind.as_str(), "resolve_approval");
assert!(
matches!(tail.kind, NodeKind::ResolveApproval { approval_id: 7 }),
"the tail carries the approval to resolve, got {:?}",
tail.kind
);
assert_eq!(
tail.deps_state(),
State::Cancelled,
"so the executor resolves the approval as cancelled, not as a failure"
);
assert_eq!(tail.deps_error(), None, "a cancelled dep carries no error");
q.complete_node(id, tail.node_id, Ok(()));
assert_eq!(state_of(&q, id), State::Cancelled);
// Unrelated later activity doesn't disturb the settled DAG.
let other = submit(&q, rebuild("agent-b", "r"));
let c = claim_one(&q);
assert_eq!(c.dag_id, other);
q.complete_node(other, c.node_id, Err("boom".to_owned()));
assert_eq!(
q.terminal_summary(id).map(|t| t.state),
Some(State::Cancelled)
);
assert_eq!(state_of(&q, id), State::Cancelled);
}
// ---- approval deploy subtree ----
@ -1006,13 +1125,12 @@ fn deploy_dag_runs_phases_in_order_and_tails_a_failed_apply() {
);
q.complete_node(id, tail.node_id, Ok(()));
let summary = q.terminal_summary(id).expect("dag terminal");
settle_approval_tail(&q, id, 7, State::Failed);
assert_eq!(
summary.state,
state_of(&q, id),
State::Failed,
"an Ok tail must not launder a failed deploy into a success"
);
assert_eq!(summary.approval_id, Some(7));
}
/// The deploy's happy path: `DeployApply` does not build. It grows the ordinary
@ -1080,9 +1198,8 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
assert!(matches!(tail.kind, NodeKind::DeployTail { .. }));
q.complete_node(id, tail.node_id, Ok(()));
let summary = q.terminal_summary(id).expect("dag terminal");
assert_eq!(summary.state, State::Done);
assert_eq!(summary.approval_id, Some(11));
settle_approval_tail(&q, id, 11, State::Done);
assert_eq!(state_of(&q, id), State::Done);
}
/// A failure *inside* the grafted rebuild is the failure mode the subgraph
@ -1132,12 +1249,14 @@ fn deploy_dag_skips_finalize_but_still_tails_a_failed_graft() {
);
q.complete_node(id, tail.node_id, Ok(()));
let summary = q.terminal_summary(id).expect("dag terminal");
assert_eq!(summary.state, State::Failed);
settle_approval_tail(&q, id, 13, State::Failed);
assert_eq!(state_of(&q, id), State::Failed);
assert_eq!(
q.first_error(id).as_deref(),
Some("profile swap failed"),
"the tail annotates failed/<id> with this"
"the tail annotates failed/<id> with this — and it is also what
`exec::failure_reason` falls back to, since the tail's own dep is a
group root that rolled up Failed and so carries no error itself"
);
}
@ -1168,9 +1287,8 @@ fn deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails() {
);
q.complete_node(id, tail.node_id, Ok(()));
let summary = q.terminal_summary(id).expect("dag terminal");
assert_eq!(summary.state, State::Failed);
assert_eq!(summary.approval_id, Some(9));
settle_approval_tail(&q, id, 9, State::Failed);
assert_eq!(state_of(&q, id), State::Failed);
}
// ---- build logs, history ----
@ -1219,8 +1337,7 @@ fn history_evicts_oldest_terminals_past_flat_cap() {
// Fail the single work node so the DAG *lingers*: a fully-`Done` DAG
// drops off the wire entirely, but a `Failed` one is retained (+
// history-capped) so the operator can still triage it. Completing the
// node rolls the container up terminal (its inline hook fires off the
// returned summary — no terminal-hook node).
// node rolls the container up terminal.
q.complete_node(id, c.node_id, Err("boom".to_owned()));
ids.push(id);
}
@ -1310,8 +1427,8 @@ fn spawn_shape_provision_create_dropin_reconcile() {
assert_eq!(c.approval_id, Some(7));
q.complete_node(id, c.node_id, Ok(()));
}
let report_terminal = state_of(&q, id);
assert_eq!(report_terminal, State::Done);
settle_approval_tail(&q, id, 7, State::Done);
assert_eq!(state_of(&q, id), State::Done);
}
#[test]
@ -1342,6 +1459,7 @@ fn perm_change_shape_prefixes_rebuild_chain() {
assert_eq!(c.kind.as_str(), expected);
q.complete_node(id, c.node_id, Ok(()));
}
settle_rebuild_tail(&q, id, "agent-a", State::Done);
assert_eq!(state_of(&q, id), State::Done);
}