feat(#2772): branch on outcome in the graph, not inside the node
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
atlas
mara
parent
affedecaa5
commit
07078b76ef
8 changed files with 365 additions and 385 deletions
|
|
@ -48,35 +48,34 @@ fn claim_one(q: &JobQueue) -> Claim {
|
|||
claims.pop().expect("one claim")
|
||||
}
|
||||
|
||||
/// Claim an approval DAG's `ResolveApproval` tail, assert which approval it
|
||||
/// carries and what it will report to that row, then complete it. Replaces the
|
||||
/// old `terminal_summary()` assertions: the outcome is no longer a struct handed
|
||||
/// to a hook, it's what this node reads off its own deps.
|
||||
fn settle_approval_tail(q: &JobQueue, dag_id: u64, approval_id: i64, expect: State) {
|
||||
/// Claim an approval DAG's `ResolveApproval` tail and complete it, asserting it
|
||||
/// is the one built for `expect`.
|
||||
///
|
||||
/// A template emits one tail per outcome and the graph runs exactly one, so the
|
||||
/// assertion is on *which node was claimed* — that alone says what the approval
|
||||
/// row is about to be resolved as. Nothing computes it.
|
||||
fn settle_approval_tail(q: &JobQueue, dag_id: u64, approval_id: i64, expect: TerminalState) {
|
||||
let tail = claim_one(q);
|
||||
assert!(
|
||||
matches!(tail.kind, NodeKind::ResolveApproval { approval_id: got } if got == approval_id),
|
||||
"expected the ResolveApproval tail for #{approval_id}, got {:?}",
|
||||
matches!(
|
||||
tail.kind,
|
||||
NodeKind::ResolveApproval { approval_id: got, outcome }
|
||||
if got == approval_id && outcome == expect
|
||||
),
|
||||
"expected the {expect:?} ResolveApproval tail for #{approval_id}, got {:?}",
|
||||
tail.kind
|
||||
);
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
expect,
|
||||
"outcome the tail reports to approval #{approval_id}"
|
||||
);
|
||||
q.complete_node(dag_id, tail.node_id, Ok(()));
|
||||
}
|
||||
|
||||
/// The `EmitRebuilt` counterpart of [`settle_approval_tail`] — claim a rebuild /
|
||||
/// perm-change DAG's tail, assert the `Rebuilt` event it will emit, complete it.
|
||||
fn settle_rebuild_tail(q: &JobQueue, dag_id: u64, agent: &str, expect: State) {
|
||||
/// The `EmitRebuilt` counterpart of [`settle_approval_tail`] — claim the tail the
|
||||
/// graph let run and assert it's the `ok` one expected.
|
||||
fn settle_rebuild_tail(q: &JobQueue, dag_id: u64, agent: &str, expect_ok: bool) {
|
||||
let tail = claim_one(q);
|
||||
assert_eq!(tail.kind.as_str(), "emit_rebuilt");
|
||||
assert_eq!(tail.agent, agent, "`Rebuilt` is emitted per agent");
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
expect,
|
||||
"ok-ness of the emitted `Rebuilt`"
|
||||
assert!(
|
||||
matches!(&tail.kind, NodeKind::EmitRebuilt { agent: a, ok } if a == agent && *ok == expect_ok),
|
||||
"expected the ok={expect_ok} EmitRebuilt tail for {agent}, got {:?}",
|
||||
tail.kind
|
||||
);
|
||||
q.complete_node(dag_id, tail.node_id, Ok(()));
|
||||
}
|
||||
|
|
@ -229,7 +228,7 @@ fn rebuild_chain_claims_in_dep_order() {
|
|||
);
|
||||
q.complete_node(id, c.node_id, Ok(()));
|
||||
}
|
||||
settle_rebuild_tail(&q, id, "agent-a", State::Done);
|
||||
settle_rebuild_tail(&q, id, "agent-a", true);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
@ -736,73 +735,6 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
|
|||
);
|
||||
}
|
||||
|
||||
// ---- dep outcomes on the claim ----
|
||||
|
||||
/// Every claimed node reports how each node it depends on finished, and those
|
||||
/// deps are always already terminal — that is what a node's edges being
|
||||
/// satisfied *means*. A tail node reads its `deps` instead of going back to the
|
||||
/// world to find out how the work below it went.
|
||||
#[test]
|
||||
fn claim_carries_terminal_dep_outcomes() {
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, rebuild("agent-a", "r"));
|
||||
let mut saw_a_dep = false;
|
||||
loop {
|
||||
let mut claims = q.claim_ready();
|
||||
let Some(claim) = claims.pop() else { break };
|
||||
assert!(claims.is_empty(), "one build slot ⇒ one claim at a time");
|
||||
for dep in &claim.deps {
|
||||
saw_a_dep = true;
|
||||
assert!(
|
||||
dep.state.is_terminal(),
|
||||
"{} was claimed with a non-terminal dep ({:?}) — a node's edges \
|
||||
being satisfied is exactly the claim that its deps have finished",
|
||||
claim.kind.as_str(),
|
||||
dep.state
|
||||
);
|
||||
assert_eq!(
|
||||
dep.state,
|
||||
State::Done,
|
||||
"on the happy path every dep of {} finished Done",
|
||||
claim.kind.as_str()
|
||||
);
|
||||
assert_eq!(dep.error, None, "a Done dep carries no error");
|
||||
}
|
||||
q.complete_node(id, claim.node_id, Ok(()));
|
||||
}
|
||||
assert!(saw_a_dep, "the rebuild DAG has at least one dependent node");
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
/// The failure direction, which is the whole point of carrying outcomes at all:
|
||||
/// `Reconcile` hangs off `Prebuild` with `AfterAny`, so a failed prebuild
|
||||
/// cancel-cascades `StopForUpdate`/`Swap`/`PostSwap` and `Reconcile` still runs
|
||||
/// — and its claim hands it the failure, including the reason, rather than
|
||||
/// leaving the executor to go and re-derive it from the world.
|
||||
#[test]
|
||||
fn claim_dep_outcome_reports_a_failed_dep_with_its_error() {
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, rebuild("agent-a", "r"));
|
||||
let meta_sync = claim_one(&q);
|
||||
q.complete_node(id, meta_sync.node_id, Ok(()));
|
||||
let prebuild = claim_one(&q);
|
||||
assert_eq!(prebuild.kind.as_str(), "prebuild");
|
||||
q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned()));
|
||||
let reconcile = claim_one(&q);
|
||||
assert_eq!(reconcile.kind.as_str(), "reconcile");
|
||||
assert_eq!(
|
||||
reconcile
|
||||
.deps
|
||||
.iter()
|
||||
.map(|d| (d.state, d.error.as_deref()))
|
||||
.collect::<Vec<_>>(),
|
||||
vec![(State::Failed, Some("nix build exploded"))],
|
||||
"reconcile's claim carries the failed prebuild and its reason"
|
||||
);
|
||||
assert_eq!(reconcile.deps_state(), State::Failed);
|
||||
assert_eq!(reconcile.deps_error(), Some("nix build exploded"));
|
||||
}
|
||||
|
||||
// ---- failure: cancel-downstream + AfterAny ----
|
||||
|
||||
#[test]
|
||||
|
|
@ -831,9 +763,19 @@ fn failed_node_cancels_downstream_but_afterany_reconcile_runs() {
|
|||
.state
|
||||
};
|
||||
assert_eq!(by_kind("prebuild"), State::Failed);
|
||||
assert_eq!(by_kind("stop_for_update"), State::Cancelled);
|
||||
assert_eq!(by_kind("swap"), State::Cancelled);
|
||||
assert_eq!(by_kind("post_swap"), State::Cancelled);
|
||||
// `StopForUpdate` / `Swap` / `PostSwap` were *ruled out* by the failed
|
||||
// `Prebuild` — `Skipped`, and skipped nodes are filtered off the wire along
|
||||
// with `Done` ones. The failure itself is still visible (the `prebuild` row
|
||||
// above, and the roll-up), which is the part an operator acts on.
|
||||
// Restoring that detail wants a real `Skipped` wire state the client renders
|
||||
// as "not run" — surfacing them as `Cancelled` instead would make a
|
||||
// *successful* DAG with a not-taken branch read as cancelled.
|
||||
for ruled_out in ["stop_for_update", "swap", "post_swap"] {
|
||||
assert!(
|
||||
dag.nodes.iter().all(|n| n.kind != ruled_out),
|
||||
"{ruled_out} was ruled out, so it is off the wire"
|
||||
);
|
||||
}
|
||||
// The AfterAny reconcile ran (claimed + completed Ok above) → it's `Done`,
|
||||
// and `Done` nodes are excluded from the wire, so it's absent here.
|
||||
assert!(
|
||||
|
|
@ -872,14 +814,18 @@ fn swap_failure_still_runs_reconcile() {
|
|||
q.complete_node(id, reconcile.node_id, Ok(()));
|
||||
let all_dags = q.snapshot();
|
||||
let dag = all_dags.iter().find(|d| d.id == id).expect("dag");
|
||||
assert!(
|
||||
dag.nodes.iter().all(|n| n.kind != "post_swap"),
|
||||
"PostSwap is ruled out by the failed Swap (`Skipped`, so off the wire)"
|
||||
);
|
||||
assert_eq!(
|
||||
dag.nodes
|
||||
.iter()
|
||||
.find(|n| n.kind == "post_swap")
|
||||
.expect("post_swap node")
|
||||
.find(|n| n.kind == "swap")
|
||||
.expect("swap node")
|
||||
.state,
|
||||
State::Cancelled,
|
||||
"PostSwap must cancel-cascade when Swap fails"
|
||||
State::Failed,
|
||||
"and the failure that ruled it out is still on the wire"
|
||||
);
|
||||
assert_eq!(state_of(&q, id), State::Failed);
|
||||
}
|
||||
|
|
@ -911,7 +857,7 @@ fn swap_ok_runs_post_swap_before_reconcile() {
|
|||
let reconcile = claim_one(&q);
|
||||
assert_eq!(reconcile.kind.as_str(), "reconcile");
|
||||
q.complete_node(id, reconcile.node_id, Ok(()));
|
||||
settle_rebuild_tail(&q, id, "agent-a", State::Done);
|
||||
settle_rebuild_tail(&q, id, "agent-a", true);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
@ -939,18 +885,14 @@ fn cancel_clears_queued_dag() {
|
|||
// operator who just cancelled it (the dashboard renders this roll-up from
|
||||
// the snapshot `post_rebuild_queue_cancel` emits synchronously).
|
||||
assert_eq!(state_of(&q, id), State::Cancelled, "no stale Queued gap");
|
||||
// Every work node is `Cancelled`, but the tail is spared so it can still
|
||||
// report the cancellation — so it is the one thing left to claim.
|
||||
let tail = claim_one(&q);
|
||||
assert_eq!(tail.kind.as_str(), "emit_rebuilt");
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
State::Cancelled,
|
||||
"the spared tail sees its deps cancelled, so it emits no `Rebuilt`"
|
||||
// Neither `EmitRebuilt` tail accepts a *dropped* dependency — the ok one is
|
||||
// `AFTER_OK`, the failure one keys on elimination — so both are cancelled
|
||||
// with the work and **nothing is emitted** for a rebuild that never ran.
|
||||
assert!(
|
||||
q.claim_ready().is_empty(),
|
||||
"a dropped rebuild reports nothing"
|
||||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
assert_eq!(state_of(&q, id), State::Cancelled);
|
||||
assert!(q.claim_ready().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -1065,21 +1007,10 @@ fn cancelled_dag_still_runs_its_approval_tail() {
|
|||
templates::approval_deploy("agent-a", 7, "approval #7".to_owned()),
|
||||
);
|
||||
assert!(q.cancel(id), "fully-queued dag cancels");
|
||||
let tail = claim_one(&q);
|
||||
assert_eq!(tail.dag_id, id);
|
||||
assert_eq!(tail.kind.as_str(), "resolve_approval");
|
||||
assert!(
|
||||
matches!(tail.kind, NodeKind::ResolveApproval { approval_id: 7 }),
|
||||
"the tail carries the approval to resolve, got {:?}",
|
||||
tail.kind
|
||||
);
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
State::Cancelled,
|
||||
"so the executor resolves the approval as cancelled, not as a failure"
|
||||
);
|
||||
assert_eq!(tail.deps_error(), None, "a cancelled dep carries no error");
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
// The `Cancelled` tail is the only node whose edge accepts a dropped
|
||||
// dependency, so it is the only one `cancel` spares — and claiming it *is*
|
||||
// the assertion that the approval gets resolved as cancelled.
|
||||
settle_approval_tail(&q, id, 7, TerminalState::Cancelled);
|
||||
assert_eq!(state_of(&q, id), State::Cancelled);
|
||||
// Unrelated later activity doesn't disturb the settled DAG.
|
||||
let other = submit(&q, rebuild("agent-b", "r"));
|
||||
|
|
@ -1130,7 +1061,7 @@ fn deploy_dag_runs_phases_in_order_and_tails_a_failed_apply() {
|
|||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
settle_approval_tail(&q, id, 7, State::Failed);
|
||||
settle_approval_tail(&q, id, 7, TerminalState::Failed);
|
||||
assert_eq!(
|
||||
state_of(&q, id),
|
||||
State::Failed,
|
||||
|
|
@ -1203,7 +1134,7 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
|
|||
assert!(matches!(tail.kind, NodeKind::DeployTail { .. }));
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
settle_approval_tail(&q, id, 11, State::Done);
|
||||
settle_approval_tail(&q, id, 11, TerminalState::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
@ -1254,7 +1185,7 @@ fn deploy_dag_skips_finalize_but_still_tails_a_failed_graft() {
|
|||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
settle_approval_tail(&q, id, 13, State::Failed);
|
||||
settle_approval_tail(&q, id, 13, TerminalState::Failed);
|
||||
assert_eq!(state_of(&q, id), State::Failed);
|
||||
assert_eq!(
|
||||
q.first_error(id).as_deref(),
|
||||
|
|
@ -1292,7 +1223,7 @@ fn deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails() {
|
|||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
settle_approval_tail(&q, id, 9, State::Failed);
|
||||
settle_approval_tail(&q, id, 9, TerminalState::Failed);
|
||||
assert_eq!(state_of(&q, id), State::Failed);
|
||||
}
|
||||
|
||||
|
|
@ -1432,7 +1363,7 @@ fn spawn_shape_provision_create_dropin_reconcile() {
|
|||
assert_eq!(c.approval_id, Some(7));
|
||||
q.complete_node(id, c.node_id, Ok(()));
|
||||
}
|
||||
settle_approval_tail(&q, id, 7, State::Done);
|
||||
settle_approval_tail(&q, id, 7, TerminalState::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
@ -1464,7 +1395,7 @@ fn perm_change_shape_prefixes_rebuild_chain() {
|
|||
assert_eq!(c.kind.as_str(), expected);
|
||||
q.complete_node(id, c.node_id, Ok(()));
|
||||
}
|
||||
settle_rebuild_tail(&q, id, "agent-a", State::Done);
|
||||
settle_rebuild_tail(&q, id, "agent-a", true);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue