feat(#2772): branch on outcome in the graph, not inside the node
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
parent
affedecaa5
commit
07078b76ef
8 changed files with 365 additions and 385 deletions
|
|
@ -66,8 +66,18 @@ pub enum TerminalState {
|
|||
Done,
|
||||
/// Own logic failed, or a sub-node did.
|
||||
Failed,
|
||||
/// Never ran — an edge it depended on became unsatisfiable.
|
||||
/// Never ran because the work was **dropped** before it could start — the
|
||||
/// caller cancelled the whole group while it was still queued. Counts as
|
||||
/// not-success when a parent rolls up.
|
||||
Cancelled,
|
||||
/// Never ran because its own **edges ruled it out**: a dependency settled on
|
||||
/// an outcome the edge doesn't accept. Expected, not a problem — the failure
|
||||
/// branch of a run that succeeded is `Skipped`.
|
||||
///
|
||||
/// A parent's roll-up **ignores** `Skipped` children entirely. Without that,
|
||||
/// branching on outcome would be self-defeating: exactly one branch is always
|
||||
/// ruled out, so every group containing one would roll up failed.
|
||||
Skipped,
|
||||
}
|
||||
|
||||
impl TerminalState {
|
||||
|
|
@ -77,6 +87,7 @@ impl TerminalState {
|
|||
TerminalState::Done => 1,
|
||||
TerminalState::Failed => 1 << 1,
|
||||
TerminalState::Cancelled => 1 << 2,
|
||||
TerminalState::Skipped => 1 << 3,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -90,6 +101,7 @@ impl State {
|
|||
State::Done => Some(TerminalState::Done),
|
||||
State::Failed => Some(TerminalState::Failed),
|
||||
State::Cancelled => Some(TerminalState::Cancelled),
|
||||
State::Skipped => Some(TerminalState::Skipped),
|
||||
State::Pending | State::Running | State::Finishing => None,
|
||||
}
|
||||
}
|
||||
|
|
@ -119,11 +131,17 @@ impl DepWhen {
|
|||
/// cancelled down the chain — e.g. a failed `Prebuild` must not let
|
||||
/// `StopForUpdate` stop a healthy container.
|
||||
pub const AFTER_OK: Self = Self(TerminalState::Done.bit());
|
||||
/// The dependency need only be terminal — any outcome satisfies. For steps
|
||||
/// that must converge regardless, e.g. `Reconcile` running even when the
|
||||
/// preceding `Swap` failed, or a tail node that reports how the work ended.
|
||||
/// Anything **except the work being dropped** — `Done`, `Failed` or
|
||||
/// `Skipped`. For steps that must converge regardless of how the run went,
|
||||
/// e.g. `Reconcile` bringing a container back up even when the preceding
|
||||
/// `Swap` failed *or* was itself ruled out by a failed `MetaSync`.
|
||||
///
|
||||
/// Deliberately excludes [`TerminalState::Cancelled`]: if the group never
|
||||
/// started at all there is nothing to converge, and running the recovery
|
||||
/// step anyway would act on work that provably never happened. A node that
|
||||
/// must report a cancellation names `Cancelled` explicitly.
|
||||
pub const AFTER_ANY: Self = Self(
|
||||
TerminalState::Done.bit() | TerminalState::Failed.bit() | TerminalState::Cancelled.bit(),
|
||||
TerminalState::Done.bit() | TerminalState::Failed.bit() | TerminalState::Skipped.bit(),
|
||||
);
|
||||
|
||||
/// An edge satisfied by exactly the listed outcomes.
|
||||
|
|
@ -204,18 +222,24 @@ pub enum State {
|
|||
Done,
|
||||
/// Completed unsuccessfully — own logic failed, or a sub-node did.
|
||||
Failed,
|
||||
/// Never ran: an `AfterOk` dependency failed, so this node (and the rest of
|
||||
/// its strong-dependent chain) is cancelled rather than run.
|
||||
/// Never ran: the work was dropped while still queued. See
|
||||
/// [`TerminalState::Cancelled`].
|
||||
Cancelled,
|
||||
/// Never ran: its own edges ruled it out. See [`TerminalState::Skipped`] —
|
||||
/// notably, a parent's roll-up ignores these.
|
||||
Skipped,
|
||||
}
|
||||
|
||||
impl State {
|
||||
/// A node is *terminal* once it has finished — successfully, unsuccessfully,
|
||||
/// or cancelled — which is when its resources are released and dependents
|
||||
/// are re-evaluated.
|
||||
/// dropped, or ruled out — which is when its resources are released and
|
||||
/// dependents are re-evaluated.
|
||||
#[must_use]
|
||||
pub fn is_terminal(self) -> bool {
|
||||
matches!(self, State::Done | State::Failed | State::Cancelled)
|
||||
matches!(
|
||||
self,
|
||||
State::Done | State::Failed | State::Cancelled | State::Skipped
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -590,10 +614,16 @@ mod tests {
|
|||
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Failed));
|
||||
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Cancelled));
|
||||
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Running));
|
||||
// AfterAny: any terminal state satisfies.
|
||||
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Skipped));
|
||||
// AfterAny: the dep reached a terminal state *some other way than being
|
||||
// dropped* — success, failure, or ruled out by its own edges.
|
||||
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Done));
|
||||
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Failed));
|
||||
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Cancelled));
|
||||
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Skipped));
|
||||
assert!(
|
||||
!DepWhen::AFTER_ANY.satisfied_by(State::Cancelled),
|
||||
"a dropped dep does not converge a weak dependent — nothing ever ran"
|
||||
);
|
||||
assert!(!DepWhen::AFTER_ANY.satisfied_by(State::Pending));
|
||||
// Finishing satisfies neither — a dependent waits until the node rolls
|
||||
// up to a terminal state (all its sub-nodes done).
|
||||
|
|
|
|||
Loading…
Reference in a new issue