vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2772): branch on outcome in the graph, not inside the node

Browse source 
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:

- `Skipped` — the node's own edges ruled it out. Expected; the failure
  branch of a run that succeeded is `Skipped`. A parent's roll-up
  **ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
  not-success for the roll-up, as before.

Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.

`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.

With that, the templates stop computing outcomes and let the graph pick:

- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
  edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
  of the two the graph let run.

Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.

Deletes, all of them #2770's host-side debt:

- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
  and the dep-snapshotting loop in `claim_ready`. Executors read their
  own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
  is derived from the edges: `cancel` keeps a node iff one of its edges
  accepts `Cancelled`. An approval tail names it and survives to resolve
  the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
  claim that this couldn't dissolve was only true while `AFTER_ANY`
  accepted cancellation.

`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.

Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
atlas 2026-07-27 16:48:14 +02:00 • committed by mara
commit 07078b76ef
8 changed files with 365 additions and 385 deletions
Download patch file Download diff file Expand all files Collapse all files

54
hive-jobq/src/lib.rs
View file

@ -66,8 +66,18 @@ pub enum TerminalState {
Done,
/// Own logic failed, or a sub-node did.
Failed,
/// Never ran — an edge it depended on became unsatisfiable.
/// Never ran because the work was **dropped** before it could start — the
/// caller cancelled the whole group while it was still queued. Counts as
/// not-success when a parent rolls up.
Cancelled,
/// Never ran because its own **edges ruled it out**: a dependency settled on
/// an outcome the edge doesn't accept. Expected, not a problem — the failure
/// branch of a run that succeeded is `Skipped`.
///
/// A parent's roll-up **ignores** `Skipped` children entirely. Without that,
/// branching on outcome would be self-defeating: exactly one branch is always
/// ruled out, so every group containing one would roll up failed.
Skipped,
}
impl TerminalState {
@ -77,6 +87,7 @@ impl TerminalState {
TerminalState::Done => 1,
TerminalState::Failed => 1 << 1,
TerminalState::Cancelled => 1 << 2,
TerminalState::Skipped => 1 << 3,
}
}
}
@ -90,6 +101,7 @@ impl State {
State::Done => Some(TerminalState::Done),
State::Failed => Some(TerminalState::Failed),
State::Cancelled => Some(TerminalState::Cancelled),
State::Skipped => Some(TerminalState::Skipped),
State::Pending | State::Running | State::Finishing => None,
}
}
@ -119,11 +131,17 @@ impl DepWhen {
/// cancelled down the chain — e.g. a failed `Prebuild` must not let
/// `StopForUpdate` stop a healthy container.
pub const AFTER_OK: Self = Self(TerminalState::Done.bit());
/// The dependency need only be terminal — any outcome satisfies. For steps
/// that must converge regardless, e.g. `Reconcile` running even when the
/// preceding `Swap` failed, or a tail node that reports how the work ended.
/// Anything **except the work being dropped** — `Done`, `Failed` or
/// `Skipped`. For steps that must converge regardless of how the run went,
/// e.g. `Reconcile` bringing a container back up even when the preceding
/// `Swap` failed *or* was itself ruled out by a failed `MetaSync`.
///
/// Deliberately excludes [`TerminalState::Cancelled`]: if the group never
/// started at all there is nothing to converge, and running the recovery
/// step anyway would act on work that provably never happened. A node that
/// must report a cancellation names `Cancelled` explicitly.
pub const AFTER_ANY: Self = Self(
TerminalState::Done.bit() | TerminalState::Failed.bit() | TerminalState::Cancelled.bit(),
TerminalState::Done.bit() | TerminalState::Failed.bit() | TerminalState::Skipped.bit(),
);
/// An edge satisfied by exactly the listed outcomes.
@ -204,18 +222,24 @@ pub enum State {
Done,
/// Completed unsuccessfully — own logic failed, or a sub-node did.
Failed,
/// Never ran: an `AfterOk` dependency failed, so this node (and the rest of
/// its strong-dependent chain) is cancelled rather than run.
/// Never ran: the work was dropped while still queued. See
/// [`TerminalState::Cancelled`].
Cancelled,
/// Never ran: its own edges ruled it out. See [`TerminalState::Skipped`] —
/// notably, a parent's roll-up ignores these.
Skipped,
}
impl State {
/// A node is *terminal* once it has finished — successfully, unsuccessfully,
/// or cancelled — which is when its resources are released and dependents
/// are re-evaluated.
/// dropped, or ruled out — which is when its resources are released and
/// dependents are re-evaluated.
#[must_use]
pub fn is_terminal(self) -> bool {
matches!(self, State::Done | State::Failed | State::Cancelled)
matches!(
self,
State::Done | State::Failed | State::Cancelled | State::Skipped
)
}
}
@ -590,10 +614,16 @@ mod tests {
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Failed));
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Cancelled));
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Running));
// AfterAny: any terminal state satisfies.
assert!(!DepWhen::AFTER_OK.satisfied_by(State::Skipped));
// AfterAny: the dep reached a terminal state *some other way than being
// dropped* — success, failure, or ruled out by its own edges.
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Done));
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Failed));
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Cancelled));
assert!(DepWhen::AFTER_ANY.satisfied_by(State::Skipped));
assert!(
!DepWhen::AFTER_ANY.satisfied_by(State::Cancelled),
"a dropped dep does not converge a weak dependent — nothing ever ran"
);
assert!(!DepWhen::AFTER_ANY.satisfied_by(State::Pending));
// Finishing satisfies neither — a dependent waits until the node rolls
// up to a terminal state (all its sub-nodes done).

67
hive-jobq/src/scheduler.rs
View file

@ -235,6 +235,9 @@ impl<N, R: Clone + Eq + Hash> Scheduler<N, R> {
/// Whether any direct child of `id` ended `Failed`/`Cancelled` — the roll-up
/// failure condition for the parent.
/// `Skipped` children are **not** counted: being ruled out by an edge is the
/// expected fate of every branch not taken, so counting it would make any
/// group that branches on outcome roll up failed no matter how the run went.
fn any_child_failed(&self, id: NodeId) -> bool {
self.graph
.nodes()
@ -321,9 +324,16 @@ impl<N, R: Clone + Eq + Hash> Scheduler<N, R> {
.collect()
}
/// Propagate cancellation out from a just-terminal `origin`: every
/// still-`Pending` node that can no longer run gets marked `Cancelled`,
/// transitively. Cancelled nodes were `Pending`, so they hold no resources.
/// Propagate elimination out from a just-terminal `origin`: every
/// still-`Pending` node that can no longer run gets marked
/// [`State::Skipped`], transitively. Skipped nodes were `Pending`, so they
/// hold no resources.
///
/// `Skipped`, not `Cancelled`: these nodes were *ruled out by their edges*,
/// which is a normal outcome, not a dropped job. `Cancelled` is reserved for
/// work the caller abandoned before it started ([`Scheduler::cancel_node`]),
/// and the two are distinguished precisely so a parent's roll-up can ignore
/// the former while still treating the latter as not-success.
///
/// One rule decides it: **a node is doomed once any edge it names can never
/// be satisfied** — the dep settled on an outcome that edge does not accept.
@ -354,7 +364,7 @@ impl<N, R: Clone + Eq + Hash> Scheduler<N, R> {
.map(|n| n.id)
.collect();
for d in doomed {
self.graph.set_state(d, State::Cancelled);
self.graph.set_state(d, State::Skipped);
stack.push(d);
}
}
@ -505,10 +515,10 @@ mod tests {
assert_eq!(n.error.as_deref(), Some("boom"));
assert!(n.finished_at.is_some());
// The `AfterOk`-cancelled node: cascade-cancelled, so finished_at is set,
// The `AFTER_OK` dependent: ruled out by its edge, so finished_at is set,
// but it never ran (no started_at) and carries no error of its own.
let n = s.graph().node(downstream).unwrap();
assert_eq!(n.state, State::Cancelled);
assert_eq!(n.state, State::Skipped);
assert!(n.started_at.is_none());
assert!(n.finished_at.is_some());
assert_eq!(n.error, None);
@ -750,8 +760,8 @@ mod tests {
.expect("weak");
assert_eq!(s.settle(), vec![root]);
s.complete(root, Outcome::Failed(String::new()));
assert_eq!(s.graph().node(strong1).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(strong2).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(strong1).unwrap().state, State::Skipped);
assert_eq!(s.graph().node(strong2).unwrap().state, State::Skipped);
assert_eq!(s.settle(), vec![weak]);
}
@ -776,8 +786,9 @@ mod tests {
s.complete(root, Outcome::Done);
assert_eq!(
s.graph().node(on_fail).unwrap().state,
State::Cancelled,
"a Failed-only branch is unsatisfiable once its dep succeeds"
State::Skipped,
"a Failed-only branch is unsatisfiable once its dep succeeds — and it is \
`Skipped`, not `Cancelled`, so the parent roll-up ignores it"
);
assert!(s.settle().is_empty(), "and nothing is left runnable");
}
@ -803,15 +814,15 @@ mod tests {
.expect("on_fail");
assert_eq!(s.settle(), vec![root]);
s.complete(root, Outcome::Failed("boom".to_owned()));
assert_eq!(s.graph().node(on_ok).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(on_ok).unwrap().state, State::Skipped);
assert_eq!(s.settle(), vec![on_fail]);
}
/// A weak edge accepts cancellation, so a tail survives the cancellation of
/// the work it reports on — the property hive-c0re's approval tails rely on,
/// held here by the edge itself rather than by any node-kind special case.
/// A weak edge accepts a dependency that was *ruled out*, so a tail still
/// runs when the work it reports on never happened — held by the edge itself
/// rather than by any node-kind special case.
#[test]
fn cancelled_dep_still_satisfies_a_weak_edge() {
fn eliminated_dep_still_satisfies_a_weak_edge() {
let mut s: Scheduler<&str, String> = Scheduler::new(Graph::new(), ResourceTable::new());
let root = s.append("root", vec![], None).expect("root");
let mid = s.append("mid", vec![after_ok(root)], None).expect("mid");
@ -827,7 +838,7 @@ mod tests {
.expect("tail");
assert_eq!(s.settle(), vec![root]);
s.complete(root, Outcome::Failed("boom".to_owned()));
assert_eq!(s.graph().node(mid).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(mid).unwrap().state, State::Skipped);
assert_eq!(
s.settle(),
vec![tail],
@ -838,11 +849,12 @@ mod tests {
/// Edges are **conjunctive**, so "any of these several nodes failed" is not
/// directly expressible — a `{Failed}` edge on each would mean *all* failed.
/// The composition that does work: the success branch depends `AFTER_OK` on
/// every node (so it runs only if all succeeded, and is cancelled the moment
/// one doesn't), and the failure branch hangs off *it* with `{Cancelled}`.
/// The success branch becomes the aggregator, and exactly one of the two runs.
/// every node (so it runs only if all succeeded, and is ruled out the moment
/// one doesn't), and the failure branch hangs off *it* with `{Skipped}` —
/// "run when the success branch was ruled out". The success branch is the
/// aggregator, and exactly one of the two runs.
#[test]
fn ok_branch_aggregates_and_failure_branch_hangs_off_its_cancellation() {
fn ok_branch_aggregates_and_failure_branch_hangs_off_its_elimination() {
let build = || {
let mut s: Scheduler<&str, String> = Scheduler::new(Graph::new(), ResourceTable::new());
let a = s.append("a", vec![], None).expect("a");
@ -855,7 +867,7 @@ mod tests {
"on_fail",
vec![Dep::Node {
id: on_ok,
when: DepWhen::of(&[TerminalState::Cancelled]),
when: DepWhen::of(&[TerminalState::Skipped]),
}],
None,
)
@ -870,16 +882,16 @@ mod tests {
s.complete(b, Outcome::Done);
assert_eq!(s.settle(), vec![on_ok]);
s.complete(on_ok, Outcome::Done);
assert_eq!(s.graph().node(on_fail).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(on_fail).unwrap().state, State::Skipped);
assert!(s.settle().is_empty());
// One of them fails: the ok branch is cancelled, which is precisely the
// One of them fails: the ok branch is ruled out, which is precisely the
// signal the failure branch waits on.
let (mut s, a, b, on_ok, on_fail) = build();
assert_eq!(s.settle(), vec![a, b]);
s.complete(a, Outcome::Failed("boom".to_owned()));
s.complete(b, Outcome::Done);
assert_eq!(s.graph().node(on_ok).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(on_ok).unwrap().state, State::Skipped);
assert_eq!(s.settle(), vec![on_fail]);
}
@ -893,8 +905,8 @@ mod tests {
let grandchild = s.append("gc", vec![], Some(child)).expect("gc");
assert_eq!(s.settle(), vec![root]);
s.complete(root, Outcome::Failed(String::new()));
assert_eq!(s.graph().node(child).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(grandchild).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(child).unwrap().state, State::Skipped);
assert_eq!(s.graph().node(grandchild).unwrap().state, State::Skipped);
}
#[test]
@ -903,8 +915,9 @@ mod tests {
let a = s.append("a", vec![], None).expect("a");
let b = s.append("b", vec![after_ok(a)], None).expect("b");
assert!(s.cancel_node(a));
// `a` was dropped by the caller; `b` was merely ruled out by its edge.
assert_eq!(s.graph().node(a).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(b).unwrap().state, State::Cancelled);
assert_eq!(s.graph().node(b).unwrap().state, State::Skipped);
let c = s.append("c", vec![], None).expect("c");
assert_eq!(s.settle(), vec![c]);
assert!(!s.cancel_node(c));