vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2772): branch on outcome in the graph, not inside the node

Browse source 
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:

- `Skipped` — the node's own edges ruled it out. Expected; the failure
  branch of a run that succeeded is `Skipped`. A parent's roll-up
  **ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
  not-success for the roll-up, as before.

Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.

`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.

With that, the templates stop computing outcomes and let the graph pick:

- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
  edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
  of the two the graph let run.

Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.

Deletes, all of them #2770's host-side debt:

- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
  and the dep-snapshotting loop in `claim_ready`. Executors read their
  own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
  is derived from the edges: `cancel` keeps a node iff one of its edges
  accepts `Cancelled`. An approval tail names it and survives to resolve
  the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
  claim that this couldn't dissolve was only true while `AFTER_ANY`
  accepted cancellation.

`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.

Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
atlas 2026-07-27 16:48:14 +02:00 • committed by mara
commit 07078b76ef
8 changed files with 365 additions and 385 deletions
Download patch file Download diff file Expand all files Collapse all files

150
hive-c0re/src/job_queue/templates.rs
View file

@ -30,7 +30,7 @@
use anyhow::{Result, bail};
use super::model::{DagSpec, Dep, DepWhen, NodeKind, NodeSpec, PermPayload, Source};
use super::model::{DagSpec, Dep, DepWhen, NodeKind, NodeSpec, PermPayload, Source, TerminalState};
use crate::coordinator::TransientKind;
/// After-ok edge on the previous node — the common chain link. Shared with
@ -43,14 +43,28 @@ pub(crate) fn after_ok(on: u64) -> Vec<Dep> {
}]
}
/// Weak edges onto every one of a DAG's other **group-roots** — how a tail node
/// (`ResolveApproval` / `EmitRebuilt`) sees the whole DAG's outcome.
/// `AfterOk` edges onto every one of a DAG's **group-roots** — the success
/// branch of a per-outcome tail pair, and the aggregator the failure branch
/// keys off.
///
/// Group-roots are the right granularity, not "every node": a root's state *is*
/// its subtree's roll-up, so edging the roots covers every descendant while
/// keeping the tail's dep list small and stable as subtrees grow. `AfterAny`
/// throughout, so the tail runs on success, failure and cancel alike and decides
/// from [`super::Claim::deps_state`].
/// keeping the dep list small and stable as subtrees grow. Because every edge is
/// `AFTER_OK`, this node runs only if *all* of them succeeded — and is ruled out
/// ([`TerminalState::Skipped`]) the moment one doesn't, which is precisely the
/// signal [`on_elimination_of`] waits for.
pub(crate) fn after_ok_all(ons: &[u64]) -> Vec<Dep> {
ons.iter()
.map(|&on| Dep {
on,
when: DepWhen::AFTER_OK,
})
.collect()
}
/// `AFTER_ANY` edges onto every group-root — "wait for all of these to finish,
/// however they went". Ordering only; it accepts any outcome except the DAG
/// being dropped.
pub(crate) fn after_any_all(ons: &[u64]) -> Vec<Dep> {
ons.iter()
.map(|&on| Dep {
@ -60,6 +74,89 @@ pub(crate) fn after_any_all(ons: &[u64]) -> Vec<Dep> {
.collect()
}
/// A single edge satisfied only when `on` was **ruled out** by its own edges.
///
/// Dependency edges are conjunctive, so "any one of these several nodes failed"
/// cannot be written directly. This is the composition that expresses it: point
/// the success branch at every root with [`after_ok_all`], then hang the failure
/// branch off *that* node's elimination. Exactly one of the pair ever runs.
///
/// Note it accepts `Skipped` and **not** `Cancelled`: if the whole DAG was
/// dropped before it started, the success branch is marked `Cancelled` directly
/// and this branch is ruled out too — a job nobody ran reports nothing.
pub(crate) fn on_elimination_of(on: u64) -> Vec<Dep> {
vec![Dep {
on,
when: DepWhen::of(&[TerminalState::Skipped]),
}]
}
/// A single edge satisfied only by the listed outcomes of `on` — for the
/// one-tail-per-outcome shape an approval DAG uses.
pub(crate) fn on_outcome(on: u64, outcomes: &[TerminalState]) -> Vec<Dep> {
vec![Dep {
on,
when: DepWhen::of(outcomes),
}]
}
/// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node
/// gated on every group-root in `roots`, and the failure node gated on *its*
/// elimination. `base` is the spec index the pair starts at.
///
/// Exactly one runs on a DAG that executed, and neither runs on one the operator
/// dropped — see [`on_elimination_of`].
fn emit_rebuilt_tails(agent: &str, roots: &[u64], base: u64) -> Vec<NodeSpec> {
// The failure branch needs *both*: the ok branch being ruled out (that is the
// "something went wrong" signal) **and** every root actually finished. The
// second half is easy to forget and gets the ordering wrong without it — a
// failed `Prebuild` eliminates the ok branch immediately, while the recovery
// `Reconcile` is still bringing the container back up, so reporting straight
// off the elimination would announce the failure mid-recovery.
let mut on_fail = after_any_all(roots);
on_fail.extend(on_elimination_of(base));
vec![
node(
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
},
after_ok_all(roots),
),
node(
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: false,
},
on_fail,
),
]
}
/// The approval-resolving tails for an approval-carrying DAG: one per outcome of
/// the DAG's single group-root `root`, each accepting only its own.
///
/// The `Cancelled` node is what keeps a dropped approval DAG from dangling its
/// row forever — its edge is the only one [`super::JobQueue::cancel`] spares.
fn resolve_approval_tails(approval_id: i64, root: u64) -> Vec<NodeSpec> {
[
TerminalState::Done,
TerminalState::Failed,
TerminalState::Cancelled,
]
.into_iter()
.map(|outcome| {
node(
NodeKind::ResolveApproval {
approval_id,
outcome,
},
on_outcome(root, &[outcome]),
)
})
.collect()
}
/// Build one **top-level (group-root)** node — `parent = None`. `kind` carries
/// the agent it targets ([`NodeKind`] is the payload directly). Shared with
/// `submit.rs`'s dynamic power-op builders. A root owns whatever resource it
@ -198,12 +295,7 @@ pub(crate) fn deploy_rebuild_nodes(agent: &str) -> Vec<NodeSpec> {
/// it reaches `Done` even after a failed swap and the tail would report success.
pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> DagSpec {
let mut nodes = rebuild_nodes(agent, relock, 0);
nodes.push(node(
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
},
after_any_all(&[0, 1, 5]),
));
nodes.extend(emit_rebuilt_tails(agent, &[0, 1, 5], 6));
DagSpec {
source,
reason,
@ -259,11 +351,10 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
when: DepWhen::AFTER_ANY,
}],
),
node(
NodeKind::ResolveApproval { approval_id },
after_any_all(&[0]),
),
],
]
.into_iter()
.chain(resolve_approval_tails(approval_id, 0))
.collect(),
}
}
@ -319,11 +410,10 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
child(0, NodeKind::Create { agent: a() }, Vec::new()),
child(1, NodeKind::WriteDropin { agent: a() }, Vec::new()),
child(1, NodeKind::Reconcile { agent: a() }, after_ok(2)),
node(
NodeKind::ResolveApproval { approval_id },
after_any_all(&[0]),
),
]
.into_iter()
.chain(resolve_approval_tails(approval_id, 0))
.collect()
},
}
}
@ -342,12 +432,7 @@ pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPay
Vec::new(),
)];
nodes.extend(rebuild_nodes(agent, true, 1));
nodes.push(node(
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
},
after_any_all(&[0, 1, 2, 6]),
));
nodes.extend(emit_rebuilt_tails(agent, &[0, 1, 2, 6], 7));
DagSpec {
source,
reason,
@ -382,14 +467,11 @@ pub fn meta_update(
Vec::new(),
)];
// The bump itself has no side effect, so an operator-driven one ends at the
// `MetaLock`; an approval-driven one still has its row to resolve and gets a
// tail edged onto that single group-root — whose roll-up covers the rebuild
// subgraphs `MetaLock` grows into itself.
// `MetaLock`; an approval-driven one still has its row to resolve and gets the
// per-outcome tails edged onto that single group-root — whose roll-up covers
// the rebuild subgraphs `MetaLock` grows into itself.
if let Some(approval_id) = approval_id {
nodes.push(node(
NodeKind::ResolveApproval { approval_id },
after_any_all(&[0]),
));
nodes.extend(resolve_approval_tails(approval_id, 0));
}
DagSpec {
source,