vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2772): branch on outcome in the graph, not inside the node

Browse source 
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:

- `Skipped` — the node's own edges ruled it out. Expected; the failure
  branch of a run that succeeded is `Skipped`. A parent's roll-up
  **ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
  not-success for the roll-up, as before.

Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.

`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.

With that, the templates stop computing outcomes and let the graph pick:

- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
  edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
  of the two the graph let run.

Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.

Deletes, all of them #2770's host-side debt:

- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
  and the dep-snapshotting loop in `claim_ready`. Executors read their
  own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
  is derived from the edges: `cancel` keeps a node iff one of its edges
  accepts `Cancelled`. An approval tail names it and survives to resolve
  the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
  claim that this couldn't dissolve was only true while `AFTER_ANY`
  accepted cancellation.

`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.

Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
atlas 2026-07-27 16:48:14 +02:00 • committed by mara
commit 07078b76ef
8 changed files with 365 additions and 385 deletions
Download patch file Download diff file Expand all files Collapse all files

131
hive-c0re/src/job_queue/mod.rs
View file

@ -49,7 +49,7 @@ use hive_sh4re::wire_time::now_unix;
use tokio::sync::Notify;
use crate::coordinator::TransientKind;
pub use model::{DagSpec, DagView, NodeKind, NodeSpec, PermPayload, Source, State};
pub use model::{DagSpec, DagView, NodeKind, NodeSpec, PermPayload, Source, State, TerminalState};
use resource::Resource;
/// How many terminal DAGs (`Done` / `Failed` / `Cancelled`) the snapshot
@ -60,31 +60,6 @@ const MAX_HISTORY_DAGS: usize = 50;
/// Cap on stored node error strings.
const MAX_ERROR_LEN: usize = 2_000;
/// How one of a claimed node's dependencies finished, snapshotted at claim time.
///
/// A node only starts once its edges are satisfied, so every dep named here is
/// already terminal: an `AfterOk` edge means [`State::Done`], an `AfterAny` edge
/// means any of `Done` / `Failed` / `Cancelled`.
///
/// This is what lets a tail node be an ordinary node. A tail that must report
/// how the work below it went — "emit `Rebuilt { ok }`", "resolve this approval
/// with the failure note" — reads its deps' outcomes off its own claim, rather
/// than re-deriving them from the world after the fact. The alternative in this
/// codebase is `DeployTail`, which infers success by re-reading *git state*; that
/// works only because a deploy happens to write its result somewhere durable, and
/// it is not a pattern to copy.
///
/// Carries no node id: a tail acts on *how* its dependencies ended, never on
/// which one it was, so an id here would be a field with no reader.
#[derive(Debug, Clone)]
pub struct DepOutcome {
/// Its terminal state, in wire terms.
pub state: State,
/// Its failure reason, when it failed with one of its own. A node that
/// rolled up `Failed` from a child, or was cancelled, carries no error.
pub error: Option<String>,
}
/// A node claimed for execution — everything the executor needs, snapshotted at
/// claim time.
#[derive(Debug, Clone)]
@ -101,38 +76,6 @@ pub struct Claim {
/// pill is currently shown is derived from live lease ownership
/// ([`JobQueue::held_transients`]), not a per-claim edge.
pub transient: Option<TransientKind>,
/// How each node this one depends on finished. Empty for a head node.
/// See [`DepOutcome`] — this is how a tail node learns the outcome of the
/// work it follows without going back to the world to ask.
pub deps: Vec<DepOutcome>,
}
impl Claim {
/// Roll this claim's dependencies up into one outcome — what a weak-edged
/// tail node acts on. `Done` only when every dep succeeded; `Failed` when any
/// failed; otherwise `Cancelled` (all terminal, none failed, so the work was
/// dropped before it ran).
///
/// A head node has no deps and rolls up `Done` — vacuously true, and never
/// reached in practice since only tail kinds consult this.
pub fn deps_state(&self) -> State {
if self.deps.iter().all(|d| d.state == State::Done) {
State::Done
} else if self.deps.iter().any(|d| d.state == State::Failed) {
State::Failed
} else {
State::Cancelled
}
}
/// The first failed dependency's error, for reporting *why* the work ended
/// badly. `None` when nothing failed.
pub fn deps_error(&self) -> Option<&str> {
self.deps
.iter()
.find(|d| d.state == State::Failed)
.and_then(|d| d.error.as_deref())
}
}
/// Per-node runtime metadata the crate graph doesn't carry. Lifecycle
@ -192,13 +135,19 @@ impl Default for JobQueue {
/// Map a crate node state onto the wire state (`Pending` ↔ `Queued`;
/// `Finishing` — own logic done, sub-nodes still running — reads as `Running`).
///
/// `Skipped` has no wire counterpart and folds into `Cancelled`: to a reader
/// both mean "this never ran". The distinction is a *scheduling* one — it
/// decides whether a parent's roll-up counts the node — and the wire carries no
/// roll-up input, only display state. In practice a client never sees it either:
/// `dag_view` drops skipped nodes from the snapshot along with `Done` ones.
fn to_wire_state(state: JobState) -> State {
match state {
JobState::Pending => State::Queued,
JobState::Running | JobState::Finishing => State::Running,
JobState::Done => State::Done,
JobState::Failed => State::Failed,
JobState::Cancelled => State::Cancelled,
JobState::Cancelled | JobState::Skipped => State::Cancelled,
}
}
@ -365,24 +314,6 @@ impl JobQueue {
};
let kind = node.payload.clone();
let agent = node.payload.agent().to_owned();
let dep_ids: Vec<NodeId> = node
.deps
.iter()
.filter_map(|d| match d {
Dep::Node { id, .. } => Some(*id),
Dep::Resource { .. } => None,
})
.collect();
let deps: Vec<DepOutcome> = dep_ids
.into_iter()
.filter_map(|dep| {
let n = inner.sched.graph().node(dep)?;
Some(DepOutcome {
state: to_wire_state(n.state),
error: n.error.clone(),
})
})
.collect();
let Some(container) = inner.dag_of(id) else {
continue;
};
@ -397,7 +328,6 @@ impl JobQueue {
approval_id: meta.approval_id,
inputs: meta.inputs,
transient: meta.transient,
deps,
});
// `started_at` is stamped on the graph `Node` by the scheduler's
// transition to `Running` — no host-side copy needed.
@ -430,17 +360,17 @@ impl JobQueue {
/// so each is cancelled. `false` once any work node is running or terminal —
/// an in-flight nix build isn't interruptible.
///
/// **Tail nodes are spared** ([`NodeKind::is_tail`]). They are weak-edged
/// (`AfterAny`), and a `Cancelled` dep satisfies a weak edge, so sparing one
/// leaves it *ready* rather than stranded: the scheduler claims it on the next
/// pass, its [`Claim::deps_state`] reads `Cancelled`, and it resolves the
/// approval as "cancelled before completion". That is what stops a queued
/// approval DAG the operator cancelled from dangling its approval forever —
/// the job the inline hook used to do from outside the graph.
/// **Nodes that explicitly observe cancellation are spared** — a node whose
/// edge names [`hive_jobq::TerminalState::Cancelled`] is asking to run when
/// the work it follows was dropped, which is exactly what an approval tail
/// needs: cancel the work, and the tail still fires to resolve the approval
/// row rather than leaving it dangling forever.
///
/// Cancelling the tail too would be the bug: `cancel_node` only cascades along
/// `AfterOk` edges and parent links, so nothing else would reach it, and the
/// approval row would simply never be touched.
/// Nothing is special-cased by node kind. `AFTER_ANY` deliberately does *not*
/// accept `Cancelled`, so an ordinary weak-edged step (rebuild's `Reconcile`,
/// say) is cancelled along with everything else — there is nothing to converge
/// when no node ever ran. Only a node that named `Cancelled` survives, and it
/// survives because it asked to.
pub fn cancel(&self, dag_id: u64) -> bool {
let mut inner = self.lock();
let Some(container) = inner.container(dag_id) else {
@ -458,12 +388,7 @@ impl JobQueue {
return false;
}
for id in work {
if inner
.sched
.graph()
.node(id)
.is_some_and(|n| n.payload.is_tail())
{
if inner.observes_cancellation(id) {
continue;
}
inner.sched.cancel_node(id);
@ -642,6 +567,18 @@ impl QueueInner {
.is_some_and(|n| n.state.is_terminal())
}
/// Whether `id` has an edge that accepts a **dropped** dependency — i.e. the
/// node exists to report on work that may never run. Used by
/// [`JobQueue::cancel`] to decide what to spare, so the decision comes from
/// the node's own declared edges rather than a hardcoded list of kinds.
fn observes_cancellation(&self, id: NodeId) -> bool {
self.sched.graph().node(id).is_some_and(|n| {
n.deps.iter().any(|d| {
matches!(d, Dep::Node { when, .. } if when.accepts(hive_jobq::TerminalState::Cancelled))
})
})
}
/// First failed work node's error (read off the graph `Node`), for the
/// dashboard's DAG-level error line.
fn dag_first_error(&self, container: NodeId) -> Option<String> {
@ -682,7 +619,11 @@ impl QueueInner {
if let Some(f) = node.finished_at {
finished.push(f);
}
if node.state == JobState::Done {
// `Done` nodes drop off the wire (a finished step isn't interesting),
// and so do `Skipped` ones: a branch that was never taken is noise on
// the dashboard, and surfacing it would also drag the client-side
// roll-up toward `Cancelled` for a run that went fine.
if matches!(node.state, JobState::Done | JobState::Skipped) {
continue;
}
let deps: Vec<u64> = node