vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2772): branch on outcome in the graph, not inside the node

Browse source 
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:

- `Skipped` — the node's own edges ruled it out. Expected; the failure
  branch of a run that succeeded is `Skipped`. A parent's roll-up
  **ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
  not-success for the roll-up, as before.

Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.

`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.

With that, the templates stop computing outcomes and let the graph pick:

- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
  edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
  of the two the graph let run.

Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.

Deletes, all of them #2770's host-side debt:

- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
  and the dep-snapshotting loop in `claim_ready`. Executors read their
  own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
  is derived from the edges: `cancel` keeps a node iff one of its edges
  accepts `Cancelled`. An approval tail names it and survives to resolve
  the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
  claim that this couldn't dissolve was only true while `AFTER_ANY`
  accepted cancellation.

`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.

Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
atlas 2026-07-27 16:48:14 +02:00 • committed by mara
commit 07078b76ef
8 changed files with 365 additions and 385 deletions
Download patch file Download diff file Expand all files Collapse all files

80
hive-c0re/src/job_queue/exec.rs
View file

@ -11,7 +11,7 @@ use std::sync::Arc;
use anyhow::{Context as _, Result};
use super::Claim;
use super::model::{NodeKind, NodeSpec, State};
use super::model::{NodeKind, NodeSpec, TerminalState};
use crate::coordinator::Coordinator;
use crate::power::{ReconcileAction, reconcile_action};
@ -95,10 +95,11 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
NodeKind::DeployApply { .. } => run_deploy_apply(coord, claim).await,
NodeKind::FinalizeDeploy { .. } => run_finalize_deploy(coord, claim).await,
NodeKind::DeployTail { .. } => run_deploy_tail(coord, claim).await,
NodeKind::ResolveApproval { approval_id, .. } => {
run_resolve_approval(coord, claim, *approval_id).await
}
NodeKind::EmitRebuilt { .. } => Ok(run_emit_rebuilt(coord, claim)),
NodeKind::ResolveApproval {
approval_id,
outcome,
} => run_resolve_approval(coord, claim, *approval_id, *outcome).await,
NodeKind::EmitRebuilt { ok, .. } => Ok(run_emit_rebuilt(coord, claim, *ok)),
NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up),
// Pure grouping container — no work; completing it lets it reach
// `Finishing` so its child work nodes start. The DAG's terminal side
@ -107,58 +108,39 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
}
}
/// Resolve the DAG's approval row from how the work it follows ended. The
/// outcome comes off this node's own dependency roll-up, not from re-reading
/// the world. Best-effort: a resolution failure is logged inside
/// [`crate::actions::resolve_approval_dag`], never surfaced as a node failure —
/// the work already happened, and failing the tail would only misreport it.
/// Resolve the DAG's approval row the way this node's own `outcome` says.
///
/// Nothing is inspected: a template emits one of these per outcome, each edged to
/// accept only that one, so *which* node the scheduler let run already is the
/// answer. Best-effort — a resolution failure is logged inside
/// [`crate::actions::resolve_approval_dag`], never surfaced as a node failure,
/// since the work already happened and failing the tail would only misreport it.
async fn run_resolve_approval(
coord: &Arc<Coordinator>,
claim: &Claim,
approval_id: i64,
outcome: TerminalState,
) -> Result<NodeOutput> {
let reason = failure_reason(coord, claim);
crate::actions::resolve_approval_dag(coord, approval_id, claim.deps_state(), reason.as_deref())
.await;
let reason = (outcome == TerminalState::Failed)
.then(|| coord.job_queue.first_error(claim.dag_id))
.flatten();
crate::actions::resolve_approval_dag(coord, approval_id, outcome, reason.as_deref()).await;
Ok(NodeOutput::default())
}
/// Why the work a tail node follows failed, as a human-readable string.
///
/// Prefers the tail's own dependency error, but a dep that is a **group root**
/// rolled up `Failed` from a child carries no error of its own (the reason lives
/// on the leaf that actually failed) — and a grafted subgraph's nodes can't be
/// edged statically anyway. So fall back to the DAG's first failing node. Still
/// the queue's own graph, not the outside world.
fn failure_reason(coord: &Arc<Coordinator>, claim: &Claim) -> Option<String> {
claim
.deps_error()
.map(str::to_owned)
.or_else(|| coord.job_queue.first_error(claim.dag_id))
}
/// Emit this agent's `Rebuilt` manager event — `ok` when the work it follows is
/// `Done`, `!ok` with the failure note when it `Failed`, and nothing at all when
/// it `Cancelled` (nothing ran, so there is no rebuild to report).
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim) -> NodeOutput {
let agent = claim.agent.clone();
match claim.deps_state() {
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent,
ok: true,
note: None,
sha: None,
tag: None,
}),
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent,
ok: false,
note: failure_reason(coord, claim),
sha: None,
tag: None,
}),
_ => {}
}
/// Emit this agent's `Rebuilt` manager event. `ok` is not computed — it is which
/// of the tail pair the graph let run. The failure note comes from the DAG's
/// first failing node, since the branch knows *that* it failed but not *why*.
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) -> NodeOutput {
coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: claim.agent.clone(),
ok,
note: (!ok)
.then(|| coord.job_queue.first_error(claim.dag_id))
.flatten(),
sha: None,
tag: None,
});
NodeOutput::default()
}