vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

refactor(#2756): replace the DAG terminal hook with real tail nodes

Browse source 
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.

`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.

So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.

Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.

Load-bearing details:

- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
  subtree, and returns `bool`. Without this a cancelled approval DAG
  would dangle its approval forever — the hazard `tests.rs` already
  named. The spared tail's deps are `Cancelled`, which satisfies its weak
  edge, so the scheduler claims it and it resolves the row as cancelled.
  `hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
  to settle afterwards so "a weak-edge terminal node observing the
  cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
  deliberately a no-op when a tail was spared (a non-terminal child parks
  the container back in `Finishing`), so power ops still settle
  synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
  cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
  group root that rolled up `Failed` from a child carries no error of its
  own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
  own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
  agent, and that is also what lets one tail close a multi-agent DAG.

Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
This commit is contained in:
atlas 2026-07-27 14:25:03 +02:00 • committed by mara
commit e8e6998ac5
11 changed files with 521 additions and 324 deletions
Download patch file Download diff file Expand all files Collapse all files

87
hive-c0re/src/job_queue/model.rs
View file

@ -17,27 +17,6 @@ use serde::Serialize;
use crate::coordinator::TransientKind;
/// The inline side effect a settled DAG fires when its container node rolls
/// up terminal (there is no hook *node*). Stated explicitly by the builder in
/// `templates.rs` / `submit.rs` rather than inferred from a DAG-level enum:
/// only the builder knows why it assembled the DAG, so only the builder can
/// say what should happen at the end of it.
///
/// A cancelled DAG deliberately gets **no** compensating hook. [`super::JobQueue::cancel`]
/// refuses unless every work node is still `Pending`, and a cancel *cascade*
/// rolls up `Failed` (see `dag_rollup`), never `Cancelled` — so on a
/// `Cancelled` DAG no node ever executed and there is nothing to undo. A power
/// op's `SetWanted` head provably never ran, so its intent is still whatever
/// the operator last set it to.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum HookKind {
/// Approval-driven DAG (spawn / opaque deploy): resolve the approval row.
ResolveApproval,
/// Rebuild / perm-change: emit one `Rebuilt` manager event per agent.
EmitRebuilt,
}
/// When a dependency edge is considered satisfied.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
@ -248,6 +227,31 @@ pub enum NodeKind {
/// means it survives a `hive-c0re` restart mid-deploy, which an in-memory
/// queue does not.
DeployTail { agent: String },
/// Tail node of an approval-carrying DAG (spawn / opaque deploy / config-PR
/// merge): resolve the approval row from how the work actually ended.
///
/// Weak-edged (`DepWhen::AfterAny`) like [`NodeKind::DeployTail`], so it runs on
/// success, failure **and cancel** alike and decides internally. It reads its
/// dependencies' terminal states off its own [`Claim::deps`] rather than
/// re-deriving them from the world the way `DeployTail` reads git: a node is
/// *told* how the work it follows ended, it does not go back out and ask.
///
/// Agentless on purpose: the approval row already names its agent, so
/// carrying one here would be a second copy free to drift. Like
/// [`NodeKind::MetaLock`] it reports `""` from [`NodeKind::agent`] and takes
/// no lease — which is also what lets one close a multi-agent DAG.
///
/// [`Claim::deps`]: super::Claim::deps
ResolveApproval { approval_id: i64 },
/// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt` manager
/// event — `ok` when its deps are `Done`, `!ok` carrying the failure note when
/// they `Failed`, and **nothing at all** when they `Cancelled` (a cancelled DAG
/// never ran, so there is no rebuild to report).
///
/// One node **per agent**, unlike the DAG-wide hook it replaces: a multi-agent
/// DAG now reports each agent's own outcome instead of painting every agent with
/// the whole DAG's roll-up.
EmitRebuilt { agent: String },
/// Write the agent's durable power intent (`wanted = Up` when `up`, else
/// `Offline`) as a first-class DAG node, at the head of a power-op
/// template so the downstream `Reconcile` reads it. Replaces the old
@ -264,15 +268,11 @@ pub enum NodeKind {
SetWanted { agent: String, up: bool },
/// The **DAG container** node: one per submitted DAG, carrying the group's
/// domain metadata. Every node hangs *under* it (its subtree), so
/// the container's `NodeId` **is** the DAG id, its rolled-up state **is** the
/// DAG state, and it reaching terminal **is** the completion signal that
/// fires the DAG's inline `hook`. Pure grouping — lease- and
/// the container's `NodeId` **is** the DAG id and its rolled-up state **is**
/// the DAG state. Pure grouping — lease- and
/// build-slot-exempt; the executor instant-completes it (`Done`) so it
/// reaches `Finishing` and its children start.
Dag {
/// The side effect to run when this DAG settles, or `None` for a DAG
/// with none (power op, meta-update, boot).
hook: Option<HookKind>,
source: Source,
reason: String,
transient: Option<TransientKind>,
@ -307,6 +307,8 @@ impl NodeKind {
NodeKind::DeployApply { .. } => "deploy_apply",
NodeKind::FinalizeDeploy { .. } => "finalize_deploy",
NodeKind::DeployTail { .. } => "deploy_tail",
NodeKind::ResolveApproval { .. } => "resolve_approval",
NodeKind::EmitRebuilt { .. } => "emit_rebuilt",
NodeKind::SetWanted { .. } => "set_wanted",
NodeKind::Dag { .. } => "dag",
}
@ -338,11 +340,29 @@ impl NodeKind {
| NodeKind::DeployApply { agent }
| NodeKind::FinalizeDeploy { agent }
| NodeKind::DeployTail { agent }
| NodeKind::EmitRebuilt { agent }
| NodeKind::SetWanted { agent, .. } => agent,
NodeKind::MetaLock { .. } | NodeKind::Reparent { .. } | NodeKind::Dag { .. } => "",
NodeKind::MetaLock { .. }
| NodeKind::Reparent { .. }
| NodeKind::ResolveApproval { .. }
| NodeKind::Dag { .. } => "",
}
}
/// Whether this is a DAG's **tail** — a node that reports how the rest of the
/// DAG ended rather than doing work of its own.
///
/// The one place this matters is [`super::JobQueue::cancel`], which spares
/// tails so they still run (and report `Cancelled`) on a cancelled DAG. Note
/// [`NodeKind::DeployTail`] is *not* one: despite the name it does real
/// compensating work, and on a cancelled DAG there is nothing to compensate.
pub fn is_tail(&self) -> bool {
matches!(
self,
NodeKind::ResolveApproval { .. } | NodeKind::EmitRebuilt { .. }
)
}
/// Nix-heavy kinds hold one of the `buildSlots` semaphore permits
/// for the node's duration.
pub fn needs_build_slot(&self) -> bool {
@ -446,14 +466,15 @@ pub struct NodeSpec {
/// ([`NodeKind::WritePermFile`]), not this generic spec.
#[derive(Debug, Clone)]
pub struct DagSpec {
/// The inline side effect to fire when this DAG settles. Explicit — the
/// builder assembling the DAG is the only thing that knows its intent.
pub hook: Option<HookKind>,
pub source: Source,
/// Free-form "why".
pub reason: String,
/// The approval row [`HookKind::ResolveApproval`] resolves. Set together
/// with that hook; carried separately because the hook needs the id.
/// The approval row this DAG belongs to, for display + the `approval_id` on
/// every [`Claim`]. The *resolving* of it rides the
/// [`NodeKind::ResolveApproval`] tail node instead — this field does not
/// drive it.
///
/// [`Claim`]: super::Claim
pub approval_id: Option<i64>,
/// Meta-update only: the inputs to bump. Display copy lives on the DAG.
pub inputs: Vec<String>,