refactor(#2756): replace the DAG terminal hook with real tail nodes
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
This commit is contained in:
atlas
mara
parent
896dfc6194
commit
e8e6998ac5
11 changed files with 521 additions and 324 deletions
|
|
@ -95,50 +95,71 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
|
|||
NodeKind::DeployApply { .. } => run_deploy_apply(coord, claim).await,
|
||||
NodeKind::FinalizeDeploy { .. } => run_finalize_deploy(coord, claim).await,
|
||||
NodeKind::DeployTail { .. } => run_deploy_tail(coord, claim).await,
|
||||
NodeKind::ResolveApproval { approval_id, .. } => {
|
||||
run_resolve_approval(coord, claim, *approval_id).await
|
||||
}
|
||||
NodeKind::EmitRebuilt { .. } => Ok(run_emit_rebuilt(coord, claim)),
|
||||
NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up),
|
||||
// Pure grouping container — no work; completing it lets it reach
|
||||
// `Finishing` so its child work nodes start. The DAG's terminal
|
||||
// hook fires (inline, via `run_terminal_hook`) when the container itself
|
||||
// rolls up terminal — not as a scheduled node.
|
||||
// `Finishing` so its child work nodes start. The DAG's terminal side
|
||||
// effect, if any, is its own tail node in the graph.
|
||||
NodeKind::Dag { .. } => Ok(NodeOutput::default()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Run a settled DAG's inline terminal hook — the container-terminal
|
||||
/// replacement for the old per-DAG hook node. Always best-effort: a hook
|
||||
/// failure is logged inside, never surfaced.
|
||||
pub(crate) async fn run_terminal_hook(coord: &Arc<Coordinator>, terminal: &super::TerminalDag) {
|
||||
match terminal.hook {
|
||||
Some(super::HookKind::ResolveApproval) => {
|
||||
crate::actions::resolve_approval_dag(coord, terminal).await;
|
||||
}
|
||||
Some(super::HookKind::EmitRebuilt) => emit_rebuilt(coord, terminal),
|
||||
None => {}
|
||||
}
|
||||
/// Resolve the DAG's approval row from how the work it follows ended. The
|
||||
/// outcome comes off this node's own dependency roll-up, not from re-reading
|
||||
/// the world. Best-effort: a resolution failure is logged inside
|
||||
/// [`crate::actions::resolve_approval_dag`], never surfaced as a node failure —
|
||||
/// the work already happened, and failing the tail would only misreport it.
|
||||
async fn run_resolve_approval(
|
||||
coord: &Arc<Coordinator>,
|
||||
claim: &Claim,
|
||||
approval_id: i64,
|
||||
) -> Result<NodeOutput> {
|
||||
let reason = failure_reason(coord, claim);
|
||||
crate::actions::resolve_approval_dag(coord, approval_id, claim.deps_state(), reason.as_deref())
|
||||
.await;
|
||||
Ok(NodeOutput::default())
|
||||
}
|
||||
|
||||
/// Rebuild / perm-change hook: emit one `Rebuilt` manager event per targeted
|
||||
/// agent — `ok` on `Done`, `!ok` on `Failed`, none on cancel.
|
||||
fn emit_rebuilt(coord: &Arc<Coordinator>, terminal: &super::TerminalDag) {
|
||||
for agent in &terminal.agents {
|
||||
match terminal.state {
|
||||
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
|
||||
agent: agent.clone(),
|
||||
ok: true,
|
||||
note: None,
|
||||
sha: None,
|
||||
tag: None,
|
||||
}),
|
||||
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
|
||||
agent: agent.clone(),
|
||||
ok: false,
|
||||
note: terminal.error.clone(),
|
||||
sha: None,
|
||||
tag: None,
|
||||
}),
|
||||
_ => {}
|
||||
}
|
||||
/// Why the work a tail node follows failed, as a human-readable string.
|
||||
///
|
||||
/// Prefers the tail's own dependency error, but a dep that is a **group root**
|
||||
/// rolled up `Failed` from a child carries no error of its own (the reason lives
|
||||
/// on the leaf that actually failed) — and a grafted subgraph's nodes can't be
|
||||
/// edged statically anyway. So fall back to the DAG's first failing node. Still
|
||||
/// the queue's own graph, not the outside world.
|
||||
fn failure_reason(coord: &Arc<Coordinator>, claim: &Claim) -> Option<String> {
|
||||
claim
|
||||
.deps_error()
|
||||
.map(str::to_owned)
|
||||
.or_else(|| coord.job_queue.first_error(claim.dag_id))
|
||||
}
|
||||
|
||||
/// Emit this agent's `Rebuilt` manager event — `ok` when the work it follows is
|
||||
/// `Done`, `!ok` with the failure note when it `Failed`, and nothing at all when
|
||||
/// it `Cancelled` (nothing ran, so there is no rebuild to report).
|
||||
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim) -> NodeOutput {
|
||||
let agent = claim.agent.clone();
|
||||
match claim.deps_state() {
|
||||
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
|
||||
agent,
|
||||
ok: true,
|
||||
note: None,
|
||||
sha: None,
|
||||
tag: None,
|
||||
}),
|
||||
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
|
||||
agent,
|
||||
ok: false,
|
||||
note: failure_reason(coord, claim),
|
||||
sha: None,
|
||||
tag: None,
|
||||
}),
|
||||
_ => {}
|
||||
}
|
||||
NodeOutput::default()
|
||||
}
|
||||
|
||||
/// Write the agent's durable power intent — the DAG-node form of the old
|
||||
|
|
@ -238,8 +259,8 @@ async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Res
|
|||
// which deps `AfterOk(Swap)`. On failure `PostSwap` is cancel-cascaded
|
||||
// and the tail `Reconcile` (`AfterAny(PostSwap)`) handles recovery; here
|
||||
// we only refresh the observed state so dashboards reflect the failed
|
||||
// swap immediately. The `Rebuilt { ok: false }` manager event fires once
|
||||
// per DAG from the terminal hook (any node may be the one that failed).
|
||||
// swap immediately. The `Rebuilt { ok: false }` manager event is emitted by
|
||||
// the DAG's `EmitRebuilt` tail (any node may be the one that failed).
|
||||
if result.is_err() {
|
||||
coord.rescan_containers_and_emit().await;
|
||||
}
|
||||
|
|
@ -258,8 +279,8 @@ async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
|
|||
{
|
||||
tracing::warn!(%name, error = ?e, "write rev marker failed");
|
||||
}
|
||||
// The `Rebuilt` manager event fires exactly once per DAG from the
|
||||
// terminal hook — emitting ok here and letting a failed tail `Reconcile`
|
||||
// The `Rebuilt` manager event is emitted exactly once per agent by the DAG's
|
||||
// `EmitRebuilt` tail — emitting ok here and letting a failed tail `Reconcile`
|
||||
// add a contradictory !ok would double-report the same rebuild.
|
||||
// Full forge + matrix sync on every successful rebuild so the rebuild
|
||||
// path is equivalent to the startup sweep: tokens, config-repo mirror,
|
||||
|
|
|
|||
Loading…
Reference in a new issue