refactor(#2756): replace the DAG terminal hook with real tail nodes
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
This commit is contained in:
atlas
mara
parent
896dfc6194
commit
e8e6998ac5
11 changed files with 521 additions and 324 deletions
|
|
@ -30,7 +30,7 @@
|
|||
|
||||
use anyhow::{Result, bail};
|
||||
|
||||
use super::model::{DagSpec, Dep, DepWhen, HookKind, NodeKind, NodeSpec, PermPayload, Source};
|
||||
use super::model::{DagSpec, Dep, DepWhen, NodeKind, NodeSpec, PermPayload, Source};
|
||||
use crate::coordinator::TransientKind;
|
||||
|
||||
/// After-ok edge on the previous node — the common chain link. Shared with
|
||||
|
|
@ -43,6 +43,23 @@ pub(crate) fn after_ok(on: u64) -> Vec<Dep> {
|
|||
}]
|
||||
}
|
||||
|
||||
/// Weak edges onto every one of a DAG's other **group-roots** — how a tail node
|
||||
/// (`ResolveApproval` / `EmitRebuilt`) sees the whole DAG's outcome.
|
||||
///
|
||||
/// Group-roots are the right granularity, not "every node": a root's state *is*
|
||||
/// its subtree's roll-up, so edging the roots covers every descendant while
|
||||
/// keeping the tail's dep list small and stable as subtrees grow. `AfterAny`
|
||||
/// throughout, so the tail runs on success, failure and cancel alike and decides
|
||||
/// from [`super::Claim::deps_state`].
|
||||
pub(crate) fn after_any_all(ons: &[u64]) -> Vec<Dep> {
|
||||
ons.iter()
|
||||
.map(|&on| Dep {
|
||||
on,
|
||||
when: DepWhen::AfterAny,
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Build one **top-level (group-root)** node — `parent = None`. `kind` carries
|
||||
/// the agent it targets ([`NodeKind`] is the payload directly). Shared with
|
||||
/// `submit.rs`'s dynamic power-op builders. A root owns whatever resource it
|
||||
|
|
@ -173,15 +190,27 @@ pub(crate) fn deploy_rebuild_nodes(agent: &str) -> Vec<NodeSpec> {
|
|||
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
||||
/// leaves it stopped). `relock = false` only for meta-update cascade
|
||||
/// children.
|
||||
///
|
||||
/// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots
|
||||
/// (`MetaSync`, `Prebuild`, `Reconcile`) — `Prebuild`'s roll-up carries the
|
||||
/// whole `StopForUpdate`→`Swap`→`PostSwap` subtree, so those three cover every
|
||||
/// node. Edging `Reconcile` alone would not do: it is `AfterAny` `Prebuild`, so
|
||||
/// it reaches `Done` even after a failed swap and the tail would report success.
|
||||
pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> DagSpec {
|
||||
let mut nodes = rebuild_nodes(agent, relock, 0);
|
||||
nodes.push(node(
|
||||
NodeKind::EmitRebuilt {
|
||||
agent: agent.to_owned(),
|
||||
},
|
||||
after_any_all(&[0, 1, 5]),
|
||||
));
|
||||
DagSpec {
|
||||
hook: Some(HookKind::EmitRebuilt),
|
||||
source,
|
||||
reason,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
transient: Some(TransientKind::Rebuilding),
|
||||
nodes: rebuild_nodes(agent, relock, 0),
|
||||
nodes,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -201,12 +230,18 @@ pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> Dag
|
|||
/// bookkeeping tail — rollback when a merge landed unfinalized, forge tag
|
||||
/// mirror, PR failure comment (see [`NodeKind::DeployTail`]).
|
||||
///
|
||||
/// - `ResolveApproval` (4, **root**, `AfterAny` `DeployWindow`): resolves the
|
||||
/// approval row. A root rather than another child, so it isn't inside the
|
||||
/// window's resource subtree — it runs once the window has released the meta
|
||||
/// window, lease and build slot. One edge suffices here: `DeployWindow` is the
|
||||
/// DAG's only other group-root, so its roll-up already *is* the whole
|
||||
/// pipeline's outcome.
|
||||
///
|
||||
/// The window still spans the container build, as it must: `prepare_deploy`
|
||||
/// leaves `flake.lock` staged-uncommitted for the build's whole duration.
|
||||
pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
||||
let a = || agent.to_owned();
|
||||
DagSpec {
|
||||
hook: Some(HookKind::ResolveApproval),
|
||||
source: Source::Approval,
|
||||
reason,
|
||||
approval_id: Some(approval_id),
|
||||
|
|
@ -224,6 +259,10 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
|
|||
when: DepWhen::AfterAny,
|
||||
}],
|
||||
),
|
||||
node(
|
||||
NodeKind::ResolveApproval { approval_id },
|
||||
after_any_all(&[0]),
|
||||
),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
|
@ -241,7 +280,6 @@ pub fn reconcile_only(
|
|||
transient: Option<TransientKind>,
|
||||
) -> DagSpec {
|
||||
DagSpec {
|
||||
hook: None,
|
||||
source,
|
||||
reason,
|
||||
approval_id: None,
|
||||
|
|
@ -264,10 +302,11 @@ pub fn reconcile_only(
|
|||
/// `Create` (child) owns the agent lease; `WriteDropin` + `Reconcile`
|
||||
/// (children of `Create`) borrow it. A failure cancel-cascades the rest —
|
||||
/// unlike rebuild there's no recovery-reconcile (nothing to converge if the
|
||||
/// container was never created).
|
||||
/// container was never created). Closed by a `ResolveApproval` tail root edged
|
||||
/// `AfterAny` onto `Provision` — the DAG's only other group-root, so its roll-up
|
||||
/// already carries the whole cascade.
|
||||
pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
hook: Some(HookKind::ResolveApproval),
|
||||
source: Source::Approval,
|
||||
reason,
|
||||
approval_id: Some(approval_id),
|
||||
|
|
@ -280,6 +319,10 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
|||
child(0, NodeKind::Create { agent: a() }, Vec::new()),
|
||||
child(1, NodeKind::WriteDropin { agent: a() }, Vec::new()),
|
||||
child(1, NodeKind::Reconcile { agent: a() }, after_ok(2)),
|
||||
node(
|
||||
NodeKind::ResolveApproval { approval_id },
|
||||
after_any_all(&[0]),
|
||||
),
|
||||
]
|
||||
},
|
||||
}
|
||||
|
|
@ -287,7 +330,9 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
|||
|
||||
/// Perm change: commit the JSON file(s), then the rebuild subgraph so
|
||||
/// the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes
|
||||
/// effect in the container.
|
||||
/// effect in the container. Group-roots are `WritePermFile`(0) plus the rebuild
|
||||
/// subgraph's `MetaSync`(1) / `Prebuild`(2) / `Reconcile`(6), so the
|
||||
/// `EmitRebuilt` tail edges all four.
|
||||
pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPayload) -> DagSpec {
|
||||
let mut nodes = vec![node(
|
||||
NodeKind::WritePermFile {
|
||||
|
|
@ -297,8 +342,13 @@ pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPay
|
|||
Vec::new(),
|
||||
)];
|
||||
nodes.extend(rebuild_nodes(agent, true, 1));
|
||||
nodes.push(node(
|
||||
NodeKind::EmitRebuilt {
|
||||
agent: agent.to_owned(),
|
||||
},
|
||||
after_any_all(&[0, 1, 2, 6]),
|
||||
));
|
||||
DagSpec {
|
||||
hook: Some(HookKind::EmitRebuilt),
|
||||
source,
|
||||
reason,
|
||||
approval_id: None,
|
||||
|
|
@ -324,22 +374,30 @@ pub fn meta_update(
|
|||
reason: String,
|
||||
approval_id: Option<i64>,
|
||||
) -> DagSpec {
|
||||
let mut nodes = vec![node(
|
||||
NodeKind::MetaLock {
|
||||
sweep: false,
|
||||
fanout: None,
|
||||
},
|
||||
Vec::new(),
|
||||
)];
|
||||
// The bump itself has no side effect, so an operator-driven one ends at the
|
||||
// `MetaLock`; an approval-driven one still has its row to resolve and gets a
|
||||
// tail edged onto that single group-root — whose roll-up covers the rebuild
|
||||
// subgraphs `MetaLock` grows into itself.
|
||||
if let Some(approval_id) = approval_id {
|
||||
nodes.push(node(
|
||||
NodeKind::ResolveApproval { approval_id },
|
||||
after_any_all(&[0]),
|
||||
));
|
||||
}
|
||||
DagSpec {
|
||||
// The bump itself has no side effect; an approval-driven one still has
|
||||
// its row to resolve.
|
||||
hook: approval_id.map(|_| HookKind::ResolveApproval),
|
||||
source,
|
||||
reason,
|
||||
approval_id,
|
||||
inputs,
|
||||
transient: Some(TransientKind::Rebuilding),
|
||||
nodes: vec![node(
|
||||
NodeKind::MetaLock {
|
||||
sweep: false,
|
||||
fanout: None,
|
||||
},
|
||||
Vec::new(),
|
||||
)],
|
||||
nodes,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -351,14 +409,13 @@ pub fn meta_update(
|
|||
/// (dashboard tree, `<parent>`/`<children>` sentinel routing, permission
|
||||
/// checks), so a parent move needs no container rebuild to take effect.
|
||||
/// No transient pill either — the node is agentless (no lease to hang one
|
||||
/// off of) and near-instant. No terminal hook: the write is the whole effect.
|
||||
/// off of) and near-instant. No tail node: the write is the whole effect.
|
||||
pub fn reparent(
|
||||
moves: Vec<(hive_types::Ident, Option<hive_types::Ident>)>,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> DagSpec {
|
||||
DagSpec {
|
||||
hook: None,
|
||||
source,
|
||||
reason,
|
||||
approval_id: None,
|
||||
|
|
|
|||
Loading…
Reference in a new issue