feat(#2772): branch on outcome in the graph, not inside the node
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
This commit is contained in:
parent
affedecaa5
commit
07078b76ef
8 changed files with 365 additions and 385 deletions
|
|
@ -20,8 +20,8 @@ use crate::coordinator::TransientKind;
|
|||
/// When a dependency edge is satisfied — re-exported from [`hive_jobq`] rather
|
||||
/// than mirrored here. It used to be a duplicate enum with a `to_crate_when`
|
||||
/// translation beside it; the copy bought nothing and had to be widened in
|
||||
/// lockstep every time the crate's edge model grew (#2772).
|
||||
pub use hive_jobq::DepWhen;
|
||||
/// lockstep every time the crate's edge model grew.
|
||||
pub use hive_jobq::{DepWhen, TerminalState};
|
||||
|
||||
/// A dependency edge (intra-DAG only — cross-DAG ordering comes from
|
||||
/// the per-agent lease + dedup, never from edges between DAGs).
|
||||
|
|
@ -233,9 +233,16 @@ pub enum NodeKind {
|
|||
/// carrying one here would be a second copy free to drift. Like
|
||||
/// [`NodeKind::MetaLock`] it reports `""` from [`NodeKind::agent`] and takes
|
||||
/// no lease — which is also what lets one close a multi-agent DAG.
|
||||
///
|
||||
/// [`Claim::deps`]: super::Claim::deps
|
||||
ResolveApproval { approval_id: i64 },
|
||||
ResolveApproval {
|
||||
approval_id: i64,
|
||||
/// Which outcome this node reports. A template emits **one per outcome**,
|
||||
/// each edged to accept only that one, so exactly one is ever runnable
|
||||
/// and the executor has nothing to decide — it resolves the row the way
|
||||
/// its own variant says. The `Cancelled` one is also the node that
|
||||
/// [`super::JobQueue::cancel`] spares, since its edge is the only one
|
||||
/// that accepts a dropped dependency.
|
||||
outcome: TerminalState,
|
||||
},
|
||||
/// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt` manager
|
||||
/// event — `ok` when its deps are `Done`, `!ok` carrying the failure note when
|
||||
/// they `Failed`, and **nothing at all** when they `Cancelled` (a cancelled DAG
|
||||
|
|
@ -243,8 +250,13 @@ pub enum NodeKind {
|
|||
///
|
||||
/// One node **per agent**, unlike the DAG-wide hook it replaces: a multi-agent
|
||||
/// DAG now reports each agent's own outcome instead of painting every agent with
|
||||
/// the whole DAG's roll-up.
|
||||
EmitRebuilt { agent: String },
|
||||
/// the whole DAG's roll-up. And one per *outcome* — `ok` isn't computed here,
|
||||
/// it's which of the pair the graph let run.
|
||||
///
|
||||
/// There is deliberately no cancel variant: a DAG dropped before it started
|
||||
/// has no rebuild to report, and neither tail's edge accepts `Cancelled`, so
|
||||
/// both are cancelled with the rest and nothing is emitted.
|
||||
EmitRebuilt { agent: String, ok: bool },
|
||||
/// Write the agent's durable power intent (`wanted = Up` when `up`, else
|
||||
/// `Offline`) as a first-class DAG node, at the head of a power-op
|
||||
/// template so the downstream `Reconcile` reads it. Replaces the old
|
||||
|
|
@ -333,7 +345,7 @@ impl NodeKind {
|
|||
| NodeKind::DeployApply { agent }
|
||||
| NodeKind::FinalizeDeploy { agent }
|
||||
| NodeKind::DeployTail { agent }
|
||||
| NodeKind::EmitRebuilt { agent }
|
||||
| NodeKind::EmitRebuilt { agent, .. }
|
||||
| NodeKind::SetWanted { agent, .. } => agent,
|
||||
NodeKind::MetaLock { .. }
|
||||
| NodeKind::Reparent { .. }
|
||||
|
|
@ -342,20 +354,6 @@ impl NodeKind {
|
|||
}
|
||||
}
|
||||
|
||||
/// Whether this is a DAG's **tail** — a node that reports how the rest of the
|
||||
/// DAG ended rather than doing work of its own.
|
||||
///
|
||||
/// The one place this matters is [`super::JobQueue::cancel`], which spares
|
||||
/// tails so they still run (and report `Cancelled`) on a cancelled DAG. Note
|
||||
/// [`NodeKind::DeployTail`] is *not* one: despite the name it does real
|
||||
/// compensating work, and on a cancelled DAG there is nothing to compensate.
|
||||
pub fn is_tail(&self) -> bool {
|
||||
matches!(
|
||||
self,
|
||||
NodeKind::ResolveApproval { .. } | NodeKind::EmitRebuilt { .. }
|
||||
)
|
||||
}
|
||||
|
||||
/// Nix-heavy kinds hold one of the `buildSlots` semaphore permits
|
||||
/// for the node's duration.
|
||||
pub fn needs_build_slot(&self) -> bool {
|
||||
|
|
|
|||
Loading…
Reference in a new issue