The roll-up treated a cancelled child the same as a failed one, so a DAG
the operator cancelled before it started reported `Failed` — it claimed to
have failed at something when nothing under it ever ran.
`Failed` still outranks `Cancelled`: a group where one step broke and the
rest were dropped in response is a failure, and that is the fact worth
surfacing. Only a group with no failed child at all reports the cancel.
This also settles a disagreement. `DagView::rollup_state` on the wire has
always ranked failed over cancelled over the rest; the graph's own roll-up
had no `Cancelled` outcome to rank, so the two described the same DAG
differently depending on which one you asked.
`JobQueue::cancel` decided whether a DAG could be cancelled by reading node
run-state, walked the subtree, judged per node whether that node had asked
to observe cancellation, and re-ran the container's roll-up. Every one of
those is a fact the scheduler owns; core was reaching across the boundary
to compute them.
`Scheduler::cancel_node` now takes the whole subtree: cancelling a node
cancels the work under it, since a group is abandoned by abandoning its
root. The existing method generalises rather than gaining a sibling — it
had one production caller, which this replaces.
The gate runs over the work *under* the node, not the node itself: a group
root's state is its subtree's roll-up rather than a step that ran, so a
container is `Finishing` and never `Pending`, and gating on it would refuse
every cancel. A node with no children is its own work, which keeps the
previous single-node behaviour.
`observes_cancellation` moves in with it — it reads a node's declared edges
and knows nothing about what the payload means.
Core keeps the one genuinely domain-specific step, resolving a wire
`dag_id` to its container node, and is three lines otherwise.
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
`DepWhen` was two named cases, so every new combination wanted a new
variant. It is now a set over the terminal outcomes: a `u8` bitset
newtype, no dependency, with `AFTER_OK` / `AFTER_ANY` kept as the two
constants the templates actually use. "Run regardless" is all outcomes,
"anything that isn't a failure" is `{Done, Cancelled}`, a compensation
branch is `{Failed}` — closed under combination, so it never needs
another variant.
`TerminalState` is its own type rather than a subset of `State`, so an
edge cannot name `Pending` / `Running` / `Finishing`. Those are
meaningless in a dependency and are better unrepresentable than
validated against. The empty set is the one thing that can't be typed
away — nothing satisfies it, so `validate` rejects it next to the cycle
check.
Two consequences worth calling out:
- `cascade_cancel` collapses to one rule: a pending node is doomed once
any edge it names can no longer be satisfied. The hardcoded `AfterOk`
special case is gone, and a weak-edged node survives its dependency's
cancellation because of its own edge rather than by exemption.
- The cascade now runs on **any** terminal outcome, `Done` included.
With sets, success rules dependents out just as failure does — a
`{Failed}` branch is unsatisfiable the moment its dependency succeeds,
and leaving it `Pending` would wedge the subtree non-terminal forever.
That is a hang, not a wrong answer, so it is the load-bearing half of
this commit.
Edges are conjunctive, so "any of these N failed" is not directly
sayable. The composition that works is in the tests: the success branch
depends `AFTER_OK` on every root, so it is itself cancelled the moment
one of them doesn't succeed, and the failure branch hangs off *that*
with `{Cancelled}`. Exactly one of the two runs.
Also deletes hive-c0re's duplicate `DepWhen` enum and the
`to_crate_when` translation beside it. The copy bought nothing and had
to be widened in lockstep with the crate's edge model — it is the
in-between layer #2772 exists to remove, and it is what broke the build
when the crate's spelling changed.
All 34 jobq tests pass, including the four new ones covering both
directions of a failure-only branch, weak-edge survival of a cancelled
dependency, and the aggregator composition.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Rework the crate's scheduling model onto an explicit parent (grouping)
axis, separate from the dep (ordering) axis.
- Node gains a structural `parent: Option<NodeId>`, set by the caller
independent of its `Dep::Node` edges. Grouping is not ordering. A
`Dep::Node` edge must stay inside the depender's own parent group
(validated) — never crossing to another group or onto the parent.
- Resource holding walks the parent tree: acquire fresh when no ancestor
holds it (the acquirer owns it, held for its whole subtree); borrow an
ancestor's grant (one branch at a time; nodes inside are covered); take
an extra unit when the grant is lent to a sibling branch, else wait. A
grant releases only once the owner and its whole subtree are terminal.
- Completion rolls up the parent tree: a node's sub-nodes run after its
own logic, and it is not terminal until they finish — it parks in
`State::Finishing`, rolling up to Done (every child Done) or Failed
(any child Failed/Cancelled). A child is gated on its parent reaching
Finishing; a downstream dep on a node therefore waits for that node's
dynamically-appended children with no explicit edge. A failed node
cancels its pending sub-nodes.
Deletes the SharedResources/ResourceGuard layer (guard.rs) and the
add_dep graph-growth hook (no longer needed). The scheduler stays
single-threaded, owning the ResourceTable directly. Early release of a
grant once no subtree node still needs it is a deferred optimization
(unsafe under dynamically-appended subnodes, #2611).
Base for the hive-c0re job_queue port (#2605), split out so that PR can
rebase onto it.
When every resource dep of a node re-enters an ancestor's lock, `owned_reqs`
is empty; the old code still called `acquire(vec![])` and stored a no-op
empty guard in `owned`. Gate the acquire + guard insertion on
`!owned_reqs.is_empty()` — one fewer `borrow_mut` + `HashMap` entry per
fully-re-entrant node in the settle loop. `node_owns` already treats a
missing `owned` entry as non-owning, so behaviour is unchanged.
Make the resource lock unmisusable from outside the crate: the public
surface is now purely declarative (build a Graph with Dep::Resource edges,
configure capacities, run the Scheduler), and the scheduler owns every
acquire/release — a consumer never holds a guard, so it cannot hold the
lock wrong.
- `guard` module + `ResourceTable::try_acquire_all`/`release_all` +
`Graph::set_state` are now `pub(crate)`.
- Remove the dead borrowed-guard layer (`ResourceGuard::borrowed`,
`Acq::Borrowed`, `is_owning`): the scheduler tracks re-entrancy via its
own single borrow slot per (holder, resource) and never constructs a
borrowed guard, so re-entrancy lives in exactly one place. `Acq`
collapses into the owning `ResourceGuard` struct.
- `#[must_use]` on `Scheduler::settle` — ignoring its ids silently drops
runnable work.
- `SharedResources::with` (test-only table observability) is `#[cfg(test)]`.
- Drop the moot borrowed-guard tests; retained owning tests are black-box,
and the redundant `set_state` test helper is gone.
Replace the concrete ResourceName(String) with a type parameter
R: Clone + Eq + Hash threaded end-to-end (Dep<R>, Node<N,R>, Graph<N,R>,
ResourceTable<R>, ResourceGuard<R>/SharedResources<R>, Scheduler<N,R>).
The crate no longer hard-codes the resource identity; the consumer picks
the concrete type (a String, or an enum like BuildSlot/Agent(name)) at
the port. Tests use String as the concrete R. Pure type-parameter
thread-through, no logic change. 25 tests green, clippy pedantic clean.
