`Scheduler::append_unchecked` was a one-line pass-through to
`Graph::insert_unchecked` with a single caller. `insert_job` is already a
method on `Scheduler`, so it can borrow `self.graph` and call the graph
directly — the wrapper bought nothing but a name.
Also fixes `insert_job`'s doc, which still claimed every node goes
through `Scheduler::append`; it goes straight to the graph's unchecked
insert, and the reason belongs in that doc rather than on a wrapper.
`Graph::insert` was public and validating, and the builder's insert loop
called it with `?`. That made job-level atomicity an accident: the loop
mutates as it goes, so a rejection at node `i` left `0..i` already in the
graph — the error fired loudly *after* the corruption, not instead of it.
It held only because `check_job_shape` happens to be exhaustive, with
nothing in the types saying so.
Make the guarantee structural instead. `Graph::insert` becomes
`pub(crate)`; the builder drains into a new infallible
`insert_unchecked` (via `Scheduler::append_unchecked`), so
`insert_with`'s sink returns a bare `NodeId` and a half-built job is no
longer expressible. Re-checking at the sink cannot add safety anyway — it
can only report after the mutation it was meant to prevent.
Drop `BuildError::Graph`: nothing in the builder path can produce a
`GraphError` any more. Clippy could not see this (an unreachable variant
of a `pub` enum is still constructible from outside the crate).
`Scheduler::append` stays public and validating — hive-c0re inserts a
DAG's container node through it. Folding that away means removing the
container/DagView indirection, which is out of scope here.
Three findings from the operator's review, all correct.
1. Two insert_job's. Graph::insert_job had no caller outside hive-jobq's
own tests -- production only ever went through Scheduler::insert_job.
It existed because the graph-level one got written first. Deleted; the
tests moved onto a Scheduler, which is where insertion belongs anyway.
2. insert_job was not atomic, and the previous commit made that worse: a
forward edge or forward parent surfaced mid-loop, leaving the nodes
before it in the graph, and resolve_wanted ran after every insert, so
an unknown handle failed once the whole job was already committed.
The module documented this under "Partial insertion" instead of fixing
it -- prose describing a hole is not a design.
All three are decidable from what the builder holds, so
check_declaration_order now runs before the first insert and the loop
indexes ids directly. A malformed job leaves the graph untouched.
What remains mid-insert is the graph's own rejection (out-of-group
dep, empty DepWhen); closing that needs a dry-run validate on Graph,
which is a separate change.
3. DagSpec no longer boxes its recipe: it is generic over the closure,
which travels from the template that built it straight into submit.
The box bought type inference, and paying for it costs annotations --
`|b: &Job|` at each declaration site (the field needs an HRTB, and an
unannotated closure binds one lifetime) and `+ use<>` on each
returning signature (or the opaque type captures the caller's borrows).
Erasure is still needed where several recipe shapes share one type:
the boxed Declare stays for the executor's append_subgraph, and a test
table uses an erase() helper.
The operator's instruction on the issue was "the closure returns an array
of guids, and enqueue_job returns the node ids in that order". What was
here instead returned a HashMap of everything inserted, and no caller used
the keys: submit dropped the return, insert_group did into_values(), and
the scheduler ignored what append_subgraph handed back. The guid-keyed
lookup was dead weight, and into_values() made that Vec arbitrarily
ordered -- harmless only because nothing read it.
insert_job now takes FnOnce(&JobBuilder) -> Vec<NodeGuid> and returns the
matching ids positionally. A handle from another job is UnknownNode rather
than a silent omission: the return is positional, so a short vector would
misalign every id after it.
c0re's Declare stays FnOnce(&Job) and the wrapper names no handles in one
place, rather than ending seven templates in an empty vector -- a DAG is
addressed by its container node, which submit inserts itself. That frees
insert_group from needing every id, so the node_rt pre-seeding goes too:
NodeRuntime is one Option field and every reader already tolerated a
missing entry (entry().or_default(), get().and_then(), iter().find()).
The tests are the argument for the shape: capturing a handle through a
mutable binding to look it up in the map afterwards collapses into
returning it and destructuring the result.
Two review findings from the operator, both about the builder being more
reachable than the design said.
**The builder must not leave the crate.** The module doc claimed "an
insertion API, not a spec factory — a builder is only ever handed to a
closure by the queue's insertion entry point", and then `new()` and
`insert_into` were public, so a caller could build one, carry it around
and insert it later. That is a spec factory with a builder's name on it.
`insert_job(root_parent, |b| …)` is now the whole API: the builder is
created inside the call, handed to the closure, and consumed there.
`new` / `insert_into` / `insert_with` are crate-private.
**A handle names the job that issued it.** `NodeGuid` was a per-builder
counter, so two jobs' first handles compared equal. `NodeRef` converts
into a bare `NodeGuid` — dropping the borrow that ties it to its
builder — so a handle carried into a second job (an inner closure
capturing an outer handle) would silently resolve to whatever that job's
first node happened to be. It is now `{ job, seq }` with a random `job`
half, so a foreign handle is a miss and the insert fails naming it. The
randomness comes from `RandomState`, which is collision-avoidance rather
than cryptography and needs no new dependency.
Tests moved onto the closure API rather than keeping their in-crate
access to the private constructor — a test that only passes because it
lives inside the crate is not testing the API a caller has. The two
forward-reference tests stopped asserting literal guid values (a random
half cannot be written down) and compare against the handles instead,
and a new test carries a handle between two jobs to pin the behaviour
that motivated the change.
`JobBuilder::node(payload)` hands back a `NodeRef` handle obtainable no
other way, and edges are handle -> handle. `insert_into(graph,
root_parent)` consumes the builder, inserts in declaration order, and
returns the id each handle was minted as. There is no intermediate
node-description type: the builder inserts through `Graph::insert`
directly, so nothing has to stay in sync with that signature.
`root_parent` is the group's attachment point — a node that declared no
parent hangs there. That is what makes a job a self-contained sub-DAG: a
template is written without knowing which container node it will live
under, and the same builder serves a runtime-emitted subgraph hanging
off its emitting node.
Generic over the same `N`/`R` as `Graph`, so it belongs to the library
rather than to any one caller's node kind. `.needs(name)` /
`.needs_units(name, count)` declare resource deps at the construction
site, next to the node that needs them.
`Scheduler::insert_job` is the same over a scheduler, via the shared
`insert_with` sink, so a caller building a job never reaches past the
scheduler at the graph underneath.
Declaration order is enforced rather than papered over: a node
referencing one declared later is a named `BuildError::ForwardEdge` /
`ForwardParent`. Sorting for the caller would silently accept a shape
the graph cannot express, and would put ordering logic in a second
place.
Additive — `Graph` is untouched.
The roll-up treated a cancelled child the same as a failed one, so a DAG
the operator cancelled before it started reported `Failed` — it claimed to
have failed at something when nothing under it ever ran.
`Failed` still outranks `Cancelled`: a group where one step broke and the
rest were dropped in response is a failure, and that is the fact worth
surfacing. Only a group with no failed child at all reports the cancel.
This also settles a disagreement. `DagView::rollup_state` on the wire has
always ranked failed over cancelled over the rest; the graph's own roll-up
had no `Cancelled` outcome to rank, so the two described the same DAG
differently depending on which one you asked.
`JobQueue::cancel` decided whether a DAG could be cancelled by reading node
run-state, walked the subtree, judged per node whether that node had asked
to observe cancellation, and re-ran the container's roll-up. Every one of
those is a fact the scheduler owns; core was reaching across the boundary
to compute them.
`Scheduler::cancel_node` now takes the whole subtree: cancelling a node
cancels the work under it, since a group is abandoned by abandoning its
root. The existing method generalises rather than gaining a sibling — it
had one production caller, which this replaces.
The gate runs over the work *under* the node, not the node itself: a group
root's state is its subtree's roll-up rather than a step that ran, so a
container is `Finishing` and never `Pending`, and gating on it would refuse
every cancel. A node with no children is its own work, which keeps the
previous single-node behaviour.
`observes_cancellation` moves in with it — it reads a node's declared edges
and knows nothing about what the payload means.
Core keeps the one genuinely domain-specific step, resolving a wire
`dag_id` to its container node, and is three lines otherwise.
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
`DepWhen` was two named cases, so every new combination wanted a new
variant. It is now a set over the terminal outcomes: a `u8` bitset
newtype, no dependency, with `AFTER_OK` / `AFTER_ANY` kept as the two
constants the templates actually use. "Run regardless" is all outcomes,
"anything that isn't a failure" is `{Done, Cancelled}`, a compensation
branch is `{Failed}` — closed under combination, so it never needs
another variant.
`TerminalState` is its own type rather than a subset of `State`, so an
edge cannot name `Pending` / `Running` / `Finishing`. Those are
meaningless in a dependency and are better unrepresentable than
validated against. The empty set is the one thing that can't be typed
away — nothing satisfies it, so `validate` rejects it next to the cycle
check.
Two consequences worth calling out:
- `cascade_cancel` collapses to one rule: a pending node is doomed once
any edge it names can no longer be satisfied. The hardcoded `AfterOk`
special case is gone, and a weak-edged node survives its dependency's
cancellation because of its own edge rather than by exemption.
- The cascade now runs on **any** terminal outcome, `Done` included.
With sets, success rules dependents out just as failure does — a
`{Failed}` branch is unsatisfiable the moment its dependency succeeds,
and leaving it `Pending` would wedge the subtree non-terminal forever.
That is a hang, not a wrong answer, so it is the load-bearing half of
this commit.
Edges are conjunctive, so "any of these N failed" is not directly
sayable. The composition that works is in the tests: the success branch
depends `AFTER_OK` on every root, so it is itself cancelled the moment
one of them doesn't succeed, and the failure branch hangs off *that*
with `{Cancelled}`. Exactly one of the two runs.
Also deletes hive-c0re's duplicate `DepWhen` enum and the
`to_crate_when` translation beside it. The copy bought nothing and had
to be widened in lockstep with the crate's edge model — it is the
in-between layer #2772 exists to remove, and it is what broke the build
when the crate's spelling changed.
All 34 jobq tests pass, including the four new ones covering both
directions of a failure-only branch, weak-edge survival of a cancelled
dependency, and the aggregator composition.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Rework the crate's scheduling model onto an explicit parent (grouping)
axis, separate from the dep (ordering) axis.
- Node gains a structural `parent: Option<NodeId>`, set by the caller
independent of its `Dep::Node` edges. Grouping is not ordering. A
`Dep::Node` edge must stay inside the depender's own parent group
(validated) — never crossing to another group or onto the parent.
- Resource holding walks the parent tree: acquire fresh when no ancestor
holds it (the acquirer owns it, held for its whole subtree); borrow an
ancestor's grant (one branch at a time; nodes inside are covered); take
an extra unit when the grant is lent to a sibling branch, else wait. A
grant releases only once the owner and its whole subtree are terminal.
- Completion rolls up the parent tree: a node's sub-nodes run after its
own logic, and it is not terminal until they finish — it parks in
`State::Finishing`, rolling up to Done (every child Done) or Failed
(any child Failed/Cancelled). A child is gated on its parent reaching
Finishing; a downstream dep on a node therefore waits for that node's
dynamically-appended children with no explicit edge. A failed node
cancels its pending sub-nodes.
Deletes the SharedResources/ResourceGuard layer (guard.rs) and the
add_dep graph-growth hook (no longer needed). The scheduler stays
single-threaded, owning the ResourceTable directly. Early release of a
grant once no subtree node still needs it is a deferred optimization
(unsafe under dynamically-appended subnodes, #2611).
Base for the hive-c0re job_queue port (#2605), split out so that PR can
rebase onto it.
When every resource dep of a node re-enters an ancestor's lock, `owned_reqs`
is empty; the old code still called `acquire(vec![])` and stored a no-op
empty guard in `owned`. Gate the acquire + guard insertion on
`!owned_reqs.is_empty()` — one fewer `borrow_mut` + `HashMap` entry per
fully-re-entrant node in the settle loop. `node_owns` already treats a
missing `owned` entry as non-owning, so behaviour is unchanged.
Make the resource lock unmisusable from outside the crate: the public
surface is now purely declarative (build a Graph with Dep::Resource edges,
configure capacities, run the Scheduler), and the scheduler owns every
acquire/release — a consumer never holds a guard, so it cannot hold the
lock wrong.
- `guard` module + `ResourceTable::try_acquire_all`/`release_all` +
`Graph::set_state` are now `pub(crate)`.
- Remove the dead borrowed-guard layer (`ResourceGuard::borrowed`,
`Acq::Borrowed`, `is_owning`): the scheduler tracks re-entrancy via its
own single borrow slot per (holder, resource) and never constructs a
borrowed guard, so re-entrancy lives in exactly one place. `Acq`
collapses into the owning `ResourceGuard` struct.
- `#[must_use]` on `Scheduler::settle` — ignoring its ids silently drops
runnable work.
- `SharedResources::with` (test-only table observability) is `#[cfg(test)]`.
- Drop the moot borrowed-guard tests; retained owning tests are black-box,
and the redundant `set_state` test helper is gone.
Replace the concrete ResourceName(String) with a type parameter
R: Clone + Eq + Hash threaded end-to-end (Dep<R>, Node<N,R>, Graph<N,R>,
ResourceTable<R>, ResourceGuard<R>/SharedResources<R>, Scheduler<N,R>).
The crate no longer hard-codes the resource identity; the consumer picks
the concrete type (a String, or an enum like BuildSlot/Agent(name)) at
the port. Tests use String as the concrete R. Pure type-parameter
thread-through, no logic change. 25 tests green, clippy pedantic clean.
