`Graph::insert` was public and validating, and the builder's insert loop
called it with `?`. That made job-level atomicity an accident: the loop
mutates as it goes, so a rejection at node `i` left `0..i` already in the
graph — the error fired loudly *after* the corruption, not instead of it.
It held only because `check_job_shape` happens to be exhaustive, with
nothing in the types saying so.
Make the guarantee structural instead. `Graph::insert` becomes
`pub(crate)`; the builder drains into a new infallible
`insert_unchecked` (via `Scheduler::append_unchecked`), so
`insert_with`'s sink returns a bare `NodeId` and a half-built job is no
longer expressible. Re-checking at the sink cannot add safety anyway — it
can only report after the mutation it was meant to prevent.
Drop `BuildError::Graph`: nothing in the builder path can produce a
`GraphError` any more. Clippy could not see this (an unreachable variant
of a `pub` enum is still constructible from outside the crate).
`Scheduler::append` stays public and validating — hive-c0re inserts a
DAG's container node through it. Folding that away means removing the
container/DagView indirection, which is out of scope here.
Three findings from the operator's review, all correct.
1. Two insert_job's. Graph::insert_job had no caller outside hive-jobq's
own tests -- production only ever went through Scheduler::insert_job.
It existed because the graph-level one got written first. Deleted; the
tests moved onto a Scheduler, which is where insertion belongs anyway.
2. insert_job was not atomic, and the previous commit made that worse: a
forward edge or forward parent surfaced mid-loop, leaving the nodes
before it in the graph, and resolve_wanted ran after every insert, so
an unknown handle failed once the whole job was already committed.
The module documented this under "Partial insertion" instead of fixing
it -- prose describing a hole is not a design.
All three are decidable from what the builder holds, so
check_declaration_order now runs before the first insert and the loop
indexes ids directly. A malformed job leaves the graph untouched.
What remains mid-insert is the graph's own rejection (out-of-group
dep, empty DepWhen); closing that needs a dry-run validate on Graph,
which is a separate change.
3. DagSpec no longer boxes its recipe: it is generic over the closure,
which travels from the template that built it straight into submit.
The box bought type inference, and paying for it costs annotations --
`|b: &Job|` at each declaration site (the field needs an HRTB, and an
unannotated closure binds one lifetime) and `+ use<>` on each
returning signature (or the opaque type captures the caller's borrows).
Erasure is still needed where several recipe shapes share one type:
the boxed Declare stays for the executor's append_subgraph, and a test
table uses an erase() helper.
The operator's instruction on the issue was "the closure returns an array
of guids, and enqueue_job returns the node ids in that order". What was
here instead returned a HashMap of everything inserted, and no caller used
the keys: submit dropped the return, insert_group did into_values(), and
the scheduler ignored what append_subgraph handed back. The guid-keyed
lookup was dead weight, and into_values() made that Vec arbitrarily
ordered -- harmless only because nothing read it.
insert_job now takes FnOnce(&JobBuilder) -> Vec<NodeGuid> and returns the
matching ids positionally. A handle from another job is UnknownNode rather
than a silent omission: the return is positional, so a short vector would
misalign every id after it.
c0re's Declare stays FnOnce(&Job) and the wrapper names no handles in one
place, rather than ending seven templates in an empty vector -- a DAG is
addressed by its container node, which submit inserts itself. That frees
insert_group from needing every id, so the node_rt pre-seeding goes too:
NodeRuntime is one Option field and every reader already tolerated a
missing entry (entry().or_default(), get().and_then(), iter().find()).
The tests are the argument for the shape: capturing a handle through a
mutable binding to look it up in the map afterwards collapses into
returning it and destructuring the result.
Two review findings from the operator, both about the builder being more
reachable than the design said.
**The builder must not leave the crate.** The module doc claimed "an
insertion API, not a spec factory — a builder is only ever handed to a
closure by the queue's insertion entry point", and then `new()` and
`insert_into` were public, so a caller could build one, carry it around
and insert it later. That is a spec factory with a builder's name on it.
`insert_job(root_parent, |b| …)` is now the whole API: the builder is
created inside the call, handed to the closure, and consumed there.
`new` / `insert_into` / `insert_with` are crate-private.
**A handle names the job that issued it.** `NodeGuid` was a per-builder
counter, so two jobs' first handles compared equal. `NodeRef` converts
into a bare `NodeGuid` — dropping the borrow that ties it to its
builder — so a handle carried into a second job (an inner closure
capturing an outer handle) would silently resolve to whatever that job's
first node happened to be. It is now `{ job, seq }` with a random `job`
half, so a foreign handle is a miss and the insert fails naming it. The
randomness comes from `RandomState`, which is collision-avoidance rather
than cryptography and needs no new dependency.
Tests moved onto the closure API rather than keeping their in-crate
access to the private constructor — a test that only passes because it
lives inside the crate is not testing the API a caller has. The two
forward-reference tests stopped asserting literal guid values (a random
half cannot be written down) and compare against the handles instead,
and a new test carries a handle between two jobs to pin the behaviour
that motivated the change.
`JobBuilder::node(payload)` hands back a `NodeRef` handle obtainable no
other way, and edges are handle -> handle. `insert_into(graph,
root_parent)` consumes the builder, inserts in declaration order, and
returns the id each handle was minted as. There is no intermediate
node-description type: the builder inserts through `Graph::insert`
directly, so nothing has to stay in sync with that signature.
`root_parent` is the group's attachment point — a node that declared no
parent hangs there. That is what makes a job a self-contained sub-DAG: a
template is written without knowing which container node it will live
under, and the same builder serves a runtime-emitted subgraph hanging
off its emitting node.
Generic over the same `N`/`R` as `Graph`, so it belongs to the library
rather than to any one caller's node kind. `.needs(name)` /
`.needs_units(name, count)` declare resource deps at the construction
site, next to the node that needs them.
`Scheduler::insert_job` is the same over a scheduler, via the shared
`insert_with` sink, so a caller building a job never reaches past the
scheduler at the graph underneath.
Declaration order is enforced rather than papered over: a node
referencing one declared later is a named `BuildError::ForwardEdge` /
`ForwardParent`. Sorting for the caller would silently accept a shape
the graph cannot express, and would put ordering logic in a second
place.
Additive — `Graph` is untouched.
mara on !2909: "shouldnt node_by_id be part of jobq?" — yes. Resolving a
raw value to a `NodeId` is the exact inverse of `NodeId::get`, which
already lives in hive-jobq, and it is only a search because the graph
owns the counter that makes ids unfabricable. Both halves of that
round-trip belong on the same side of the crate boundary.
Placing it in `QueueInner` also put it in a layer slated for removal, so
the c0re-side helper would have had to move later anyway — and it was
private there, leaving any other caller needing the same resolution to
write the same `nodes().find_map(…)` by hand.
`Graph::resolve_id` replaces it, with a unit test covering the
round-trip and the rejection of a value that was never an id.
While re-reading the diff for that question: the doc comment added in
the previous commit landed *between* `container`'s doc comment and its
signature, silently reattaching "The container node of `dag_id`" to the
new helper and leaving `container` undocumented. Restored.
No behaviour change and no wire change — same search, same call site.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-jobq -p hive-c0re` (41 + 322 passed) and `nix fmt`.
`hive_host_sock::jobs::State` was a hand-maintained copy of
`hive_jobq::State` — five variants spelled the same in both, kept in sync
by whoever remembered. Adding `Skipped` last week meant adding it twice.
The wire crate now re-exports the scheduler's enum and `to_wire_state` is
gone.
Two states that were hidden now reach clients. `to_wire_state` renamed
`Pending` to `Queued` and folded `Finishing` into `Running`, so the
dashboard could not distinguish a node waiting on its dependencies from
one whose own work is done while its sub-nodes still run. Both are now
visible, and consumers say which they mean.
Every consumer had to move with it, and only the Rust ones said so: the
exhaustive matches in `hivectl` and `DagView::rollup_state` failed to
compile, while the dashboard's fourteen string comparisons would have
gone quietly wrong — a `finishing` node no longer counting as running,
a `pending` node no longer as queued.
The frontend also builds CSS class names out of the state string
(`rqe-` + state, `rqe-node-` + state) and keys its glyph map on it, all
lowercase. Those go through a `stateSlug` helper now; comparisons use the
wire spelling, presentation lowercases. Without that split every queue
entry and node chip would have silently lost its styling.
Dropping the `State as JobState` alias in hive-c0re falls out of this:
the alias only existed to tell two `State` types apart, and there is one
now.
Returning `false` for an unknown id gave the same answer as a node that is
merely still running, so a caller polling a stale id would wait forever for
a state that can never arrive. `Option<bool>` makes the two cases separate,
matching `node()`'s convention that `None` means the id isn't in the graph.
hive-c0re hand-rolled four traversals over a graph it doesn't own, because
`Graph` exposed only `node()` and `nodes()`. They are generic — nothing in
them knows what a hyperhive DAG is — so they move to `hive-jobq` and core
delegates.
`Graph` gains `root_of`, `descendants`, `roots`, `is_settled` and
`first_error`; they reuse the private `is_descendant` the crate already had
for its dep-scope rule. `subtree` gets faster on the way: core walked every
node's whole parent chain to the root for every node in the graph, where
`is_descendant` stops as soon as it sees the ancestor.
`first_error` deliberately looks for the first `Failed` descendant that
*carries* an error rather than the first `Failed` one. A node that rolled
its failure up from a child holds no error of its own and sorts before that
child, so the simpler version reports `None` for the common case and the
dashboard loses the reason. The distinction has its own test.
`dag_is_terminal` is deleted rather than moved: it was already a plain
`state.is_terminal()` read, and its three call sites now ask the graph.
Review follow-ups on #2785.
`DepWhen` wraps `BitFlags<TerminalState>` instead of a hand-rolled `u8`,
so the bit manipulation belongs to the library and `TerminalState` gains
its flag value from `#[bitflags]` rather than a `bit()` match anyone
could get wrong. `of`/`accepts`/`is_empty` become one-liners over it.
Serialization is written out by hand rather than derived: clippy's
`unsafe_derive_deserialize` fires on deriving over a type with unsafe
internals, and the honest fix is to say what the wire form is. It is now
the list of accepted outcomes — `["done","failed"]` — which reads better
than a bitmask and survives the bits being renumbered.
Also drops the `pub use` re-export of `DepWhen` / `TerminalState` from
hive-c0re's `model`. It existed so that `use super::model::…` kept
compiling, which is a shim for one consumer's convenience; the sites
import from `hive_jobq` directly now.
And removes comments narrating what the code used to be. Git holds that.
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
`DepWhen` was two named cases, so every new combination wanted a new
variant. It is now a set over the terminal outcomes: a `u8` bitset
newtype, no dependency, with `AFTER_OK` / `AFTER_ANY` kept as the two
constants the templates actually use. "Run regardless" is all outcomes,
"anything that isn't a failure" is `{Done, Cancelled}`, a compensation
branch is `{Failed}` — closed under combination, so it never needs
another variant.
`TerminalState` is its own type rather than a subset of `State`, so an
edge cannot name `Pending` / `Running` / `Finishing`. Those are
meaningless in a dependency and are better unrepresentable than
validated against. The empty set is the one thing that can't be typed
away — nothing satisfies it, so `validate` rejects it next to the cycle
check.
Two consequences worth calling out:
- `cascade_cancel` collapses to one rule: a pending node is doomed once
any edge it names can no longer be satisfied. The hardcoded `AfterOk`
special case is gone, and a weak-edged node survives its dependency's
cancellation because of its own edge rather than by exemption.
- The cascade now runs on **any** terminal outcome, `Done` included.
With sets, success rules dependents out just as failure does — a
`{Failed}` branch is unsatisfiable the moment its dependency succeeds,
and leaving it `Pending` would wedge the subtree non-terminal forever.
That is a hang, not a wrong answer, so it is the load-bearing half of
this commit.
Edges are conjunctive, so "any of these N failed" is not directly
sayable. The composition that works is in the tests: the success branch
depends `AFTER_OK` on every root, so it is itself cancelled the moment
one of them doesn't succeed, and the failure branch hangs off *that*
with `{Cancelled}`. Exactly one of the two runs.
Also deletes hive-c0re's duplicate `DepWhen` enum and the
`to_crate_when` translation beside it. The copy bought nothing and had
to be widened in lockstep with the crate's edge model — it is the
in-between layer #2772 exists to remove, and it is what broke the build
when the crate's spelling changed.
All 34 jobq tests pass, including the four new ones covering both
directions of a failure-only branch, weak-edge survival of a cancelled
dependency, and the aggregator composition.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Rework the crate's scheduling model onto an explicit parent (grouping)
axis, separate from the dep (ordering) axis.
- Node gains a structural `parent: Option<NodeId>`, set by the caller
independent of its `Dep::Node` edges. Grouping is not ordering. A
`Dep::Node` edge must stay inside the depender's own parent group
(validated) — never crossing to another group or onto the parent.
- Resource holding walks the parent tree: acquire fresh when no ancestor
holds it (the acquirer owns it, held for its whole subtree); borrow an
ancestor's grant (one branch at a time; nodes inside are covered); take
an extra unit when the grant is lent to a sibling branch, else wait. A
grant releases only once the owner and its whole subtree are terminal.
- Completion rolls up the parent tree: a node's sub-nodes run after its
own logic, and it is not terminal until they finish — it parks in
`State::Finishing`, rolling up to Done (every child Done) or Failed
(any child Failed/Cancelled). A child is gated on its parent reaching
Finishing; a downstream dep on a node therefore waits for that node's
dynamically-appended children with no explicit edge. A failed node
cancels its pending sub-nodes.
Deletes the SharedResources/ResourceGuard layer (guard.rs) and the
add_dep graph-growth hook (no longer needed). The scheduler stays
single-threaded, owning the ResourceTable directly. Early release of a
grant once no subtree node still needs it is a deferred optimization
(unsafe under dynamically-appended subnodes, #2611).
Base for the hive-c0re job_queue port (#2605), split out so that PR can
rebase onto it.
Make the resource lock unmisusable from outside the crate: the public
surface is now purely declarative (build a Graph with Dep::Resource edges,
configure capacities, run the Scheduler), and the scheduler owns every
acquire/release — a consumer never holds a guard, so it cannot hold the
lock wrong.
- `guard` module + `ResourceTable::try_acquire_all`/`release_all` +
`Graph::set_state` are now `pub(crate)`.
- Remove the dead borrowed-guard layer (`ResourceGuard::borrowed`,
`Acq::Borrowed`, `is_owning`): the scheduler tracks re-entrancy via its
own single borrow slot per (holder, resource) and never constructs a
borrowed guard, so re-entrancy lives in exactly one place. `Acq`
collapses into the owning `ResourceGuard` struct.
- `#[must_use]` on `Scheduler::settle` — ignoring its ids silently drops
runnable work.
- `SharedResources::with` (test-only table observability) is `#[cfg(test)]`.
- Drop the moot borrowed-guard tests; retained owning tests are black-box,
and the redundant `set_state` test helper is gone.
Replace the concrete ResourceName(String) with a type parameter
R: Clone + Eq + Hash threaded end-to-end (Dep<R>, Node<N,R>, Graph<N,R>,
ResourceTable<R>, ResourceGuard<R>/SharedResources<R>, Scheduler<N,R>).
The crate no longer hard-codes the resource identity; the consumer picks
the concrete type (a String, or an enum like BuildSlot/Agent(name)) at
the port. Tests use String as the concrete R. Pure type-parameter
thread-through, no logic change. 25 tests green, clippy pedantic clean.
Named counting-semaphore resources — the Dep::Resource side of the v2
model. A ResourceTable tracks per-name capacity + held counts; unconfigured
names default to capacity 1 (created lazily). try_acquire_all grants every
requested unit or none, leaving the table untouched on failure — so a node
never holds one resource while waiting for another, which is what makes the
scheduler deadlock-free without cycle detection. Duplicate names in a
request are summed; over-capacity requests can never acquire. release_all
saturates rather than underflowing. Runtime scheduler state, not persisted:
held counts are rederived from running nodes on restart. Guard objects
(RAII release, recursive re-entrancy) wrap this in a follow-up.
Per mara's direction — validate ids as they enter the graph so internal
iteration can trust every id the graph holds; the generational route for
removal comes later. Adds GraphError; insert() now rejects a dangling
Dep::Node / parent id (it is fallible); validate() checks all internal id
references resolve and that next_id is past the largest existing id;
deserialization runs validate() via #[serde(try_from = "GraphData<N>")], so
a loaded graph can never carry a dangling reference. 5 new tests; serde_json
added as a dev-dependency for the round-trip cases.
argus flagged that `NodeId(pub u64)` contradicted the "opaque" doc — a pub
inner field lets callers fabricate `NodeId(42)`. Make the field `pub(crate)`
so an id can only originate from the graph's monotonic counter or serde
deserialization, never a caller. Tests construct ids in-crate (unaffected);
the derived Serialize/Deserialize round-trips fine. Doc keeps "opaque" — now
accurate — with a line explaining the enforcement.
group_terminal checked only that every child was terminal, never the group
node's own state — so an empty group whose node is still Running returned
true (empty .all()), making a running node that has yet to append its
subgraph look already-finished. Now it requires the group node itself
terminal AND every child recursively terminal. Deciding when to settle a
group node to terminal once its children are done stays a scheduler concern;
this answers the dependents' question — is the whole group, node included,
finished. Adds group-node-pending-with-child-done + empty-running-group tests.
First step of extracting the job-DAG queue into a domain-agnostic
`hive-jobq` library, per the operator's v2 design: one persistent
graph, named-counter resources, recursive node groups, opaque stable
node ids, guard-object locks, a slot-filling scheduler.
This commit lands only the data model, so the shape can be reviewed
before the machinery is built on it:
- NodeId: opaque, stable, monotonic; group membership is a parent
edge, not encoded in the id (the 1/1/2 hierarchy is a derived UI
label).
- ResourceName, Dep (Node | Resource{name,count}), State.
- Node<N>: caller-defined payload N so the library stays
container-agnostic.
- Graph<N>: insert (mints stable ids), node lookup, children,
recursive group-terminal check. Retains completed groups (no
pruning in v1).
The resource-acquisition machinery (atomic all-or-nothing acquire),
the recursive-lock guards, and the scheduler loop are follow-ups.
Tests cover id minting, group terminality, and state terminality;
clippy + rustdoc clean.
