Watch

History

Repository files (latest commit first)
Filename	Latest commit message	Latest commit date
atlas fe0906c043 jobq-wire: name every outcome, keep the enum spellings, document the schema Three things, all from review: Accepted outcomes were built from a hand-listed [Done, Failed, Cancelled, Skipped] array. Exhaustive today, silently short the day someone adds a variant — the new outcome would vanish from every edge that accepts it. BitFlags::ALL asks the type instead. TerminalState carried rename_all = "snake_case" while its sibling State did not, so one enum shipped "done" and the other "Done". A rename is a second spelling of a name that then has to be kept in agreement by hand; both now serialise their variant names verbatim. Nothing else reads TerminalState off a wire, so no consumer moves. GraphDep's tag values likewise. The endpoint documented its body as serde_json::Value, which tells a spec reader nothing. hive-jobq-wire now derives ToSchema. State and TerminalState are foreign types here and utoipa stays out of the scheduler crate, so the schema points at local mirror enums. A mirror that drifts is worse than none: the conversions are exhaustive (a new upstream variant fails the build) and a test asserts each documented name equals the serialised one, since an exhaustive match still compiles when only the spellings diverge.		2026-08-03 00:39:14 +02:00
..
src	jobq-wire: name every outcome, keep the enum spellings, document the schema	2026-08-03 00:39:14 +02:00
Cargo.toml	jobq: make NodeGuid an actual guid	2026-08-02 15:32:05 +02:00
README.md	jobq: a job asks for the ids it wants back	2026-08-02 15:32:05 +02:00

README.md

hive-jobq

A persistent job-DAG scheduler, extracted from hive-c0re's in-tree job_queue as a domain-agnostic library. It schedules a single persistent graph of nodes over named resources; it knows nothing about containers, rebuilds, or any hyperhive type — the node payload N and resource name R are both generic, so the caller supplies its own domain.

When to use it

Reach for this crate whenever you need to run a DAG of interdependent work items under bounded, named concurrency — the hive-c0re rebuild/lifecycle queue is the first consumer, but nothing here is specific to it. The caller defines the node kinds, wires deps, and supplies a runner; the scheduler decides what can start.

Model

One persistent graph for the whole system, not a DAG per job. Enqueuing inserts a self-contained sub-DAG and returns the ids of the nodes the job asked for, in the order it named them; the scheduler runs a continuous loop, starting every node whose deps are satisfied:

Resource deps are named counting semaphores over a caller-chosen type R — e.g. build-slot (capacity N), agent/<name> (capacity 1), or any unconfigured name (capacity 1, created on use). A node acquires all its resource deps atomically at start (all-or-nothing) — no hold-and-wait, so no deadlock.
Node deps wait on another node per DepWhen: AfterOk needs success (a failed dep cancels the dependent), AfterAny only needs terminal.

A node carries two independent axes: its Deps (ordering + resource needs) and its parent (structural grouping). The parent chain, not the node edges, is what the scheduler consults for resource re-entrancy: a resource unit is held for the acquiring node plus its whole parent subtree, and a descendant needing a resource an ancestor already holds re-uses that grant (a re-entrant borrow, one branch at a time) rather than taking a fresh unit.

A NodeId is opaque, stable, and monotonic (safe to persist). The scheduler is single-threaded — it owns the resource table and mutates it directly.

Shape

Graph<N, R> — the persistent node store. insert mints ids and validates dep/parent references; set_state is the single state-transition choke point (and where each node's lifecycle timestamps — started_at / finished_at, DateTime<Utc> — are stamped).
Node<N, R> — { id, parent, payload, deps, state, started_at, finished_at, error }. All fields public; derives serde for persistence + the wire.
Scheduler<N, R> — drives the graph: settle() starts every ready node (acquiring resources atomically), complete(id, outcome) reports a finished node's result and rolls terminality up the parent chain, releasing grants once a subtree is done. Outcome::{Done, Failed(String)} — the failure reason rides Failed onto the node's error.
ResourceTable<R> — per-name capacities; unconfigured names default to capacity 1.

See the crate-root and scheduler module //! docs for the full borrow/release model.