d1ee3d8cc5
The future `Scheduler::claim_next` hands back completed its node only
after `run`'s future returned. A panic while running (the workspace
unwinds) skipped that completion, so the node stayed `Running` and its
resource units stayed held until the process restarted; later nodes
needing the same resource never started.
The run future is now polled under `catch_unwind`. A panic completes
the node as `Outcome::Failed("runner panicked: <msg>")` with nothing
grown, through the same `complete_growing` path as an ordinary failure:
`AFTER_OK` dependents are skipped, pending children cascade, and the
node's units are released.
The node's future is boxed: wrapping it inline pushed the futures
swarm-controller and hive-c0re spawn over clippy's `large_futures`
limit.
Closes #5058
68 lines
3.6 KiB
Markdown
68 lines
3.6 KiB
Markdown
# hive-jobq
|
|
|
|
A job-DAG scheduler, extracted from hive-c0re's in-tree `job_queue`
|
|
as a **domain-agnostic** library. It schedules a single in-memory graph of
|
|
nodes over named resources; it knows nothing about containers, rebuilds, or any
|
|
hyperhive type — the node payload `N` and resource name `R` are both generic, so
|
|
the caller supplies its own domain.
|
|
|
|
## When to use it
|
|
|
|
Reach for this crate whenever you need to run a DAG of interdependent work items
|
|
under bounded, named concurrency — the hive-c0re rebuild/lifecycle queue is the
|
|
first consumer, but nothing here is specific to it. The caller defines the node
|
|
kinds, wires deps, and supplies a runner; the scheduler decides what can start.
|
|
|
|
## Model
|
|
|
|
**Runtime-only: nothing writes this graph to disk.** The serde impls exist for
|
|
the wire projection (`hive-jobq-wire`) and a possible future store; no caller
|
|
loads one, so ids and timestamps are stable within a run, not across restarts.
|
|
hive-c0re constructs an empty graph every boot and re-derives desired state with
|
|
its reconcile sweep.
|
|
|
|
One **shared graph** for the whole system, not a DAG per job. Enqueuing
|
|
inserts a self-contained sub-DAG and returns the ids of the nodes the job
|
|
_asked_ for, in the order it named them; the scheduler runs a continuous loop,
|
|
starting every node whose deps are satisfied:
|
|
|
|
- **Resource deps** are named counting semaphores over a caller-chosen type `R`
|
|
— e.g. `build-slot` (capacity N), `agent/<name>` (capacity 1), or any
|
|
unconfigured name (capacity 1, created on use). A node acquires _all_ its
|
|
resource deps atomically at start (all-or-nothing) — no hold-and-wait, so no
|
|
deadlock.
|
|
- **Node deps** wait on another node per `DepWhen`: `AfterOk` needs success (a
|
|
failed dep cancels the dependent), `AfterAny` only needs terminal.
|
|
|
|
A node carries two independent axes: its `Dep`s (ordering + resource needs) and
|
|
its `parent` (structural grouping). The **parent chain**, not the node edges, is
|
|
what the scheduler consults for resource re-entrancy: a resource unit is held
|
|
for the acquiring node _plus its whole parent subtree_, and a descendant needing
|
|
a resource an ancestor already holds re-uses that grant (a re-entrant borrow,
|
|
one branch at a time) rather than taking a fresh unit.
|
|
|
|
A `NodeId` is opaque, stable and monotonic **within a run** — a fresh process
|
|
mints ids from zero, so an id stored outside it is a historical record, not a
|
|
handle that will resolve later. The scheduler is
|
|
single-threaded — it owns the resource table and mutates it directly.
|
|
|
|
## Shape
|
|
|
|
- **`Graph<N, R>`** — the in-memory node store. `insert` mints ids and
|
|
validates dep/parent references; `set_state` is the single state-transition
|
|
choke point (and where each node's lifecycle timestamps —
|
|
`started_at` / `finished_at`, `DateTime<Utc>` — are stamped).
|
|
- **`Node<N, R>`** — `{ id, parent, payload, deps, state, started_at,
|
|
finished_at, error }`. All fields public; derives serde for the wire
|
|
projection (and so a store could be added — nothing calls one today).
|
|
- **`Scheduler<N, R>`** — drives the graph: `settle()` starts every ready node
|
|
(acquiring resources atomically), `complete(id, outcome)` reports a finished
|
|
node's result and rolls terminality up the parent chain, releasing grants once
|
|
a subtree is done. `Outcome::{Done, Failed(String)}` — the failure reason
|
|
rides `Failed` onto the node's `error`. A runner that panics completes its node as
|
|
`Failed` with the panic message, which releases its grants like any failure.
|
|
- **`ResourceTable<R>`** — per-name capacities; unconfigured names default to
|
|
capacity 1.
|
|
|
|
See the crate-root and `scheduler` module `//!` docs for the full borrow/release
|
|
model.
|