vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

feat(#2591): hive-jobq parent-axis grouping + borrow + roll-up scheduler

Browse source 
Rework the crate's scheduling model onto an explicit parent (grouping)
axis, separate from the dep (ordering) axis.

- Node gains a structural `parent: Option<NodeId>`, set by the caller
  independent of its `Dep::Node` edges. Grouping is not ordering. A
  `Dep::Node` edge must stay inside the depender's own parent group
  (validated) — never crossing to another group or onto the parent.
- Resource holding walks the parent tree: acquire fresh when no ancestor
  holds it (the acquirer owns it, held for its whole subtree); borrow an
  ancestor's grant (one branch at a time; nodes inside are covered); take
  an extra unit when the grant is lent to a sibling branch, else wait. A
  grant releases only once the owner and its whole subtree are terminal.
- Completion rolls up the parent tree: a node's sub-nodes run after its
  own logic, and it is not terminal until they finish — it parks in
  `State::Finishing`, rolling up to Done (every child Done) or Failed
  (any child Failed/Cancelled). A child is gated on its parent reaching
  Finishing; a downstream dep on a node therefore waits for that node's
  dynamically-appended children with no explicit edge. A failed node
  cancels its pending sub-nodes.

Deletes the SharedResources/ResourceGuard layer (guard.rs) and the
add_dep graph-growth hook (no longer needed). The scheduler stays
single-threaded, owning the ResourceTable directly. Early release of a
grant once no subtree node still needs it is a deferred optimization
(unsafe under dynamically-appended subnodes, #2611).

Base for the hive-c0re job_queue port (#2605), split out so that PR can
rebase onto it.
This commit is contained in:
atlas 2026-07-20 19:45:53 +02:00 • committed by mara
commit 5906cc2f2b
3 changed files with 734 additions and 481 deletions
Download patch file Download diff file Expand all files Collapse all files

343
hive-jobq/src/lib.rs
View file

@ -4,42 +4,35 @@
//! # Model (v2)
//!
//! One **persistent graph** for the whole system, not a DAG per job. Enqueuing
//! inserts a self-contained **node group** and returns its id; the scheduler
//! runs a continuous loop, starting every node whose [`Dep`]s are satisfied:
//! inserts a self-contained sub-DAG of nodes and returns their ids; the
//! scheduler runs a continuous loop, starting every node whose [`Dep`]s are
//! satisfied:
//!
//! - **Resource** deps are named counting semaphores over a caller-chosen
//! type `R` (a `String` or an enum): `build-slot` (cap N), `agent/<name>`
//! (cap 1), or any name (cap 1, created on use). A node acquires *all* its
//! resource deps atomically at start (all-or-nothing) — no hold-and-wait,
//! so no deadlock and no cycle detection needed.
//! - **Node** deps wait on a node/group per [`DepWhen`]: `AfterOk` needs
//! type `R`: `build-slot` (cap N), `agent/<name>` (cap 1), or any name
//! (cap 1, created on use). A node acquires *all* its resource deps
//! atomically at start (all-or-nothing) — no hold-and-wait, no deadlock.
//! - **Node** deps wait on another node per [`DepWhen`]: `AfterOk` needs
//! success (a failed dep cancels the dependent), `AfterAny` only terminal.
//!
//! A **node group** is a self-contained sub-graph; things depend on it as a
//! whole (done = every inner node terminal), never on an inner node. Groups
//! nest; a running node may grow its own group but not reach outside it.
//! A node carries two independent axes: its [`Dep`]s (ordering + resource
//! needs) and its [`Node::parent`] (structural grouping) — the parent chain,
//! not the [`Dep::Node`] edges, is what the [`scheduler`] consults for resource
//! re-entrancy. A [`NodeId`] is opaque, stable, and monotonic (persisted). The
//! payload `N` is generic so the library stays container-agnostic.
//!
//! A [`NodeId`] is opaque, stable, and monotonic — persisted, so it survives
//! restarts. Group membership is a parent edge ([`Node::parent`]), *not* in the
//! id; the `1/1/2` hierarchy is a derived UI label. The node payload is generic
//! (`N`) so the library stays container-agnostic — the caller supplies its own
//! node kind. Resources are held by the acquiring node and released on
//! completion via guard objects, recursive within a group.
//!
//! The [`scheduler`] settle loop drives execution; the resource machinery
//! lives in [`resources`] and the RAII lock guards over it in `guard`.
//! A resource unit is held for the acquiring node + its whole [`Node::parent`]
//! subtree; a node needing a resource an ancestor holds re-uses that grant (a
//! re-entrant borrow, one branch at a time). Single-threaded — the scheduler
//! owns the resource table and mutates it directly. See [`scheduler`].
pub(crate) mod guard;
pub mod resources;
pub mod scheduler;
/// Opaque, stable, monotonic node identifier.
///
/// Assigned by the [`Graph`] on insert and persisted, so it is stable across
/// restarts. Group membership is a separate parent edge ([`Node::parent`]) — it
/// is deliberately *not* encoded in the id, so the id never changes as the tree
/// grows or collapses. The hierarchical `1/1/2` path used in the UI is derived
/// from the parent tree at render time.
/// restarts.
///
/// The inner field is crate-private: an id can only originate from the graph's
/// monotonic counter (or deserialization of a persisted graph), never be
@ -49,6 +42,18 @@ pub mod scheduler;
)]
pub struct NodeId(pub(crate) u64);
impl NodeId {
/// The underlying monotonic value, for carrying this id across a boundary
/// that cannot hold the opaque `NodeId` type — e.g. serializing it onto a
/// wire protocol. The inverse (fabricating a `NodeId` from a raw value)
/// stays impossible by construction: an id only ever originates from the
/// graph's counter, which is what makes it opaque.
#[must_use]
pub fn get(self) -> u64 {
self.0
}
}
/// When a [`Dep::Node`] edge is satisfied — the strong/weak distinction the
/// current queue carries as `DepWhen`, load-bearing for failure safety.
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
@ -80,19 +85,22 @@ impl DepWhen {
/// it names can be acquired (all of them, atomically).
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub enum Dep<R> {
/// Depend on another node (or a group, by its group node's id). Whether a
/// *failed* dependency satisfies the edge is decided by `when`: `AfterOk`
/// requires success (and cancels this node if the dep fails), `AfterAny`
/// only requires the dep to be terminal.
/// Depend on another node. Whether a *failed* dependency satisfies the edge
/// is decided by `when`: `AfterOk` requires success (and cancels this node
/// if the dep fails), `AfterAny` only requires the dep to be terminal.
Node {
/// The node (or group) depended on.
/// The node depended on.
id: NodeId,
/// Strong (`AfterOk`) vs weak (`AfterAny`).
when: DepWhen,
},
/// Hold `count` units of a named resource for the duration of this node's
/// run. Acquired atomically with the node's other resource deps at start,
/// released when the node completes.
/// Need `count` units of a named resource to run. Declared on every node
/// that needs it, even when a [`Node::parent`]-ancestor already holds it.
/// Acquired atomically with the node's other resource deps at start; the
/// acquired unit is held for the acquirer's whole subtree (released only
/// once the acquirer and all its sub-nodes are terminal). A node whose
/// parent-ancestor already holds this resource re-uses that grant (a
/// re-entrant borrow) instead of taking a fresh unit.
Resource {
/// The resource to acquire.
name: R,
@ -106,11 +114,17 @@ pub enum Dep<R> {
pub enum State {
/// Waiting on dependencies (node or resource).
Pending,
/// Dependencies satisfied, resources held, currently executing.
/// Dependencies satisfied, resources held, currently executing its own logic.
Running,
/// Completed successfully.
/// Own logic finished successfully, but the node is *not yet terminal*: it
/// waits here until all its sub-nodes ([`Node::parent`] children) are
/// terminal, then rolls up to [`State::Done`] (every child `Done`) or
/// [`State::Failed`] (any child `Failed`/`Cancelled`). A node with no
/// children never rests here — it goes straight to a terminal state.
Finishing,
/// Completed successfully — own logic done *and* every sub-node `Done`.
Done,
/// Completed unsuccessfully.
/// Completed unsuccessfully — own logic failed, or a sub-node did.
Failed,
/// Never ran: an `AfterOk` dependency failed, so this node (and the rest of
/// its strong-dependent chain) is cancelled rather than run.
@ -131,13 +145,19 @@ impl State {
///
/// The library schedules over `Node`s and resources without interpreting the
/// payload; the caller supplies `N` (its own node kind) and a runner to execute
/// a claimed node.
/// a claimed node. A node carries two independent axes: its [`Dep`]s (ordering +
/// resource needs) and its [`Node::parent`] (structural grouping), both set by
/// the caller/submit layer.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct Node<N, R> {
/// Stable identity, assigned on insert.
pub id: NodeId,
/// The group this node belongs to, if any. `None` for a top-level group
/// node. Group membership lives here, not in the id.
/// Structural grouping: the node this one is a sub-node of, or `None` for a
/// group root. Independent of [`Node::deps`] — grouping is *not* ordering.
/// The [`scheduler`] uses the parent chain to decide resource re-entrancy: a
/// node needing a resource a parent-ancestor holds re-uses that grant rather
/// than acquiring a fresh unit, and a held unit stays reserved for the
/// acquirer's whole subtree. A node's sub-nodes run *after* its own logic.
pub parent: Option<NodeId>,
/// Caller-defined payload (the node's kind / work description).
pub payload: N,
@ -158,9 +178,20 @@ pub enum GraphError {
/// A node's dependency named an id not present in the graph.
#[error("dependency references unknown node {0:?}")]
UnknownDep(NodeId),
/// A node's parent named an id not present in the graph.
/// A node's `parent` named an id not present in the graph.
#[error("parent references unknown node {0:?}")]
UnknownParent(NodeId),
/// A node's [`Dep::Node`] edge points outside its own parent group — the
/// target must be a proper descendant of the depender's `parent` (a sibling
/// or a sibling's sub-node), never the parent itself or a node in another
/// group. Top-level nodes may only depend on top-level nodes.
#[error("dependency {dep:?} is outside the depender's parent group {parent:?}")]
DepOutsideParent {
/// The out-of-group dependency target.
dep: NodeId,
/// The depender's parent (the group the target had to be inside).
parent: Option<NodeId>,
},
/// A loaded graph's `next_id` counter is not past the largest existing id,
/// so the next minted id would collide with one already in the graph.
#[error("next_id {next_id} must exceed the largest existing node id {max_id}")]
@ -174,9 +205,8 @@ pub enum GraphError {
/// The single persistent graph of all nodes.
///
/// New jobs are inserted as node groups; the scheduler (added in a follow-up)
/// walks this graph filling open slots. Completed groups are retained (no
/// pruning in v1).
/// New jobs are inserted as sub-DAGs of nodes; the scheduler walks this graph
/// filling open slots. Completed nodes are retained (no pruning in v1).
#[derive(Debug, serde::Serialize, serde::Deserialize)]
#[serde(
try_from = "GraphData<N, R>",
@ -238,33 +268,37 @@ impl<N, R> Graph<N, R> {
id
}
/// Insert a node with the given payload, deps, and parent group, returning
/// its freshly-minted id. The node starts [`State::Pending`].
/// Insert a node with the given payload, deps, and `parent`, returning its
/// freshly-minted id. The node starts [`State::Pending`].
///
/// Every [`Dep::Node`] id and the `parent` id (if any) must already resolve
/// to a node in the graph — an id is only meaningful against the graph that
/// minted it, so a dangling reference is rejected here rather than surfacing
/// as a broken edge later.
/// Every [`Dep::Node`] id and the `parent` id (when `Some`) must already
/// resolve to a node in the graph — an id is only meaningful against the
/// graph that minted it, so a dangling reference is rejected here rather than
/// surfacing as a broken edge later.
///
/// # Errors
/// Returns [`GraphError::UnknownParent`] / [`GraphError::UnknownDep`] if the
/// parent or a dependency references a node not in the graph.
/// Returns [`GraphError::UnknownDep`] / [`GraphError::UnknownParent`] for a
/// dangling dependency or parent id, or [`GraphError::DepOutsideParent`] if a
/// `Dep::Node` edge points outside the node's own parent group.
pub fn insert(
&mut self,
payload: N,
deps: Vec<Dep<R>>,
parent: Option<NodeId>,
) -> Result<NodeId, GraphError> {
if let Some(parent_id) = parent
&& self.node(parent_id).is_none()
if let Some(p) = parent
&& self.node(p).is_none()
{
return Err(GraphError::UnknownParent(parent_id));
return Err(GraphError::UnknownParent(p));
}
for dep in &deps {
if let Dep::Node { id, .. } = dep
&& self.node(*id).is_none()
{
return Err(GraphError::UnknownDep(*id));
if let Dep::Node { id, .. } = dep {
if self.node(*id).is_none() {
return Err(GraphError::UnknownDep(*id));
}
if !self.dep_target_in_group(parent, *id) {
return Err(GraphError::DepOutsideParent { dep: *id, parent });
}
}
}
let id = self.mint_id();
@ -284,17 +318,39 @@ impl<N, R> Graph<N, R> {
self.nodes.iter().find(|n| n.id == id)
}
/// The direct children of a group node (nodes whose `parent` is `id`).
pub fn children(&self, id: NodeId) -> impl Iterator<Item = &Node<N, R>> {
self.nodes.iter().filter(move |n| n.parent == Some(id))
}
/// Every node in the graph, in insertion order. The scheduler iterates
/// this to find runnable pending nodes.
pub fn nodes(&self) -> impl Iterator<Item = &Node<N, R>> {
self.nodes.iter()
}
/// Whether `ancestor` lies on `node`'s [`Node::parent`] chain (i.e. `node` is
/// in `ancestor`'s subtree). `node` is not its own ancestor.
fn is_descendant(&self, node: NodeId, ancestor: NodeId) -> bool {
let mut cur = self.node(node).and_then(|n| n.parent);
while let Some(p) = cur {
if p == ancestor {
return true;
}
cur = self.node(p).and_then(|n| n.parent);
}
false
}
/// Whether a node whose parent is `node_parent` may depend on `target` — the
/// grouping rule: a [`Dep::Node`] edge must stay inside the depender's own
/// parent group. `target` must be a proper descendant of `node_parent` (a
/// sibling or a sibling's sub-node), never the parent itself (which would
/// deadlock: the parent stays [`State::Finishing`] until its children finish,
/// so a child that waited on the parent could never run). Top-level nodes
/// (`parent == None`) may only depend on other top-level nodes.
fn dep_target_in_group(&self, node_parent: Option<NodeId>, target: NodeId) -> bool {
match node_parent {
Some(p) => self.is_descendant(target, p),
None => self.node(target).is_some_and(|n| n.parent.is_none()),
}
}
/// Set a node's lifecycle state, returning `false` for an unknown id. The
/// scheduler drives every state transition — nothing else mutates state,
/// which is what keeps the resource guards + terminality in sync.
@ -307,40 +363,32 @@ impl<N, R> Graph<N, R> {
}
}
/// A group is terminal once the group node itself is terminal *and* every
/// node inside it (recursively) is terminal. The node's own state matters:
/// a group node still `Pending`/`Running` is not terminal even with no
/// children yet, since a running node may still append some. (Deciding when
/// to *settle* a group node to terminal once its children are all done is a
/// separate concern the scheduler owns.) An unknown id is not terminal.
#[must_use]
pub fn group_terminal(&self, id: NodeId) -> bool {
let Some(node) = self.node(id) else {
return false;
};
node.state.is_terminal() && self.children(id).all(|child| self.group_terminal(child.id))
}
/// Check that every id the graph holds resolves: each node's `parent` and
/// every [`Dep::Node`] id names a node present in the graph, and `next_id`
/// is past the largest existing id. Deserialization runs this, so a loaded
/// graph is internally consistent and internal iteration can trust its ids.
/// Check that every id the graph holds resolves: every [`Dep::Node`] id
/// names a node present in the graph, and `next_id` is past the largest
/// existing id. Deserialization runs this, so a loaded graph is internally
/// consistent and internal iteration can trust its ids.
///
/// # Errors
/// Returns [`GraphError`] on a dangling parent / dependency reference, or a
/// `next_id` that would remint an id already in the graph.
/// Returns [`GraphError`] on a dangling dependency reference, or a `next_id`
/// that would remint an id already in the graph.
pub fn validate(&self) -> Result<(), GraphError> {
for node in &self.nodes {
if let Some(parent_id) = node.parent
&& self.node(parent_id).is_none()
if let Some(p) = node.parent
&& self.node(p).is_none()
{
return Err(GraphError::UnknownParent(parent_id));
return Err(GraphError::UnknownParent(p));
}
for dep in &node.deps {
if let Dep::Node { id, .. } = dep
&& self.node(*id).is_none()
{
return Err(GraphError::UnknownDep(*id));
if let Dep::Node { id, .. } = dep {
if self.node(*id).is_none() {
return Err(GraphError::UnknownDep(*id));
}
if !self.dep_target_in_group(node.parent, *id) {
return Err(GraphError::DepOutsideParent {
dep: *id,
parent: node.parent,
});
}
}
}
}
@ -371,43 +419,16 @@ mod tests {
id: a,
when: DepWhen::AfterOk,
}],
Some(a),
None,
)
.unwrap();
assert_eq!(a, NodeId(0));
assert_eq!(b, NodeId(1));
// Membership is the parent edge, not the id.
assert_eq!(g.node(b).unwrap().parent, Some(a));
assert_eq!(g.node(a).unwrap().parent, None);
}
#[test]
fn group_terminal_requires_the_group_node_and_all_children_terminal() {
let mut g: Graph<&str, String> = Graph::new();
let group = g.insert("group", vec![], None).unwrap();
let child = g.insert("child", vec![], Some(group)).unwrap();
// Both pending → not terminal.
assert!(!g.group_terminal(group));
// Child done, but the group node itself is still pending → NOT terminal:
// the group node's own state is load-bearing, not just its children.
g.set_state(child, State::Done);
assert!(!g.group_terminal(group));
// Group node terminal too → the whole group is terminal.
g.set_state(group, State::Done);
assert!(g.group_terminal(group));
}
#[test]
fn empty_running_group_is_not_terminal() {
// A running node with no children yet may still append some, so it must
// not read as terminal just because its child set is currently empty.
let mut g: Graph<&str, String> = Graph::new();
let group = g.insert("group", vec![], None).unwrap();
g.set_state(group, State::Running);
assert!(!g.group_terminal(group));
// Once it finishes (having grown no children), it is terminal.
g.set_state(group, State::Done);
assert!(g.group_terminal(group));
// The dep edge references the earlier node; ids are stable + monotonic.
assert!(matches!(
g.node(b).unwrap().deps.first(),
Some(Dep::Node { id, .. }) if *id == a
));
}
#[test]
@ -417,6 +438,8 @@ mod tests {
assert!(State::Cancelled.is_terminal());
assert!(!State::Pending.is_terminal());
assert!(!State::Running.is_terminal());
// Finishing (logic done, children still running) is NOT terminal.
assert!(!State::Finishing.is_terminal());
}
#[test]
@ -432,16 +455,10 @@ mod tests {
assert!(DepWhen::AfterAny.satisfied_by(State::Failed));
assert!(DepWhen::AfterAny.satisfied_by(State::Cancelled));
assert!(!DepWhen::AfterAny.satisfied_by(State::Pending));
}
#[test]
fn insert_rejects_unknown_parent() {
let mut g: Graph<&str, String> = Graph::new();
let bogus = NodeId(7);
assert_eq!(
g.insert("x", vec![], Some(bogus)).unwrap_err(),
GraphError::UnknownParent(bogus)
);
// Finishing satisfies neither — a dependent waits until the node rolls
// up to a terminal state (all its sub-nodes done).
assert!(!DepWhen::AfterOk.satisfied_by(State::Finishing));
assert!(!DepWhen::AfterAny.satisfied_by(State::Finishing));
}
#[test]
@ -458,9 +475,25 @@ mod tests {
);
}
#[test]
fn insert_rejects_unknown_parent() {
let mut g: Graph<&str, String> = Graph::new();
let bogus = NodeId(7);
assert_eq!(
g.insert("x", vec![], Some(bogus)).unwrap_err(),
GraphError::UnknownParent(bogus)
);
// A resolvable parent is accepted and recorded.
let a = g.insert("a", vec![], None).unwrap();
let b = g.insert("b", vec![], Some(a)).unwrap();
assert_eq!(g.node(b).unwrap().parent, Some(a));
}
#[test]
fn valid_graph_round_trips_through_serde() {
let mut g: Graph<String, String> = Graph::new();
// `a` (top-level) and `b` (top-level, depends on its sibling `a`), plus
// `c` — a sub-node of `a` (grouping, no dep on its parent).
let a = g.insert("a".to_owned(), vec![], None).unwrap();
g.insert(
"b".to_owned(),
@ -468,13 +501,57 @@ mod tests {
id: a,
when: DepWhen::AfterAny,
}],
Some(a),
None,
)
.unwrap();
let c = g.insert("c".to_owned(), vec![], Some(a)).unwrap();
let json = serde_json::to_string(&g).unwrap();
let back: Graph<String, String> = serde_json::from_str(&json).unwrap();
assert!(back.validate().is_ok());
assert_eq!(back.node(a).unwrap().payload, "a");
assert_eq!(back.node(c).unwrap().parent, Some(a));
}
#[test]
fn insert_rejects_dep_on_parent_and_cross_group() {
let mut g: Graph<&str, String> = Graph::new();
let root = g.insert("root", vec![], None).unwrap();
// A child cannot depend on its own parent (would deadlock under the
// roll-up model — the parent stays `Finishing` awaiting its children).
let on_parent = vec![Dep::Node {
id: root,
when: DepWhen::AfterOk,
}];
assert_eq!(
g.insert("child", on_parent, Some(root)).unwrap_err(),
GraphError::DepOutsideParent {
dep: root,
parent: Some(root),
}
);
// A sibling dep IS allowed: two children of `root`, the second on the first.
let c1 = g.insert("c1", vec![], Some(root)).unwrap();
let c2 = g
.insert("c2", vec![after_ok_dep(c1)], Some(root))
.expect("sibling dep is in-group");
assert_eq!(g.node(c2).unwrap().parent, Some(root));
// But a node in another group cannot be depended on across the boundary.
let other = g.insert("other", vec![], None).unwrap();
assert_eq!(
g.insert("x", vec![after_ok_dep(other)], Some(root))
.unwrap_err(),
GraphError::DepOutsideParent {
dep: other,
parent: Some(root),
}
);
}
fn after_ok_dep(on: NodeId) -> Dep<String> {
Dep::Node {
id: on,
when: DepWhen::AfterOk,
}
}
#[test]