vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2591): port hive-c0re job_queue onto the hive-jobq crate

Browse source 
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).

Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
  sibling ordering directly (deps order execution, parent groups a subtree
  whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
  for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
  agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
  separate top-level root (AfterAny Prebuild) so it survives the cancel-
  cascade of any failed step (recovery-start invariant) and converges to
  the persisted `wanted` on a fresh lease. This is the multi-root
  correction to the single-root-chain sketch: node0=root broke lease-
  exemption (hoisting the lease onto Prebuild) and recovery-reconcile
  (root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
  same way; per-agent power-op subgraphs stay independent roots so a
  multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
  DAG terminal node deps AfterAny on every group root and runs once the
  whole op rolls up. Drop the old Graph::add_dep terminal wiring.

36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
This commit is contained in:
atlas 2026-07-20 21:46:08 +02:00 • committed by mara
commit a5c321a1a0
14 changed files with 1111 additions and 893 deletions
Download patch file Download diff file Expand all files Collapse all files

119
hive-c0re/src/job_queue/scheduler.rs
View file

@ -1,18 +1,25 @@
//! The single scheduler task that drives all DAGs: claim every ready
//! node (as many as the build slots / leases allow), spawn one
//! executor task per claim, and on any completion re-evaluate.
//! Concurrency comes from the build-slot count, not multiple workers.
//! The single scheduler task that drives all DAGs: claim every ready node (as
//! many as the build slots / leases allow), spawn one executor task per claim,
//! and on any completion re-evaluate. Concurrency comes from the build-slot
//! count, not multiple workers.
//!
//! Also owns the per-DAG transient guard (dashboard pill + crash-watch
//! suppression) that the sync queue core can't hold itself — created when a
//! DAG acquires its agent lease, dropped when the DAG settles terminal.
//! Owns the per-DAG transient guard (dashboard pill + crash-watch suppression)
//! that the sync queue core can't hold itself. The guard set is *reconciled*
//! from live lease ownership ([`super::JobQueue::held_transients`]) each loop:
//! a `(dag, agent)` pill exists for exactly as long as that agent's lease is
//! held, so it appears when the agent's owner node starts and disappears when
//! its subgraph settles — one pill per agent a DAG touches.
//!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs after the lock
//! bump, a `Reconcile` fanning its `Start`/`Stop`) flows through
//! `NodeOutput.append_subgraph`, applied before the emitting node
//! completes — see `handle_completion`.
//! Per-DAG terminal work (approval resolution, `Rebuilt`, cancelled-power-op
//! intent revert) is not drained here: it runs as the DAG's focused terminal
//! node (`ResolveApproval` / `EmitRebuilt` / `RevertIntent`), dispatched through
//! `exec::run_node` like any other node once the DAG settles.
//!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning
//! its `Start`/`Stop`) flows through `NodeOutput.append_subgraph`, applied
//! before the emitting node completes — see `handle_completion`.
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use super::Claim;
@ -26,38 +33,24 @@ struct NodeDone {
/// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`.
///
/// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true
/// signal the loop exits immediately; already-running node tasks ride
/// the runtime down with the process, and pending `Queued` DAGs are
/// dropped — desired state is re-derived on next boot (boot sweep +
/// reconcile), so the in-memory queue is deliberately not durable.
/// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true signal
/// the loop exits immediately; already-running node tasks ride the runtime down
/// with the process, and pending `Queued` DAGs are dropped — desired state is
/// re-derived on next boot (boot sweep + reconcile), so the in-memory queue is
/// deliberately not durable.
pub async fn run_worker(coord: Arc<Coordinator>) {
let mut shutdown = coord.shutdown_rx();
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<NodeDone>();
// (DAG id, agent) → transient guard held for that agent's lease
// window. Keyed per-agent so a multi-agent DAG shows one transient
// pill per agent it touches.
// (DAG id, agent) → transient guard held for that agent's lease window.
let mut transients: HashMap<(u64, String), crate::coordinator::TransientGuard> = HashMap::new();
loop {
// Terminal roll-ups can appear without a node completion —
// the cancel surfaces settle DAGs directly and wake this loop
// via notify — so drain on every iteration, not just inside
// handle_completion.
process_terminals(&coord, &mut transients).await;
reconcile_transients(&coord, &mut transients);
let claims = coord.job_queue.claim_ready();
if !claims.is_empty() {
for claim in claims {
if claim.lease_acquired
&& let Some(kind) = claim.transient
{
transients.insert(
(claim.dag_id, claim.agent.clone()),
coord.transient_guard(&claim.agent, kind),
);
}
tracing::info!(
dag = claim.dag_id,
node = claim.node_id,
node = claim.node_id.get(),
kind = claim.kind.as_str(),
agent = %claim.agent,
template = claim.template.as_str(),
@ -71,6 +64,8 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
let _ = tx.send(NodeDone { claim, result });
});
}
// Newly-started owner nodes now hold their leases — surface the pills.
reconcile_transients(&coord, &mut transients);
coord.emit_rebuild_queue_snapshot();
continue;
}
@ -83,35 +78,30 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
}
}
Some(done) = rx.recv() => {
handle_completion(&coord, &mut transients, done).await;
handle_completion(&coord, done);
}
() = coord.job_queue.notify.notified() => {}
}
}
}
async fn handle_completion(
coord: &Arc<Coordinator>,
transients: &mut HashMap<(u64, String), crate::coordinator::TransientGuard>,
done: NodeDone,
) {
fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) {
let NodeDone { claim, result } = done;
match result {
Ok(output) => {
tracing::info!(
dag = claim.dag_id,
node = claim.node_id,
node = claim.node_id.get(),
"job_queue: node done"
);
// Append any in-DAG subgraphs BEFORE completing this node, so
// completing it doesn't roll the DAG terminal while the appended
// work is still pending — that keeps the lease-window transient
// held across it. Each subgraph is independent, rooted on this
// node (`AfterOk`), so it becomes ready the instant this one
// settles `Done` just below. Covers both the multi-node case (a
// `MetaLock` growing per-agent rebuild subgraphs) and the
// single-node case (a `Reconcile` planner's `Start` / `Stop`).
for subgraph in output.append_subgraph {
// work is still pending. Each subgraph roots on this node
// (`AfterOk`), so it becomes ready the instant this one settles
// `Done` just below — covers both the multi-node case (a `MetaLock`
// growing per-agent rebuild subgraphs) and the single-node case (a
// `Reconcile` planner's `Start` / `Stop`).
for subgraph in &output.append_subgraph {
coord
.job_queue
.append_subgraph(claim.dag_id, subgraph, claim.node_id);
@ -124,7 +114,7 @@ async fn handle_completion(
let msg = format!("{e:#}");
tracing::warn!(
dag = claim.dag_id,
node = claim.node_id,
node = claim.node_id.get(),
kind = claim.kind.as_str(),
agent = %claim.agent,
error = %msg,
@ -135,30 +125,23 @@ async fn handle_completion(
.complete_node(claim.dag_id, claim.node_id, Err(msg));
}
}
process_terminals(coord, transients).await;
// The next loop iteration re-reconciles the transient pills against the
// post-completion lease state (a settled subgraph drops its pill).
coord.emit_rebuild_queue_snapshot();
}
/// Drain per-agent lease releases and buffered terminal roll-ups.
///
/// Per-agent first: an agent's subgraph within a DAG went terminal (its
/// lease was freed in `settle`), so drop that agent's `(dag, agent)`
/// transient pill now — ahead of whole-DAG terminal for a multi-agent
/// DAG. Then the whole-DAG terminals: drop any remaining transient the
/// DAG still held and run the terminal hook (approval resolution,
/// `Rebuilt` events, cancelled-power-op intent revert).
async fn process_terminals(
/// Reconcile the transient-guard set against live lease ownership: drop pills
/// whose lease is no longer held, create one for each newly-held `(dag, agent)`.
fn reconcile_transients(
coord: &Arc<Coordinator>,
transients: &mut HashMap<(u64, String), crate::coordinator::TransientGuard>,
) {
for rel in coord.job_queue.drain_agent_releases() {
transients.remove(&(rel.dag_id, rel.agent));
}
for terminal in coord.job_queue.drain_terminal() {
// Drop any per-agent transient guard the DAG still held (the
// per-agent pass above already dropped the ones whose subgraphs
// settled early).
transients.retain(|(dag_id, _), _| *dag_id != terminal.dag_id);
exec::on_dag_terminal(coord, &terminal).await;
let held = coord.job_queue.held_transients();
let keys: HashSet<(u64, String)> = held.iter().map(|(d, a, _)| (*d, a.clone())).collect();
transients.retain(|k, _| keys.contains(k));
for (dag_id, agent, kind) in held {
transients
.entry((dag_id, agent.clone()))
.or_insert_with(|| coord.transient_guard(&agent, kind));
}
}