vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: drop Claim, claim_ready and the completion wrappers

Browse source 
c0re's run loop now goes through hive_jobq's claim_next seam, so the
host layer no longer needs its own claim/complete vocabulary.

exec::run_node takes (NodeId, &NodeKind) instead of a &Claim snapshot.
The agent already rides the payload, and the DAG id is a derived read
(JobQueue::dag_of) that only three arms want, so it is taken per-arm
rather than eagerly for every node. Two arms (WritePermFile, Reparent)
re-matched the kind behind a bail! that could never fire; the match arm
already destructures the payload, so they take it directly now.

Deleted from the c0re layer:
  - struct Claim
  - JobQueue::claim_ready
  - JobQueue::complete_node / complete_node_growing
  - scheduler::NodeDone / handle_completion

Completion happens inside the future claim_next hands back, so "ran the
node but forgot to complete it" is not expressible on the production
path any more. The node done / node failed logging moved with it -- it
lived in handle_completion but is not dead code.

claim_ready and the completion wrappers were left with no non-test
callers, so the tests carry them as ClaimReady / CompleteNode extension
traits over the crate primitives. JobQueue::new_job stays: run_worker
still mints an empty builder on a failed outcome.
This commit is contained in:
atlas 2026-08-02 19:01:53 +02:00 • committed by mara
commit ab53f6710d
4 changed files with 263 additions and 255 deletions
Download patch file Download diff file Expand all files Collapse all files

157
hive-c0re/src/job_queue/scheduler.rs
View file

@ -17,23 +17,17 @@
//!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning
//! its `Start`/`Stop`) is declared onto the builder each node is handed, and
//! inserted as part of completing that node — see `handle_completion`.
//! inserted as part of completing that node. Completion itself is not this
//! module's job any more: it happens *inside* the future
//! [`hive_jobq::scheduler::Scheduler::claim_next`] hands back, so a node that
//! ran but was never completed is not an expressible state here.
use std::collections::HashMap;
use std::sync::Arc;
use super::exec;
use super::{Claim, Job};
use crate::coordinator::Coordinator;
struct NodeDone {
claim: Claim,
/// Whatever the node declared into its builder while running — usually
/// nothing. Inserted under the node as part of completing it.
grown: Job,
result: anyhow::Result<()>,
}
/// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`.
///
/// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true signal
@ -56,7 +50,6 @@ struct NodeDone {
/// reconverging silently.
pub async fn run_worker(coord: Arc<Coordinator>) {
let mut shutdown = coord.shutdown_rx();
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<NodeDone>();
// Last derived pill set we published, keyed by agent (its lease is cap-1,
// so one pill each). Purely the previous value of a *derived* quantity —
// it exists to spot transitions, since the dashboard wants edges
@ -73,33 +66,69 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
return;
}
reconcile_transients(&coord, &mut transients);
let claims = coord.job_queue.claim_ready();
if !claims.is_empty() {
for claim in claims {
tracing::info!(
dag = claim.dag_id,
node = claim.node_id.get(),
kind = claim.kind.as_str(),
agent = %claim.agent,
"job_queue: node running"
);
let coord = Arc::clone(&coord);
let tx = tx.clone();
tokio::spawn(async move {
// The node's growth channel. Local state, so it costs
// nothing to carry and holds no lock while the node runs.
// The builder is passed by value and handed back: owned it
// is `Send`, a `&Job` held across an await is not.
let job = coord.job_queue.new_job();
let (grown, result) = exec::run_node(&coord, job, &claim).await;
// Send failure = scheduler gone (shutdown); drop.
let _ = tx.send(NodeDone {
claim,
grown,
result,
});
});
}
// Claim exactly one node and get back the work that runs it. `Some`
// means something started, so there may be more runnable right now —
// loop again immediately. `None` means nothing is runnable and the
// loop parks below. That decision is the whole reason the crate hands
// back a task rather than an id.
let runner = {
// Two handles, deliberately: `sched` is the scheduler the crate
// locks, `node_coord` is what the node's own future captures. One
// binding can't do both — passing `coord.job_queue.sched()` borrows
// `coord` for the whole call while the `move` closure wants to take
// it.
let sched = Arc::clone(coord.job_queue.sched());
let node_coord = Arc::clone(&coord);
hive_jobq::scheduler::Scheduler::claim_next(&sched, move |id, kind, job| {
let coord = node_coord;
async move {
tracing::info!(
dag = coord.job_queue.dag_of(id).unwrap_or_default(),
node = id.get(),
kind = kind.as_str(),
agent = %kind.agent(),
"job_queue: node running"
);
let (grown, result) = exec::run_node(&coord, job, id, &kind).await;
match &result {
Ok(()) => tracing::info!(node = id.get(), "job_queue: node done"),
Err(e) => tracing::warn!(
node = id.get(),
kind = kind.as_str(),
agent = %kind.agent(),
error = %format!("{e:#}"),
grown_nodes = !grown.is_empty(),
"job_queue: node failed"
),
}
let outcome = super::outcome_of(result.map_err(|e| format!("{e:#}")));
// Growth is dropped on failure: a node that declared
// follow-up work and *then* failed does not want it run —
// failure cancel-cascades, so inserting it would only add
// nodes to immediately cancel.
let grown = if matches!(outcome, hive_jobq::scheduler::Outcome::Failed(_)) {
coord.job_queue.new_job()
} else {
grown
};
(grown, outcome)
}
})
};
if let Some(runner) = runner {
let done_coord = Arc::clone(&coord);
tokio::spawn(async move {
// Completion happens inside `runner` — it cannot be forgotten
// here, which is why there is no completion channel any more.
let (id, grew) = runner.await;
if let Err(e) = grew {
tracing::warn!(node = id.get(), error = %e, "job_queue: grown job rejected");
}
done_coord.emit_rebuild_queue_snapshot();
// Wake the loop: this node's completion may have unblocked
// dependents. Previously the completion channel did this.
done_coord.job_queue.notify.notify_one();
});
// Newly-started owner nodes now hold their leases — surface the pills.
reconcile_transients(&coord, &mut transients);
coord.emit_rebuild_queue_snapshot();
@ -113,63 +142,11 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
return;
}
}
Some(done) = rx.recv() => {
handle_completion(&coord, done);
}
() = coord.job_queue.notify.notified() => {}
}
}
}
fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) {
let NodeDone {
claim,
grown,
result,
} = done;
match result {
Ok(()) => {
tracing::info!(
dag = claim.dag_id,
node = claim.node_id.get(),
"job_queue: node done"
);
// Whatever the node declared goes in under it as part of this
// completion, so the DAG cannot roll terminal while the appended
// work is still pending. Covers the multi-node case (a `MetaLock`
// growing per-agent rebuild subgraphs) and the single-node case (a
// `Reconcile` planner's `Start` / `Stop`) identically.
coord
.job_queue
.complete_node_growing(claim.node_id, Ok(()), grown);
}
Err(e) => {
let msg = format!("{e:#}");
tracing::warn!(
dag = claim.dag_id,
node = claim.node_id.get(),
kind = claim.kind.as_str(),
agent = %claim.agent,
error = %msg,
grown_nodes = !grown.is_empty(),
"job_queue: node failed"
);
// `grown` is deliberately dropped on failure. A node that declared
// follow-up work and *then* failed does not want that work run —
// failure cancel-cascades downstream, so inserting it would only
// add nodes to immediately cancel. This preserves the old shape,
// where growth could only be expressed on the success path at all;
// the difference is that it is now possible to declare and then
// fail, so the drop has to be a decision rather than an accident.
drop(grown);
coord.job_queue.complete_node(claim.node_id, Err(msg));
}
}
// The next loop iteration re-reconciles the transient pills against the
// post-completion lease state (a settled subgraph drops its pill).
coord.emit_rebuild_queue_snapshot();
}
/// Publish the transitions between the previously-derived pill set and the
/// current one. `prev` is last loop's derived value, keyed by agent (an agent's
/// lease is cap-1, so at most one pill each).