refactor(job-queue): a job is a recipe, not a value you carry
Follows the jobq change: a builder can no longer be constructed or inserted outside `hive_jobq`, so `DagSpec` cannot hold one. It carries a `Declare` — `Box<dyn FnOnce(&Job) + Send>` — and the queue runs it against a builder jobq owns, at the moment it inserts. `NodeOutput.append_subgraph` becomes `Vec<Declare>` for the same reason, and this is where the shape was always heading: that field's doc already said an executor "cannot reach the queue, so it hands the declaration back", while its type was a `Vec<Job>` the executor had built itself. The rejected `build_nodes -> Vec<NodeSpec>` was the first version of that escape hatch; a recipe is the last one, because there is no job-shaped value to hand over at all. Templates and the power-op assemblers move their owned data into the closure and are otherwise unchanged — `rebuild_nodes`, `node` and the tail helpers already took `&Job` and returned handles, so only each template's outermost frame moved. Two `Debug` impls are hand-written: a closure has nothing to show, and its nodes do not exist until the queue runs it. `NodeOutput` reports how many subgraphs were emitted, `DagSpec` its source and reason. `append_subgraph`'s `is_empty()` early-return is gone — you cannot ask a recipe whether it will declare anything without running it. It now inserts and returns an empty id list if nothing was declared, which takes the queue lock in a case that previously skipped it. The two in-DAG-growth tests build `Declare`s now, so they exercise the shape an executor actually produces rather than one only a test could construct. 45 job-queue tests unchanged and passing.
This commit is contained in:
parent
f7548e4535
commit
9c97365f8f
8 changed files with 336 additions and 284 deletions
|
|
@ -51,10 +51,20 @@ pub use model::{DagSpec, DagView, NodeKind, PermPayload, Source, State};
|
|||
use resource::Resource;
|
||||
|
||||
/// A job under construction: `hive_jobq`'s builder over this queue's payload
|
||||
/// ([`NodeKind`]) and resource ([`Resource`]) types. Templates declare into
|
||||
/// one of these; [`JobQueue::submit`] inserts it.
|
||||
/// ([`NodeKind`]) and resource ([`Resource`]) types. Templates declare into a
|
||||
/// borrowed one; only `hive_jobq` can make or insert it.
|
||||
pub type Job = hive_jobq::JobBuilder<NodeKind, Resource>;
|
||||
|
||||
/// A job's shape as a **recipe**: given a builder, declare the nodes.
|
||||
///
|
||||
/// What a template returns and what an executor hands back, because neither
|
||||
/// can build a job itself — `hive_jobq` creates the builder inside its own
|
||||
/// insertion call and never lets one out. So the transferable thing is the
|
||||
/// declaring closure, and the queue runs it at the moment it inserts.
|
||||
///
|
||||
/// `Send` because an executor's output crosses the scheduler's task boundary.
|
||||
pub type Declare = Box<dyn FnOnce(&Job) + Send>;
|
||||
|
||||
/// A handle to one node a template declared — where its edges, grouping and
|
||||
/// resources are declared. `Copy`; naming a node as a dependency does not
|
||||
/// consume the ability to name it again.
|
||||
|
|
@ -169,12 +179,12 @@ impl Default for JobQueue {
|
|||
/// Propagates a crate graph-insert error (malformed dep/parent / dep-scope).
|
||||
fn insert_group(
|
||||
inner: &mut QueueInner,
|
||||
job: Job,
|
||||
declare: Declare,
|
||||
group_parent: Option<NodeId>,
|
||||
) -> anyhow::Result<Vec<NodeId>> {
|
||||
let ids = inner
|
||||
.sched
|
||||
.insert_job(job, group_parent)
|
||||
.insert_job(group_parent, declare)
|
||||
.map_err(|e| anyhow::anyhow!("job_queue: graph insert failed: {e}"))?;
|
||||
for &id in ids.values() {
|
||||
inner.node_rt.insert(id, NodeRuntime::default());
|
||||
|
|
@ -226,7 +236,7 @@ impl JobQueue {
|
|||
)
|
||||
.map_err(|e| anyhow::anyhow!("job_queue: container insert failed: {e}"))?;
|
||||
inner.node_rt.insert(container, NodeRuntime::default());
|
||||
insert_group(&mut inner, spec.job, Some(container))?;
|
||||
insert_group(&mut inner, spec.declare, Some(container))?;
|
||||
// Settle the container's own (no-op) logic immediately so it parks in
|
||||
// `Finishing` and its children become runnable — it never needs claiming
|
||||
// or executing, and stays out of `claim_ready`. It rolls up terminal when
|
||||
|
|
@ -247,10 +257,7 @@ impl JobQueue {
|
|||
/// the DAG's terminal node deps on the top root, roll-up keeps the DAG from
|
||||
/// settling early with no explicit wiring. Returns the new node ids; empty if
|
||||
/// the DAG is gone or `nodes` is empty.
|
||||
pub fn append_subgraph(&self, dag_id: u64, job: Job, dep_on: NodeId) -> Vec<NodeId> {
|
||||
if job.is_empty() {
|
||||
return Vec::new();
|
||||
}
|
||||
pub fn append_subgraph(&self, dag_id: u64, declare: Declare, dep_on: NodeId) -> Vec<NodeId> {
|
||||
let mut inner = self.lock();
|
||||
if inner.container(dag_id).is_none() {
|
||||
return Vec::new();
|
||||
|
|
@ -261,7 +268,7 @@ impl JobQueue {
|
|||
// emitter stays `Finishing` until this appended subtree settles, and the
|
||||
// container node rolls up terminal only once its whole subtree (incl. this
|
||||
// appended work) has settled, so the DAG hook waits for free.
|
||||
let ids = match insert_group(&mut inner, job, Some(dep_on)) {
|
||||
let ids = match insert_group(&mut inner, declare, Some(dep_on)) {
|
||||
Ok(ids) => ids,
|
||||
Err(e) => {
|
||||
tracing::error!(
|
||||
|
|
|
|||
Loading…
Reference in a new issue