refactor(job-queue): a job is a recipe, not a value you carry

Follows the jobq change: a builder can no longer be constructed or
inserted outside `hive_jobq`, so `DagSpec` cannot hold one. It carries a
`Declare` — `Box<dyn FnOnce(&Job) + Send>` — and the queue runs it
against a builder jobq owns, at the moment it inserts.

`NodeOutput.append_subgraph` becomes `Vec<Declare>` for the same reason,
and this is where the shape was always heading: that field's doc already
said an executor "cannot reach the queue, so it hands the declaration
back", while its type was a `Vec<Job>` the executor had built itself.
The rejected `build_nodes -> Vec<NodeSpec>` was the first version of that
escape hatch; a recipe is the last one, because there is no job-shaped
value to hand over at all.

Templates and the power-op assemblers move their owned data into the
closure and are otherwise unchanged — `rebuild_nodes`, `node` and the
tail helpers already took `&Job` and returned handles, so only each
template's outermost frame moved.

Two `Debug` impls are hand-written: a closure has nothing to show, and
its nodes do not exist until the queue runs it. `NodeOutput` reports how
many subgraphs were emitted, `DagSpec` its source and reason.

`append_subgraph`'s `is_empty()` early-return is gone — you cannot ask a
recipe whether it will declare anything without running it. It now
inserts and returns an empty id list if nothing was declared, which
takes the queue lock in a case that previously skipped it.

The two in-DAG-growth tests build `Declare`s now, so they exercise the
shape an executor actually produces rather than one only a test could
construct. 45 job-queue tests unchanged and passing.
This commit is contained in:
atlas 2026-08-02 13:40:03 +02:00 committed by mara
commit 9c97365f8f
8 changed files with 336 additions and 284 deletions

View file

@ -470,15 +470,27 @@ impl NodeKind {
/// Type-specific payloads (`PermChange`'s file payload) ride the node that
/// consumes them ([`NodeKind::WritePermFile`]), not this generic spec.
///
/// There is no separate per-node spec type: the nodes live in the builder,
/// which inserts them itself. A shape that has been declared is therefore
/// always insertable — a dangling edge or a cycle cannot be expressed, so
/// there is nothing left for a submit-time validation pass to reject.
#[derive(Debug)]
/// There is no separate per-node spec type, and no built job either: `declare`
/// is a *recipe* the queue runs against a builder `hive_jobq` owns, at the
/// moment it inserts. A shape that has been declared is therefore always
/// insertable — a dangling edge or a cycle cannot be expressed, so there is
/// nothing left for a submit-time validation pass to reject.
pub struct DagSpec {
pub source: Source,
/// Free-form "why".
pub reason: String,
/// The DAG's declared nodes, with their edges, grouping and resources.
pub job: super::Job,
/// Declares the DAG's nodes — their edges, grouping and resources — onto
/// the builder the queue hands it.
pub declare: super::Declare,
}
impl std::fmt::Debug for DagSpec {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
// The recipe is a closure; there is nothing to show of it, and its
// nodes do not exist until the queue runs it.
f.debug_struct("DagSpec")
.field("source", &self.source)
.field("reason", &self.reason)
.finish_non_exhaustive()
}
}