refactor(job-queue): build DAGs by naming nodes, not counting them
Every template built a `Vec<NodeSpec>` whose edges and parents were positional indices into that vector, so a shape was expressed as arithmetic: `base + 1`, `stop_root + 2`, `sfu + 1`, and a `reconcile_index()` helper that read the emitted vector's length to find out where its own last node had landed. `concat_subgraphs` existed solely to rebase one per-agent subgraph's indices onto another's. Templates now declare into a `hive_jobq::JobBuilder` and hold the handles they get back, so an edge names the node it waits on. The arithmetic is gone, and with it: - `NodeSpec` and the job-queue's own index-based `Dep`. - `insert_group`'s index resolution — it wraps `Scheduler::insert_job`. - `concat_subgraphs` — per-agent chains share one builder and each keeps its own root, so independence is structural rather than computed. - `reconcile_index` and `dep_index`. - `templates::validate` and its petgraph toposort. It rejected dangling deps and cycles; both are now unrepresentable, since a handle only exists for an already-declared node and every edge therefore points backwards. (petgraph stays in the tree for `agent_config::topology`.) `NodeOutput.append_subgraph` becomes `Vec<Job>`: an executor cannot reach the queue, so it hands back declarations and the scheduler inserts them under its own lock. That is what the in-DAG growth path always wanted — a transferable declaration, not a vector of specs. Resource declaration is unchanged in behaviour: the `templates::node` helper applies `NodeKind::resource_deps()` at the construction site, so every node still declares what its kind needs. Moving that declaration to the call sites is #2818's job; this leaves it one place to delete. Three tests went with the guard they covered — they hand-built malformed specs out of indices, which is the representation that made those shapes possible. Two more now read a DAG's shape off the queue rather than out of a spec vector, which is where it is observable. The remaining 45 job-queue tests are unchanged and still pass: lease serialization, roll-up, cancel-cascade, in-DAG growth and per-agent concurrency all behave as before.
This commit is contained in:
parent
9be7731c5e
commit
e7c3cf5a3d
9 changed files with 528 additions and 696 deletions
|
|
@ -13,18 +13,10 @@
|
|||
//! full design.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
pub use hive_host_sock::jobs::{DagView, NodeId, PermPayload, Source, State};
|
||||
pub use hive_host_sock::jobs::{DagView, PermPayload, Source, State};
|
||||
use serde::Serialize;
|
||||
|
||||
use hive_jobq::{DepWhen, TerminalState};
|
||||
|
||||
/// A dependency edge (intra-DAG only — cross-DAG ordering comes from
|
||||
/// the per-agent lease + dedup, never from edges between DAGs).
|
||||
#[derive(Debug, Clone, Copy, Serialize)]
|
||||
pub struct Dep {
|
||||
pub on: NodeId,
|
||||
pub when: DepWhen,
|
||||
}
|
||||
use hive_jobq::TerminalState;
|
||||
|
||||
/// The primitive operations — each kind maps to one executor fn in
|
||||
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta.rs`
|
||||
|
|
@ -469,35 +461,24 @@ impl NodeKind {
|
|||
}
|
||||
}
|
||||
|
||||
/// Submit-time spec for one node.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NodeSpec {
|
||||
/// The node's payload — [`NodeKind`] is the queue's payload type directly,
|
||||
/// and each variant carries the agent it targets (a DAG can span agents;
|
||||
/// the queue derives per-agent leasing from [`NodeKind::agent`]).
|
||||
pub kind: NodeKind,
|
||||
pub deps: Vec<Dep>,
|
||||
/// The **structural parent** axis — the spec-local index of this node's
|
||||
/// group parent, or `None` for a top-level (group-root) node. Independent
|
||||
/// of `deps`: `deps` order execution, `parent` groups nodes into a subtree
|
||||
/// whose resource the whole subtree borrows (the agent lease is owned by a
|
||||
/// group root and re-entered by its descendants for continuity). A child
|
||||
/// runs once its parent reaches `Finishing` (the parent gate), so a child
|
||||
/// never `deps` on its own parent (that would deadlock — dep-scope
|
||||
/// validation rejects it).
|
||||
pub parent: Option<u64>,
|
||||
}
|
||||
|
||||
/// Submit-time spec for a whole DAG. Built by `templates.rs`; validated
|
||||
/// (cycle rejection) by `JobQueue::submit`. No DAG-level `agent` — every
|
||||
/// node carries its own (a DAG can span agents), and the queue derives
|
||||
/// per-agent leasing from [`NodeKind::agent`]. Type-specific payloads
|
||||
/// (`PermChange`'s file payload) ride the node that consumes them
|
||||
/// ([`NodeKind::WritePermFile`]), not this generic spec.
|
||||
#[derive(Debug, Clone)]
|
||||
/// Submit-time spec for a whole DAG: the group's metadata plus the declared —
|
||||
/// not yet inserted — nodes. Built by `templates.rs`, inserted by
|
||||
/// `JobQueue::submit`.
|
||||
///
|
||||
/// No DAG-level `agent` — every node carries its own (a DAG can span agents),
|
||||
/// and the queue derives per-agent leasing from [`NodeKind::agent`].
|
||||
/// Type-specific payloads (`PermChange`'s file payload) ride the node that
|
||||
/// consumes them ([`NodeKind::WritePermFile`]), not this generic spec.
|
||||
///
|
||||
/// There is no separate per-node spec type: the nodes live in the builder,
|
||||
/// which inserts them itself. A shape that has been declared is therefore
|
||||
/// always insertable — a dangling edge or a cycle cannot be expressed, so
|
||||
/// there is nothing left for a submit-time validation pass to reject.
|
||||
#[derive(Debug)]
|
||||
pub struct DagSpec {
|
||||
pub source: Source,
|
||||
/// Free-form "why".
|
||||
pub reason: String,
|
||||
pub nodes: Vec<NodeSpec>,
|
||||
/// The DAG's declared nodes, with their edges, grouping and resources.
|
||||
pub job: super::Job,
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue