vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

jobq: one insertion entry point, and make it atomic

Browse source 
Three findings from the operator's review, all correct.

1. Two insert_job's. Graph::insert_job had no caller outside hive-jobq's
   own tests -- production only ever went through Scheduler::insert_job.
   It existed because the graph-level one got written first. Deleted; the
   tests moved onto a Scheduler, which is where insertion belongs anyway.

2. insert_job was not atomic, and the previous commit made that worse: a
   forward edge or forward parent surfaced mid-loop, leaving the nodes
   before it in the graph, and resolve_wanted ran after every insert, so
   an unknown handle failed once the whole job was already committed.
   The module documented this under "Partial insertion" instead of fixing
   it -- prose describing a hole is not a design.

   All three are decidable from what the builder holds, so
   check_declaration_order now runs before the first insert and the loop
   indexes ids directly. A malformed job leaves the graph untouched.
   What remains mid-insert is the graph's own rejection (out-of-group
   dep, empty DepWhen); closing that needs a dry-run validate on Graph,
   which is a separate change.

3. DagSpec no longer boxes its recipe: it is generic over the closure,
   which travels from the template that built it straight into submit.
   The box bought type inference, and paying for it costs annotations --
   `|b: &Job|` at each declaration site (the field needs an HRTB, and an
   unannotated closure binds one lifetime) and `+ use<>` on each
   returning signature (or the opaque type captures the caller's borrows).
   Erasure is still needed where several recipe shapes share one type:
   the boxed Declare stays for the executor's append_subgraph, and a test
   table uses an erase() helper.
This commit is contained in:
atlas 2026-08-02 14:39:37 +02:00 • committed by mara
commit f035b63b9a
8 changed files with 211 additions and 176 deletions
Download patch file Download diff file Expand all files Collapse all files

18
hive-c0re/src/job_queue/submit.rs
View file

@ -26,11 +26,11 @@ use std::sync::Arc;
use super::model::{DagSpec, NodeKind};
use super::templates::{RebuildOpts, node, rebuild_nodes};
use super::{Declare, Job, Source, templates};
use super::{Job, Source, templates};
use crate::coordinator::Coordinator;
use crate::lifecycle;
fn submit_and_emit(coord: &Arc<Coordinator>, spec: super::DagSpec) -> u64 {
fn submit_and_emit<F: FnOnce(&Job)>(coord: &Arc<Coordinator>, spec: super::DagSpec<F>) -> u64 {
let id = coord
.job_queue
.submit(spec)
@ -169,7 +169,7 @@ fn restart_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
/// and each keeps its own root, so the per-agent subgraphs are independent and
/// run concurrently, each on its own lease. Rebasing one subgraph's indices
/// onto another's used to be a function.
fn power_dag(source: Source, reason: String, declare: Declare) -> DagSpec {
fn power_dag<F: FnOnce(&Job)>(source: Source, reason: String, declare: F) -> DagSpec<F> {
DagSpec {
source,
reason,
@ -188,12 +188,12 @@ pub(crate) fn stop_spec(
graceful: bool,
source: Source,
reason: String,
) -> DagSpec {
) -> DagSpec<impl FnOnce(&Job) + use<>> {
let targets = targets.to_vec();
power_dag(
source,
reason,
Box::new(move |b| {
Box::new(move |b: &Job| {
for (agent, running) in targets {
stop_chain(b, &agent, graceful, running);
}
@ -211,12 +211,12 @@ pub(crate) fn start_spec(
targets: &[(String, bool, bool)],
source: Source,
reason: String,
) -> DagSpec {
) -> DagSpec<impl FnOnce(&Job) + use<>> {
let targets = targets.to_vec();
power_dag(
source,
reason,
Box::new(move |b| {
Box::new(move |b: &Job| {
for (agent, running, stale) in targets {
start_chain(b, &agent, running, stale);
}
@ -230,12 +230,12 @@ pub(crate) fn restart_spec(
graceful: bool,
source: Source,
reason: String,
) -> DagSpec {
) -> DagSpec<impl FnOnce(&Job) + use<>> {
let targets = targets.to_vec();
power_dag(
source,
reason,
Box::new(move |b| {
Box::new(move |b: &Job| {
for (agent, running) in targets {
restart_chain(b, &agent, graceful, running);
}