vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

refactor(#2949): kill Declare — a running node declares onto its own builder

Browse source 
A node no longer hands back a recipe for the scheduler to replay later. It
declares straight onto a builder it was given, and that builder is inserted
as part of completing the node.

Deleted: `pub type Declare`, `struct NodeOutput` (+ its hand-written `Debug`),
`JobQueue::append_subgraph`. Nothing added to `Dag` / `DagView`.

jobq gains `Scheduler::new_job()` (the only way to obtain a `JobBuilder`) and
`complete_growing(id, outcome, grown)`, which inserts under `id` and *then*
completes it, so a DAG cannot roll terminal while grown work is still pending.
`complete()` and `complete_growing()` share a private `finish()` rather than
one redirecting through the other. The DAG-gone guard lives beside the graph
now, where it cannot be skipped, instead of being a caller-side lookup.

The growth executors return data (`run_meta_lock -> (Vec<String>, RebuildOpts)`,
`run_reconcile -> Option<NodeKind>`) rather than taking the builder: a `&Job`
parameter is live for the whole function body, and `&RefCell<T>` is never
`Send`, so an async fn taking one cannot be spawned. `run_node` threads the
builder by value and hands it back.

A node can now declare work and then fail, which was previously inexpressible.
`grown` is dropped in that case — failure cancel-cascades downstream, so
inserting it would only add nodes to immediately cancel — and the log line
carries `grown_nodes` so the drop is visible.
This commit is contained in:
atlas 2026-08-02 17:20:43 +02:00 • committed by mara
commit 82ef06f445
8 changed files with 388 additions and 344 deletions
Download patch file Download diff file Expand all files Collapse all files

138
hive-c0re/src/job_queue/tests.rs
View file

@ -13,13 +13,19 @@ fn submit<F: FnOnce(&Job)>(q: &JobQueue, spec: DagSpec<F>) -> u64 {
q.submit(spec).expect("valid spec")
}
/// Erase a spec's recipe to the boxed [`Declare`] so specs of *different*
/// shapes can share one type — e.g. a table of `(name, spec)` cases.
/// A spec recipe with its concrete closure type erased. **Test-only** — the
/// module used to export this alias for the executor's growth path too, which
/// is exactly what a node declaring onto its own builder removed: nothing in
/// production stores a recipe to replay later, so nothing needs to box one.
type ErasedRecipe = Box<dyn FnOnce(&Job) + Send>;
/// Erase a spec's recipe so specs of *different* shapes can share one type —
/// e.g. a table of `(name, spec)` cases.
///
/// Production never needs this: each submit path builds one spec and hands it
/// straight to `submit`, so the concrete closure type is known end to end. A
/// test table is the case where several shapes must be one type.
fn erase<F: FnOnce(&Job) + Send + 'static>(spec: DagSpec<F>) -> DagSpec<Declare> {
/// test table is the one case where several shapes must be one type.
fn erase<F: FnOnce(&Job) + Send + 'static>(spec: DagSpec<F>) -> DagSpec<ErasedRecipe> {
DagSpec {
source: spec.source,
reason: spec.reason,
@ -758,19 +764,16 @@ fn a_fanned_out_start_declares_the_lease_and_re_enters_its_reconciles_grant() {
assert_eq!(reconcile.dag_id, id);
assert_eq!(reconcile.kind.as_str(), "reconcile");
// What `run_reconcile` does on observing a down container with wanted=Up.
q.append_subgraph(
id,
Box::new(|b: &Job| {
let kind = NodeKind::Start {
agent: "agent-a".to_owned(),
};
let lease = Resource::Agent(kind.agent().to_owned());
let _ = b.node(kind).needs(lease);
}),
reconcile.node_id,
);
q.complete_node(reconcile.node_id, Ok(()));
// What `run_reconcile` does on observing a down container with wanted=Up:
// declare into the builder it was handed, then hand it back with the
// completion. Same two calls the scheduler makes, in the same order.
let grown = q.new_job();
let kind = NodeKind::Start {
agent: "agent-a".to_owned(),
};
let lease = Resource::Agent(kind.agent().to_owned());
let _ = grown.node(kind).needs(lease);
q.complete_node_growing(reconcile.node_id, Ok(()), grown);
// (a) + (b): the child runs, under the parent that parked in `Finishing`.
let start = claim_one(&q);
@ -850,7 +853,7 @@ fn boot_sweep_nodes_declare_their_own_resources() {
}
#[test]
fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
fn grown_subgraph_roots_on_emitter_and_rebases_local_deps() {
// The startup-sweep mechanism: a `MetaLock` emitter grows one rebuild
// subgraph per stale agent into its OWN DAG. Each subgraph is rooted on
// the emitter and its LOCAL 0-based deps are rebased onto the DAG.
@ -866,31 +869,29 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
});
}),
};
let id = submit(&q, spec);
submit(&q, spec);
let emitter = claim_one(&q);
assert_eq!(emitter.kind.as_str(), "meta_lock");
// Two independent per-agent subgraphs — the REAL production shape the
// sweep MetaLock grows: root MetaSync → root Prebuild → Signal → Drain →
// StopForUpdate → Swap → Reconcile, local 0-based deps. `graceful` must
// match the sweep arm of `run_meta_lock` or this stops tracking production.
let subgraph = |agent: &str| -> Declare {
let agent = agent.to_owned();
Box::new(move |b: &Job| {
templates::rebuild_nodes(
b,
&agent,
templates::RebuildOpts {
relock: true,
graceful: true,
},
None,
);
})
};
// Must append BEFORE completing the emitter (the documented contract).
q.append_subgraph(id, subgraph("a"), emitter.node_id);
q.append_subgraph(id, subgraph("b"), emitter.node_id);
q.complete_node(emitter.node_id, Ok(()));
// Both subgraphs go into the emitter's own builder, exactly as
// `run_meta_lock`'s sweep arm does. Insert-before-complete is no longer the
// caller's job to remember: it is one call, and the ordering is inside it.
let grown = q.new_job();
for agent in ["a", "b"] {
templates::rebuild_nodes(
&grown,
agent,
templates::RebuildOpts {
relock: true,
graceful: true,
},
None,
);
}
q.complete_node_growing(emitter.node_id, Ok(()), grown);
// Still ONE DAG; both subgraph roots become ready once the emitter is
// Done (rooted on it), each on its own agent lease. Their `MetaSync` heads
// take turns on the cap-1 global meta window, so drain those first — what
@ -967,7 +968,7 @@ fn rebuild_chain_nodes_suppress_crash_watch() {
#[test]
fn meta_update_grows_cascade_in_dag() {
// The meta-update `MetaLock` grows one rebuild subgraph per affected
// agent into its OWN DAG (via append_subgraph), not child DAGs.
// agent into its OWN DAG (via the builder it is handed), not child DAGs.
let spec = templates::meta_update(
vec!["nixpkgs".to_owned()],
Source::Manual,
@ -975,26 +976,26 @@ fn meta_update_grows_cascade_in_dag() {
None,
);
let q = JobQueue::new(4);
let id = submit(&q, spec);
submit(&q, spec);
let meta_lock = claim_one(&q);
assert_eq!(meta_lock.kind.as_str(), "meta_lock");
// Simulate the executor growing the cascade in-DAG (`relock = false` — a
// cascade child must not re-lock and revert the parent's bump).
// cascade child must not re-lock and revert the parent's bump). Both
// agents go into the one builder the node was handed, which is what
// `run_meta_lock`'s fanout arm does.
let grown = q.new_job();
for agent in ["alice", "bob"] {
let declare: Declare = Box::new(move |b: &Job| {
templates::rebuild_nodes(
b,
agent,
templates::RebuildOpts {
relock: false,
graceful: false,
},
None,
);
});
q.append_subgraph(id, declare, meta_lock.node_id);
templates::rebuild_nodes(
&grown,
agent,
templates::RebuildOpts {
relock: false,
graceful: false,
},
None,
);
}
q.complete_node(meta_lock.node_id, Ok(()));
q.complete_node_growing(meta_lock.node_id, Ok(()), grown);
// Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root
// on the MetaLock, each on its own agent lease. The per-agent `MetaSync`
// heads serialize on the global meta window (they commit to the meta repo);
@ -1233,8 +1234,8 @@ fn cancelled_power_op_runs_no_compensating_node() {
for graceful in [false, true] {
for running in [false, true] {
let targets = vec![("agent-a".to_owned(), running)];
// Erased to `DagSpec<Declare>`: three different recipe types have to
// sit in one array.
// Erased to one boxed recipe type: three different recipe types
// have to sit in one array.
let cases = [
(
"restart",
@ -1420,16 +1421,14 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
let apply = claim_one(&q);
assert!(matches!(apply.kind, NodeKind::DeployApply { .. }));
// Mirrors the scheduler: the executor's `NodeOutput` subgraphs are grafted
// BEFORE the emitting node is completed. Completing first would settle the
// apply node `Done` with nothing under it, opening the tail's `AfterAny`
// gate immediately and letting the deploy "finish" before it had built.
q.append_subgraph(
id,
templates::deploy_rebuild_nodes("agent-a", 11),
apply.node_id,
);
q.complete_node(apply.node_id, Ok(()));
// Mirrors the scheduler. The graft lands BEFORE the emitting node settles,
// and that ordering is now structural rather than a rule this call site has
// to follow: completing first would settle the apply node `Done` with
// nothing under it, opening the tail's `AfterAny` gate immediately and
// letting the deploy "finish" before it had built.
let grown = q.new_job();
templates::deploy_rebuild_nodes(&grown, "agent-a", 11);
q.complete_node_growing(apply.node_id, Ok(()), grown);
// The grafted chain runs in rebuild order. `claim_one` asserts exactly one
// claimable node at each step, which also proves the `AfterAny` tail stays
@ -1481,12 +1480,9 @@ fn deploy_dag_skips_finalize_but_still_tails_a_failed_graft() {
let verify = claim_one(&q);
q.complete_node(verify.node_id, Ok(()));
let apply = claim_one(&q);
q.append_subgraph(
id,
templates::deploy_rebuild_nodes("agent-a", 13),
apply.node_id,
);
q.complete_node(apply.node_id, Ok(()));
let grown = q.new_job();
templates::deploy_rebuild_nodes(&grown, "agent-a", 13);
q.complete_node_growing(apply.node_id, Ok(()), grown);
for expected in ["meta_sync", "prebuild", "stop_for_update"] {
let c = claim_one(&q);