vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

job_queue: test grafted work by declaring it, not by grafting it

Browse source 
Reproducing the runtime path is not needed to test what the runtime path
declares. What DeployApply grows is deploy_rebuild_nodes' output, and
that is a pure declaration -- so declare it directly and read the shape,
instead of running a deploy far enough to graft it.

That makes the finalize gate visible without any of the walking:
finalize_deploy declares AfterOk on both prebuild and reconcile, so
either root failing skips it, and reconcile hangs off prebuild with
AfterAny so a failed swap still reaches it.

deploy_dag_skips_finalize_but_still_tails_a_failed_graft is gone with
it. Its three declared claims are rows in that table, and its runtime
claims belong to hive-jobq: cascade on failure, roll-up, and first_error
digging past a group root that rolled up Failed while carrying no error
of its own -- which is why the DAG reports the swap's error rather than
nothing.

The grafting mechanism itself is also hive-jobq's and tested there: work
lands under the emitter before it settles, and the emitter parks in
Finishing so a downstream AfterAny gate stays shut while its new
children run.
This commit is contained in:
atlas 2026-08-02 20:12:30 +02:00 • committed by mara
commit e9a84310fe
1 changed files with 58 additions and 101 deletions
Download patch file Download diff file Expand all files Collapse all files

159
hive-c0re/src/job_queue/tests.rs
View file

@ -1642,116 +1642,73 @@ fn deploy_dag_runs_phases_in_order_and_tails_a_failed_apply() {
/// a resource its own DAG owns and deadlock — this test is what pins that down.
#[test]
fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
// What `DeployApply` grows is `deploy_rebuild_nodes`' output, and that is a
// pure declaration — so it is declared here directly rather than by running
// a deploy far enough to graft it. **Reproducing the runtime path is not
// needed to test what the runtime path declares.**
//
// The grafting mechanism itself is hive_jobq's and tested there: the work
// lands under the emitter *before* it settles, and the emitter parks in
// `Finishing` so a downstream `AfterAny` gate stays shut while the new
// children run (`a_completing_node_grows_the_work_it_declared`,
// `parent_parks_in_finishing_until_children_roll_up`).
let q = JobQueue::new(1);
let id = submit(
&q,
templates::approval_deploy("agent-a", 11, "approval 11".to_owned()),
DagSpec {
source: Source::Manual,
reason: "deploy graft".to_owned(),
declare: Box::new(|b: &Job| templates::deploy_rebuild_nodes(b, "agent-a", 11)),
},
);
let root = claim_one(&q);
assert!(matches!(root.kind, NodeKind::DeployWindow { .. }));
q.complete_node(root.node_id, Ok(()));
let verify = claim_one(&q);
q.complete_node(verify.node_id, Ok(()));
let apply = claim_one(&q);
assert!(matches!(apply.kind, NodeKind::DeployApply { .. }));
// Mirrors the scheduler. The graft lands BEFORE the emitting node settles,
// and that ordering is now structural rather than a rule this call site has
// to follow: completing first would settle the apply node `Done` with
// nothing under it, opening the tail's `AfterAny` gate immediately and
// letting the deploy "finish" before it had built.
let grown = q.new_job();
templates::deploy_rebuild_nodes(&grown, "agent-a", 11);
q.complete_node_growing(apply.node_id, Ok(()), grown);
// The grafted chain runs in rebuild order. `claim_one` asserts exactly one
// claimable node at each step, which also proves the `AfterAny` tail stays
// shut: `DeployApply` is `Finishing` (not terminal) while its new children
// run, and `Finishing` satisfies neither dep kind.
for expected in [
"meta_sync",
"prebuild",
"stop_for_update",
"swap",
"post_swap",
"reconcile",
] {
let c = claim_one(&q);
assert_eq!(c.kind.as_str(), expected, "grafted phase order");
q.complete_node(c.node_id, Ok(()));
}
let finalize = claim_one(&q);
assert!(
matches!(finalize.kind, NodeKind::FinalizeDeploy { .. }),
"the deploy tag is planted only after the rebuild came up clean"
);
q.complete_node(finalize.node_id, Ok(()));
let tail = claim_one(&q);
assert!(matches!(tail.kind, NodeKind::DeployTail { .. }));
q.complete_node(tail.node_id, Ok(()));
settle_approval_tail(&q, 11, TerminalState::Done);
assert_eq!(state_of(&q, id), State::Done);
}
/// A failure *inside* the grafted rebuild is the failure mode the subgraph
/// growth introduces: the deploy is already merged and the container half-swapped.
/// `FinalizeDeploy` must be cancel-cascaded (its `AfterOk` gate never opens) so
/// no `deployed/<id>` tag is planted, while the tail still runs to compensate.
/// `Reconcile` is deliberately still reached — it boots the container back up.
#[test]
fn deploy_dag_skips_finalize_but_still_tails_a_failed_graft() {
let q = JobQueue::new(1);
let id = submit(
&q,
templates::approval_deploy("agent-a", 13, "approval 13".to_owned()),
);
let root = claim_one(&q);
q.complete_node(root.node_id, Ok(()));
let verify = claim_one(&q);
q.complete_node(verify.node_id, Ok(()));
let apply = claim_one(&q);
let grown = q.new_job();
templates::deploy_rebuild_nodes(&grown, "agent-a", 13);
q.complete_node_growing(apply.node_id, Ok(()), grown);
for expected in ["meta_sync", "prebuild", "stop_for_update"] {
let c = claim_one(&q);
assert_eq!(c.kind.as_str(), expected);
q.complete_node(c.node_id, Ok(()));
}
let swap = claim_one(&q);
assert_eq!(swap.kind.as_str(), "swap");
q.complete_node(swap.node_id, Err("profile swap failed".into()));
// `Reconcile` hangs off `Prebuild` with `AfterAny`, so a failed swap still
// reaches it — bringing the container back up is exactly what it's for.
let reconcile = claim_one(&q);
assert_eq!(reconcile.kind.as_str(), "reconcile");
q.complete_node(reconcile.node_id, Ok(()));
let tail = claim_one(&q);
assert!(
matches!(tail.kind, NodeKind::DeployTail { .. }),
"finalize is cancel-cascaded, so the tail is the next claimable node"
);
q.complete_node(tail.node_id, Ok(()));
settle_approval_tail(&q, 13, TerminalState::Failed);
assert_eq!(state_of(&q, id), State::Failed);
assert_eq!(
q.first_error(id).as_deref(),
Some("profile swap failed"),
"the tail annotates failed/<id> with this — and it is also what
`exec::failure_reason` falls back to, since the tail's own dep is a
group root that rolled up Failed and so carries no error itself"
declared_shape(&q, id),
vec![
row("meta_sync", None, &[]),
row("prebuild", None, &[("meta_sync", "done")]),
row("stop_for_update", Some("prebuild"), &[]),
row("swap", Some("stop_for_update"), &[]),
row("post_swap", Some("stop_for_update"), &[("swap", "done")]),
row("reconcile", None, &[("prebuild", "done|failed|skipped")]),
// The deploy tag is planted only after the rebuild came up clean:
// `AfterOk` on **both** roots, so either one failing skips it. That
// pair of edges is the whole "skips finalize on a failed graft"
// behaviour — no run needed to see it.
row(
"finalize_deploy",
None,
&[("prebuild", "done"), ("reconcile", "done")]
),
]
);
}
// `deploy_dag_skips_finalize_but_still_tails_a_failed_graft` lived here.
//
// A failure *inside* the grafted rebuild is the failure mode subgraph growth
// introduces: the deploy is already merged and the container half-swapped, so
// `FinalizeDeploy` must be cancel-cascaded (no `deployed/<id>` tag planted)
// while the tail still runs to compensate, and `Reconcile` is deliberately
// still reached — it boots the container back up.
//
// Every declared half of that is asserted by
// `deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it`, which reads
// `deploy_rebuild_nodes`' shape directly:
//
// - "a failed swap still reaches Reconcile" is the `reconcile` row's
// `AfterAny` edge on `prebuild` (`done|failed|skipped`);
// - "finalize is cancel-cascaded" is `finalize_deploy`'s `AfterOk` pair —
// either root failing skips it;
// - "the tail still runs" is `deploy_tail`'s own `done|failed|skipped` edge
// on apply, asserted in the `approval_deploy` table above.
//
// The runtime halves are hive_jobq's: cascade on failure, roll-up, and
// `first_error` digging past a group root that rolled up `Failed` while
// carrying no error of its own (`first_error_skips_a_rolled_up_failure_
// carrying_no_error`). That last one is why the DAG reports "profile swap
// failed" rather than nothing — the mechanism, not this shape.
// `deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails` lived here.
//
// A pre-merge rejection (drift gate, eval failure) cancel-cascades the