vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix(#3034): run prebuild beside the graceful-stop window, braced by AgentWindow

Browse source 
The graceful path hung `Signal` under `Prebuild`, and a child only starts once
its parent's own logic completes — so the drain window waited for the entire nix
build before the agent was even asked to checkpoint. Up to the full
GRACEFUL_STOP_TIMEOUT hidden behind the build, per agent, on every boot sweep.
`Prebuild` needs the build slot and `Signal`/`Drain` need the agent lease, so
there was never any contention to justify the nesting.

Adds `NodeKind::AgentWindow`, a pure resource holder in the `DeployWindow`
pattern. It declares the build slot and the agent lease atomically and holds
both for its whole subtree; `Prebuild` and the quiesce chain hang off it as
siblings and run concurrently. `StopForUpdate` is AfterOk *both*, so the
container still goes down only once the build is ready and the agent has
checkpointed — running the drain early is the win, stopping early would just be
downtime.

Two things this deliberately reverses, both documented in place:

* The coordinated children now declare no resources. `templates.rs`'s module doc
  said each node must declare its own, precisely so one running under a holding
  ancestor could not get away with declaring nothing. That rule stands; the
  brace is named as its one exception, because declaring a resource means "I
  need this exclusively" and the lease is single-unit — two siblings that both
  declared it could never overlap, which is the whole point of the shape.
* `rebuild_chain_declares_the_slot_where_the_nix_work_is` asserted the old
  principle in its name. Renamed to `..._declares_its_resources_on_the_brace`
  rather than left saying something the code no longer does.

Hoisting the build slot is not new serialisation: a unit is held until the
acquirer's subtree settles, and everything downstream already sat inside
`Prebuild`, so the slot already spanned the entire rebuild.

`graceful_rebuild_chain_drains_before_stopping` now asserts full rows instead of
the kind list — the kind list is identical whether the chain runs beside the
build or under it, so it could not see this bug. Verified by mutation: re-nesting
`Signal` under `Prebuild` fails exactly that one test out of 317.
This commit is contained in:
atlas 2026-08-04 12:51:45 +02:00
commit 31a1853a45
4 changed files with 295 additions and 140 deletions
Download patch file Download diff file Expand all files Collapse all files

222
hive-c0re/src/job_queue/tests.rs
View file

@ -390,23 +390,40 @@ fn rebuild_chain_is_declared_serial() {
// observe an order that is fully declared the moment `submit` returns.
//
// ⚠️ The old name was also wrong about the mechanism, and reading it rather
// than the graph is how you'd stay wrong: **only half this chain is dep
// edges.** `stop_for_update` and `swap` declare no deps at all — they are
// ordered by *parent nesting* ("a node's sub-nodes run after its own
// logic"). Both axes are asserted below because a template can break either
// one independently.
// than the graph is how you'd stay wrong: **only part of this chain is dep
// edges.** `swap` declares no deps at all — it is ordered by *parent
// nesting* ("a node's sub-nodes run after its own logic"). Both axes are
// asserted below because a template can break either one independently.
//
// ⚠️ "Serial" is now about the *declared* order, not about concurrency:
// `prebuild` and the graceful quiesce chain are siblings under the brace and
// run **in parallel** (see `graceful_rebuild_chain_drains_before_stopping`).
// The non-graceful shape asserted here has no quiesce chain, so nothing here
// is concurrent — but the name would mislead about the graceful one.
let q = JobQueue::new(1);
let id = submit(&q, "r", |builder| rebuild(builder, "agent-a"));
assert_eq!(
declared_shape(&q, id),
vec![
row("meta_sync", None, &[]),
row("prebuild", None, &[("meta_sync", "done")]),
// No dep: ordered by hanging under `prebuild`.
row("stop_for_update", Some("prebuild"), &[]),
// The brace: holds the lease + slot for everything nested below it.
row("agent_window", None, &[("meta_sync", "done")]),
// No dep: ordered by hanging under the brace.
row("prebuild", Some("agent_window"), &[]),
// A real dep, not nesting: the container must not go down until the
// build that replaces it has succeeded.
row(
"stop_for_update",
Some("agent_window"),
&[("prebuild", "done")]
),
row("swap", Some("stop_for_update"), &[]),
row("post_swap", Some("stop_for_update"), &[("swap", "done")]),
row("reconcile", None, &[("prebuild", "done|failed|skipped")]),
row(
"reconcile",
None,
&[("agent_window", "done|failed|skipped")]
),
// The tail pair. The ok tail needs every root to succeed; the !ok
// tail hangs off the ok tail's *elimination* (`skipped`), which is
// what makes exactly one of them run.
@ -415,7 +432,7 @@ fn rebuild_chain_is_declared_serial() {
None,
&[
("meta_sync", "done"),
("prebuild", "done"),
("agent_window", "done"),
("reconcile", "done"),
],
),
@ -425,7 +442,7 @@ fn rebuild_chain_is_declared_serial() {
&[
("emit_rebuilt", "skipped"),
("meta_sync", "done|failed|skipped"),
("prebuild", "done|failed|skipped"),
("agent_window", "done|failed|skipped"),
("reconcile", "done|failed|skipped"),
],
),
@ -433,12 +450,17 @@ fn rebuild_chain_is_declared_serial() {
);
}
/// The boot sweep's graceful shape: the agent gets `Signal` → `Drain` to
/// finish its turn before `StopForUpdate` takes the container down. `Signal`
/// *parents* the rest of the stop rather than sitting beside it, so the agent
/// lease is held continuously across the whole bounce — as siblings, each of
/// `Signal` / `Drain` / `StopForUpdate` would acquire the lease separately and
/// leave a window for another DAG to claim the agent mid-stop.
/// The boot sweep's graceful shape: the agent gets `Signal` → `Drain` to finish
/// its turn before `StopForUpdate` takes the container down — **concurrently
/// with the nix build**, which is the whole reason the brace exists. The drain
/// window costs nothing it doesn't already cost; nesting it under `Prebuild`
/// used to hide the entire `GRACEFUL_STOP_TIMEOUT` behind the build.
///
/// Lease continuity across the bounce is preserved by `AgentWindow` holding the
/// lease above all of them, which is what makes them safe as siblings — the
/// older shape had to nest `Signal` over the rest of the stop for exactly that
/// reason, since otherwise each would acquire the lease separately and leave a
/// window for another DAG to claim the agent mid-stop.
#[test]
fn graceful_rebuild_chain_drains_before_stopping() {
let q = JobQueue::new(1);
@ -451,30 +473,47 @@ fn graceful_rebuild_chain_drains_before_stopping() {
},
)
.expect("valid shape");
// Asserted as full rows, not just kinds: the kind list is identical whether
// the quiesce chain runs beside the build or nested under it, so a
// kind-only assertion cannot see the bug this shape exists to fix.
assert_eq!(
declared_shape(&q, id)
.iter()
.map(|d| d.kind)
.collect::<Vec<_>>(),
declared_shape(&q, id),
vec![
"meta_sync",
"prebuild",
// The graceful window goes between the build and the stop: the
// agent gets its turn to finish before the container goes down.
"signal",
"drain",
"stop_for_update",
"swap",
"post_swap",
"reconcile",
row("meta_sync", None, &[]),
row("agent_window", None, &[("meta_sync", "done")]),
row("prebuild", Some("agent_window"), &[]),
// 🎯 **The fix, and the thing to guard.** `signal` hangs off the
// *brace*, not off `prebuild`, and declares no dep on it — so the
// drain window runs concurrently with the nix build instead of
// behind it. Both halves matter: re-parenting it under `prebuild`
// OR adding an `AfterOk(prebuild)` edge would each silently restore
// the original defect (up to GRACEFUL_STOP_TIMEOUT hidden behind
// every agent's build, on every boot sweep).
row("signal", Some("agent_window"), &[]),
row("drain", Some("signal"), &[]),
// The container still goes down only when *both* are ready: the
// build succeeded and the agent has checkpointed. Running the drain
// early is the win; stopping early would just be downtime.
row(
"stop_for_update",
Some("agent_window"),
&[("prebuild", "done"), ("drain", "done")]
),
row("swap", Some("stop_for_update"), &[]),
row("post_swap", Some("stop_for_update"), &[("swap", "done")]),
row(
"reconcile",
None,
&[("agent_window", "done|failed|skipped")]
),
],
"graceful inserts signal + drain ahead of the stop, and nothing else"
"graceful runs signal + drain beside the build, and stops only after both"
);
}
/// The non-graceful shape is the default everywhere except the boot sweep:
/// a manual rebuild, a meta-update cascade child and a deploy must NOT spend a
/// drain window, so `StopForUpdate` still hangs straight off `Prebuild`.
/// drain window, so `StopForUpdate` waits only on `Prebuild`.
#[test]
fn non_graceful_rebuild_has_no_signal_or_drain() {
// Read the shape off the queue rather than out of a node list: a declared
@ -491,32 +530,38 @@ fn non_graceful_rebuild_has_no_signal_or_drain() {
.collect::<Vec<_>>(),
vec![
"meta_sync",
"agent_window",
"prebuild",
"stop_for_update",
"swap",
"post_swap",
"reconcile"
],
"exactly six nodes, and neither of them is signal or drain"
"exactly seven nodes, and none of them is signal or drain"
);
}
// ---- build slots ----
#[test]
fn rebuild_chain_declares_the_slot_where_the_nix_work_is() {
// Was `fifo_fairness_for_the_slot`, which submitted three rebuilds and
// drove one to completion to watch the freed slot go to the earlier
// waiter. **That fairness guarantee is hive_jobq's**, and it had no test
// there at all — its claim primitive scans nodes in insertion order and
// takes the first satisfiable one, and nothing pinned that. It does now:
// `a_contended_resource_goes_to_the_oldest_waiter`.
fn rebuild_chain_declares_its_resources_on_the_brace() {
// Was `rebuild_chain_declares_the_slot_where_the_nix_work_is` (and before
// that `fifo_fairness_for_the_slot`).
//
// What is c0re's is *which* nodes contend for the slot in the first place,
// and that is a declaration. "Uniform hold across the chain" then follows
// from the parent nesting asserted in `rebuild_chain_is_declared_serial`:
// a resource unit is held for the acquirer's whole subtree, so the slot
// `Prebuild` takes covers `StopForUpdate` → `Swap` → `PostSwap` beneath it.
// ⚠️ **This rename is a reversal, not tidying.** The old assertion read
// "the slot follows the nix work and the lease follows the container": each
// node declared the resource it personally needed, deliberately, so that the
// requirement belonged to the node rather than to one DAG shape. The brace
// inverts that for a coordinated subtree, and the reason is forced:
// declaring a resource means *"I need this exclusively"* and the lease is
// single-unit, so `Prebuild` and the quiesce chain — which exist to run
// **concurrently** — could never overlap if both declared it. One holder
// above them speaks for the subtree. `templates.rs`'s module doc carries the
// general rule and states this as its one sanctioned exception.
//
// (`hive_jobq` owns slot *fairness*, pinned there by
// `a_contended_resource_goes_to_the_oldest_waiter`. What is c0re's is
// *which* nodes contend in the first place — a declaration, asserted here.)
let q = JobQueue::new(1);
let id = submit(&q, "r", |builder| rebuild(builder, "agent-a"));
let res = |kind: &str| declared_resources(&q, node_of(&q, id, kind));
@ -525,6 +570,7 @@ fn rebuild_chain_declares_the_slot_where_the_nix_work_is() {
assert_eq!(
[
res("meta_sync"),
res("agent_window"),
res("prebuild"),
res("stop_for_update"),
res("swap"),
@ -532,21 +578,28 @@ fn rebuild_chain_declares_the_slot_where_the_nix_work_is() {
],
[
// The meta preamble takes the global window and *nothing else* —
// no slot (it does no nix work) and no lease.
// no slot (it does no nix work) and no lease. It stays a sibling
// root so that hive-global window is not held across any build.
vec![Resource::MetaWindow],
// The nix build is the slot-needer, and takes **no lease**. That is
// what lets a prebuild overlap another DAG on the same agent: the
// container is still up and untouched while it builds.
vec![Resource::BuildSlot],
// The lease starts here — the first node that touches the
// container — and not one node earlier.
vec![agent()],
// Swap needs both. It re-enters the slot its `Prebuild` ancestor
// holds rather than acquiring a second unit.
// The brace takes both, atomically, and holds them for its whole
// subtree. Hoisting the slot here is not a widening: it already
// spanned the entire rebuild when `Prebuild` held it, because a unit
// is held until the acquirer's subtree settles and everything below
// was inside `Prebuild`.
vec![Resource::BuildSlot, agent()],
// The coordinated children declare nothing and re-enter the brace's
// grants. An empty vec here is the *point* of the shape, not an
// omission — if any of these regains a declaration it will silently
// stop running in parallel with its siblings.
vec![],
vec![],
vec![],
// `Reconcile` is the exception that stays: a top-level root outside
// the brace, so it takes a genuinely fresh lease after the window
// has released.
vec![agent()],
],
"the slot follows the nix work and the lease follows the container"
"the brace declares for the subtree; coordinated children declare nothing"
);
}
@ -652,11 +705,20 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() {
vec![
row("set_wanted", None, &[]),
row("meta_sync", None, &[("set_wanted", "done")]),
row("prebuild", None, &[("meta_sync", "done")]),
row("stop_for_update", Some("prebuild"), &[]),
row("agent_window", None, &[("meta_sync", "done")]),
row("prebuild", Some("agent_window"), &[]),
row(
"stop_for_update",
Some("agent_window"),
&[("prebuild", "done")]
),
row("swap", Some("stop_for_update"), &[]),
row("post_swap", Some("stop_for_update"), &[("swap", "done")]),
row("reconcile", None, &[("prebuild", "done|failed|skipped")]),
row(
"reconcile",
None,
&[("agent_window", "done|failed|skipped")]
),
],
"a stale agent gets the whole rebuild chain wedged between intent and \
convergence — same DAG, same head kind, more in the middle"
@ -818,15 +880,23 @@ fn rebuild_reconcile_waits_for_the_whole_build_subtree() {
//
// The interesting claim was "Reconcile must wait for PostSwap, not race
// it" — and it does *not* come from an edge between them. `reconcile` deps
// `AfterAny(prebuild)`, while `post_swap` sits inside prebuild's subtree
// (post_swap → stop_for_update → prebuild). A parent is not terminal until
// its subtree is, so prebuild cannot satisfy that edge while post_swap is
// outstanding. **The ordering is the parent chain, not a dependency.**
// `AfterAny(agent_window)`, while `post_swap` sits inside the brace's
// subtree (post_swap → stop_for_update → agent_window). A parent is not
// terminal until its subtree is, so the brace cannot satisfy that edge while
// post_swap is outstanding. **The ordering is the parent chain, not a
// dependency.**
//
// Both facts are asserted in `rebuild_chain_is_declared_serial`; this test
// states the derived property explicitly because the indirection is the
// easy thing to break — someone flattening the chain would keep every edge
// and still lose the guarantee.
//
// 🎯 That warning earned itself: introducing `AgentWindow` **did** flatten
// this chain (`stop_for_update` moved off `prebuild` and onto the brace),
// and the guarantee survives only because the roll-up point moved with it.
// The same roll-up rule is also what makes the obvious-looking alternative
// — `stop_for_update` `AfterOk` `prebuild` *with* `swap` still nested under
// `prebuild` — a **cycle**: the two would wait on each other.
let q = JobQueue::new(1);
let id = submit(&q, "r", |builder| rebuild(builder, "agent-a"));
let shape = declared_shape(&q, id);
@ -838,15 +908,15 @@ fn rebuild_reconcile_waits_for_the_whole_build_subtree() {
.parent
};
assert_eq!(parent_of("post_swap"), Some("stop_for_update"));
assert_eq!(parent_of("stop_for_update"), Some("prebuild"));
assert_eq!(parent_of("stop_for_update"), Some("agent_window"));
assert_eq!(
shape
.iter()
.find(|d| d.kind == "reconcile")
.expect("reconcile node")
.after,
vec![("prebuild", "done|failed|skipped".to_owned())],
"reconcile gates on prebuild's roll-up, which covers the whole build \
vec![("agent_window", "done|failed|skipped".to_owned())],
"reconcile gates on the brace's roll-up, which covers the whole \
subtree — including post_swap — and runs on failure too"
);
}
@ -1222,11 +1292,20 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
declared_shape(&q, id),
vec![
row("meta_sync", None, &[]),
row("prebuild", None, &[("meta_sync", "done")]),
row("stop_for_update", Some("prebuild"), &[]),
row("agent_window", None, &[("meta_sync", "done")]),
row("prebuild", Some("agent_window"), &[]),
row(
"stop_for_update",
Some("agent_window"),
&[("prebuild", "done")]
),
row("swap", Some("stop_for_update"), &[]),
row("post_swap", Some("stop_for_update"), &[("swap", "done")]),
row("reconcile", None, &[("prebuild", "done|failed|skipped")]),
row(
"reconcile",
None,
&[("agent_window", "done|failed|skipped")]
),
// The deploy tag is planted only after the rebuild came up clean:
// `AfterOk` on **both** roots, so either one failing skips it. That
// pair of edges is the whole "skips finalize on a failed graft"
@ -1234,7 +1313,7 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
row(
"finalize_deploy",
None,
&[("prebuild", "done"), ("reconcile", "done")]
&[("agent_window", "done"), ("reconcile", "done")]
),
]
);
@ -1456,6 +1535,7 @@ fn perm_change_shape_prefixes_rebuild_chain() {
vec![
"write_perm_file",
"meta_sync",
"agent_window",
"prebuild",
"stop_for_update",
"swap",