vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix(#3034): run prebuild beside the graceful-stop window, braced by AgentWindow

Browse source 
The graceful path hung `Signal` under `Prebuild`, and a child only starts once
its parent's own logic completes — so the drain window waited for the entire nix
build before the agent was even asked to checkpoint. Up to the full
GRACEFUL_STOP_TIMEOUT hidden behind the build, per agent, on every boot sweep.
`Prebuild` needs the build slot and `Signal`/`Drain` need the agent lease, so
there was never any contention to justify the nesting.

Adds `NodeKind::AgentWindow`, a pure resource holder in the `DeployWindow`
pattern. It declares the build slot and the agent lease atomically and holds
both for its whole subtree; `Prebuild` and the quiesce chain hang off it as
siblings and run concurrently. `StopForUpdate` is AfterOk *both*, so the
container still goes down only once the build is ready and the agent has
checkpointed — running the drain early is the win, stopping early would just be
downtime.

Two things this deliberately reverses, both documented in place:

* The coordinated children now declare no resources. `templates.rs`'s module doc
  said each node must declare its own, precisely so one running under a holding
  ancestor could not get away with declaring nothing. That rule stands; the
  brace is named as its one exception, because declaring a resource means "I
  need this exclusively" and the lease is single-unit — two siblings that both
  declared it could never overlap, which is the whole point of the shape.
* `rebuild_chain_declares_the_slot_where_the_nix_work_is` asserted the old
  principle in its name. Renamed to `..._declares_its_resources_on_the_brace`
  rather than left saying something the code no longer does.

Hoisting the build slot is not new serialisation: a unit is held until the
acquirer's subtree settles, and everything downstream already sat inside
`Prebuild`, so the slot already spanned the entire rebuild.

`graceful_rebuild_chain_drains_before_stopping` now asserts full rows instead of
the kind list — the kind list is identical whether the chain runs beside the
build or under it, so it could not see this bug. Verified by mutation: re-nesting
`Signal` under `Prebuild` fails exactly that one test out of 317.
This commit is contained in:
atlas 2026-08-04 12:51:45 +02:00
commit 31a1853a45
4 changed files with 295 additions and 140 deletions
Download patch file Download diff file Expand all files Collapse all files

161
hive-c0re/src/job_queue/templates.rs
View file

@ -6,9 +6,24 @@
//! happened to run under an ancestor already holding the resource could get
//! away with declaring nothing.
//!
//! **The one sanctioned exception is a brace** — a pure-resource-holder root
//! ([`NodeKind::AgentWindow`], [`NodeKind::DeployWindow`]) that declares for a
//! subtree of nodes coordinated with each other, which then declare nothing.
//! This is the opposite of the failure above, not a relapse into it: there the
//! requirement was *implicit*, inferred from a kind and true only by accident of
//! placement; here it is declared explicitly on one node, and the omission below
//! it is deliberate and documented on the brace.
//!
//! It has to work this way, because declaring a resource means *"I need this
//! exclusively"* and the agent lease is single-unit: **two siblings that both
//! declared it could never run concurrently.** So for a subtree whose whole
//! point is concurrency, declaring the requirement truthfully on every node and
//! running those nodes in parallel are mutually exclusive. The brace is how the
//! shape says "this subtree is coordinated, one holder speaks for it".
//!
//! ```text
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
//! rebuild(a) graceful: … Prebuild(a) → Signal(a) → Drain(a) → StopForUpdate(a) → … [boot sweep only]
//! rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) } →(any) Reconcile(a)
//! rebuild(a) graceful: … AgentWindow(a){ Prebuild(a) ∥ Signal(a) → Drain(a); both →(ok) StopForUpdate(a) → … } [boot sweep only]
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
//! perm-change(a): WritePermFile(a) → «rebuild subgraph»
//! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a»
@ -132,9 +147,10 @@ pub(crate) fn fanned_out_mechanical(builder: &JobBuilder, kind: NodeKind) {
pub(crate) struct RebuildRoots<'a> {
/// The meta-repo preamble.
pub meta_sync: Handle<'a>,
/// The build root — its roll-up carries the whole
/// `StopForUpdate` → `Swap` → `PostSwap` subtree.
pub prebuild: Handle<'a>,
/// The brace holding the agent lease and the build slot — its roll-up
/// carries the whole mechanical subtree (`Prebuild`, the quiesce chain,
/// `StopForUpdate` → `Swap` → `PostSwap`).
pub agent_window: Handle<'a>,
/// The recovery/convergence tail root.
pub reconcile: Handle<'a>,
}
@ -142,40 +158,50 @@ pub(crate) struct RebuildRoots<'a> {
impl<'a> RebuildRoots<'a> {
/// The three roots as a slice, for edging a tail onto all of them.
fn all(self) -> [Handle<'a>; 3] {
[self.meta_sync, self.prebuild, self.reconcile]
[self.meta_sync, self.agent_window, self.reconcile]
}
}
/// The rebuild node subtree (nested, three group roots). `after`, when given, is
/// the node this subgraph chains behind. Structure:
/// The rebuild node subtree (three group roots). `after`, when given, is the
/// node this subgraph chains behind. Structure:
/// - `MetaSync` (**root**): the meta-repo preamble (dir prep, agent sync,
/// optional relock). Owns the global `MetaWindow` — and *only* for its own
/// short duration, which is why it is a sibling root rather than `Prebuild`'s
/// short duration, which is why it is a sibling root rather than the brace's
/// parent: a resource is held across the holder's whole subtree, so parenting
/// the build under it would extend a hive-global window over every rebuild's
/// nix build.
/// - `Prebuild` (**root**): `AfterOk` `MetaSync`. Owns the build slot for the
/// whole mechanical subtree below it. Lease-exempt — the nix build overlaps
/// other DAGs on the same agent.
/// - the **stop root** (child of `Prebuild`): owns the agent lease and runs once
/// `Prebuild` reaches `Finishing` (parent gate). Non-graceful that is
/// `StopForUpdate` itself; graceful it is `Signal`, with `Drain` and then
/// `StopForUpdate` as its children so the lease stays continuous across the
/// whole stop — siblings would each take the lease separately and leave a gap
/// another DAG could claim the agent in, mid-bounce.
/// - `Swap` (child of `StopForUpdate`): borrows the agent lease from its
/// ancestors and the build slot from `Prebuild` — both continuous.
/// - `PostSwap` (child of `StopForUpdate`): the swap's Ok-only bookkeeping tail
/// (rev marker, forge/matrix sync, kick, rescan), `AfterOk` its sibling
/// `Swap`.
/// - `Reconcile` (**last, root**): `AfterAny` `Prebuild`, which rolls up
/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has
/// settled — so `Reconcile` runs after the swap regardless of outcome, and as
/// a top-level root it survives the cancel-cascade of a failed `Prebuild`
/// (recovery-start invariant, which also covers a failed `MetaSync`: that
/// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It
/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to
/// the persisted `wanted` idempotently.
/// the rebuild under it would extend a hive-global window over every
/// rebuild's nix build.
/// - `AgentWindow` (**root**): `AfterOk` `MetaSync`. The brace — declares the
/// build slot *and* the agent lease, atomically, and holds both for the whole
/// subtree. Everything below it declares **nothing** and re-enters these
/// grants.
/// - `Prebuild` and the quiesce chain are **siblings under the brace, and run
/// concurrently.** That is the point of the brace: they contend for different
/// resources (slot vs. agent), so nesting the stop under the build — as this
/// template did — hid the entire graceful-stop timeout behind the nix build,
/// per agent, on every sweep.
/// - the **quiesce chain** (graceful only): `Signal` then `Drain` as its child.
/// Asks the agent to checkpoint and waits for it to go quiet; the container
/// is still *up* throughout.
/// - `StopForUpdate` (child of the brace): `AfterOk` **both** `Prebuild` and
/// `Drain`. Waiting on the build is deliberate — running the drain window
/// early is the win, taking the container *down* early would be pure
/// downtime. This is the node that actually stops the container.
/// - `Swap` (child of `StopForUpdate`), then `PostSwap` (`AfterOk` its sibling
/// `Swap`): the Ok-only bookkeeping tail (rev marker, forge/matrix sync,
/// kick, rescan).
/// - `Reconcile` (**last, root**): `AfterAny` `AgentWindow`, which rolls up
/// terminal only once its whole subtree has settled — so `Reconcile` runs
/// after the swap regardless of outcome, and as a top-level root it survives
/// the cancel-cascade of a failed brace (recovery-start invariant, which also
/// covers a failed `MetaSync`: that cancel-cascades the brace, i.e. terminal,
/// so the tail still runs). It takes a fresh lease; the tiny gap is harmless —
/// `Reconcile` converges to the persisted `wanted` idempotently.
///
/// The lease-continuity argument that used to justify nesting the stop chain
/// (*"siblings would each take the lease separately and leave a gap another DAG
/// could claim the agent in, mid-bounce"*) is **satisfied by the brace instead**
/// — one holder above them all, so siblings share one continuous grant and
/// there is no gap to claim. That is what makes flattening them safe.
fn rebuild_subtree<'a>(
builder: &'a JobBuilder,
agent: &str,
@ -191,59 +217,62 @@ fn rebuild_subtree<'a>(
if let Some(after) = after {
meta_sync = meta_sync.after_ok(after);
}
let prebuild = builder
.node(NodeKind::Prebuild { agent: a() })
// The brace. Both resources are declared here, on one node, on purpose —
// the queue acquires a node's resources atomically, so a single
// multi-resource root can never hold one and block on another. Hoisting the
// slot up costs nothing: a resource is held for the acquirer's whole
// subtree, and the build slot already spanned the entire rebuild when
// `Prebuild` was the one holding it.
let agent_window = builder
.node(NodeKind::AgentWindow { agent: a() })
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.after_ok(meta_sync);
// The stop root hangs off `Prebuild` and owns the agent lease for
// everything below it. `StopForUpdate` parents the swap pair either way.
let stop_for_update = if graceful {
// Siblings under the brace: the build and the quiesce chain run
// concurrently, borrowing the brace's grants rather than declaring their
// own. Declaring the lease on both would make them mutually exclusive —
// it is single-unit — which is exactly what this shape exists to avoid.
let prebuild = builder
.node(NodeKind::Prebuild { agent: a() })
.part_of(agent_window);
let drain = graceful.then(|| {
let signal = builder
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a()))
.part_of(prebuild);
.part_of(agent_window);
// `Drain` is a *child* of `Signal`, so the parent gate already orders
// it — a child must not dep on its own parent (dep-scope).
let drain = builder
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal);
builder
.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal)
.after_ok(drain)
} else {
builder
.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(prebuild)
};
builder.node(NodeKind::Drain { agent: a() }).part_of(signal)
});
// The container goes down here, not earlier: `AfterOk` the build so a
// failed build never stops a healthy container, and `AfterOk` the drain so
// the agent has checkpointed.
let mut stop_for_update = builder
.node(NodeKind::StopForUpdate { agent: a() })
.part_of(agent_window)
.after_ok(prebuild);
if let Some(drain) = drain {
stop_for_update = stop_for_update.after_ok(drain);
}
let swap = builder
.node(NodeKind::Swap { agent: a() })
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.part_of(stop_for_update);
// `PostSwap` declares the lease it actually runs under. It is a child of
// `StopForUpdate`, which holds it, so this is a re-entrant borrow — no
// second unit, no deadlock. Declaring it is what stops the requirement
// being true only of this one DAG shape.
let _post_swap = builder
.node(NodeKind::PostSwap { agent: a() })
.needs(Resource::Agent(a()))
.part_of(stop_for_update)
.after_ok(swap);
let reconcile = builder
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.after_any(prebuild);
.after_any(agent_window);
RebuildRoots {
meta_sync,
prebuild,
agent_window,
reconcile,
}
}
@ -309,7 +338,7 @@ pub(crate) fn deploy_rebuild_nodes(builder: &JobBuilder, agent: &str, approval_i
approval_id,
})
.needs(Resource::MetaWindow)
.after_ok(roots.prebuild)
.after_ok(roots.agent_window)
.after_ok(roots.reconcile);
}
@ -444,7 +473,7 @@ pub fn perm_change(builder: &JobBuilder, agent: &str, payload: PermPayload) {
emit_rebuilt_tails(
builder,
agent,
&[write, roots.meta_sync, roots.prebuild, roots.reconcile],
&[write, roots.meta_sync, roots.agent_window, roots.reconcile],
);
}