vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(job-queue): promote the meta-repo deploy window to a queue resource

Browse source 
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 • committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

44
hive-c0re/src/job_queue/templates.rs
View file

@ -18,7 +18,7 @@
//! agent's existing `wanted`.
//!
//! ```text
//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
//! perm-change(a): WritePermFile(a) → «rebuild subgraph»
//! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a»
@ -68,45 +68,55 @@ pub(crate) fn child(parent: u64, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
}
}
/// The rebuild node subtree (nested, two group roots). `base` is the spec index
/// of the first node (`Prebuild`). Structure:
/// - `Prebuild` (base+0, **root**): owns the build slot for the whole subtree.
/// Lease-exempt — the nix build overlaps other DAGs on the same agent.
/// - `StopForUpdate` (base+1, child of `Prebuild`): owns the agent lease. Runs
/// The rebuild node subtree (nested, three group roots). `base` is the spec
/// index of the first node (`MetaSync`). Structure:
/// - `MetaSync` (base+0, **root**): the meta-repo preamble (dir prep, agent
/// sync, optional relock). Owns the global `MetaWindow` — and *only* for its
/// own short duration, which is why it is a sibling root rather than
/// `Prebuild`'s parent: a resource is held across the holder's whole subtree,
/// so parenting the build under it would extend a hive-global window over
/// every rebuild's nix build.
/// - `Prebuild` (base+1, **root**): `AfterOk` `MetaSync`. Owns the build slot
/// for the whole mechanical subtree below it. Lease-exempt — the nix build
/// overlaps other DAGs on the same agent.
/// - `StopForUpdate` (base+2, child of `Prebuild`): owns the agent lease. Runs
/// once `Prebuild` reaches `Finishing` (parent gate).
/// - `Swap` (base+2, child of `StopForUpdate`): borrows the agent lease from its
/// - `Swap` (base+3, child of `StopForUpdate`): borrows the agent lease from its
/// parent and the build slot from grand-ancestor `Prebuild` — both continuous.
/// - `PostSwap` (base+3, child of `StopForUpdate`): the swap's Ok-only
/// - `PostSwap` (base+4, child of `StopForUpdate`): the swap's Ok-only
/// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk`
/// its sibling `Swap`.
/// - `Reconcile` (base+4, **root**): `AfterAny` `Prebuild`, which rolls up
/// - `Reconcile` (base+5, **root**): `AfterAny` `Prebuild`, which rolls up
/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has
/// settled — so `Reconcile` runs after the swap regardless of outcome, and as
/// a top-level root it survives the cancel-cascade of a failed `Prebuild`
/// (recovery-start invariant). It takes a fresh lease; the tiny gap is
/// harmless — `Reconcile` converges to the persisted `wanted` idempotently.
/// (recovery-start invariant, which also covers a failed `MetaSync`: that
/// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It
/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to
/// the persisted `wanted` idempotently.
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec<NodeSpec> {
let a = || agent.to_owned();
vec![
node(
NodeKind::Prebuild { agent: a(), relock },
NodeKind::MetaSync { agent: a(), relock },
if base == 0 {
Vec::new()
} else {
after_ok(base - 1)
},
),
child(base, NodeKind::StopForUpdate { agent: a() }, Vec::new()),
child(base + 1, NodeKind::Swap { agent: a() }, Vec::new()),
node(NodeKind::Prebuild { agent: a() }, after_ok(base)),
child(base + 1, NodeKind::StopForUpdate { agent: a() }, Vec::new()),
child(base + 2, NodeKind::Swap { agent: a() }, Vec::new()),
child(
base + 1,
base + 2,
NodeKind::PostSwap { agent: a() },
after_ok(base + 2),
after_ok(base + 3),
),
node(
NodeKind::Reconcile { agent: a() },
vec![Dep {
on: base,
on: base + 1,
when: DepWhen::AfterAny,
}],
),