diff --git a/docs/coordinator.md b/docs/coordinator.md index 02160057..a16f11eb 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -33,7 +33,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration: | Node | Wraps | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving | +| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) | | `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node | | `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) | | `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion | @@ -43,6 +43,7 @@ Cheap — no build slot: | Node | Behavior | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | | `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event still fires once per DAG from the terminal hook, not here) | @@ -60,13 +61,19 @@ Two further layers protect the meta repo across *windows* that span multiple span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole container build: -- **The deploy-window gate** (`meta::exclusive()`): every executor that - mutates the meta repo (`Prebuild`'s sync+relock, `MetaLock`, - `WritePermFile`, `Create`'s agent registration, and `ApprovalDeploy` for - its whole span) holds this async mutex for its mutation span, so no commit - can land inside another node's staged window. `Prebuild` drops it before - the long toplevel build (store reads only), preserving `buildSlots > 1` - concurrency. +- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue + resource declared by every node kind that mutates the meta repo — `MetaSync`, + `MetaLock`, `WritePermFile`, `Provision`'s agent registration, and + `ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta + mutations can therefore never interleave, so no commit lands inside another + node's staged window. It is a queue resource rather than a runtime mutex + because a resource is held by a subtree root across its whole subtree, which + a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a + multi-node deploy own one window. For the same reason the window must stay + *off* long store-only work: the rebuild's meta preamble is its own + `MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the + toplevel build runs outside the window and `buildSlots > 1` still gives + concurrent rebuilds across agents. - **Path-limited commits**: the targeted meta committers (perm files, topology, lock bumps, finalize) commit `-- ` with path-scoped dirty checks, so even a non-queue caller (boot migration, destroy's @@ -100,7 +107,7 @@ sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale* agent's subgraph is a rebuild-then-start). ```text -rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a) +rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a) stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a) offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG) restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted) @@ -178,8 +185,8 @@ resources are free. Resources: touching several agents holds one lease per agent. (`SetWanted` is a store write, not a container op, but takes the lease anyway so a power-op DAG's intent write + reconcile is atomic — two racing ops can't clobber intent - before either reconciles.) **Lease-exempt**: `Prebuild`, `MetaLock`, - `WritePermFile` — + before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`, + `MetaLock`, `WritePermFile` — they touch the store / meta, not the running container, which is exactly why a stop can land while another DAG's prebuild is still building. @@ -311,16 +318,20 @@ one go, rather than the double-bounce a live `update` would trigger. Sequence for a rebuild DAG (each step is its own queue node): -1. `Prebuild` — build the new `system.build.toplevel` **before** stopping. +1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a + meta-update cascade child) the per-agent relock. Short, and the only step + that mutates the meta repo, so it is the only one holding the global deploy + window. +2. `Prebuild` — build the new `system.build.toplevel` **before** stopping. The container keeps serving the previous generation while eval + fetch + build happen out-of-band. `nixos-container update` then finds the result cached and skips straight to the profile-swap. Build failures surface here, before the running container is touched. (Runs even for a stopped container — same total nix work, one uniform DAG shape.) -2. `StopForUpdate` — bring the container down (noop when already stopped). -3. `Swap` — `nixos-container update --flake meta#` profile-swap +3. `StopForUpdate` — bring the container down (noop when already stopped). +4. `Swap` — `nixos-container update --flake meta#` profile-swap (near-instant after the prebuild). -4. `Reconcile` — boot into the new generation when `wanted = Up`; the +5. `Reconcile` — boot into the new generation when `wanted = Up`; the in-container activation script transitions old → new. Holds no build slot, so the next DAG's `Prebuild` overlaps the container boot — the old "deferred start" split, now structural. diff --git a/hive-c0re/src/job_queue/exec.rs b/hive-c0re/src/job_queue/exec.rs index 3a232a3b..58f3ef1c 100644 --- a/hive-c0re/src/job_queue/exec.rs +++ b/hive-c0re/src/job_queue/exec.rs @@ -82,7 +82,8 @@ pub(super) async fn run_node(coord: &Arc, claim: &Claim) -> Result< node_id: claim.node_id, }; match &claim.kind { - NodeKind::Prebuild { relock, .. } => run_prebuild(coord, claim, &ctx, *relock).await, + NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await, + NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await, NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await, NodeKind::PostSwap { .. } => run_post_swap(coord, claim, &ctx).await, NodeKind::Provision { .. } => run_provision(coord, claim, &ctx).await, @@ -186,24 +187,24 @@ fn run_set_wanted(coord: &Arc, claim: &Claim, up: bool) -> Result, claim: &Claim, - ctx: &Ctx<'_>, relock: bool, ) -> Result { let name = &claim.agent; - // Prebuild runs while the agent is still up — the runtime dir and - // MCP listener already exist. Use the pure path accessor; no need - // to re-register the listener (event-driven: registered at start/create). + // Runs while the agent is still up — the runtime dir and MCP listener + // already exist. Use the pure path accessor; no need to re-register the + // listener (event-driven: registered at start/create). let agent_dir = crate::paths::agent_runtime_dir(name); let hive = coord.hive_env(); let paths = Coordinator::agent_paths(name, agent_dir); @@ -211,18 +212,24 @@ async fn run_prebuild( // Idempotent meta sync so a manual rebuild can also recover from a // divergent meta repo; then bump just this agent's input. `relock = // false` only for meta-update cascade children, where re-locking - // would revert the bump the cascade just committed. Both run under - // the deploy-window gate so they can never land inside another - // node's staged prepare→finalize window; the gate drops before the - // (long) toplevel build, which only reads the store. - { - let _window = crate::meta::exclusive().await; - let agents = crate::lifecycle::agents_for_meta_listing().await?; - crate::meta::sync_agents(&hive, &agents).await?; - if relock { - crate::meta::lock_update_for_rebuild(name).await?; - } + // would revert the bump the cascade just committed. + let agents = crate::lifecycle::agents_for_meta_listing().await?; + crate::meta::sync_agents(&hive, &agents).await?; + if relock { + crate::meta::lock_update_for_rebuild(name).await?; } + Ok(NodeOutput::default()) +} + +/// Out-of-band toplevel build while the container keeps serving: warm +/// `system.build.toplevel` so the later `Swap` hits cache and skips +/// straight to the profile-swap, against a meta repo the upstream +/// `MetaSync` node has already synced. The warm build is skipped when the +/// container is already down: its only purpose is to shrink the swap's +/// downtime window, so a stopped agent (no uptime to preserve) doesn't +/// pay the double eval — `Swap` builds inline instead. +async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result { + let name = &claim.agent; // Warm the toplevel build only when the container is up — the whole // point of prebuild is to shrink the swap's downtime window. A // stopped agent has no uptime to preserve, so skip the (expensive) @@ -303,10 +310,9 @@ async fn run_post_swap( } /// First-spawn pre-create provisioning: proposed/applied repos, state -/// subvolume, and the meta `sync_agents` registration. Holds the -/// deploy-window gate for the commit so it can't land inside another -/// node's staged deploy window — same discipline as `Prebuild`, which -/// commits under the gate then drops it before the (store-only) build. +/// subvolume, and the meta `sync_agents` registration. Runs under the +/// deploy window (`NodeKind::needs_meta_window`) so its commit can't +/// land inside another node's staged deploy window. async fn run_provision( coord: &Arc, claim: &Claim, @@ -317,7 +323,6 @@ async fn run_provision( let hive = coord.hive_env(); let paths = Coordinator::agent_paths(name, agent_dir); ctx.step("provisioning"); - let _window = crate::meta::exclusive().await; crate::lifecycle::provision_container(name, &hive, &paths).await?; Ok(NodeOutput::default()) } @@ -347,7 +352,6 @@ async fn run_meta_lock( ) -> Result { if sweep { ctx.step("nix flake update hyperhive"); - let _window = crate::meta::exclusive().await; if let Err(e) = crate::meta::lock_update_hyperhive().await { tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed"); } @@ -364,10 +368,7 @@ async fn run_meta_lock( } let _progress = coord.meta_update_guard(); ctx.step("nix flake update"); - { - let _window = crate::meta::exclusive().await; - crate::meta::lock_update(&claim.inputs).await?; - } + crate::meta::lock_update(&claim.inputs).await?; // Lock file changed — meta-inputs panel re-renders. crate::dashboard::emit_meta_inputs_snapshot(coord); let cascade = match fanout { @@ -545,11 +546,10 @@ async fn run_write_perm_file( anyhow::bail!("run_write_perm_file on a non-WritePermFile node"); }; ctx.step("writing + committing perm file"); - // Deploy-window gate: a perm commit landing inside another node's - // staged prepare→finalize window would sweep the staged deploy - // lock into its commit (the commits are also path-limited in - // meta.rs — belt and braces). - let _window = crate::meta::exclusive().await; + // Runs under the deploy window (`NodeKind::needs_meta_window`): a + // perm commit landing inside another node's staged prepare→finalize + // window would sweep the staged deploy lock into its commit (the + // commits are also path-limited in meta.rs — belt and braces). match payload { PermPayload::ToolGroups { groups } => { crate::meta::commit_tool_groups(name, groups) @@ -586,11 +586,14 @@ async fn run_approval_deploy(coord: &Arc, claim: &Claim) -> Result< let approval_id = claim .approval_id .with_context(|| format!("approval_deploy dag {} has no approval_id", claim.dag_id))?; - // Hold the deploy-window gate for the whole prepare→finalize span: - // `prepare_deploy` stages `flake.lock` uncommitted for the entire - // container build, and no other meta mutation may land inside that - // window (it would sweep the staged lock and neuter `abort_deploy`). - let _window = crate::meta::exclusive().await; + // The deploy window covers the whole prepare→finalize span + // (`NodeKind::needs_meta_window`, held by the scheduler for this + // node): `prepare_deploy` stages `flake.lock` uncommitted for the + // entire container build, and no other meta mutation may land inside + // that window (it would sweep the staged lock and neuter + // `abort_deploy`). Holding it as a queue resource — rather than a + // `MutexGuard` that cannot outlive this fn — is what lets increment 2 + // decompose this node into sub-nodes under a window-holding parent. crate::actions::run_approval_merge_config_pr(coord, Some(claim.dag_id), approval_id) .await .map(|()| NodeOutput::default()) diff --git a/hive-c0re/src/job_queue/model.rs b/hive-c0re/src/job_queue/model.rs index 192ea881..d0ba1157 100644 --- a/hive-c0re/src/job_queue/model.rs +++ b/hive-c0re/src/job_queue/model.rs @@ -99,15 +99,27 @@ pub struct Dep { #[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum NodeKind { + /// The rebuild's meta-repo preamble: `lifecycle::prepare_rebuild_dirs`, + /// an idempotent meta `sync_agents`, and an optional per-agent relock. + /// `relock = false` only for meta-update cascade rebuilds (re-locking + /// would revert the bump the cascade just committed). + /// + /// Its own node — ahead of, and *not* an ancestor of, [`NodeKind::Prebuild`] + /// — precisely because it is the only part of the rebuild that mutates the + /// meta repo and so holds the global + /// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow). Fusing + /// it into `Prebuild` (or making it `Prebuild`'s parent, which holds a + /// resource across the whole subtree) would extend that global window over + /// the multi-minute toplevel build and serialize rebuilds hive-wide. + /// Store/meta work only — build-slot- and lease-exempt. + MetaSync { agent: String, relock: bool }, /// Out-of-band toplevel build while the container keeps serving: - /// meta `sync_agents`, optional per-agent relock, then - /// `lifecycle::prebuild_toplevel`. `relock = false` only for - /// meta-update cascade rebuilds (re-locking would revert the bump - /// the cascade just committed). The `prebuild_toplevel` warm is - /// skipped when the container is already down — it only exists to - /// shrink the swap's downtime, which a stopped agent doesn't need - /// (the sync + dir prep still run; `Swap` builds inline). - Prebuild { agent: String, relock: bool }, + /// `lifecycle::prebuild_toplevel`, reading a meta repo the upstream + /// [`NodeKind::MetaSync`] has already synced. The warm build is skipped + /// when the container is already down — it only exists to shrink the + /// swap's downtime, which a stopped agent doesn't need (`Swap` builds + /// inline instead). + Prebuild { agent: String }, /// `nixos-container update` profile-swap (requires the container /// stopped). Re-applies nspawn flags + resource limits first — /// rebuild is the reconcile verb. The post-rebuild bookkeeping tail @@ -216,6 +228,7 @@ impl NodeKind { /// Wire string for `NodeView.kind`. pub fn as_str(&self) -> &'static str { match self { + NodeKind::MetaSync { .. } => "meta_sync", NodeKind::Prebuild { .. } => "prebuild", NodeKind::Swap { .. } => "swap", NodeKind::PostSwap { .. } => "post_swap", @@ -242,7 +255,8 @@ impl NodeKind { #[must_use] pub fn agent(&self) -> &str { match self { - NodeKind::Prebuild { agent, .. } + NodeKind::MetaSync { agent, .. } + | NodeKind::Prebuild { agent } | NodeKind::Swap { agent } | NodeKind::PostSwap { agent } | NodeKind::Provision { agent } @@ -276,8 +290,8 @@ impl NodeKind { /// Container-affecting kinds require the DAG to hold the agent's /// lifecycle lease (acquired at the first such node, held until the - /// DAG is terminal). Lease-exempt kinds (`Prebuild`, `Provision`, - /// `MetaLock`, `WritePermFile`) touch the store / meta repo, not the + /// DAG is terminal). Lease-exempt kinds (`MetaSync`, `Prebuild`, + /// `Provision`, `MetaLock`, `WritePermFile`) touch the store / meta repo, not the /// running container — which is exactly why a `Prebuild` can overlap /// another DAG's work on the same agent. `Provision` precedes the /// container's existence entirely, so the lease is first taken at the @@ -296,6 +310,36 @@ impl NodeKind { | NodeKind::SetWanted { .. } ) } + + /// Kinds that **mutate the meta repo** and so must hold the global + /// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow) for + /// their duration: no two meta mutations may interleave, because a commit + /// landing inside another node's staged `prepare_deploy`→`finalize_deploy` + /// window would sweep the staged `flake.lock` into its own commit and + /// neuter `abort_deploy`. + /// + /// This is the queue-primitive replacement for the former runtime + /// `meta::exclusive()` mutex — same global serialisation, but held by the + /// scheduler and therefore able to span a whole subtree, which a + /// `MutexGuard` cannot. + /// + /// Note what is **not** here: [`NodeKind::Prebuild`]. The window must stay + /// off the multi-minute toplevel build, which only *reads* the store — the + /// old mutex was scoped to drop before it, and holding a cap-1 global + /// across it would serialize every agent's rebuild behind every other's. + /// That is why the meta preamble is its own [`NodeKind::MetaSync`] node, + /// and why that node is a sibling rather than `Prebuild`'s parent (a + /// resource held by a parent covers its whole subtree). + pub fn needs_meta_window(&self) -> bool { + matches!( + self, + NodeKind::MetaSync { .. } + | NodeKind::Provision { .. } + | NodeKind::MetaLock { .. } + | NodeKind::WritePermFile { .. } + | NodeKind::ApprovalDeploy { .. } + ) + } } /// Submit-time spec for one node. diff --git a/hive-c0re/src/job_queue/resource.rs b/hive-c0re/src/job_queue/resource.rs index a59a428f..0b46ab9d 100644 --- a/hive-c0re/src/job_queue/resource.rs +++ b/hive-c0re/src/job_queue/resource.rs @@ -22,6 +22,17 @@ pub enum Resource { /// the rest of that agent's subtree via the crate's recursive lock, so two /// DAGs never interleave container ops on one agent. Agent(String), + /// The meta-repo mutation window — a global singleton (default capacity 1) + /// held by any node that mutates the meta repo, so two meta mutations never + /// interleave. Replaces the former runtime `meta::exclusive()` mutex: a + /// `MutexGuard` cannot span scheduler nodes, but a resource held by a + /// subtree root *can* — which is what lets the two-phase deploy + /// (`prepare_deploy` stages `flake.lock` uncommitted across the whole + /// container build, `finalize_deploy`/`abort_deploy` resolve it) be + /// decomposed into sub-nodes instead of one opaque node. Descendants of a + /// holder re-enter it through the crate's recursive lock, exactly like + /// [`Resource::Agent`]. + MetaWindow, } impl NodeKind { @@ -31,7 +42,14 @@ impl NodeKind { /// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt /// container ops (`Start` / `Stop`, fanned out by a lease-holding /// `Reconcile`) hold no lease of their own — they re-enter the ancestor's - /// `Agent` lock through the crate's recursive re-entrancy. + /// `Agent` lock through the crate's recursive re-entrancy. Meta-mutating + /// kinds ([`NodeKind::needs_meta_window`]) additionally take the global + /// [`Resource::MetaWindow`]. + /// + /// All of a node's resource edges are acquired **atomically** + /// (`try_acquire_all`) — a node never holds one resource while waiting on + /// another, so the multi-resource kinds (a `MetaLock` wants a build slot + /// *and* the meta window) cannot deadlock against each other. pub fn resource_deps(&self) -> Vec> { let mut deps = Vec::new(); if self.needs_build_slot() { @@ -46,6 +64,12 @@ impl NodeKind { count: 1, }); } + if self.needs_meta_window() { + deps.push(Dep::Resource { + name: Resource::MetaWindow, + count: 1, + }); + } deps } } diff --git a/hive-c0re/src/job_queue/submit.rs b/hive-c0re/src/job_queue/submit.rs index b4d85415..6f010c4b 100644 --- a/hive-c0re/src/job_queue/submit.rs +++ b/hive-c0re/src/job_queue/submit.rs @@ -95,8 +95,9 @@ fn start_chain(agent: &str, running: bool, stale: bool) -> Vec { )]; if !running && stale { // Rebuild subtree after the SetWanted head (base = 1, so the rebuild's - // `Prebuild` root deps `after_ok(0)` = the head). `Prebuild` + - // `Reconcile` are their own group roots (top-level, per `rebuild_nodes`). + // `MetaSync` root deps `after_ok(0)` = the head). `MetaSync`, + // `Prebuild` + `Reconcile` are their own group roots (top-level, per + // `rebuild_nodes`). n.extend(rebuild_nodes(agent, true, 1)); } else { n.push(child( diff --git a/hive-c0re/src/job_queue/templates.rs b/hive-c0re/src/job_queue/templates.rs index f456da8e..f2b3747c 100644 --- a/hive-c0re/src/job_queue/templates.rs +++ b/hive-c0re/src/job_queue/templates.rs @@ -18,7 +18,7 @@ //! agent's existing `wanted`. //! //! ```text -//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a) +//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a) //! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve] //! perm-change(a): WritePermFile(a) → «rebuild subgraph» //! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a» @@ -68,45 +68,55 @@ pub(crate) fn child(parent: u64, kind: NodeKind, deps: Vec) -> NodeSpec { } } -/// The rebuild node subtree (nested, two group roots). `base` is the spec index -/// of the first node (`Prebuild`). Structure: -/// - `Prebuild` (base+0, **root**): owns the build slot for the whole subtree. -/// Lease-exempt — the nix build overlaps other DAGs on the same agent. -/// - `StopForUpdate` (base+1, child of `Prebuild`): owns the agent lease. Runs +/// The rebuild node subtree (nested, three group roots). `base` is the spec +/// index of the first node (`MetaSync`). Structure: +/// - `MetaSync` (base+0, **root**): the meta-repo preamble (dir prep, agent +/// sync, optional relock). Owns the global `MetaWindow` — and *only* for its +/// own short duration, which is why it is a sibling root rather than +/// `Prebuild`'s parent: a resource is held across the holder's whole subtree, +/// so parenting the build under it would extend a hive-global window over +/// every rebuild's nix build. +/// - `Prebuild` (base+1, **root**): `AfterOk` `MetaSync`. Owns the build slot +/// for the whole mechanical subtree below it. Lease-exempt — the nix build +/// overlaps other DAGs on the same agent. +/// - `StopForUpdate` (base+2, child of `Prebuild`): owns the agent lease. Runs /// once `Prebuild` reaches `Finishing` (parent gate). -/// - `Swap` (base+2, child of `StopForUpdate`): borrows the agent lease from its +/// - `Swap` (base+3, child of `StopForUpdate`): borrows the agent lease from its /// parent and the build slot from grand-ancestor `Prebuild` — both continuous. -/// - `PostSwap` (base+3, child of `StopForUpdate`): the swap's Ok-only +/// - `PostSwap` (base+4, child of `StopForUpdate`): the swap's Ok-only /// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk` /// its sibling `Swap`. -/// - `Reconcile` (base+4, **root**): `AfterAny` `Prebuild`, which rolls up +/// - `Reconcile` (base+5, **root**): `AfterAny` `Prebuild`, which rolls up /// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has /// settled — so `Reconcile` runs after the swap regardless of outcome, and as /// a top-level root it survives the cancel-cascade of a failed `Prebuild` -/// (recovery-start invariant). It takes a fresh lease; the tiny gap is -/// harmless — `Reconcile` converges to the persisted `wanted` idempotently. +/// (recovery-start invariant, which also covers a failed `MetaSync`: that +/// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It +/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to +/// the persisted `wanted` idempotently. pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec { let a = || agent.to_owned(); vec![ node( - NodeKind::Prebuild { agent: a(), relock }, + NodeKind::MetaSync { agent: a(), relock }, if base == 0 { Vec::new() } else { after_ok(base - 1) }, ), - child(base, NodeKind::StopForUpdate { agent: a() }, Vec::new()), - child(base + 1, NodeKind::Swap { agent: a() }, Vec::new()), + node(NodeKind::Prebuild { agent: a() }, after_ok(base)), + child(base + 1, NodeKind::StopForUpdate { agent: a() }, Vec::new()), + child(base + 2, NodeKind::Swap { agent: a() }, Vec::new()), child( - base + 1, + base + 2, NodeKind::PostSwap { agent: a() }, - after_ok(base + 2), + after_ok(base + 3), ), node( NodeKind::Reconcile { agent: a() }, vec![Dep { - on: base, + on: base + 1, when: DepWhen::AfterAny, }], ), diff --git a/hive-c0re/src/job_queue/tests.rs b/hive-c0re/src/job_queue/tests.rs index 6adac2e9..ba46586c 100644 --- a/hive-c0re/src/job_queue/tests.rs +++ b/hive-c0re/src/job_queue/tests.rs @@ -176,6 +176,7 @@ fn rebuild_chain_claims_in_dep_order() { let q = JobQueue::new(1); let id = submit(&q, rebuild("agent-a", "r")); for expected in [ + "meta_sync", "prebuild", "stop_for_update", "swap", @@ -201,15 +202,29 @@ fn build_slot_serializes_nix_heavy_nodes() { let q = JobQueue::new(1); let a = submit(&q, rebuild("agent-a", "r")); let b = submit(&q, rebuild("agent-b", "r")); - let first = claim_one(&q); // a's Prebuild takes the only slot - assert_eq!(first.dag_id, a); - assert_eq!(first.kind.as_str(), "prebuild"); - q.complete_node(a, first.node_id, Ok(())); + // The rebuild heads are `MetaSync` (slot-free, but serialized on the + // global meta window), so drive each chain's head out of the way first. + let head_a = claim_one(&q); + assert_eq!(head_a.dag_id, a); + assert_eq!(head_a.kind.as_str(), "meta_sync"); + q.complete_node(a, head_a.node_id, Ok(())); + // a's Prebuild takes the only slot; b's MetaSync is free to run beside it + // (different resources), but b's Prebuild is not. + let claims = q.claim_ready(); + let mut kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect(); + kinds.sort_unstable(); + assert_eq!(kinds, vec![(a, "prebuild"), (b, "meta_sync")]); + for c in &claims { + q.complete_node(c.dag_id, c.node_id, Ok(())); + } // Uniform hold: agent-a keeps the build slot across its whole build chain // (Swap re-enters it), so a's StopForUpdate (lease, slot-free) runs but b's // Prebuild must wait for a's slot-needers (through Swap) to finish. - let claims = q.claim_ready(); - let kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect(); + let kinds: Vec<(u64, &str)> = q + .claim_ready() + .iter() + .map(|c| (c.dag_id, c.kind.as_str())) + .collect(); assert_eq!(kinds, vec![(a, "stop_for_update")]); assert!( !kinds.iter().any(|&(d, _)| d == b), @@ -222,9 +237,23 @@ fn two_build_slots_run_two_prebuilds() { let q = JobQueue::new(2); submit(&q, rebuild("agent-a", "r")); submit(&q, rebuild("agent-b", "r")); - let claims = q.claim_ready(); - assert_eq!(claims.len(), 2, "two slots → two concurrent prebuilds"); - assert!(claims.iter().all(|c| c.kind.as_str() == "prebuild")); + // Each rebuild's head `MetaSync` holds the cap-1 global meta window, so the + // two heads take turns — exactly the serialization the old runtime + // `meta::exclusive()` mutex imposed inside the prebuild executor. What must + // NOT serialize is the build itself: complete only the meta heads and watch + // both prebuilds end up in flight together, neither of them completed. + let mut prebuilds = Vec::new(); + for _ in 0..3 { + for c in q.claim_ready() { + if c.kind.as_str() == "meta_sync" { + q.complete_node(c.dag_id, c.node_id, Ok(())); + } else { + prebuilds.push(c); + } + } + } + assert_eq!(prebuilds.len(), 2, "two slots → two concurrent prebuilds"); + assert!(prebuilds.iter().all(|c| c.kind.as_str() == "prebuild")); } #[test] @@ -312,12 +341,23 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() { None, ), ); - // Prebuild is lease-exempt: the stop's Reconcile takes the lease + // Both DAGs' heads are lease-independent of each other: the rebuild's + // MetaSync (meta window) and the stop's Reconcile (agent lease). + let heads = q.claim_ready(); + let head_kinds: Vec<&str> = heads.iter().map(|c| c.kind.as_str()).collect(); + assert!(head_kinds.contains(&"meta_sync")); + assert!(head_kinds.contains(&"reconcile")); + let meta_sync = heads + .iter() + .find(|c| c.kind.as_str() == "meta_sync") + .expect("meta_sync claim") + .clone(); + q.complete_node(meta_sync.dag_id, meta_sync.node_id, Ok(())); + // Prebuild is lease-exempt: the stop's Reconcile keeps the lease // and runs concurrently with the rebuild's out-of-band nix build. let claims = q.claim_ready(); let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect(); assert!(kinds.contains(&"prebuild")); - assert!(kinds.contains(&"reconcile")); // But the rebuild's StopForUpdate must then wait for the stop DAG // to finish (lease). let prebuild = claims @@ -330,7 +370,7 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() { q.claim_ready().is_empty(), "StopForUpdate blocked while stop DAG holds the lease" ); - let reconcile = claims + let reconcile = heads .iter() .find(|c| c.kind.as_str() == "reconcile") .expect("reconcile claim") @@ -484,7 +524,7 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() { head_agents.sort_unstable(); assert_eq!(head_agents, vec!["fresh", "stale"]); // Complete both heads; the fresh agent then reconciles directly while - // the stale agent's subgraph is the rebuild chain (prebuild first). + // the stale agent's subgraph is the rebuild chain (meta_sync first). for c in &heads { q.complete_node(id, c.node_id, Ok(())); } @@ -496,7 +536,7 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() { kinds.sort_unstable(); assert_eq!( kinds, - vec![("fresh", "reconcile"), ("stale", "prebuild")], + vec![("fresh", "reconcile"), ("stale", "meta_sync")], "fresh agent starts directly; stale agent rebuilds first, all in one DAG" ); } @@ -578,30 +618,48 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() { let emitter = claim_one(&q); assert_eq!(emitter.kind.as_str(), "meta_lock"); // Two independent per-agent subgraphs — the REAL production shape the - // sweep MetaLock grows (`rebuild_nodes(_, true, 0)`: root Prebuild → - // StopForUpdate → Swap → Reconcile, local 0-based deps), so this test - // tracks any drift in that builder's root-first (`base = 0`) shape. + // sweep MetaLock grows (`rebuild_nodes(_, true, 0)`: root MetaSync → root + // Prebuild → StopForUpdate → Swap → Reconcile, local 0-based deps), so this + // test tracks any drift in that builder's root-first (`base = 0`) shape. let subgraph = |agent: &str| templates::rebuild_nodes(agent, true, 0); // Must append BEFORE completing the emitter (the documented contract). q.append_subgraph(id, &subgraph("a"), emitter.node_id); q.append_subgraph(id, &subgraph("b"), emitter.node_id); q.complete_node(id, emitter.node_id, Ok(())); // Still ONE DAG; both subgraph roots become ready once the emitter is - // Done (rooted on it), each on its own agent lease. + // Done (rooted on it), each on its own agent lease. Their `MetaSync` heads + // take turns on the cap-1 global meta window, so drain those first — what + // must be concurrent is the builds. assert_eq!(q.snapshot().len(), 1); - let next = q.claim_ready(); - let mut kinds: Vec<(&str, &str)> = next - .iter() - .map(|c| (c.agent.as_str(), c.kind.as_str())) - .collect(); + let mut kinds = drain_meta_syncs(&q, id); kinds.sort_unstable(); assert_eq!( kinds, - vec![("a", "prebuild"), ("b", "prebuild")], + vec![ + ("a".to_owned(), "prebuild".to_owned()), + ("b".to_owned(), "prebuild".to_owned()) + ], "both rebuild subgraphs root on the emitter and run concurrently in one DAG" ); } +/// Complete every `MetaSync` head the queue offers (they take turns on the +/// cap-1 global meta window) and return whatever else got claimed alongside +/// them, as `(agent, kind)` pairs left in flight. +fn drain_meta_syncs(q: &JobQueue, dag: u64) -> Vec<(String, String)> { + let mut rest = Vec::new(); + for _ in 0..3 { + for c in q.claim_ready() { + if c.kind.as_str() == "meta_sync" { + q.complete_node(dag, c.node_id, Ok(())); + } else { + rest.push((c.agent.clone(), c.kind.as_str().to_owned())); + } + } + } + rest +} + #[test] fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() { // The meta-update `MetaLock` grows one rebuild subgraph per affected @@ -637,17 +695,18 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() { } q.complete_node(id, meta_lock.node_id, Ok(())); // Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root - // on the MetaLock, each on its own agent lease. + // on the MetaLock, each on its own agent lease. The per-agent `MetaSync` + // heads serialize on the global meta window (they commit to the meta repo); + // the builds behind them do not. assert_eq!(q.snapshot().len(), 1); - let next = q.claim_ready(); - let mut kinds: Vec<(&str, &str)> = next - .iter() - .map(|c| (c.agent.as_str(), c.kind.as_str())) - .collect(); + let mut kinds = drain_meta_syncs(&q, id); kinds.sort_unstable(); assert_eq!( kinds, - vec![("alice", "prebuild"), ("bob", "prebuild")], + vec![ + ("alice".to_owned(), "prebuild".to_owned()), + ("bob".to_owned(), "prebuild".to_owned()) + ], "cascade rebuilds grow in the meta-update DAG, concurrent per agent" ); } @@ -658,7 +717,11 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() { fn failed_node_cancels_downstream_but_afterany_reconcile_runs() { let q = JobQueue::new(1); let id = submit(&q, rebuild("agent-a", "r")); + let meta_sync = claim_one(&q); + assert_eq!(meta_sync.kind.as_str(), "meta_sync"); + q.complete_node(id, meta_sync.node_id, Ok(())); let prebuild = claim_one(&q); + assert_eq!(prebuild.kind.as_str(), "prebuild"); q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned())); // StopForUpdate + Swap are cancelled (AfterOk on a failed chain); // the AfterAny Reconcile still runs once Swap is terminal. @@ -702,7 +765,8 @@ fn failed_node_cancels_downstream_but_afterany_reconcile_runs() { fn swap_failure_still_runs_reconcile() { let q = JobQueue::new(1); let id = submit(&q, rebuild("agent-a", "r")); - for _ in 0..2 { + // meta_sync + prebuild + stop_for_update + for _ in 0..3 { let c = claim_one(&q); q.complete_node(id, c.node_id, Ok(())); } @@ -735,8 +799,8 @@ fn swap_failure_still_runs_reconcile() { fn swap_ok_runs_post_swap_before_reconcile() { let q = JobQueue::new(1); let id = submit(&q, rebuild("agent-a", "r")); - // prebuild + stop_for_update - for _ in 0..2 { + // meta_sync + prebuild + stop_for_update + for _ in 0..3 { let c = claim_one(&q); q.complete_node(id, c.node_id, Ok(())); } @@ -979,13 +1043,25 @@ fn graceful_signal_and_drain_hold_no_build_slot() { submit(&q, rebuild("builder", "slot hog")); submit(&q, stop_online(&["agent-a"], true, "g")); submit(&q, stop_online(&["agent-b"], true, "g")); - let claims = q.claim_ready(); - let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect(); + // All three DAG heads are build-slot-exempt, so they run at once. + let heads = q.claim_ready(); + let kinds: Vec<&str> = heads.iter().map(|c| c.kind.as_str()).collect(); + assert_eq!(kinds, vec!["meta_sync", "set_wanted", "set_wanted"]); + for c in &heads { + q.complete_node(c.dag_id, c.node_id, Ok(())); + } + // Now the rebuild's Prebuild holds the single slot — and both graceful + // stops still proceed to their Signal beside it. + let kinds: Vec<&str> = q + .claim_ready() + .iter() + .map(|c| c.kind.as_str()) + .collect::>(); assert_eq!( kinds, - vec!["prebuild", "set_wanted", "set_wanted"], - "both agents' graceful-stop heads (SetWanted, build-slot-exempt) run \ - while the slot is held; their signals follow" + vec!["prebuild", "signal", "signal"], + "both agents' graceful-stop signals (build-slot-exempt) run while the \ + rebuild holds the slot" ); } @@ -1023,6 +1099,7 @@ fn perm_change_shape_prefixes_rebuild_chain() { ); for expected in [ "write_perm_file", + "meta_sync", "prebuild", "stop_for_update", "swap", diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index 55f4edf5..66796212 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -27,24 +27,17 @@ const GIT_EMAIL: &str = "c0re@hyperhive.local"; /// take turns instead of colliding. static META_LOCK: Mutex<()> = Mutex::const_new(()); -/// Coarse exclusivity for meta-repo *windows* that span multiple -/// `META_LOCK` acquisitions — above all the two-phase deploy -/// (`prepare_deploy` stages `flake.lock` uncommitted for the whole -/// container build; `finalize_deploy` / `abort_deploy` resolve it). -/// `META_LOCK` serializes individual git ops but cannot keep another -/// op out of that staged window: a perm-file or lock-bump commit -/// landing mid-window would sweep the staged deploy lock into its own -/// commit and neuter `abort_deploy`. Job-queue executors that mutate -/// the meta repo hold this gate for their mutation span; the opaque -/// approval-deploy node holds it across its whole prepare→finalize -/// span. Never acquired inside this module's functions (they run -/// *under* a caller's window — nesting would deadlock). -static DEPLOY_GATE: Mutex<()> = Mutex::const_new(()); - -/// Acquire the deploy/meta-mutation window gate. See [`DEPLOY_GATE`]. -pub async fn exclusive() -> tokio::sync::MutexGuard<'static, ()> { - DEPLOY_GATE.lock().await -} +// Exclusivity for meta-repo *windows* that span multiple `META_LOCK` +// acquisitions — above all the two-phase deploy (`prepare_deploy` stages +// `flake.lock` uncommitted for the whole container build; +// `finalize_deploy` / `abort_deploy` resolve it) — is **not** a mutex in +// this module. `META_LOCK` above serializes individual git ops but cannot +// keep another op out of that staged window; that window is owned by the +// job queue instead, as `Resource::MetaWindow`, declared by every +// meta-mutating node kind (`NodeKind::needs_meta_window`). A resource can +// be held by a subtree root across its children, which a `MutexGuard` +// (bounded by one executor fn) cannot — that's what lets the deploy be +// modelled as sub-nodes rather than one opaque node. /// Where the manager sees this directory inside its container (RO bind). pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta";