diff --git a/docs/approvals.md b/docs/approvals.md index c1c3f244..998cbab8 100644 --- a/docs/approvals.md +++ b/docs/approvals.md @@ -319,9 +319,8 @@ running and a restart resumes at node granularity: `applied//main` (which the deploy already fast-forwarded to the reviewed PR head). 2. The rebuild subgraph `DeployApply` grows into the DAG builds and - swaps the container (`AgentWindow` bracing `Prebuild → StopForUpdate - → Swap → RebuildBookkeeping`, plus `Reconcile`). Nix evaluates - against the staged lock. + swaps the container (`Prebuild → StopForUpdate → Swap → PostSwap`, + plus `Reconcile`). Nix evaluates against the staged lock. 3. On success — `FinalizeDeploy` drops the rollback ref, plants `deployed/`, then `meta::finalize_deploy(name, sha, "deployed/ ")` stages `flake.lock` and commits with diff --git a/docs/coordinator.md b/docs/coordinator.md index 115d8ec2..e06d77fb 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -40,7 +40,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration: | Node | Wraps | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) | -| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node | +| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node | | `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) | | `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion | | `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) | @@ -55,8 +55,7 @@ Cheap — no build slot: | `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | -| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here) | -| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ | +| `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here) | | `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | | `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | | `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload | @@ -119,7 +118,7 @@ sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale* agent's subgraph is a rebuild-then-start). ```text -rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) ∥ [Signal(a)→Drain(a) if graceful]; both →(after-ok) StopForUpdate(a) → Swap(a) →(after-ok) RebuildBookkeeping(a) } →(after-any) Reconcile(a) +rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a) stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a) offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG) restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted) @@ -191,56 +190,17 @@ resources are free. Resources: held by nix-heavy nodes for the node's duration. 2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is per-node; a DAG can span agents) and globally exclusive per agent across - all DAGs: acquired either at a container-affecting node (`SetWanted`, - `Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`, - `DeployWindow`) on behalf of a whole coordinated subtree; held by the owning - DAG until it's terminal, so two DAGs never interleave container ops on the - same agent. A DAG touching several agents holds one lease per agent. - (`SetWanted` is a store write, not a container op, but takes the lease anyway - so a power-op DAG's intent write + reconcile is atomic — two racing ops can't - clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`, - `Prebuild`, `MetaLock`, `WritePermFile`, `Reparent` — they touch the store / - meta, not the running container, which is exactly why a stop can land while - another DAG's prebuild is still building. Also exempt, for a different - reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`, - `Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the - container, but their `AgentWindow` brace holds the lease above them — see - _Braces_ below. - -#### Braces - -Templates otherwise declare a resource on **every** node that needs it, even -when a parent already holds it, so the requirement belongs to the node rather -than to one DAG shape it happens to appear in. A **brace** is the one sanctioned -exception: a pure-resource-holder root that declares on behalf of a subtree -coordinated with itself, whose members then declare nothing. - -It is forced rather than stylistic. Declaring a resource means *"I need this -exclusively"*, and the agent lease is single-unit — so **two siblings that both -declared it could never run concurrently.** For a subtree whose whole point is -concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring -the requirement truthfully on every node and running those nodes in parallel are -mutually exclusive. One holder above them speaks for the subtree. - -This is the opposite of the failure the declare-your-own rule exists to prevent, -not a relapse into it: there the requirement was *implicit*, inferred from a -node's kind and true only by accident of placement. Here it is explicit, on one -node, with the omission below it documented on the brace itself. - -Two consequences worth knowing: - -- **Flattening a chain under a brace is safe.** The stop chain used to nest - `Signal` over `Drain` over `StopForUpdate` specifically so the lease stayed - continuous — as independent siblings each would acquire it separately and - leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies - that continuity directly, so the nesting is no longer load-bearing. -- **Observability is unaffected.** `running_transients` keys off a node's - *payload* agent, not off a declared lease edge, so every child still lights its - own dashboard pill and still reports its own `takes_container_down` to the - crash watcher. A brace itself reports `false`: it parents the stopping nodes - but does not stop anything, and claiming otherwise would widen crash - suppression across the build and tail, where a vanished container is still a - real crash. + all DAGs: acquired at a container-affecting node (`SetWanted`, + `StopForUpdate`, `Swap`, `Signal`, `Drain`, `Reconcile`, `WriteDropin`, + `Create`, `DeployWindow`), held by the owning DAG until it's terminal, + so two DAGs never interleave container ops on the same agent. A DAG + touching several agents holds one lease per agent. (`SetWanted` is a store + write, not a container op, but takes the lease anyway so a power-op DAG's + intent write + reconcile is atomic — two racing ops can't clobber intent + before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`, + `MetaLock`, `WritePermFile`, `Reparent` — + they touch the store / meta, not the running container, which is exactly + why a stop can land while another DAG's prebuild is still building. Among simultaneously-ready nodes competing for a resource, DAG-submit order wins (FIFO) so bulk operations drain predictably. The scheduler also owns the diff --git a/hive-c0re/src/actions.rs b/hive-c0re/src/actions.rs index 7333f387..83f5bd07 100644 --- a/hive-c0re/src/actions.rs +++ b/hive-c0re/src/actions.rs @@ -867,7 +867,7 @@ async fn prepare_applied_target( /// [`run_deploy_tail`] must not roll `main` back. Ordering that ahead of the tag /// plant is what makes the tail's `deployed/` cross-check a second line of /// defence rather than the only one. No agent kick — the rebuild's own -/// `RebuildBookkeeping` already did it. +/// `PostSwap` already did it. /// /// # Errors /// diff --git a/hive-c0re/src/job_queue/exec.rs b/hive-c0re/src/job_queue/exec.rs index fe0c4074..a7f30146 100644 --- a/hive-c0re/src/job_queue/exec.rs +++ b/hive-c0re/src/job_queue/exec.rs @@ -63,7 +63,7 @@ pub(super) async fn run_node( NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, agent, *relock).await, NodeKind::Prebuild { .. } => run_prebuild(agent, id).await, NodeKind::Swap { .. } => run_swap(coord, agent, id).await, - NodeKind::RebuildBookkeeping { .. } => run_rebuild_bookkeeping(coord, agent).await, + NodeKind::PostSwap { .. } => run_post_swap(coord, agent).await, NodeKind::Provision { .. } => run_provision(coord, agent).await, NodeKind::Create { .. } => run_create(agent).await, NodeKind::MetaLock { @@ -126,15 +126,13 @@ pub(super) async fn run_node( Ok(()) } NodeKind::SetWanted { up, .. } => run_set_wanted(coord, agent, *up), - // The nodes that carry no work of their own; completing one lets it - // reach `Finishing` so the nodes under it start. + // The two nodes that carry no work of their own; completing either + // lets it reach `Finishing` so the nodes under it start. // - `Dag`: pure grouping container. The DAG's terminal side effect, if // any, is its own tail node in the graph. - // - `DeployWindow` / `AgentWindow`: pure resource holders (braces) — - // what they declare stays held until their subtree settles. - NodeKind::Dag { .. } | NodeKind::DeployWindow { .. } | NodeKind::AgentWindow { .. } => { - Ok(()) - } + // - `DeployWindow`: pure resource holder — the meta window, agent lease + // and build slot it declares stay held until its subtree settles. + NodeKind::Dag { .. } | NodeKind::DeployWindow { .. } => Ok(()), }; (builder, result) } @@ -260,9 +258,9 @@ async fn run_swap(coord: &Arc, name: &str, id: NodeId) -> Result<() let paths = Coordinator::agent_paths(name, agent_dir); let result = crate::lifecycle::swap_update(name, &hive, &paths, Some(id.get())).await; // On success the Ok-only bookkeeping tail (rev marker, forge/matrix - // sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node, - // which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded - // and the tail `Reconcile` (`AfterAny(RebuildBookkeeping)`) handles recovery; here + // sync, kick, rescan, snapshot) runs in the sibling `PostSwap` node, + // which deps `AfterOk(Swap)`. On failure `PostSwap` is cancel-cascaded + // and the tail `Reconcile` (`AfterAny(PostSwap)`) handles recovery; here // we only refresh the observed state so dashboards reflect the failed // swap immediately. The `Rebuilt { ok: false }` manager event is emitted by // the DAG's `EmitRebuilt` tail (any node may be the one that failed). @@ -277,7 +275,7 @@ async fn run_swap(coord: &Arc, name: &str, id: NodeId) -> Result<() /// means the profile swap succeeded. Store/forge/matrix work only — no nix /// build (build-slot-exempt); the agent lease taken at `Swap` is still held /// (the whole chain up to `Reconcile` is one agent's subgraph). -async fn run_rebuild_bookkeeping(coord: &Arc, name: &str) -> Result<()> { +async fn run_post_swap(coord: &Arc, name: &str) -> Result<()> { if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake) && let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev) { diff --git a/hive-c0re/src/job_queue/model.rs b/hive-c0re/src/job_queue/model.rs index f1f7936d..769b86b6 100644 --- a/hive-c0re/src/job_queue/model.rs +++ b/hive-c0re/src/job_queue/model.rs @@ -53,22 +53,20 @@ pub enum NodeKind { /// `nixos-container update` profile-swap (requires the container /// stopped). Re-applies nspawn flags + resource limits first — /// rebuild is the reconcile verb. The post-rebuild bookkeeping tail - /// lives in the sibling `RebuildBookkeeping` node. + /// lives in the sibling `PostSwap` node. Swap { agent: String }, /// The post-`Swap` bookkeeping tail as a first-class node: rev marker, /// forge + matrix sync, manager kick, container rescan, meta-inputs /// snapshot. Split out of `Swap` for dashboard visibility + retry /// granularity. Deps `AfterOk(Swap)`, so it runs only when the profile - /// swap succeeded; the tail `Reconcile` deps `AfterAny(RebuildBookkeeping)`, so on + /// swap succeeded; the tail `Reconcile` deps `AfterAny(PostSwap)`, so on /// swap failure this node is cancel-cascaded (a terminal state) and - /// recovery still runs. Store/forge/matrix work only — no nix build. - /// - /// Declares **no resources**: it is a coordinated child of - /// [`NodeKind::AgentWindow`], which holds the agent lease (and the build - /// slot) for the whole rebuild subtree. See `templates.rs`'s module doc for - /// why the brace is the one place a resource is declared on behalf of - /// others. - RebuildBookkeeping { agent: String }, + /// recovery still runs. Store/forge/matrix work only — no nix build, so + /// build-slot-exempt. It *does* declare the agent lease: an ancestor in the + /// stop chain already holds it, so this is a re-entrant borrow rather than a + /// second unit — declaring it keeps the requirement true of this node rather + /// than of the one DAG shape it happens to be used in. + PostSwap { agent: String }, /// First-spawn pre-create provisioning: proposed/applied repos, /// state subvolume, and meta registration (`sync_agents`). Runs /// ahead of `Create` so the `nixos-container create --flake @@ -159,15 +157,6 @@ pub enum NodeKind { /// those phases does — the id is the node's own payload, not something a /// DAG-level catch-all hands down. DeployWindow { agent: String, approval_id: i64 }, - /// Group root of a rebuild subtree — the **brace**: declares the agent - /// lease and the build slot, holds both for the whole subtree, and performs - /// no work of its own. Same pure-resource-holder shape as - /// [`NodeKind::DeployWindow`], scoped to one agent. - /// - /// Its children declare no resources and borrow these grants, which is what - /// lets `Prebuild` run beside the `Signal` → `Drain` window. Why braces - /// exist and what they cost: `docs/coordinator.md`, _Braces_. - AgentWindow { agent: String }, /// Deploy phase 1 — **verify only, mutates nothing.** Drift-gate the /// approval's PR head, fetch it into the applied repo, and eval-verify the /// merge head. Any failure here aborts the deploy with the forge state @@ -340,7 +329,7 @@ impl NodeKind { NodeKind::MetaSync { .. } => "meta_sync", NodeKind::Prebuild { .. } => "prebuild", NodeKind::Swap { .. } => "swap", - NodeKind::RebuildBookkeeping { .. } => "rebuild_bookkeeping", + NodeKind::PostSwap { .. } => "post_swap", NodeKind::Provision { .. } => "provision", NodeKind::Create { .. } => "create", NodeKind::MetaLock { .. } => "meta_lock", @@ -354,7 +343,6 @@ impl NodeKind { NodeKind::WritePermFile { .. } => "write_perm_file", NodeKind::Reparent { .. } => "reparent", NodeKind::DeployWindow { .. } => "deploy_window", - NodeKind::AgentWindow { .. } => "agent_window", NodeKind::MergeVerify { .. } => "merge_verify", NodeKind::DeployApply { .. } => "deploy_apply", NodeKind::FinalizeDeploy { .. } => "finalize_deploy", @@ -376,7 +364,7 @@ impl NodeKind { NodeKind::MetaSync { agent, .. } | NodeKind::Prebuild { agent } | NodeKind::Swap { agent } - | NodeKind::RebuildBookkeeping { agent } + | NodeKind::PostSwap { agent } | NodeKind::Provision { agent } | NodeKind::Create { agent } | NodeKind::Reconcile { agent } @@ -388,7 +376,6 @@ impl NodeKind { | NodeKind::WriteDropin { agent } | NodeKind::WritePermFile { agent, .. } | NodeKind::DeployWindow { agent, .. } - | NodeKind::AgentWindow { agent } | NodeKind::MergeVerify { agent, .. } | NodeKind::DeployApply { agent, .. } | NodeKind::FinalizeDeploy { agent, .. } @@ -438,10 +425,5 @@ impl NodeKind { // - `Reconcile` is a planner; it fans out `Start` / `Stop`, which carry // their own answer. // - `DeployWindow` brackets a deploy without itself stopping anything. - // - `AgentWindow` likewise, though it *parents* nodes that answer - // `true`. This is per-node, not per-subtree, and each of those - // children reports for itself — so `true` here would only widen - // suppression over the build and tail, where a vanished container is - // still a real crash. } } diff --git a/hive-c0re/src/job_queue/submit.rs b/hive-c0re/src/job_queue/submit.rs index 3ec3e11b..95a36260 100644 --- a/hive-c0re/src/job_queue/submit.rs +++ b/hive-c0re/src/job_queue/submit.rs @@ -81,10 +81,15 @@ fn stop_chain(builder: &JobBuilder, agent: &str, graceful: bool, running: bool) // Declaration order is dependency order: the quiesce steps come first so // the `Reconcile` that waits on them can name them. if graceful && running { - // `SetWanted` is the brace here, so the quiesce pair borrows its grant - // rather than declaring the lease itself — same shape the rebuild - // template uses, one definition. - let drain = super::templates::quiesce(builder, agent, wanted); + let signal = builder + .node(NodeKind::Signal { agent: a() }) + .needs(Resource::Agent(a())) + .part_of(wanted); + let drain = builder + .node(NodeKind::Drain { agent: a() }) + .needs(Resource::Agent(a())) + .part_of(wanted) + .after_ok(signal); let _ = builder .node(NodeKind::Reconcile { agent: a() }) .needs(Resource::Agent(a())) @@ -150,14 +155,6 @@ fn restart_chain(builder: &JobBuilder, agent: &str, graceful: bool, running: boo // `Reconcile` gates on the last mechanical step. For a non-graceful bounce // that step *is* the root, and the parent gate already orders it — a child // must NOT dep on its own parent (dep-scope), so it takes no sibling edge. - // - // ⚠️ This is the one quiesce site that does NOT use `templates::quiesce`. - // The helper needs a brace holding the lease above the pair; here `Signal` - // *is* the holder, and the nesting under it is what keeps the grant - // continuous across the bounce. Giving this chain its own brace would - // unify all three sites — at the cost of one extra no-op node on every - // graceful restart, which an operator would see. Deliberately not done as - // a side effect of a rebuild-shape change. if graceful { let signal = builder .node(NodeKind::Signal { agent: a() }) diff --git a/hive-c0re/src/job_queue/templates.rs b/hive-c0re/src/job_queue/templates.rs index 536093fc..89a9b206 100644 --- a/hive-c0re/src/job_queue/templates.rs +++ b/hive-c0re/src/job_queue/templates.rs @@ -1,15 +1,19 @@ -//! DAG shape builders — every operation as a template over the shared node -//! primitives. Pure (no I/O); each node carries its own `agent` (there is no -//! DAG-level agent) and **declares its own resources** with `.needs(…)` right -//! where it is constructed, rather than derived from its kind. Deriving made the -//! requirement a property of the *kind*, so a kind running under an ancestor -//! that already held the resource could get away with declaring nothing. +//! DAG shape builders — every operation as a template over the shared +//! node primitives. Pure (no I/O); each node carries its own `agent` (there is +//! no DAG-level agent) and **declares its own resources** with `.needs(…)` +//! right where it is constructed, rather than having them derived from its +//! kind. Deriving made the requirement a property of the *kind*, so a kind that +//! happened to run under an ancestor already holding the resource could get +//! away with declaring nothing. //! -//! **The one sanctioned exception is a brace** — a pure-resource-holder root -//! ([`NodeKind::AgentWindow`], [`NodeKind::DeployWindow`]) declaring for a -//! coordinated subtree whose members then declare nothing. See -//! `docs/coordinator.md`, _Braces_ — which also carries the per-operation DAG -//! shapes, so they are not restated here. +//! ```text +//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a) +//! rebuild(a) graceful: … Prebuild(a) → Signal(a) → Drain(a) → StopForUpdate(a) → … [boot sweep only] +//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve] +//! perm-change(a): WritePermFile(a) → «rebuild subgraph» +//! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a» +//! reparent(moves): Reparent(moves) [no rebuild — topology.json is read live] +//! ``` //! //! Nodes are **named, not counted** — a template holds the handle //! [`JobBuilder::node`] hands back, so an edge says which node it waits on. Why that @@ -18,7 +22,7 @@ //! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here: //! their per-agent shape depends on live running state (an async //! `lifecycle::is_running` read), so `submit.rs` assembles them out of the -//! primitives this module exports ([`rebuild_nodes`]). +//! primitives this module exports ([`rebuild_nodes`]) over `JobBuilder::node`. use hive_jobq::TerminalState; @@ -119,24 +123,6 @@ pub(crate) fn fanned_out_mechanical(builder: &JobBuilder, kind: NodeKind) { let _ = builder.node(kind).needs(lease); } -/// The graceful quiesce pair — `Signal` then `Drain`: ask the agent to run its -/// stop-checkpoint turn, then wait for it to go quiet. Returns the `Drain` -/// handle, which is what a caller edges its stop onto. The container stays -/// **up** throughout; the actual stop is the caller's next node. -/// -/// `brace` must already hold [`Resource::Agent`] for its subtree; the pair -/// declares nothing and borrows it (see the module header). They are -/// dep-ordered **siblings**, not nested — nesting is only load-bearing where -/// `Signal` itself holds the lease, as in `submit.rs`'s `restart_chain`. -pub(crate) fn quiesce<'a>(builder: &'a JobBuilder, agent: &str, brace: Handle<'a>) -> Handle<'a> { - let a = || agent.to_owned(); - let signal = builder.node(NodeKind::Signal { agent: a() }).part_of(brace); - builder - .node(NodeKind::Drain { agent: a() }) - .part_of(brace) - .after_ok(signal) -} - /// The group-roots a [`rebuild_nodes`] subgraph exposes to its caller: what a /// tail node edges onto, and what a follow-up node waits for. /// @@ -146,10 +132,9 @@ pub(crate) fn quiesce<'a>(builder: &'a JobBuilder, agent: &str, brace: Handle<'a pub(crate) struct RebuildRoots<'a> { /// The meta-repo preamble. pub meta_sync: Handle<'a>, - /// The brace holding the agent lease and the build slot — its roll-up - /// carries the whole mechanical subtree (`Prebuild`, the quiesce chain, - /// `StopForUpdate` → `Swap` → `RebuildBookkeeping`). - pub agent_window: Handle<'a>, + /// The build root — its roll-up carries the whole + /// `StopForUpdate` → `Swap` → `PostSwap` subtree. + pub prebuild: Handle<'a>, /// The recovery/convergence tail root. pub reconcile: Handle<'a>, } @@ -157,26 +142,40 @@ pub(crate) struct RebuildRoots<'a> { impl<'a> RebuildRoots<'a> { /// The three roots as a slice, for edging a tail onto all of them. fn all(self) -> [Handle<'a>; 3] { - [self.meta_sync, self.agent_window, self.reconcile] + [self.meta_sync, self.prebuild, self.reconcile] } } -/// The rebuild node subtree — three group roots (`MetaSync`, the `AgentWindow` -/// brace, `Reconcile`). `after`, when given, is the node it chains behind. The -/// shape itself is in `docs/coordinator.md`; the code below is the source of -/// truth for it, so only the three choices a reader would otherwise undo are -/// called out here: -/// -/// - **`MetaSync` is a sibling root, not the brace's parent** — it owns the -/// *global* `MetaWindow`, and a resource is held across the holder's whole -/// subtree, so parenting the rebuild under it would serialise every agent's -/// nix build behind one hive-wide window. -/// - **`StopForUpdate` waits on `Prebuild` as well as `Drain`** — running the -/// drain early is the win; taking the container *down* early is pure downtime. -/// - **`Reconcile` is a top-level root, not a child** — so it survives the -/// cancel-cascade of a failed brace and still converges the container -/// (recovery-start invariant). It takes a fresh lease; the gap is harmless -/// because it is idempotent. +/// The rebuild node subtree (nested, three group roots). `after`, when given, is +/// the node this subgraph chains behind. Structure: +/// - `MetaSync` (**root**): the meta-repo preamble (dir prep, agent sync, +/// optional relock). Owns the global `MetaWindow` — and *only* for its own +/// short duration, which is why it is a sibling root rather than `Prebuild`'s +/// parent: a resource is held across the holder's whole subtree, so parenting +/// the build under it would extend a hive-global window over every rebuild's +/// nix build. +/// - `Prebuild` (**root**): `AfterOk` `MetaSync`. Owns the build slot for the +/// whole mechanical subtree below it. Lease-exempt — the nix build overlaps +/// other DAGs on the same agent. +/// - the **stop root** (child of `Prebuild`): owns the agent lease and runs once +/// `Prebuild` reaches `Finishing` (parent gate). Non-graceful that is +/// `StopForUpdate` itself; graceful it is `Signal`, with `Drain` and then +/// `StopForUpdate` as its children so the lease stays continuous across the +/// whole stop — siblings would each take the lease separately and leave a gap +/// another DAG could claim the agent in, mid-bounce. +/// - `Swap` (child of `StopForUpdate`): borrows the agent lease from its +/// ancestors and the build slot from `Prebuild` — both continuous. +/// - `PostSwap` (child of `StopForUpdate`): the swap's Ok-only bookkeeping tail +/// (rev marker, forge/matrix sync, kick, rescan), `AfterOk` its sibling +/// `Swap`. +/// - `Reconcile` (**last, root**): `AfterAny` `Prebuild`, which rolls up +/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has +/// settled — so `Reconcile` runs after the swap regardless of outcome, and as +/// a top-level root it survives the cancel-cascade of a failed `Prebuild` +/// (recovery-start invariant, which also covers a failed `MetaSync`: that +/// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It +/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to +/// the persisted `wanted` idempotently. fn rebuild_subtree<'a>( builder: &'a JobBuilder, agent: &str, @@ -192,55 +191,59 @@ fn rebuild_subtree<'a>( if let Some(after) = after { meta_sync = meta_sync.after_ok(after); } - // The brace. Both resources are declared here, on one node, on purpose — - // the queue acquires a node's resources atomically, so a single - // multi-resource root can never hold one and block on another. Hoisting the - // slot up costs nothing: a resource is held for the acquirer's whole - // subtree, and the build slot already spanned the entire rebuild when - // `Prebuild` was the one holding it. - let agent_window = builder - .node(NodeKind::AgentWindow { agent: a() }) - .needs(Resource::BuildSlot) - .needs(Resource::Agent(a())) - .after_ok(meta_sync); - - // Siblings under the brace: the build and the quiesce chain run - // concurrently, borrowing the brace's grants rather than declaring their - // own. Declaring the lease on both would make them mutually exclusive — - // it is single-unit — which is exactly what this shape exists to avoid. let prebuild = builder .node(NodeKind::Prebuild { agent: a() }) - .part_of(agent_window); + .needs(Resource::BuildSlot) + .after_ok(meta_sync); - let drain = graceful.then(|| quiesce(builder, agent, agent_window)); - - // The container goes down here, not earlier: `AfterOk` the build so a - // failed build never stops a healthy container, and `AfterOk` the drain so - // the agent has checkpointed. - let mut stop_for_update = builder - .node(NodeKind::StopForUpdate { agent: a() }) - .part_of(agent_window) - .after_ok(prebuild); - if let Some(drain) = drain { - stop_for_update = stop_for_update.after_ok(drain); - } + // The stop root hangs off `Prebuild` and owns the agent lease for + // everything below it. `StopForUpdate` parents the swap pair either way. + let stop_for_update = if graceful { + let signal = builder + .node(NodeKind::Signal { agent: a() }) + .needs(Resource::Agent(a())) + .part_of(prebuild); + // `Drain` is a *child* of `Signal`, so the parent gate already orders + // it — a child must not dep on its own parent (dep-scope). + let drain = builder + .node(NodeKind::Drain { agent: a() }) + .needs(Resource::Agent(a())) + .part_of(signal); + builder + .node(NodeKind::StopForUpdate { agent: a() }) + .needs(Resource::Agent(a())) + .part_of(signal) + .after_ok(drain) + } else { + builder + .node(NodeKind::StopForUpdate { agent: a() }) + .needs(Resource::Agent(a())) + .part_of(prebuild) + }; let swap = builder .node(NodeKind::Swap { agent: a() }) + .needs(Resource::BuildSlot) + .needs(Resource::Agent(a())) .part_of(stop_for_update); - let _rebuild_bookkeeping = builder - .node(NodeKind::RebuildBookkeeping { agent: a() }) + // `PostSwap` declares the lease it actually runs under. It is a child of + // `StopForUpdate`, which holds it, so this is a re-entrant borrow — no + // second unit, no deadlock. Declaring it is what stops the requirement + // being true only of this one DAG shape. + let _post_swap = builder + .node(NodeKind::PostSwap { agent: a() }) + .needs(Resource::Agent(a())) .part_of(stop_for_update) .after_ok(swap); let reconcile = builder .node(NodeKind::Reconcile { agent: a() }) .needs(Resource::Agent(a())) - .after_any(agent_window); + .after_any(prebuild); RebuildRoots { meta_sync, - agent_window, + prebuild, reconcile, } } @@ -285,7 +288,7 @@ pub(crate) fn graceful_rebuild_nodes<'a>( /// `FinalizeDeploy` waits on **two** roots, which together reproduce the gate /// the old fused node had around its inline `rebuild_no_meta` call: /// - `AfterOk` `Prebuild` — a parent's state is its roll-up, so this is `Done` -/// only once `StopForUpdate` → `Swap` → `RebuildBookkeeping` all are (a failed *or* +/// only once `StopForUpdate` → `Swap` → `PostSwap` all are (a failed *or* /// cancelled child rolls the parent up `Failed`). That's the old /// `build_result`. /// - `AfterOk` `Reconcile` — the old call passed `deferred_start = false` on @@ -306,7 +309,7 @@ pub(crate) fn deploy_rebuild_nodes(builder: &JobBuilder, agent: &str, approval_i approval_id, }) .needs(Resource::MetaWindow) - .after_ok(roots.agent_window) + .after_ok(roots.prebuild) .after_ok(roots.reconcile); } @@ -318,7 +321,7 @@ pub(crate) fn deploy_rebuild_nodes(builder: &JobBuilder, agent: &str, approval_i /// /// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots /// (`MetaSync`, `Prebuild`, `Reconcile`) — `Prebuild`'s roll-up carries the -/// whole `StopForUpdate`→`Swap`→`RebuildBookkeeping` subtree, so those three cover every +/// whole `StopForUpdate`→`Swap`→`PostSwap` subtree, so those three cover every /// node. Edging `Reconcile` alone would not do: it is `AfterAny` `Prebuild`, so /// it reaches `Done` even after a failed swap and the tail would report success. pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) { @@ -441,7 +444,7 @@ pub fn perm_change(builder: &JobBuilder, agent: &str, payload: PermPayload) { emit_rebuilt_tails( builder, agent, - &[write, roots.meta_sync, roots.agent_window, roots.reconcile], + &[write, roots.meta_sync, roots.prebuild, roots.reconcile], ); } diff --git a/hive-c0re/src/job_queue/tests.rs b/hive-c0re/src/job_queue/tests.rs index 4bc199a9..10759af5 100644 --- a/hive-c0re/src/job_queue/tests.rs +++ b/hive-c0re/src/job_queue/tests.rs @@ -390,44 +390,23 @@ fn rebuild_chain_is_declared_serial() { // observe an order that is fully declared the moment `submit` returns. // // ⚠️ The old name was also wrong about the mechanism, and reading it rather - // than the graph is how you'd stay wrong: **only part of this chain is dep - // edges.** `swap` declares no deps at all — it is ordered by *parent - // nesting* ("a node's sub-nodes run after its own logic"). Both axes are - // asserted below because a template can break either one independently. - // - // ⚠️ "Serial" is now about the *declared* order, not about concurrency: - // `prebuild` and the graceful quiesce chain are siblings under the brace and - // run **in parallel** (see `graceful_rebuild_chain_drains_before_stopping`). - // The non-graceful shape asserted here has no quiesce chain, so nothing here - // is concurrent — but the name would mislead about the graceful one. + // than the graph is how you'd stay wrong: **only half this chain is dep + // edges.** `stop_for_update` and `swap` declare no deps at all — they are + // ordered by *parent nesting* ("a node's sub-nodes run after its own + // logic"). Both axes are asserted below because a template can break either + // one independently. let q = JobQueue::new(1); let id = submit(&q, "r", |builder| rebuild(builder, "agent-a")); assert_eq!( declared_shape(&q, id), vec![ row("meta_sync", None, &[]), - // The brace: holds the lease + slot for everything nested below it. - row("agent_window", None, &[("meta_sync", "done")]), - // No dep: ordered by hanging under the brace. - row("prebuild", Some("agent_window"), &[]), - // A real dep, not nesting: the container must not go down until the - // build that replaces it has succeeded. - row( - "stop_for_update", - Some("agent_window"), - &[("prebuild", "done")] - ), + row("prebuild", None, &[("meta_sync", "done")]), + // No dep: ordered by hanging under `prebuild`. + row("stop_for_update", Some("prebuild"), &[]), row("swap", Some("stop_for_update"), &[]), - row( - "rebuild_bookkeeping", - Some("stop_for_update"), - &[("swap", "done")] - ), - row( - "reconcile", - None, - &[("agent_window", "done|failed|skipped")] - ), + row("post_swap", Some("stop_for_update"), &[("swap", "done")]), + row("reconcile", None, &[("prebuild", "done|failed|skipped")]), // The tail pair. The ok tail needs every root to succeed; the !ok // tail hangs off the ok tail's *elimination* (`skipped`), which is // what makes exactly one of them run. @@ -436,7 +415,7 @@ fn rebuild_chain_is_declared_serial() { None, &[ ("meta_sync", "done"), - ("agent_window", "done"), + ("prebuild", "done"), ("reconcile", "done"), ], ), @@ -446,7 +425,7 @@ fn rebuild_chain_is_declared_serial() { &[ ("emit_rebuilt", "skipped"), ("meta_sync", "done|failed|skipped"), - ("agent_window", "done|failed|skipped"), + ("prebuild", "done|failed|skipped"), ("reconcile", "done|failed|skipped"), ], ), @@ -454,17 +433,12 @@ fn rebuild_chain_is_declared_serial() { ); } -/// The boot sweep's graceful shape: the agent gets `Signal` → `Drain` to finish -/// its turn before `StopForUpdate` takes the container down — **concurrently -/// with the nix build**, which is the whole reason the brace exists. The drain -/// window costs nothing it doesn't already cost; nesting it under `Prebuild` -/// used to hide the entire `GRACEFUL_STOP_TIMEOUT` behind the build. -/// -/// Lease continuity across the bounce is preserved by `AgentWindow` holding the -/// lease above all of them, which is what makes them safe as siblings — the -/// older shape had to nest `Signal` over the rest of the stop for exactly that -/// reason, since otherwise each would acquire the lease separately and leave a -/// window for another DAG to claim the agent mid-stop. +/// The boot sweep's graceful shape: the agent gets `Signal` → `Drain` to +/// finish its turn before `StopForUpdate` takes the container down. `Signal` +/// *parents* the rest of the stop rather than sitting beside it, so the agent +/// lease is held continuously across the whole bounce — as siblings, each of +/// `Signal` / `Drain` / `StopForUpdate` would acquire the lease separately and +/// leave a window for another DAG to claim the agent mid-stop. #[test] fn graceful_rebuild_chain_drains_before_stopping() { let q = JobQueue::new(1); @@ -477,53 +451,30 @@ fn graceful_rebuild_chain_drains_before_stopping() { }, ) .expect("valid shape"); - // Asserted as full rows, not just kinds: the kind list is identical whether - // the quiesce chain runs beside the build or nested under it, so a - // kind-only assertion cannot see the bug this shape exists to fix. assert_eq!( - declared_shape(&q, id), + declared_shape(&q, id) + .iter() + .map(|d| d.kind) + .collect::>(), vec![ - row("meta_sync", None, &[]), - row("agent_window", None, &[("meta_sync", "done")]), - row("prebuild", Some("agent_window"), &[]), - // 🎯 **The fix, and the thing to guard.** `signal` hangs off the - // *brace*, not off `prebuild`, and declares no dep on it — so the - // drain window runs concurrently with the nix build instead of - // behind it. Both halves matter: re-parenting it under `prebuild` - // OR adding an `AfterOk(prebuild)` edge would each silently restore - // the original defect (up to GRACEFUL_STOP_TIMEOUT hidden behind - // every agent's build, on every boot sweep). - row("signal", Some("agent_window"), &[]), - // Siblings under the brace, dep-ordered — not nested. Nesting is - // only needed where `Signal` itself holds the lease. - row("drain", Some("agent_window"), &[("signal", "done")]), - // The container still goes down only when *both* are ready: the - // build succeeded and the agent has checkpointed. Running the drain - // early is the win; stopping early would just be downtime. - row( - "stop_for_update", - Some("agent_window"), - &[("prebuild", "done"), ("drain", "done")] - ), - row("swap", Some("stop_for_update"), &[]), - row( - "rebuild_bookkeeping", - Some("stop_for_update"), - &[("swap", "done")] - ), - row( - "reconcile", - None, - &[("agent_window", "done|failed|skipped")] - ), + "meta_sync", + "prebuild", + // The graceful window goes between the build and the stop: the + // agent gets its turn to finish before the container goes down. + "signal", + "drain", + "stop_for_update", + "swap", + "post_swap", + "reconcile", ], - "graceful runs signal + drain beside the build, and stops only after both" + "graceful inserts signal + drain ahead of the stop, and nothing else" ); } /// The non-graceful shape is the default everywhere except the boot sweep: /// a manual rebuild, a meta-update cascade child and a deploy must NOT spend a -/// drain window, so `StopForUpdate` waits only on `Prebuild`. +/// drain window, so `StopForUpdate` still hangs straight off `Prebuild`. #[test] fn non_graceful_rebuild_has_no_signal_or_drain() { // Read the shape off the queue rather than out of a node list: a declared @@ -540,32 +491,32 @@ fn non_graceful_rebuild_has_no_signal_or_drain() { .collect::>(), vec![ "meta_sync", - "agent_window", "prebuild", "stop_for_update", "swap", - "rebuild_bookkeeping", + "post_swap", "reconcile" ], - "exactly seven nodes, and none of them is signal or drain" + "exactly six nodes, and neither of them is signal or drain" ); } // ---- build slots ---- #[test] -fn rebuild_chain_declares_its_resources_on_the_brace() { - // Was `rebuild_chain_declares_the_slot_where_the_nix_work_is` (and before - // that `fifo_fairness_for_the_slot`). +fn rebuild_chain_declares_the_slot_where_the_nix_work_is() { + // Was `fifo_fairness_for_the_slot`, which submitted three rebuilds and + // drove one to completion to watch the freed slot go to the earlier + // waiter. **That fairness guarantee is hive_jobq's**, and it had no test + // there at all — its claim primitive scans nodes in insertion order and + // takes the first satisfiable one, and nothing pinned that. It does now: + // `a_contended_resource_goes_to_the_oldest_waiter`. // - // ⚠️ **The rename is a reversal, not tidying.** The old name asserted "the - // slot follows the nix work and the lease follows the container" — each node - // declaring what it personally needed. The brace inverts that for a - // coordinated subtree; see `docs/coordinator.md`, _Braces_. - // - // (`hive_jobq` owns slot *fairness*, pinned there by - // `a_contended_resource_goes_to_the_oldest_waiter`. What is c0re's is - // *which* nodes contend in the first place — a declaration, asserted here.) + // What is c0re's is *which* nodes contend for the slot in the first place, + // and that is a declaration. "Uniform hold across the chain" then follows + // from the parent nesting asserted in `rebuild_chain_is_declared_serial`: + // a resource unit is held for the acquirer's whole subtree, so the slot + // `Prebuild` takes covers `StopForUpdate` → `Swap` → `PostSwap` beneath it. let q = JobQueue::new(1); let id = submit(&q, "r", |builder| rebuild(builder, "agent-a")); let res = |kind: &str| declared_resources(&q, node_of(&q, id, kind)); @@ -574,7 +525,6 @@ fn rebuild_chain_declares_its_resources_on_the_brace() { assert_eq!( [ res("meta_sync"), - res("agent_window"), res("prebuild"), res("stop_for_update"), res("swap"), @@ -582,28 +532,21 @@ fn rebuild_chain_declares_its_resources_on_the_brace() { ], [ // The meta preamble takes the global window and *nothing else* — - // no slot (it does no nix work) and no lease. It stays a sibling - // root so that hive-global window is not held across any build. + // no slot (it does no nix work) and no lease. vec![Resource::MetaWindow], - // The brace takes both, atomically, and holds them for its whole - // subtree. Hoisting the slot here is not a widening: it already - // spanned the entire rebuild when `Prebuild` held it, because a unit - // is held until the acquirer's subtree settles and everything below - // was inside `Prebuild`. + // The nix build is the slot-needer, and takes **no lease**. That is + // what lets a prebuild overlap another DAG on the same agent: the + // container is still up and untouched while it builds. + vec![Resource::BuildSlot], + // The lease starts here — the first node that touches the + // container — and not one node earlier. + vec![agent()], + // Swap needs both. It re-enters the slot its `Prebuild` ancestor + // holds rather than acquiring a second unit. vec![Resource::BuildSlot, agent()], - // The coordinated children declare nothing and re-enter the brace's - // grants. An empty vec here is the *point* of the shape, not an - // omission — if any of these regains a declaration it will silently - // stop running in parallel with its siblings. - vec![], - vec![], - vec![], - // `Reconcile` is the exception that stays: a top-level root outside - // the brace, so it takes a genuinely fresh lease after the window - // has released. vec![agent()], ], - "the brace declares for the subtree; coordinated children declare nothing" + "the slot follows the nix work and the lease follows the container" ); } @@ -709,24 +652,11 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() { vec![ row("set_wanted", None, &[]), row("meta_sync", None, &[("set_wanted", "done")]), - row("agent_window", None, &[("meta_sync", "done")]), - row("prebuild", Some("agent_window"), &[]), - row( - "stop_for_update", - Some("agent_window"), - &[("prebuild", "done")] - ), + row("prebuild", None, &[("meta_sync", "done")]), + row("stop_for_update", Some("prebuild"), &[]), row("swap", Some("stop_for_update"), &[]), - row( - "rebuild_bookkeeping", - Some("stop_for_update"), - &[("swap", "done")] - ), - row( - "reconcile", - None, - &[("agent_window", "done|failed|skipped")] - ), + row("post_swap", Some("stop_for_update"), &[("swap", "done")]), + row("reconcile", None, &[("prebuild", "done|failed|skipped")]), ], "a stale agent gets the whole rebuild chain wedged between intent and \ convergence — same DAG, same head kind, more in the middle" @@ -877,29 +807,26 @@ fn rebuild_chain_nodes_suppress_crash_watch() { // individually legible, and the arrangement was the only reason this module // needed to claim and complete nodes. -/// The swap-success path: `Swap` ok → the `AfterOk` `RebuildBookkeeping` (bookkeeping +/// The swap-success path: `Swap` ok → the `AfterOk` `PostSwap` (bookkeeping /// tail) runs, and only then does `Reconcile` fire — serialized behind -/// `RebuildBookkeeping` (not racing it) because `Reconcile` deps `AfterAny(RebuildBookkeeping)`. +/// `PostSwap` (not racing it) because `Reconcile` deps `AfterAny(PostSwap)`. #[test] fn rebuild_reconcile_waits_for_the_whole_build_subtree() { // Replaces `swap_ok_runs_post_swap_before_reconcile` and // `swap_failure_still_runs_reconcile`, which walked the same DAG with the // swap succeeding in one and failing in the other. // - // The interesting claim was "Reconcile must wait for RebuildBookkeeping, not race + // The interesting claim was "Reconcile must wait for PostSwap, not race // it" — and it does *not* come from an edge between them. `reconcile` deps - // `AfterAny(agent_window)`, while `rebuild_bookkeeping` sits inside the brace's - // subtree (rebuild_bookkeeping → stop_for_update → agent_window). A parent is not - // terminal until its subtree is, so the brace cannot satisfy that edge while - // rebuild_bookkeeping is outstanding. **The ordering is the parent chain, not a - // dependency.** + // `AfterAny(prebuild)`, while `post_swap` sits inside prebuild's subtree + // (post_swap → stop_for_update → prebuild). A parent is not terminal until + // its subtree is, so prebuild cannot satisfy that edge while post_swap is + // outstanding. **The ordering is the parent chain, not a dependency.** // // Both facts are asserted in `rebuild_chain_is_declared_serial`; this test // states the derived property explicitly because the indirection is the // easy thing to break — someone flattening the chain would keep every edge - // and still lose the guarantee. That warning earned itself: `AgentWindow` - // **did** flatten this chain, and the guarantee survives only because the - // roll-up point moved with it. + // and still lose the guarantee. let q = JobQueue::new(1); let id = submit(&q, "r", |builder| rebuild(builder, "agent-a")); let shape = declared_shape(&q, id); @@ -910,23 +837,23 @@ fn rebuild_reconcile_waits_for_the_whole_build_subtree() { .unwrap_or_else(|| panic!("{kind} node")) .parent }; - assert_eq!(parent_of("rebuild_bookkeeping"), Some("stop_for_update")); - assert_eq!(parent_of("stop_for_update"), Some("agent_window")); + assert_eq!(parent_of("post_swap"), Some("stop_for_update")); + assert_eq!(parent_of("stop_for_update"), Some("prebuild")); assert_eq!( shape .iter() .find(|d| d.kind == "reconcile") .expect("reconcile node") .after, - vec![("agent_window", "done|failed|skipped".to_owned())], - "reconcile gates on the brace's roll-up, which covers the whole \ - subtree — including rebuild_bookkeeping — and runs on failure too" + vec![("prebuild", "done|failed|skipped".to_owned())], + "reconcile gates on prebuild's roll-up, which covers the whole build \ + subtree — including post_swap — and runs on failure too" ); } // `swap_failure_still_runs_reconcile` lived here. // -// It asserted that a failed swap leaves `rebuild_bookkeeping` `Skipped` and `swap` +// It asserted that a failed swap leaves `post_swap` `Skipped` and `swap` // `Failed`, and that reconcile still runs. All three are hive_jobq's cascade // (`failed_after_ok_dep_cancels_dependents_but_after_any_still_runs`), and the // "says so on the wire" half turned out to be nothing: `snapshot` fills @@ -1295,24 +1222,11 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() { declared_shape(&q, id), vec![ row("meta_sync", None, &[]), - row("agent_window", None, &[("meta_sync", "done")]), - row("prebuild", Some("agent_window"), &[]), - row( - "stop_for_update", - Some("agent_window"), - &[("prebuild", "done")] - ), + row("prebuild", None, &[("meta_sync", "done")]), + row("stop_for_update", Some("prebuild"), &[]), row("swap", Some("stop_for_update"), &[]), - row( - "rebuild_bookkeeping", - Some("stop_for_update"), - &[("swap", "done")] - ), - row( - "reconcile", - None, - &[("agent_window", "done|failed|skipped")] - ), + row("post_swap", Some("stop_for_update"), &[("swap", "done")]), + row("reconcile", None, &[("prebuild", "done|failed|skipped")]), // The deploy tag is planted only after the rebuild came up clean: // `AfterOk` on **both** roots, so either one failing skips it. That // pair of edges is the whole "skips finalize on a failed graft" @@ -1320,7 +1234,7 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() { row( "finalize_deploy", None, - &[("agent_window", "done"), ("reconcile", "done")] + &[("prebuild", "done"), ("reconcile", "done")] ), ] ); @@ -1481,24 +1395,18 @@ fn graceful_stop_shape_signal_drain_reconcile() { row("reconcile", Some("set_wanted"), &[("drain", "done")]), ] ); - // The quiesce pair declares **nothing**: `set_wanted` is the brace and holds - // the lease for the whole subtree, so both borrow that grant. Same shape, - // same helper (`templates::quiesce`) as the rebuild's window. - // - // ⚠️ In particular neither takes a build slot — which is what lets a - // whole-hive graceful *stop* overlap every agent's drain even at - // `buildSlots = 1`: the ceiling is one `GRACEFUL_STOP_TIMEOUT` in total, - // not one per agent. That holds here because nothing in a stop chain is - // slot-needing. It does **not** hold for the boot *sweep*, whose rebuilds - // do hold the slot across their drains — see the note on the sweep in - // `exec.rs`. + // Neither half of the graceful window takes a build slot. That is what lets + // a whole-hive graceful stop overlap every agent's drain even at + // `buildSlots = 1` while a rebuild hogs the slot — the cost ceiling is one + // `GRACEFUL_STOP_TIMEOUT` in total, not one per agent. + let agent = Resource::Agent("agent-a".to_owned()); assert_eq!( [ declared_resources(&q, node_of(&q, id, "signal")), declared_resources(&q, node_of(&q, id, "drain")), ], - [vec![], vec![]], - "the quiesce pair borrows the brace's lease and declares nothing" + [vec![agent.clone()], vec![agent]], + "signal and drain hold the lease but never a build slot" ); } @@ -1548,11 +1456,10 @@ fn perm_change_shape_prefixes_rebuild_chain() { vec![ "write_perm_file", "meta_sync", - "agent_window", "prebuild", "stop_for_update", "swap", - "rebuild_bookkeeping", + "post_swap", "reconcile", // the ok / !ok tail pair "emit_rebuilt",