feat(job-queue): promote the meta-repo deploy window to a queue resource

The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions

View file

@ -33,7 +33,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node |
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
@ -43,6 +43,7 @@ Cheap — no build slot:
| Node | Behavior |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event still fires once per DAG from the terminal hook, not here) |
@ -60,13 +61,19 @@ Two further layers protect the meta repo across *windows* that span multiple
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
container build:
- **The deploy-window gate** (`meta::exclusive()`): every executor that
mutates the meta repo (`Prebuild`'s sync+relock, `MetaLock`,
`WritePermFile`, `Create`'s agent registration, and `ApprovalDeploy` for
its whole span) holds this async mutex for its mutation span, so no commit
can land inside another node's staged window. `Prebuild` drops it before
the long toplevel build (store reads only), preserving `buildSlots > 1`
concurrency.
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
resource declared by every node kind that mutates the meta repo — `MetaSync`,
`MetaLock`, `WritePermFile`, `Provision`'s agent registration, and
`ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta
mutations can therefore never interleave, so no commit lands inside another
node's staged window. It is a queue resource rather than a runtime mutex
because a resource is held by a subtree root across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
multi-node deploy own one window. For the same reason the window must stay
*off* long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
- **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's
@ -100,7 +107,7 @@ sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
agent's subgraph is a rebuild-then-start).
```text
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
@ -178,8 +185,8 @@ resources are free. Resources:
touching several agents holds one lease per agent. (`SetWanted` is a store
write, not a container op, but takes the lease anyway so a power-op DAG's
intent write + reconcile is atomic — two racing ops can't clobber intent
before either reconciles.) **Lease-exempt**: `Prebuild`, `MetaLock`,
`WritePermFile` —
before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`,
`MetaLock`, `WritePermFile` —
they touch the store / meta, not the running container, which is exactly
why a stop can land while another DAG's prebuild is still building.
@ -311,16 +318,20 @@ one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node):
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
meta-update cascade child) the per-agent relock. Short, and the only step
that mutates the meta repo, so it is the only one holding the global deploy
window.
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface
here, before the running container is touched. (Runs even for a stopped
container — same total nix work, one uniform DAG shape.)
2. `StopForUpdate` — bring the container down (noop when already stopped).
3. `Swap``nixos-container update --flake meta#<name>` profile-swap
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap``nixos-container update --flake meta#<name>` profile-swap
(near-instant after the prebuild).
4. `Reconcile` — boot into the new generation when `wanted = Up`; the
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
"deferred start" split, now structural.