feat(job-queue): promote the meta-repo deploy window to a queue resource
The two-phase approval deploy keeps a bumped `flake.lock` staged uncommitted for the whole container build, so no other meta mutation may land inside that span — until now enforced by a process-global `meta::exclusive()` mutex held inside each executor fn. A `MutexGuard` cannot outlive the fn that takes it, which is what blocks decomposing the opaque `ApprovalDeploy` node into scheduler-visible sub-nodes: the window has to span them. Replace the mutex with `Resource::MetaWindow`, a global capacity-1 queue resource declared by every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources are held by a subtree root across its whole subtree, so a later increment can hang the deploy's phases under one window-holding parent. Same global serialisation as before, and the scheduler now blocks a node from being claimed rather than parking a worker on a mutex. Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync` node. `Prebuild` must NOT hold the window: the old mutex was deliberately scoped to drop before the multi-minute toplevel build, which only reads the store, and a cap-1 global held across it would serialise every agent's rebuild behind every other's. `MetaSync` is a sibling root that `Prebuild` deps `AfterOk` on — not its parent, since a parent's resource covers its whole subtree and would reintroduce exactly that problem. Queue tests: shape assertions gain the extra node, which is the point of the change (phases become nodes). The concurrency invariants are intact but observed one step later — the `MetaSync` heads take turns on the window, exactly as the runtime mutex made them, so those tests now complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
parent
2316287327
commit
dfadacd45f
8 changed files with 311 additions and 148 deletions
|
|
@ -33,7 +33,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
|||
|
||||
| Node | Wraps |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node |
|
||||
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
|
|
@ -43,6 +43,7 @@ Cheap — no build slot:
|
|||
|
||||
| Node | Behavior |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event still fires once per DAG from the terminal hook, not here) |
|
||||
|
|
@ -60,13 +61,19 @@ Two further layers protect the meta repo across *windows* that span multiple
|
|||
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
|
||||
container build:
|
||||
|
||||
- **The deploy-window gate** (`meta::exclusive()`): every executor that
|
||||
mutates the meta repo (`Prebuild`'s sync+relock, `MetaLock`,
|
||||
`WritePermFile`, `Create`'s agent registration, and `ApprovalDeploy` for
|
||||
its whole span) holds this async mutex for its mutation span, so no commit
|
||||
can land inside another node's staged window. `Prebuild` drops it before
|
||||
the long toplevel build (store reads only), preserving `buildSlots > 1`
|
||||
concurrency.
|
||||
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
|
||||
resource declared by every node kind that mutates the meta repo — `MetaSync`,
|
||||
`MetaLock`, `WritePermFile`, `Provision`'s agent registration, and
|
||||
`ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta
|
||||
mutations can therefore never interleave, so no commit lands inside another
|
||||
node's staged window. It is a queue resource rather than a runtime mutex
|
||||
because a resource is held by a subtree root across its whole subtree, which
|
||||
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
|
||||
multi-node deploy own one window. For the same reason the window must stay
|
||||
*off* long store-only work: the rebuild's meta preamble is its own
|
||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
||||
toplevel build runs outside the window and `buildSlots > 1` still gives
|
||||
concurrent rebuilds across agents.
|
||||
- **Path-limited commits**: the targeted meta committers (perm files,
|
||||
topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
|
||||
dirty checks, so even a non-queue caller (boot migration, destroy's
|
||||
|
|
@ -100,7 +107,7 @@ sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
|
|||
agent's subgraph is a rebuild-then-start).
|
||||
|
||||
```text
|
||||
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
|
||||
rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
|
||||
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
|
||||
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
|
||||
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
|
||||
|
|
@ -178,8 +185,8 @@ resources are free. Resources:
|
|||
touching several agents holds one lease per agent. (`SetWanted` is a store
|
||||
write, not a container op, but takes the lease anyway so a power-op DAG's
|
||||
intent write + reconcile is atomic — two racing ops can't clobber intent
|
||||
before either reconciles.) **Lease-exempt**: `Prebuild`, `MetaLock`,
|
||||
`WritePermFile` —
|
||||
before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`,
|
||||
`MetaLock`, `WritePermFile` —
|
||||
they touch the store / meta, not the running container, which is exactly
|
||||
why a stop can land while another DAG's prebuild is still building.
|
||||
|
||||
|
|
@ -311,16 +318,20 @@ one go, rather than the double-bounce a live `update` would trigger.
|
|||
|
||||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
|
||||
meta-update cascade child) the per-agent relock. Short, and the only step
|
||||
that mutates the meta repo, so it is the only one holding the global deploy
|
||||
window.
|
||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
build happen out-of-band. `nixos-container update` then finds the result
|
||||
cached and skips straight to the profile-swap. Build failures surface
|
||||
here, before the running container is touched. (Runs even for a stopped
|
||||
container — same total nix work, one uniform DAG shape.)
|
||||
2. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
3. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
||||
(near-instant after the prebuild).
|
||||
4. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
|
||||
"deferred start" split, now structural.
|
||||
|
|
|
|||
Loading…
Reference in a new issue