feat(job-queue): promote the meta-repo deploy window to a queue resource

The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions

View file

@ -27,24 +27,17 @@ const GIT_EMAIL: &str = "c0re@hyperhive.local";
/// take turns instead of colliding.
static META_LOCK: Mutex<()> = Mutex::const_new(());
/// Coarse exclusivity for meta-repo *windows* that span multiple
/// `META_LOCK` acquisitions — above all the two-phase deploy
/// (`prepare_deploy` stages `flake.lock` uncommitted for the whole
/// container build; `finalize_deploy` / `abort_deploy` resolve it).
/// `META_LOCK` serializes individual git ops but cannot keep another
/// op out of that staged window: a perm-file or lock-bump commit
/// landing mid-window would sweep the staged deploy lock into its own
/// commit and neuter `abort_deploy`. Job-queue executors that mutate
/// the meta repo hold this gate for their mutation span; the opaque
/// approval-deploy node holds it across its whole prepare→finalize
/// span. Never acquired inside this module's functions (they run
/// *under* a caller's window — nesting would deadlock).
static DEPLOY_GATE: Mutex<()> = Mutex::const_new(());
/// Acquire the deploy/meta-mutation window gate. See [`DEPLOY_GATE`].
pub async fn exclusive() -> tokio::sync::MutexGuard<'static, ()> {
DEPLOY_GATE.lock().await
}
// Exclusivity for meta-repo *windows* that span multiple `META_LOCK`
// acquisitions — above all the two-phase deploy (`prepare_deploy` stages
// `flake.lock` uncommitted for the whole container build;
// `finalize_deploy` / `abort_deploy` resolve it) — is **not** a mutex in
// this module. `META_LOCK` above serializes individual git ops but cannot
// keep another op out of that staged window; that window is owned by the
// job queue instead, as `Resource::MetaWindow`, declared by every
// meta-mutating node kind (`NodeKind::needs_meta_window`). A resource can
// be held by a subtree root across its children, which a `MutexGuard`
// (bounded by one executor fn) cannot — that's what lets the deploy be
// modelled as sub-nodes rather than one opaque node.
/// Where the manager sees this directory inside its container (RO bind).
pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta";