vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(job-queue): promote the meta-repo deploy window to a queue resource

Browse source 
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 • committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

41
docs/coordinator.md
View file

@ -33,7 +33,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps | | Node | Wraps |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving | | `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node | | `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node |
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) | | `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion | | `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
@ -43,6 +43,7 @@ Cheap — no build slot:
| Node | Behavior | | Node | Behavior |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event still fires once per DAG from the terminal hook, not here) | | `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event still fires once per DAG from the terminal hook, not here) |
@ -60,13 +61,19 @@ Two further layers protect the meta repo across *windows* that span multiple
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
container build: container build:
- **The deploy-window gate** (`meta::exclusive()`): every executor that - **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
mutates the meta repo (`Prebuild`'s sync+relock, `MetaLock`, resource declared by every node kind that mutates the meta repo — `MetaSync`,
`WritePermFile`, `Create`'s agent registration, and `ApprovalDeploy` for `MetaLock`, `WritePermFile`, `Provision`'s agent registration, and
its whole span) holds this async mutex for its mutation span, so no commit `ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta
can land inside another node's staged window. `Prebuild` drops it before mutations can therefore never interleave, so no commit lands inside another
the long toplevel build (store reads only), preserving `buildSlots > 1` node's staged window. It is a queue resource rather than a runtime mutex
concurrency. because a resource is held by a subtree root across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
multi-node deploy own one window. For the same reason the window must stay
*off* long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
- **Path-limited commits**: the targeted meta committers (perm files, - **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's dirty checks, so even a non-queue caller (boot migration, destroy's
@ -100,7 +107,7 @@ sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
agent's subgraph is a rebuild-then-start). agent's subgraph is a rebuild-then-start).
```text ```text
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a) rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a) stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG) offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted) restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
@ -178,8 +185,8 @@ resources are free. Resources:
touching several agents holds one lease per agent. (`SetWanted` is a store touching several agents holds one lease per agent. (`SetWanted` is a store
write, not a container op, but takes the lease anyway so a power-op DAG's write, not a container op, but takes the lease anyway so a power-op DAG's
intent write + reconcile is atomic — two racing ops can't clobber intent intent write + reconcile is atomic — two racing ops can't clobber intent
before either reconciles.) **Lease-exempt**: `Prebuild`, `MetaLock`, before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`,
`WritePermFile` — `MetaLock`, `WritePermFile` —
they touch the store / meta, not the running container, which is exactly they touch the store / meta, not the running container, which is exactly
why a stop can land while another DAG's prebuild is still building. why a stop can land while another DAG's prebuild is still building.
@ -311,16 +318,20 @@ one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node): Sequence for a rebuild DAG (each step is its own queue node):
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping. 1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
meta-update cascade child) the per-agent relock. Short, and the only step
that mutates the meta repo, so it is the only one holding the global deploy
window.
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch + The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface cached and skips straight to the profile-swap. Build failures surface
here, before the running container is touched. (Runs even for a stopped here, before the running container is touched. (Runs even for a stopped
container — same total nix work, one uniform DAG shape.) container — same total nix work, one uniform DAG shape.)
2. `StopForUpdate` — bring the container down (noop when already stopped). 3. `StopForUpdate` — bring the container down (noop when already stopped).
3. `Swap` — `nixos-container update --flake meta#<name>` profile-swap 4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
(near-instant after the prebuild). (near-instant after the prebuild).
4. `Reconcile` — boot into the new generation when `wanted = Up`; the 5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot — the old slot, so the next DAG's `Prebuild` overlaps the container boot — the old
"deferred start" split, now structural. "deferred start" split, now structural.

93
hive-c0re/src/job_queue/exec.rs
View file

@ -82,7 +82,8 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
node_id: claim.node_id, node_id: claim.node_id,
}; };
match &claim.kind { match &claim.kind {
NodeKind::Prebuild { relock, .. } => run_prebuild(coord, claim, &ctx, *relock).await, NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await,
NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await, NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, claim, &ctx).await, NodeKind::PostSwap { .. } => run_post_swap(coord, claim, &ctx).await,
NodeKind::Provision { .. } => run_provision(coord, claim, &ctx).await, NodeKind::Provision { .. } => run_provision(coord, claim, &ctx).await,
@ -186,24 +187,24 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<N
Ok(NodeOutput::default()) Ok(NodeOutput::default())
} }
/// Out-of-band toplevel build while the container keeps serving: meta /// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
/// sync + optional per-agent relock, then warm /// `sync_agents`, and the optional per-agent relock. Runs under the deploy
/// `system.build.toplevel` so the later `Swap` hits cache and skips /// window (`NodeKind::needs_meta_window`, held by the scheduler for this
/// straight to the profile-swap. The warm build is skipped when the /// node) so its commits can never land inside another node's staged
/// container is already down: its only purpose is to shrink the swap's /// prepare→finalize window.
/// downtime window, so a stopped agent (no uptime to preserve) doesn't ///
/// pay the double eval — `Swap` builds inline instead. The meta sync + /// Deliberately a separate node from the [`run_prebuild`] it feeds: that
/// dir prep still run unconditionally (the `Swap` depends on them). /// build takes minutes and only *reads* the store, so keeping the global
async fn run_prebuild( /// window off it is what lets rebuilds of different agents overlap.
async fn run_meta_sync(
coord: &Arc<Coordinator>, coord: &Arc<Coordinator>,
claim: &Claim, claim: &Claim,
ctx: &Ctx<'_>,
relock: bool, relock: bool,
) -> Result<NodeOutput> { ) -> Result<NodeOutput> {
let name = &claim.agent; let name = &claim.agent;
// Prebuild runs while the agent is still up — the runtime dir and // Runs while the agent is still up — the runtime dir and MCP listener
// MCP listener already exist. Use the pure path accessor; no need // already exist. Use the pure path accessor; no need to re-register the
// to re-register the listener (event-driven: registered at start/create). // listener (event-driven: registered at start/create).
let agent_dir = crate::paths::agent_runtime_dir(name); let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env(); let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir); let paths = Coordinator::agent_paths(name, agent_dir);
@ -211,18 +212,24 @@ async fn run_prebuild(
// Idempotent meta sync so a manual rebuild can also recover from a // Idempotent meta sync so a manual rebuild can also recover from a
// divergent meta repo; then bump just this agent's input. `relock = // divergent meta repo; then bump just this agent's input. `relock =
// false` only for meta-update cascade children, where re-locking // false` only for meta-update cascade children, where re-locking
// would revert the bump the cascade just committed. Both run under // would revert the bump the cascade just committed.
// the deploy-window gate so they can never land inside another let agents = crate::lifecycle::agents_for_meta_listing().await?;
// node's staged prepare→finalize window; the gate drops before the crate::meta::sync_agents(&hive, &agents).await?;
// (long) toplevel build, which only reads the store. if relock {
{ crate::meta::lock_update_for_rebuild(name).await?;
let _window = crate::meta::exclusive().await;
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?;
if relock {
crate::meta::lock_update_for_rebuild(name).await?;
}
} }
Ok(NodeOutput::default())
}
/// Out-of-band toplevel build while the container keeps serving: warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap, against a meta repo the upstream
/// `MetaSync` node has already synced. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead.
async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
let name = &claim.agent;
// Warm the toplevel build only when the container is up — the whole // Warm the toplevel build only when the container is up — the whole
// point of prebuild is to shrink the swap's downtime window. A // point of prebuild is to shrink the swap's downtime window. A
// stopped agent has no uptime to preserve, so skip the (expensive) // stopped agent has no uptime to preserve, so skip the (expensive)
@ -303,10 +310,9 @@ async fn run_post_swap(
} }
/// First-spawn pre-create provisioning: proposed/applied repos, state /// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Holds the /// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy-window gate for the commit so it can't land inside another /// deploy window (`NodeKind::needs_meta_window`) so its commit can't
/// node's staged deploy window — same discipline as `Prebuild`, which /// land inside another node's staged deploy window.
/// commits under the gate then drops it before the (store-only) build.
async fn run_provision( async fn run_provision(
coord: &Arc<Coordinator>, coord: &Arc<Coordinator>,
claim: &Claim, claim: &Claim,
@ -317,7 +323,6 @@ async fn run_provision(
let hive = coord.hive_env(); let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir); let paths = Coordinator::agent_paths(name, agent_dir);
ctx.step("provisioning"); ctx.step("provisioning");
let _window = crate::meta::exclusive().await;
crate::lifecycle::provision_container(name, &hive, &paths).await?; crate::lifecycle::provision_container(name, &hive, &paths).await?;
Ok(NodeOutput::default()) Ok(NodeOutput::default())
} }
@ -347,7 +352,6 @@ async fn run_meta_lock(
) -> Result<NodeOutput> { ) -> Result<NodeOutput> {
if sweep { if sweep {
ctx.step("nix flake update hyperhive"); ctx.step("nix flake update hyperhive");
let _window = crate::meta::exclusive().await;
if let Err(e) = crate::meta::lock_update_hyperhive().await { if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed"); tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
} }
@ -364,10 +368,7 @@ async fn run_meta_lock(
} }
let _progress = coord.meta_update_guard(); let _progress = coord.meta_update_guard();
ctx.step("nix flake update"); ctx.step("nix flake update");
{ crate::meta::lock_update(&claim.inputs).await?;
let _window = crate::meta::exclusive().await;
crate::meta::lock_update(&claim.inputs).await?;
}
// Lock file changed — meta-inputs panel re-renders. // Lock file changed — meta-inputs panel re-renders.
crate::dashboard::emit_meta_inputs_snapshot(coord); crate::dashboard::emit_meta_inputs_snapshot(coord);
let cascade = match fanout { let cascade = match fanout {
@ -545,11 +546,10 @@ async fn run_write_perm_file(
anyhow::bail!("run_write_perm_file on a non-WritePermFile node"); anyhow::bail!("run_write_perm_file on a non-WritePermFile node");
}; };
ctx.step("writing + committing perm file"); ctx.step("writing + committing perm file");
// Deploy-window gate: a perm commit landing inside another node's // Runs under the deploy window (`NodeKind::needs_meta_window`): a
// staged prepare→finalize window would sweep the staged deploy // perm commit landing inside another node's staged prepare→finalize
// lock into its commit (the commits are also path-limited in // window would sweep the staged deploy lock into its commit (the
// meta.rs — belt and braces). // commits are also path-limited in meta.rs — belt and braces).
let _window = crate::meta::exclusive().await;
match payload { match payload {
PermPayload::ToolGroups { groups } => { PermPayload::ToolGroups { groups } => {
crate::meta::commit_tool_groups(name, groups) crate::meta::commit_tool_groups(name, groups)
@ -586,11 +586,14 @@ async fn run_approval_deploy(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
let approval_id = claim let approval_id = claim
.approval_id .approval_id
.with_context(|| format!("approval_deploy dag {} has no approval_id", claim.dag_id))?; .with_context(|| format!("approval_deploy dag {} has no approval_id", claim.dag_id))?;
// Hold the deploy-window gate for the whole prepare→finalize span: // The deploy window covers the whole prepare→finalize span
// `prepare_deploy` stages `flake.lock` uncommitted for the entire // (`NodeKind::needs_meta_window`, held by the scheduler for this
// container build, and no other meta mutation may land inside that // node): `prepare_deploy` stages `flake.lock` uncommitted for the
// window (it would sweep the staged lock and neuter `abort_deploy`). // entire container build, and no other meta mutation may land inside
let _window = crate::meta::exclusive().await; // that window (it would sweep the staged lock and neuter
// `abort_deploy`). Holding it as a queue resource — rather than a
// `MutexGuard` that cannot outlive this fn — is what lets increment 2
// decompose this node into sub-nodes under a window-holding parent.
crate::actions::run_approval_merge_config_pr(coord, Some(claim.dag_id), approval_id) crate::actions::run_approval_merge_config_pr(coord, Some(claim.dag_id), approval_id)
.await .await
.map(|()| NodeOutput::default()) .map(|()| NodeOutput::default())

66
hive-c0re/src/job_queue/model.rs
View file

@ -99,15 +99,27 @@ pub struct Dep {
#[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(tag = "kind", rename_all = "snake_case")] #[serde(tag = "kind", rename_all = "snake_case")]
pub enum NodeKind { pub enum NodeKind {
/// The rebuild's meta-repo preamble: `lifecycle::prepare_rebuild_dirs`,
/// an idempotent meta `sync_agents`, and an optional per-agent relock.
/// `relock = false` only for meta-update cascade rebuilds (re-locking
/// would revert the bump the cascade just committed).
///
/// Its own node — ahead of, and *not* an ancestor of, [`NodeKind::Prebuild`]
/// — precisely because it is the only part of the rebuild that mutates the
/// meta repo and so holds the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow). Fusing
/// it into `Prebuild` (or making it `Prebuild`'s parent, which holds a
/// resource across the whole subtree) would extend that global window over
/// the multi-minute toplevel build and serialize rebuilds hive-wide.
/// Store/meta work only — build-slot- and lease-exempt.
MetaSync { agent: String, relock: bool },
/// Out-of-band toplevel build while the container keeps serving: /// Out-of-band toplevel build while the container keeps serving:
/// meta `sync_agents`, optional per-agent relock, then /// `lifecycle::prebuild_toplevel`, reading a meta repo the upstream
/// `lifecycle::prebuild_toplevel`. `relock = false` only for /// [`NodeKind::MetaSync`] has already synced. The warm build is skipped
/// meta-update cascade rebuilds (re-locking would revert the bump /// when the container is already down — it only exists to shrink the
/// the cascade just committed). The `prebuild_toplevel` warm is /// swap's downtime, which a stopped agent doesn't need (`Swap` builds
/// skipped when the container is already down — it only exists to /// inline instead).
/// shrink the swap's downtime, which a stopped agent doesn't need Prebuild { agent: String },
/// (the sync + dir prep still run; `Swap` builds inline).
Prebuild { agent: String, relock: bool },
/// `nixos-container update` profile-swap (requires the container /// `nixos-container update` profile-swap (requires the container
/// stopped). Re-applies nspawn flags + resource limits first — /// stopped). Re-applies nspawn flags + resource limits first —
/// rebuild is the reconcile verb. The post-rebuild bookkeeping tail /// rebuild is the reconcile verb. The post-rebuild bookkeeping tail
@ -216,6 +228,7 @@ impl NodeKind {
/// Wire string for `NodeView.kind`. /// Wire string for `NodeView.kind`.
pub fn as_str(&self) -> &'static str { pub fn as_str(&self) -> &'static str {
match self { match self {
NodeKind::MetaSync { .. } => "meta_sync",
NodeKind::Prebuild { .. } => "prebuild", NodeKind::Prebuild { .. } => "prebuild",
NodeKind::Swap { .. } => "swap", NodeKind::Swap { .. } => "swap",
NodeKind::PostSwap { .. } => "post_swap", NodeKind::PostSwap { .. } => "post_swap",
@ -242,7 +255,8 @@ impl NodeKind {
#[must_use] #[must_use]
pub fn agent(&self) -> &str { pub fn agent(&self) -> &str {
match self { match self {
NodeKind::Prebuild { agent, .. } NodeKind::MetaSync { agent, .. }
| NodeKind::Prebuild { agent }
| NodeKind::Swap { agent } | NodeKind::Swap { agent }
| NodeKind::PostSwap { agent } | NodeKind::PostSwap { agent }
| NodeKind::Provision { agent } | NodeKind::Provision { agent }
@ -276,8 +290,8 @@ impl NodeKind {
/// Container-affecting kinds require the DAG to hold the agent's /// Container-affecting kinds require the DAG to hold the agent's
/// lifecycle lease (acquired at the first such node, held until the /// lifecycle lease (acquired at the first such node, held until the
/// DAG is terminal). Lease-exempt kinds (`Prebuild`, `Provision`, /// DAG is terminal). Lease-exempt kinds (`MetaSync`, `Prebuild`,
/// `MetaLock`, `WritePermFile`) touch the store / meta repo, not the /// `Provision`, `MetaLock`, `WritePermFile`) touch the store / meta repo, not the
/// running container — which is exactly why a `Prebuild` can overlap /// running container — which is exactly why a `Prebuild` can overlap
/// another DAG's work on the same agent. `Provision` precedes the /// another DAG's work on the same agent. `Provision` precedes the
/// container's existence entirely, so the lease is first taken at the /// container's existence entirely, so the lease is first taken at the
@ -296,6 +310,36 @@ impl NodeKind {
| NodeKind::SetWanted { .. } | NodeKind::SetWanted { .. }
) )
} }
/// Kinds that **mutate the meta repo** and so must hold the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow) for
/// their duration: no two meta mutations may interleave, because a commit
/// landing inside another node's staged `prepare_deploy`→`finalize_deploy`
/// window would sweep the staged `flake.lock` into its own commit and
/// neuter `abort_deploy`.
///
/// This is the queue-primitive replacement for the former runtime
/// `meta::exclusive()` mutex — same global serialisation, but held by the
/// scheduler and therefore able to span a whole subtree, which a
/// `MutexGuard` cannot.
///
/// Note what is **not** here: [`NodeKind::Prebuild`]. The window must stay
/// off the multi-minute toplevel build, which only *reads* the store — the
/// old mutex was scoped to drop before it, and holding a cap-1 global
/// across it would serialize every agent's rebuild behind every other's.
/// That is why the meta preamble is its own [`NodeKind::MetaSync`] node,
/// and why that node is a sibling rather than `Prebuild`'s parent (a
/// resource held by a parent covers its whole subtree).
pub fn needs_meta_window(&self) -> bool {
matches!(
self,
NodeKind::MetaSync { .. }
| NodeKind::Provision { .. }
| NodeKind::MetaLock { .. }
| NodeKind::WritePermFile { .. }
| NodeKind::ApprovalDeploy { .. }
)
}
} }
/// Submit-time spec for one node. /// Submit-time spec for one node.

26
hive-c0re/src/job_queue/resource.rs
View file

@ -22,6 +22,17 @@ pub enum Resource {
/// the rest of that agent's subtree via the crate's recursive lock, so two /// the rest of that agent's subtree via the crate's recursive lock, so two
/// DAGs never interleave container ops on one agent. /// DAGs never interleave container ops on one agent.
Agent(String), Agent(String),
/// The meta-repo mutation window — a global singleton (default capacity 1)
/// held by any node that mutates the meta repo, so two meta mutations never
/// interleave. Replaces the former runtime `meta::exclusive()` mutex: a
/// `MutexGuard` cannot span scheduler nodes, but a resource held by a
/// subtree root *can* — which is what lets the two-phase deploy
/// (`prepare_deploy` stages `flake.lock` uncommitted across the whole
/// container build, `finalize_deploy`/`abort_deploy` resolve it) be
/// decomposed into sub-nodes instead of one opaque node. Descendants of a
/// holder re-enter it through the crate's recursive lock, exactly like
/// [`Resource::Agent`].
MetaWindow,
} }
impl NodeKind { impl NodeKind {
@ -31,7 +42,14 @@ impl NodeKind {
/// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt /// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt
/// container ops (`Start` / `Stop`, fanned out by a lease-holding /// container ops (`Start` / `Stop`, fanned out by a lease-holding
/// `Reconcile`) hold no lease of their own — they re-enter the ancestor's /// `Reconcile`) hold no lease of their own — they re-enter the ancestor's
/// `Agent` lock through the crate's recursive re-entrancy. /// `Agent` lock through the crate's recursive re-entrancy. Meta-mutating
/// kinds ([`NodeKind::needs_meta_window`]) additionally take the global
/// [`Resource::MetaWindow`].
///
/// All of a node's resource edges are acquired **atomically**
/// (`try_acquire_all`) — a node never holds one resource while waiting on
/// another, so the multi-resource kinds (a `MetaLock` wants a build slot
/// *and* the meta window) cannot deadlock against each other.
pub fn resource_deps(&self) -> Vec<Dep<Resource>> { pub fn resource_deps(&self) -> Vec<Dep<Resource>> {
let mut deps = Vec::new(); let mut deps = Vec::new();
if self.needs_build_slot() { if self.needs_build_slot() {
@ -46,6 +64,12 @@ impl NodeKind {
count: 1, count: 1,
}); });
} }
if self.needs_meta_window() {
deps.push(Dep::Resource {
name: Resource::MetaWindow,
count: 1,
});
}
deps deps
} }
} }

5
hive-c0re/src/job_queue/submit.rs
View file

@ -95,8 +95,9 @@ fn start_chain(agent: &str, running: bool, stale: bool) -> Vec<NodeSpec> {
)]; )];
if !running && stale { if !running && stale {
// Rebuild subtree after the SetWanted head (base = 1, so the rebuild's // Rebuild subtree after the SetWanted head (base = 1, so the rebuild's
// `Prebuild` root deps `after_ok(0)` = the head). `Prebuild` + // `MetaSync` root deps `after_ok(0)` = the head). `MetaSync`,
// `Reconcile` are their own group roots (top-level, per `rebuild_nodes`). // `Prebuild` + `Reconcile` are their own group roots (top-level, per
// `rebuild_nodes`).
n.extend(rebuild_nodes(agent, true, 1)); n.extend(rebuild_nodes(agent, true, 1));
} else { } else {
n.push(child( n.push(child(

44
hive-c0re/src/job_queue/templates.rs
View file

@ -18,7 +18,7 @@
//! agent's existing `wanted`. //! agent's existing `wanted`.
//! //!
//! ```text //! ```text
//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a) //! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve] //! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
//! perm-change(a): WritePermFile(a) → «rebuild subgraph» //! perm-change(a): WritePermFile(a) → «rebuild subgraph»
//! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a» //! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a»
@ -68,45 +68,55 @@ pub(crate) fn child(parent: u64, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
} }
} }
/// The rebuild node subtree (nested, two group roots). `base` is the spec index /// The rebuild node subtree (nested, three group roots). `base` is the spec
/// of the first node (`Prebuild`). Structure: /// index of the first node (`MetaSync`). Structure:
/// - `Prebuild` (base+0, **root**): owns the build slot for the whole subtree. /// - `MetaSync` (base+0, **root**): the meta-repo preamble (dir prep, agent
/// Lease-exempt — the nix build overlaps other DAGs on the same agent. /// sync, optional relock). Owns the global `MetaWindow` — and *only* for its
/// - `StopForUpdate` (base+1, child of `Prebuild`): owns the agent lease. Runs /// own short duration, which is why it is a sibling root rather than
/// `Prebuild`'s parent: a resource is held across the holder's whole subtree,
/// so parenting the build under it would extend a hive-global window over
/// every rebuild's nix build.
/// - `Prebuild` (base+1, **root**): `AfterOk` `MetaSync`. Owns the build slot
/// for the whole mechanical subtree below it. Lease-exempt — the nix build
/// overlaps other DAGs on the same agent.
/// - `StopForUpdate` (base+2, child of `Prebuild`): owns the agent lease. Runs
/// once `Prebuild` reaches `Finishing` (parent gate). /// once `Prebuild` reaches `Finishing` (parent gate).
/// - `Swap` (base+2, child of `StopForUpdate`): borrows the agent lease from its /// - `Swap` (base+3, child of `StopForUpdate`): borrows the agent lease from its
/// parent and the build slot from grand-ancestor `Prebuild` — both continuous. /// parent and the build slot from grand-ancestor `Prebuild` — both continuous.
/// - `PostSwap` (base+3, child of `StopForUpdate`): the swap's Ok-only /// - `PostSwap` (base+4, child of `StopForUpdate`): the swap's Ok-only
/// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk` /// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk`
/// its sibling `Swap`. /// its sibling `Swap`.
/// - `Reconcile` (base+4, **root**): `AfterAny` `Prebuild`, which rolls up /// - `Reconcile` (base+5, **root**): `AfterAny` `Prebuild`, which rolls up
/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has /// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has
/// settled — so `Reconcile` runs after the swap regardless of outcome, and as /// settled — so `Reconcile` runs after the swap regardless of outcome, and as
/// a top-level root it survives the cancel-cascade of a failed `Prebuild` /// a top-level root it survives the cancel-cascade of a failed `Prebuild`
/// (recovery-start invariant). It takes a fresh lease; the tiny gap is /// (recovery-start invariant, which also covers a failed `MetaSync`: that
/// harmless — `Reconcile` converges to the persisted `wanted` idempotently. /// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It
/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to
/// the persisted `wanted` idempotently.
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec<NodeSpec> { pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec<NodeSpec> {
let a = || agent.to_owned(); let a = || agent.to_owned();
vec![ vec![
node( node(
NodeKind::Prebuild { agent: a(), relock }, NodeKind::MetaSync { agent: a(), relock },
if base == 0 { if base == 0 {
Vec::new() Vec::new()
} else { } else {
after_ok(base - 1) after_ok(base - 1)
}, },
), ),
child(base, NodeKind::StopForUpdate { agent: a() }, Vec::new()), node(NodeKind::Prebuild { agent: a() }, after_ok(base)),
child(base + 1, NodeKind::Swap { agent: a() }, Vec::new()), child(base + 1, NodeKind::StopForUpdate { agent: a() }, Vec::new()),
child(base + 2, NodeKind::Swap { agent: a() }, Vec::new()),
child( child(
base + 1, base + 2,
NodeKind::PostSwap { agent: a() }, NodeKind::PostSwap { agent: a() },
after_ok(base + 2), after_ok(base + 3),
), ),
node( node(
NodeKind::Reconcile { agent: a() }, NodeKind::Reconcile { agent: a() },
vec![Dep { vec![Dep {
on: base, on: base + 1,
when: DepWhen::AfterAny, when: DepWhen::AfterAny,
}], }],
), ),

155
hive-c0re/src/job_queue/tests.rs
View file

@ -176,6 +176,7 @@ fn rebuild_chain_claims_in_dep_order() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r")); let id = submit(&q, rebuild("agent-a", "r"));
for expected in [ for expected in [
"meta_sync",
"prebuild", "prebuild",
"stop_for_update", "stop_for_update",
"swap", "swap",
@ -201,15 +202,29 @@ fn build_slot_serializes_nix_heavy_nodes() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
let a = submit(&q, rebuild("agent-a", "r")); let a = submit(&q, rebuild("agent-a", "r"));
let b = submit(&q, rebuild("agent-b", "r")); let b = submit(&q, rebuild("agent-b", "r"));
let first = claim_one(&q); // a's Prebuild takes the only slot // The rebuild heads are `MetaSync` (slot-free, but serialized on the
assert_eq!(first.dag_id, a); // global meta window), so drive each chain's head out of the way first.
assert_eq!(first.kind.as_str(), "prebuild"); let head_a = claim_one(&q);
q.complete_node(a, first.node_id, Ok(())); assert_eq!(head_a.dag_id, a);
assert_eq!(head_a.kind.as_str(), "meta_sync");
q.complete_node(a, head_a.node_id, Ok(()));
// a's Prebuild takes the only slot; b's MetaSync is free to run beside it
// (different resources), but b's Prebuild is not.
let claims = q.claim_ready();
let mut kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect();
kinds.sort_unstable();
assert_eq!(kinds, vec![(a, "prebuild"), (b, "meta_sync")]);
for c in &claims {
q.complete_node(c.dag_id, c.node_id, Ok(()));
}
// Uniform hold: agent-a keeps the build slot across its whole build chain // Uniform hold: agent-a keeps the build slot across its whole build chain
// (Swap re-enters it), so a's StopForUpdate (lease, slot-free) runs but b's // (Swap re-enters it), so a's StopForUpdate (lease, slot-free) runs but b's
// Prebuild must wait for a's slot-needers (through Swap) to finish. // Prebuild must wait for a's slot-needers (through Swap) to finish.
let claims = q.claim_ready(); let kinds: Vec<(u64, &str)> = q
let kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect(); .claim_ready()
.iter()
.map(|c| (c.dag_id, c.kind.as_str()))
.collect();
assert_eq!(kinds, vec![(a, "stop_for_update")]); assert_eq!(kinds, vec![(a, "stop_for_update")]);
assert!( assert!(
!kinds.iter().any(|&(d, _)| d == b), !kinds.iter().any(|&(d, _)| d == b),
@ -222,9 +237,23 @@ fn two_build_slots_run_two_prebuilds() {
let q = JobQueue::new(2); let q = JobQueue::new(2);
submit(&q, rebuild("agent-a", "r")); submit(&q, rebuild("agent-a", "r"));
submit(&q, rebuild("agent-b", "r")); submit(&q, rebuild("agent-b", "r"));
let claims = q.claim_ready(); // Each rebuild's head `MetaSync` holds the cap-1 global meta window, so the
assert_eq!(claims.len(), 2, "two slots → two concurrent prebuilds"); // two heads take turns — exactly the serialization the old runtime
assert!(claims.iter().all(|c| c.kind.as_str() == "prebuild")); // `meta::exclusive()` mutex imposed inside the prebuild executor. What must
// NOT serialize is the build itself: complete only the meta heads and watch
// both prebuilds end up in flight together, neither of them completed.
let mut prebuilds = Vec::new();
for _ in 0..3 {
for c in q.claim_ready() {
if c.kind.as_str() == "meta_sync" {
q.complete_node(c.dag_id, c.node_id, Ok(()));
} else {
prebuilds.push(c);
}
}
}
assert_eq!(prebuilds.len(), 2, "two slots → two concurrent prebuilds");
assert!(prebuilds.iter().all(|c| c.kind.as_str() == "prebuild"));
} }
#[test] #[test]
@ -312,12 +341,23 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
None, None,
), ),
); );
// Prebuild is lease-exempt: the stop's Reconcile takes the lease // Both DAGs' heads are lease-independent of each other: the rebuild's
// MetaSync (meta window) and the stop's Reconcile (agent lease).
let heads = q.claim_ready();
let head_kinds: Vec<&str> = heads.iter().map(|c| c.kind.as_str()).collect();
assert!(head_kinds.contains(&"meta_sync"));
assert!(head_kinds.contains(&"reconcile"));
let meta_sync = heads
.iter()
.find(|c| c.kind.as_str() == "meta_sync")
.expect("meta_sync claim")
.clone();
q.complete_node(meta_sync.dag_id, meta_sync.node_id, Ok(()));
// Prebuild is lease-exempt: the stop's Reconcile keeps the lease
// and runs concurrently with the rebuild's out-of-band nix build. // and runs concurrently with the rebuild's out-of-band nix build.
let claims = q.claim_ready(); let claims = q.claim_ready();
let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect(); let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect();
assert!(kinds.contains(&"prebuild")); assert!(kinds.contains(&"prebuild"));
assert!(kinds.contains(&"reconcile"));
// But the rebuild's StopForUpdate must then wait for the stop DAG // But the rebuild's StopForUpdate must then wait for the stop DAG
// to finish (lease). // to finish (lease).
let prebuild = claims let prebuild = claims
@ -330,7 +370,7 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
q.claim_ready().is_empty(), q.claim_ready().is_empty(),
"StopForUpdate blocked while stop DAG holds the lease" "StopForUpdate blocked while stop DAG holds the lease"
); );
let reconcile = claims let reconcile = heads
.iter() .iter()
.find(|c| c.kind.as_str() == "reconcile") .find(|c| c.kind.as_str() == "reconcile")
.expect("reconcile claim") .expect("reconcile claim")
@ -484,7 +524,7 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() {
head_agents.sort_unstable(); head_agents.sort_unstable();
assert_eq!(head_agents, vec!["fresh", "stale"]); assert_eq!(head_agents, vec!["fresh", "stale"]);
// Complete both heads; the fresh agent then reconciles directly while // Complete both heads; the fresh agent then reconciles directly while
// the stale agent's subgraph is the rebuild chain (prebuild first). // the stale agent's subgraph is the rebuild chain (meta_sync first).
for c in &heads { for c in &heads {
q.complete_node(id, c.node_id, Ok(())); q.complete_node(id, c.node_id, Ok(()));
} }
@ -496,7 +536,7 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() {
kinds.sort_unstable(); kinds.sort_unstable();
assert_eq!( assert_eq!(
kinds, kinds,
vec![("fresh", "reconcile"), ("stale", "prebuild")], vec![("fresh", "reconcile"), ("stale", "meta_sync")],
"fresh agent starts directly; stale agent rebuilds first, all in one DAG" "fresh agent starts directly; stale agent rebuilds first, all in one DAG"
); );
} }
@ -578,30 +618,48 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
let emitter = claim_one(&q); let emitter = claim_one(&q);
assert_eq!(emitter.kind.as_str(), "meta_lock"); assert_eq!(emitter.kind.as_str(), "meta_lock");
// Two independent per-agent subgraphs — the REAL production shape the // Two independent per-agent subgraphs — the REAL production shape the
// sweep MetaLock grows (`rebuild_nodes(_, true, 0)`: root Prebuild → // sweep MetaLock grows (`rebuild_nodes(_, true, 0)`: root MetaSync → root
// StopForUpdate → Swap → Reconcile, local 0-based deps), so this test // Prebuild → StopForUpdate → Swap → Reconcile, local 0-based deps), so this
// tracks any drift in that builder's root-first (`base = 0`) shape. // test tracks any drift in that builder's root-first (`base = 0`) shape.
let subgraph = |agent: &str| templates::rebuild_nodes(agent, true, 0); let subgraph = |agent: &str| templates::rebuild_nodes(agent, true, 0);
// Must append BEFORE completing the emitter (the documented contract). // Must append BEFORE completing the emitter (the documented contract).
q.append_subgraph(id, &subgraph("a"), emitter.node_id); q.append_subgraph(id, &subgraph("a"), emitter.node_id);
q.append_subgraph(id, &subgraph("b"), emitter.node_id); q.append_subgraph(id, &subgraph("b"), emitter.node_id);
q.complete_node(id, emitter.node_id, Ok(())); q.complete_node(id, emitter.node_id, Ok(()));
// Still ONE DAG; both subgraph roots become ready once the emitter is // Still ONE DAG; both subgraph roots become ready once the emitter is
// Done (rooted on it), each on its own agent lease. // Done (rooted on it), each on its own agent lease. Their `MetaSync` heads
// take turns on the cap-1 global meta window, so drain those first — what
// must be concurrent is the builds.
assert_eq!(q.snapshot().len(), 1); assert_eq!(q.snapshot().len(), 1);
let next = q.claim_ready(); let mut kinds = drain_meta_syncs(&q, id);
let mut kinds: Vec<(&str, &str)> = next
.iter()
.map(|c| (c.agent.as_str(), c.kind.as_str()))
.collect();
kinds.sort_unstable(); kinds.sort_unstable();
assert_eq!( assert_eq!(
kinds, kinds,
vec![("a", "prebuild"), ("b", "prebuild")], vec![
("a".to_owned(), "prebuild".to_owned()),
("b".to_owned(), "prebuild".to_owned())
],
"both rebuild subgraphs root on the emitter and run concurrently in one DAG" "both rebuild subgraphs root on the emitter and run concurrently in one DAG"
); );
} }
/// Complete every `MetaSync` head the queue offers (they take turns on the
/// cap-1 global meta window) and return whatever else got claimed alongside
/// them, as `(agent, kind)` pairs left in flight.
fn drain_meta_syncs(q: &JobQueue, dag: u64) -> Vec<(String, String)> {
let mut rest = Vec::new();
for _ in 0..3 {
for c in q.claim_ready() {
if c.kind.as_str() == "meta_sync" {
q.complete_node(dag, c.node_id, Ok(()));
} else {
rest.push((c.agent.clone(), c.kind.as_str().to_owned()));
}
}
}
rest
}
#[test] #[test]
fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() { fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
// The meta-update `MetaLock` grows one rebuild subgraph per affected // The meta-update `MetaLock` grows one rebuild subgraph per affected
@ -637,17 +695,18 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
} }
q.complete_node(id, meta_lock.node_id, Ok(())); q.complete_node(id, meta_lock.node_id, Ok(()));
// Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root // Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root
// on the MetaLock, each on its own agent lease. // on the MetaLock, each on its own agent lease. The per-agent `MetaSync`
// heads serialize on the global meta window (they commit to the meta repo);
// the builds behind them do not.
assert_eq!(q.snapshot().len(), 1); assert_eq!(q.snapshot().len(), 1);
let next = q.claim_ready(); let mut kinds = drain_meta_syncs(&q, id);
let mut kinds: Vec<(&str, &str)> = next
.iter()
.map(|c| (c.agent.as_str(), c.kind.as_str()))
.collect();
kinds.sort_unstable(); kinds.sort_unstable();
assert_eq!( assert_eq!(
kinds, kinds,
vec![("alice", "prebuild"), ("bob", "prebuild")], vec![
("alice".to_owned(), "prebuild".to_owned()),
("bob".to_owned(), "prebuild".to_owned())
],
"cascade rebuilds grow in the meta-update DAG, concurrent per agent" "cascade rebuilds grow in the meta-update DAG, concurrent per agent"
); );
} }
@ -658,7 +717,11 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
fn failed_node_cancels_downstream_but_afterany_reconcile_runs() { fn failed_node_cancels_downstream_but_afterany_reconcile_runs() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r")); let id = submit(&q, rebuild("agent-a", "r"));
let meta_sync = claim_one(&q);
assert_eq!(meta_sync.kind.as_str(), "meta_sync");
q.complete_node(id, meta_sync.node_id, Ok(()));
let prebuild = claim_one(&q); let prebuild = claim_one(&q);
assert_eq!(prebuild.kind.as_str(), "prebuild");
q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned())); q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned()));
// StopForUpdate + Swap are cancelled (AfterOk on a failed chain); // StopForUpdate + Swap are cancelled (AfterOk on a failed chain);
// the AfterAny Reconcile still runs once Swap is terminal. // the AfterAny Reconcile still runs once Swap is terminal.
@ -702,7 +765,8 @@ fn failed_node_cancels_downstream_but_afterany_reconcile_runs() {
fn swap_failure_still_runs_reconcile() { fn swap_failure_still_runs_reconcile() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r")); let id = submit(&q, rebuild("agent-a", "r"));
for _ in 0..2 { // meta_sync + prebuild + stop_for_update
for _ in 0..3 {
let c = claim_one(&q); let c = claim_one(&q);
q.complete_node(id, c.node_id, Ok(())); q.complete_node(id, c.node_id, Ok(()));
} }
@ -735,8 +799,8 @@ fn swap_failure_still_runs_reconcile() {
fn swap_ok_runs_post_swap_before_reconcile() { fn swap_ok_runs_post_swap_before_reconcile() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r")); let id = submit(&q, rebuild("agent-a", "r"));
// prebuild + stop_for_update // meta_sync + prebuild + stop_for_update
for _ in 0..2 { for _ in 0..3 {
let c = claim_one(&q); let c = claim_one(&q);
q.complete_node(id, c.node_id, Ok(())); q.complete_node(id, c.node_id, Ok(()));
} }
@ -979,13 +1043,25 @@ fn graceful_signal_and_drain_hold_no_build_slot() {
submit(&q, rebuild("builder", "slot hog")); submit(&q, rebuild("builder", "slot hog"));
submit(&q, stop_online(&["agent-a"], true, "g")); submit(&q, stop_online(&["agent-a"], true, "g"));
submit(&q, stop_online(&["agent-b"], true, "g")); submit(&q, stop_online(&["agent-b"], true, "g"));
let claims = q.claim_ready(); // All three DAG heads are build-slot-exempt, so they run at once.
let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect(); let heads = q.claim_ready();
let kinds: Vec<&str> = heads.iter().map(|c| c.kind.as_str()).collect();
assert_eq!(kinds, vec!["meta_sync", "set_wanted", "set_wanted"]);
for c in &heads {
q.complete_node(c.dag_id, c.node_id, Ok(()));
}
// Now the rebuild's Prebuild holds the single slot — and both graceful
// stops still proceed to their Signal beside it.
let kinds: Vec<&str> = q
.claim_ready()
.iter()
.map(|c| c.kind.as_str())
.collect::<Vec<_>>();
assert_eq!( assert_eq!(
kinds, kinds,
vec!["prebuild", "set_wanted", "set_wanted"], vec!["prebuild", "signal", "signal"],
"both agents' graceful-stop heads (SetWanted, build-slot-exempt) run \ "both agents' graceful-stop signals (build-slot-exempt) run while the \
while the slot is held; their signals follow" rebuild holds the slot"
); );
} }
@ -1023,6 +1099,7 @@ fn perm_change_shape_prefixes_rebuild_chain() {
); );
for expected in [ for expected in [
"write_perm_file", "write_perm_file",
"meta_sync",
"prebuild", "prebuild",
"stop_for_update", "stop_for_update",
"swap", "swap",

29
hive-c0re/src/meta.rs
View file

@ -27,24 +27,17 @@ const GIT_EMAIL: &str = "c0re@hyperhive.local";
/// take turns instead of colliding. /// take turns instead of colliding.
static META_LOCK: Mutex<()> = Mutex::const_new(()); static META_LOCK: Mutex<()> = Mutex::const_new(());
/// Coarse exclusivity for meta-repo *windows* that span multiple // Exclusivity for meta-repo *windows* that span multiple `META_LOCK`
/// `META_LOCK` acquisitions — above all the two-phase deploy // acquisitions — above all the two-phase deploy (`prepare_deploy` stages
/// (`prepare_deploy` stages `flake.lock` uncommitted for the whole // `flake.lock` uncommitted for the whole container build;
/// container build; `finalize_deploy` / `abort_deploy` resolve it). // `finalize_deploy` / `abort_deploy` resolve it) — is **not** a mutex in
/// `META_LOCK` serializes individual git ops but cannot keep another // this module. `META_LOCK` above serializes individual git ops but cannot
/// op out of that staged window: a perm-file or lock-bump commit // keep another op out of that staged window; that window is owned by the
/// landing mid-window would sweep the staged deploy lock into its own // job queue instead, as `Resource::MetaWindow`, declared by every
/// commit and neuter `abort_deploy`. Job-queue executors that mutate // meta-mutating node kind (`NodeKind::needs_meta_window`). A resource can
/// the meta repo hold this gate for their mutation span; the opaque // be held by a subtree root across its children, which a `MutexGuard`
/// approval-deploy node holds it across its whole prepare→finalize // (bounded by one executor fn) cannot — that's what lets the deploy be
/// span. Never acquired inside this module's functions (they run // modelled as sub-nodes rather than one opaque node.
/// *under* a caller's window — nesting would deadlock).
static DEPLOY_GATE: Mutex<()> = Mutex::const_new(());
/// Acquire the deploy/meta-mutation window gate. See [`DEPLOY_GATE`].
pub async fn exclusive() -> tokio::sync::MutexGuard<'static, ()> {
DEPLOY_GATE.lock().await
}
/// Where the manager sees this directory inside its container (RO bind). /// Where the manager sees this directory inside its container (RO bind).
pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta"; pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta";