vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(job-queue): promote the meta-repo deploy window to a queue resource

Browse source 
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 • committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

41
docs/coordinator.md
View file

@ -33,7 +33,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node |
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
@ -43,6 +43,7 @@ Cheap — no build slot:
| Node | Behavior |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `PostSwap` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event still fires once per DAG from the terminal hook, not here) |
@ -60,13 +61,19 @@ Two further layers protect the meta repo across *windows* that span multiple
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
container build:
- **The deploy-window gate** (`meta::exclusive()`): every executor that
mutates the meta repo (`Prebuild`'s sync+relock, `MetaLock`,
`WritePermFile`, `Create`'s agent registration, and `ApprovalDeploy` for
its whole span) holds this async mutex for its mutation span, so no commit
can land inside another node's staged window. `Prebuild` drops it before
the long toplevel build (store reads only), preserving `buildSlots > 1`
concurrency.
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
resource declared by every node kind that mutates the meta repo — `MetaSync`,
`MetaLock`, `WritePermFile`, `Provision`'s agent registration, and
`ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta
mutations can therefore never interleave, so no commit lands inside another
node's staged window. It is a queue resource rather than a runtime mutex
because a resource is held by a subtree root across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
multi-node deploy own one window. For the same reason the window must stay
*off* long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
- **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's
@ -100,7 +107,7 @@ sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
agent's subgraph is a rebuild-then-start).
```text
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
@ -178,8 +185,8 @@ resources are free. Resources:
touching several agents holds one lease per agent. (`SetWanted` is a store
write, not a container op, but takes the lease anyway so a power-op DAG's
intent write + reconcile is atomic — two racing ops can't clobber intent
before either reconciles.) **Lease-exempt**: `Prebuild`, `MetaLock`,
`WritePermFile` —
before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`,
`MetaLock`, `WritePermFile` —
they touch the store / meta, not the running container, which is exactly
why a stop can land while another DAG's prebuild is still building.
@ -311,16 +318,20 @@ one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node):
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
meta-update cascade child) the per-agent relock. Short, and the only step
that mutates the meta repo, so it is the only one holding the global deploy
window.
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface
here, before the running container is touched. (Runs even for a stopped
container — same total nix work, one uniform DAG shape.)
2. `StopForUpdate` — bring the container down (noop when already stopped).
3. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
(near-instant after the prebuild).
4. `Reconcile` — boot into the new generation when `wanted = Up`; the
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
"deferred start" split, now structural.

93
hive-c0re/src/job_queue/exec.rs
View file

@ -82,7 +82,8 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
node_id: claim.node_id,
};
match &claim.kind {
NodeKind::Prebuild { relock, .. } => run_prebuild(coord, claim, &ctx, *relock).await,
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await,
NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, claim, &ctx).await,
NodeKind::Provision { .. } => run_provision(coord, claim, &ctx).await,
@ -186,24 +187,24 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<N
Ok(NodeOutput::default())
}
/// Out-of-band toplevel build while the container keeps serving: meta
/// sync + optional per-agent relock, then warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead. The meta sync +
/// dir prep still run unconditionally (the `Swap` depends on them).
async fn run_prebuild(
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
/// `sync_agents`, and the optional per-agent relock. Runs under the deploy
/// window (`NodeKind::needs_meta_window`, held by the scheduler for this
/// node) so its commits can never land inside another node's staged
/// prepare→finalize window.
///
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that
/// build takes minutes and only *reads* the store, so keeping the global
/// window off it is what lets rebuilds of different agents overlap.
async fn run_meta_sync(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
relock: bool,
) -> Result<NodeOutput> {
let name = &claim.agent;
// Prebuild runs while the agent is still up — the runtime dir and
// MCP listener already exist. Use the pure path accessor; no need
// to re-register the listener (event-driven: registered at start/create).
// Runs while the agent is still up — the runtime dir and MCP listener
// already exist. Use the pure path accessor; no need to re-register the
// listener (event-driven: registered at start/create).
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
@ -211,18 +212,24 @@ async fn run_prebuild(
// Idempotent meta sync so a manual rebuild can also recover from a
// divergent meta repo; then bump just this agent's input. `relock =
// false` only for meta-update cascade children, where re-locking
// would revert the bump the cascade just committed. Both run under
// the deploy-window gate so they can never land inside another
// node's staged prepare→finalize window; the gate drops before the
// (long) toplevel build, which only reads the store.
{
let _window = crate::meta::exclusive().await;
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?;
if relock {
crate::meta::lock_update_for_rebuild(name).await?;
}
// would revert the bump the cascade just committed.
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?;
if relock {
crate::meta::lock_update_for_rebuild(name).await?;
}
Ok(NodeOutput::default())
}
/// Out-of-band toplevel build while the container keeps serving: warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap, against a meta repo the upstream
/// `MetaSync` node has already synced. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead.
async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
let name = &claim.agent;
// Warm the toplevel build only when the container is up — the whole
// point of prebuild is to shrink the swap's downtime window. A
// stopped agent has no uptime to preserve, so skip the (expensive)
@ -303,10 +310,9 @@ async fn run_post_swap(
}
/// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Holds the
/// deploy-window gate for the commit so it can't land inside another
/// node's staged deploy window — same discipline as `Prebuild`, which
/// commits under the gate then drops it before the (store-only) build.
/// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (`NodeKind::needs_meta_window`) so its commit can't
/// land inside another node's staged deploy window.
async fn run_provision(
coord: &Arc<Coordinator>,
claim: &Claim,
@ -317,7 +323,6 @@ async fn run_provision(
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
ctx.step("provisioning");
let _window = crate::meta::exclusive().await;
crate::lifecycle::provision_container(name, &hive, &paths).await?;
Ok(NodeOutput::default())
}
@ -347,7 +352,6 @@ async fn run_meta_lock(
) -> Result<NodeOutput> {
if sweep {
ctx.step("nix flake update hyperhive");
let _window = crate::meta::exclusive().await;
if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
}
@ -364,10 +368,7 @@ async fn run_meta_lock(
}
let _progress = coord.meta_update_guard();
ctx.step("nix flake update");
{
let _window = crate::meta::exclusive().await;
crate::meta::lock_update(&claim.inputs).await?;
}
crate::meta::lock_update(&claim.inputs).await?;
// Lock file changed — meta-inputs panel re-renders.
crate::dashboard::emit_meta_inputs_snapshot(coord);
let cascade = match fanout {
@ -545,11 +546,10 @@ async fn run_write_perm_file(
anyhow::bail!("run_write_perm_file on a non-WritePermFile node");
};
ctx.step("writing + committing perm file");
// Deploy-window gate: a perm commit landing inside another node's
// staged prepare→finalize window would sweep the staged deploy
// lock into its commit (the commits are also path-limited in
// meta.rs — belt and braces).
let _window = crate::meta::exclusive().await;
// Runs under the deploy window (`NodeKind::needs_meta_window`): a
// perm commit landing inside another node's staged prepare→finalize
// window would sweep the staged deploy lock into its commit (the
// commits are also path-limited in meta.rs — belt and braces).
match payload {
PermPayload::ToolGroups { groups } => {
crate::meta::commit_tool_groups(name, groups)
@ -586,11 +586,14 @@ async fn run_approval_deploy(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
let approval_id = claim
.approval_id
.with_context(|| format!("approval_deploy dag {} has no approval_id", claim.dag_id))?;
// Hold the deploy-window gate for the whole prepare→finalize span:
// `prepare_deploy` stages `flake.lock` uncommitted for the entire
// container build, and no other meta mutation may land inside that
// window (it would sweep the staged lock and neuter `abort_deploy`).
let _window = crate::meta::exclusive().await;
// The deploy window covers the whole prepare→finalize span
// (`NodeKind::needs_meta_window`, held by the scheduler for this
// node): `prepare_deploy` stages `flake.lock` uncommitted for the
// entire container build, and no other meta mutation may land inside
// that window (it would sweep the staged lock and neuter
// `abort_deploy`). Holding it as a queue resource — rather than a
// `MutexGuard` that cannot outlive this fn — is what lets increment 2
// decompose this node into sub-nodes under a window-holding parent.
crate::actions::run_approval_merge_config_pr(coord, Some(claim.dag_id), approval_id)
.await
.map(|()| NodeOutput::default())

66
hive-c0re/src/job_queue/model.rs
View file

@ -99,15 +99,27 @@ pub struct Dep {
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum NodeKind {
/// The rebuild's meta-repo preamble: `lifecycle::prepare_rebuild_dirs`,
/// an idempotent meta `sync_agents`, and an optional per-agent relock.
/// `relock = false` only for meta-update cascade rebuilds (re-locking
/// would revert the bump the cascade just committed).
///
/// Its own node — ahead of, and *not* an ancestor of, [`NodeKind::Prebuild`]
/// — precisely because it is the only part of the rebuild that mutates the
/// meta repo and so holds the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow). Fusing
/// it into `Prebuild` (or making it `Prebuild`'s parent, which holds a
/// resource across the whole subtree) would extend that global window over
/// the multi-minute toplevel build and serialize rebuilds hive-wide.
/// Store/meta work only — build-slot- and lease-exempt.
MetaSync { agent: String, relock: bool },
/// Out-of-band toplevel build while the container keeps serving:
/// meta `sync_agents`, optional per-agent relock, then
/// `lifecycle::prebuild_toplevel`. `relock = false` only for
/// meta-update cascade rebuilds (re-locking would revert the bump
/// the cascade just committed). The `prebuild_toplevel` warm is
/// skipped when the container is already down — it only exists to
/// shrink the swap's downtime, which a stopped agent doesn't need
/// (the sync + dir prep still run; `Swap` builds inline).
Prebuild { agent: String, relock: bool },
/// `lifecycle::prebuild_toplevel`, reading a meta repo the upstream
/// [`NodeKind::MetaSync`] has already synced. The warm build is skipped
/// when the container is already down — it only exists to shrink the
/// swap's downtime, which a stopped agent doesn't need (`Swap` builds
/// inline instead).
Prebuild { agent: String },
/// `nixos-container update` profile-swap (requires the container
/// stopped). Re-applies nspawn flags + resource limits first —
/// rebuild is the reconcile verb. The post-rebuild bookkeeping tail
@ -216,6 +228,7 @@ impl NodeKind {
/// Wire string for `NodeView.kind`.
pub fn as_str(&self) -> &'static str {
match self {
NodeKind::MetaSync { .. } => "meta_sync",
NodeKind::Prebuild { .. } => "prebuild",
NodeKind::Swap { .. } => "swap",
NodeKind::PostSwap { .. } => "post_swap",
@ -242,7 +255,8 @@ impl NodeKind {
#[must_use]
pub fn agent(&self) -> &str {
match self {
NodeKind::Prebuild { agent, .. }
NodeKind::MetaSync { agent, .. }
| NodeKind::Prebuild { agent }
| NodeKind::Swap { agent }
| NodeKind::PostSwap { agent }
| NodeKind::Provision { agent }
@ -276,8 +290,8 @@ impl NodeKind {
/// Container-affecting kinds require the DAG to hold the agent's
/// lifecycle lease (acquired at the first such node, held until the
/// DAG is terminal). Lease-exempt kinds (`Prebuild`, `Provision`,
/// `MetaLock`, `WritePermFile`) touch the store / meta repo, not the
/// DAG is terminal). Lease-exempt kinds (`MetaSync`, `Prebuild`,
/// `Provision`, `MetaLock`, `WritePermFile`) touch the store / meta repo, not the
/// running container — which is exactly why a `Prebuild` can overlap
/// another DAG's work on the same agent. `Provision` precedes the
/// container's existence entirely, so the lease is first taken at the
@ -296,6 +310,36 @@ impl NodeKind {
| NodeKind::SetWanted { .. }
)
}
/// Kinds that **mutate the meta repo** and so must hold the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow) for
/// their duration: no two meta mutations may interleave, because a commit
/// landing inside another node's staged `prepare_deploy`→`finalize_deploy`
/// window would sweep the staged `flake.lock` into its own commit and
/// neuter `abort_deploy`.
///
/// This is the queue-primitive replacement for the former runtime
/// `meta::exclusive()` mutex — same global serialisation, but held by the
/// scheduler and therefore able to span a whole subtree, which a
/// `MutexGuard` cannot.
///
/// Note what is **not** here: [`NodeKind::Prebuild`]. The window must stay
/// off the multi-minute toplevel build, which only *reads* the store — the
/// old mutex was scoped to drop before it, and holding a cap-1 global
/// across it would serialize every agent's rebuild behind every other's.
/// That is why the meta preamble is its own [`NodeKind::MetaSync`] node,
/// and why that node is a sibling rather than `Prebuild`'s parent (a
/// resource held by a parent covers its whole subtree).
pub fn needs_meta_window(&self) -> bool {
matches!(
self,
NodeKind::MetaSync { .. }
| NodeKind::Provision { .. }
| NodeKind::MetaLock { .. }
| NodeKind::WritePermFile { .. }
| NodeKind::ApprovalDeploy { .. }
)
}
}
/// Submit-time spec for one node.

26
hive-c0re/src/job_queue/resource.rs
View file

@ -22,6 +22,17 @@ pub enum Resource {
/// the rest of that agent's subtree via the crate's recursive lock, so two
/// DAGs never interleave container ops on one agent.
Agent(String),
/// The meta-repo mutation window — a global singleton (default capacity 1)
/// held by any node that mutates the meta repo, so two meta mutations never
/// interleave. Replaces the former runtime `meta::exclusive()` mutex: a
/// `MutexGuard` cannot span scheduler nodes, but a resource held by a
/// subtree root *can* — which is what lets the two-phase deploy
/// (`prepare_deploy` stages `flake.lock` uncommitted across the whole
/// container build, `finalize_deploy`/`abort_deploy` resolve it) be
/// decomposed into sub-nodes instead of one opaque node. Descendants of a
/// holder re-enter it through the crate's recursive lock, exactly like
/// [`Resource::Agent`].
MetaWindow,
}
impl NodeKind {
@ -31,7 +42,14 @@ impl NodeKind {
/// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt
/// container ops (`Start` / `Stop`, fanned out by a lease-holding
/// `Reconcile`) hold no lease of their own — they re-enter the ancestor's
/// `Agent` lock through the crate's recursive re-entrancy.
/// `Agent` lock through the crate's recursive re-entrancy. Meta-mutating
/// kinds ([`NodeKind::needs_meta_window`]) additionally take the global
/// [`Resource::MetaWindow`].
///
/// All of a node's resource edges are acquired **atomically**
/// (`try_acquire_all`) — a node never holds one resource while waiting on
/// another, so the multi-resource kinds (a `MetaLock` wants a build slot
/// *and* the meta window) cannot deadlock against each other.
pub fn resource_deps(&self) -> Vec<Dep<Resource>> {
let mut deps = Vec::new();
if self.needs_build_slot() {
@ -46,6 +64,12 @@ impl NodeKind {
count: 1,
});
}
if self.needs_meta_window() {
deps.push(Dep::Resource {
name: Resource::MetaWindow,
count: 1,
});
}
deps
}
}

5
hive-c0re/src/job_queue/submit.rs
View file

@ -95,8 +95,9 @@ fn start_chain(agent: &str, running: bool, stale: bool) -> Vec<NodeSpec> {
)];
if !running && stale {
// Rebuild subtree after the SetWanted head (base = 1, so the rebuild's
// `Prebuild` root deps `after_ok(0)` = the head). `Prebuild` +
// `Reconcile` are their own group roots (top-level, per `rebuild_nodes`).
// `MetaSync` root deps `after_ok(0)` = the head). `MetaSync`,
// `Prebuild` + `Reconcile` are their own group roots (top-level, per
// `rebuild_nodes`).
n.extend(rebuild_nodes(agent, true, 1));
} else {
n.push(child(

44
hive-c0re/src/job_queue/templates.rs
View file

@ -18,7 +18,7 @@
//! agent's existing `wanted`.
//!
//! ```text
//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
//! perm-change(a): WritePermFile(a) → «rebuild subgraph»
//! meta-update(inp): MetaLock(inp) →«in-DAG rebuild subgraph per affected a»
@ -68,45 +68,55 @@ pub(crate) fn child(parent: u64, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
}
}
/// The rebuild node subtree (nested, two group roots). `base` is the spec index
/// of the first node (`Prebuild`). Structure:
/// - `Prebuild` (base+0, **root**): owns the build slot for the whole subtree.
/// Lease-exempt — the nix build overlaps other DAGs on the same agent.
/// - `StopForUpdate` (base+1, child of `Prebuild`): owns the agent lease. Runs
/// The rebuild node subtree (nested, three group roots). `base` is the spec
/// index of the first node (`MetaSync`). Structure:
/// - `MetaSync` (base+0, **root**): the meta-repo preamble (dir prep, agent
/// sync, optional relock). Owns the global `MetaWindow` — and *only* for its
/// own short duration, which is why it is a sibling root rather than
/// `Prebuild`'s parent: a resource is held across the holder's whole subtree,
/// so parenting the build under it would extend a hive-global window over
/// every rebuild's nix build.
/// - `Prebuild` (base+1, **root**): `AfterOk` `MetaSync`. Owns the build slot
/// for the whole mechanical subtree below it. Lease-exempt — the nix build
/// overlaps other DAGs on the same agent.
/// - `StopForUpdate` (base+2, child of `Prebuild`): owns the agent lease. Runs
/// once `Prebuild` reaches `Finishing` (parent gate).
/// - `Swap` (base+2, child of `StopForUpdate`): borrows the agent lease from its
/// - `Swap` (base+3, child of `StopForUpdate`): borrows the agent lease from its
/// parent and the build slot from grand-ancestor `Prebuild` — both continuous.
/// - `PostSwap` (base+3, child of `StopForUpdate`): the swap's Ok-only
/// - `PostSwap` (base+4, child of `StopForUpdate`): the swap's Ok-only
/// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk`
/// its sibling `Swap`.
/// - `Reconcile` (base+4, **root**): `AfterAny` `Prebuild`, which rolls up
/// - `Reconcile` (base+5, **root**): `AfterAny` `Prebuild`, which rolls up
/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has
/// settled — so `Reconcile` runs after the swap regardless of outcome, and as
/// a top-level root it survives the cancel-cascade of a failed `Prebuild`
/// (recovery-start invariant). It takes a fresh lease; the tiny gap is
/// harmless — `Reconcile` converges to the persisted `wanted` idempotently.
/// (recovery-start invariant, which also covers a failed `MetaSync`: that
/// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It
/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to
/// the persisted `wanted` idempotently.
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec<NodeSpec> {
let a = || agent.to_owned();
vec![
node(
NodeKind::Prebuild { agent: a(), relock },
NodeKind::MetaSync { agent: a(), relock },
if base == 0 {
Vec::new()
} else {
after_ok(base - 1)
},
),
child(base, NodeKind::StopForUpdate { agent: a() }, Vec::new()),
child(base + 1, NodeKind::Swap { agent: a() }, Vec::new()),
node(NodeKind::Prebuild { agent: a() }, after_ok(base)),
child(base + 1, NodeKind::StopForUpdate { agent: a() }, Vec::new()),
child(base + 2, NodeKind::Swap { agent: a() }, Vec::new()),
child(
base + 1,
base + 2,
NodeKind::PostSwap { agent: a() },
after_ok(base + 2),
after_ok(base + 3),
),
node(
NodeKind::Reconcile { agent: a() },
vec![Dep {
on: base,
on: base + 1,
when: DepWhen::AfterAny,
}],
),

155
hive-c0re/src/job_queue/tests.rs
View file

@ -176,6 +176,7 @@ fn rebuild_chain_claims_in_dep_order() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
for expected in [
"meta_sync",
"prebuild",
"stop_for_update",
"swap",
@ -201,15 +202,29 @@ fn build_slot_serializes_nix_heavy_nodes() {
let q = JobQueue::new(1);
let a = submit(&q, rebuild("agent-a", "r"));
let b = submit(&q, rebuild("agent-b", "r"));
let first = claim_one(&q); // a's Prebuild takes the only slot
assert_eq!(first.dag_id, a);
assert_eq!(first.kind.as_str(), "prebuild");
q.complete_node(a, first.node_id, Ok(()));
// The rebuild heads are `MetaSync` (slot-free, but serialized on the
// global meta window), so drive each chain's head out of the way first.
let head_a = claim_one(&q);
assert_eq!(head_a.dag_id, a);
assert_eq!(head_a.kind.as_str(), "meta_sync");
q.complete_node(a, head_a.node_id, Ok(()));
// a's Prebuild takes the only slot; b's MetaSync is free to run beside it
// (different resources), but b's Prebuild is not.
let claims = q.claim_ready();
let mut kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect();
kinds.sort_unstable();
assert_eq!(kinds, vec![(a, "prebuild"), (b, "meta_sync")]);
for c in &claims {
q.complete_node(c.dag_id, c.node_id, Ok(()));
}
// Uniform hold: agent-a keeps the build slot across its whole build chain
// (Swap re-enters it), so a's StopForUpdate (lease, slot-free) runs but b's
// Prebuild must wait for a's slot-needers (through Swap) to finish.
let claims = q.claim_ready();
let kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect();
let kinds: Vec<(u64, &str)> = q
.claim_ready()
.iter()
.map(|c| (c.dag_id, c.kind.as_str()))
.collect();
assert_eq!(kinds, vec![(a, "stop_for_update")]);
assert!(
!kinds.iter().any(|&(d, _)| d == b),
@ -222,9 +237,23 @@ fn two_build_slots_run_two_prebuilds() {
let q = JobQueue::new(2);
submit(&q, rebuild("agent-a", "r"));
submit(&q, rebuild("agent-b", "r"));
let claims = q.claim_ready();
assert_eq!(claims.len(), 2, "two slots → two concurrent prebuilds");
assert!(claims.iter().all(|c| c.kind.as_str() == "prebuild"));
// Each rebuild's head `MetaSync` holds the cap-1 global meta window, so the
// two heads take turns — exactly the serialization the old runtime
// `meta::exclusive()` mutex imposed inside the prebuild executor. What must
// NOT serialize is the build itself: complete only the meta heads and watch
// both prebuilds end up in flight together, neither of them completed.
let mut prebuilds = Vec::new();
for _ in 0..3 {
for c in q.claim_ready() {
if c.kind.as_str() == "meta_sync" {
q.complete_node(c.dag_id, c.node_id, Ok(()));
} else {
prebuilds.push(c);
}
}
}
assert_eq!(prebuilds.len(), 2, "two slots → two concurrent prebuilds");
assert!(prebuilds.iter().all(|c| c.kind.as_str() == "prebuild"));
}
#[test]
@ -312,12 +341,23 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
None,
),
);
// Prebuild is lease-exempt: the stop's Reconcile takes the lease
// Both DAGs' heads are lease-independent of each other: the rebuild's
// MetaSync (meta window) and the stop's Reconcile (agent lease).
let heads = q.claim_ready();
let head_kinds: Vec<&str> = heads.iter().map(|c| c.kind.as_str()).collect();
assert!(head_kinds.contains(&"meta_sync"));
assert!(head_kinds.contains(&"reconcile"));
let meta_sync = heads
.iter()
.find(|c| c.kind.as_str() == "meta_sync")
.expect("meta_sync claim")
.clone();
q.complete_node(meta_sync.dag_id, meta_sync.node_id, Ok(()));
// Prebuild is lease-exempt: the stop's Reconcile keeps the lease
// and runs concurrently with the rebuild's out-of-band nix build.
let claims = q.claim_ready();
let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect();
assert!(kinds.contains(&"prebuild"));
assert!(kinds.contains(&"reconcile"));
// But the rebuild's StopForUpdate must then wait for the stop DAG
// to finish (lease).
let prebuild = claims
@ -330,7 +370,7 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
q.claim_ready().is_empty(),
"StopForUpdate blocked while stop DAG holds the lease"
);
let reconcile = claims
let reconcile = heads
.iter()
.find(|c| c.kind.as_str() == "reconcile")
.expect("reconcile claim")
@ -484,7 +524,7 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() {
head_agents.sort_unstable();
assert_eq!(head_agents, vec!["fresh", "stale"]);
// Complete both heads; the fresh agent then reconciles directly while
// the stale agent's subgraph is the rebuild chain (prebuild first).
// the stale agent's subgraph is the rebuild chain (meta_sync first).
for c in &heads {
q.complete_node(id, c.node_id, Ok(()));
}
@ -496,7 +536,7 @@ fn multi_agent_start_one_dag_folds_per_agent_stale_rebuild() {
kinds.sort_unstable();
assert_eq!(
kinds,
vec![("fresh", "reconcile"), ("stale", "prebuild")],
vec![("fresh", "reconcile"), ("stale", "meta_sync")],
"fresh agent starts directly; stale agent rebuilds first, all in one DAG"
);
}
@ -578,30 +618,48 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
let emitter = claim_one(&q);
assert_eq!(emitter.kind.as_str(), "meta_lock");
// Two independent per-agent subgraphs — the REAL production shape the
// sweep MetaLock grows (`rebuild_nodes(_, true, 0)`: root Prebuild →
// StopForUpdate → Swap → Reconcile, local 0-based deps), so this test
// tracks any drift in that builder's root-first (`base = 0`) shape.
// sweep MetaLock grows (`rebuild_nodes(_, true, 0)`: root MetaSync → root
// Prebuild → StopForUpdate → Swap → Reconcile, local 0-based deps), so this
// test tracks any drift in that builder's root-first (`base = 0`) shape.
let subgraph = |agent: &str| templates::rebuild_nodes(agent, true, 0);
// Must append BEFORE completing the emitter (the documented contract).
q.append_subgraph(id, &subgraph("a"), emitter.node_id);
q.append_subgraph(id, &subgraph("b"), emitter.node_id);
q.complete_node(id, emitter.node_id, Ok(()));
// Still ONE DAG; both subgraph roots become ready once the emitter is
// Done (rooted on it), each on its own agent lease.
// Done (rooted on it), each on its own agent lease. Their `MetaSync` heads
// take turns on the cap-1 global meta window, so drain those first — what
// must be concurrent is the builds.
assert_eq!(q.snapshot().len(), 1);
let next = q.claim_ready();
let mut kinds: Vec<(&str, &str)> = next
.iter()
.map(|c| (c.agent.as_str(), c.kind.as_str()))
.collect();
let mut kinds = drain_meta_syncs(&q, id);
kinds.sort_unstable();
assert_eq!(
kinds,
vec![("a", "prebuild"), ("b", "prebuild")],
vec![
("a".to_owned(), "prebuild".to_owned()),
("b".to_owned(), "prebuild".to_owned())
],
"both rebuild subgraphs root on the emitter and run concurrently in one DAG"
);
}
/// Complete every `MetaSync` head the queue offers (they take turns on the
/// cap-1 global meta window) and return whatever else got claimed alongside
/// them, as `(agent, kind)` pairs left in flight.
fn drain_meta_syncs(q: &JobQueue, dag: u64) -> Vec<(String, String)> {
let mut rest = Vec::new();
for _ in 0..3 {
for c in q.claim_ready() {
if c.kind.as_str() == "meta_sync" {
q.complete_node(dag, c.node_id, Ok(()));
} else {
rest.push((c.agent.clone(), c.kind.as_str().to_owned()));
}
}
}
rest
}
#[test]
fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
// The meta-update `MetaLock` grows one rebuild subgraph per affected
@ -637,17 +695,18 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
}
q.complete_node(id, meta_lock.node_id, Ok(()));
// Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root
// on the MetaLock, each on its own agent lease.
// on the MetaLock, each on its own agent lease. The per-agent `MetaSync`
// heads serialize on the global meta window (they commit to the meta repo);
// the builds behind them do not.
assert_eq!(q.snapshot().len(), 1);
let next = q.claim_ready();
let mut kinds: Vec<(&str, &str)> = next
.iter()
.map(|c| (c.agent.as_str(), c.kind.as_str()))
.collect();
let mut kinds = drain_meta_syncs(&q, id);
kinds.sort_unstable();
assert_eq!(
kinds,
vec![("alice", "prebuild"), ("bob", "prebuild")],
vec![
("alice".to_owned(), "prebuild".to_owned()),
("bob".to_owned(), "prebuild".to_owned())
],
"cascade rebuilds grow in the meta-update DAG, concurrent per agent"
);
}
@ -658,7 +717,11 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
fn failed_node_cancels_downstream_but_afterany_reconcile_runs() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
let meta_sync = claim_one(&q);
assert_eq!(meta_sync.kind.as_str(), "meta_sync");
q.complete_node(id, meta_sync.node_id, Ok(()));
let prebuild = claim_one(&q);
assert_eq!(prebuild.kind.as_str(), "prebuild");
q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned()));
// StopForUpdate + Swap are cancelled (AfterOk on a failed chain);
// the AfterAny Reconcile still runs once Swap is terminal.
@ -702,7 +765,8 @@ fn failed_node_cancels_downstream_but_afterany_reconcile_runs() {
fn swap_failure_still_runs_reconcile() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
for _ in 0..2 {
// meta_sync + prebuild + stop_for_update
for _ in 0..3 {
let c = claim_one(&q);
q.complete_node(id, c.node_id, Ok(()));
}
@ -735,8 +799,8 @@ fn swap_failure_still_runs_reconcile() {
fn swap_ok_runs_post_swap_before_reconcile() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
// prebuild + stop_for_update
for _ in 0..2 {
// meta_sync + prebuild + stop_for_update
for _ in 0..3 {
let c = claim_one(&q);
q.complete_node(id, c.node_id, Ok(()));
}
@ -979,13 +1043,25 @@ fn graceful_signal_and_drain_hold_no_build_slot() {
submit(&q, rebuild("builder", "slot hog"));
submit(&q, stop_online(&["agent-a"], true, "g"));
submit(&q, stop_online(&["agent-b"], true, "g"));
let claims = q.claim_ready();
let kinds: Vec<&str> = claims.iter().map(|c| c.kind.as_str()).collect();
// All three DAG heads are build-slot-exempt, so they run at once.
let heads = q.claim_ready();
let kinds: Vec<&str> = heads.iter().map(|c| c.kind.as_str()).collect();
assert_eq!(kinds, vec!["meta_sync", "set_wanted", "set_wanted"]);
for c in &heads {
q.complete_node(c.dag_id, c.node_id, Ok(()));
}
// Now the rebuild's Prebuild holds the single slot — and both graceful
// stops still proceed to their Signal beside it.
let kinds: Vec<&str> = q
.claim_ready()
.iter()
.map(|c| c.kind.as_str())
.collect::<Vec<_>>();
assert_eq!(
kinds,
vec!["prebuild", "set_wanted", "set_wanted"],
"both agents' graceful-stop heads (SetWanted, build-slot-exempt) run \
while the slot is held; their signals follow"
vec!["prebuild", "signal", "signal"],
"both agents' graceful-stop signals (build-slot-exempt) run while the \
rebuild holds the slot"
);
}
@ -1023,6 +1099,7 @@ fn perm_change_shape_prefixes_rebuild_chain() {
);
for expected in [
"write_perm_file",
"meta_sync",
"prebuild",
"stop_for_update",
"swap",

29
hive-c0re/src/meta.rs
View file

@ -27,24 +27,17 @@ const GIT_EMAIL: &str = "c0re@hyperhive.local";
/// take turns instead of colliding.
static META_LOCK: Mutex<()> = Mutex::const_new(());
/// Coarse exclusivity for meta-repo *windows* that span multiple
/// `META_LOCK` acquisitions — above all the two-phase deploy
/// (`prepare_deploy` stages `flake.lock` uncommitted for the whole
/// container build; `finalize_deploy` / `abort_deploy` resolve it).
/// `META_LOCK` serializes individual git ops but cannot keep another
/// op out of that staged window: a perm-file or lock-bump commit
/// landing mid-window would sweep the staged deploy lock into its own
/// commit and neuter `abort_deploy`. Job-queue executors that mutate
/// the meta repo hold this gate for their mutation span; the opaque
/// approval-deploy node holds it across its whole prepare→finalize
/// span. Never acquired inside this module's functions (they run
/// *under* a caller's window — nesting would deadlock).
static DEPLOY_GATE: Mutex<()> = Mutex::const_new(());
/// Acquire the deploy/meta-mutation window gate. See [`DEPLOY_GATE`].
pub async fn exclusive() -> tokio::sync::MutexGuard<'static, ()> {
DEPLOY_GATE.lock().await
}
// Exclusivity for meta-repo *windows* that span multiple `META_LOCK`
// acquisitions — above all the two-phase deploy (`prepare_deploy` stages
// `flake.lock` uncommitted for the whole container build;
// `finalize_deploy` / `abort_deploy` resolve it) — is **not** a mutex in
// this module. `META_LOCK` above serializes individual git ops but cannot
// keep another op out of that staged window; that window is owned by the
// job queue instead, as `Resource::MetaWindow`, declared by every
// meta-mutating node kind (`NodeKind::needs_meta_window`). A resource can
// be held by a subtree root across its children, which a `MutexGuard`
// (bounded by one executor fn) cannot — that's what lets the deploy be
// modelled as sub-nodes rather than one opaque node.
/// Where the manager sees this directory inside its container (RO bind).
pub const CONTAINER_MANAGER_META_MOUNT: &str = "/meta";