vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(job-queue): promote the meta-repo deploy window to a queue resource

Browse source 
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 • committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

93
hive-c0re/src/job_queue/exec.rs
View file

@ -82,7 +82,8 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
node_id: claim.node_id,
};
match &claim.kind {
NodeKind::Prebuild { relock, .. } => run_prebuild(coord, claim, &ctx, *relock).await,
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await,
NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, claim, &ctx).await,
NodeKind::Provision { .. } => run_provision(coord, claim, &ctx).await,
@ -186,24 +187,24 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<N
Ok(NodeOutput::default())
}
/// Out-of-band toplevel build while the container keeps serving: meta
/// sync + optional per-agent relock, then warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead. The meta sync +
/// dir prep still run unconditionally (the `Swap` depends on them).
async fn run_prebuild(
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
/// `sync_agents`, and the optional per-agent relock. Runs under the deploy
/// window (`NodeKind::needs_meta_window`, held by the scheduler for this
/// node) so its commits can never land inside another node's staged
/// prepare→finalize window.
///
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that
/// build takes minutes and only *reads* the store, so keeping the global
/// window off it is what lets rebuilds of different agents overlap.
async fn run_meta_sync(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
relock: bool,
) -> Result<NodeOutput> {
let name = &claim.agent;
// Prebuild runs while the agent is still up — the runtime dir and
// MCP listener already exist. Use the pure path accessor; no need
// to re-register the listener (event-driven: registered at start/create).
// Runs while the agent is still up — the runtime dir and MCP listener
// already exist. Use the pure path accessor; no need to re-register the
// listener (event-driven: registered at start/create).
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
@ -211,18 +212,24 @@ async fn run_prebuild(
// Idempotent meta sync so a manual rebuild can also recover from a
// divergent meta repo; then bump just this agent's input. `relock =
// false` only for meta-update cascade children, where re-locking
// would revert the bump the cascade just committed. Both run under
// the deploy-window gate so they can never land inside another
// node's staged prepare→finalize window; the gate drops before the
// (long) toplevel build, which only reads the store.
{
let _window = crate::meta::exclusive().await;
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?;
if relock {
crate::meta::lock_update_for_rebuild(name).await?;
}
// would revert the bump the cascade just committed.
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?;
if relock {
crate::meta::lock_update_for_rebuild(name).await?;
}
Ok(NodeOutput::default())
}
/// Out-of-band toplevel build while the container keeps serving: warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap, against a meta repo the upstream
/// `MetaSync` node has already synced. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead.
async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
let name = &claim.agent;
// Warm the toplevel build only when the container is up — the whole
// point of prebuild is to shrink the swap's downtime window. A
// stopped agent has no uptime to preserve, so skip the (expensive)
@ -303,10 +310,9 @@ async fn run_post_swap(
}
/// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Holds the
/// deploy-window gate for the commit so it can't land inside another
/// node's staged deploy window — same discipline as `Prebuild`, which
/// commits under the gate then drops it before the (store-only) build.
/// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (`NodeKind::needs_meta_window`) so its commit can't
/// land inside another node's staged deploy window.
async fn run_provision(
coord: &Arc<Coordinator>,
claim: &Claim,
@ -317,7 +323,6 @@ async fn run_provision(
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
ctx.step("provisioning");
let _window = crate::meta::exclusive().await;
crate::lifecycle::provision_container(name, &hive, &paths).await?;
Ok(NodeOutput::default())
}
@ -347,7 +352,6 @@ async fn run_meta_lock(
) -> Result<NodeOutput> {
if sweep {
ctx.step("nix flake update hyperhive");
let _window = crate::meta::exclusive().await;
if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
}
@ -364,10 +368,7 @@ async fn run_meta_lock(
}
let _progress = coord.meta_update_guard();
ctx.step("nix flake update");
{
let _window = crate::meta::exclusive().await;
crate::meta::lock_update(&claim.inputs).await?;
}
crate::meta::lock_update(&claim.inputs).await?;
// Lock file changed — meta-inputs panel re-renders.
crate::dashboard::emit_meta_inputs_snapshot(coord);
let cascade = match fanout {
@ -545,11 +546,10 @@ async fn run_write_perm_file(
anyhow::bail!("run_write_perm_file on a non-WritePermFile node");
};
ctx.step("writing + committing perm file");
// Deploy-window gate: a perm commit landing inside another node's
// staged prepare→finalize window would sweep the staged deploy
// lock into its commit (the commits are also path-limited in
// meta.rs — belt and braces).
let _window = crate::meta::exclusive().await;
// Runs under the deploy window (`NodeKind::needs_meta_window`): a
// perm commit landing inside another node's staged prepare→finalize
// window would sweep the staged deploy lock into its commit (the
// commits are also path-limited in meta.rs — belt and braces).
match payload {
PermPayload::ToolGroups { groups } => {
crate::meta::commit_tool_groups(name, groups)
@ -586,11 +586,14 @@ async fn run_approval_deploy(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
let approval_id = claim
.approval_id
.with_context(|| format!("approval_deploy dag {} has no approval_id", claim.dag_id))?;
// Hold the deploy-window gate for the whole prepare→finalize span:
// `prepare_deploy` stages `flake.lock` uncommitted for the entire
// container build, and no other meta mutation may land inside that
// window (it would sweep the staged lock and neuter `abort_deploy`).
let _window = crate::meta::exclusive().await;
// The deploy window covers the whole prepare→finalize span
// (`NodeKind::needs_meta_window`, held by the scheduler for this
// node): `prepare_deploy` stages `flake.lock` uncommitted for the
// entire container build, and no other meta mutation may land inside
// that window (it would sweep the staged lock and neuter
// `abort_deploy`). Holding it as a queue resource — rather than a
// `MutexGuard` that cannot outlive this fn — is what lets increment 2
// decompose this node into sub-nodes under a window-holding parent.
crate::actions::run_approval_merge_config_pr(coord, Some(claim.dag_id), approval_id)
.await
.map(|()| NodeOutput::default())