vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(job-queue): promote the meta-repo deploy window to a queue resource

Browse source 
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.

A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.

Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.

Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.

Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
This commit is contained in:
atlas 2026-07-25 20:00:29 +02:00 • committed by mara
commit dfadacd45f
8 changed files with 311 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

66
hive-c0re/src/job_queue/model.rs
View file

@ -99,15 +99,27 @@ pub struct Dep {
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum NodeKind {
/// The rebuild's meta-repo preamble: `lifecycle::prepare_rebuild_dirs`,
/// an idempotent meta `sync_agents`, and an optional per-agent relock.
/// `relock = false` only for meta-update cascade rebuilds (re-locking
/// would revert the bump the cascade just committed).
///
/// Its own node — ahead of, and *not* an ancestor of, [`NodeKind::Prebuild`]
/// — precisely because it is the only part of the rebuild that mutates the
/// meta repo and so holds the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow). Fusing
/// it into `Prebuild` (or making it `Prebuild`'s parent, which holds a
/// resource across the whole subtree) would extend that global window over
/// the multi-minute toplevel build and serialize rebuilds hive-wide.
/// Store/meta work only — build-slot- and lease-exempt.
MetaSync { agent: String, relock: bool },
/// Out-of-band toplevel build while the container keeps serving:
/// meta `sync_agents`, optional per-agent relock, then
/// `lifecycle::prebuild_toplevel`. `relock = false` only for
/// meta-update cascade rebuilds (re-locking would revert the bump
/// the cascade just committed). The `prebuild_toplevel` warm is
/// skipped when the container is already down — it only exists to
/// shrink the swap's downtime, which a stopped agent doesn't need
/// (the sync + dir prep still run; `Swap` builds inline).
Prebuild { agent: String, relock: bool },
/// `lifecycle::prebuild_toplevel`, reading a meta repo the upstream
/// [`NodeKind::MetaSync`] has already synced. The warm build is skipped
/// when the container is already down — it only exists to shrink the
/// swap's downtime, which a stopped agent doesn't need (`Swap` builds
/// inline instead).
Prebuild { agent: String },
/// `nixos-container update` profile-swap (requires the container
/// stopped). Re-applies nspawn flags + resource limits first —
/// rebuild is the reconcile verb. The post-rebuild bookkeeping tail
@ -216,6 +228,7 @@ impl NodeKind {
/// Wire string for `NodeView.kind`.
pub fn as_str(&self) -> &'static str {
match self {
NodeKind::MetaSync { .. } => "meta_sync",
NodeKind::Prebuild { .. } => "prebuild",
NodeKind::Swap { .. } => "swap",
NodeKind::PostSwap { .. } => "post_swap",
@ -242,7 +255,8 @@ impl NodeKind {
#[must_use]
pub fn agent(&self) -> &str {
match self {
NodeKind::Prebuild { agent, .. }
NodeKind::MetaSync { agent, .. }
| NodeKind::Prebuild { agent }
| NodeKind::Swap { agent }
| NodeKind::PostSwap { agent }
| NodeKind::Provision { agent }
@ -276,8 +290,8 @@ impl NodeKind {
/// Container-affecting kinds require the DAG to hold the agent's
/// lifecycle lease (acquired at the first such node, held until the
/// DAG is terminal). Lease-exempt kinds (`Prebuild`, `Provision`,
/// `MetaLock`, `WritePermFile`) touch the store / meta repo, not the
/// DAG is terminal). Lease-exempt kinds (`MetaSync`, `Prebuild`,
/// `Provision`, `MetaLock`, `WritePermFile`) touch the store / meta repo, not the
/// running container — which is exactly why a `Prebuild` can overlap
/// another DAG's work on the same agent. `Provision` precedes the
/// container's existence entirely, so the lease is first taken at the
@ -296,6 +310,36 @@ impl NodeKind {
| NodeKind::SetWanted { .. }
)
}
/// Kinds that **mutate the meta repo** and so must hold the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow) for
/// their duration: no two meta mutations may interleave, because a commit
/// landing inside another node's staged `prepare_deploy`→`finalize_deploy`
/// window would sweep the staged `flake.lock` into its own commit and
/// neuter `abort_deploy`.
///
/// This is the queue-primitive replacement for the former runtime
/// `meta::exclusive()` mutex — same global serialisation, but held by the
/// scheduler and therefore able to span a whole subtree, which a
/// `MutexGuard` cannot.
///
/// Note what is **not** here: [`NodeKind::Prebuild`]. The window must stay
/// off the multi-minute toplevel build, which only *reads* the store — the
/// old mutex was scoped to drop before it, and holding a cap-1 global
/// across it would serialize every agent's rebuild behind every other's.
/// That is why the meta preamble is its own [`NodeKind::MetaSync`] node,
/// and why that node is a sibling rather than `Prebuild`'s parent (a
/// resource held by a parent covers its whole subtree).
pub fn needs_meta_window(&self) -> bool {
matches!(
self,
NodeKind::MetaSync { .. }
| NodeKind::Provision { .. }
| NodeKind::MetaLock { .. }
| NodeKind::WritePermFile { .. }
| NodeKind::ApprovalDeploy { .. }
)
}
}
/// Submit-time spec for one node.