dfadacd45f
The two-phase approval deploy keeps a bumped `flake.lock` staged uncommitted for the whole container build, so no other meta mutation may land inside that span — until now enforced by a process-global `meta::exclusive()` mutex held inside each executor fn. A `MutexGuard` cannot outlive the fn that takes it, which is what blocks decomposing the opaque `ApprovalDeploy` node into scheduler-visible sub-nodes: the window has to span them. Replace the mutex with `Resource::MetaWindow`, a global capacity-1 queue resource declared by every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources are held by a subtree root across its whole subtree, so a later increment can hang the deploy's phases under one window-holding parent. Same global serialisation as before, and the scheduler now blocks a node from being claimed rather than parking a worker on a mutex. Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync` node. `Prebuild` must NOT hold the window: the old mutex was deliberately scoped to drop before the multi-minute toplevel build, which only reads the store, and a cap-1 global held across it would serialise every agent's rebuild behind every other's. `MetaSync` is a sibling root that `Prebuild` deps `AfterOk` on — not its parent, since a parent's resource covers its whole subtree and would reintroduce exactly that problem. Queue tests: shape assertions gain the extra node, which is the point of the change (phases become nodes). The concurrency invariants are intact but observed one step later — the `MetaSync` heads take turns on the window, exactly as the runtime mutex made them, so those tests now complete the heads before asserting that the prebuilds overlap.
75 lines
3.5 KiB
Rust
75 lines
3.5 KiB
Rust
//! The concrete resource type the rebuild queue schedules over — the bridge
|
|
//! from hive-c0re's [`NodeKind`] onto the domain-agnostic `hive-jobq` crate.
|
|
//! `hive-jobq` is generic over a resource type `R: Clone + Eq + Hash` and a node
|
|
//! payload `N`; here `R` is [`Resource`] and `N` is [`NodeKind`] directly (each
|
|
//! variant carries the agent it targets).
|
|
|
|
use hive_jobq::Dep;
|
|
|
|
use super::model::NodeKind;
|
|
|
|
/// The two resource classes the queue gates concurrency on, as the crate's
|
|
/// generic resource type `R`.
|
|
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
|
|
pub enum Resource {
|
|
/// One of the `buildSlots` permits, held by a nix-heavy node for its
|
|
/// duration. Capacity is `services.hyperhive.c0re.buildSlots` (default 1),
|
|
/// set on the [`hive_jobq::resources::ResourceTable`] at construction.
|
|
BuildSlot,
|
|
/// The per-agent lifecycle lease — globally exclusive per agent across all
|
|
/// DAGs (unconfigured, so the crate's default capacity 1 applies). Held by
|
|
/// a DAG's first container-affecting node for that agent and re-entered by
|
|
/// the rest of that agent's subtree via the crate's recursive lock, so two
|
|
/// DAGs never interleave container ops on one agent.
|
|
Agent(String),
|
|
/// The meta-repo mutation window — a global singleton (default capacity 1)
|
|
/// held by any node that mutates the meta repo, so two meta mutations never
|
|
/// interleave. Replaces the former runtime `meta::exclusive()` mutex: a
|
|
/// `MutexGuard` cannot span scheduler nodes, but a resource held by a
|
|
/// subtree root *can* — which is what lets the two-phase deploy
|
|
/// (`prepare_deploy` stages `flake.lock` uncommitted across the whole
|
|
/// container build, `finalize_deploy`/`abort_deploy` resolve it) be
|
|
/// decomposed into sub-nodes instead of one opaque node. Descendants of a
|
|
/// holder re-enter it through the crate's recursive lock, exactly like
|
|
/// [`Resource::Agent`].
|
|
MetaWindow,
|
|
}
|
|
|
|
impl NodeKind {
|
|
/// The [`Dep::Resource`] edges this node must acquire to run, derived from
|
|
/// its kind + agent: a build slot for nix-heavy kinds
|
|
/// ([`NodeKind::needs_build_slot`]) and the agent lease for
|
|
/// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt
|
|
/// container ops (`Start` / `Stop`, fanned out by a lease-holding
|
|
/// `Reconcile`) hold no lease of their own — they re-enter the ancestor's
|
|
/// `Agent` lock through the crate's recursive re-entrancy. Meta-mutating
|
|
/// kinds ([`NodeKind::needs_meta_window`]) additionally take the global
|
|
/// [`Resource::MetaWindow`].
|
|
///
|
|
/// All of a node's resource edges are acquired **atomically**
|
|
/// (`try_acquire_all`) — a node never holds one resource while waiting on
|
|
/// another, so the multi-resource kinds (a `MetaLock` wants a build slot
|
|
/// *and* the meta window) cannot deadlock against each other.
|
|
pub fn resource_deps(&self) -> Vec<Dep<Resource>> {
|
|
let mut deps = Vec::new();
|
|
if self.needs_build_slot() {
|
|
deps.push(Dep::Resource {
|
|
name: Resource::BuildSlot,
|
|
count: 1,
|
|
});
|
|
}
|
|
if self.needs_lease() {
|
|
deps.push(Dep::Resource {
|
|
name: Resource::Agent(self.agent().to_owned()),
|
|
count: 1,
|
|
});
|
|
}
|
|
if self.needs_meta_window() {
|
|
deps.push(Dep::Resource {
|
|
name: Resource::MetaWindow,
|
|
count: 1,
|
|
});
|
|
}
|
|
deps
|
|
}
|
|
}
|