Compare commits

...
Author SHA1 Message Date
atlas
58a9f218f2 job_queue: fix the boot sweep's lost declarations, drop the node wrapper
Two review findings on the resources-at-construction change.

argus: `workers::auto_update`'s boot sweep constructs nodes through
`templates::node` too, and it was not converted. With the kind-derived
declaration gone, its sweep `MetaLock` and its per-agent `Reconcile`
silently declared no resources at all — so a boot reconcile no longer
held the agent lease and could race another DAG's container ops, and the
sweep's meta commit could land inside another node's staged deploy
window. Nothing failed to compile: removing an implicit behaviour from a
helper is invisible at every call site that relied on it.

The declarations now live in a pure `boot_nodes`, split out of
`submit_boot_tree` so they can be exercised without a `Coordinator`.
That path is the only place job nodes are built outside `job_queue/`,
which is exactly why it had no coverage; `boot_sweep_nodes_declare_
their_own_resources` closes that, asserting against declared graph edges
rather than against the kind.

mara: `templates::node` is a redundant redirect now that it no longer
derives resources — deleted, and its 43 call sites use `Job::node`
directly. The reasoning it documented moved to the module docs of
`templates.rs` and `resource.rs`, which is where it stays true.
2026-08-02 16:29:06 +02:00
atlas
10dbdb444d job_queue: declare a node's resources where the node is constructed
Resources were derived from the node's kind: `templates::node` called
`NodeKind::resource_deps()`, which fanned out to `needs_build_slot` /
`needs_lease` / `needs_meta_window`. That made the requirement a property
of the *kind*, so a kind that happened to run under an ancestor already
holding the resource could get away with declaring nothing.

Three did. `Start`, `Stop` and `PostSwap` appear in none of the three
predicates, and that was only safe because one construction site fans
them out from inside a lease-holding `Reconcile` — a fact about today's
DAG shape, not about the nodes.

Each of the 41 construction sites now says what it holds. `Start` /
`Stop` / `PostSwap` declare the agent lease; per the contract that is a
re-entrant borrow, which a new test pins rather than argues.

`running_transients` reads the node's declared deps instead of
re-deriving from the kind. That closes the blank-pill gap: the pill went
blank during container start, stop and the post-swap tail because the
declaration was missing, not because the filter was wrong.

The deleted predicates carried the only written record of three design
decisions; each moved to the `Resource` variant it constrains rather than
dying with its function.
2026-08-02 16:29:06 +02:00
10 changed files with 432 additions and 302 deletions

View file

@ -75,7 +75,7 @@ container build:
resource declared by every node kind that mutates the meta repo — `MetaSync`,
`MetaLock`, `WritePermFile`, `Reparent`, `Provision`'s agent registration, and
`DeployWindow` — the deploy subtree's root, which holds it across every
phase below it (`NodeKind::needs_meta_window`). Two meta
phase below it (it declares `Resource::MetaWindow`). Two meta
mutations can therefore never interleave, so no commit lands inside another
node's staged window. It is a queue resource rather than a runtime mutex
because a resource is held by a subtree root across its whole subtree, which

View file

@ -14,6 +14,7 @@ use super::{Claim, Declare};
use hive_jobq::TerminalState;
use super::model::NodeKind;
use super::resource::Resource;
use crate::coordinator::Coordinator;
use crate::power::{ReconcileAction, reconcile_action};
@ -168,8 +169,8 @@ fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) -> NodeOu
/// Write the agent's durable power intent — the DAG-node form of the old
/// pre-submit `set_wanted` side effect. Store-only (no container touch), so
/// build-slot-exempt; but it takes the agent's lifecycle lease (see
/// `NodeKind::needs_lease`) so the whole power-op DAG is atomic per-agent.
/// build-slot-exempt; but it declares the agent's lifecycle lease
/// (`Resource::Agent`) so the whole power-op DAG is atomic per-agent.
/// The downstream `Reconcile` reads the intent this writes. Unlike the old
/// warn-and-continue write, a failed write fails the node (cancel-downstream
/// cancels the `Reconcile`) rather than letting it converge to a stale
@ -189,7 +190,7 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<N
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
/// `sync_agents`, and the optional per-agent relock. Runs under the deploy
/// window (`NodeKind::needs_meta_window`, held by the scheduler for this
/// window (`Resource::MetaWindow`, held by the scheduler for this
/// node) so its commits can never land inside another node's staged
/// prepare→finalize window.
///
@ -302,7 +303,7 @@ async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
/// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (`NodeKind::needs_meta_window`) so its commit can't
/// deploy window (it declares `Resource::MetaWindow`) so its commit can't
/// land inside another node's staged deploy window.
async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
@ -417,8 +418,13 @@ async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
// carries the agent it targets, so stamp `claim.agent` into the fanned-out
// Start/Stop kind (one in-DAG-growth channel).
let sub = |kind: NodeKind| {
// `Start` / `Stop` declare the lease they run under. This node is their
// parent and holds it, so the declaration is a re-entrant borrow — no
// second unit, no deadlock. It exists so the requirement belongs to the
// node rather than to the fact that a `Reconcile` happens to fan it out.
let lease = Resource::Agent(kind.agent().to_owned());
vec![Box::new(move |b: &super::Job| {
let _ = super::templates::node(b, kind);
let _ = b.node(kind).needs(lease);
}) as Declare]
};
let append_subgraph = match reconcile_action(wanted, running) {
@ -556,7 +562,7 @@ async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
let NodeKind::WritePermFile { payload, .. } = &claim.kind else {
anyhow::bail!("run_write_perm_file on a non-WritePermFile node");
};
// Runs under the deploy window (`NodeKind::needs_meta_window`): a
// Runs under the deploy window (it declares `Resource::MetaWindow`): a
// perm commit landing inside another node's staged prepare→finalize
// window would sweep the staged deploy lock into its commit (the
// commits are also path-limited in meta.rs — belt and braces).
@ -592,7 +598,7 @@ async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
/// commit (`Coordinator::reparent_bulk_with_notify`, which already handles
/// both the single- and bulk-move case, sends the per-agent move
/// notifications, and rescans + diff-emits the container tree). Runs under
/// the deploy window (`NodeKind::needs_meta_window`), same reasoning as
/// the deploy window (it declares `Resource::MetaWindow`), same reasoning as
/// `run_write_perm_file`: a topology commit landing inside another node's
/// staged deploy window would sweep the staged lock into its commit.
async fn run_reparent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {

View file

@ -9,7 +9,7 @@
//! - [`model::NodeKind`] **is** the crate payload `N` directly — each variant
//! carries the agent it targets ([`NodeKind::agent`]); the two resource
//! classes are [`resource::Resource`] (`BuildSlot` node-held, `Agent` lease
//! subtree-held), derived per node by [`NodeKind::resource_deps`];
//! subtree-held), declared per node at its construction site;
//! - a **DAG is a single container node** ([`NodeKind::Dag`], `parent = None`)
//! carrying the group's metadata, with the work nodes hung under it as
//! its subtree (the **parent axis** groups; `deps` order). So the container's
@ -436,10 +436,13 @@ impl JobQueue {
/// `Stop` pill are both pills; only one means a vanished container is
/// expected.
///
/// By design, `Start` / `Stop` / `PostSwap` run inside a lease-holding
/// ancestor and re-declare nothing, so they light no pill; closing that is
/// the resources-where-constructed work, not this function. An agent's lease
/// is cap-1, so at most one entry per agent.
/// Read off the node's **declared** resource edges, not off its kind. Those
/// are the same thing now that every construction site states what it holds,
/// and the distinction is the whole point: `Start` / `Stop` / `PostSwap` run
/// inside a lease-holding ancestor, and while the declaration was derived
/// from the kind they re-declared nothing and lit no pill. Asking the node
/// what it holds cannot go stale that way. An agent's lease is cap-1, so at
/// most one entry per agent.
#[must_use]
pub fn running_transients(&self) -> Vec<RunningTransient> {
let inner = self.lock();
@ -449,14 +452,13 @@ impl JobQueue {
.nodes()
.filter(|n| matches!(n.state, State::Running))
.filter_map(|n| {
let agent =
n.payload
.resource_deps()
.into_iter()
.find_map(|(name, _)| match name {
Resource::Agent(a) => Some(a),
_ => None,
})?;
let agent = n.deps.iter().find_map(|dep| match dep {
hive_jobq::Dep::Resource {
name: Resource::Agent(a),
..
} => Some(a.clone()),
_ => None,
})?;
Some(RunningTransient {
agent,
label: n.payload.as_str().to_owned(),

View file

@ -21,7 +21,8 @@ use hive_jobq::TerminalState;
/// The primitive operations — each kind maps to one executor fn in
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta.rs`
/// code. Concurrency is gated by two resource classes (see
/// [`NodeKind::needs_build_slot`] / [`NodeKind::needs_lease`]); the
/// [`Resource`](super::resource::Resource), declared per node where the node
/// is constructed rather than derived from its kind); the
/// meta *repo* is serialized by `meta::META_LOCK` inside the wrapped
/// functions themselves, which is why there is no `GitCommit` node —
/// a standalone commit node would open a dirty-working-tree window
@ -62,8 +63,10 @@ pub enum NodeKind {
/// swap succeeded; the tail `Reconcile` deps `AfterAny(PostSwap)`, so on
/// swap failure this node is cancel-cascaded (a terminal state) and
/// recovery still runs. Store/forge/matrix work only — no nix build, so
/// build-slot-exempt; the agent lease taken at `Swap` is held across the
/// whole chain until `Reconcile` settles, so it's not re-declared here.
/// build-slot-exempt. It *does* declare the agent lease: an ancestor in the
/// stop chain already holds it, so this is a re-entrant borrow rather than a
/// second unit — declaring it keeps the requirement true of this node rather
/// than of the one DAG shape it happens to be used in.
PostSwap { agent: String },
/// First-spawn pre-create provisioning: proposed/applied repos,
/// state subvolume, and meta registration (`sync_agents`). Runs
@ -124,7 +127,7 @@ pub enum NodeKind {
/// Topology move(s) — `set-parent` (len 1) or `set-parent-bulk` (len N) —
/// as a single queue node. Agentless like [`NodeKind::MetaLock`]: a
/// reparent touches the meta repo, not any one container, and a bulk
/// move spans multiple agents anyway. `needs_meta_window() = true`, same
/// move spans multiple agents anyway. Declares the meta window, same
/// precedent as [`NodeKind::WritePermFile`] (also a small
/// git-commit-under-`META_LOCK` op) — a reparent's commit must not land
/// inside another node's staged deploy `prepare_deploy`→`finalize_deploy`
@ -347,42 +350,6 @@ impl NodeKind {
}
}
/// Nix-heavy kinds hold one of the `buildSlots` semaphore permits
/// for the node's duration.
pub fn needs_build_slot(&self) -> bool {
matches!(
self,
NodeKind::Prebuild { .. }
| NodeKind::Swap { .. }
| NodeKind::Create { .. }
| NodeKind::MetaLock { .. }
| NodeKind::DeployWindow { .. }
)
}
/// Container-affecting kinds require the DAG to hold the agent's
/// lifecycle lease (acquired at the first such node, held until the
/// DAG is terminal). Lease-exempt kinds (`MetaSync`, `Prebuild`,
/// `Provision`, `MetaLock`, `WritePermFile`, `Reparent`) touch the store / meta repo, not the
/// running container — which is exactly why a `Prebuild` can overlap
/// another DAG's work on the same agent. `Provision` precedes the
/// container's existence entirely, so the lease is first taken at the
/// `Create` node it feeds.
pub fn needs_lease(&self) -> bool {
matches!(
self,
NodeKind::Swap { .. }
| NodeKind::Create { .. }
| NodeKind::Reconcile { .. }
| NodeKind::StopForUpdate { .. }
| NodeKind::Signal { .. }
| NodeKind::Drain { .. }
| NodeKind::WriteDropin { .. }
| NodeKind::DeployWindow { .. }
| NodeKind::SetWanted { .. }
)
}
/// Whether running this node is *expected* to take the agent's container
/// down. Feeds `TransientState::deliberate_stop`, which the crash watcher
/// reads to tell an intentional stop from a crash.
@ -420,45 +387,6 @@ impl NodeKind {
// their own answer.
// - `DeployWindow` brackets a deploy without itself stopping anything.
}
/// Kinds that **mutate the meta repo** and so must hold the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow) for
/// their duration: no two meta mutations may interleave, because a commit
/// landing inside another node's staged `prepare_deploy`→`finalize_deploy`
/// window would sweep the staged `flake.lock` into its own commit and
/// neuter `abort_deploy`.
///
/// This is the queue-primitive replacement for the former runtime
/// `meta::exclusive()` mutex — same global serialisation, but held by the
/// scheduler and therefore able to span a whole subtree, which a
/// `MutexGuard` cannot.
///
/// Note what is **not** here: [`NodeKind::Prebuild`]. The window must stay
/// off the multi-minute toplevel build, which only *reads* the store — the
/// old mutex was scoped to drop before it, and holding a cap-1 global
/// across it would serialize every agent's rebuild behind every other's.
/// That is why the meta preamble is its own [`NodeKind::MetaSync`] node,
/// and why that node is a sibling rather than `Prebuild`'s parent (a
/// resource held by a parent covers its whole subtree).
///
/// Two of these kinds run *inside* a [`NodeKind::DeployWindow`]'s subtree
/// (the appended rebuild's `MetaSync`, and [`NodeKind::FinalizeDeploy`]).
/// They still declare the window: a descendant re-enters an ancestor's hold
/// through the crate's recursive lock, exactly as `Start` / `Stop` re-enter
/// a `Reconcile`'s agent lease. Declaring it is what keeps the requirement
/// true of the *node* rather than of one particular DAG shape.
pub fn needs_meta_window(&self) -> bool {
matches!(
self,
NodeKind::MetaSync { .. }
| NodeKind::Provision { .. }
| NodeKind::MetaLock { .. }
| NodeKind::WritePermFile { .. }
| NodeKind::Reparent { .. }
| NodeKind::DeployWindow { .. }
| NodeKind::FinalizeDeploy { .. }
)
}
}
/// Submit-time spec for a whole DAG: the group's metadata plus the declared —

View file

@ -3,8 +3,16 @@
//! `hive-jobq` is generic over a resource type `R: Clone + Eq + Hash` and a node
//! payload `N`; here `R` is [`Resource`] and `N` is [`NodeKind`] directly (each
//! variant carries the agent it targets).
use super::model::NodeKind;
//!
//! **A node's resources are declared where the node is constructed** with
//! `.needs(…)`, not derived from its kind. Deriving them made the requirement a
//! property of the *kind*, which let a kind that happened to run under a
//! holding ancestor declare nothing at all.
//!
//! All of a node's resource edges are acquired **atomically**
//! (`try_acquire_all`) — a node never holds one resource while waiting on
//! another, so the multi-resource nodes (a `MetaLock` wants a build slot *and*
//! the meta window) cannot deadlock against each other.
/// The two resource classes the queue gates concurrency on, as the crate's
/// generic resource type `R`.
@ -19,10 +27,20 @@ pub enum Resource {
/// a DAG's first container-affecting node for that agent and re-entered by
/// the rest of that agent's subtree via the crate's recursive lock, so two
/// DAGs never interleave container ops on one agent.
///
/// Nodes that touch the *store or meta repo* rather than the running
/// container do not declare it — `MetaSync`, `Prebuild`, `Provision`,
/// `MetaLock`, `WritePermFile`, `Reparent`. That exemption is what lets a
/// `Prebuild` overlap another DAG's work on the same agent. `Provision`
/// precedes the container's existence entirely, so the lease is first taken
/// at the `Create` it feeds.
Agent(String),
/// The meta-repo mutation window — a global singleton (default capacity 1)
/// held by any node that mutates the meta repo, so two meta mutations never
/// interleave. Replaces the former runtime `meta::exclusive()` mutex: a
/// interleave. That exclusion is load-bearing: a commit landing inside
/// another node's staged `prepare_deploy`→`finalize_deploy` window would
/// sweep the staged `flake.lock` into its own commit and neuter
/// `abort_deploy`. Replaces the former runtime `meta::exclusive()` mutex: a
/// `MutexGuard` cannot span scheduler nodes, but a resource held by a
/// subtree root *can* — which is what lets the two-phase deploy
/// (`prepare_deploy` stages `flake.lock` uncommitted across the whole
@ -30,35 +48,12 @@ pub enum Resource {
/// decomposed into sub-nodes instead of one opaque node. Descendants of a
/// holder re-enter it through the crate's recursive lock, exactly like
/// [`Resource::Agent`].
///
/// Deliberately **not** declared by `Prebuild`: the window must stay off the
/// multi-minute toplevel build, which only *reads* the store. Holding a
/// hive-global cap-1 across it would serialize every agent's rebuild behind
/// every other's — which is why the meta preamble is its own `MetaSync`
/// node, and a sibling of `Prebuild` rather than its parent (a resource is
/// held across the holder's whole subtree).
MetaWindow,
}
impl NodeKind {
/// The resources this node must acquire to run — `(name, units)` — derived from
/// its kind + agent: a build slot for nix-heavy kinds
/// ([`NodeKind::needs_build_slot`]) and the agent lease for
/// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt
/// container ops (`Start` / `Stop`, fanned out by a lease-holding
/// `Reconcile`) hold no lease of their own — they re-enter the ancestor's
/// `Agent` lock through the crate's recursive re-entrancy. Meta-mutating
/// kinds ([`NodeKind::needs_meta_window`]) additionally take the global
/// [`Resource::MetaWindow`].
///
/// All of a node's resource edges are acquired **atomically**
/// (`try_acquire_all`) — a node never holds one resource while waiting on
/// another, so the multi-resource kinds (a `MetaLock` wants a build slot
/// *and* the meta window) cannot deadlock against each other.
pub fn resource_deps(&self) -> Vec<(Resource, u32)> {
let mut deps = Vec::new();
if self.needs_build_slot() {
deps.push((Resource::BuildSlot, 1));
}
if self.needs_lease() {
deps.push((Resource::Agent(self.agent().to_owned()), 1));
}
if self.needs_meta_window() {
deps.push((Resource::MetaWindow, 1));
}
deps
}
}

View file

@ -6,7 +6,7 @@
//! state, which needs an async `lifecycle::is_running` read that a pure/sync
//! template can't do. So these fns are async — they read each agent's state,
//! assemble a per-agent subgraph out of the shared pure primitives
//! (`templates::{node, rebuild_nodes}`), all declaring into ONE job
//! (`Job::node` + `templates::rebuild_nodes`), all declaring into ONE job
//! (independent per-agent roots, concurrent on their own leases).
//!
//! Dynamic shape rule: `stop`/`start` carry a head `SetWanted(w)` (durable
@ -25,7 +25,8 @@
use std::sync::Arc;
use super::model::{DagSpec, NodeKind};
use super::templates::{RebuildOpts, node, rebuild_nodes};
use super::resource::Resource;
use super::templates::{RebuildOpts, rebuild_nodes};
use super::{Job, Source, templates};
use crate::coordinator::Coordinator;
use crate::lifecycle;
@ -64,25 +65,34 @@ fn stop_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
// steps are its children (borrow the lease, run once it reaches `Finishing`,
// dep-ordered among themselves).
let a = || agent.to_owned();
let wanted = node(
b,
NodeKind::SetWanted {
let wanted = b
.node(NodeKind::SetWanted {
agent: a(),
up: false,
},
);
})
.needs(Resource::Agent(a()));
// Declaration order is dependency order: the quiesce steps come first so
// the `Reconcile` that waits on them can name them.
if graceful && running {
let signal = node(b, NodeKind::Signal { agent: a() }).part_of(wanted);
let drain = node(b, NodeKind::Drain { agent: a() })
let signal = b
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a()))
.part_of(wanted);
let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a()))
.part_of(wanted)
.after_ok(signal);
let _ = node(b, NodeKind::Reconcile { agent: a() })
let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.part_of(wanted)
.after_ok(drain);
} else {
let _ = node(b, NodeKind::Reconcile { agent: a() }).part_of(wanted);
let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.part_of(wanted);
}
}
@ -91,13 +101,12 @@ fn stop_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
/// current derivations), otherwise a plain `Reconcile` (which starts a down
/// agent and noops an already-running one).
fn start_chain(b: &Job, agent: &str, running: bool, stale: bool) {
let wanted = node(
b,
NodeKind::SetWanted {
let wanted = b
.node(NodeKind::SetWanted {
agent: agent.to_owned(),
up: true,
},
);
})
.needs(Resource::Agent(agent.to_owned()));
if !running && stale {
// Rebuild subtree chained behind the `SetWanted` head. `MetaSync`,
// `Prebuild` + `Reconcile` are their own group roots (top-level, per
@ -112,13 +121,12 @@ fn start_chain(b: &Job, agent: &str, running: bool, stale: bool) {
Some(wanted),
);
} else {
let _ = node(
b,
NodeKind::Reconcile {
let _ = b
.node(NodeKind::Reconcile {
agent: agent.to_owned(),
},
)
.part_of(wanted);
})
.needs(Resource::Agent(agent.to_owned()))
.part_of(wanted);
}
}
@ -135,7 +143,9 @@ fn restart_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
let a = || agent.to_owned();
if !running {
// Nothing to bounce — a lone Reconcile converges to intent.
let _ = node(b, NodeKind::Reconcile { agent: a() });
let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()));
return;
}
// Running: mechanical stop then Reconcile. The first stop node is the group
@ -147,17 +157,31 @@ fn restart_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
// that step *is* the root, and the parent gate already orders it — a child
// must NOT dep on its own parent (dep-scope), so it takes no sibling edge.
if graceful {
let signal = node(b, NodeKind::Signal { agent: a() });
let drain = node(b, NodeKind::Drain { agent: a() }).part_of(signal);
let stop = node(b, NodeKind::StopForUpdate { agent: a() })
let signal = b
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a()));
let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal);
let stop = b
.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal)
.after_ok(drain);
let _ = node(b, NodeKind::Reconcile { agent: a() })
let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal)
.after_ok(stop);
} else {
let stop = node(b, NodeKind::StopForUpdate { agent: a() });
let _ = node(b, NodeKind::Reconcile { agent: a() }).part_of(stop);
let stop = b
.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()));
let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.part_of(stop);
}
}

View file

@ -1,7 +1,10 @@
//! DAG shape builders — every operation as a template over the shared
//! node primitives. Pure (no I/O); each node carries its own `agent` (there is
//! no DAG-level agent), stamped by the [`node`] helper along with the
//! resources that node's kind needs.
//! no DAG-level agent) and **declares its own resources** with `.needs(…)`
//! right where it is constructed, rather than having them derived from its
//! kind. Deriving made the requirement a property of the *kind*, so a kind that
//! happened to run under an ancestor already holding the resource could get
//! away with declaring nothing.
//!
//! ```text
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
@ -19,35 +22,14 @@
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
//! their per-agent shape depends on live running state (an async
//! `lifecycle::is_running` read), so `submit.rs` assembles them out of the
//! primitives this module exports ([`node`], [`rebuild_nodes`]).
//! primitives this module exports ([`rebuild_nodes`]) over `Job::node`.
use hive_jobq::TerminalState;
use super::model::{DagSpec, NodeKind, PermPayload, Source};
use super::resource::Resource;
use super::{Declare, Handle, Job};
/// Declare one node carrying `kind`, with the resources that kind needs.
///
/// The resource declaration is [`NodeKind::resource_deps`] applied at the
/// construction site — a build slot for nix-heavy kinds, the agent lease for
/// container-affecting ones, the global meta window for meta-mutating ones. A
/// node that declares a resource an ancestor already holds re-enters that
/// grant rather than taking a fresh unit, so declaring costs nothing.
///
/// The returned handle is where edges and grouping are declared, and is `Copy`
/// — naming a node as a dependency does not consume the ability to name it
/// again.
pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
// Read the resources off the kind before handing it over — the payload is
// moved into the node, not cloned for it.
let resources = kind.resource_deps();
let mut handle = b.node(kind);
for (name, count) in resources {
handle = handle.needs_units(name, count);
}
handle
}
/// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node
/// gated on every group-root in `roots`, and the failure node gated on *its*
/// elimination.
@ -56,13 +38,10 @@ pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
/// dropped — see [`hive_jobq::NodeRef::on_elimination_of`].
fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
let ok = roots.iter().fold(
node(
b,
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
},
),
b.node(NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
}),
hive_jobq::NodeRef::after_ok,
);
// The failure branch needs *both*: the ok branch being ruled out (that is the
@ -72,13 +51,10 @@ fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
// `Reconcile` is still bringing the container back up, so reporting straight
// off the elimination would announce the failure mid-recovery.
let _failed = roots.iter().fold(
node(
b,
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: false,
},
)
b.node(NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: false,
})
.on_elimination_of(ok),
hive_jobq::NodeRef::after_any,
);
@ -95,14 +71,12 @@ fn resolve_approval_tails(b: &Job, approval_id: i64, root: Handle<'_>) {
TerminalState::Failed,
TerminalState::Cancelled,
] {
let _ = node(
b,
NodeKind::ResolveApproval {
let _ = b
.node(NodeKind::ResolveApproval {
approval_id,
outcome,
},
)
.on_outcome(root, &[outcome]);
})
.on_outcome(root, &[outcome]);
}
}
@ -181,32 +155,59 @@ pub(crate) fn rebuild_nodes<'a>(
let a = || agent.to_owned();
let RebuildOpts { relock, graceful } = opts;
let mut meta_sync = node(b, NodeKind::MetaSync { agent: a(), relock });
let mut meta_sync = b
.node(NodeKind::MetaSync { agent: a(), relock })
.needs(Resource::MetaWindow);
if let Some(after) = after {
meta_sync = meta_sync.after_ok(after);
}
let prebuild = node(b, NodeKind::Prebuild { agent: a() }).after_ok(meta_sync);
let prebuild = b
.node(NodeKind::Prebuild { agent: a() })
.needs(Resource::BuildSlot)
.after_ok(meta_sync);
// The stop root hangs off `Prebuild` and owns the agent lease for
// everything below it. `StopForUpdate` parents the swap pair either way.
let stop_for_update = if graceful {
let signal = node(b, NodeKind::Signal { agent: a() }).part_of(prebuild);
let signal = b
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a()))
.part_of(prebuild);
// `Drain` is a *child* of `Signal`, so the parent gate already orders
// it — a child must not dep on its own parent (dep-scope).
let drain = node(b, NodeKind::Drain { agent: a() }).part_of(signal);
node(b, NodeKind::StopForUpdate { agent: a() })
let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal);
b.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal)
.after_ok(drain)
} else {
node(b, NodeKind::StopForUpdate { agent: a() }).part_of(prebuild)
b.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(prebuild)
};
let swap = node(b, NodeKind::Swap { agent: a() }).part_of(stop_for_update);
let _post_swap = node(b, NodeKind::PostSwap { agent: a() })
let swap = b
.node(NodeKind::Swap { agent: a() })
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.part_of(stop_for_update);
// `PostSwap` declares the lease it actually runs under. It is a child of
// `StopForUpdate`, which holds it, so this is a re-entrant borrow — no
// second unit, no deadlock. Declaring it is what stops the requirement
// being true only of this one DAG shape.
let _post_swap = b
.node(NodeKind::PostSwap { agent: a() })
.needs(Resource::Agent(a()))
.part_of(stop_for_update)
.after_ok(swap);
let reconcile = node(b, NodeKind::Reconcile { agent: a() }).after_any(prebuild);
let reconcile = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.after_any(prebuild);
RebuildRoots {
meta_sync,
@ -251,15 +252,14 @@ pub(crate) fn deploy_rebuild_nodes(agent: &str, approval_id: i64) -> Declare {
},
None,
);
let _finalize = node(
b,
NodeKind::FinalizeDeploy {
let _finalize = b
.node(NodeKind::FinalizeDeploy {
agent: agent.clone(),
approval_id,
},
)
.after_ok(roots.prebuild)
.after_ok(roots.reconcile);
})
.needs(Resource::MetaWindow)
.after_ok(roots.prebuild)
.after_ok(roots.reconcile);
})
}
@ -335,39 +335,40 @@ pub fn approval_deploy(
reason,
declare: Box::new(move |b: &Job| {
let a = || agent.clone();
let window = node(
b,
NodeKind::DeployWindow {
// The window is the widest holder in the tree: it brackets a nix
// build (`BuildSlot`), takes the container down across the swap
// (`Agent`), and serialises the meta mutation its subtree performs
// (`MetaWindow`). All three are held for its whole subtree, which
// is what lets the appended rebuild's `MetaSync` and the
// `FinalizeDeploy` re-enter rather than contend.
let window = b
.node(NodeKind::DeployWindow {
agent: a(),
approval_id,
},
);
let verify = node(
b,
NodeKind::MergeVerify {
})
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.needs(Resource::MetaWindow);
let verify = b
.node(NodeKind::MergeVerify {
agent: a(),
approval_id,
},
)
.part_of(window);
let apply = node(
b,
NodeKind::DeployApply {
})
.part_of(window);
let apply = b
.node(NodeKind::DeployApply {
agent: a(),
approval_id,
},
)
.part_of(window)
.after_ok(verify);
let _tail = node(
b,
NodeKind::DeployTail {
})
.part_of(window)
.after_ok(verify);
let _tail = b
.node(NodeKind::DeployTail {
agent: a(),
approval_id,
},
)
.part_of(window)
.after_any(apply);
})
.part_of(window)
.after_any(apply);
resolve_approval_tails(b, approval_id, window);
}),
@ -390,7 +391,9 @@ pub fn reconcile_only(
source,
reason,
declare: Box::new(move |b: &Job| {
let _reconcile = node(b, NodeKind::Reconcile { agent });
// Name the lease before the agent string is moved into the kind.
let lease = Resource::Agent(agent.clone());
let _reconcile = b.node(NodeKind::Reconcile { agent }).needs(lease);
}),
}
}
@ -413,10 +416,21 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec<impl FnOn
reason,
declare: Box::new(move |b: &Job| {
let a = || agent.clone();
let provision = node(b, NodeKind::Provision { agent: a() });
let create = node(b, NodeKind::Create { agent: a() }).part_of(provision);
let dropin = node(b, NodeKind::WriteDropin { agent: a() }).part_of(create);
let _reconcile = node(b, NodeKind::Reconcile { agent: a() })
let provision = b
.node(NodeKind::Provision { agent: a() })
.needs(Resource::MetaWindow);
let create = b
.node(NodeKind::Create { agent: a() })
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.part_of(provision);
let dropin = b
.node(NodeKind::WriteDropin { agent: a() })
.needs(Resource::Agent(a()))
.part_of(create);
let _reconcile = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.part_of(create)
.after_ok(dropin);
@ -441,13 +455,12 @@ pub fn perm_change(
source,
reason,
declare: Box::new(move |b: &Job| {
let write = node(
b,
NodeKind::WritePermFile {
let write = b
.node(NodeKind::WritePermFile {
agent: agent.clone(),
payload,
},
);
})
.needs(Resource::MetaWindow);
let roots = rebuild_nodes(
b,
&agent,
@ -486,14 +499,14 @@ pub fn meta_update(
source,
reason,
declare: Box::new(move |b: &Job| {
let lock = node(
b,
NodeKind::MetaLock {
let lock = b
.node(NodeKind::MetaLock {
sweep: false,
fanout: None,
inputs,
},
);
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
// The bump itself has no side effect, so an operator-driven one ends
// at the `MetaLock`; an approval-driven one still has its row to
// resolve and gets the per-outcome tails edged onto that single
@ -524,7 +537,9 @@ pub fn reparent(
source,
reason,
declare: Box::new(move |b: &Job| {
let _reparent = node(b, NodeKind::Reparent { moves });
let _reparent = b
.node(NodeKind::Reparent { moves })
.needs(Resource::MetaWindow);
}),
}
}

View file

@ -102,6 +102,28 @@ fn settle_rebuild_tail(q: &JobQueue, agent: &str, expect_ok: bool) {
q.complete_node(tail.node_id, Ok(()));
}
/// The resources a node **declared**, read off its graph edges.
///
/// The declaration is the thing under test now that construction sites state
/// their own holdings: asking the `NodeKind` what it "should" need would just
/// re-run the derivation this module removed, and would pass even if the
/// construction site declared nothing.
fn declared_resources(q: &JobQueue, node_id: hive_jobq::NodeId) -> Vec<Resource> {
let inner = q.lock();
inner
.sched
.graph()
.node(node_id)
.expect("node exists")
.deps
.iter()
.filter_map(|dep| match dep {
hive_jobq::Dep::Resource { name, .. } => Some(name.clone()),
hive_jobq::Dep::Node { .. } => None,
})
.collect()
}
fn state_of(q: &JobQueue, dag_id: u64) -> State {
// A DAG whose nodes have all settled `Done` or `Skipped` drops out of the
// snapshot — absence is the completion signal, so map it to `Done`.
@ -708,6 +730,125 @@ fn offline_agents_skip_mechanical_nodes_but_keep_reconcile() {
);
}
#[test]
fn a_fanned_out_start_declares_the_lease_and_re_enters_its_reconciles_grant() {
// `Start` / `Stop` / `PostSwap` were lease-exempt *as kinds*, which was only
// safe because every construction site fans them out from inside a
// lease-holding ancestor. Now they declare the lease themselves.
//
// The contract says that costs nothing — a descendant re-enters the
// ancestor's grant instead of taking a fresh unit. That is exactly the sort
// of claim that is true until a node is used from a second site, so it is
// pinned here rather than argued: the fanned-out `Start` must (a) actually
// carry the declaration, (b) still run under its parent's grant, and
// (c) not have consumed a second unit of a cap-1 lease.
let q = JobQueue::new(4);
let id = submit(
&q,
templates::reconcile_only("agent-a", Source::Manual, "converge".to_owned()),
);
// A competing DAG on the same agent, to prove the lease is genuinely held
// (and held *once*) across the fan-out.
let rival = submit(
&q,
templates::reconcile_only("agent-a", Source::Manual, "rival".to_owned()),
);
let reconcile = claim_one(&q);
assert_eq!(reconcile.dag_id, id);
assert_eq!(reconcile.kind.as_str(), "reconcile");
// What `run_reconcile` does on observing a down container with wanted=Up.
q.append_subgraph(
id,
Box::new(|b: &Job| {
let kind = NodeKind::Start {
agent: "agent-a".to_owned(),
};
let lease = Resource::Agent(kind.agent().to_owned());
let _ = b.node(kind).needs(lease);
}),
reconcile.node_id,
);
q.complete_node(reconcile.node_id, Ok(()));
// (a) + (b): the child runs, under the parent that parked in `Finishing`.
let start = claim_one(&q);
assert_eq!(start.kind.as_str(), "start");
assert_eq!(
declared_resources(&q, start.node_id),
vec![Resource::Agent("agent-a".to_owned())],
"a fanned-out Start declares the lease it runs under"
);
// (c): one unit, not two. `claim_one` above already asserted the rival did
// not come back in the same pass; make the reason explicit.
assert!(
q.claim_ready().is_empty(),
"the rival DAG's Reconcile must still be blocked — the appended Start \
borrowed the grant rather than acquiring a second unit"
);
q.complete_node(start.node_id, Ok(()));
// Subtree terminal → the grant releases and the rival finally runs.
let rival_reconcile = claim_one(&q);
assert_eq!(rival_reconcile.dag_id, rival);
q.complete_node(rival_reconcile.node_id, Ok(()));
assert_eq!(state_of(&q, id), State::Done);
assert_eq!(state_of(&q, rival), State::Done);
}
#[test]
fn boot_sweep_nodes_declare_their_own_resources() {
// Regression, and the reason it needs its own test: `workers::auto_update`
// is the only place job nodes are constructed *outside* `job_queue/`, so
// nothing in this module covered it. When resource derivation moved to the
// construction sites, this path was missed and both kinds silently declared
// nothing — dropping the agent lease a boot `Reconcile` needs to not race
// another DAG's container ops, and letting the sweep `MetaLock` land its
// meta commit inside another node's staged deploy window. Nothing failed to
// compile; only an exhaustive caller list would have caught it.
let q = JobQueue::new(4);
let _id = submit(
&q,
DagSpec {
source: Source::AutoUpdate,
reason: "boot".to_owned(),
declare: Box::new(|b: &Job| {
crate::workers::auto_update::boot_nodes(
b,
true,
vec!["stale-agent".to_owned()],
vec!["drifted-agent".to_owned()],
);
}),
},
);
// Both are independent roots on disjoint resources, so both start at once.
let claims = q.claim_ready();
let by_kind = |kind: &str| {
claims
.iter()
.find(|c| c.kind.as_str() == kind)
.unwrap_or_else(|| panic!("no {kind} claim in {claims:?}"))
};
let mut lock = declared_resources(&q, by_kind("meta_lock").node_id);
lock.sort_by_key(|r| format!("{r:?}"));
assert_eq!(
lock,
vec![Resource::BuildSlot, Resource::MetaWindow],
"the sweep MetaLock runs a nix lock bump and commits to meta"
);
assert_eq!(
declared_resources(&q, by_kind("reconcile").node_id),
vec![Resource::Agent("drifted-agent".to_owned())],
"a boot Reconcile touches the container, so it holds that agent's lease"
);
}
#[test]
fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
// The startup-sweep mechanism: a `MetaLock` emitter grows one rebuild
@ -718,14 +859,11 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
source: Source::AutoUpdate,
reason: "sweep".to_owned(),
declare: Box::new(|b: &Job| {
let _lock = templates::node(
b,
NodeKind::MetaLock {
sweep: true,
fanout: None,
inputs: Vec::new(),
},
);
let _lock = b.node(NodeKind::MetaLock {
sweep: true,
fanout: None,
inputs: Vec::new(),
});
}),
};
let id = submit(&q, spec);
@ -1594,12 +1732,12 @@ fn reparent_shape_is_a_lone_agentless_meta_window_node() {
let c = claim_one(&q);
assert_eq!(c.kind.as_str(), "reparent");
assert_eq!(c.agent, "", "Reparent is agentless — no per-agent lease");
assert!(
c.kind.needs_meta_window(),
"a topology commit must hold the same MetaWindow as WritePermFile"
assert_eq!(
declared_resources(&q, c.node_id),
vec![Resource::MetaWindow],
"a topology commit must declare the same MetaWindow as WritePermFile, \
and nothing else no lease (agentless), no build slot (no nix work)"
);
assert!(!c.kind.needs_lease());
assert!(!c.kind.needs_build_slot());
q.complete_node(c.node_id, Ok(()));
assert_eq!(state_of(&q, id), State::Done);
}

View file

@ -34,7 +34,7 @@ static META_LOCK: Mutex<()> = Mutex::const_new(());
// this module. `META_LOCK` above serializes individual git ops but cannot
// keep another op out of that staged window; that window is owned by the
// job queue instead, as `Resource::MetaWindow`, declared by every
// meta-mutating node kind (`NodeKind::needs_meta_window`). A resource can
// meta-mutating node kind (it declares `Resource::MetaWindow`). A resource can
// be held by a subtree root across its children, which a `MutexGuard`
// (bounded by one executor fn) cannot — that's what lets the deploy be
// modelled as sub-nodes rather than one opaque node.

View file

@ -318,6 +318,49 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
Ok(())
}
/// The boot DAG's node declarations, split out of [`submit_boot_tree`] so they
/// can be exercised without a live [`Coordinator`].
///
/// That split is not cosmetic: this path constructs nodes outside
/// `job_queue/`, so it is the one place a resource declaration can be forgotten
/// without any in-module test noticing. It has happened once already — the
/// sweep `MetaLock` and the boot `Reconcile`s silently declared nothing when
/// kind-derived resources were removed, which drops the agent lease a boot
/// reconcile needs to not race another DAG's container ops.
pub(crate) fn boot_nodes(
b: &crate::job_queue::Job,
any_stale: bool,
fanout: Vec<String>,
drifted: Vec<String>,
) {
use crate::job_queue::NodeKind;
use crate::job_queue::resource::Resource;
// Sweep whenever ANY marker is stale — even when every stale agent is
// wanted-offline: the hyperhive lock bump must land now so their later
// start-upgrade rebuilds build against it. No stale agents ⇒ no MetaLock
// ⇒ no meta commit on a no-change boot. The `fanout` list rides the
// MetaLock into `run_meta_lock`, which appends the rebuild subgraphs.
if any_stale {
let _ = b
.node(NodeKind::MetaLock {
sweep: true,
fanout: Some(fanout),
// A sweep bumps `hyperhive` alone (`lock_update_hyperhive`),
// so it names no inputs.
inputs: Vec::new(),
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
}
// One boot Reconcile per drifted agent — independent roots.
for name in drifted {
// Name the lease before the agent string moves into the kind.
let lease = Resource::Agent(name.clone());
let _ = b.node(NodeKind::Reconcile { agent: name }).needs(lease);
}
}
/// Submit this boot's work as **one DAG** (no anchor node, no per-agent
/// child DAGs). Node 0 is the sweep `MetaLock` (only when
/// something is stale) — its executor bumps the hyperhive lock, then grows
@ -334,7 +377,7 @@ fn submit_boot_tree(
n_deferred: usize,
n_skipped: usize,
) {
use crate::job_queue::{DagSpec, NodeKind, Source, templates};
use crate::job_queue::{DagSpec, Source};
// Fully-quiet boot (nothing stale, nothing drifted) submits nothing.
if !any_stale && drifted.is_empty() {
@ -348,29 +391,8 @@ fn submit_boot_tree(
n_skipped,
);
let declare: crate::job_queue::Declare = Box::new(move |b| {
// Sweep whenever ANY marker is stale — even when every stale agent is
// wanted-offline: the hyperhive lock bump must land now so their later
// start-upgrade rebuilds build against it. No stale agents ⇒ no MetaLock
// ⇒ no meta commit on a no-change boot. The `fanout` list rides the
// MetaLock into `run_meta_lock`, which appends the rebuild subgraphs.
if any_stale {
let _ = templates::node(
b,
NodeKind::MetaLock {
sweep: true,
fanout: Some(fanout),
// A sweep bumps `hyperhive` alone (`lock_update_hyperhive`),
// so it names no inputs.
inputs: Vec::new(),
},
);
}
// One boot Reconcile per drifted agent — independent roots.
for name in drifted {
let _ = templates::node(b, NodeKind::Reconcile { agent: name });
}
});
let declare: crate::job_queue::Declare =
Box::new(move |b| boot_nodes(b, any_stale, fanout, drifted));
let spec = DagSpec {
// The sweep's own rebuild subgraphs emit their `Rebuilt` events as they