vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: fix the boot sweep's lost declarations, drop the node wrapper

Browse source 
Two review findings on the resources-at-construction change.

argus: `workers::auto_update`'s boot sweep constructs nodes through
`templates::node` too, and it was not converted. With the kind-derived
declaration gone, its sweep `MetaLock` and its per-agent `Reconcile`
silently declared no resources at all — so a boot reconcile no longer
held the agent lease and could race another DAG's container ops, and the
sweep's meta commit could land inside another node's staged deploy
window. Nothing failed to compile: removing an implicit behaviour from a
helper is invisible at every call site that relied on it.

The declarations now live in a pure `boot_nodes`, split out of
`submit_boot_tree` so they can be exercised without a `Coordinator`.
That path is the only place job nodes are built outside `job_queue/`,
which is exactly why it had no coverage; `boot_sweep_nodes_declare_
their_own_resources` closes that, asserting against declared graph edges
rather than against the kind.

mara: `templates::node` is a redundant redirect now that it no longer
derives resources — deleted, and its 43 call sites use `Job::node`
directly. The reasoning it documented moved to the module docs of
`templates.rs` and `resource.rs`, which is where it stays true.
This commit is contained in:
atlas 2026-08-02 16:22:56 +02:00 • committed by mara
commit 58a9f218f2
6 changed files with 231 additions and 180 deletions
Download patch file Download diff file Expand all files Collapse all files

2
hive-c0re/src/job_queue/exec.rs
View file

@ -424,7 +424,7 @@ async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
// node rather than to the fact that a `Reconcile` happens to fan it out. // node rather than to the fact that a `Reconcile` happens to fan it out.
let lease = Resource::Agent(kind.agent().to_owned()); let lease = Resource::Agent(kind.agent().to_owned());
vec![Box::new(move |b: &super::Job| { vec![Box::new(move |b: &super::Job| {
let _ = super::templates::node(b, kind).needs(lease); let _ = b.node(kind).needs(lease);
}) as Declare] }) as Declare]
}; };
let append_subgraph = match reconcile_action(wanted, running) { let append_subgraph = match reconcile_action(wanted, running) {

8
hive-c0re/src/job_queue/resource.rs
View file

@ -4,10 +4,10 @@
//! payload `N`; here `R` is [`Resource`] and `N` is [`NodeKind`] directly (each //! payload `N`; here `R` is [`Resource`] and `N` is [`NodeKind`] directly (each
//! variant carries the agent it targets). //! variant carries the agent it targets).
//! //!
//! **A node's resources are declared where the node is constructed**, not //! **A node's resources are declared where the node is constructed** with
//! derived from its kind — see `templates::node`. Deriving them made the //! `.needs(…)`, not derived from its kind. Deriving them made the requirement a
//! requirement a property of the *kind*, which let a kind that happened to run //! property of the *kind*, which let a kind that happened to run under a
//! under a holding ancestor declare nothing at all. //! holding ancestor declare nothing at all.
//! //!
//! All of a node's resource edges are acquired **atomically** //! All of a node's resource edges are acquired **atomically**
//! (`try_acquire_all`) — a node never holds one resource while waiting on //! (`try_acquire_all`) — a node never holds one resource while waiting on

72
hive-c0re/src/job_queue/submit.rs
View file

@ -6,7 +6,7 @@
//! state, which needs an async `lifecycle::is_running` read that a pure/sync //! state, which needs an async `lifecycle::is_running` read that a pure/sync
//! template can't do. So these fns are async — they read each agent's state, //! template can't do. So these fns are async — they read each agent's state,
//! assemble a per-agent subgraph out of the shared pure primitives //! assemble a per-agent subgraph out of the shared pure primitives
//! (`templates::{node, rebuild_nodes}`), all declaring into ONE job //! (`Job::node` + `templates::rebuild_nodes`), all declaring into ONE job
//! (independent per-agent roots, concurrent on their own leases). //! (independent per-agent roots, concurrent on their own leases).
//! //!
//! Dynamic shape rule: `stop`/`start` carry a head `SetWanted(w)` (durable //! Dynamic shape rule: `stop`/`start` carry a head `SetWanted(w)` (durable
@ -26,7 +26,7 @@ use std::sync::Arc;
use super::model::{DagSpec, NodeKind}; use super::model::{DagSpec, NodeKind};
use super::resource::Resource; use super::resource::Resource;
use super::templates::{RebuildOpts, node, rebuild_nodes}; use super::templates::{RebuildOpts, rebuild_nodes};
use super::{Job, Source, templates}; use super::{Job, Source, templates};
use crate::coordinator::Coordinator; use crate::coordinator::Coordinator;
use crate::lifecycle; use crate::lifecycle;
@ -65,30 +65,32 @@ fn stop_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
// steps are its children (borrow the lease, run once it reaches `Finishing`, // steps are its children (borrow the lease, run once it reaches `Finishing`,
// dep-ordered among themselves). // dep-ordered among themselves).
let a = || agent.to_owned(); let a = || agent.to_owned();
let wanted = node( let wanted = b
b, .node(NodeKind::SetWanted {
NodeKind::SetWanted {
agent: a(), agent: a(),
up: false, up: false,
}, })
) .needs(Resource::Agent(a()));
.needs(Resource::Agent(a()));
// Declaration order is dependency order: the quiesce steps come first so // Declaration order is dependency order: the quiesce steps come first so
// the `Reconcile` that waits on them can name them. // the `Reconcile` that waits on them can name them.
if graceful && running { if graceful && running {
let signal = node(b, NodeKind::Signal { agent: a() }) let signal = b
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(wanted); .part_of(wanted);
let drain = node(b, NodeKind::Drain { agent: a() }) let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(wanted) .part_of(wanted)
.after_ok(signal); .after_ok(signal);
let _ = node(b, NodeKind::Reconcile { agent: a() }) let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(wanted) .part_of(wanted)
.after_ok(drain); .after_ok(drain);
} else { } else {
let _ = node(b, NodeKind::Reconcile { agent: a() }) let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(wanted); .part_of(wanted);
} }
@ -99,14 +101,12 @@ fn stop_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
/// current derivations), otherwise a plain `Reconcile` (which starts a down /// current derivations), otherwise a plain `Reconcile` (which starts a down
/// agent and noops an already-running one). /// agent and noops an already-running one).
fn start_chain(b: &Job, agent: &str, running: bool, stale: bool) { fn start_chain(b: &Job, agent: &str, running: bool, stale: bool) {
let wanted = node( let wanted = b
b, .node(NodeKind::SetWanted {
NodeKind::SetWanted {
agent: agent.to_owned(), agent: agent.to_owned(),
up: true, up: true,
}, })
) .needs(Resource::Agent(agent.to_owned()));
.needs(Resource::Agent(agent.to_owned()));
if !running && stale { if !running && stale {
// Rebuild subtree chained behind the `SetWanted` head. `MetaSync`, // Rebuild subtree chained behind the `SetWanted` head. `MetaSync`,
// `Prebuild` + `Reconcile` are their own group roots (top-level, per // `Prebuild` + `Reconcile` are their own group roots (top-level, per
@ -121,14 +121,12 @@ fn start_chain(b: &Job, agent: &str, running: bool, stale: bool) {
Some(wanted), Some(wanted),
); );
} else { } else {
let _ = node( let _ = b
b, .node(NodeKind::Reconcile {
NodeKind::Reconcile {
agent: agent.to_owned(), agent: agent.to_owned(),
}, })
) .needs(Resource::Agent(agent.to_owned()))
.needs(Resource::Agent(agent.to_owned())) .part_of(wanted);
.part_of(wanted);
} }
} }
@ -145,7 +143,9 @@ fn restart_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
let a = || agent.to_owned(); let a = || agent.to_owned();
if !running { if !running {
// Nothing to bounce — a lone Reconcile converges to intent. // Nothing to bounce — a lone Reconcile converges to intent.
let _ = node(b, NodeKind::Reconcile { agent: a() }).needs(Resource::Agent(a())); let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()));
return; return;
} }
// Running: mechanical stop then Reconcile. The first stop node is the group // Running: mechanical stop then Reconcile. The first stop node is the group
@ -157,21 +157,29 @@ fn restart_chain(b: &Job, agent: &str, graceful: bool, running: bool) {
// that step *is* the root, and the parent gate already orders it — a child // that step *is* the root, and the parent gate already orders it — a child
// must NOT dep on its own parent (dep-scope), so it takes no sibling edge. // must NOT dep on its own parent (dep-scope), so it takes no sibling edge.
if graceful { if graceful {
let signal = node(b, NodeKind::Signal { agent: a() }).needs(Resource::Agent(a())); let signal = b
let drain = node(b, NodeKind::Drain { agent: a() }) .node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a()));
let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(signal); .part_of(signal);
let stop = node(b, NodeKind::StopForUpdate { agent: a() }) let stop = b
.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(signal) .part_of(signal)
.after_ok(drain); .after_ok(drain);
let _ = node(b, NodeKind::Reconcile { agent: a() }) let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(signal) .part_of(signal)
.after_ok(stop); .after_ok(stop);
} else { } else {
let stop = node(b, NodeKind::StopForUpdate { agent: a() }).needs(Resource::Agent(a())); let stop = b
let _ = node(b, NodeKind::Reconcile { agent: a() }) .node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()));
let _ = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(stop); .part_of(stop);
} }

193
hive-c0re/src/job_queue/templates.rs
View file

@ -1,7 +1,10 @@
//! DAG shape builders — every operation as a template over the shared //! DAG shape builders — every operation as a template over the shared
//! node primitives. Pure (no I/O); each node carries its own `agent` (there is //! node primitives. Pure (no I/O); each node carries its own `agent` (there is
//! no DAG-level agent), stamped by the [`node`] helper along with the //! no DAG-level agent) and **declares its own resources** with `.needs(…)`
//! resources that node's kind needs. //! right where it is constructed, rather than having them derived from its
//! kind. Deriving made the requirement a property of the *kind*, so a kind that
//! happened to run under an ancestor already holding the resource could get
//! away with declaring nothing.
//! //!
//! ```text //! ```text
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a) //! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
@ -19,7 +22,7 @@
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here: //! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
//! their per-agent shape depends on live running state (an async //! their per-agent shape depends on live running state (an async
//! `lifecycle::is_running` read), so `submit.rs` assembles them out of the //! `lifecycle::is_running` read), so `submit.rs` assembles them out of the
//! primitives this module exports ([`node`], [`rebuild_nodes`]). //! primitives this module exports ([`rebuild_nodes`]) over `Job::node`.
use hive_jobq::TerminalState; use hive_jobq::TerminalState;
@ -27,28 +30,6 @@ use super::model::{DagSpec, NodeKind, PermPayload, Source};
use super::resource::Resource; use super::resource::Resource;
use super::{Declare, Handle, Job}; use super::{Declare, Handle, Job};
/// Declare one node carrying `kind`. **Resources are not derived here** — the
/// construction site says what the node holds, with `.needs(…)`.
///
/// That is the point rather than an omission. Deriving `(name, units)` from the
/// kind made the declaration a property of the *kind*, so a kind that happened
/// to run under an ancestor holding the resource could get away with declaring
/// nothing — which is precisely how `Start` / `Stop` / `PostSwap` ended up
/// lease-exempt: one construction site fans them out from inside a
/// lease-holding `Reconcile`. The requirement is a property of the node, not of
/// the one DAG shape it is used in today.
///
/// Declaring a resource an ancestor already holds is free: a descendant
/// re-enters that grant through the crate's recursive lock rather than taking a
/// fresh unit.
///
/// The returned handle is where edges and grouping are declared, and is `Copy`
/// — naming a node as a dependency does not consume the ability to name it
/// again.
pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
b.node(kind)
}
/// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node /// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node
/// gated on every group-root in `roots`, and the failure node gated on *its* /// gated on every group-root in `roots`, and the failure node gated on *its*
/// elimination. /// elimination.
@ -57,13 +38,10 @@ pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
/// dropped — see [`hive_jobq::NodeRef::on_elimination_of`]. /// dropped — see [`hive_jobq::NodeRef::on_elimination_of`].
fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) { fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
let ok = roots.iter().fold( let ok = roots.iter().fold(
node( b.node(NodeKind::EmitRebuilt {
b, agent: agent.to_owned(),
NodeKind::EmitRebuilt { ok: true,
agent: agent.to_owned(), }),
ok: true,
},
),
hive_jobq::NodeRef::after_ok, hive_jobq::NodeRef::after_ok,
); );
// The failure branch needs *both*: the ok branch being ruled out (that is the // The failure branch needs *both*: the ok branch being ruled out (that is the
@ -73,13 +51,10 @@ fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
// `Reconcile` is still bringing the container back up, so reporting straight // `Reconcile` is still bringing the container back up, so reporting straight
// off the elimination would announce the failure mid-recovery. // off the elimination would announce the failure mid-recovery.
let _failed = roots.iter().fold( let _failed = roots.iter().fold(
node( b.node(NodeKind::EmitRebuilt {
b, agent: agent.to_owned(),
NodeKind::EmitRebuilt { ok: false,
agent: agent.to_owned(), })
ok: false,
},
)
.on_elimination_of(ok), .on_elimination_of(ok),
hive_jobq::NodeRef::after_any, hive_jobq::NodeRef::after_any,
); );
@ -96,14 +71,12 @@ fn resolve_approval_tails(b: &Job, approval_id: i64, root: Handle<'_>) {
TerminalState::Failed, TerminalState::Failed,
TerminalState::Cancelled, TerminalState::Cancelled,
] { ] {
let _ = node( let _ = b
b, .node(NodeKind::ResolveApproval {
NodeKind::ResolveApproval {
approval_id, approval_id,
outcome, outcome,
}, })
) .on_outcome(root, &[outcome]);
.on_outcome(root, &[outcome]);
} }
} }
@ -182,37 +155,42 @@ pub(crate) fn rebuild_nodes<'a>(
let a = || agent.to_owned(); let a = || agent.to_owned();
let RebuildOpts { relock, graceful } = opts; let RebuildOpts { relock, graceful } = opts;
let mut meta_sync = let mut meta_sync = b
node(b, NodeKind::MetaSync { agent: a(), relock }).needs(Resource::MetaWindow); .node(NodeKind::MetaSync { agent: a(), relock })
.needs(Resource::MetaWindow);
if let Some(after) = after { if let Some(after) = after {
meta_sync = meta_sync.after_ok(after); meta_sync = meta_sync.after_ok(after);
} }
let prebuild = node(b, NodeKind::Prebuild { agent: a() }) let prebuild = b
.node(NodeKind::Prebuild { agent: a() })
.needs(Resource::BuildSlot) .needs(Resource::BuildSlot)
.after_ok(meta_sync); .after_ok(meta_sync);
// The stop root hangs off `Prebuild` and owns the agent lease for // The stop root hangs off `Prebuild` and owns the agent lease for
// everything below it. `StopForUpdate` parents the swap pair either way. // everything below it. `StopForUpdate` parents the swap pair either way.
let stop_for_update = if graceful { let stop_for_update = if graceful {
let signal = node(b, NodeKind::Signal { agent: a() }) let signal = b
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(prebuild); .part_of(prebuild);
// `Drain` is a *child* of `Signal`, so the parent gate already orders // `Drain` is a *child* of `Signal`, so the parent gate already orders
// it — a child must not dep on its own parent (dep-scope). // it — a child must not dep on its own parent (dep-scope).
let drain = node(b, NodeKind::Drain { agent: a() }) let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(signal); .part_of(signal);
node(b, NodeKind::StopForUpdate { agent: a() }) b.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(signal) .part_of(signal)
.after_ok(drain) .after_ok(drain)
} else { } else {
node(b, NodeKind::StopForUpdate { agent: a() }) b.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(prebuild) .part_of(prebuild)
}; };
let swap = node(b, NodeKind::Swap { agent: a() }) let swap = b
.node(NodeKind::Swap { agent: a() })
.needs(Resource::BuildSlot) .needs(Resource::BuildSlot)
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(stop_for_update); .part_of(stop_for_update);
@ -220,12 +198,14 @@ pub(crate) fn rebuild_nodes<'a>(
// `StopForUpdate`, which holds it, so this is a re-entrant borrow — no // `StopForUpdate`, which holds it, so this is a re-entrant borrow — no
// second unit, no deadlock. Declaring it is what stops the requirement // second unit, no deadlock. Declaring it is what stops the requirement
// being true only of this one DAG shape. // being true only of this one DAG shape.
let _post_swap = node(b, NodeKind::PostSwap { agent: a() }) let _post_swap = b
.node(NodeKind::PostSwap { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(stop_for_update) .part_of(stop_for_update)
.after_ok(swap); .after_ok(swap);
let reconcile = node(b, NodeKind::Reconcile { agent: a() }) let reconcile = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.after_any(prebuild); .after_any(prebuild);
@ -272,16 +252,14 @@ pub(crate) fn deploy_rebuild_nodes(agent: &str, approval_id: i64) -> Declare {
}, },
None, None,
); );
let _finalize = node( let _finalize = b
b, .node(NodeKind::FinalizeDeploy {
NodeKind::FinalizeDeploy {
agent: agent.clone(), agent: agent.clone(),
approval_id, approval_id,
}, })
) .needs(Resource::MetaWindow)
.needs(Resource::MetaWindow) .after_ok(roots.prebuild)
.after_ok(roots.prebuild) .after_ok(roots.reconcile);
.after_ok(roots.reconcile);
}) })
} }
@ -363,42 +341,34 @@ pub fn approval_deploy(
// (`MetaWindow`). All three are held for its whole subtree, which // (`MetaWindow`). All three are held for its whole subtree, which
// is what lets the appended rebuild's `MetaSync` and the // is what lets the appended rebuild's `MetaSync` and the
// `FinalizeDeploy` re-enter rather than contend. // `FinalizeDeploy` re-enter rather than contend.
let window = node( let window = b
b, .node(NodeKind::DeployWindow {
NodeKind::DeployWindow {
agent: a(), agent: a(),
approval_id, approval_id,
}, })
) .needs(Resource::BuildSlot)
.needs(Resource::BuildSlot) .needs(Resource::Agent(a()))
.needs(Resource::Agent(a())) .needs(Resource::MetaWindow);
.needs(Resource::MetaWindow); let verify = b
let verify = node( .node(NodeKind::MergeVerify {
b,
NodeKind::MergeVerify {
agent: a(), agent: a(),
approval_id, approval_id,
}, })
) .part_of(window);
.part_of(window); let apply = b
let apply = node( .node(NodeKind::DeployApply {
b,
NodeKind::DeployApply {
agent: a(), agent: a(),
approval_id, approval_id,
}, })
) .part_of(window)
.part_of(window) .after_ok(verify);
.after_ok(verify); let _tail = b
let _tail = node( .node(NodeKind::DeployTail {
b,
NodeKind::DeployTail {
agent: a(), agent: a(),
approval_id, approval_id,
}, })
) .part_of(window)
.part_of(window) .after_any(apply);
.after_any(apply);
resolve_approval_tails(b, approval_id, window); resolve_approval_tails(b, approval_id, window);
}), }),
@ -423,7 +393,7 @@ pub fn reconcile_only(
declare: Box::new(move |b: &Job| { declare: Box::new(move |b: &Job| {
// Name the lease before the agent string is moved into the kind. // Name the lease before the agent string is moved into the kind.
let lease = Resource::Agent(agent.clone()); let lease = Resource::Agent(agent.clone());
let _reconcile = node(b, NodeKind::Reconcile { agent }).needs(lease); let _reconcile = b.node(NodeKind::Reconcile { agent }).needs(lease);
}), }),
} }
} }
@ -446,15 +416,20 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec<impl FnOn
reason, reason,
declare: Box::new(move |b: &Job| { declare: Box::new(move |b: &Job| {
let a = || agent.clone(); let a = || agent.clone();
let provision = node(b, NodeKind::Provision { agent: a() }).needs(Resource::MetaWindow); let provision = b
let create = node(b, NodeKind::Create { agent: a() }) .node(NodeKind::Provision { agent: a() })
.needs(Resource::MetaWindow);
let create = b
.node(NodeKind::Create { agent: a() })
.needs(Resource::BuildSlot) .needs(Resource::BuildSlot)
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(provision); .part_of(provision);
let dropin = node(b, NodeKind::WriteDropin { agent: a() }) let dropin = b
.node(NodeKind::WriteDropin { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(create); .part_of(create);
let _reconcile = node(b, NodeKind::Reconcile { agent: a() }) let _reconcile = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a())) .needs(Resource::Agent(a()))
.part_of(create) .part_of(create)
.after_ok(dropin); .after_ok(dropin);
@ -480,14 +455,12 @@ pub fn perm_change(
source, source,
reason, reason,
declare: Box::new(move |b: &Job| { declare: Box::new(move |b: &Job| {
let write = node( let write = b
b, .node(NodeKind::WritePermFile {
NodeKind::WritePermFile {
agent: agent.clone(), agent: agent.clone(),
payload, payload,
}, })
) .needs(Resource::MetaWindow);
.needs(Resource::MetaWindow);
let roots = rebuild_nodes( let roots = rebuild_nodes(
b, b,
&agent, &agent,
@ -526,16 +499,14 @@ pub fn meta_update(
source, source,
reason, reason,
declare: Box::new(move |b: &Job| { declare: Box::new(move |b: &Job| {
let lock = node( let lock = b
b, .node(NodeKind::MetaLock {
NodeKind::MetaLock {
sweep: false, sweep: false,
fanout: None, fanout: None,
inputs, inputs,
}, })
) .needs(Resource::BuildSlot)
.needs(Resource::BuildSlot) .needs(Resource::MetaWindow);
.needs(Resource::MetaWindow);
// The bump itself has no side effect, so an operator-driven one ends // The bump itself has no side effect, so an operator-driven one ends
// at the `MetaLock`; an approval-driven one still has its row to // at the `MetaLock`; an approval-driven one still has its row to
// resolve and gets the per-outcome tails edged onto that single // resolve and gets the per-outcome tails edged onto that single
@ -566,7 +537,9 @@ pub fn reparent(
source, source,
reason, reason,
declare: Box::new(move |b: &Job| { declare: Box::new(move |b: &Job| {
let _reparent = node(b, NodeKind::Reparent { moves }).needs(Resource::MetaWindow); let _reparent = b
.node(NodeKind::Reparent { moves })
.needs(Resource::MetaWindow);
}), }),
} }
} }

66
hive-c0re/src/job_queue/tests.rs
View file

@ -766,7 +766,7 @@ fn a_fanned_out_start_declares_the_lease_and_re_enters_its_reconciles_grant() {
agent: "agent-a".to_owned(), agent: "agent-a".to_owned(),
}; };
let lease = Resource::Agent(kind.agent().to_owned()); let lease = Resource::Agent(kind.agent().to_owned());
let _ = templates::node(b, kind).needs(lease); let _ = b.node(kind).needs(lease);
}), }),
reconcile.node_id, reconcile.node_id,
); );
@ -798,6 +798,57 @@ fn a_fanned_out_start_declares_the_lease_and_re_enters_its_reconciles_grant() {
assert_eq!(state_of(&q, rival), State::Done); assert_eq!(state_of(&q, rival), State::Done);
} }
#[test]
fn boot_sweep_nodes_declare_their_own_resources() {
// Regression, and the reason it needs its own test: `workers::auto_update`
// is the only place job nodes are constructed *outside* `job_queue/`, so
// nothing in this module covered it. When resource derivation moved to the
// construction sites, this path was missed and both kinds silently declared
// nothing — dropping the agent lease a boot `Reconcile` needs to not race
// another DAG's container ops, and letting the sweep `MetaLock` land its
// meta commit inside another node's staged deploy window. Nothing failed to
// compile; only an exhaustive caller list would have caught it.
let q = JobQueue::new(4);
let _id = submit(
&q,
DagSpec {
source: Source::AutoUpdate,
reason: "boot".to_owned(),
declare: Box::new(|b: &Job| {
crate::workers::auto_update::boot_nodes(
b,
true,
vec!["stale-agent".to_owned()],
vec!["drifted-agent".to_owned()],
);
}),
},
);
// Both are independent roots on disjoint resources, so both start at once.
let claims = q.claim_ready();
let by_kind = |kind: &str| {
claims
.iter()
.find(|c| c.kind.as_str() == kind)
.unwrap_or_else(|| panic!("no {kind} claim in {claims:?}"))
};
let mut lock = declared_resources(&q, by_kind("meta_lock").node_id);
lock.sort_by_key(|r| format!("{r:?}"));
assert_eq!(
lock,
vec![Resource::BuildSlot, Resource::MetaWindow],
"the sweep MetaLock runs a nix lock bump and commits to meta"
);
assert_eq!(
declared_resources(&q, by_kind("reconcile").node_id),
vec![Resource::Agent("drifted-agent".to_owned())],
"a boot Reconcile touches the container, so it holds that agent's lease"
);
}
#[test] #[test]
fn append_subgraph_roots_on_emitter_and_rebases_local_deps() { fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
// The startup-sweep mechanism: a `MetaLock` emitter grows one rebuild // The startup-sweep mechanism: a `MetaLock` emitter grows one rebuild
@ -808,14 +859,11 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
source: Source::AutoUpdate, source: Source::AutoUpdate,
reason: "sweep".to_owned(), reason: "sweep".to_owned(),
declare: Box::new(|b: &Job| { declare: Box::new(|b: &Job| {
let _lock = templates::node( let _lock = b.node(NodeKind::MetaLock {
b, sweep: true,
NodeKind::MetaLock { fanout: None,
sweep: true, inputs: Vec::new(),
fanout: None, });
inputs: Vec::new(),
},
);
}), }),
}; };
let id = submit(&q, spec); let id = submit(&q, spec);

70
hive-c0re/src/workers/auto_update.rs
View file

@ -318,6 +318,49 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
Ok(()) Ok(())
} }
/// The boot DAG's node declarations, split out of [`submit_boot_tree`] so they
/// can be exercised without a live [`Coordinator`].
///
/// That split is not cosmetic: this path constructs nodes outside
/// `job_queue/`, so it is the one place a resource declaration can be forgotten
/// without any in-module test noticing. It has happened once already — the
/// sweep `MetaLock` and the boot `Reconcile`s silently declared nothing when
/// kind-derived resources were removed, which drops the agent lease a boot
/// reconcile needs to not race another DAG's container ops.
pub(crate) fn boot_nodes(
b: &crate::job_queue::Job,
any_stale: bool,
fanout: Vec<String>,
drifted: Vec<String>,
) {
use crate::job_queue::NodeKind;
use crate::job_queue::resource::Resource;
// Sweep whenever ANY marker is stale — even when every stale agent is
// wanted-offline: the hyperhive lock bump must land now so their later
// start-upgrade rebuilds build against it. No stale agents ⇒ no MetaLock
// ⇒ no meta commit on a no-change boot. The `fanout` list rides the
// MetaLock into `run_meta_lock`, which appends the rebuild subgraphs.
if any_stale {
let _ = b
.node(NodeKind::MetaLock {
sweep: true,
fanout: Some(fanout),
// A sweep bumps `hyperhive` alone (`lock_update_hyperhive`),
// so it names no inputs.
inputs: Vec::new(),
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
}
// One boot Reconcile per drifted agent — independent roots.
for name in drifted {
// Name the lease before the agent string moves into the kind.
let lease = Resource::Agent(name.clone());
let _ = b.node(NodeKind::Reconcile { agent: name }).needs(lease);
}
}
/// Submit this boot's work as **one DAG** (no anchor node, no per-agent /// Submit this boot's work as **one DAG** (no anchor node, no per-agent
/// child DAGs). Node 0 is the sweep `MetaLock` (only when /// child DAGs). Node 0 is the sweep `MetaLock` (only when
/// something is stale) — its executor bumps the hyperhive lock, then grows /// something is stale) — its executor bumps the hyperhive lock, then grows
@ -334,7 +377,7 @@ fn submit_boot_tree(
n_deferred: usize, n_deferred: usize,
n_skipped: usize, n_skipped: usize,
) { ) {
use crate::job_queue::{DagSpec, NodeKind, Source, templates}; use crate::job_queue::{DagSpec, Source};
// Fully-quiet boot (nothing stale, nothing drifted) submits nothing. // Fully-quiet boot (nothing stale, nothing drifted) submits nothing.
if !any_stale && drifted.is_empty() { if !any_stale && drifted.is_empty() {
@ -348,29 +391,8 @@ fn submit_boot_tree(
n_skipped, n_skipped,
); );
let declare: crate::job_queue::Declare = Box::new(move |b| { let declare: crate::job_queue::Declare =
// Sweep whenever ANY marker is stale — even when every stale agent is Box::new(move |b| boot_nodes(b, any_stale, fanout, drifted));
// wanted-offline: the hyperhive lock bump must land now so their later
// start-upgrade rebuilds build against it. No stale agents ⇒ no MetaLock
// ⇒ no meta commit on a no-change boot. The `fanout` list rides the
// MetaLock into `run_meta_lock`, which appends the rebuild subgraphs.
if any_stale {
let _ = templates::node(
b,
NodeKind::MetaLock {
sweep: true,
fanout: Some(fanout),
// A sweep bumps `hyperhive` alone (`lock_update_hyperhive`),
// so it names no inputs.
inputs: Vec::new(),
},
);
}
// One boot Reconcile per drifted agent — independent roots.
for name in drifted {
let _ = templates::node(b, NodeKind::Reconcile { agent: name });
}
});
let spec = DagSpec { let spec = DagSpec {
// The sweep's own rebuild subgraphs emit their `Rebuilt` events as they // The sweep's own rebuild subgraphs emit their `Rebuilt` events as they