vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: fix the boot sweep's lost declarations, drop the node wrapper

Browse source 
Two review findings on the resources-at-construction change.

argus: `workers::auto_update`'s boot sweep constructs nodes through
`templates::node` too, and it was not converted. With the kind-derived
declaration gone, its sweep `MetaLock` and its per-agent `Reconcile`
silently declared no resources at all — so a boot reconcile no longer
held the agent lease and could race another DAG's container ops, and the
sweep's meta commit could land inside another node's staged deploy
window. Nothing failed to compile: removing an implicit behaviour from a
helper is invisible at every call site that relied on it.

The declarations now live in a pure `boot_nodes`, split out of
`submit_boot_tree` so they can be exercised without a `Coordinator`.
That path is the only place job nodes are built outside `job_queue/`,
which is exactly why it had no coverage; `boot_sweep_nodes_declare_
their_own_resources` closes that, asserting against declared graph edges
rather than against the kind.

mara: `templates::node` is a redundant redirect now that it no longer
derives resources — deleted, and its 43 call sites use `Job::node`
directly. The reasoning it documented moved to the module docs of
`templates.rs` and `resource.rs`, which is where it stays true.
This commit is contained in:
atlas 2026-08-02 16:22:56 +02:00 • committed by mara
commit 58a9f218f2
6 changed files with 231 additions and 180 deletions
Download patch file Download diff file Expand all files Collapse all files

193
hive-c0re/src/job_queue/templates.rs
View file

@ -1,7 +1,10 @@
//! DAG shape builders — every operation as a template over the shared
//! node primitives. Pure (no I/O); each node carries its own `agent` (there is
//! no DAG-level agent), stamped by the [`node`] helper along with the
//! resources that node's kind needs.
//! no DAG-level agent) and **declares its own resources** with `.needs(…)`
//! right where it is constructed, rather than having them derived from its
//! kind. Deriving made the requirement a property of the *kind*, so a kind that
//! happened to run under an ancestor already holding the resource could get
//! away with declaring nothing.
//!
//! ```text
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
@ -19,7 +22,7 @@
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
//! their per-agent shape depends on live running state (an async
//! `lifecycle::is_running` read), so `submit.rs` assembles them out of the
//! primitives this module exports ([`node`], [`rebuild_nodes`]).
//! primitives this module exports ([`rebuild_nodes`]) over `Job::node`.
use hive_jobq::TerminalState;
@ -27,28 +30,6 @@ use super::model::{DagSpec, NodeKind, PermPayload, Source};
use super::resource::Resource;
use super::{Declare, Handle, Job};
/// Declare one node carrying `kind`. **Resources are not derived here** — the
/// construction site says what the node holds, with `.needs(…)`.
///
/// That is the point rather than an omission. Deriving `(name, units)` from the
/// kind made the declaration a property of the *kind*, so a kind that happened
/// to run under an ancestor holding the resource could get away with declaring
/// nothing — which is precisely how `Start` / `Stop` / `PostSwap` ended up
/// lease-exempt: one construction site fans them out from inside a
/// lease-holding `Reconcile`. The requirement is a property of the node, not of
/// the one DAG shape it is used in today.
///
/// Declaring a resource an ancestor already holds is free: a descendant
/// re-enters that grant through the crate's recursive lock rather than taking a
/// fresh unit.
///
/// The returned handle is where edges and grouping are declared, and is `Copy`
/// — naming a node as a dependency does not consume the ability to name it
/// again.
pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
b.node(kind)
}
/// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node
/// gated on every group-root in `roots`, and the failure node gated on *its*
/// elimination.
@ -57,13 +38,10 @@ pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
/// dropped — see [`hive_jobq::NodeRef::on_elimination_of`].
fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
let ok = roots.iter().fold(
node(
b,
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
},
),
b.node(NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
}),
hive_jobq::NodeRef::after_ok,
);
// The failure branch needs *both*: the ok branch being ruled out (that is the
@ -73,13 +51,10 @@ fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
// `Reconcile` is still bringing the container back up, so reporting straight
// off the elimination would announce the failure mid-recovery.
let _failed = roots.iter().fold(
node(
b,
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: false,
},
)
b.node(NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: false,
})
.on_elimination_of(ok),
hive_jobq::NodeRef::after_any,
);
@ -96,14 +71,12 @@ fn resolve_approval_tails(b: &Job, approval_id: i64, root: Handle<'_>) {
TerminalState::Failed,
TerminalState::Cancelled,
] {
let _ = node(
b,
NodeKind::ResolveApproval {
let _ = b
.node(NodeKind::ResolveApproval {
approval_id,
outcome,
},
)
.on_outcome(root, &[outcome]);
})
.on_outcome(root, &[outcome]);
}
}
@ -182,37 +155,42 @@ pub(crate) fn rebuild_nodes<'a>(
let a = || agent.to_owned();
let RebuildOpts { relock, graceful } = opts;
let mut meta_sync =
node(b, NodeKind::MetaSync { agent: a(), relock }).needs(Resource::MetaWindow);
let mut meta_sync = b
.node(NodeKind::MetaSync { agent: a(), relock })
.needs(Resource::MetaWindow);
if let Some(after) = after {
meta_sync = meta_sync.after_ok(after);
}
let prebuild = node(b, NodeKind::Prebuild { agent: a() })
let prebuild = b
.node(NodeKind::Prebuild { agent: a() })
.needs(Resource::BuildSlot)
.after_ok(meta_sync);
// The stop root hangs off `Prebuild` and owns the agent lease for
// everything below it. `StopForUpdate` parents the swap pair either way.
let stop_for_update = if graceful {
let signal = node(b, NodeKind::Signal { agent: a() })
let signal = b
.node(NodeKind::Signal { agent: a() })
.needs(Resource::Agent(a()))
.part_of(prebuild);
// `Drain` is a *child* of `Signal`, so the parent gate already orders
// it — a child must not dep on its own parent (dep-scope).
let drain = node(b, NodeKind::Drain { agent: a() })
let drain = b
.node(NodeKind::Drain { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal);
node(b, NodeKind::StopForUpdate { agent: a() })
b.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(signal)
.after_ok(drain)
} else {
node(b, NodeKind::StopForUpdate { agent: a() })
b.node(NodeKind::StopForUpdate { agent: a() })
.needs(Resource::Agent(a()))
.part_of(prebuild)
};
let swap = node(b, NodeKind::Swap { agent: a() })
let swap = b
.node(NodeKind::Swap { agent: a() })
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.part_of(stop_for_update);
@ -220,12 +198,14 @@ pub(crate) fn rebuild_nodes<'a>(
// `StopForUpdate`, which holds it, so this is a re-entrant borrow — no
// second unit, no deadlock. Declaring it is what stops the requirement
// being true only of this one DAG shape.
let _post_swap = node(b, NodeKind::PostSwap { agent: a() })
let _post_swap = b
.node(NodeKind::PostSwap { agent: a() })
.needs(Resource::Agent(a()))
.part_of(stop_for_update)
.after_ok(swap);
let reconcile = node(b, NodeKind::Reconcile { agent: a() })
let reconcile = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.after_any(prebuild);
@ -272,16 +252,14 @@ pub(crate) fn deploy_rebuild_nodes(agent: &str, approval_id: i64) -> Declare {
},
None,
);
let _finalize = node(
b,
NodeKind::FinalizeDeploy {
let _finalize = b
.node(NodeKind::FinalizeDeploy {
agent: agent.clone(),
approval_id,
},
)
.needs(Resource::MetaWindow)
.after_ok(roots.prebuild)
.after_ok(roots.reconcile);
})
.needs(Resource::MetaWindow)
.after_ok(roots.prebuild)
.after_ok(roots.reconcile);
})
}
@ -363,42 +341,34 @@ pub fn approval_deploy(
// (`MetaWindow`). All three are held for its whole subtree, which
// is what lets the appended rebuild's `MetaSync` and the
// `FinalizeDeploy` re-enter rather than contend.
let window = node(
b,
NodeKind::DeployWindow {
let window = b
.node(NodeKind::DeployWindow {
agent: a(),
approval_id,
},
)
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.needs(Resource::MetaWindow);
let verify = node(
b,
NodeKind::MergeVerify {
})
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.needs(Resource::MetaWindow);
let verify = b
.node(NodeKind::MergeVerify {
agent: a(),
approval_id,
},
)
.part_of(window);
let apply = node(
b,
NodeKind::DeployApply {
})
.part_of(window);
let apply = b
.node(NodeKind::DeployApply {
agent: a(),
approval_id,
},
)
.part_of(window)
.after_ok(verify);
let _tail = node(
b,
NodeKind::DeployTail {
})
.part_of(window)
.after_ok(verify);
let _tail = b
.node(NodeKind::DeployTail {
agent: a(),
approval_id,
},
)
.part_of(window)
.after_any(apply);
})
.part_of(window)
.after_any(apply);
resolve_approval_tails(b, approval_id, window);
}),
@ -423,7 +393,7 @@ pub fn reconcile_only(
declare: Box::new(move |b: &Job| {
// Name the lease before the agent string is moved into the kind.
let lease = Resource::Agent(agent.clone());
let _reconcile = node(b, NodeKind::Reconcile { agent }).needs(lease);
let _reconcile = b.node(NodeKind::Reconcile { agent }).needs(lease);
}),
}
}
@ -446,15 +416,20 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec<impl FnOn
reason,
declare: Box::new(move |b: &Job| {
let a = || agent.clone();
let provision = node(b, NodeKind::Provision { agent: a() }).needs(Resource::MetaWindow);
let create = node(b, NodeKind::Create { agent: a() })
let provision = b
.node(NodeKind::Provision { agent: a() })
.needs(Resource::MetaWindow);
let create = b
.node(NodeKind::Create { agent: a() })
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.part_of(provision);
let dropin = node(b, NodeKind::WriteDropin { agent: a() })
let dropin = b
.node(NodeKind::WriteDropin { agent: a() })
.needs(Resource::Agent(a()))
.part_of(create);
let _reconcile = node(b, NodeKind::Reconcile { agent: a() })
let _reconcile = b
.node(NodeKind::Reconcile { agent: a() })
.needs(Resource::Agent(a()))
.part_of(create)
.after_ok(dropin);
@ -480,14 +455,12 @@ pub fn perm_change(
source,
reason,
declare: Box::new(move |b: &Job| {
let write = node(
b,
NodeKind::WritePermFile {
let write = b
.node(NodeKind::WritePermFile {
agent: agent.clone(),
payload,
},
)
.needs(Resource::MetaWindow);
})
.needs(Resource::MetaWindow);
let roots = rebuild_nodes(
b,
&agent,
@ -526,16 +499,14 @@ pub fn meta_update(
source,
reason,
declare: Box::new(move |b: &Job| {
let lock = node(
b,
NodeKind::MetaLock {
let lock = b
.node(NodeKind::MetaLock {
sweep: false,
fanout: None,
inputs,
},
)
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
// The bump itself has no side effect, so an operator-driven one ends
// at the `MetaLock`; an approval-driven one still has its row to
// resolve and gets the per-outcome tails edged onto that single
@ -566,7 +537,9 @@ pub fn reparent(
source,
reason,
declare: Box::new(move |b: &Job| {
let _reparent = node(b, NodeKind::Reparent { moves }).needs(Resource::MetaWindow);
let _reparent = b
.node(NodeKind::Reparent { moves })
.needs(Resource::MetaWindow);
}),
}
}