refactor(#2949): kill Declare — a running node declares onto its own builder

A node no longer hands back a recipe for the scheduler to replay later. It
declares straight onto a builder it was given, and that builder is inserted
as part of completing the node.

Deleted: `pub type Declare`, `struct NodeOutput` (+ its hand-written `Debug`),
`JobQueue::append_subgraph`. Nothing added to `Dag` / `DagView`.

jobq gains `Scheduler::new_job()` (the only way to obtain a `JobBuilder`) and
`complete_growing(id, outcome, grown)`, which inserts under `id` and *then*
completes it, so a DAG cannot roll terminal while grown work is still pending.
`complete()` and `complete_growing()` share a private `finish()` rather than
one redirecting through the other. The DAG-gone guard lives beside the graph
now, where it cannot be skipped, instead of being a caller-side lookup.

The growth executors return data (`run_meta_lock -> (Vec<String>, RebuildOpts)`,
`run_reconcile -> Option<NodeKind>`) rather than taking the builder: a `&Job`
parameter is live for the whole function body, and `&RefCell<T>` is never
`Send`, so an async fn taking one cannot be spawned. `run_node` threads the
builder by value and hands it back.

A node can now declare work and then fail, which was previously inexpressible.
`grown` is dropped in that case — failure cancel-cascades downstream, so
inserting it would only add nodes to immediately cancel — and the log line
carries `grown_nodes` so the drop is visible.
This commit is contained in:
atlas 2026-08-02 17:20:43 +02:00 committed by mara
commit 82ef06f445
8 changed files with 388 additions and 344 deletions

View file

@ -10,7 +10,7 @@ use std::sync::Arc;
use anyhow::{Context as _, Result}; use anyhow::{Context as _, Result};
use super::{Claim, Declare}; use super::Claim;
use hive_jobq::TerminalState; use hive_jobq::TerminalState;
use super::model::NodeKind; use super::model::NodeKind;
@ -26,36 +26,6 @@ use crate::power::{ReconcileAction, reconcile_action};
/// N × this timeout. /// N × this timeout.
pub const GRACEFUL_STOP_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(3); pub const GRACEFUL_STOP_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(3);
/// Extra signal an executor hands back to the scheduler alongside
/// success.
#[derive(Default)]
pub struct NodeOutput {
/// Whole per-agent *subgraphs* to append into *this same* DAG at
/// runtime — the single in-DAG-growth channel. Each [`Job`] is one
/// independent subgraph, declared but not yet inserted: an executor cannot
/// reach the queue, so it hands the declaration back and the scheduler
/// inserts it via [`super::JobQueue::append_subgraph`] under its own lock,
/// rooted on the emitting node. Used both for the multi-node case
/// (`MetaLock` growing one rebuild subgraph per agent — the startup
/// sweep's stale agents, the meta-update cascade's affected agents) and
/// the single-node case (a `Reconcile` planner emitting its mechanical
/// `Start` / `Stop` as a one-node subgraph). The scheduler applies these
/// *before* the emitting node's completion so the DAG never rolls terminal
/// with the appended work still pending — keeping the lease-window
/// transient held across the sub-step.
pub append_subgraph: Vec<Declare>,
}
impl std::fmt::Debug for NodeOutput {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
// The subgraphs are closures — how many were emitted is the only thing
// there is to say about them before the queue runs them.
f.debug_struct("NodeOutput")
.field("append_subgraph", &self.append_subgraph.len())
.finish()
}
}
/// Build-log sink for one claimed node. /// Build-log sink for one claimed node.
struct Ctx<'a> { struct Ctx<'a> {
coord: &'a Arc<Coordinator>, coord: &'a Arc<Coordinator>,
@ -78,13 +48,35 @@ impl Ctx<'_> {
/// Run one claimed node to completion. Called from a task the /// Run one claimed node to completion. Called from a task the
/// scheduler spawns per claim; the `Result` (stringified) becomes the /// scheduler spawns per claim; the `Result` (stringified) becomes the
/// node's terminal state. /// node's terminal state.
pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { ///
/// `job` is the node's own growth channel: an executor that decides more work
/// is needed declares it here, and the scheduler inserts it under this node
/// when the node completes. Most executors never touch it. Nothing is inserted
/// while the node runs — the builder is local state, so this stays outside the
/// queue's lock for the whole (often multi-minute) execution.
///
/// ⚠️ Taken **by value and handed back**, not by reference. A `JobBuilder` is
/// `RefCell`-backed: owned it is `Send`, but `&JobBuilder` is not (a shared ref
/// is `Send` only if the referent is `Sync`, and `RefCell` never is). A `&Job`
/// parameter would be live across every `.await` in this fn and make the whole
/// future non-`Send`, which the scheduler's `tokio::spawn` rejects. So the
/// growth executors below return *what to grow* and the declaration happens
/// here, synchronously, between awaits.
pub(super) async fn run_node(
coord: &Arc<Coordinator>,
job: super::Job,
claim: &Claim,
) -> (super::Job, Result<()>) {
let ctx = Ctx { let ctx = Ctx {
coord, coord,
dag_id: claim.dag_id, dag_id: claim.dag_id,
node_id: claim.node_id, node_id: claim.node_id,
}; };
match &claim.kind { // Every arm is `Result<()>`; the three that grow work declare into `job`
// *synchronously*, after their own awaits have finished. Borrowing `&job`
// inside an `.await` would make this future non-`Send` (see above), so the
// growth executors return what to grow rather than taking the builder.
let result = match &claim.kind {
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await, NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await, NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await,
NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await, NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await,
@ -95,19 +87,40 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
sweep, sweep,
fanout, fanout,
inputs, inputs,
} => run_meta_lock(coord, *sweep, fanout.clone(), inputs).await, } => run_meta_lock(coord, *sweep, fanout.clone(), inputs)
NodeKind::Reconcile { .. } => run_reconcile(coord, claim).await, .await
.map(|(agents, opts)| {
for agent in agents {
super::templates::rebuild_nodes(&job, &agent, opts, None);
}
}),
NodeKind::Reconcile { .. } => run_reconcile(coord, claim).await.map(|sub| {
if let Some(kind) = sub {
// `Start` / `Stop` declare the lease they run under. This node
// is their parent and holds it, so the declaration is a
// re-entrant borrow — no second unit, no deadlock. It exists so
// the requirement belongs to the node rather than to the fact
// that a `Reconcile` happens to fan it out.
let lease = Resource::Agent(kind.agent().to_owned());
let _ = job.node(kind).needs(lease);
}
}),
NodeKind::Start { .. } => run_start(coord, claim).await, NodeKind::Start { .. } => run_start(coord, claim).await,
NodeKind::Stop { .. } => run_stop(coord, claim).await, NodeKind::Stop { .. } => run_stop(coord, claim).await,
NodeKind::StopForUpdate { .. } => run_stop_for_update(coord, claim).await, NodeKind::StopForUpdate { .. } => run_stop_for_update(coord, claim).await,
NodeKind::Signal { .. } => Ok(run_signal(coord, claim)), NodeKind::Signal { .. } => {
run_signal(coord, claim);
Ok(())
}
NodeKind::Drain { .. } => run_drain(coord, claim).await, NodeKind::Drain { .. } => run_drain(coord, claim).await,
NodeKind::WriteDropin { .. } => run_write_dropin(coord, claim).await, NodeKind::WriteDropin { .. } => run_write_dropin(coord, claim).await,
NodeKind::WritePermFile { .. } => run_write_perm_file(coord, claim).await, NodeKind::WritePermFile { .. } => run_write_perm_file(coord, claim).await,
NodeKind::Reparent { .. } => run_reparent(coord, claim).await, NodeKind::Reparent { .. } => run_reparent(coord, claim).await,
NodeKind::MergeVerify { approval_id, .. } => run_merge_verify(coord, *approval_id).await, NodeKind::MergeVerify { approval_id, .. } => run_merge_verify(coord, *approval_id).await,
NodeKind::DeployApply { approval_id, .. } => { NodeKind::DeployApply { approval_id, .. } => {
run_deploy_apply(coord, claim, *approval_id).await run_deploy_apply(coord, *approval_id).await.map(|()| {
super::templates::deploy_rebuild_nodes(&job, claim.kind.agent(), *approval_id);
})
} }
NodeKind::FinalizeDeploy { approval_id, .. } => { NodeKind::FinalizeDeploy { approval_id, .. } => {
run_finalize_deploy(coord, *approval_id).await run_finalize_deploy(coord, *approval_id).await
@ -119,7 +132,10 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
approval_id, approval_id,
outcome, outcome,
} => run_resolve_approval(coord, claim, *approval_id, *outcome).await, } => run_resolve_approval(coord, claim, *approval_id, *outcome).await,
NodeKind::EmitRebuilt { ok, .. } => Ok(run_emit_rebuilt(coord, claim, *ok)), NodeKind::EmitRebuilt { ok, .. } => {
run_emit_rebuilt(coord, claim, *ok);
Ok(())
}
NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up), NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up),
// The two nodes that carry no work of their own; completing either // The two nodes that carry no work of their own; completing either
// lets it reach `Finishing` so the nodes under it start. // lets it reach `Finishing` so the nodes under it start.
@ -127,8 +143,9 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
// any, is its own tail node in the graph. // any, is its own tail node in the graph.
// - `DeployWindow`: pure resource holder — the meta window, agent lease // - `DeployWindow`: pure resource holder — the meta window, agent lease
// and build slot it declares stay held until its subtree settles. // and build slot it declares stay held until its subtree settles.
NodeKind::Dag { .. } | NodeKind::DeployWindow { .. } => Ok(NodeOutput::default()), NodeKind::Dag { .. } | NodeKind::DeployWindow { .. } => Ok(()),
} };
(job, result)
} }
/// Resolve the DAG's approval row the way this node's own `outcome` says. /// Resolve the DAG's approval row the way this node's own `outcome` says.
@ -143,18 +160,18 @@ async fn run_resolve_approval(
claim: &Claim, claim: &Claim,
approval_id: i64, approval_id: i64,
outcome: TerminalState, outcome: TerminalState,
) -> Result<NodeOutput> { ) -> Result<()> {
let reason = (outcome == TerminalState::Failed) let reason = (outcome == TerminalState::Failed)
.then(|| coord.job_queue.first_error(claim.dag_id)) .then(|| coord.job_queue.first_error(claim.dag_id))
.flatten(); .flatten();
crate::actions::resolve_approval_dag(coord, approval_id, outcome, reason.as_deref()).await; crate::actions::resolve_approval_dag(coord, approval_id, outcome, reason.as_deref()).await;
Ok(NodeOutput::default()) Ok(())
} }
/// Emit this agent's `Rebuilt` manager event. `ok` is not computed — it is which /// Emit this agent's `Rebuilt` manager event. `ok` is not computed — it is which
/// of the tail pair the graph let run. The failure note comes from the DAG's /// of the tail pair the graph let run. The failure note comes from the DAG's
/// first failing node, since the branch knows *that* it failed but not *why*. /// first failing node, since the branch knows *that* it failed but not *why*.
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) -> NodeOutput { fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) {
coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt { coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: claim.agent.clone(), agent: claim.agent.clone(),
ok, ok,
@ -164,7 +181,6 @@ fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) -> NodeOu
sha: None, sha: None,
tag: None, tag: None,
}); });
NodeOutput::default()
} }
/// Write the agent's durable power intent — the DAG-node form of the old /// Write the agent's durable power intent — the DAG-node form of the old
@ -175,7 +191,7 @@ fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) -> NodeOu
/// warn-and-continue write, a failed write fails the node (cancel-downstream /// warn-and-continue write, a failed write fails the node (cancel-downstream
/// cancels the `Reconcile`) rather than letting it converge to a stale /// cancels the `Reconcile`) rather than letting it converge to a stale
/// intent — that atomicity is the point of moving it into the DAG. /// intent — that atomicity is the point of moving it into the DAG.
fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<NodeOutput> { fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<()> {
let wanted = if up { let wanted = if up {
crate::power::Wanted::Up crate::power::Wanted::Up
} else { } else {
@ -185,7 +201,7 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<N
.power .power
.set(&claim.agent, wanted) .set(&claim.agent, wanted)
.with_context(|| format!("set wanted={} for agent {}", wanted.as_str(), claim.agent))?; .with_context(|| format!("set wanted={} for agent {}", wanted.as_str(), claim.agent))?;
Ok(NodeOutput::default()) Ok(())
} }
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta /// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
@ -197,11 +213,7 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<N
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that /// Deliberately a separate node from the [`run_prebuild`] it feeds: that
/// build takes minutes and only *reads* the store, so keeping the global /// build takes minutes and only *reads* the store, so keeping the global
/// window off it is what lets rebuilds of different agents overlap. /// window off it is what lets rebuilds of different agents overlap.
async fn run_meta_sync( async fn run_meta_sync(coord: &Arc<Coordinator>, claim: &Claim, relock: bool) -> Result<()> {
coord: &Arc<Coordinator>,
claim: &Claim,
relock: bool,
) -> Result<NodeOutput> {
let name = &claim.agent; let name = &claim.agent;
// Runs while the agent is still up — the runtime dir and MCP listener // Runs while the agent is still up — the runtime dir and MCP listener
// already exist. Use the pure path accessor; no need to re-register the // already exist. Use the pure path accessor; no need to re-register the
@ -219,7 +231,7 @@ async fn run_meta_sync(
if relock { if relock {
crate::meta::lock_update_for_rebuild(name).await?; crate::meta::lock_update_for_rebuild(name).await?;
} }
Ok(NodeOutput::default()) Ok(())
} }
/// Out-of-band toplevel build while the container keeps serving: warm /// Out-of-band toplevel build while the container keeps serving: warm
@ -229,7 +241,7 @@ async fn run_meta_sync(
/// container is already down: its only purpose is to shrink the swap's /// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't /// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead. /// pay the double eval — `Swap` builds inline instead.
async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> { async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
// Warm the toplevel build only when the container is up — the whole // Warm the toplevel build only when the container is up — the whole
// point of prebuild is to shrink the swap's downtime window. A // point of prebuild is to shrink the swap's downtime window. A
@ -240,7 +252,7 @@ async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
crate::lifecycle::prebuild_toplevel(name, &flake_ref, &|log_id| ctx.build_log(log_id)) crate::lifecycle::prebuild_toplevel(name, &flake_ref, &|log_id| ctx.build_log(log_id))
.await?; .await?;
} }
Ok(NodeOutput::default()) Ok(())
} }
/// Profile-swap: re-apply drop-ins (rebuild is the reconcile verb), /// Profile-swap: re-apply drop-ins (rebuild is the reconcile verb),
@ -248,7 +260,7 @@ async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
/// (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan). /// (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan).
/// The recovery-start on failure is NOT here — the DAG's tail /// The recovery-start on failure is NOT here — the DAG's tail
/// `Reconcile` runs after this node terminal ok *or* fail. /// `Reconcile` runs after this node terminal ok *or* fail.
async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> { async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
// Swap runs on an already-existing (stopped) container — runtime dir // Swap runs on an already-existing (stopped) container — runtime dir
// and listener were created earlier. Pure path accessor suffices. // and listener were created earlier. Pure path accessor suffices.
@ -269,7 +281,7 @@ async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Res
if result.is_err() { if result.is_err() {
coord.rescan_containers_and_emit().await; coord.rescan_containers_and_emit().await;
} }
result.map(|()| NodeOutput::default()) result
} }
/// The post-`Swap` bookkeeping tail, split into its own node for dashboard /// The post-`Swap` bookkeeping tail, split into its own node for dashboard
@ -277,7 +289,7 @@ async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Res
/// means the profile swap succeeded. Store/forge/matrix work only — no nix /// means the profile swap succeeded. Store/forge/matrix work only — no nix
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held /// build (build-slot-exempt); the agent lease taken at `Swap` is still held
/// (the whole chain up to `Reconcile` is one agent's subgraph). /// (the whole chain up to `Reconcile` is one agent's subgraph).
async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake) if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake)
&& let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev) && let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev)
@ -298,20 +310,20 @@ async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
coord.kick_agent(name, "container rebuilt"); coord.kick_agent(name, "container rebuilt");
coord.rescan_containers_and_emit().await; coord.rescan_containers_and_emit().await;
crate::dashboard::emit_meta_inputs_snapshot(coord); crate::dashboard::emit_meta_inputs_snapshot(coord);
Ok(NodeOutput::default()) Ok(())
} }
/// First-spawn pre-create provisioning: proposed/applied repos, state /// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Runs under the /// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (it declares `Resource::MetaWindow`) so its commit can't /// deploy window (it declares `Resource::MetaWindow`) so its commit can't
/// land inside another node's staged deploy window. /// land inside another node's staged deploy window.
async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
let agent_dir = crate::paths::agent_runtime_dir(name); let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env(); let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir); let paths = Coordinator::agent_paths(name, agent_dir);
crate::lifecycle::provision_container(name, &hive, &paths).await?; crate::lifecycle::provision_container(name, &hive, &paths).await?;
Ok(NodeOutput::default()) Ok(())
} }
/// `nixos-container create` proper — the upstream `Provision` node /// `nixos-container create` proper — the upstream `Provision` node
@ -320,21 +332,24 @@ async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
/// dir creation and MCP listener registration are deferred to the tail /// dir creation and MCP listener registration are deferred to the tail
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this /// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
/// node stays purely "create", not "create + start". /// node stays purely "create", not "create + start".
async fn run_create(claim: &Claim) -> Result<NodeOutput> { async fn run_create(claim: &Claim) -> Result<()> {
crate::lifecycle::create_only(&claim.agent).await?; crate::lifecycle::create_only(&claim.agent).await?;
Ok(NodeOutput::default()) Ok(())
} }
/// Meta flake lock bump. Boot-sweep flavour is non-fatal (a failed /// Meta flake lock bump. Boot-sweep flavour is non-fatal (a failed
/// bump must not cancel the fan-out rebuilds — they proceed against /// bump must not cancel the fan-out rebuilds — they proceed against
/// the current lock, exactly like today's sweep); the meta-update /// the current lock, exactly like today's sweep); the meta-update
/// flavour propagates errors, and a failed bump fans out nothing. /// flavour propagates errors, and a failed bump fans out nothing.
/// Returns the agents whose rebuild subgraphs the caller should grow into this
/// node, and the options to build them with — rather than declaring them here.
/// The declaration has to happen outside any `.await` (see [`run_node`]).
async fn run_meta_lock( async fn run_meta_lock(
coord: &Arc<Coordinator>, coord: &Arc<Coordinator>,
sweep: bool, sweep: bool,
fanout: Option<Vec<String>>, fanout: Option<Vec<String>>,
inputs: &[String], inputs: &[String],
) -> Result<NodeOutput> { ) -> Result<(Vec<String>, super::templates::RebuildOpts)> {
if sweep { if sweep {
if let Err(e) = crate::meta::lock_update_hyperhive().await { if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed"); tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
@ -349,25 +364,13 @@ async fn run_meta_lock(
// drain window rather than being cut off. The per-agent drains overlap, // drain window rather than being cut off. The per-agent drains overlap,
// so the sweep's cost ceiling is one `GRACEFUL_STOP_TIMEOUT` in total, // so the sweep's cost ceiling is one `GRACEFUL_STOP_TIMEOUT` in total,
// not one per agent. // not one per agent.
let append_subgraph = fanout return Ok((
.unwrap_or_default() fanout.unwrap_or_default(),
.iter() super::templates::RebuildOpts {
.map(|agent| { relock: true,
let agent = agent.clone(); graceful: true,
Box::new(move |b: &super::Job| { },
super::templates::rebuild_nodes( ));
b,
&agent,
super::templates::RebuildOpts {
relock: true,
graceful: true,
},
None,
);
}) as Declare
})
.collect();
return Ok(NodeOutput { append_subgraph });
} }
let _progress = coord.meta_update_guard(); let _progress = coord.meta_update_guard();
crate::meta::lock_update(inputs).await?; crate::meta::lock_update(inputs).await?;
@ -383,68 +386,47 @@ async fn run_meta_lock(
// cascade children must NOT re-lock, which would revert the bump this // cascade children must NOT re-lock, which would revert the bump this
// node just committed (the property the old `fanout_specs` meta-update // node just committed (the property the old `fanout_specs` meta-update
// branch encoded). // branch encoded).
let append_subgraph = cascade Ok((
.iter() cascade,
.map(|agent| { super::templates::RebuildOpts {
let agent = agent.clone(); relock: false,
Box::new(move |b: &super::Job| { graceful: false,
super::templates::rebuild_nodes( },
b, ))
&agent,
super::templates::RebuildOpts {
relock: false,
graceful: false,
},
None,
);
}) as Declare
})
.collect();
Ok(NodeOutput { append_subgraph })
} }
/// Idempotent power-converge *planner*: compare `wanted` (durable /// Idempotent power-converge *planner*: compare `wanted` (durable
/// intent) against observed state and, when they diverge, fan the /// intent) against observed state and, when they diverge, fan the
/// mechanical `Start` / `Stop` out as a first-class node appended to /// mechanical `Start` / `Stop` out as a first-class node appended to
/// *this* DAG (a single-node `NodeOutput::append_subgraph` rooted on /// *this* DAG (a single node declared into `job`, rooted on this node).
/// this node). Does no container work itself — the sub-step becomes /// Does no container work itself — the sub-step becomes visible in the
/// visible in the DAG and the lease-window transient (or the sub-step's /// DAG and the lease-window transient (or the sub-step's own node-local
/// own node-local guard) rides across it. /// guard) rides across it.
async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { /// Returns the mechanical node to fan out (`None` on a noop) rather than
/// declaring it — the declaration has to happen outside any `.await`, see
/// [`run_node`]. `NodeKind` carries the agent it targets, so `claim.agent` is
/// stamped into the kind here.
async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Option<NodeKind>> {
let name = &claim.agent; let name = &claim.agent;
let running = crate::lifecycle::is_running(name).await; let running = crate::lifecycle::is_running(name).await;
let wanted = coord.power.get_or_seed(name, running)?; let wanted = coord.power.get_or_seed(name, running)?;
// One node targeting this agent, rooted on this reconcile node. `NodeKind` Ok(match reconcile_action(wanted, running) {
// carries the agent it targets, so stamp `claim.agent` into the fanned-out ReconcileAction::Start => Some(NodeKind::Start {
// Start/Stop kind (one in-DAG-growth channel).
let sub = |kind: NodeKind| {
// `Start` / `Stop` declare the lease they run under. This node is their
// parent and holds it, so the declaration is a re-entrant borrow — no
// second unit, no deadlock. It exists so the requirement belongs to the
// node rather than to the fact that a `Reconcile` happens to fan it out.
let lease = Resource::Agent(kind.agent().to_owned());
vec![Box::new(move |b: &super::Job| {
let _ = b.node(kind).needs(lease);
}) as Declare]
};
let append_subgraph = match reconcile_action(wanted, running) {
ReconcileAction::Start => sub(NodeKind::Start {
agent: name.clone(), agent: name.clone(),
}), }),
ReconcileAction::Stop => sub(NodeKind::Stop { ReconcileAction::Stop => Some(NodeKind::Stop {
agent: name.clone(), agent: name.clone(),
}), }),
ReconcileAction::Noop => { ReconcileAction::Noop => {
tracing::debug!(%name, wanted = wanted.as_str(), running, "reconcile: noop"); tracing::debug!(%name, wanted = wanted.as_str(), running, "reconcile: noop");
Vec::new() None
} }
}; })
Ok(NodeOutput { append_subgraph })
} }
/// Mechanical container start — the sub-step a `Reconcile` planner fans /// Mechanical container start — the sub-step a `Reconcile` planner fans
/// out when it observes `wanted = Up` and the container down. /// out when it observes `wanted = Up` and the container down.
async fn run_start(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_start(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
// No node-local transient guard: the pill is derived from the running node // No node-local transient guard: the pill is derived from the running node
// set, and `Start` reports `Starting` via `NodeKind::transient_kind`. This // set, and `Start` reports `Starting` via `NodeKind::transient_kind`. This
@ -468,12 +450,12 @@ async fn run_start(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput
coord.register_agent(name)?; coord.register_agent(name)?;
coord.kick_agent(name, "container started"); coord.kick_agent(name, "container started");
coord.rescan_containers_and_emit().await; coord.rescan_containers_and_emit().await;
Ok(NodeOutput::default()) Ok(())
} }
/// Mechanical container stop — the sub-step a `Reconcile` planner fans /// Mechanical container stop — the sub-step a `Reconcile` planner fans
/// out when it observes `wanted = Offline` and the container up. /// out when it observes `wanted = Offline` and the container up.
async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
// See `run_start`: no node-local guard — `Stop` reports `Stopping` from its // See `run_start`: no node-local guard — `Stop` reports `Stopping` from its
// own kind now. // own kind now.
@ -483,12 +465,12 @@ async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput>
agent: name.clone(), agent: name.clone(),
}); });
coord.rescan_containers_and_emit().await; coord.rescan_containers_and_emit().await;
Ok(NodeOutput::default()) Ok(())
} }
/// Mechanical stop for the profile swap. Never *changes* `wanted`; /// Mechanical stop for the profile swap. Never *changes* `wanted`;
/// noop when already stopped. /// noop when already stopped.
async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
if crate::lifecycle::is_running(name).await { if crate::lifecycle::is_running(name).await {
// Seed a missing agent_power row from the PRE-stop observation // Seed a missing agent_power row from the PRE-stop observation
@ -501,7 +483,7 @@ async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
crate::lifecycle::kill(name).await?; crate::lifecycle::kill(name).await?;
coord.rescan_containers_and_emit().await; coord.rescan_containers_and_emit().await;
} }
Ok(NodeOutput::default()) Ok(())
} }
/// Set the graceful fence + kick so the harness sees it promptly and /// Set the graceful fence + kick so the harness sees it promptly and
@ -513,19 +495,18 @@ async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
/// `GRACEFUL_STOP_TIMEOUT`. Safe because the harness tests the marker at /// `GRACEFUL_STOP_TIMEOUT`. Safe because the harness tests the marker at
/// the top of its loop — a paused agent has no turn in flight, so there /// the top of its loop — a paused agent has no turn in flight, so there
/// is nothing to checkpoint. /// is nothing to checkpoint.
fn run_signal(coord: &Arc<Coordinator>, claim: &Claim) -> NodeOutput { fn run_signal(coord: &Arc<Coordinator>, claim: &Claim) {
if hive_types::Ident::parse(&claim.agent).is_ok_and(|a| Coordinator::is_paused(&a)) { if hive_types::Ident::parse(&claim.agent).is_ok_and(|a| Coordinator::is_paused(&a)) {
return NodeOutput::default(); return;
} }
coord.mark_graceful_stop(&claim.agent); coord.mark_graceful_stop(&claim.agent);
coord.kick_agent(&claim.agent, "graceful stop requested"); coord.kick_agent(&claim.agent, "graceful stop requested");
NodeOutput::default()
} }
/// Await the harness clearing the fence (`GracefulStopComplete`) or /// Await the harness clearing the fence (`GracefulStopComplete`) or
/// the timeout — either way the downstream `Reconcile` proceeds with /// the timeout — either way the downstream `Reconcile` proceeds with
/// the actual stop. /// the actual stop.
async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
let deadline = std::time::Instant::now() + GRACEFUL_STOP_TIMEOUT; let deadline = std::time::Instant::now() + GRACEFUL_STOP_TIMEOUT;
while coord.is_graceful_stop_pending(name) { while coord.is_graceful_stop_pending(name) {
@ -536,11 +517,11 @@ async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput
tokio::time::sleep(std::time::Duration::from_millis(500)).await; tokio::time::sleep(std::time::Duration::from_millis(500)).await;
} }
coord.clear_graceful_stop(name); coord.clear_graceful_stop(name);
Ok(NodeOutput::default()) Ok(())
} }
/// `set_nspawn_flags` + `set_resource_limits` + daemon-reload. /// `set_nspawn_flags` + `set_resource_limits` + daemon-reload.
async fn run_write_dropin(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_write_dropin(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent; let name = &claim.agent;
// write_dropins only needs the path value to build AgentPaths; the // write_dropins only needs the path value to build AgentPaths; the
// dir doesn't need to exist at this point (created by ensure_agent_runtime_dir // dir doesn't need to exist at this point (created by ensure_agent_runtime_dir
@ -549,13 +530,13 @@ async fn run_write_dropin(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Nod
let hive = coord.hive_env(); let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir); let paths = Coordinator::agent_paths(name, agent_dir);
crate::lifecycle::write_dropins(name, &hive, &paths).await?; crate::lifecycle::write_dropins(name, &hive, &paths).await?;
Ok(NodeOutput::default()) Ok(())
} }
/// Write + commit the perm file(s) (fused under `META_LOCK` so the /// Write + commit the perm file(s) (fused under `META_LOCK` so the
/// working tree is never left dirty), then emit the P3RM1SS10NS-tab /// working tree is never left dirty), then emit the P3RM1SS10NS-tab
/// snapshots so the dashboard reflects the new assignment. /// snapshots so the dashboard reflects the new assignment.
async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
use super::model::PermPayload; use super::model::PermPayload;
let name = &claim.agent; let name = &claim.agent;
// The perm file payload rides the node itself (the only consumer). // The perm file payload rides the node itself (the only consumer).
@ -591,7 +572,7 @@ async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
} }
} }
} }
Ok(NodeOutput::default()) Ok(())
} }
/// Apply the node's `(child, new_parent)` moves as one `META_LOCK`-fused /// Apply the node's `(child, new_parent)` moves as one `META_LOCK`-fused
@ -601,7 +582,7 @@ async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
/// the deploy window (it declares `Resource::MetaWindow`), same reasoning as /// the deploy window (it declares `Resource::MetaWindow`), same reasoning as
/// `run_write_perm_file`: a topology commit landing inside another node's /// `run_write_perm_file`: a topology commit landing inside another node's
/// staged deploy window would sweep the staged lock into its commit. /// staged deploy window would sweep the staged lock into its commit.
async fn run_reparent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> { async fn run_reparent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let NodeKind::Reparent { moves } = &claim.kind else { let NodeKind::Reparent { moves } = &claim.kind else {
anyhow::bail!("run_reparent on a non-Reparent node"); anyhow::bail!("run_reparent on a non-Reparent node");
}; };
@ -618,16 +599,14 @@ async fn run_reparent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOut
.reparent_bulk_with_notify(&refs) .reparent_bulk_with_notify(&refs)
.await .await
.map_err(|e| anyhow::anyhow!(e))?; .map_err(|e| anyhow::anyhow!(e))?;
Ok(NodeOutput::default()) Ok(())
} }
/// Deploy phase 1 — drift gate, fetch, eval-verify. Mutates nothing, so a /// Deploy phase 1 — drift gate, fetch, eval-verify. Mutates nothing, so a
/// failure here cancel-cascades the rest of the subtree with the forge and the /// failure here cancel-cascades the rest of the subtree with the forge and the
/// applied repo exactly as they were. /// applied repo exactly as they were.
async fn run_merge_verify(coord: &Arc<Coordinator>, approval_id: i64) -> Result<NodeOutput> { async fn run_merge_verify(coord: &Arc<Coordinator>, approval_id: i64) -> Result<()> {
crate::actions::run_deploy_merge_verify(coord, approval_id) crate::actions::run_deploy_merge_verify(coord, approval_id).await
.await
.map(|()| NodeOutput::default())
} }
/// Deploy phase 2 — the irreversible half: ff-merge, then phase 1 of the /// Deploy phase 2 — the irreversible half: ff-merge, then phase 1 of the
@ -639,27 +618,15 @@ async fn run_merge_verify(coord: &Arc<Coordinator>, approval_id: i64) -> Result<
/// their `MetaSync` declares is re-entered rather than deadlocked against the /// their `MetaSync` declares is re-entered rather than deadlocked against the
/// ancestor already holding it. On failure nothing is appended and the tail /// ancestor already holding it. On failure nothing is appended and the tail
/// compensates, exactly as before. /// compensates, exactly as before.
async fn run_deploy_apply( async fn run_deploy_apply(coord: &Arc<Coordinator>, approval_id: i64) -> Result<()> {
coord: &Arc<Coordinator>, crate::actions::run_deploy_apply(coord, approval_id).await
claim: &Claim,
approval_id: i64,
) -> Result<NodeOutput> {
crate::actions::run_deploy_apply(coord, approval_id).await?;
Ok(NodeOutput {
append_subgraph: vec![super::templates::deploy_rebuild_nodes(
claim.kind.agent(),
approval_id,
)],
})
} }
/// Deploy phase 3 — close the staged-lock window once the appended rebuild has /// Deploy phase 3 — close the staged-lock window once the appended rebuild has
/// come up clean: drop the rollback ref, plant the `deployed/<id>` tag, commit /// come up clean: drop the rollback ref, plant the `deployed/<id>` tag, commit
/// the staged lock. /// the staged lock.
async fn run_finalize_deploy(coord: &Arc<Coordinator>, approval_id: i64) -> Result<NodeOutput> { async fn run_finalize_deploy(coord: &Arc<Coordinator>, approval_id: i64) -> Result<()> {
crate::actions::run_finalize_deploy(coord, approval_id) crate::actions::run_finalize_deploy(coord, approval_id).await
.await
.map(|()| NodeOutput::default())
} }
/// Deploy compensation + bookkeeping tail. `AfterAny` the apply node, so it /// Deploy compensation + bookkeeping tail. `AfterAny` the apply node, so it
@ -669,14 +636,10 @@ async fn run_finalize_deploy(coord: &Arc<Coordinator>, approval_id: i64) -> Resu
/// ///
/// Takes the agent from the node payload so the tail can still compensate when /// Takes the agent from the node payload so the tail can still compensate when
/// the approval row is gone (deny race, purge). /// the approval row is gone (deny race, purge).
async fn run_deploy_tail( async fn run_deploy_tail(coord: &Arc<Coordinator>, claim: &Claim, approval_id: i64) -> Result<()> {
coord: &Arc<Coordinator>,
claim: &Claim,
approval_id: i64,
) -> Result<NodeOutput> {
crate::actions::run_deploy_tail(coord, Some(claim.dag_id), claim.kind.agent(), approval_id) crate::actions::run_deploy_tail(coord, Some(claim.dag_id), claim.kind.agent(), approval_id)
.await; .await;
Ok(NodeOutput::default()) Ok(())
} }
/// Compute which agents a `nix flake update <inputs>` on the meta /// Compute which agents a `nix flake update <inputs>` on the meta

View file

@ -55,16 +55,6 @@ use resource::Resource;
/// borrowed one; only `hive_jobq` can make or insert it. /// borrowed one; only `hive_jobq` can make or insert it.
pub type Job = hive_jobq::JobBuilder<NodeKind, Resource>; pub type Job = hive_jobq::JobBuilder<NodeKind, Resource>;
/// A job's shape as a **recipe**: given a builder, declare the nodes.
///
/// What a template returns and what an executor hands back, because neither
/// can build a job itself — `hive_jobq` creates the builder inside its own
/// insertion call and never lets one out. So the transferable thing is the
/// declaring closure, and the queue runs it at the moment it inserts.
///
/// `Send` because an executor's output crosses the scheduler's task boundary.
pub type Declare = Box<dyn FnOnce(&Job) + Send>;
/// A handle to one node a template declared — where its edges, grouping and /// A handle to one node a template declared — where its edges, grouping and
/// resources are declared. `Copy`; naming a node as a dependency does not /// resources are declared. `Copy`; naming a node as a dependency does not
/// consume the ability to name it again. /// consume the ability to name it again.
@ -163,6 +153,18 @@ impl Default for JobQueue {
} }
} }
/// A node runner's `Result` as the scheduler's [`Outcome`].
///
/// The failure reason + `finished_at` are stamped onto the graph `Node` by the
/// scheduler (the reason rides `Outcome::Failed`); there is no host-side copy,
/// so nothing needs clearing on success.
fn outcome_of(result: Result<(), String>) -> Outcome {
match result {
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(truncate_error(&e)),
}
}
/// Insert a declared `job` into the shared graph and record its per-node /// Insert a declared `job` into the shared graph and record its per-node
/// `node_rt`, returning the inserted ids. /// `node_rt`, returning the inserted ids.
/// ///
@ -221,7 +223,7 @@ impl JobQueue {
/// roots re-parented to the container). Returns the container's id as the /// roots re-parented to the container). Returns the container's id as the
/// DAG id — its rolled-up state is the DAG state. /// DAG id — its rolled-up state is the DAG state.
/// ///
/// Takes the spec's recipe by generic, not as a boxed [`Declare`]: a spec /// Takes the spec's recipe by generic, not as a boxed closure: a spec
/// travels from the template that built it directly into this call, so /// travels from the template that built it directly into this call, so
/// there is nothing to allocate for. /// there is nothing to allocate for.
/// ///
@ -254,38 +256,6 @@ impl JobQueue {
Ok(container.get()) Ok(container.get())
} }
/// Append a whole *subgraph* into a live DAG at runtime — the single
/// in-DAG-growth primitive. The subgraph is inserted as a [`insert_group`]
/// rooted under `dep_on` (the emitting node): the subgraph's own root becomes
/// a *child* of `dep_on`, its steps children of that root, and the group's
/// agent lease is hoisted onto that root. Ordering root→`dep_on` is the parent
/// gate — the children run once `dep_on` reaches `Finishing`. Because the
/// emitting node stays `Finishing` until this appended subtree is terminal and
/// the DAG's terminal node deps on the top root, roll-up keeps the DAG from
/// settling early with no explicit wiring. A no-op if the DAG is gone.
pub fn append_subgraph(&self, dag_id: u64, declare: Declare, dep_on: NodeId) {
let mut inner = self.lock();
if inner.container(dag_id).is_none() {
return;
}
// Insert the subgraph as a group rooted under the emitting node: the
// subgraph's own root becomes a child of `dep_on`, its steps children of
// that root. No terminal-node wiring — roll-up carries terminality: the
// emitter stays `Finishing` until this appended subtree settles, and the
// container node rolls up terminal only once its whole subtree (incl. this
// appended work) has settled, so the DAG hook waits for free.
if let Err(e) = insert_group(&mut inner, declare, Some(dep_on)) {
tracing::error!(
dag = dag_id,
error = %e,
"job_queue: append_subgraph insert failed"
);
return;
}
drop(inner);
self.notify.notify_one();
}
/// Claim every currently-runnable node, acquiring its resources, and mark it /// Claim every currently-runnable node, acquiring its resources, and mark it
/// `Running`. Delegates readiness + resource acquisition to the crate's /// `Running`. Delegates readiness + resource acquisition to the crate's
/// settle loop; builds a [`Claim`] per started node from its payload + its /// settle loop; builds a [`Claim`] per started node from its payload + its
@ -325,15 +295,48 @@ impl JobQueue {
/// ([`NodeKind::ResolveApproval`] / [`NodeKind::EmitRebuilt`]), which the /// ([`NodeKind::ResolveApproval`] / [`NodeKind::EmitRebuilt`]), which the
/// scheduler claims and runs like any other node. /// scheduler claims and runs like any other node.
pub fn complete_node(&self, node_id: NodeId, result: Result<(), String>) { pub fn complete_node(&self, node_id: NodeId, result: Result<(), String>) {
// Deliberately not `complete_node_growing(.., self.new_job())`: that
// would take the lock twice (once to mint an empty builder, once to
// complete) to express "grew nothing". The shared part is the outcome
// mapping, and that's a free fn.
let mut inner = self.lock(); let mut inner = self.lock();
// The failure reason + `finished_at` are stamped onto the graph `Node` inner.sched.complete(node_id, outcome_of(result));
// by the scheduler (the reason rides `Outcome::Failed`); no host-side drop(inner);
// copy, so there is nothing to clear here. self.notify.notify_one();
let outcome = match result { }
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(truncate_error(&e)), /// A builder for a node to declare more work into while it runs.
}; ///
inner.sched.complete(node_id, outcome); /// Handed to [`exec::run_node`] and returned to
/// [`JobQueue::complete_node_growing`]. Only `hive_jobq` can construct one,
/// which is why this goes through the scheduler rather than
/// `Job::default()`.
#[must_use]
pub fn new_job(&self) -> Job {
self.lock().sched.new_job()
}
/// [`JobQueue::complete_node`] plus the work the node declared while it ran.
///
/// `grown` is inserted **under `node_id`** before the completion, so the DAG
/// cannot roll terminal with the appended work still pending — the property
/// the old two-call `append_subgraph` + `complete_node` sequence had to
/// arrange by hand at every call site.
pub fn complete_node_growing(&self, node_id: NodeId, result: Result<(), String>, grown: Job) {
let mut inner = self.lock();
// A rejected grown job is logged, not propagated: the node's own work
// already ran, and refusing to complete it here would both misreport
// that and wedge the DAG on a node stuck `Running`.
if let Err(e) = inner
.sched
.complete_growing(node_id, outcome_of(result), grown)
{
tracing::error!(
node = node_id.get(),
error = %e,
"job_queue: work grown by a completing node was rejected"
);
}
drop(inner); drop(inner);
self.notify.notify_one(); self.notify.notify_one();
} }

View file

@ -407,9 +407,9 @@ impl NodeKind {
/// Generic over the recipe rather than boxing it: a spec goes from the template /// Generic over the recipe rather than boxing it: a spec goes from the template
/// that returns it straight to the `submit` that consumes it, so the closure's /// that returns it straight to the `submit` that consumes it, so the closure's
/// concrete type is known the whole way and needs neither an allocation nor a /// concrete type is known the whole way and needs neither an allocation nor a
/// `Send` bound. (The executor's `append_subgraph` is the case that *does* need /// `Send` bound. Nothing boxes a recipe any more — a running node grows its DAG
/// a boxed [`super::Declare`] — its recipes are collected into a `Vec` and /// by declaring straight onto the builder it was handed, so there is no recipe
/// applied later, across a task boundary.) /// to store and replay across a task boundary.
pub struct DagSpec<F> { pub struct DagSpec<F> {
pub source: Source, pub source: Source,
/// Free-form "why". /// Free-form "why".

View file

@ -16,19 +16,22 @@
//! the DAG settles. //! the DAG settles.
//! //!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning //! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning
//! its `Start`/`Stop`) flows through `NodeOutput.append_subgraph`, applied //! its `Start`/`Stop`) is declared onto the builder each node is handed, and
//! before the emitting node completes — see `handle_completion`. //! inserted as part of completing that node — see `handle_completion`.
use std::collections::HashMap; use std::collections::HashMap;
use std::sync::Arc; use std::sync::Arc;
use super::Claim; use super::exec;
use super::exec::{self, NodeOutput}; use super::{Claim, Job};
use crate::coordinator::Coordinator; use crate::coordinator::Coordinator;
struct NodeDone { struct NodeDone {
claim: Claim, claim: Claim,
result: anyhow::Result<NodeOutput>, /// Whatever the node declared into its builder while running — usually
/// nothing. Inserted under the node as part of completing it.
grown: Job,
result: anyhow::Result<()>,
} }
/// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`. /// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`.
@ -83,9 +86,18 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
let coord = Arc::clone(&coord); let coord = Arc::clone(&coord);
let tx = tx.clone(); let tx = tx.clone();
tokio::spawn(async move { tokio::spawn(async move {
let result = exec::run_node(&coord, &claim).await; // The node's growth channel. Local state, so it costs
// nothing to carry and holds no lock while the node runs.
// The builder is passed by value and handed back: owned it
// is `Send`, a `&Job` held across an await is not.
let job = coord.job_queue.new_job();
let (grown, result) = exec::run_node(&coord, job, &claim).await;
// Send failure = scheduler gone (shutdown); drop. // Send failure = scheduler gone (shutdown); drop.
let _ = tx.send(NodeDone { claim, result }); let _ = tx.send(NodeDone {
claim,
grown,
result,
});
}); });
} }
// Newly-started owner nodes now hold their leases — surface the pills. // Newly-started owner nodes now hold their leases — surface the pills.
@ -110,27 +122,26 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
} }
fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) { fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) {
let NodeDone { claim, result } = done; let NodeDone {
claim,
grown,
result,
} = done;
match result { match result {
Ok(output) => { Ok(()) => {
tracing::info!( tracing::info!(
dag = claim.dag_id, dag = claim.dag_id,
node = claim.node_id.get(), node = claim.node_id.get(),
"job_queue: node done" "job_queue: node done"
); );
// Append any in-DAG subgraphs BEFORE completing this node, so // Whatever the node declared goes in under it as part of this
// completing it doesn't roll the DAG terminal while the appended // completion, so the DAG cannot roll terminal while the appended
// work is still pending. Each subgraph roots on this node // work is still pending. Covers the multi-node case (a `MetaLock`
// (`AfterOk`), so it becomes ready the instant this one settles
// `Done` just below — covers both the multi-node case (a `MetaLock`
// growing per-agent rebuild subgraphs) and the single-node case (a // growing per-agent rebuild subgraphs) and the single-node case (a
// `Reconcile` planner's `Start` / `Stop`). // `Reconcile` planner's `Start` / `Stop`) identically.
for subgraph in output.append_subgraph { coord
coord .job_queue
.job_queue .complete_node_growing(claim.node_id, Ok(()), grown);
.append_subgraph(claim.dag_id, subgraph, claim.node_id);
}
coord.job_queue.complete_node(claim.node_id, Ok(()));
} }
Err(e) => { Err(e) => {
let msg = format!("{e:#}"); let msg = format!("{e:#}");
@ -140,8 +151,17 @@ fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) {
kind = claim.kind.as_str(), kind = claim.kind.as_str(),
agent = %claim.agent, agent = %claim.agent,
error = %msg, error = %msg,
grown_nodes = !grown.is_empty(),
"job_queue: node failed" "job_queue: node failed"
); );
// `grown` is deliberately dropped on failure. A node that declared
// follow-up work and *then* failed does not want that work run —
// failure cancel-cascades downstream, so inserting it would only
// add nodes to immediately cancel. This preserves the old shape,
// where growth could only be expressed on the success path at all;
// the difference is that it is now possible to declare and then
// fail, so the drop has to be a decision rather than an accident.
drop(grown);
coord.job_queue.complete_node(claim.node_id, Err(msg)); coord.job_queue.complete_node(claim.node_id, Err(msg));
} }
} }

View file

@ -28,7 +28,7 @@ use hive_jobq::TerminalState;
use super::model::{DagSpec, NodeKind, PermPayload, Source}; use super::model::{DagSpec, NodeKind, PermPayload, Source};
use super::resource::Resource; use super::resource::Resource;
use super::{Declare, Handle, Job}; use super::{Handle, Job};
/// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node /// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node
/// gated on every group-root in `roots`, and the failure node gated on *its* /// gated on every group-root in `roots`, and the failure node gated on *its*
@ -235,32 +235,29 @@ pub(crate) fn rebuild_nodes<'a>(
/// finalized. `Reconcile` alone would not do, being `AfterAny` — it reaches /// finalized. `Reconcile` alone would not do, being `AfterAny` — it reaches
/// `Done` even after a failed `Swap`. /// `Done` even after a failed `Swap`.
/// ///
/// Appended, not submitted: the roots below become children of the emitting /// Declared into a **running** `DeployApply`'s own builder, not submitted: the
/// `DeployApply` (see [`super::JobQueue::append_subgraph`]), which puts them /// roots below become children of that node, which puts them inside the
/// inside the `DeployWindow`'s subtree — so the `MetaWindow` this subgraph's /// `DeployWindow`'s subtree — so the `MetaWindow` this subgraph's `MetaSync`
/// `MetaSync` and `FinalizeDeploy` declare is re-entered from the ancestor /// and `FinalizeDeploy` declare is re-entered from the ancestor already holding
/// already holding it rather than deadlocking against it. /// it rather than deadlocking against it.
pub(crate) fn deploy_rebuild_nodes(agent: &str, approval_id: i64) -> Declare { pub(crate) fn deploy_rebuild_nodes(b: &Job, agent: &str, approval_id: i64) {
let agent = agent.to_owned(); let roots = rebuild_nodes(
Box::new(move |b: &Job| { b,
let roots = rebuild_nodes( agent,
b, RebuildOpts {
&agent, relock: false,
RebuildOpts { graceful: false,
relock: false, },
graceful: false, None,
}, );
None, let _finalize = b
); .node(NodeKind::FinalizeDeploy {
let _finalize = b agent: agent.to_owned(),
.node(NodeKind::FinalizeDeploy { approval_id,
agent: agent.clone(), })
approval_id, .needs(Resource::MetaWindow)
}) .after_ok(roots.prebuild)
.needs(Resource::MetaWindow) .after_ok(roots.reconcile);
.after_ok(roots.prebuild)
.after_ok(roots.reconcile);
})
} }
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate` /// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
@ -480,8 +477,8 @@ pub fn perm_change(
} }
/// Meta-input lock bump. The `MetaLock` executor grows one rebuild subgraph /// Meta-input lock bump. The `MetaLock` executor grows one rebuild subgraph
/// per affected agent into *this same* DAG on completion (via /// per affected agent into *this same* DAG on completion (declared onto the
/// `append_subgraph`) — appended *after* the bump lands so their prebuilds /// builder it was handed) — appended *after* the bump lands so their prebuilds
/// run against the post-bump lock, and a failed bump appends nothing /// run against the post-bump lock, and a failed bump appends nothing
/// (replacing the old fan-out-child-DAGs dance). /// (replacing the old fan-out-child-DAGs dance).
/// `transient = Rebuilding` because those appended subgraphs are rebuilds: /// `transient = Rebuilding` because those appended subgraphs are rebuilds:

View file

@ -13,13 +13,19 @@ fn submit<F: FnOnce(&Job)>(q: &JobQueue, spec: DagSpec<F>) -> u64 {
q.submit(spec).expect("valid spec") q.submit(spec).expect("valid spec")
} }
/// Erase a spec's recipe to the boxed [`Declare`] so specs of *different* /// A spec recipe with its concrete closure type erased. **Test-only** — the
/// shapes can share one type — e.g. a table of `(name, spec)` cases. /// module used to export this alias for the executor's growth path too, which
/// is exactly what a node declaring onto its own builder removed: nothing in
/// production stores a recipe to replay later, so nothing needs to box one.
type ErasedRecipe = Box<dyn FnOnce(&Job) + Send>;
/// Erase a spec's recipe so specs of *different* shapes can share one type —
/// e.g. a table of `(name, spec)` cases.
/// ///
/// Production never needs this: each submit path builds one spec and hands it /// Production never needs this: each submit path builds one spec and hands it
/// straight to `submit`, so the concrete closure type is known end to end. A /// straight to `submit`, so the concrete closure type is known end to end. A
/// test table is the case where several shapes must be one type. /// test table is the one case where several shapes must be one type.
fn erase<F: FnOnce(&Job) + Send + 'static>(spec: DagSpec<F>) -> DagSpec<Declare> { fn erase<F: FnOnce(&Job) + Send + 'static>(spec: DagSpec<F>) -> DagSpec<ErasedRecipe> {
DagSpec { DagSpec {
source: spec.source, source: spec.source,
reason: spec.reason, reason: spec.reason,
@ -758,19 +764,16 @@ fn a_fanned_out_start_declares_the_lease_and_re_enters_its_reconciles_grant() {
assert_eq!(reconcile.dag_id, id); assert_eq!(reconcile.dag_id, id);
assert_eq!(reconcile.kind.as_str(), "reconcile"); assert_eq!(reconcile.kind.as_str(), "reconcile");
// What `run_reconcile` does on observing a down container with wanted=Up. // What `run_reconcile` does on observing a down container with wanted=Up:
q.append_subgraph( // declare into the builder it was handed, then hand it back with the
id, // completion. Same two calls the scheduler makes, in the same order.
Box::new(|b: &Job| { let grown = q.new_job();
let kind = NodeKind::Start { let kind = NodeKind::Start {
agent: "agent-a".to_owned(), agent: "agent-a".to_owned(),
}; };
let lease = Resource::Agent(kind.agent().to_owned()); let lease = Resource::Agent(kind.agent().to_owned());
let _ = b.node(kind).needs(lease); let _ = grown.node(kind).needs(lease);
}), q.complete_node_growing(reconcile.node_id, Ok(()), grown);
reconcile.node_id,
);
q.complete_node(reconcile.node_id, Ok(()));
// (a) + (b): the child runs, under the parent that parked in `Finishing`. // (a) + (b): the child runs, under the parent that parked in `Finishing`.
let start = claim_one(&q); let start = claim_one(&q);
@ -850,7 +853,7 @@ fn boot_sweep_nodes_declare_their_own_resources() {
} }
#[test] #[test]
fn append_subgraph_roots_on_emitter_and_rebases_local_deps() { fn grown_subgraph_roots_on_emitter_and_rebases_local_deps() {
// The startup-sweep mechanism: a `MetaLock` emitter grows one rebuild // The startup-sweep mechanism: a `MetaLock` emitter grows one rebuild
// subgraph per stale agent into its OWN DAG. Each subgraph is rooted on // subgraph per stale agent into its OWN DAG. Each subgraph is rooted on
// the emitter and its LOCAL 0-based deps are rebased onto the DAG. // the emitter and its LOCAL 0-based deps are rebased onto the DAG.
@ -866,31 +869,29 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
}); });
}), }),
}; };
let id = submit(&q, spec); submit(&q, spec);
let emitter = claim_one(&q); let emitter = claim_one(&q);
assert_eq!(emitter.kind.as_str(), "meta_lock"); assert_eq!(emitter.kind.as_str(), "meta_lock");
// Two independent per-agent subgraphs — the REAL production shape the // Two independent per-agent subgraphs — the REAL production shape the
// sweep MetaLock grows: root MetaSync → root Prebuild → Signal → Drain → // sweep MetaLock grows: root MetaSync → root Prebuild → Signal → Drain →
// StopForUpdate → Swap → Reconcile, local 0-based deps. `graceful` must // StopForUpdate → Swap → Reconcile, local 0-based deps. `graceful` must
// match the sweep arm of `run_meta_lock` or this stops tracking production. // match the sweep arm of `run_meta_lock` or this stops tracking production.
let subgraph = |agent: &str| -> Declare { // Both subgraphs go into the emitter's own builder, exactly as
let agent = agent.to_owned(); // `run_meta_lock`'s sweep arm does. Insert-before-complete is no longer the
Box::new(move |b: &Job| { // caller's job to remember: it is one call, and the ordering is inside it.
templates::rebuild_nodes( let grown = q.new_job();
b, for agent in ["a", "b"] {
&agent, templates::rebuild_nodes(
templates::RebuildOpts { &grown,
relock: true, agent,
graceful: true, templates::RebuildOpts {
}, relock: true,
None, graceful: true,
); },
}) None,
}; );
// Must append BEFORE completing the emitter (the documented contract). }
q.append_subgraph(id, subgraph("a"), emitter.node_id); q.complete_node_growing(emitter.node_id, Ok(()), grown);
q.append_subgraph(id, subgraph("b"), emitter.node_id);
q.complete_node(emitter.node_id, Ok(()));
// Still ONE DAG; both subgraph roots become ready once the emitter is // Still ONE DAG; both subgraph roots become ready once the emitter is
// Done (rooted on it), each on its own agent lease. Their `MetaSync` heads // Done (rooted on it), each on its own agent lease. Their `MetaSync` heads
// take turns on the cap-1 global meta window, so drain those first — what // take turns on the cap-1 global meta window, so drain those first — what
@ -967,7 +968,7 @@ fn rebuild_chain_nodes_suppress_crash_watch() {
#[test] #[test]
fn meta_update_grows_cascade_in_dag() { fn meta_update_grows_cascade_in_dag() {
// The meta-update `MetaLock` grows one rebuild subgraph per affected // The meta-update `MetaLock` grows one rebuild subgraph per affected
// agent into its OWN DAG (via append_subgraph), not child DAGs. // agent into its OWN DAG (via the builder it is handed), not child DAGs.
let spec = templates::meta_update( let spec = templates::meta_update(
vec!["nixpkgs".to_owned()], vec!["nixpkgs".to_owned()],
Source::Manual, Source::Manual,
@ -975,26 +976,26 @@ fn meta_update_grows_cascade_in_dag() {
None, None,
); );
let q = JobQueue::new(4); let q = JobQueue::new(4);
let id = submit(&q, spec); submit(&q, spec);
let meta_lock = claim_one(&q); let meta_lock = claim_one(&q);
assert_eq!(meta_lock.kind.as_str(), "meta_lock"); assert_eq!(meta_lock.kind.as_str(), "meta_lock");
// Simulate the executor growing the cascade in-DAG (`relock = false` — a // Simulate the executor growing the cascade in-DAG (`relock = false` — a
// cascade child must not re-lock and revert the parent's bump). // cascade child must not re-lock and revert the parent's bump). Both
// agents go into the one builder the node was handed, which is what
// `run_meta_lock`'s fanout arm does.
let grown = q.new_job();
for agent in ["alice", "bob"] { for agent in ["alice", "bob"] {
let declare: Declare = Box::new(move |b: &Job| { templates::rebuild_nodes(
templates::rebuild_nodes( &grown,
b, agent,
agent, templates::RebuildOpts {
templates::RebuildOpts { relock: false,
relock: false, graceful: false,
graceful: false, },
}, None,
None, );
);
});
q.append_subgraph(id, declare, meta_lock.node_id);
} }
q.complete_node(meta_lock.node_id, Ok(())); q.complete_node_growing(meta_lock.node_id, Ok(()), grown);
// Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root // Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root
// on the MetaLock, each on its own agent lease. The per-agent `MetaSync` // on the MetaLock, each on its own agent lease. The per-agent `MetaSync`
// heads serialize on the global meta window (they commit to the meta repo); // heads serialize on the global meta window (they commit to the meta repo);
@ -1233,8 +1234,8 @@ fn cancelled_power_op_runs_no_compensating_node() {
for graceful in [false, true] { for graceful in [false, true] {
for running in [false, true] { for running in [false, true] {
let targets = vec![("agent-a".to_owned(), running)]; let targets = vec![("agent-a".to_owned(), running)];
// Erased to `DagSpec<Declare>`: three different recipe types have to // Erased to one boxed recipe type: three different recipe types
// sit in one array. // have to sit in one array.
let cases = [ let cases = [
( (
"restart", "restart",
@ -1420,16 +1421,14 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
let apply = claim_one(&q); let apply = claim_one(&q);
assert!(matches!(apply.kind, NodeKind::DeployApply { .. })); assert!(matches!(apply.kind, NodeKind::DeployApply { .. }));
// Mirrors the scheduler: the executor's `NodeOutput` subgraphs are grafted // Mirrors the scheduler. The graft lands BEFORE the emitting node settles,
// BEFORE the emitting node is completed. Completing first would settle the // and that ordering is now structural rather than a rule this call site has
// apply node `Done` with nothing under it, opening the tail's `AfterAny` // to follow: completing first would settle the apply node `Done` with
// gate immediately and letting the deploy "finish" before it had built. // nothing under it, opening the tail's `AfterAny` gate immediately and
q.append_subgraph( // letting the deploy "finish" before it had built.
id, let grown = q.new_job();
templates::deploy_rebuild_nodes("agent-a", 11), templates::deploy_rebuild_nodes(&grown, "agent-a", 11);
apply.node_id, q.complete_node_growing(apply.node_id, Ok(()), grown);
);
q.complete_node(apply.node_id, Ok(()));
// The grafted chain runs in rebuild order. `claim_one` asserts exactly one // The grafted chain runs in rebuild order. `claim_one` asserts exactly one
// claimable node at each step, which also proves the `AfterAny` tail stays // claimable node at each step, which also proves the `AfterAny` tail stays
@ -1481,12 +1480,9 @@ fn deploy_dag_skips_finalize_but_still_tails_a_failed_graft() {
let verify = claim_one(&q); let verify = claim_one(&q);
q.complete_node(verify.node_id, Ok(())); q.complete_node(verify.node_id, Ok(()));
let apply = claim_one(&q); let apply = claim_one(&q);
q.append_subgraph( let grown = q.new_job();
id, templates::deploy_rebuild_nodes(&grown, "agent-a", 13);
templates::deploy_rebuild_nodes("agent-a", 13), q.complete_node_growing(apply.node_id, Ok(()), grown);
apply.node_id,
);
q.complete_node(apply.node_id, Ok(()));
for expected in ["meta_sync", "prebuild", "stop_for_update"] { for expected in ["meta_sync", "prebuild", "stop_for_update"] {
let c = claim_one(&q); let c = claim_one(&q);

View file

@ -391,8 +391,7 @@ fn submit_boot_tree(
n_skipped, n_skipped,
); );
let declare: crate::job_queue::Declare = let declare = move |b: &crate::job_queue::Job| boot_nodes(b, any_stale, fanout, drifted);
Box::new(move |b| boot_nodes(b, any_stale, fanout, drifted));
let spec = DagSpec { let spec = DagSpec {
// The sweep's own rebuild subgraphs emit their `Rebuilt` events as they // The sweep's own rebuild subgraphs emit their `Rebuilt` events as they

View file

@ -251,6 +251,72 @@ impl<N, R: Clone + Eq + Hash> Scheduler<N, R> {
/// propagates up the parent chain. Call [`Scheduler::settle`] again afterwards /// propagates up the parent chain. Call [`Scheduler::settle`] again afterwards
/// to start newly-unblocked work. /// to start newly-unblocked work.
pub fn complete(&mut self, id: NodeId, outcome: Outcome) { pub fn complete(&mut self, id: NodeId, outcome: Outcome) {
self.finish(id, outcome);
}
/// A fresh builder for a **running** node to declare more work into.
///
/// The node runs outside this scheduler's lock — often for minutes — so it
/// cannot hold a graph reference while it works. It doesn't need one: a
/// builder is pure local state (locally-minted guids, resolved to
/// [`NodeId`]s only at insert), so it can be filled in freely and handed
/// back to [`Scheduler::complete_growing`], which inserts it under the lock.
///
/// This is the only way to get one — [`JobBuilder::new`] is `pub(crate)` and
/// there is no `Default` impl — so a caller can declare work but never
/// insert it itself.
#[must_use]
pub fn new_job(&self) -> JobBuilder<N, R> {
JobBuilder::new()
}
/// [`Scheduler::complete`], plus whatever the node declared into the builder
/// it was handed while running.
///
/// `grown`'s nodes are inserted **under `id`** and *before* the completion,
/// so the node cannot roll terminal with its own appended work still
/// pending — the same ordering the caller previously had to arrange by
/// hand. A job that declares nothing costs nothing: the insert is skipped
/// outright, which is the overwhelmingly common case (most nodes grow no
/// work at all).
///
/// # Errors
/// [`BuildError`] if `grown` is malformed — **and the node is still
/// completed**. Its own work already happened; refusing to complete it
/// would misreport that, and leaving it `Running` forever would wedge the
/// DAG. So the error is returned for the caller to log, not used to abort
/// the completion. This crate has no logger of its own; the caller does.
pub fn complete_growing(
&mut self,
id: NodeId,
outcome: Outcome,
grown: JobBuilder<N, R>,
) -> Result<(), BuildError> {
// A node that is no longer in the graph grows nothing. The DAG it
// belonged to can be cancelled or evicted while it runs, and the insert
// below is *unchecked* — rooting on a departed parent would plant a
// dangling `parent` edge rather than being rejected. The host used to
// carry this guard itself, as a lookup before a separate append call;
// it belongs here, where the graph is and where it cannot be skipped.
let grew = if grown.is_empty() || self.graph.node(id).is_none() {
Ok(())
} else {
let graph = &mut self.graph;
grown
.insert_with(Some(id), &[], |payload, deps, parent| {
graph.insert_unchecked(payload, deps, parent)
})
.map(|_ids| ())
};
self.finish(id, outcome);
grew
}
/// The completion half, shared by [`Scheduler::complete`] and
/// [`Scheduler::complete_growing`] so neither is a redirect through the
/// other: the growing form must insert *before* this runs, and the plain
/// form must not pay for an empty job.
fn finish(&mut self, id: NodeId, outcome: Outcome) {
match outcome { match outcome {
Outcome::Failed(error) => { Outcome::Failed(error) => {
// Record the reason before the terminal transition so it's set // Record the reason before the terminal transition so it's set