vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: drop Claim, claim_ready and the completion wrappers

Browse source 
c0re's run loop now goes through hive_jobq's claim_next seam, so the
host layer no longer needs its own claim/complete vocabulary.

exec::run_node takes (NodeId, &NodeKind) instead of a &Claim snapshot.
The agent already rides the payload, and the DAG id is a derived read
(JobQueue::dag_of) that only three arms want, so it is taken per-arm
rather than eagerly for every node. Two arms (WritePermFile, Reparent)
re-matched the kind behind a bail! that could never fire; the match arm
already destructures the payload, so they take it directly now.

Deleted from the c0re layer:
  - struct Claim
  - JobQueue::claim_ready
  - JobQueue::complete_node / complete_node_growing
  - scheduler::NodeDone / handle_completion

Completion happens inside the future claim_next hands back, so "ran the
node but forgot to complete it" is not expressible on the production
path any more. The node done / node failed logging moved with it -- it
lived in handle_completion but is not dead code.

claim_ready and the completion wrappers were left with no non-test
callers, so the tests carry them as ClaimReady / CompleteNode extension
traits over the crate primitives. JobQueue::new_job stays: run_worker
still mints an empty builder on a failed outcome.
This commit is contained in:
atlas 2026-08-02 19:01:53 +02:00 • committed by mara
commit ab53f6710d
4 changed files with 263 additions and 255 deletions
Download patch file Download diff file Expand all files Collapse all files

164
hive-c0re/src/job_queue/exec.rs
View file

@ -10,8 +10,7 @@ use std::sync::Arc;
use anyhow::{Context as _, Result};
use super::Claim;
use hive_jobq::TerminalState;
use hive_jobq::{NodeId, TerminalState};
use super::model::NodeKind;
use super::resource::Resource;
@ -43,22 +42,31 @@ pub const GRACEFUL_STOP_TIMEOUT: std::time::Duration = std::time::Duration::from
/// future non-`Send`, which the scheduler's `tokio::spawn` rejects. So the
/// growth executors below return *what to grow* and the declaration happens
/// here, synchronously, between awaits.
///
/// The node is identified by its own id + payload rather than by a `Claim`
/// side-struct: `kind` already carries the agent, and the DAG id is a
/// derived read (`JobQueue::dag_of`) the three arms that need it take
/// themselves. Nothing here needs a claim to exist as a type.
pub(super) async fn run_node(
coord: &Arc<Coordinator>,
job: super::Job,
claim: &Claim,
id: NodeId,
kind: &NodeKind,
) -> (super::Job, Result<()>) {
// The agent this node targets rides the payload — empty for the agentless
// container kinds (`MetaLock`, `Dag`), which never read it.
let agent = kind.agent();
// Every arm is `Result<()>`; the three that grow work declare into `job`
// *synchronously*, after their own awaits have finished. Borrowing `&job`
// inside an `.await` would make this future non-`Send` (see above), so the
// growth executors return what to grow rather than taking the builder.
let result = match &claim.kind {
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(claim).await,
NodeKind::Swap { .. } => run_swap(coord, claim).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, claim).await,
NodeKind::Provision { .. } => run_provision(coord, claim).await,
NodeKind::Create { .. } => run_create(claim).await,
let result = match kind {
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, agent, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(agent, id).await,
NodeKind::Swap { .. } => run_swap(coord, agent, id).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, agent).await,
NodeKind::Provision { .. } => run_provision(coord, agent).await,
NodeKind::Create { .. } => run_create(agent).await,
NodeKind::MetaLock {
sweep,
fanout,
@ -70,7 +78,7 @@ pub(super) async fn run_node(
super::templates::rebuild_nodes(&job, &agent, opts, None);
}
}),
NodeKind::Reconcile { .. } => run_reconcile(coord, claim).await.map(|sub| {
NodeKind::Reconcile { .. } => run_reconcile(coord, agent).await.map(|sub| {
if let Some(kind) = sub {
// `Start` / `Stop` declare the lease they run under. This node
// is their parent and holds it, so the declaration is a
@ -81,38 +89,41 @@ pub(super) async fn run_node(
let _ = job.node(kind).needs(lease);
}
}),
NodeKind::Start { .. } => run_start(coord, claim).await,
NodeKind::Stop { .. } => run_stop(coord, claim).await,
NodeKind::StopForUpdate { .. } => run_stop_for_update(coord, claim).await,
NodeKind::Start { .. } => run_start(coord, agent).await,
NodeKind::Stop { .. } => run_stop(coord, agent).await,
NodeKind::StopForUpdate { .. } => run_stop_for_update(coord, agent).await,
NodeKind::Signal { .. } => {
run_signal(coord, claim);
run_signal(coord, agent);
Ok(())
}
NodeKind::Drain { .. } => run_drain(coord, claim).await,
NodeKind::WriteDropin { .. } => run_write_dropin(coord, claim).await,
NodeKind::WritePermFile { .. } => run_write_perm_file(coord, claim).await,
NodeKind::Reparent { .. } => run_reparent(coord, claim).await,
NodeKind::Drain { .. } => run_drain(coord, agent).await,
NodeKind::WriteDropin { .. } => run_write_dropin(coord, agent).await,
// The payload rides the node and is destructured here, so the executor
// takes it directly instead of re-matching the kind behind a `bail!`
// that could never fire.
NodeKind::WritePermFile { payload, .. } => run_write_perm_file(coord, agent, payload).await,
NodeKind::Reparent { moves } => run_reparent(coord, moves).await,
NodeKind::MergeVerify { approval_id, .. } => run_merge_verify(coord, *approval_id).await,
NodeKind::DeployApply { approval_id, .. } => {
run_deploy_apply(coord, *approval_id).await.map(|()| {
super::templates::deploy_rebuild_nodes(&job, claim.kind.agent(), *approval_id);
super::templates::deploy_rebuild_nodes(&job, agent, *approval_id);
})
}
NodeKind::FinalizeDeploy { approval_id, .. } => {
run_finalize_deploy(coord, *approval_id).await
}
NodeKind::DeployTail { approval_id, .. } => {
run_deploy_tail(coord, claim, *approval_id).await
run_deploy_tail(coord, coord.job_queue.dag_of(id), agent, *approval_id).await
}
NodeKind::ResolveApproval {
approval_id,
outcome,
} => run_resolve_approval(coord, claim, *approval_id, *outcome).await,
} => run_resolve_approval(coord, coord.job_queue.dag_of(id), *approval_id, *outcome).await,
NodeKind::EmitRebuilt { ok, .. } => {
run_emit_rebuilt(coord, claim, *ok);
run_emit_rebuilt(coord, agent, coord.job_queue.dag_of(id), *ok);
Ok(())
}
NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up),
NodeKind::SetWanted { up, .. } => run_set_wanted(coord, agent, *up),
// The two nodes that carry no work of their own; completing either
// lets it reach `Finishing` so the nodes under it start.
// - `Dag`: pure grouping container. The DAG's terminal side effect, if
@ -133,12 +144,12 @@ pub(super) async fn run_node(
/// since the work already happened and failing the tail would only misreport it.
async fn run_resolve_approval(
coord: &Arc<Coordinator>,
claim: &Claim,
dag_id: Option<u64>,
approval_id: i64,
outcome: TerminalState,
) -> Result<()> {
let reason = (outcome == TerminalState::Failed)
.then(|| coord.job_queue.first_error(claim.dag_id))
.then(|| dag_id.and_then(|dag| coord.job_queue.first_error(dag)))
.flatten();
crate::actions::resolve_approval_dag(coord, approval_id, outcome, reason.as_deref()).await;
Ok(())
@ -147,12 +158,12 @@ async fn run_resolve_approval(
/// Emit this agent's `Rebuilt` manager event. `ok` is not computed — it is which
/// of the tail pair the graph let run. The failure note comes from the DAG's
/// first failing node, since the branch knows *that* it failed but not *why*.
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) {
fn run_emit_rebuilt(coord: &Arc<Coordinator>, agent: &str, dag_id: Option<u64>, ok: bool) {
coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: claim.agent.clone(),
agent: agent.to_owned(),
ok,
note: (!ok)
.then(|| coord.job_queue.first_error(claim.dag_id))
.then(|| dag_id.and_then(|dag| coord.job_queue.first_error(dag)))
.flatten(),
sha: None,
tag: None,
@ -167,7 +178,7 @@ fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim, ok: bool) {
/// warn-and-continue write, a failed write fails the node (cancel-downstream
/// cancels the `Reconcile`) rather than letting it converge to a stale
/// intent — that atomicity is the point of moving it into the DAG.
fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<()> {
fn run_set_wanted(coord: &Arc<Coordinator>, agent: &str, up: bool) -> Result<()> {
let wanted = if up {
crate::power::Wanted::Up
} else {
@ -175,8 +186,8 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<(
};
coord
.power
.set(&claim.agent, wanted)
.with_context(|| format!("set wanted={} for agent {}", wanted.as_str(), claim.agent))?;
.set(agent, wanted)
.with_context(|| format!("set wanted={} for agent {agent}", wanted.as_str()))?;
Ok(())
}
@ -189,8 +200,7 @@ fn run_set_wanted(coord: &Arc<Coordinator>, claim: &Claim, up: bool) -> Result<(
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that
/// build takes minutes and only *reads* the store, so keeping the global
/// window off it is what lets rebuilds of different agents overlap.
async fn run_meta_sync(coord: &Arc<Coordinator>, claim: &Claim, relock: bool) -> Result<()> {
let name = &claim.agent;
async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Result<()> {
// Runs while the agent is still up — the runtime dir and MCP listener
// already exist. Use the pure path accessor; no need to re-register the
// listener (event-driven: registered at start/create).
@ -217,15 +227,14 @@ async fn run_meta_sync(coord: &Arc<Coordinator>, claim: &Claim, relock: bool) ->
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead.
async fn run_prebuild(claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_prebuild(name: &str, id: NodeId) -> Result<()> {
// Warm the toplevel build only when the container is up — the whole
// point of prebuild is to shrink the swap's downtime window. A
// stopped agent has no uptime to preserve, so skip the (expensive)
// eval and let the downstream `Swap` build inline.
if crate::lifecycle::is_running(name).await {
let flake_ref = format!("{}#{name}", crate::paths::meta_root().display());
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(claim.node_id.get())).await?;
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?;
}
Ok(())
}
@ -235,15 +244,13 @@ async fn run_prebuild(claim: &Claim) -> Result<()> {
/// (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan).
/// The recovery-start on failure is NOT here — the DAG's tail
/// `Reconcile` runs after this node terminal ok *or* fail.
async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()> {
// Swap runs on an already-existing (stopped) container — runtime dir
// and listener were created earlier. Pure path accessor suffices.
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
let result =
crate::lifecycle::swap_update(name, &hive, &paths, Some(claim.node_id.get())).await;
let result = crate::lifecycle::swap_update(name, &hive, &paths, Some(id.get())).await;
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix
// sync, kick, rescan, snapshot) runs in the sibling `PostSwap` node,
// which deps `AfterOk(Swap)`. On failure `PostSwap` is cancel-cascaded
@ -262,8 +269,7 @@ async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
/// means the profile swap succeeded. Store/forge/matrix work only — no nix
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held
/// (the whole chain up to `Reconcile` is one agent's subgraph).
async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_post_swap(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake)
&& let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev)
{
@ -290,8 +296,7 @@ async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
/// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (it declares `Resource::MetaWindow`) so its commit can't
/// land inside another node's staged deploy window.
async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_provision(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
@ -305,8 +310,8 @@ async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
/// dir creation and MCP listener registration are deferred to the tail
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
/// node stays purely "create", not "create + start".
async fn run_create(claim: &Claim) -> Result<()> {
crate::lifecycle::create_only(&claim.agent).await?;
async fn run_create(name: &str) -> Result<()> {
crate::lifecycle::create_only(name).await?;
Ok(())
}
@ -377,18 +382,17 @@ async fn run_meta_lock(
/// guard) rides across it.
/// Returns the mechanical node to fan out (`None` on a noop) rather than
/// declaring it — the declaration has to happen outside any `.await`, see
/// [`run_node`]. `NodeKind` carries the agent it targets, so `claim.agent` is
/// stamped into the kind here.
async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Option<NodeKind>> {
let name = &claim.agent;
/// [`run_node`]. `NodeKind` carries the agent it targets, so this node's agent
/// is stamped into the fanned-out kind here.
async fn run_reconcile(coord: &Arc<Coordinator>, name: &str) -> Result<Option<NodeKind>> {
let running = crate::lifecycle::is_running(name).await;
let wanted = coord.power.get_or_seed(name, running)?;
Ok(match reconcile_action(wanted, running) {
ReconcileAction::Start => Some(NodeKind::Start {
agent: name.clone(),
agent: name.to_owned(),
}),
ReconcileAction::Stop => Some(NodeKind::Stop {
agent: name.clone(),
agent: name.to_owned(),
}),
ReconcileAction::Noop => {
tracing::debug!(%name, wanted = wanted.as_str(), running, "reconcile: noop");
@ -399,8 +403,7 @@ async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Option
/// Mechanical container start — the sub-step a `Reconcile` planner fans
/// out when it observes `wanted = Up` and the container down.
async fn run_start(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_start(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
// No node-local transient guard: the pill is derived from the running node
// set, and `Start` reports `Starting` via `NodeKind::transient_kind`. This
// used to take one "only when the DAG holds none", which was a second
@ -428,14 +431,13 @@ async fn run_start(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
/// Mechanical container stop — the sub-step a `Reconcile` planner fans
/// out when it observes `wanted = Offline` and the container up.
async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_stop(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
// See `run_start`: no node-local guard — `Stop` reports `Stopping` from its
// own kind now.
crate::lifecycle::kill(name).await?;
coord.unregister_agent(name);
coord.notify_manager(&hive_sh4re::HelperEvent::Killed {
agent: name.clone(),
agent: name.to_owned(),
});
coord.rescan_containers_and_emit().await;
Ok(())
@ -443,8 +445,7 @@ async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
/// Mechanical stop for the profile swap. Never *changes* `wanted`;
/// noop when already stopped.
async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_stop_for_update(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
if crate::lifecycle::is_running(name).await {
// Seed a missing agent_power row from the PRE-stop observation
// — the DAG's tail `Reconcile` observes only the mechanically
@ -468,19 +469,18 @@ async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
/// `GRACEFUL_STOP_TIMEOUT`. Safe because the harness tests the marker at
/// the top of its loop — a paused agent has no turn in flight, so there
/// is nothing to checkpoint.
fn run_signal(coord: &Arc<Coordinator>, claim: &Claim) {
if hive_types::Ident::parse(&claim.agent).is_ok_and(|a| Coordinator::is_paused(&a)) {
fn run_signal(coord: &Arc<Coordinator>, name: &str) {
if hive_types::Ident::parse(name).is_ok_and(|a| Coordinator::is_paused(&a)) {
return;
}
coord.mark_graceful_stop(&claim.agent);
coord.kick_agent(&claim.agent, "graceful stop requested");
coord.mark_graceful_stop(name);
coord.kick_agent(name, "graceful stop requested");
}
/// Await the harness clearing the fence (`GracefulStopComplete`) or
/// the timeout — either way the downstream `Reconcile` proceeds with
/// the actual stop.
async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_drain(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
let deadline = std::time::Instant::now() + GRACEFUL_STOP_TIMEOUT;
while coord.is_graceful_stop_pending(name) {
if std::time::Instant::now() >= deadline {
@ -494,8 +494,7 @@ async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
}
/// `set_nspawn_flags` + `set_resource_limits` + daemon-reload.
async fn run_write_dropin(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let name = &claim.agent;
async fn run_write_dropin(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
// write_dropins only needs the path value to build AgentPaths; the
// dir doesn't need to exist at this point (created by ensure_agent_runtime_dir
// on the upstream Prebuild/Start node).
@ -509,13 +508,12 @@ async fn run_write_dropin(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()>
/// Write + commit the perm file(s) (fused under `META_LOCK` so the
/// working tree is never left dirty), then emit the P3RM1SS10NS-tab
/// snapshots so the dashboard reflects the new assignment.
async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
async fn run_write_perm_file(
coord: &Arc<Coordinator>,
name: &str,
payload: &super::model::PermPayload,
) -> Result<()> {
use super::model::PermPayload;
let name = &claim.agent;
// The perm file payload rides the node itself (the only consumer).
let NodeKind::WritePermFile { payload, .. } = &claim.kind else {
anyhow::bail!("run_write_perm_file on a non-WritePermFile node");
};
// Runs under the deploy window (it declares `Resource::MetaWindow`): a
// perm commit landing inside another node's staged prepare→finalize
// window would sweep the staged deploy lock into its commit (the
@ -555,10 +553,10 @@ async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
/// the deploy window (it declares `Resource::MetaWindow`), same reasoning as
/// `run_write_perm_file`: a topology commit landing inside another node's
/// staged deploy window would sweep the staged lock into its commit.
async fn run_reparent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<()> {
let NodeKind::Reparent { moves } = &claim.kind else {
anyhow::bail!("run_reparent on a non-Reparent node");
};
async fn run_reparent(
coord: &Arc<Coordinator>,
moves: &[(hive_types::Ident, Option<hive_types::Ident>)],
) -> Result<()> {
let refs: Vec<(&str, Option<&str>)> = moves
.iter()
.map(|(child, parent)| {
@ -609,9 +607,13 @@ async fn run_finalize_deploy(coord: &Arc<Coordinator>, approval_id: i64) -> Resu
///
/// Takes the agent from the node payload so the tail can still compensate when
/// the approval row is gone (deny race, purge).
async fn run_deploy_tail(coord: &Arc<Coordinator>, claim: &Claim, approval_id: i64) -> Result<()> {
crate::actions::run_deploy_tail(coord, Some(claim.dag_id), claim.kind.agent(), approval_id)
.await;
async fn run_deploy_tail(
coord: &Arc<Coordinator>,
dag_id: Option<u64>,
agent: &str,
approval_id: i64,
) -> Result<()> {
crate::actions::run_deploy_tail(coord, dag_id, agent, approval_id).await;
Ok(())
}