vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2591): port hive-c0re job_queue onto the hive-jobq crate

Browse source 
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).

Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
  sibling ordering directly (deps order execution, parent groups a subtree
  whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
  for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
  agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
  separate top-level root (AfterAny Prebuild) so it survives the cancel-
  cascade of any failed step (recovery-start invariant) and converges to
  the persisted `wanted` on a fresh lease. This is the multi-root
  correction to the single-root-chain sketch: node0=root broke lease-
  exemption (hoisting the lease onto Prebuild) and recovery-reconcile
  (root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
  same way; per-agent power-op subgraphs stay independent roots so a
  multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
  DAG terminal node deps AfterAny on every group root and runs once the
  whole op rolls up. Drop the old Graph::add_dep terminal wiring.

36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
This commit is contained in:
atlas 2026-07-20 21:46:08 +02:00 • committed by mara
commit a5c321a1a0
14 changed files with 1111 additions and 893 deletions
Download patch file Download diff file Expand all files Collapse all files

134
hive-c0re/src/job_queue/exec.rs
View file

@ -10,8 +10,8 @@ use std::sync::Arc;
use anyhow::{Context as _, Result};
use super::model::{NodeKind, NodeSpec, State, Template};
use super::{Claim, TerminalDag};
use super::Claim;
use super::model::{NodeKind, NodeSpec, State};
use crate::coordinator::Coordinator;
use crate::power::{ReconcileAction, reconcile_action};
@ -100,9 +100,74 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
NodeKind::WritePermFile => run_write_perm_file(coord, claim, &ctx).await,
NodeKind::ApprovalDeploy => run_approval_deploy(coord, claim).await,
NodeKind::SetWanted { up } => run_set_wanted(coord, claim, *up),
NodeKind::ResolveApproval => run_resolve_approval(coord, claim).await,
NodeKind::EmitRebuilt => Ok(run_emit_rebuilt(coord, claim)),
NodeKind::RevertIntent => run_revert_intent(coord, claim).await,
}
}
/// Terminal hook (approval DAGs — spawn / opaque deploy): resolve the DAG's
/// approval row from its rolled-up outcome. Its own graph node, weak-dep on the
/// DAG tails, so it runs once everything has settled (any outcome, incl. a
/// cancel before starting — the fallback that resolves a queued-then-cancelled
/// approval whose node never ran). Always succeeds — a hook failure is logged
/// inside, not surfaced as a node failure.
async fn run_resolve_approval(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
if let Some(terminal) = coord.job_queue.terminal_summary(claim.dag_id) {
crate::actions::resolve_approval_dag(coord, &terminal).await;
}
Ok(NodeOutput::default())
}
/// Terminal hook (rebuild / perm-change DAGs): emit one `Rebuilt` manager event
/// per targeted agent — `ok` on `Done`, `!ok` on `Failed`, none on cancel.
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim) -> NodeOutput {
if let Some(terminal) = coord.job_queue.terminal_summary(claim.dag_id) {
for agent in &terminal.agents {
match terminal.state {
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: true,
note: None,
sha: None,
tag: None,
}),
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: false,
note: terminal.error.clone(),
sha: None,
tag: None,
}),
_ => {}
}
}
}
NodeOutput::default()
}
/// Terminal hook (power-op DAGs): on a *cancelled* DAG, revert each targeted
/// agent's `wanted` intent to its observed state — the operator's cancel means
/// "don't do it", so the intent snaps back instead of the flip executing as a
/// surprise side effect of some later reconcile. Noop on any non-cancelled
/// outcome. Always succeeds — a revert failure is logged, not surfaced.
async fn run_revert_intent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
if let Some(terminal) = coord.job_queue.terminal_summary(claim.dag_id)
&& terminal.state == State::Cancelled
{
for agent in &terminal.agents {
let running = crate::lifecycle::is_running(agent).await;
if let Err(e) = coord
.power
.set(agent, crate::power::Wanted::from_running(running))
{
tracing::warn!(%agent, error = ?e, "agent_power: cancel revert failed");
}
}
}
Ok(NodeOutput::default())
}
/// Write the agent's durable power intent — the DAG-node form of the old
/// pre-submit `set_wanted` side effect. Store-only (no container touch), so
/// build-slot-exempt; but it takes the agent's lifecycle lease (see
@ -531,71 +596,6 @@ async fn run_approval_deploy(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
.map(|()| NodeOutput::default())
}
/// Terminal-roll-up hook, fired exactly once per DAG (node completion
/// and cancel paths alike — the queue buffers roll-ups and the
/// scheduler drains them). Three concerns:
/// - approval DAGs resolve their approval row (except the opaque
/// deploy pipeline, which resolves inside its node — unless it was
/// cancelled while still queued and the node never ran);
/// - non-approval rebuild-shaped DAGs emit exactly one `Rebuilt`
/// manager event: ok on `Done`, !ok on `Failed`, none on cancel;
/// - a cancelled power-op DAG reverts the `wanted` intent its submit
/// wrote: the operator's cancel means "don't do it", so intent
/// snaps back to the observed state instead of the flip executing
/// as a surprise side effect of some later reconcile.
pub(super) async fn on_dag_terminal(coord: &Arc<Coordinator>, terminal: &TerminalDag) {
if terminal.state == State::Cancelled
&& matches!(
terminal.template,
Template::Start
| Template::Stop
| Template::GracefulStop
| Template::Restart
| Template::GracefulRestart
)
{
// Revert each targeted agent's power intent to its observed state —
// the operator's cancel means "don't do it". Single-agent power-op
// DAGs have one agent here.
for agent in &terminal.agents {
let running = crate::lifecycle::is_running(agent).await;
if let Err(e) = coord
.power
.set(agent, crate::power::Wanted::from_running(running))
{
tracing::warn!(%agent, error = ?e, "agent_power: cancel revert failed");
}
}
}
if terminal.approval_id.is_some() {
crate::actions::resolve_approval_dag(coord, terminal).await;
return;
}
if matches!(terminal.template, Template::Rebuild | Template::PermChange) {
// Rebuild / PermChange are single-agent; emit one `Rebuilt` per
// targeted agent (exactly one today).
for agent in &terminal.agents {
match terminal.state {
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: true,
note: None,
sha: None,
tag: None,
}),
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: false,
note: terminal.error.clone(),
sha: None,
tag: None,
}),
_ => {}
}
}
}
}
/// Compute which agents a `nix flake update <inputs>` on the meta
/// flake affects — the fan-out set for `MetaUpdate` DAGs. Empty
/// `inputs` or any input under `hyperhive` → every container;