vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#2591): port hive-c0re job_queue onto the hive-jobq crate

Browse source 
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).

Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
  sibling ordering directly (deps order execution, parent groups a subtree
  whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
  for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
  agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
  separate top-level root (AfterAny Prebuild) so it survives the cancel-
  cascade of any failed step (recovery-start invariant) and converges to
  the persisted `wanted` on a fresh lease. This is the multi-root
  correction to the single-root-chain sketch: node0=root broke lease-
  exemption (hoisting the lease onto Prebuild) and recovery-reconcile
  (root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
  same way; per-agent power-op subgraphs stay independent roots so a
  multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
  DAG terminal node deps AfterAny on every group root and runs once the
  whole op rolls up. Drop the old Graph::add_dep terminal wiring.

36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
This commit is contained in:
atlas 2026-07-20 21:46:08 +02:00 • committed by mara
commit a5c321a1a0
14 changed files with 1111 additions and 893 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Cargo.lock generated
View file

@ -1620,6 +1620,7 @@ dependencies = [
"forgejo-api",
"hive-core-agent-sock",
"hive-host-sock",
"hive-jobq",
"hive-priv-sock",
"hive-sh4re",
"hive-types",

1
Cargo.toml
View file

@ -53,6 +53,7 @@ clap_complete = "4"
indicatif = "0.18"
hive-sh4re = { path = "hive-sh4re" }
hive-agent-sock = { path = "hive-agent-sock" }
hive-jobq = { path = "hive-jobq" }
hive-core-agent-sock = { path = "hive-core-agent-sock" }
hive-claude = { path = "hive-claude" }
hive-host-sock = { path = "hive-host-sock" }

1
hive-c0re/Cargo.toml
View file

@ -33,6 +33,7 @@ indicatif.workspace = true
hive-core-agent-sock.workspace = true
hive-sh4re.workspace = true
hive-host-sock.workspace = true
hive-jobq.workspace = true
hive-priv-sock.workspace = true
hive-types.workspace = true
libc.workspace = true

134
hive-c0re/src/job_queue/exec.rs
View file

@ -10,8 +10,8 @@ use std::sync::Arc;
use anyhow::{Context as _, Result};
use super::model::{NodeKind, NodeSpec, State, Template};
use super::{Claim, TerminalDag};
use super::Claim;
use super::model::{NodeKind, NodeSpec, State};
use crate::coordinator::Coordinator;
use crate::power::{ReconcileAction, reconcile_action};
@ -100,9 +100,74 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
NodeKind::WritePermFile => run_write_perm_file(coord, claim, &ctx).await,
NodeKind::ApprovalDeploy => run_approval_deploy(coord, claim).await,
NodeKind::SetWanted { up } => run_set_wanted(coord, claim, *up),
NodeKind::ResolveApproval => run_resolve_approval(coord, claim).await,
NodeKind::EmitRebuilt => Ok(run_emit_rebuilt(coord, claim)),
NodeKind::RevertIntent => run_revert_intent(coord, claim).await,
}
}
/// Terminal hook (approval DAGs — spawn / opaque deploy): resolve the DAG's
/// approval row from its rolled-up outcome. Its own graph node, weak-dep on the
/// DAG tails, so it runs once everything has settled (any outcome, incl. a
/// cancel before starting — the fallback that resolves a queued-then-cancelled
/// approval whose node never ran). Always succeeds — a hook failure is logged
/// inside, not surfaced as a node failure.
async fn run_resolve_approval(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
if let Some(terminal) = coord.job_queue.terminal_summary(claim.dag_id) {
crate::actions::resolve_approval_dag(coord, &terminal).await;
}
Ok(NodeOutput::default())
}
/// Terminal hook (rebuild / perm-change DAGs): emit one `Rebuilt` manager event
/// per targeted agent — `ok` on `Done`, `!ok` on `Failed`, none on cancel.
fn run_emit_rebuilt(coord: &Arc<Coordinator>, claim: &Claim) -> NodeOutput {
if let Some(terminal) = coord.job_queue.terminal_summary(claim.dag_id) {
for agent in &terminal.agents {
match terminal.state {
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: true,
note: None,
sha: None,
tag: None,
}),
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: false,
note: terminal.error.clone(),
sha: None,
tag: None,
}),
_ => {}
}
}
}
NodeOutput::default()
}
/// Terminal hook (power-op DAGs): on a *cancelled* DAG, revert each targeted
/// agent's `wanted` intent to its observed state — the operator's cancel means
/// "don't do it", so the intent snaps back instead of the flip executing as a
/// surprise side effect of some later reconcile. Noop on any non-cancelled
/// outcome. Always succeeds — a revert failure is logged, not surfaced.
async fn run_revert_intent(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
if let Some(terminal) = coord.job_queue.terminal_summary(claim.dag_id)
&& terminal.state == State::Cancelled
{
for agent in &terminal.agents {
let running = crate::lifecycle::is_running(agent).await;
if let Err(e) = coord
.power
.set(agent, crate::power::Wanted::from_running(running))
{
tracing::warn!(%agent, error = ?e, "agent_power: cancel revert failed");
}
}
}
Ok(NodeOutput::default())
}
/// Write the agent's durable power intent — the DAG-node form of the old
/// pre-submit `set_wanted` side effect. Store-only (no container touch), so
/// build-slot-exempt; but it takes the agent's lifecycle lease (see
@ -531,71 +596,6 @@ async fn run_approval_deploy(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
.map(|()| NodeOutput::default())
}
/// Terminal-roll-up hook, fired exactly once per DAG (node completion
/// and cancel paths alike — the queue buffers roll-ups and the
/// scheduler drains them). Three concerns:
/// - approval DAGs resolve their approval row (except the opaque
/// deploy pipeline, which resolves inside its node — unless it was
/// cancelled while still queued and the node never ran);
/// - non-approval rebuild-shaped DAGs emit exactly one `Rebuilt`
/// manager event: ok on `Done`, !ok on `Failed`, none on cancel;
/// - a cancelled power-op DAG reverts the `wanted` intent its submit
/// wrote: the operator's cancel means "don't do it", so intent
/// snaps back to the observed state instead of the flip executing
/// as a surprise side effect of some later reconcile.
pub(super) async fn on_dag_terminal(coord: &Arc<Coordinator>, terminal: &TerminalDag) {
if terminal.state == State::Cancelled
&& matches!(
terminal.template,
Template::Start
| Template::Stop
| Template::GracefulStop
| Template::Restart
| Template::GracefulRestart
)
{
// Revert each targeted agent's power intent to its observed state —
// the operator's cancel means "don't do it". Single-agent power-op
// DAGs have one agent here.
for agent in &terminal.agents {
let running = crate::lifecycle::is_running(agent).await;
if let Err(e) = coord
.power
.set(agent, crate::power::Wanted::from_running(running))
{
tracing::warn!(%agent, error = ?e, "agent_power: cancel revert failed");
}
}
}
if terminal.approval_id.is_some() {
crate::actions::resolve_approval_dag(coord, terminal).await;
return;
}
if matches!(terminal.template, Template::Rebuild | Template::PermChange) {
// Rebuild / PermChange are single-agent; emit one `Rebuilt` per
// targeted agent (exactly one today).
for agent in &terminal.agents {
match terminal.state {
State::Done => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: true,
note: None,
sha: None,
tag: None,
}),
State::Failed => coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: agent.clone(),
ok: false,
note: terminal.error.clone(),
sha: None,
tag: None,
}),
_ => {}
}
}
}
}
/// Compute which agents a `nix flake update <inputs>` on the meta
/// flake affects — the fan-out set for `MetaUpdate` DAGs. Empty
/// `inputs` or any input under `hyperhive` → every container;

1214
hive-c0re/src/job_queue/mod.rs
View file

File diff suppressed because it is too large Load diff

195
hive-c0re/src/job_queue/model.rs
View file

@ -11,7 +11,7 @@
//! DAG can span agents). See `docs/coordinator.md::Job queue` for the
//! full design.
pub use hive_sh4re::jobs::{DagView, NodeId, NodeView, PermPayload, Source, State, Template};
pub use hive_sh4re::jobs::{DagView, NodeId, PermPayload, Source, State, Template};
use serde::Serialize;
/// When a dependency edge is considered satisfied.
@ -139,6 +139,21 @@ pub enum NodeKind {
/// `Prebuild`, but that's a no-op there — the agent is down, so prebuild
/// is skipped.)
SetWanted { up: bool },
/// Per-DAG terminal hook (approval-driven DAGs — spawn / opaque deploy):
/// resolve the DAG's approval row from the rolled-up outcome. Appended once
/// with a weak (`AfterAny`) edge on the DAG's tails, so it runs exactly when
/// the DAG has settled (any outcome, including a cancel before starting).
/// Build-slot- and lease-exempt; always runs (weak edge ⇒ never cascaded).
ResolveApproval,
/// Per-DAG terminal hook (rebuild / perm-change DAGs): emit one `Rebuilt`
/// manager event per targeted agent — `ok` on `Done`, `!ok` on `Failed`,
/// none on cancel. Appended weak-dep on the tails; slot/lease-exempt.
EmitRebuilt,
/// Per-DAG terminal hook (power-op DAGs): on a *cancelled* DAG, revert each
/// agent's `wanted` intent to its observed state — the operator's cancel
/// means "don't do it". Noop on any non-cancelled outcome. Appended weak-dep
/// on the tails; slot/lease-exempt.
RevertIntent,
}
impl NodeKind {
@ -161,6 +176,9 @@ impl NodeKind {
NodeKind::WritePermFile => "write_perm_file",
NodeKind::ApprovalDeploy => "approval_deploy",
NodeKind::SetWanted { .. } => "set_wanted",
NodeKind::ResolveApproval => "resolve_approval",
NodeKind::EmitRebuilt => "emit_rebuilt",
NodeKind::RevertIntent => "revert_intent",
}
}
@ -201,39 +219,23 @@ impl NodeKind {
}
}
/// One schedulable unit inside a DAG.
#[derive(Debug, Clone)]
pub struct Node {
pub id: NodeId,
/// The agent this node's work targets. Per-node so a single DAG can
/// span agents (e.g. a hive-wide restart); the lifecycle lease is
/// acquired against *this* agent (still globally exclusive per agent
/// across all DAGs). `"hyperhive"` for meta-level nodes.
pub agent: String,
pub kind: NodeKind,
pub deps: Vec<Dep>,
pub state: State,
/// Live sub-label while `Running` (kept for parity with the old
/// per-entry `step`).
pub step: Option<String>,
/// Row id of the `build_logs` entry this node opened (`Prebuild` /
/// `Swap` / `ApprovalDeploy`), for the dashboard's live-stream link.
pub build_log_id: Option<i64>,
pub started_at: Option<i64>,
pub finished_at: Option<i64>,
/// Populated when `state == Failed` (truncated by the queue).
pub error: Option<String>,
}
/// Submit-time spec for one node.
#[derive(Debug, Clone)]
pub struct NodeSpec {
/// The agent this node targets (see [`Node::agent`]). Built by the
/// `templates.rs` `node` helper, which stamps the template's agent
/// onto every node.
/// The agent this node's work targets. Built by the `templates.rs` `node`
/// helper, which stamps the template's agent onto every node.
pub agent: String,
pub kind: NodeKind,
pub deps: Vec<Dep>,
/// The **structural parent** axis — the spec-local index of this node's
/// group parent, or `None` for a top-level (group-root) node. Independent
/// of `deps`: `deps` order execution, `parent` groups nodes into a subtree
/// whose resource the whole subtree borrows (the agent lease is owned by a
/// group root and re-entered by its descendants for continuity). A child
/// runs once its parent reaches `Finishing` (the parent gate), so a child
/// never `deps` on its own parent (that would deadlock — dep-scope
/// validation rejects it).
pub parent: Option<u64>,
}
/// Submit-time spec for a whole DAG. Built by `templates.rs`; validated
@ -258,140 +260,3 @@ pub struct DagSpec {
pub transient: Option<crate::coordinator::TransientKind>,
pub nodes: Vec<NodeSpec>,
}
/// A live DAG in the queue. No DAG-level `agent`: agent is per-[`Node`],
/// so a DAG can span agents. Per-agent leasing is derived from the
/// nodes' agents.
#[derive(Debug, Clone)]
pub struct Dag {
pub id: u64,
pub template: Template,
pub source: Source,
pub reason: String,
pub approval_id: Option<i64>,
pub inputs: Vec<String>,
pub perm_payload: Option<PermPayload>,
pub transient: Option<crate::coordinator::TransientKind>,
pub created_at: i64,
pub nodes: Vec<Node>,
/// Terminal roll-up already reported to the scheduler's hooks
/// (approval resolution, transient release). Internal bookkeeping,
/// never serialized.
pub terminal_reported: bool,
}
impl Dag {
/// Roll-up state: `Failed` if any node failed; else `Running` if
/// any running; else `Queued` if any queued; else `Cancelled` if
/// any cancelled; else `Done`.
pub fn rollup(&self) -> State {
let mut any_cancelled = false;
let mut any_queued = false;
let mut any_running = false;
for n in &self.nodes {
match n.state {
State::Failed => return State::Failed,
State::Running => any_running = true,
State::Queued => any_queued = true,
State::Cancelled => any_cancelled = true,
State::Done => {}
}
}
if any_running {
State::Running
} else if any_queued {
State::Queued
} else if any_cancelled {
State::Cancelled
} else {
State::Done
}
}
/// True when every node is terminal.
pub fn is_terminal(&self) -> bool {
self.nodes.iter().all(|n| n.state.is_terminal())
}
/// True when no live (non-terminal) node of this DAG still targets
/// `agent` — i.e. that agent's subgraph within the DAG has settled.
/// Used to release an agent's lifecycle lease the moment its own
/// work is done, rather than waiting for the whole DAG to terminate.
/// Vacuously true for an agent the DAG has no node for; callers gate
/// on actually holding that agent's lease first.
pub fn agent_subgraph_terminal(&self, agent: &str) -> bool {
self.nodes
.iter()
.filter(|n| n.agent == agent)
.all(|n| n.state.is_terminal())
}
/// First failed node's error, for the roll-up `error` field.
pub fn first_error(&self) -> Option<&str> {
self.nodes
.iter()
.find(|n| n.state == State::Failed)
.and_then(|n| n.error.as_deref())
}
pub fn node(&self, id: NodeId) -> Option<&Node> {
self.nodes.iter().find(|n| n.id == id)
}
pub fn node_mut(&mut self, id: NodeId) -> Option<&mut Node> {
self.nodes.iter_mut().find(|n| n.id == id)
}
/// Distinct agents this DAG's nodes target, in first-seen order.
/// Used for per-agent lease release and the terminal cancel-revert —
/// a single-agent DAG yields one, a multi-agent DAG yields several.
pub fn agents(&self) -> Vec<String> {
let mut seen: Vec<String> = Vec::new();
for n in &self.nodes {
if !seen.iter().any(|a| a == &n.agent) {
seen.push(n.agent.clone());
}
}
seen
}
}
impl Dag {
pub fn view(&self) -> DagView {
let started_at = self.nodes.iter().filter_map(|n| n.started_at).min();
let finished_at = if self.is_terminal() {
self.nodes.iter().filter_map(|n| n.finished_at).max()
} else {
None
};
DagView {
id: self.id,
kind: self.template,
state: self.rollup(),
source: self.source,
reason: self.reason.clone(),
enqueued_at: self.created_at,
started_at,
finished_at,
inputs: self.inputs.clone(),
approval_id: self.approval_id,
perm_payload: self.perm_payload.clone(),
nodes: self
.nodes
.iter()
.map(|n| NodeView {
id: n.id,
agent: n.agent.clone(),
kind: n.kind.as_str().to_owned(),
deps: n.deps.iter().map(|d| d.on).collect(),
state: n.state,
step: n.step.clone(),
build_log_id: n.build_log_id,
started_at: n.started_at,
finished_at: n.finished_at,
error: n.error.clone(),
})
.collect(),
}
}
}

60
hive-c0re/src/job_queue/resource.rs Normal file
View file

@ -0,0 +1,60 @@
//! The concrete resource + payload types the rebuild queue schedules over —
//! the bridge from hive-c0re's [`NodeKind`] onto the domain-agnostic
//! `hive-jobq` crate. `hive-jobq` is generic over a resource type
//! `R: Clone + Eq + Hash` and a node payload `N`; here `R` is [`Resource`] and
//! `N` is [`JobPayload`].
use hive_jobq::Dep;
use super::model::NodeKind;
/// The two resource classes the queue gates concurrency on, as the crate's
/// generic resource type `R`.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum Resource {
/// One of the `buildSlots` permits, held by a nix-heavy node for its
/// duration. Capacity is `services.hyperhive.c0re.buildSlots` (default 1),
/// set on the [`hive_jobq::resources::ResourceTable`] at construction.
BuildSlot,
/// The per-agent lifecycle lease — globally exclusive per agent across all
/// DAGs (unconfigured, so the crate's default capacity 1 applies). Held by
/// a DAG's first container-affecting node for that agent and re-entered by
/// the rest of that agent's subtree via the crate's recursive lock, so two
/// DAGs never interleave container ops on one agent.
Agent(String),
}
/// A schedulable node's payload — the crate's generic `N`. Carries the
/// primitive operation and the agent it targets. The agent is per-node (a DAG
/// spans agents), and the lease [`Resource::Agent`] is keyed on it.
#[derive(Debug, Clone)]
pub struct JobPayload {
pub kind: NodeKind,
pub agent: String,
}
impl JobPayload {
/// The [`Dep::Resource`] edges this node must acquire to run, derived from
/// its kind + agent: a build slot for nix-heavy kinds
/// ([`NodeKind::needs_build_slot`]) and the agent lease for
/// container-affecting kinds ([`NodeKind::needs_lease`]). Lease-exempt
/// container ops (`Start` / `Stop`, fanned out by a lease-holding
/// `Reconcile`) hold no lease of their own — they re-enter the ancestor's
/// `Agent` lock through the crate's recursive re-entrancy.
pub fn resource_deps(&self) -> Vec<Dep<Resource>> {
let mut deps = Vec::new();
if self.kind.needs_build_slot() {
deps.push(Dep::Resource {
name: Resource::BuildSlot,
count: 1,
});
}
if self.kind.needs_lease() {
deps.push(Dep::Resource {
name: Resource::Agent(self.agent.clone()),
count: 1,
});
}
deps
}
}

119
hive-c0re/src/job_queue/scheduler.rs
View file

@ -1,18 +1,25 @@
//! The single scheduler task that drives all DAGs: claim every ready
//! node (as many as the build slots / leases allow), spawn one
//! executor task per claim, and on any completion re-evaluate.
//! Concurrency comes from the build-slot count, not multiple workers.
//! The single scheduler task that drives all DAGs: claim every ready node (as
//! many as the build slots / leases allow), spawn one executor task per claim,
//! and on any completion re-evaluate. Concurrency comes from the build-slot
//! count, not multiple workers.
//!
//! Also owns the per-DAG transient guard (dashboard pill + crash-watch
//! suppression) that the sync queue core can't hold itself — created when a
//! DAG acquires its agent lease, dropped when the DAG settles terminal.
//! Owns the per-DAG transient guard (dashboard pill + crash-watch suppression)
//! that the sync queue core can't hold itself. The guard set is *reconciled*
//! from live lease ownership ([`super::JobQueue::held_transients`]) each loop:
//! a `(dag, agent)` pill exists for exactly as long as that agent's lease is
//! held, so it appears when the agent's owner node starts and disappears when
//! its subgraph settles — one pill per agent a DAG touches.
//!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs after the lock
//! bump, a `Reconcile` fanning its `Start`/`Stop`) flows through
//! `NodeOutput.append_subgraph`, applied before the emitting node
//! completes — see `handle_completion`.
//! Per-DAG terminal work (approval resolution, `Rebuilt`, cancelled-power-op
//! intent revert) is not drained here: it runs as the DAG's focused terminal
//! node (`ResolveApproval` / `EmitRebuilt` / `RevertIntent`), dispatched through
//! `exec::run_node` like any other node once the DAG settles.
//!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning
//! its `Start`/`Stop`) flows through `NodeOutput.append_subgraph`, applied
//! before the emitting node completes — see `handle_completion`.
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use super::Claim;
@ -26,38 +33,24 @@ struct NodeDone {
/// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`.
///
/// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true
/// signal the loop exits immediately; already-running node tasks ride
/// the runtime down with the process, and pending `Queued` DAGs are
/// dropped — desired state is re-derived on next boot (boot sweep +
/// reconcile), so the in-memory queue is deliberately not durable.
/// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true signal
/// the loop exits immediately; already-running node tasks ride the runtime down
/// with the process, and pending `Queued` DAGs are dropped — desired state is
/// re-derived on next boot (boot sweep + reconcile), so the in-memory queue is
/// deliberately not durable.
pub async fn run_worker(coord: Arc<Coordinator>) {
let mut shutdown = coord.shutdown_rx();
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<NodeDone>();
// (DAG id, agent) → transient guard held for that agent's lease
// window. Keyed per-agent so a multi-agent DAG shows one transient
// pill per agent it touches.
// (DAG id, agent) → transient guard held for that agent's lease window.
let mut transients: HashMap<(u64, String), crate::coordinator::TransientGuard> = HashMap::new();
loop {
// Terminal roll-ups can appear without a node completion —
// the cancel surfaces settle DAGs directly and wake this loop
// via notify — so drain on every iteration, not just inside
// handle_completion.
process_terminals(&coord, &mut transients).await;
reconcile_transients(&coord, &mut transients);
let claims = coord.job_queue.claim_ready();
if !claims.is_empty() {
for claim in claims {
if claim.lease_acquired
&& let Some(kind) = claim.transient
{
transients.insert(
(claim.dag_id, claim.agent.clone()),
coord.transient_guard(&claim.agent, kind),
);
}
tracing::info!(
dag = claim.dag_id,
node = claim.node_id,
node = claim.node_id.get(),
kind = claim.kind.as_str(),
agent = %claim.agent,
template = claim.template.as_str(),
@ -71,6 +64,8 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
let _ = tx.send(NodeDone { claim, result });
});
}
// Newly-started owner nodes now hold their leases — surface the pills.
reconcile_transients(&coord, &mut transients);
coord.emit_rebuild_queue_snapshot();
continue;
}
@ -83,35 +78,30 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
}
}
Some(done) = rx.recv() => {
handle_completion(&coord, &mut transients, done).await;
handle_completion(&coord, done);
}
() = coord.job_queue.notify.notified() => {}
}
}
}
async fn handle_completion(
coord: &Arc<Coordinator>,
transients: &mut HashMap<(u64, String), crate::coordinator::TransientGuard>,
done: NodeDone,
) {
fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) {
let NodeDone { claim, result } = done;
match result {
Ok(output) => {
tracing::info!(
dag = claim.dag_id,
node = claim.node_id,
node = claim.node_id.get(),
"job_queue: node done"
);
// Append any in-DAG subgraphs BEFORE completing this node, so
// completing it doesn't roll the DAG terminal while the appended
// work is still pending — that keeps the lease-window transient
// held across it. Each subgraph is independent, rooted on this
// node (`AfterOk`), so it becomes ready the instant this one
// settles `Done` just below. Covers both the multi-node case (a
// `MetaLock` growing per-agent rebuild subgraphs) and the
// single-node case (a `Reconcile` planner's `Start` / `Stop`).
for subgraph in output.append_subgraph {
// work is still pending. Each subgraph roots on this node
// (`AfterOk`), so it becomes ready the instant this one settles
// `Done` just below — covers both the multi-node case (a `MetaLock`
// growing per-agent rebuild subgraphs) and the single-node case (a
// `Reconcile` planner's `Start` / `Stop`).
for subgraph in &output.append_subgraph {
coord
.job_queue
.append_subgraph(claim.dag_id, subgraph, claim.node_id);
@ -124,7 +114,7 @@ async fn handle_completion(
let msg = format!("{e:#}");
tracing::warn!(
dag = claim.dag_id,
node = claim.node_id,
node = claim.node_id.get(),
kind = claim.kind.as_str(),
agent = %claim.agent,
error = %msg,
@ -135,30 +125,23 @@ async fn handle_completion(
.complete_node(claim.dag_id, claim.node_id, Err(msg));
}
}
process_terminals(coord, transients).await;
// The next loop iteration re-reconciles the transient pills against the
// post-completion lease state (a settled subgraph drops its pill).
coord.emit_rebuild_queue_snapshot();
}
/// Drain per-agent lease releases and buffered terminal roll-ups.
///
/// Per-agent first: an agent's subgraph within a DAG went terminal (its
/// lease was freed in `settle`), so drop that agent's `(dag, agent)`
/// transient pill now — ahead of whole-DAG terminal for a multi-agent
/// DAG. Then the whole-DAG terminals: drop any remaining transient the
/// DAG still held and run the terminal hook (approval resolution,
/// `Rebuilt` events, cancelled-power-op intent revert).
async fn process_terminals(
/// Reconcile the transient-guard set against live lease ownership: drop pills
/// whose lease is no longer held, create one for each newly-held `(dag, agent)`.
fn reconcile_transients(
coord: &Arc<Coordinator>,
transients: &mut HashMap<(u64, String), crate::coordinator::TransientGuard>,
) {
for rel in coord.job_queue.drain_agent_releases() {
transients.remove(&(rel.dag_id, rel.agent));
}
for terminal in coord.job_queue.drain_terminal() {
// Drop any per-agent transient guard the DAG still held (the
// per-agent pass above already dropped the ones whose subgraphs
// settled early).
transients.retain(|(dag_id, _), _| *dag_id != terminal.dag_id);
exec::on_dag_terminal(coord, &terminal).await;
let held = coord.job_queue.held_transients();
let keys: HashSet<(u64, String)> = held.iter().map(|(d, a, _)| (*d, a.clone())).collect();
transients.retain(|k, _| keys.contains(k));
for (dag_id, agent, kind) in held {
transients
.entry((dag_id, agent.clone()))
.or_insert_with(|| coord.transient_guard(&agent, kind));
}
}

58
hive-c0re/src/job_queue/submit.rs
View file

@ -25,7 +25,7 @@
use std::sync::Arc;
use super::model::{DagSpec, Dep, NodeKind, NodeSpec, Template};
use super::templates::{after_ok, node, rebuild_nodes};
use super::templates::{after_ok, child, node, rebuild_nodes};
use super::{Source, templates};
use crate::coordinator::{Coordinator, TransientKind};
use crate::lifecycle;
@ -60,13 +60,16 @@ pub fn rebuild(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: St
/// stays even for a down agent so a race-up between the state read and exec
/// is still stopped in-DAG.
fn stop_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
// `SetWanted` is the group root and owns the agent lease; the mechanical
// steps are its children (borrow the lease, run once it reaches `Finishing`,
// dep-ordered among themselves).
let mut n = vec![node(agent, NodeKind::SetWanted { up: false }, Vec::new())];
if graceful && running {
n.push(node(agent, NodeKind::Signal, after_ok(0)));
n.push(node(agent, NodeKind::Drain, after_ok(1)));
n.push(node(agent, NodeKind::Reconcile, after_ok(2)));
n.push(child(0, agent, NodeKind::Signal, Vec::new()));
n.push(child(0, agent, NodeKind::Drain, after_ok(1)));
n.push(child(0, agent, NodeKind::Reconcile, after_ok(2)));
} else {
n.push(node(agent, NodeKind::Reconcile, after_ok(0)));
n.push(child(0, agent, NodeKind::Reconcile, Vec::new()));
}
n
}
@ -78,11 +81,12 @@ fn stop_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
fn start_chain(agent: &str, running: bool, stale: bool) -> Vec<NodeSpec> {
let mut n = vec![node(agent, NodeKind::SetWanted { up: true }, Vec::new())];
if !running && stale {
// Rebuild subgraph rooted at the SetWanted head (base = 1, so
// `Prebuild` deps `after_ok(0)` = the head).
// Rebuild subtree after the SetWanted head (base = 1, so the rebuild's
// `Prebuild` root deps `after_ok(0)` = the head). `Prebuild` +
// `Reconcile` are their own group roots (top-level, per `rebuild_nodes`).
n.extend(rebuild_nodes(agent, true, 1));
} else {
n.push(node(agent, NodeKind::Reconcile, after_ok(0)));
n.push(child(0, agent, NodeKind::Reconcile, Vec::new()));
}
n
}
@ -101,18 +105,30 @@ fn restart_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
// Nothing to bounce — a lone Reconcile converges to intent.
return vec![node(agent, NodeKind::Reconcile, Vec::new())];
}
// Running: mechanical stop then Reconcile. The first stop node is the
// subgraph root (no SetWanted head) and acquires the agent lease.
let mut n = Vec::new();
if graceful {
n.push(node(agent, NodeKind::Signal, Vec::new()));
n.push(node(agent, NodeKind::Drain, after_ok(0)));
n.push(node(agent, NodeKind::StopForUpdate, after_ok(1)));
// Running: mechanical stop then Reconcile. The first stop node is the group
// ROOT (no SetWanted head) and owns the agent lease; the rest are its
// children (borrow the lease, dep-ordered), so the bounce holds one
// continuous lease and `Reconcile` cancel-cascades if a stop step fails.
let mut n = vec![if graceful {
node(agent, NodeKind::Signal, Vec::new())
} else {
n.push(node(agent, NodeKind::StopForUpdate, Vec::new()));
node(agent, NodeKind::StopForUpdate, Vec::new())
}];
if graceful {
n.push(child(0, agent, NodeKind::Drain, Vec::new()));
n.push(child(0, agent, NodeKind::StopForUpdate, after_ok(1)));
}
let stop_idx = u32::try_from(n.len() - 1).unwrap_or(0);
n.push(node(agent, NodeKind::Reconcile, after_ok(stop_idx)));
// `Reconcile` gates on the last mechanical step. When the only step is the
// root itself (non-graceful, `StopForUpdate` == index 0), the parent gate
// already orders `Reconcile` after it — a child must NOT dep on its own
// parent (dep-scope). So the sibling dep is added only for a graceful
// bounce, where the last step is a sibling child.
let deps = if n.len() > 1 {
after_ok(u64::try_from(n.len() - 1).unwrap_or(0))
} else {
Vec::new()
};
n.push(child(0, agent, NodeKind::Reconcile, deps));
n
}
@ -124,7 +140,7 @@ fn restart_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
fn concat_subgraphs(chains: Vec<Vec<NodeSpec>>) -> Vec<NodeSpec> {
let mut out: Vec<NodeSpec> = Vec::new();
for chain in chains {
let base = u32::try_from(out.len()).unwrap_or(u32::MAX);
let base = u64::try_from(out.len()).unwrap_or(u64::MAX);
for spec in chain {
let deps = spec
.deps
@ -138,6 +154,10 @@ fn concat_subgraphs(chains: Vec<Vec<NodeSpec>>) -> Vec<NodeSpec> {
agent: spec.agent,
kind: spec.kind,
deps,
// Rebase the structural parent by the same offset (a subgraph
// root keeps `parent = None`, so the per-agent groups stay
// independent + concurrent).
parent: spec.parent.map(|p| base + p),
});
}
}

75
hive-c0re/src/job_queue/templates.rs
View file

@ -35,34 +35,58 @@ use crate::coordinator::TransientKind;
/// After-ok edge on the previous node — the common chain link. Shared with
/// the async power-op builders in `submit.rs` (which assemble per-agent
/// chains dynamically from live container state).
pub(crate) fn after_ok(on: u32) -> Vec<Dep> {
pub(crate) fn after_ok(on: u64) -> Vec<Dep> {
vec![Dep {
on,
when: DepWhen::AfterOk,
}]
}
/// Build one node targeting `agent`. The single place a node's agent is
/// stamped. Shared with `submit.rs`'s dynamic power-op builders.
/// Build one **top-level (group-root)** node targeting `agent` — `parent =
/// None`. The single place a node's agent is stamped. Shared with `submit.rs`'s
/// dynamic power-op builders. A root owns whatever resource it declares for its
/// whole subtree; its descendants borrow it (agent-lease / build-slot
/// continuity). Ordering vs other nodes is `deps`; grouping is `parent`.
pub(crate) fn node(agent: &str, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
NodeSpec {
agent: agent.to_owned(),
kind,
deps,
parent: None,
}
}
/// The rebuild node chain. `PostSwap` carries the swap's Ok-only
/// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan) and deps
/// `Swap` with `AfterOk`. `Reconcile` then deps on `PostSwap` with
/// `AfterAny`: it must run even when the swap failed, so a previously-up
/// agent comes back on its old config (today's recovery-start). On swap
/// failure the `AfterOk` `PostSwap` is cancel-cascaded to a terminal state,
/// which still satisfies `Reconcile`'s `AfterAny` edge — the only `AfterAny`
/// edge in v1. Pointing `Reconcile` at `PostSwap` (not `Swap`) also
/// serializes the tail ahead of the reconcile, so there's no double
/// rescan/kick race.
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u32) -> Vec<NodeSpec> {
/// Build a **child** node whose structural parent is spec-index `parent`. The
/// child runs once its parent reaches `Finishing` (the parent gate), so it must
/// NOT `deps` on `parent` (dep-scope validation rejects a dep on one's own
/// parent). `deps` here order the child against its *siblings* only.
pub(crate) fn child(parent: u64, agent: &str, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
NodeSpec {
agent: agent.to_owned(),
kind,
deps,
parent: Some(parent),
}
}
/// The rebuild node subtree (nested, two group roots). `base` is the spec index
/// of the first node (`Prebuild`). Structure:
/// - `Prebuild` (base+0, **root**): owns the build slot for the whole subtree.
/// Lease-exempt — the nix build overlaps other DAGs on the same agent.
/// - `StopForUpdate` (base+1, child of `Prebuild`): owns the agent lease. Runs
/// once `Prebuild` reaches `Finishing` (parent gate).
/// - `Swap` (base+2, child of `StopForUpdate`): borrows the agent lease from its
/// parent and the build slot from grand-ancestor `Prebuild` — both continuous.
/// - `PostSwap` (base+3, child of `StopForUpdate`): the swap's Ok-only
/// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk`
/// its sibling `Swap`.
/// - `Reconcile` (base+4, **root**): `AfterAny` `Prebuild`, which rolls up
/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has
/// settled — so `Reconcile` runs after the swap regardless of outcome, and as
/// a top-level root it survives the cancel-cascade of a failed `Prebuild`
/// (recovery-start invariant). It takes a fresh lease; the tiny gap is
/// harmless — `Reconcile` converges to the persisted `wanted` idempotently.
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u64) -> Vec<NodeSpec> {
vec![
node(
agent,
@ -73,14 +97,14 @@ pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u32) -> Vec<NodeSpe
after_ok(base - 1)
},
),
node(agent, NodeKind::StopForUpdate, after_ok(base)),
node(agent, NodeKind::Swap, after_ok(base + 1)),
node(agent, NodeKind::PostSwap, after_ok(base + 2)),
child(base, agent, NodeKind::StopForUpdate, Vec::new()),
child(base + 1, agent, NodeKind::Swap, Vec::new()),
child(base + 1, agent, NodeKind::PostSwap, after_ok(base + 2)),
node(
agent,
NodeKind::Reconcile,
vec![Dep {
on: base + 3,
on: base,
when: DepWhen::AfterAny,
}],
),
@ -149,7 +173,12 @@ pub fn reconcile_only(
/// First-deploy spawn (approval-driven): `Provision` (proposed/applied
/// repos, state subvolume, meta registration) then `Create`
/// (`nixos-container create`), drop-in write, then `Reconcile` starts
/// the container (`wanted = Up` written at approve time).
/// the container (`wanted = Up` written at approve time). All-or-nothing:
/// `Provision` (lease-exempt, precedes the container) is the group root;
/// `Create` (child) owns the agent lease; `WriteDropin` + `Reconcile`
/// (children of `Create`) borrow it. A failure cancel-cascades the rest —
/// unlike rebuild there's no recovery-reconcile (nothing to converge if the
/// container was never created).
pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
DagSpec {
template: Template::Spawn,
@ -161,9 +190,9 @@ pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
transient: Some(TransientKind::Spawning),
nodes: vec![
node(agent, NodeKind::Provision, Vec::new()),
node(agent, NodeKind::Create, after_ok(0)),
node(agent, NodeKind::WriteDropin, after_ok(1)),
node(agent, NodeKind::Reconcile, after_ok(2)),
child(0, agent, NodeKind::Create, Vec::new()),
child(1, agent, NodeKind::WriteDropin, Vec::new()),
child(1, agent, NodeKind::Reconcile, after_ok(2)),
],
}
}
@ -241,7 +270,7 @@ pub fn validate(spec: &DagSpec) -> Result<()> {
.collect();
for (i, node) in spec.nodes.iter().enumerate() {
for dep in &node.deps {
let Some(&dep_idx) = idx.get(dep.on as usize) else {
let Some(&dep_idx) = usize::try_from(dep.on).ok().and_then(|i| idx.get(i)) else {
bail!(
"dag spec {:?} node {i} depends on unknown node {}",
spec.template,

173
hive-c0re/src/job_queue/tests.rs
View file

@ -115,6 +115,7 @@ fn cyclic_dag_is_rejected_at_submit() {
on: 1,
when: DepWhen::AfterOk,
}],
parent: None,
},
NodeSpec {
agent: "agent-a".to_owned(),
@ -123,6 +124,7 @@ fn cyclic_dag_is_rejected_at_submit() {
on: 0,
when: DepWhen::AfterOk,
}],
parent: None,
},
];
assert!(q.submit(spec).is_err(), "cyclic spec must be refused");
@ -140,6 +142,7 @@ fn unknown_dep_is_rejected_at_submit() {
on: 9,
when: DepWhen::AfterOk,
}],
parent: None,
}];
assert!(q.submit(spec).is_err());
}
@ -180,13 +183,16 @@ fn build_slot_serializes_nix_heavy_nodes() {
assert_eq!(first.dag_id, a);
assert_eq!(first.kind.as_str(), "prebuild");
q.complete_node(a, first.node_id, Ok(()));
// With the slot free again, FIFO gives... a's StopForUpdate is
// slot-free (lease) and b's Prebuild takes the slot — both run.
// Uniform hold: agent-a keeps the build slot across its whole build chain
// (Swap re-enters it), so a's StopForUpdate (lease, slot-free) runs but b's
// Prebuild must wait for a's slot-needers (through Swap) to finish.
let claims = q.claim_ready();
let kinds: Vec<(u64, &str)> = claims.iter().map(|c| (c.dag_id, c.kind.as_str())).collect();
assert!(kinds.contains(&(a, "stop_for_update")));
assert!(kinds.contains(&(b, "prebuild")));
assert_eq!(claims.len(), 2);
assert_eq!(kinds, vec![(a, "stop_for_update")]);
assert!(
!kinds.iter().any(|&(d, _)| d == b),
"b's build waits — slot held across a's chain"
);
}
#[test]
@ -208,9 +214,27 @@ fn fifo_fairness_for_the_slot() {
let first = claim_one(&q);
assert_eq!(first.dag_id, a, "submit order wins the slot");
q.complete_node(a, first.node_id, Ok(()));
let next: Vec<u64> = q.claim_ready().iter().map(|cl| cl.dag_id).collect();
assert!(next.contains(&b), "b's prebuild before c's");
assert!(!next.contains(&c));
// Uniform hold: the slot stays with agent-a until its Swap (the last
// slot-needer) completes. Drive a's chain; the moment its slot frees,
// submit order (b before c) wins it.
let mut freed_to = None;
for _ in 0..6 {
let claims = q.claim_ready();
if let Some(nb) = claims.iter().find(|cl| cl.dag_id == b || cl.dag_id == c) {
freed_to = Some(nb.dag_id);
break;
}
for cl in claims {
if cl.dag_id == a {
q.complete_node(a, cl.node_id, Ok(()));
}
}
}
assert_eq!(
freed_to,
Some(b),
"b's prebuild wins the freed slot before c's"
);
}
// ---- per-agent lease ----
@ -234,18 +258,31 @@ fn lease_serializes_two_lifecycle_dags_for_same_agent() {
let first = claim_one(&q);
assert_eq!(first.dag_id, restart);
assert_eq!(first.kind.as_str(), "stop_for_update");
assert!(first.lease_acquired);
q.complete_node(restart, first.node_id, Ok(()));
// Same DAG keeps the lease through the tail Reconcile.
// Same DAG keeps the lease through the tail Reconcile (re-entered from the
// dep graph — no fresh acquire), since stop's Reconcile can't re-enter it.
let second = claim_one(&q);
assert_eq!(second.dag_id, restart);
assert_eq!(second.kind.as_str(), "reconcile");
assert!(!second.lease_acquired, "lease already held by this DAG");
q.complete_node(restart, second.node_id, Ok(()));
// Restart terminal → lease released → stop's Reconcile runs.
let third = claim_one(&q);
assert_eq!(third.dag_id, stop);
// Restart's work is terminal → its lease releases. Its terminal node and
// stop's now-unblocked Reconcile both become ready in the same pass.
let ready = q.claim_ready();
let restart_fin = ready
.iter()
.find(|c| c.dag_id == restart && c.kind.as_str() == "revert_intent")
.expect("restart finalize ready");
q.complete_node(restart, restart_fin.node_id, Ok(()));
let third = ready
.iter()
.find(|c| c.dag_id == stop)
.expect("stop reconcile ready once the lease is freed");
assert_eq!(third.kind.as_str(), "reconcile");
q.complete_node(stop, third.node_id, Ok(()));
// stop's terminal node (revert-intent) then runs.
let stop_fin = claim_one(&q);
assert_eq!(stop_fin.kind.as_str(), "revert_intent");
q.complete_node(stop, stop_fin.node_id, Ok(()));
assert_eq!(state_of(&q, restart), State::Done);
assert_eq!(state_of(&q, stop), State::Done);
}
@ -288,8 +325,20 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
.expect("reconcile claim")
.clone();
q.complete_node(stop, reconcile.node_id, Ok(()));
let next = claim_one(&q);
assert_eq!(next.kind.as_str(), "stop_for_update");
// stop's Reconcile done → its lease frees (rebuild's StopForUpdate unblocks)
// and its terminal node becomes ready; both surface in the same pass.
let after = q.claim_ready();
assert!(
after
.iter()
.any(|c| c.dag_id == stop && c.kind.as_str() == "revert_intent"),
"stop DAG's finalize runs once its work settles"
);
let sfu = after
.iter()
.find(|c| c.kind.as_str() == "stop_for_update")
.expect("rebuild StopForUpdate unblocked once the lease frees");
assert_eq!(sfu.agent, "agent-a");
}
#[test]
@ -315,16 +364,16 @@ fn multi_agent_restart_is_one_dag_with_concurrent_per_agent_subgraphs() {
// lease (no contention across distinct agents), all inside the single DAG.
let claims = q.claim_ready();
assert!(claims.iter().all(|c| c.dag_id == id));
let mut heads: Vec<(&str, &str, bool)> = claims
let mut heads: Vec<(&str, &str)> = claims
.iter()
.map(|c| (c.agent.as_str(), c.kind.as_str(), c.lease_acquired))
.map(|c| (c.agent.as_str(), c.kind.as_str()))
.collect();
heads.sort_unstable();
assert_eq!(
heads,
vec![
("agent-a", "stop_for_update", true),
("agent-b", "stop_for_update", true),
("agent-a", "stop_for_update"),
("agent-b", "stop_for_update"),
],
"both per-agent subgraphs start concurrently, each acquiring its own lease"
);
@ -387,17 +436,14 @@ fn multi_agent_stop_is_one_dag_with_concurrent_per_agent_subgraphs() {
assert_eq!(q.snapshot().len(), 1);
let claims = q.claim_ready();
assert!(claims.iter().all(|c| c.dag_id == id));
let mut heads: Vec<(&str, &str, bool)> = claims
let mut heads: Vec<(&str, &str)> = claims
.iter()
.map(|c| (c.agent.as_str(), c.kind.as_str(), c.lease_acquired))
.map(|c| (c.agent.as_str(), c.kind.as_str()))
.collect();
heads.sort_unstable();
assert_eq!(
heads,
vec![
("agent-a", "set_wanted", true),
("agent-b", "set_wanted", true),
],
vec![("agent-a", "set_wanted"), ("agent-b", "set_wanted")],
"both per-agent stop subgraphs start concurrently, each on its own lease"
);
}
@ -521,6 +567,7 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
fanout: None,
},
deps: Vec::new(),
parent: None,
}],
};
let id = submit(&q, spec);
@ -532,8 +579,8 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
// tracks any drift in that builder's root-first (`base = 0`) shape.
let subgraph = |agent: &str| templates::rebuild_nodes(agent, true, 0);
// Must append BEFORE completing the emitter (the documented contract).
q.append_subgraph(id, subgraph("a"), emitter.node_id);
q.append_subgraph(id, subgraph("b"), emitter.node_id);
q.append_subgraph(id, &subgraph("a"), emitter.node_id);
q.append_subgraph(id, &subgraph("b"), emitter.node_id);
q.complete_node(id, emitter.node_id, Ok(()));
// Still ONE DAG; both subgraph roots become ready once the emitter is
// Done (rooted on it), each on its own agent lease.
@ -580,7 +627,7 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
for agent in ["alice", "bob"] {
q.append_subgraph(
id,
templates::rebuild_nodes(agent, false, 0),
&templates::rebuild_nodes(agent, false, 0),
meta_lock.node_id,
);
}
@ -728,6 +775,11 @@ fn cancel_clears_queued_dag() {
let id = submit(&q, rebuild("agent-a", "r"));
assert!(q.cancel(id));
assert_eq!(state_of(&q, id), State::Cancelled);
// The terminal node's weak edges are satisfied by the cancelled (terminal)
// work nodes, so it still runs its hooks — it's the one thing left claimable.
let fin = claim_one(&q);
assert_eq!(fin.kind.as_str(), "emit_rebuilt");
q.complete_node(id, fin.node_id, Ok(()));
assert!(q.claim_ready().is_empty());
}
@ -743,22 +795,25 @@ fn cancel_refuses_running_dag() {
// ---- terminal reporting + lease release ----
#[test]
fn terminal_dag_reported_exactly_once_and_lease_released() {
fn terminal_node_runs_after_work_settles_and_lease_released() {
let q = JobQueue::new(1);
let id = submit(&q, restart_online(&["agent-a"], false, "r"));
// restart = StopForUpdate → Reconcile; not terminal until the last
// node completes.
// restart = StopForUpdate → Reconcile; the terminal node (which weak-deps on
// the chain tail) isn't runnable until the whole chain is terminal.
let stop = claim_one(&q);
q.complete_node(id, stop.node_id, Ok(()));
assert!(q.drain_terminal().is_empty(), "dag not terminal yet");
let rec = claim_one(&q);
assert_eq!(rec.kind.as_str(), "reconcile");
q.complete_node(id, rec.node_id, Ok(()));
let reports = q.drain_terminal();
assert_eq!(reports.len(), 1);
assert_eq!(reports[0].dag_id, id);
assert_eq!(reports[0].state, State::Done);
assert!(q.drain_terminal().is_empty(), "reported exactly once");
// Lease released: a new DAG for the agent can claim immediately.
// Work settled → the terminal node is now the runnable one; it carries the
// terminal roll-up the hook consumes via `terminal_summary`.
let fin = claim_one(&q);
assert_eq!(fin.kind.as_str(), "revert_intent");
let summary = q.terminal_summary(id).expect("terminal summary");
assert_eq!(summary.state, State::Done);
q.complete_node(id, fin.node_id, Ok(()));
// Lease released (freed when the work chain settled, ahead of finalize): a
// new DAG for the agent claims immediately.
let next = submit(
&q,
templates::reconcile_only(
@ -771,31 +826,38 @@ fn terminal_dag_reported_exactly_once_and_lease_released() {
);
let c = claim_one(&q);
assert_eq!(c.dag_id, next);
assert!(c.lease_acquired);
}
/// A DAG cancelled while fully queued must still surface a terminal
/// roll-up for the scheduler's hooks — otherwise a queued approval
/// DAG cancelled by the operator would dangle its approval forever.
#[test]
fn cancelled_dag_reports_terminal_once() {
fn cancelled_dag_finalizes_with_terminal_rollup() {
let q = JobQueue::new(1);
let id = submit(
&q,
templates::approval_deploy("agent-a", 7, "approval #7".to_owned()),
);
assert!(q.cancel(id));
let reports = q.drain_terminal();
assert_eq!(reports.len(), 1);
assert_eq!(reports[0].dag_id, id);
assert_eq!(reports[0].state, State::Cancelled);
assert_eq!(reports[0].approval_id, Some(7));
// Never re-reported by later activity.
// The terminal node's weak edges still fire on a fully-cancelled DAG, so its
// hook (approval resolution) runs — surfaced here as a claimable resolve-
// approval node whose `terminal_summary` is Cancelled + carries the approval id.
let fin = claim_one(&q);
assert_eq!(fin.kind.as_str(), "resolve_approval");
let summary = q.terminal_summary(id).expect("terminal summary");
assert_eq!(summary.state, State::Cancelled);
assert_eq!(summary.approval_id, Some(7));
q.complete_node(id, fin.node_id, Ok(()));
// The cancelled DAG's summary stays available (until history-trimmed) and
// unrelated later activity doesn't disturb it.
let other = submit(&q, rebuild("agent-b", "r"));
let c = claim_one(&q);
assert_eq!(c.dag_id, other);
q.complete_node(other, c.node_id, Err("boom".to_owned()));
assert!(q.drain_terminal().iter().all(|t| t.dag_id != id));
assert_eq!(
q.terminal_summary(id).map(|t| t.state),
Some(State::Cancelled)
);
}
// ---- steps, build logs, history ----
@ -804,7 +866,10 @@ fn cancelled_dag_reports_terminal_once() {
fn set_step_only_on_running_and_signals_change() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
assert!(!q.set_step(id, 0, "too early"), "queued node refuses step");
assert!(
!q.set_step_running(id, "too early"),
"no running node yet → refused"
);
let c = claim_one(&q);
assert!(q.set_step(id, c.node_id, "nix build"));
assert!(
@ -823,7 +888,10 @@ fn set_step_only_on_running_and_signals_change() {
fn set_build_log_id_links_running_node() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
assert!(!q.set_build_log_id(id, 0, 41), "queued node refuses log id");
assert!(
!q.set_build_log_id_running(id, 41),
"no running node yet → refused"
);
let c = claim_one(&q);
assert!(q.set_build_log_id(id, c.node_id, 42));
assert!(q.set_build_log_id_running(id, 43));
@ -849,6 +917,11 @@ fn history_evicts_old_terminals_per_template() {
);
let c = claim_one(&q);
q.complete_node(id, c.node_id, Ok(()));
// Drain the DAG's terminal node too, so the next iteration's claim sees
// only its own work (the terminal node is excluded from the view + rollup).
let fin = claim_one(&q);
assert_eq!(fin.kind.as_str(), "revert_intent");
q.complete_node(id, fin.node_id, Ok(()));
}
// Fresh terminals are inside the grace window: nothing evicts yet,
// so a ~1s QueueDag poller can still observe every terminal state

2
hive-c0re/src/workers/auto_update.rs
View file

@ -331,6 +331,7 @@ fn submit_boot_tree(
fanout: Some(fanout),
},
deps: Vec::new(),
parent: None,
});
}
// One boot Reconcile per drifted agent — independent roots.
@ -339,6 +340,7 @@ fn submit_boot_tree(
agent: name,
kind: NodeKind::Reconcile,
deps: Vec::new(),
parent: None,
});
}

7
hive-sh4re/src/jobs.rs
View file

@ -134,8 +134,11 @@ pub enum PermPayload {
},
}
/// Node id, unique within its DAG.
pub type NodeId = u32;
/// Node id. Carries the scheduler crate's globally-monotonic node id
/// (`hive_jobq::NodeId`) verbatim on the wire — unique across all DAGs, not
/// just within one. Consumers treat it opaquely (grouping + dep matching),
/// so the widening from the old dag-local `u32` is transparent.
pub type NodeId = u64;
/// One node of a queued DAG, as serialized. Step labels, build-log
/// links, errors, and timestamps are per-node; the DAG-level `state`

2
hivectl/src/dag_progress.rs
View file

@ -325,7 +325,7 @@ mod tests {
use super::render_dag_line;
fn node(id: u32, agent: &str, kind: &str, state: State, step: Option<&str>) -> NodeView {
fn node(id: u64, agent: &str, kind: &str, state: State, step: Option<&str>) -> NodeView {
NodeView {
id,
agent: agent.to_owned(),