vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(#2756): declare the terminal hook instead of inferring it

Browse source 
`Template` was a DAG-level enum that three different things read back
out: `terminal_hook()` mapped it to a side effect, the retention pass
bucketed history by it, and a tracing field printed it. None of those
needed a *label* — they needed the two facts the label happened to
encode. So the enum was a lossy stand-in for intent, and every new DAG
shape had to pick the variant whose inferred behaviour matched, whether
or not the name fit (`reparent` rode `MetaUpdate` for exactly this
reason, with a 10-line comment apologising for it).

Replace the inference with a declaration: `DagSpec.hook:
Option<HookKind>`. Only the builder assembling a DAG knows why it did
so, so only the builder can say what should happen when it settles.
`run_terminal_hook` becomes a field read, and `reparent`'s apology
becomes `hook: None`.

Hook assignment is byte-identical to the old precedence rule
(`approval_id.is_some()` wins, then `Rebuild | PermChange`), checked
site by site; `meta_update` is the only builder with a variable
approval id and so the only remaining conditional.

Retention loses the per-template bucket with the enum that keyed it.
The dashboard renders one recent-builds list, so one flat newest-first
cap (`MAX_HISTORY_DAGS`) bounds it. `HISTORY_GRACE_SECS` goes too — it
existed to stop a burst of same-template DAGs evicting each other
inside one poll interval, which is not a failure mode a flat cap has.
That takes `snapshot_capped()` and the `snapshot_no_grace()` test hook
with it.

The queue is runtime-only (empty graph on boot), so the serde changes
carry no migration risk.
This commit is contained in:
atlas 2026-07-27 12:37:03 +02:00 • committed by mara
commit ca7146e4f0
8 changed files with 113 additions and 241 deletions
Download patch file Download diff file Expand all files Collapse all files

10
hive-c0re/src/job_queue/exec.rs
View file

@ -97,18 +97,18 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
NodeKind::DeployTail { .. } => run_deploy_tail(coord, claim).await, NodeKind::DeployTail { .. } => run_deploy_tail(coord, claim).await,
NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up), NodeKind::SetWanted { up, .. } => run_set_wanted(coord, claim, *up),
// Pure grouping container — no work; completing it lets it reach // Pure grouping container — no work; completing it lets it reach
// `Finishing` so its child template nodes start. The DAG's terminal // `Finishing` so its child work nodes start. The DAG's terminal
// hook fires (inline, via `run_terminal_hook`) when the container itself // hook fires (inline, via `run_terminal_hook`) when the container itself
// rolls up terminal — not as a scheduled node. // rolls up terminal — not as a scheduled node.
NodeKind::Dag { .. } => Ok(NodeOutput::default()), NodeKind::Dag { .. } => Ok(NodeOutput::default()),
} }
} }
/// Run a settled DAG's inline terminal hook, dispatched off its rolled-up /// Run a settled DAG's inline terminal hook — the container-terminal
/// summary — the container-terminal replacement for the old per-DAG hook node. /// replacement for the old per-DAG hook node. Always best-effort: a hook
/// Always best-effort: a hook failure is logged inside, never surfaced. /// failure is logged inside, never surfaced.
pub(crate) async fn run_terminal_hook(coord: &Arc<Coordinator>, terminal: &super::TerminalDag) { pub(crate) async fn run_terminal_hook(coord: &Arc<Coordinator>, terminal: &super::TerminalDag) {
match super::terminal_hook(terminal.template, terminal.approval_id) { match terminal.hook {
Some(super::HookKind::ResolveApproval) => { Some(super::HookKind::ResolveApproval) => {
crate::actions::resolve_approval_dag(coord, terminal).await; crate::actions::resolve_approval_dag(coord, terminal).await;
} }

113
hive-c0re/src/job_queue/mod.rs
View file

@ -11,15 +11,15 @@
//! classes are [`resource::Resource`] (`BuildSlot` node-held, `Agent` lease //! classes are [`resource::Resource`] (`BuildSlot` node-held, `Agent` lease
//! subtree-held), derived per node by [`NodeKind::resource_deps`]; //! subtree-held), derived per node by [`NodeKind::resource_deps`];
//! - a **DAG is a single container node** ([`NodeKind::Dag`], `parent = None`) //! - a **DAG is a single container node** ([`NodeKind::Dag`], `parent = None`)
//! carrying the group's metadata, with the template's nodes hung under it as //! carrying the group's metadata, with the work nodes hung under it as
//! its subtree (the **parent axis** groups; `deps` order). So the container's //! its subtree (the **parent axis** groups; `deps` order). So the container's
//! `NodeId` is the DAG id, its rolled-up state is the DAG state, and membership //! `NodeId` is the DAG id, its rolled-up state is the DAG state, and membership
//! is a graph walk — there are no host grouping side-tables. The lease is owned //! is a graph walk — there are no host grouping side-tables. The lease is owned
//! by a subtree root and borrowed by its descendants (continuity); //! by a subtree root and borrowed by its descendants (continuity);
//! - per-DAG terminal work runs **inline** ([`exec::run_terminal_hook`]) when the //! - per-DAG terminal work runs **inline** ([`exec::run_terminal_hook`]) when the
//! container rolls up terminal — dispatched off its template //! container rolls up terminal, off the [`HookKind`] the *builder* stated on
//! ([`terminal_hook`]): approval-resolve, `Rebuilt`-emit, or power-intent //! the spec: approval-resolve or `Rebuilt`-emit. No terminal-hook node, no
//! revert. No terminal-hook node, no drained event stream. //! drained event stream.
//! //!
//! The queue is runtime-only (no persistence): an empty graph on boot; desired //! The queue is runtime-only (no persistence): an empty graph on boot; desired
//! state is re-derived by the reconcile sweep. A single scheduler task //! state is re-derived by the reconcile sweep. A single scheduler task
@ -48,18 +48,14 @@ use tokio::sync::Notify;
use crate::coordinator::TransientKind; use crate::coordinator::TransientKind;
pub use model::{ pub use model::{
DagSpec, DagView, DepWhen, NodeKind, NodeSpec, PermPayload, Source, State, Template, DagSpec, DagView, DepWhen, HookKind, NodeKind, NodeSpec, PermPayload, Source, State,
}; };
use resource::Resource; use resource::Resource;
/// How many terminal DAGs (`Done` / `Failed` / `Cancelled`) to retain per /// How many terminal DAGs (`Done` / `Failed` / `Cancelled`) the snapshot
/// template in the snapshot, matching the old per-kind history cap. /// retains, newest first. A flat cap over the whole sorted list: the
const MAX_HISTORY_PER_TEMPLATE: usize = 5; /// dashboard renders one recent-builds list, so one number bounds it.
const MAX_HISTORY_DAGS: usize = 50;
/// Terminal DAGs younger than this are exempt from the per-template history
/// cap, so a burst of same-template DAGs that settle within one `QueueDag`
/// poll interval isn't evicted before the poller observes their terminal state.
const HISTORY_GRACE_SECS: i64 = 300;
/// Cap on stored node error strings. /// Cap on stored node error strings.
const MAX_ERROR_LEN: usize = 2_000; const MAX_ERROR_LEN: usize = 2_000;
@ -74,7 +70,6 @@ pub struct Claim {
/// The agent this node targets (its own, not a DAG-level field). Empty for /// The agent this node targets (its own, not a DAG-level field). Empty for
/// the agentless [`NodeKind::MetaLock`] + [`NodeKind::Dag`] container nodes. /// the agentless [`NodeKind::MetaLock`] + [`NodeKind::Dag`] container nodes.
pub agent: String, pub agent: String,
pub template: Template,
pub approval_id: Option<i64>, pub approval_id: Option<i64>,
pub inputs: Vec<String>, pub inputs: Vec<String>,
/// Transient pill kind for the lease window (from the spec). Whether the /// Transient pill kind for the lease window (from the spec). Whether the
@ -88,7 +83,8 @@ pub struct Claim {
/// revert). Computed on demand from live graph state, not drained. /// revert). Computed on demand from live graph state, not drained.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct TerminalDag { pub struct TerminalDag {
pub template: Template, /// The side effect to fire, carried from the DAG container's payload.
pub hook: Option<HookKind>,
/// Distinct agents this DAG's nodes targeted (one for a single-agent DAG). /// Distinct agents this DAG's nodes targeted (one for a single-agent DAG).
pub agents: Vec<String>, pub agents: Vec<String>,
pub approval_id: Option<i64>, pub approval_id: Option<i64>,
@ -110,7 +106,7 @@ struct NodeRuntime {
/// Derived on read from the container node — the data has a single home (the /// Derived on read from the container node — the data has a single home (the
/// node payload); this is not a stored side-table. /// node payload); this is not a stored side-table.
struct DagMeta { struct DagMeta {
template: Template, hook: Option<HookKind>,
source: Source, source: Source,
reason: String, reason: String,
transient: Option<TransientKind>, transient: Option<TransientKind>,
@ -161,37 +157,6 @@ fn to_crate_when(when: DepWhen) -> JobDepWhen {
} }
} }
/// The inline terminal-hook a settled DAG fires — dispatched off its container's
/// template + approval id when the container rolls up terminal (no hook node).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HookKind {
/// Approval-driven DAG (spawn / opaque deploy): resolve the approval row.
ResolveApproval,
/// Rebuild / perm-change: emit one `Rebuilt` manager event per agent.
EmitRebuilt,
}
/// The terminal hook a DAG needs, from its template + approval id — or `None`
/// for a DAG with no terminal side effect (power-op, meta-update, boot, bare
/// reconcile).
///
/// A cancelled DAG deliberately gets **no** compensating hook. [`JobQueue::cancel`]
/// refuses unless every work node is still `Pending`, and a cancel *cascade*
/// rolls up `Failed` (see `dag_rollup`), never `Cancelled` — so on a
/// `Cancelled` DAG no node ever executed and there is nothing to undo. A power
/// op's `SetWanted` head provably never ran, so its intent is still whatever
/// the operator last set it to.
#[must_use]
pub fn terminal_hook(template: Template, approval_id: Option<i64>) -> Option<HookKind> {
if approval_id.is_some() {
return Some(HookKind::ResolveApproval);
}
match template {
Template::Rebuild | Template::PermChange => Some(HookKind::EmitRebuilt),
_ => None,
}
}
/// Map a crate node state onto the wire state (`Pending` ↔ `Queued`; /// Map a crate node state onto the wire state (`Pending` ↔ `Queued`;
/// `Finishing` — own logic done, sub-nodes still running — reads as `Running`). /// `Finishing` — own logic done, sub-nodes still running — reads as `Running`).
fn to_wire_state(state: JobState) -> State { fn to_wire_state(state: JobState) -> State {
@ -289,7 +254,7 @@ impl JobQueue {
.sched .sched
.append( .append(
NodeKind::Dag { NodeKind::Dag {
template: spec.template, hook: spec.hook,
source: spec.source, source: spec.source,
reason: spec.reason, reason: spec.reason,
transient: spec.transient, transient: spec.transient,
@ -380,7 +345,6 @@ impl JobQueue {
node_id: id, node_id: id,
kind, kind,
agent, agent,
template: meta.template,
approval_id: meta.approval_id, approval_id: meta.approval_id,
inputs: meta.inputs, inputs: meta.inputs,
transient: meta.transient, transient: meta.transient,
@ -535,14 +499,8 @@ impl JobQueue {
/// Snapshot every live + retained DAG for `/api/state` + `RebuildQueueChanged`. /// Snapshot every live + retained DAG for `/api/state` + `RebuildQueueChanged`.
#[must_use] #[must_use]
pub fn snapshot(&self) -> Vec<DagView> { pub fn snapshot(&self) -> Vec<DagView> {
self.snapshot_capped(now_unix() - HISTORY_GRACE_SECS)
}
/// Snapshot the visible DAG set (live + newest-per-template terminal, terminal
/// ones after `grace_cutoff` always kept), sorted by container id.
fn snapshot_capped(&self, grace_cutoff: i64) -> Vec<DagView> {
let inner = self.lock(); let inner = self.lock();
let mut ids = inner.visible_dags(grace_cutoff); let mut ids = inner.visible_dags();
ids.sort_unstable_by_key(|c| c.get()); ids.sort_unstable_by_key(|c| c.get());
ids.into_iter().filter_map(|c| inner.dag_view(c)).collect() ids.into_iter().filter_map(|c| inner.dag_view(c)).collect()
} }
@ -558,14 +516,6 @@ impl JobQueue {
.filter(|&c| !inner.dag_is_terminal(c)) .filter(|&c| !inner.dag_is_terminal(c))
.count() .count()
} }
/// Test hook: snapshot with the history grace window disabled, so the
/// per-template cap applies to just-finished terminal DAGs too.
#[cfg(test)]
#[must_use]
pub(crate) fn snapshot_no_grace(&self) -> Vec<DagView> {
self.snapshot_capped(i64::MAX)
}
} }
impl QueueInner { impl QueueInner {
@ -616,7 +566,7 @@ impl QueueInner {
/// not a stored side-table. /// not a stored side-table.
fn dag_meta(&self, container: NodeId) -> Option<DagMeta> { fn dag_meta(&self, container: NodeId) -> Option<DagMeta> {
let NodeKind::Dag { let NodeKind::Dag {
template, hook,
source, source,
reason, reason,
transient, transient,
@ -628,7 +578,7 @@ impl QueueInner {
return None; return None;
}; };
Some(DagMeta { Some(DagMeta {
template: *template, hook: *hook,
source: *source, source: *source,
reason: reason.clone(), reason: reason.clone(),
transient: *transient, transient: *transient,
@ -708,7 +658,7 @@ impl QueueInner {
fn terminal_dag(&self, container: NodeId) -> Option<TerminalDag> { fn terminal_dag(&self, container: NodeId) -> Option<TerminalDag> {
let meta = self.dag_meta(container)?; let meta = self.dag_meta(container)?;
Some(TerminalDag { Some(TerminalDag {
template: meta.template, hook: meta.hook,
agents: self.dag_agents(container), agents: self.dag_agents(container),
approval_id: meta.approval_id, approval_id: meta.approval_id,
state: self.dag_rollup(container), state: self.dag_rollup(container),
@ -829,33 +779,24 @@ impl QueueInner {
} }
/// The **visible** DAG set for the snapshot: every live (non-terminal) DAG, /// The **visible** DAG set for the snapshot: every live (non-terminal) DAG,
/// plus the newest [`MAX_HISTORY_PER_TEMPLATE`] terminal DAGs per template /// plus the newest [`MAX_HISTORY_DAGS`] terminal ones. Crate nodes for
/// (terminal DAGs finished after `grace_cutoff` are always kept). Crate nodes /// evicted DAGs linger in the graph (bounded-prune is a Stage-C follow-up);
/// for evicted DAGs linger in the graph (bounded-prune is a Stage-C /// this filter is what bounds what the dashboard sees.
/// follow-up); this filter is what bounds what the dashboard sees. fn visible_dags(&self) -> Vec<NodeId> {
fn visible_dags(&self, grace_cutoff: i64) -> Vec<NodeId> {
let mut live: Vec<NodeId> = Vec::new(); let mut live: Vec<NodeId> = Vec::new();
let mut terminal: Vec<(NodeId, Template, i64)> = Vec::new(); let mut terminal: Vec<(NodeId, i64)> = Vec::new();
for c in self.containers() { for c in self.containers() {
if self.dag_is_terminal(c) { if self.dag_is_terminal(c) {
if let Some(meta) = self.dag_meta(c) { terminal.push((c, self.dag_finished_at(c)));
terminal.push((c, meta.template, self.dag_finished_at(c)));
}
} else { } else {
live.push(c); live.push(c);
} }
} }
// Newest first so the per-template cap keeps the most recent. // Newest first, so truncating to the cap keeps the most recent.
terminal.sort_by(|a, b| b.2.cmp(&a.2).then(b.0.get().cmp(&a.0.get()))); terminal.sort_by(|a, b| b.1.cmp(&a.1).then(b.0.get().cmp(&a.0.get())));
let mut counts: HashMap<Template, usize> = HashMap::new(); terminal.truncate(MAX_HISTORY_DAGS);
let mut kept = live; let mut kept = live;
for (c, template, finished) in terminal { kept.extend(terminal.into_iter().map(|(c, _)| c));
let n = counts.entry(template).or_insert(0);
*n += 1;
if *n <= MAX_HISTORY_PER_TEMPLATE || finished > grace_cutoff {
kept.push(c);
}
}
kept kept
} }
} }

85
hive-c0re/src/job_queue/model.rs
View file

@ -17,55 +17,25 @@ use serde::Serialize;
use crate::coordinator::TransientKind; use crate::coordinator::TransientKind;
/// What a DAG *means* — the request-level shape. Internal to the queue now: /// The inline side effect a settled DAG fires when its container node rolls
/// it drives the terminal-hook dispatch ([`crate::job_queue`]'s `dag_hook`) /// up terminal (there is no hook *node*). Stated explicitly by the builder in
/// and the meta-update dedup key, and is **no longer sent on the wire** — the /// `templates.rs` / `submit.rs` rather than inferred from a DAG-level enum:
/// dashboard derives a DAG's label from its node kinds (see `DagView`). The /// only the builder knows why it assembled the DAG, so only the builder can
/// `NodeKind::Dag` container carries it in its payload. /// say what should happen at the end of it.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize)] ///
/// A cancelled DAG deliberately gets **no** compensating hook. [`super::JobQueue::cancel`]
/// refuses unless every work node is still `Pending`, and a cancel *cascade*
/// rolls up `Failed` (see `dag_rollup`), never `Cancelled` — so on a
/// `Cancelled` DAG no node ever executed and there is nothing to undo. A power
/// op's `SetWanted` head provably never ran, so its intent is still whatever
/// the operator last set it to.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")] #[serde(rename_all = "snake_case")]
pub enum Template { pub enum HookKind {
/// Rebuild one agent's container (prebuild → stop → profile-swap → /// Approval-driven DAG (spawn / opaque deploy): resolve the approval row.
/// reconcile). ResolveApproval,
Rebuild, /// Rebuild / perm-change: emit one `Rebuilt` manager event per agent.
/// Bump meta flake locks; grows a rebuild subgraph per affected EmitRebuilt,
/// agent into the same DAG on completion.
MetaUpdate,
/// First-deploy spawn (approval-driven).
Spawn,
/// Mechanical stop + converge to `wanted = Up` (a restart).
Restart,
/// Signal → drain → mechanical stop → converge to `wanted = Up` — a
/// graceful restart as one atomic DAG.
GracefulRestart,
/// Perm-file commit followed by the rebuild subgraph.
PermChange,
/// Quiesce the harness, drain, then stop (`wanted = Offline`).
GracefulStop,
/// Converge to `wanted = Up`.
Start,
/// Converge to `wanted = Offline`.
Stop,
/// Boot-time config sweep as one DAG.
Boot,
}
impl Template {
#[must_use]
pub fn as_str(self) -> &'static str {
match self {
Template::Rebuild => "rebuild",
Template::MetaUpdate => "meta_update",
Template::Spawn => "spawn",
Template::Restart => "restart",
Template::GracefulRestart => "graceful_restart",
Template::PermChange => "perm_change",
Template::GracefulStop => "graceful_stop",
Template::Start => "start",
Template::Stop => "stop",
Template::Boot => "boot",
}
}
} }
/// When a dependency edge is considered satisfied. /// When a dependency edge is considered satisfied.
@ -293,15 +263,16 @@ pub enum NodeKind {
/// is skipped.) /// is skipped.)
SetWanted { agent: String, up: bool }, SetWanted { agent: String, up: bool },
/// The **DAG container** node: one per submitted DAG, carrying the group's /// The **DAG container** node: one per submitted DAG, carrying the group's
/// domain metadata. Every template node hangs *under* it (its subtree), so /// domain metadata. Every node hangs *under* it (its subtree), so
/// the container's `NodeId` **is** the DAG id, its rolled-up state **is** the /// the container's `NodeId` **is** the DAG id, its rolled-up state **is** the
/// DAG state, and it reaching terminal **is** the completion signal that /// DAG state, and it reaching terminal **is** the completion signal that
/// fires the DAG's inline hook (approval-resolve / rebuilt-emit / /// fires the DAG's inline `hook`. Pure grouping — lease- and
/// intent-revert, dispatched off `template`). Pure grouping — lease- and
/// build-slot-exempt; the executor instant-completes it (`Done`) so it /// build-slot-exempt; the executor instant-completes it (`Done`) so it
/// reaches `Finishing` and its children start. /// reaches `Finishing` and its children start.
Dag { Dag {
template: Template, /// The side effect to run when this DAG settles, or `None` for a DAG
/// with none (power op, meta-update, boot).
hook: Option<HookKind>,
source: Source, source: Source,
reason: String, reason: String,
transient: Option<TransientKind>, transient: Option<TransientKind>,
@ -475,14 +446,16 @@ pub struct NodeSpec {
/// ([`NodeKind::WritePermFile`]), not this generic spec. /// ([`NodeKind::WritePermFile`]), not this generic spec.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct DagSpec { pub struct DagSpec {
pub template: Template, /// The inline side effect to fire when this DAG settles. Explicit — the
/// builder assembling the DAG is the only thing that knows its intent.
pub hook: Option<HookKind>,
pub source: Source, pub source: Source,
/// Free-form "why". /// Free-form "why".
pub reason: String, pub reason: String,
/// Fires the approval-resolution hook on DAG terminal. /// The approval row [`HookKind::ResolveApproval`] resolves. Set together
/// with that hook; carried separately because the hook needs the id.
pub approval_id: Option<i64>, pub approval_id: Option<i64>,
/// `MetaUpdate`-only: the inputs to bump (also part of the dedup /// Meta-update only: the inputs to bump. Display copy lives on the DAG.
/// key for that template). Display copy lives on the DAG.
pub inputs: Vec<String>, pub inputs: Vec<String>,
/// Dashboard transient pill (and crash-watch suppression) held for /// Dashboard transient pill (and crash-watch suppression) held for
/// the lease window — from lease acquisition to DAG terminal. /// the lease window — from lease acquisition to DAG terminal.

1
hive-c0re/src/job_queue/scheduler.rs
View file

@ -53,7 +53,6 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
node = claim.node_id.get(), node = claim.node_id.get(),
kind = claim.kind.as_str(), kind = claim.kind.as_str(),
agent = %claim.agent, agent = %claim.agent,
template = claim.template.as_str(),
"job_queue: node running" "job_queue: node running"
); );
let coord = Arc::clone(&coord); let coord = Arc::clone(&coord);

29
hive-c0re/src/job_queue/submit.rs
View file

@ -24,7 +24,7 @@
use std::sync::Arc; use std::sync::Arc;
use super::model::{DagSpec, Dep, NodeKind, NodeSpec, Template}; use super::model::{DagSpec, Dep, NodeKind, NodeSpec};
use super::templates::{after_ok, child, node, rebuild_nodes}; use super::templates::{after_ok, child, node, rebuild_nodes};
use super::{Source, templates}; use super::{Source, templates};
use crate::coordinator::{Coordinator, TransientKind}; use crate::coordinator::{Coordinator, TransientKind};
@ -188,16 +188,17 @@ fn concat_subgraphs(chains: Vec<Vec<NodeSpec>>) -> Vec<NodeSpec> {
out out
} }
/// Wrap assembled power-op `nodes` in a `DagSpec`. /// Wrap assembled power-op `nodes` in a `DagSpec`. No terminal hook: a power
/// op's effect is its nodes (`SetWanted` + `Reconcile`), with nothing left to
/// do once they settle.
fn power_dag( fn power_dag(
template: Template,
transient: TransientKind, transient: TransientKind,
source: Source, source: Source,
reason: String, reason: String,
nodes: Vec<NodeSpec>, nodes: Vec<NodeSpec>,
) -> DagSpec { ) -> DagSpec {
DagSpec { DagSpec {
template, hook: None,
source, source,
reason, reason,
approval_id: None, approval_id: None,
@ -223,13 +224,7 @@ pub(crate) fn stop_spec(
.iter() .iter()
.map(|(agent, running)| stop_chain(agent, graceful, *running)) .map(|(agent, running)| stop_chain(agent, graceful, *running))
.collect(); .collect();
let template = if graceful {
Template::GracefulStop
} else {
Template::Stop
};
power_dag( power_dag(
template,
TransientKind::Stopping, TransientKind::Stopping,
source, source,
reason, reason,
@ -256,13 +251,7 @@ pub(crate) fn start_spec(
} else { } else {
TransientKind::Starting TransientKind::Starting
}; };
power_dag( power_dag(transient, source, reason, concat_subgraphs(chains))
Template::Start,
transient,
source,
reason,
concat_subgraphs(chains),
)
} }
/// Assemble the restart DAG from explicit `(agent, running)` targets. /// Assemble the restart DAG from explicit `(agent, running)` targets.
@ -276,13 +265,7 @@ pub(crate) fn restart_spec(
.iter() .iter()
.map(|(agent, running)| restart_chain(agent, graceful, *running)) .map(|(agent, running)| restart_chain(agent, graceful, *running))
.collect(); .collect();
let template = if graceful {
Template::GracefulRestart
} else {
Template::Restart
};
power_dag( power_dag(
template,
TransientKind::Restarting, TransientKind::Restarting,
source, source,
reason, reason,

39
hive-c0re/src/job_queue/templates.rs
View file

@ -30,7 +30,7 @@
use anyhow::{Result, bail}; use anyhow::{Result, bail};
use super::model::{DagSpec, Dep, DepWhen, NodeKind, NodeSpec, PermPayload, Source, Template}; use super::model::{DagSpec, Dep, DepWhen, HookKind, NodeKind, NodeSpec, PermPayload, Source};
use crate::coordinator::TransientKind; use crate::coordinator::TransientKind;
/// After-ok edge on the previous node — the common chain link. Shared with /// After-ok edge on the previous node — the common chain link. Shared with
@ -175,7 +175,7 @@ pub(crate) fn deploy_rebuild_nodes(agent: &str) -> Vec<NodeSpec> {
/// children. /// children.
pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> DagSpec { pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> DagSpec {
DagSpec { DagSpec {
template: Template::Rebuild, hook: Some(HookKind::EmitRebuilt),
source, source,
reason, reason,
approval_id: None, approval_id: None,
@ -206,7 +206,7 @@ pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> Dag
pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec { pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec {
let a = || agent.to_owned(); let a = || agent.to_owned();
DagSpec { DagSpec {
template: Template::Rebuild, hook: Some(HookKind::ResolveApproval),
source: Source::Approval, source: Source::Approval,
reason, reason,
approval_id: Some(approval_id), approval_id: Some(approval_id),
@ -235,14 +235,13 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
/// in the queue tests); production paths no longer emit a bare reconcile. /// in the queue tests); production paths no longer emit a bare reconcile.
#[cfg(test)] #[cfg(test)]
pub fn reconcile_only( pub fn reconcile_only(
template: Template,
agent: &str, agent: &str,
source: Source, source: Source,
reason: String, reason: String,
transient: Option<TransientKind>, transient: Option<TransientKind>,
) -> DagSpec { ) -> DagSpec {
DagSpec { DagSpec {
template, hook: None,
source, source,
reason, reason,
approval_id: None, approval_id: None,
@ -268,7 +267,7 @@ pub fn reconcile_only(
/// container was never created). /// container was never created).
pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec { pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
DagSpec { DagSpec {
template: Template::Spawn, hook: Some(HookKind::ResolveApproval),
source: Source::Approval, source: Source::Approval,
reason, reason,
approval_id: Some(approval_id), approval_id: Some(approval_id),
@ -299,7 +298,7 @@ pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPay
)]; )];
nodes.extend(rebuild_nodes(agent, true, 1)); nodes.extend(rebuild_nodes(agent, true, 1));
DagSpec { DagSpec {
template: Template::PermChange, hook: Some(HookKind::EmitRebuilt),
source, source,
reason, reason,
approval_id: None, approval_id: None,
@ -326,7 +325,9 @@ pub fn meta_update(
approval_id: Option<i64>, approval_id: Option<i64>,
) -> DagSpec { ) -> DagSpec {
DagSpec { DagSpec {
template: Template::MetaUpdate, // The bump itself has no side effect; an approval-driven one still has
// its row to resolve.
hook: approval_id.map(|_| HookKind::ResolveApproval),
source, source,
reason, reason,
approval_id, approval_id,
@ -350,24 +351,14 @@ pub fn meta_update(
/// (dashboard tree, `<parent>`/`<children>` sentinel routing, permission /// (dashboard tree, `<parent>`/`<children>` sentinel routing, permission
/// checks), so a parent move needs no container rebuild to take effect. /// checks), so a parent move needs no container rebuild to take effect.
/// No transient pill either — the node is agentless (no lease to hang one /// No transient pill either — the node is agentless (no lease to hang one
/// off of) and near-instant. /// off of) and near-instant. No terminal hook: the write is the whole effect.
///
/// Rides `Template::MetaUpdate` rather than a dedicated variant because
/// `Template` is being removed and nothing should dispatch on it — a fresh
/// variant would just be more surface to delete later. `Template` is already
/// internal-only (not on `DagView`'s wire shape — the dashboard derives its
/// label from `nodes`), so the choice of stand-in variant only affects
/// `terminal_hook` dispatch (`MetaUpdate` resolves to `None`, same as a
/// dedicated variant would) and the per-template history-retention bucket —
/// both cosmetic. Swap this to whatever the eventual node-kind-derived
/// dispatch lands with, whenever it lands.
pub fn reparent( pub fn reparent(
moves: Vec<(hive_types::Ident, Option<hive_types::Ident>)>, moves: Vec<(hive_types::Ident, Option<hive_types::Ident>)>,
source: Source, source: Source,
reason: String, reason: String,
) -> DagSpec { ) -> DagSpec {
DagSpec { DagSpec {
template: Template::MetaUpdate, hook: None,
source, source,
reason, reason,
approval_id: None, approval_id: None,
@ -388,7 +379,7 @@ pub fn reparent(
/// old queue's documented "circular dep silently deadlocks forever" caveat. /// old queue's documented "circular dep silently deadlocks forever" caveat.
pub fn validate(spec: &DagSpec) -> Result<()> { pub fn validate(spec: &DagSpec) -> Result<()> {
if spec.nodes.is_empty() { if spec.nodes.is_empty() {
bail!("dag spec {:?} has no nodes", spec.template); bail!("dag spec {:?} has no nodes", spec.reason);
} }
let n = spec.nodes.len(); let n = spec.nodes.len();
let mut graph = petgraph::graph::DiGraph::<u32, ()>::new(); let mut graph = petgraph::graph::DiGraph::<u32, ()>::new();
@ -404,14 +395,14 @@ pub fn validate(spec: &DagSpec) -> Result<()> {
{ {
bail!( bail!(
"dag spec {:?} node {i} has invalid parent {p} (must be an earlier node)", "dag spec {:?} node {i} has invalid parent {p} (must be an earlier node)",
spec.template spec.reason
); );
} }
for dep in &node.deps { for dep in &node.deps {
let Some(&dep_idx) = usize::try_from(dep.on).ok().and_then(|i| idx.get(i)) else { let Some(&dep_idx) = usize::try_from(dep.on).ok().and_then(|i| idx.get(i)) else {
bail!( bail!(
"dag spec {:?} node {i} depends on unknown node {}", "dag spec {:?} node {i} depends on unknown node {}",
spec.template, spec.reason,
dep.on dep.on
); );
}; };
@ -419,7 +410,7 @@ pub fn validate(spec: &DagSpec) -> Result<()> {
} }
} }
if petgraph::algo::toposort(&graph, None).is_err() { if petgraph::algo::toposort(&graph, None).is_err() {
bail!("dag spec {:?} contains a dependency cycle", spec.template); bail!("dag spec {:?} contains a dependency cycle", spec.reason);
} }
Ok(()) Ok(())
} }

71
hive-c0re/src/job_queue/tests.rs
View file

@ -300,13 +300,7 @@ fn lease_serializes_two_lifecycle_dags_for_same_agent() {
let restart = submit(&q, restart_online(&["agent-a"], false, "restart")); let restart = submit(&q, restart_online(&["agent-a"], false, "restart"));
let stop = submit( let stop = submit(
&q, &q,
templates::reconcile_only( templates::reconcile_only("agent-a", Source::Manual, "stop".to_owned(), None),
Template::Stop,
"agent-a",
Source::Manual,
"stop".to_owned(),
None,
),
); );
// Restart's first node (StopForUpdate) takes the lease; stop's // Restart's first node (StopForUpdate) takes the lease; stop's
// Reconcile must wait even though slots are free. // Reconcile must wait even though slots are free.
@ -337,13 +331,7 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
submit(&q, rebuild("agent-a", "rebuild")); submit(&q, rebuild("agent-a", "rebuild"));
let stop = submit( let stop = submit(
&q, &q,
templates::reconcile_only( templates::reconcile_only("agent-a", Source::Manual, "stop".to_owned(), None),
Template::Stop,
"agent-a",
Source::Manual,
"stop".to_owned(),
None,
),
); );
// Both DAGs' heads are lease-independent of each other: the rebuild's // Both DAGs' heads are lease-independent of each other: the rebuild's
// MetaSync (meta window) and the stop's Reconcile (agent lease). // MetaSync (meta window) and the stop's Reconcile (agent lease).
@ -603,7 +591,7 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
// the emitter and its LOCAL 0-based deps are rebased onto the DAG. // the emitter and its LOCAL 0-based deps are rebased onto the DAG.
let q = JobQueue::new(4); let q = JobQueue::new(4);
let spec = DagSpec { let spec = DagSpec {
template: Template::Boot, hook: None,
source: Source::AutoUpdate, source: Source::AutoUpdate,
reason: "sweep".to_owned(), reason: "sweep".to_owned(),
approval_id: None, approval_id: None,
@ -831,13 +819,7 @@ fn failed_reconcile_marks_dag_failed() {
let q = JobQueue::new(1); let q = JobQueue::new(1);
let id = submit( let id = submit(
&q, &q,
templates::reconcile_only( templates::reconcile_only("agent-a", Source::Manual, "start".to_owned(), None),
Template::Start,
"agent-a",
Source::Manual,
"start".to_owned(),
None,
),
); );
let c = claim_one(&q); let c = claim_one(&q);
q.complete_node(id, c.node_id, Err("start failed".to_owned())); q.complete_node(id, c.node_id, Err("start failed".to_owned()));
@ -917,8 +899,7 @@ fn cancelled_power_op_fires_no_hook() {
let summary = q.cancel(id).expect("cancelled while queued"); let summary = q.cancel(id).expect("cancelled while queued");
assert_eq!(summary.state, State::Cancelled); assert_eq!(summary.state, State::Cancelled);
assert_eq!( assert_eq!(
terminal_hook(summary.template, summary.approval_id), summary.hook, None,
None,
"cancelled {name} (graceful={graceful}, running={running}) must \ "cancelled {name} (graceful={graceful}, running={running}) must \
fire no hook — no node of it ever ran" fire no hook — no node of it ever ran"
); );
@ -951,13 +932,7 @@ fn dag_settles_terminal_and_releases_lease_after_work() {
// immediately. // immediately.
let next = submit( let next = submit(
&q, &q,
templates::reconcile_only( templates::reconcile_only("agent-a", Source::Manual, "stop".to_owned(), None),
Template::Stop,
"agent-a",
Source::Manual,
"stop".to_owned(),
None,
),
); );
let c = claim_one(&q); let c = claim_one(&q);
assert_eq!(c.dag_id, next); assert_eq!(c.dag_id, next);
@ -1220,14 +1195,20 @@ fn set_build_log_id_links_running_node() {
); );
} }
/// History retention is a **flat** newest-first cap over all terminal DAGs
/// (`MAX_HISTORY_DAGS`), not a per-template bucket behind a grace window.
/// The dashboard renders one recent-builds list, so one number bounds it —
/// and with no bucketing there's nothing for a burst of same-shaped DAGs to
/// evict early, which is what the grace window used to paper over.
#[test] #[test]
fn history_evicts_old_terminals_per_template() { fn history_evicts_oldest_terminals_past_flat_cap() {
const OVERFLOW: usize = 8;
let q = JobQueue::new(1); let q = JobQueue::new(1);
for i in 0..8 { let mut ids = Vec::new();
for i in 0..(MAX_HISTORY_DAGS + OVERFLOW) {
let id = submit( let id = submit(
&q, &q,
templates::reconcile_only( templates::reconcile_only(
Template::Start,
&format!("agent-{i}"), &format!("agent-{i}"),
Source::Manual, Source::Manual,
"start".to_owned(), "start".to_owned(),
@ -1241,17 +1222,19 @@ fn history_evicts_old_terminals_per_template() {
// node rolls the container up terminal (its inline hook fires off the // node rolls the container up terminal (its inline hook fires off the
// returned summary — no terminal-hook node). // returned summary — no terminal-hook node).
q.complete_node(id, c.node_id, Err("boom".to_owned())); q.complete_node(id, c.node_id, Err("boom".to_owned()));
ids.push(id);
}
let kept: std::collections::HashSet<u64> = q.snapshot().iter().map(|d| d.id).collect();
assert_eq!(kept.len(), MAX_HISTORY_DAGS, "flat history cap");
// Newest-first: the oldest `OVERFLOW` fall off, everything after survives.
// These DAGs settle within the same wall-clock second, so this also pins
// the `NodeId`-descending tiebreak that orders them when `finished_at` ties.
for old in &ids[..OVERFLOW] {
assert!(!kept.contains(old), "oldest terminal {old} evicted");
}
for recent in &ids[OVERFLOW..] {
assert!(kept.contains(recent), "recent terminal {recent} retained");
} }
// Fresh terminals are inside the grace window: nothing evicts yet,
// so a ~1s QueueDag poller can still observe every terminal state
// (a broad stop/start settles many same-template DAGs at once).
assert_eq!(
q.snapshot().len(),
8,
"grace window protects fresh terminals"
);
// Past the grace window the per-template cap applies.
assert_eq!(q.snapshot_no_grace().len(), 5, "per-template history cap");
assert_eq!(q.live_count(), 0); assert_eq!(q.live_count(), 0);
} }

6
hive-c0re/src/workers/auto_update.rs
View file

@ -303,7 +303,7 @@ fn submit_boot_tree(
n_deferred: usize, n_deferred: usize,
n_skipped: usize, n_skipped: usize,
) { ) {
use crate::job_queue::{DagSpec, NodeKind, NodeSpec, Source, Template}; use crate::job_queue::{DagSpec, NodeKind, NodeSpec, Source};
// Fully-quiet boot (nothing stale, nothing drifted) submits nothing. // Fully-quiet boot (nothing stale, nothing drifted) submits nothing.
if !any_stale && drifted.is_empty() { if !any_stale && drifted.is_empty() {
@ -343,7 +343,9 @@ fn submit_boot_tree(
} }
let spec = DagSpec { let spec = DagSpec {
template: Template::Boot, // The sweep's own rebuild subgraphs emit their `Rebuilt` events as they
// land; the boot DAG as a whole has no terminal side effect.
hook: None,
source: Source::AutoUpdate, source: Source::AutoUpdate,
reason, reason,
approval_id: None, approval_id: None,