Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent subgraph each (concurrent on their own leases) instead of N DAGs — and each subgraph is now built dynamically from the agent's live running state rather than a fixed template shape: - online agent: the full stop→reconcile (restart: stop-for-update→reconcile) chain; `graceful` prepends signal→drain. - offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a restart of a down agent is really a start). The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee) are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate` nodes are state-conditional. Keeping `Reconcile` in every shape closes the TOCTOU window — a race-up between the `is_running` read and node exec is still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no reliance on an external reconcile sweep. The state-aware assembly needs an async `is_running` read, so it moves out of the pure/sync `templates.rs` into `submit.rs`, layered as pure `*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async `*_many` (reads live state + submits). `templates.rs` keeps only the shared pure primitives (`node`/`after_ok`/`rebuild_nodes`). Callers await the now-async submit fns (server, dashboard, socket_server). Tests exercise both the online and offline shapes via the pure `*_spec` seam. docs/coordinator.md shapes updated.
292 lines
10 KiB
Rust
292 lines
10 KiB
Rust
//! DAG shape builders — every operation as a template over the shared
|
|
//! node primitives — plus submit-time cycle validation (petgraph is
|
|
//! confined to this validation; the runtime store stays the plain
|
|
//! `Vec<Node>` + `deps`).
|
|
//!
|
|
//! Every node carries its own `agent` (there is no DAG-level agent) — the
|
|
//! `node` helper stamps each node's agent. This module holds the *pure*
|
|
//! shape builders (no I/O). The hive-wide **power ops** (`stop` / `start` /
|
|
//! `restart`) are NOT here: their per-agent shape depends on each agent's
|
|
//! live running state (an async `lifecycle::is_running` read), so they are
|
|
//! assembled dynamically in `submit.rs` out of the shared pure primitives
|
|
//! this module exports (`node`, `after_ok`, `rebuild_nodes`) — one
|
|
//! independent per-agent subgraph each, concurrent on its own lease, ONE
|
|
//! DAG for the whole hive-wide op. Power ops write the durable `wanted`
|
|
//! intent via a head `SetWanted(w)` node (holding the agent lease, so
|
|
//! intent+reconcile is atomic per-agent).
|
|
//!
|
|
//! ```text
|
|
//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(any) Reconcile(a)
|
|
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
|
|
//! perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
|
//! meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected a»
|
|
//! ```
|
|
//!
|
|
//! For the dynamic power-op shapes (`stop` / `start` / `restart`, built from
|
|
//! live online/offline state), see `submit.rs`.
|
|
|
|
use anyhow::{Result, bail};
|
|
|
|
use super::model::{DagSpec, Dep, DepWhen, NodeKind, NodeSpec, PermPayload, Source, Template};
|
|
use crate::coordinator::TransientKind;
|
|
|
|
/// After-ok edge on the previous node — the common chain link. Shared with
|
|
/// the async power-op builders in `submit.rs` (which assemble per-agent
|
|
/// chains dynamically from live container state).
|
|
pub(crate) fn after_ok(on: u32) -> Vec<Dep> {
|
|
vec![Dep {
|
|
on,
|
|
when: DepWhen::AfterOk,
|
|
}]
|
|
}
|
|
|
|
/// Build one node targeting `agent`. The single place a node's agent is
|
|
/// stamped. Shared with `submit.rs`'s dynamic power-op builders.
|
|
pub(crate) fn node(agent: &str, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
|
|
NodeSpec {
|
|
agent: agent.to_owned(),
|
|
kind,
|
|
deps,
|
|
}
|
|
}
|
|
|
|
/// The rebuild node chain. `Reconcile` deps on `Swap` with `AfterAny`:
|
|
/// it must run even when the profile swap failed, so a previously-up
|
|
/// agent comes back on its old config (today's recovery-start). This
|
|
/// is the only `AfterAny` edge in v1.
|
|
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u32) -> Vec<NodeSpec> {
|
|
vec![
|
|
node(
|
|
agent,
|
|
NodeKind::Prebuild { relock },
|
|
if base == 0 {
|
|
Vec::new()
|
|
} else {
|
|
after_ok(base - 1)
|
|
},
|
|
),
|
|
node(agent, NodeKind::StopForUpdate, after_ok(base)),
|
|
node(agent, NodeKind::Swap, after_ok(base + 1)),
|
|
node(
|
|
agent,
|
|
NodeKind::Reconcile,
|
|
vec![Dep {
|
|
on: base + 2,
|
|
when: DepWhen::AfterAny,
|
|
}],
|
|
),
|
|
]
|
|
}
|
|
|
|
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
|
|
/// noops when already down; the tail `Reconcile` auto-noops the start
|
|
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
|
/// leaves it stopped). `relock = false` only for meta-update cascade
|
|
/// children.
|
|
pub fn rebuild(
|
|
agent: &str,
|
|
source: Source,
|
|
reason: String,
|
|
parent_id: Option<u64>,
|
|
relock: bool,
|
|
) -> DagSpec {
|
|
DagSpec {
|
|
template: Template::Rebuild,
|
|
source,
|
|
reason,
|
|
parent_id,
|
|
approval_id: None,
|
|
inputs: Vec::new(),
|
|
perm_payload: None,
|
|
transient: Some(TransientKind::Rebuilding),
|
|
nodes: rebuild_nodes(agent, relock, 0),
|
|
}
|
|
}
|
|
|
|
/// Approval-driven deploy (`ApplyCommit` / `MergeConfigPr`): the whole
|
|
/// two-phase pipeline stays one opaque node in v1 (design doc §9) —
|
|
/// wire-visible as a `rebuild` card like today.
|
|
pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
|
DagSpec {
|
|
template: Template::Rebuild,
|
|
source: Source::Approval,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id: Some(approval_id),
|
|
inputs: Vec::new(),
|
|
perm_payload: None,
|
|
transient: Some(TransientKind::Rebuilding),
|
|
nodes: vec![node(agent, NodeKind::ApprovalDeploy, Vec::new())],
|
|
}
|
|
}
|
|
|
|
/// Boot-time reconcile: a single `Reconcile` node that converges observed
|
|
/// power state to the persisted intent — `wanted` is untouched (no
|
|
/// `SetWanted`), unlike the operator `start`/`stop` templates. Used only
|
|
/// by the boot sweep now.
|
|
pub fn reconcile_only(
|
|
template: Template,
|
|
agent: &str,
|
|
source: Source,
|
|
reason: String,
|
|
transient: Option<TransientKind>,
|
|
) -> DagSpec {
|
|
DagSpec {
|
|
template,
|
|
source,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id: None,
|
|
inputs: Vec::new(),
|
|
perm_payload: None,
|
|
transient,
|
|
nodes: vec![node(agent, NodeKind::Reconcile, Vec::new())],
|
|
}
|
|
}
|
|
|
|
/// First-deploy spawn (approval-driven): `Provision` (proposed/applied
|
|
/// repos, state subvolume, meta registration) then `Create`
|
|
/// (`nixos-container create`), drop-in write, then `Reconcile` starts
|
|
/// the container (`wanted = Up` written at approve time).
|
|
pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
|
DagSpec {
|
|
template: Template::Spawn,
|
|
source: Source::Approval,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id: Some(approval_id),
|
|
inputs: Vec::new(),
|
|
perm_payload: None,
|
|
transient: Some(TransientKind::Spawning),
|
|
nodes: vec![
|
|
node(agent, NodeKind::Provision, Vec::new()),
|
|
node(agent, NodeKind::Create, after_ok(0)),
|
|
node(agent, NodeKind::WriteDropin, after_ok(1)),
|
|
node(agent, NodeKind::Reconcile, after_ok(2)),
|
|
],
|
|
}
|
|
}
|
|
|
|
/// Perm change: commit the JSON file(s), then the rebuild subgraph so
|
|
/// the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes
|
|
/// effect in the container.
|
|
pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPayload) -> DagSpec {
|
|
let mut nodes = vec![node(agent, NodeKind::WritePermFile, Vec::new())];
|
|
nodes.extend(rebuild_nodes(agent, true, 1));
|
|
DagSpec {
|
|
template: Template::PermChange,
|
|
source,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id: None,
|
|
inputs: Vec::new(),
|
|
perm_payload: Some(payload),
|
|
transient: Some(TransientKind::Rebuilding),
|
|
nodes,
|
|
}
|
|
}
|
|
|
|
/// Meta-input lock bump. Child `Rebuild` DAGs fan out on completion —
|
|
/// appended *after* the bump lands so their prebuilds run against the
|
|
/// post-bump lock (and so a failed bump simply fans out nothing,
|
|
/// replacing the old pre-enqueue + `cancel_children` dance).
|
|
pub fn meta_update(
|
|
inputs: Vec<String>,
|
|
source: Source,
|
|
reason: String,
|
|
approval_id: Option<i64>,
|
|
) -> DagSpec {
|
|
DagSpec {
|
|
template: Template::MetaUpdate,
|
|
source,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id,
|
|
inputs,
|
|
perm_payload: None,
|
|
transient: None,
|
|
nodes: vec![node(
|
|
"hyperhive",
|
|
NodeKind::MetaLock {
|
|
sweep: false,
|
|
fanout: None,
|
|
},
|
|
Vec::new(),
|
|
)],
|
|
}
|
|
}
|
|
|
|
/// Boot-time root anchor DAG: a single [`NodeKind::Noop`] node that groups
|
|
/// this boot's `StartupSweep` + per-agent `Reconcile` child DAGs (linked via
|
|
/// `parent_id`) into one tree so the dashboard renders the boot as one entry.
|
|
/// Holds no lease and does no work — the children it anchors still run
|
|
/// concurrently. `auto_update::run` submits it first (when there's any boot
|
|
/// work), then parents the sweep + reconciles onto its id.
|
|
pub fn boot_root(reason: String) -> DagSpec {
|
|
DagSpec {
|
|
template: Template::Boot,
|
|
source: Source::AutoUpdate,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id: None,
|
|
inputs: Vec::new(),
|
|
perm_payload: None,
|
|
transient: None,
|
|
nodes: vec![node("hyperhive", NodeKind::Noop, Vec::new())],
|
|
}
|
|
}
|
|
|
|
/// Boot-time sweep parent: bump meta's hyperhive input (non-fatal),
|
|
/// then fan out `Rebuild` children for the precomputed stale agent
|
|
/// list (topology-sorted by the caller).
|
|
pub fn startup_sweep(reason: String, stale_agents: Vec<String>) -> DagSpec {
|
|
DagSpec {
|
|
template: Template::StartupSweep,
|
|
source: Source::AutoUpdate,
|
|
reason,
|
|
parent_id: None,
|
|
approval_id: None,
|
|
inputs: Vec::new(),
|
|
perm_payload: None,
|
|
transient: None,
|
|
nodes: vec![node(
|
|
"hyperhive",
|
|
NodeKind::MetaLock {
|
|
sweep: true,
|
|
fanout: Some(stale_agents),
|
|
},
|
|
Vec::new(),
|
|
)],
|
|
}
|
|
}
|
|
|
|
/// Validate a spec before it enters the queue: node ids are dense
|
|
/// (index = id), deps reference existing nodes, and the dep graph is
|
|
/// acyclic (petgraph `toposort`). Rejecting cycles here fixes the old
|
|
/// queue's documented "circular dep silently deadlocks forever" caveat.
|
|
pub fn validate(spec: &DagSpec) -> Result<()> {
|
|
if spec.nodes.is_empty() {
|
|
bail!("dag spec {:?} has no nodes", spec.template);
|
|
}
|
|
let n = spec.nodes.len();
|
|
let mut graph = petgraph::graph::DiGraph::<u32, ()>::new();
|
|
let idx: Vec<_> = (0..n)
|
|
.map(|i| graph.add_node(u32::try_from(i).unwrap_or(u32::MAX)))
|
|
.collect();
|
|
for (i, node) in spec.nodes.iter().enumerate() {
|
|
for dep in &node.deps {
|
|
let Some(&dep_idx) = idx.get(dep.on as usize) else {
|
|
bail!(
|
|
"dag spec {:?} node {i} depends on unknown node {}",
|
|
spec.template,
|
|
dep.on
|
|
);
|
|
};
|
|
graph.add_edge(dep_idx, idx[i], ());
|
|
}
|
|
}
|
|
if petgraph::algo::toposort(&graph, None).is_err() {
|
|
bail!("dag spec {:?} contains a dependency cycle", spec.template);
|
|
}
|
|
Ok(())
|
|
}
|