refactor(#2439): build hive-wide stop/start/restart DAGs dynamically
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent subgraph each (concurrent on their own leases) instead of N DAGs — and each subgraph is now built dynamically from the agent's live running state rather than a fixed template shape: - online agent: the full stop→reconcile (restart: stop-for-update→reconcile) chain; `graceful` prepends signal→drain. - offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a restart of a down agent is really a start). The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee) are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate` nodes are state-conditional. Keeping `Reconcile` in every shape closes the TOCTOU window — a race-up between the `is_running` read and node exec is still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no reliance on an external reconcile sweep. The state-aware assembly needs an async `is_running` read, so it moves out of the pure/sync `templates.rs` into `submit.rs`, layered as pure `*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async `*_many` (reads live state + submits). `templates.rs` keeps only the shared pure primitives (`node`/`after_ok`/`rebuild_nodes`). Callers await the now-async submit fns (server, dashboard, socket_server). Tests exercise both the online and offline shapes via the pure `*_spec` seam. docs/coordinator.md shapes updated.
This commit is contained in:
atlas
parent
3797177e7f
commit
860484a193
8 changed files with 574 additions and 335 deletions
|
|
@ -4,47 +4,45 @@
|
|||
//! `Vec<Node>` + `deps`).
|
||||
//!
|
||||
//! Every node carries its own `agent` (there is no DAG-level agent) — the
|
||||
//! `node` helper stamps the template's agent onto each. `restart` takes an
|
||||
//! agent *list* and stamps each agent onto its own subgraph, so a hive-wide
|
||||
//! `hivectl restart` is ONE DAG with N independent per-agent subgraphs
|
||||
//! (each a root chain, run concurrently on its own lease) rather than N
|
||||
//! separate DAGs. The other templates are still single-agent.
|
||||
//!
|
||||
//! The power ops write the durable `wanted` intent via a head
|
||||
//! `SetWanted(w)` node (not a pre-submit side effect); it holds the agent
|
||||
//! lease so intent+reconcile is atomic per-agent.
|
||||
//! `node` helper stamps each node's agent. This module holds the *pure*
|
||||
//! shape builders (no I/O). The hive-wide **power ops** (`stop` / `start` /
|
||||
//! `restart`) are NOT here: their per-agent shape depends on each agent's
|
||||
//! live running state (an async `lifecycle::is_running` read), so they are
|
||||
//! assembled dynamically in `submit.rs` out of the shared pure primitives
|
||||
//! this module exports (`node`, `after_ok`, `rebuild_nodes`) — one
|
||||
//! independent per-agent subgraph each, concurrent on its own lease, ONE
|
||||
//! DAG for the whole hive-wide op. Power ops write the durable `wanted`
|
||||
//! intent via a head `SetWanted(w)` node (holding the agent lease, so
|
||||
//! intent+reconcile is atomic per-agent).
|
||||
//!
|
||||
//! ```text
|
||||
//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(any) Reconcile(a)
|
||||
//! graceful-stop(a): SetWanted(a,Off) → Signal(a) → Drain(a) → Reconcile(a)
|
||||
//! restart(a): SetWanted(a,Up) → StopForUpdate(a) → Reconcile(a)
|
||||
//! graceful-restart(a): SetWanted(a,Up) → Signal(a) → Drain(a) → StopForUpdate(a) → Reconcile(a)
|
||||
//! start(a): SetWanted(a,Up) → Reconcile(a)
|
||||
//! stop(a): SetWanted(a,Off) → Reconcile(a)
|
||||
//! stale-start(a): SetWanted(a,Up) → «rebuild subgraph» (rev stale; prebuild noops, agent down)
|
||||
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
|
||||
//! perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
||||
//! meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected a»
|
||||
//! startup sweep: MetaLock(hyperhive, non-fatal) → «fan-out rebuild(stale a)»
|
||||
//! ```
|
||||
//!
|
||||
//! For the dynamic power-op shapes (`stop` / `start` / `restart`, built from
|
||||
//! live online/offline state), see `submit.rs`.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
|
||||
use super::model::{DagSpec, Dep, DepWhen, NodeKind, NodeSpec, PermPayload, Source, Template};
|
||||
use crate::coordinator::TransientKind;
|
||||
|
||||
/// After-ok edge on the previous node — the common chain link.
|
||||
fn after_ok(on: u32) -> Vec<Dep> {
|
||||
/// After-ok edge on the previous node — the common chain link. Shared with
|
||||
/// the async power-op builders in `submit.rs` (which assemble per-agent
|
||||
/// chains dynamically from live container state).
|
||||
pub(crate) fn after_ok(on: u32) -> Vec<Dep> {
|
||||
vec![Dep {
|
||||
on,
|
||||
when: DepWhen::AfterOk,
|
||||
}]
|
||||
}
|
||||
|
||||
/// Build one node targeting `agent`. The single place templates stamp a
|
||||
/// node's agent, so a whole template is single-agent by passing the same
|
||||
/// `agent` to every call.
|
||||
fn node(agent: &str, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
|
||||
/// Build one node targeting `agent`. The single place a node's agent is
|
||||
/// stamped. Shared with `submit.rs`'s dynamic power-op builders.
|
||||
pub(crate) fn node(agent: &str, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
|
||||
NodeSpec {
|
||||
agent: agent.to_owned(),
|
||||
kind,
|
||||
|
|
@ -56,7 +54,7 @@ fn node(agent: &str, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
|
|||
/// it must run even when the profile swap failed, so a previously-up
|
||||
/// agent comes back on its old config (today's recovery-start). This
|
||||
/// is the only `AfterAny` edge in v1.
|
||||
fn rebuild_nodes(agent: &str, relock: bool, base: u32) -> Vec<NodeSpec> {
|
||||
pub(crate) fn rebuild_nodes(agent: &str, relock: bool, base: u32) -> Vec<NodeSpec> {
|
||||
vec![
|
||||
node(
|
||||
agent,
|
||||
|
|
@ -122,73 +120,6 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
|
|||
}
|
||||
}
|
||||
|
||||
/// Graceful stop: cheap `Signal` fires immediately (no build slot), the
|
||||
/// `Drain` awaits the harness checkpoint (bounded), and the tail
|
||||
/// `Reconcile` performs the actual container stop. The head `SetWanted`
|
||||
/// node writes `wanted = Offline` as part of the DAG (was a pre-submit
|
||||
/// side effect). A whole-hive graceful stop therefore signals every agent
|
||||
/// up front and overlaps every drain, replacing the old detached-watcher
|
||||
/// thread structurally.
|
||||
pub fn graceful_stop(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::GracefulStop,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Stopping),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::SetWanted { up: false }, Vec::new()),
|
||||
node(agent, NodeKind::Signal, after_ok(0)),
|
||||
node(agent, NodeKind::Drain, after_ok(1)),
|
||||
node(agent, NodeKind::Reconcile, after_ok(2)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Restart one or more agents in a **single** DAG. Each agent gets an
|
||||
/// independent subgraph — a head `SetWanted(Up)` (a root: no cross-agent
|
||||
/// dep) then its restart chain — so all agents' restarts run concurrently,
|
||||
/// each taking its own agent lease. `graceful` inserts `Signal → Drain`
|
||||
/// before the mechanical `StopForUpdate` (each agent's subgraph is still one
|
||||
/// atomic restart). One agent = the ordinary single-agent restart; many =
|
||||
/// a hive-wide `hivectl restart` as one DAG instead of N separate ones.
|
||||
pub fn restart(agents: &[String], graceful: bool, source: Source, reason: String) -> DagSpec {
|
||||
let mut nodes = Vec::new();
|
||||
for agent in agents {
|
||||
let base = u32::try_from(nodes.len()).unwrap_or(u32::MAX);
|
||||
// Head of this agent's subgraph — a root (empty deps), so the N
|
||||
// per-agent subgraphs are independent and run concurrently.
|
||||
nodes.push(node(agent, NodeKind::SetWanted { up: true }, Vec::new()));
|
||||
if graceful {
|
||||
nodes.push(node(agent, NodeKind::Signal, after_ok(base)));
|
||||
nodes.push(node(agent, NodeKind::Drain, after_ok(base + 1)));
|
||||
nodes.push(node(agent, NodeKind::StopForUpdate, after_ok(base + 2)));
|
||||
nodes.push(node(agent, NodeKind::Reconcile, after_ok(base + 3)));
|
||||
} else {
|
||||
nodes.push(node(agent, NodeKind::StopForUpdate, after_ok(base)));
|
||||
nodes.push(node(agent, NodeKind::Reconcile, after_ok(base + 1)));
|
||||
}
|
||||
}
|
||||
DagSpec {
|
||||
template: if graceful {
|
||||
Template::GracefulRestart
|
||||
} else {
|
||||
Template::Restart
|
||||
},
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Restarting),
|
||||
nodes,
|
||||
}
|
||||
}
|
||||
|
||||
/// Boot-time reconcile: a single `Reconcile` node that converges observed
|
||||
/// power state to the persisted intent — `wanted` is untouched (no
|
||||
/// `SetWanted`), unlike the operator `start`/`stop` templates. Used only
|
||||
|
|
@ -213,68 +144,6 @@ pub fn reconcile_only(
|
|||
}
|
||||
}
|
||||
|
||||
/// Start: write `wanted = Up` (head `SetWanted`), then reconcile (which
|
||||
/// starts the container). The intent write is a DAG node now, not a
|
||||
/// pre-submit side effect.
|
||||
pub fn start(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::Start,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Starting),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::SetWanted { up: true }, Vec::new()),
|
||||
node(agent, NodeKind::Reconcile, after_ok(0)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Stop: write `wanted = Offline` (head `SetWanted`), then reconcile
|
||||
/// (kill + unregister + `Killed` event).
|
||||
pub fn stop(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::Stop,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Stopping),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::SetWanted { up: false }, Vec::new()),
|
||||
node(agent, NodeKind::Reconcile, after_ok(0)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Stale start: a `start` whose rev marker is stale, so it rebuilds
|
||||
/// before coming up — `SetWanted(Up)` → «rebuild subgraph» → `Reconcile`
|
||||
/// (the rebuild's tail `Reconcile` starts it, since `wanted = Up`). The
|
||||
/// rebuild nodes are the same `rebuild_nodes` chain a manual rebuild uses
|
||||
/// (reused, not a variant); only the leading `SetWanted(Up)` intent
|
||||
/// differs. Shows as a `Rebuild` on the dashboard like the old
|
||||
/// submit-time upgrade did.
|
||||
pub fn stale_start(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
let mut nodes = vec![node(agent, NodeKind::SetWanted { up: true }, Vec::new())];
|
||||
nodes.extend(rebuild_nodes(agent, true, 1));
|
||||
DagSpec {
|
||||
template: Template::Rebuild,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Rebuilding),
|
||||
nodes,
|
||||
}
|
||||
}
|
||||
|
||||
/// First-deploy spawn (approval-driven): `Provision` (proposed/applied
|
||||
/// repos, state subvolume, meta registration) then `Create`
|
||||
/// (`nixos-container create`), drop-in write, then `Reconcile` starts
|
||||
|
|
|
|||
Loading…
Reference in a new issue