refactor(#2439): build hive-wide stop/start/restart DAGs dynamically
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent subgraph each (concurrent on their own leases) instead of N DAGs — and each subgraph is now built dynamically from the agent's live running state rather than a fixed template shape: - online agent: the full stop→reconcile (restart: stop-for-update→reconcile) chain; `graceful` prepends signal→drain. - offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a restart of a down agent is really a start). The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee) are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate` nodes are state-conditional. Keeping `Reconcile` in every shape closes the TOCTOU window — a race-up between the `is_running` read and node exec is still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no reliance on an external reconcile sweep. The state-aware assembly needs an async `is_running` read, so it moves out of the pure/sync `templates.rs` into `submit.rs`, layered as pure `*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async `*_many` (reads live state + submits). `templates.rs` keeps only the shared pure primitives (`node`/`after_ok`/`rebuild_nodes`). Callers await the now-async submit fns (server, dashboard, socket_server). Tests exercise both the online and offline shapes via the pure `*_spec` seam. docs/coordinator.md shapes updated.
This commit is contained in:
atlas
parent
3797177e7f
commit
860484a193
8 changed files with 574 additions and 335 deletions
|
|
@ -1,15 +1,32 @@
|
|||
//! Request-level submit API — the surface the dashboard POST handlers,
|
||||
//! the MCP socket handlers, and `hivectl` paths call. The durable
|
||||
//! `wanted` power intent is now written by a `SetWanted` DAG node at the
|
||||
//! head of each power-op template (not a pre-submit side effect); the
|
||||
//! only submit-time logic left is the stale-start *shape* decision
|
||||
//! (`start` vs `stale_start`). Every helper emits a fresh queue snapshot
|
||||
//! so the dashboard shows the new DAG immediately.
|
||||
//! the MCP socket handlers, and `hivectl` paths call.
|
||||
//!
|
||||
//! The **power ops** (`stop` / `start` / `restart`) are built here, not in
|
||||
//! `templates.rs`: each agent's subgraph shape depends on its *live* running
|
||||
//! state, which needs an async `lifecycle::is_running` read that a pure/sync
|
||||
//! template can't do. So these fns are async — they read each agent's state,
|
||||
//! assemble a per-agent subgraph out of the shared pure primitives
|
||||
//! (`templates::{node, after_ok, rebuild_nodes}`), and concatenate them into
|
||||
//! ONE DAG (independent per-agent roots, concurrent on their own leases).
|
||||
//!
|
||||
//! Dynamic shape rule: the head `SetWanted(w)` (durable intent) and the tail
|
||||
//! `Reconcile` (the convergence guarantee — cheap, noops when already
|
||||
//! converged) are ALWAYS present; only the *mechanical* nodes
|
||||
//! (`Signal`/`Drain`/`StopForUpdate`) are state-conditional (skipped for a
|
||||
//! down agent — nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
//! closes the TOCTOU window: if an agent flips state between the `is_running`
|
||||
//! read and node execution, the tail `Reconcile` still converges it in-DAG,
|
||||
//! with `StopForUpdate`-noop as the backstop — no reliance on an external
|
||||
//! reconcile sweep. Every helper emits a fresh queue snapshot so the
|
||||
//! dashboard shows the new DAG immediately.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::model::{DagSpec, Dep, NodeKind, NodeSpec, Template};
|
||||
use super::templates::{after_ok, node, rebuild_nodes};
|
||||
use super::{Source, templates};
|
||||
use crate::coordinator::Coordinator;
|
||||
use crate::coordinator::{Coordinator, TransientKind};
|
||||
use crate::lifecycle;
|
||||
|
||||
fn submit_and_emit(coord: &Arc<Coordinator>, spec: super::DagSpec) -> u64 {
|
||||
let id = coord
|
||||
|
|
@ -27,73 +44,305 @@ pub fn rebuild(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: St
|
|||
submit_and_emit(coord, templates::rebuild(agent, source, reason, None, true))
|
||||
}
|
||||
|
||||
/// Restart a single agent: mechanical stop + converge to `wanted = Up`.
|
||||
/// The intent write is the template's head `SetWanted(Up)` node — it
|
||||
/// matters when `wanted` drifted `Offline` under a running agent (an
|
||||
/// operator asking for a restart plainly wants it running, not a stop).
|
||||
/// Thin wrapper over [`restart_many`] with a one-agent slice.
|
||||
pub fn restart(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
restart_many(coord, &[agent.to_owned()], false, source, reason)
|
||||
// ---- dynamic power-op DAG assembly ----------------------------------------
|
||||
//
|
||||
// The pure per-agent chain builders below take `running` (and `stale`)
|
||||
// explicitly so they stay pure + unit-testable without a live container;
|
||||
// the async `*_many` fns read the real state via `lifecycle::is_running`
|
||||
// then hand it in. Each chain uses LOCAL (0-based) deps; `concat_subgraphs`
|
||||
// rebases them into one DAG.
|
||||
|
||||
/// One agent's **stop** subgraph. `SetWanted(Off)` head + `Reconcile` tail
|
||||
/// always; the graceful `Signal → Drain` quiesce only when the agent is
|
||||
/// actually running (nothing to drain on a down container). The `Reconcile`
|
||||
/// stays even for a down agent so a race-up between the state read and exec
|
||||
/// is still stopped in-DAG.
|
||||
fn stop_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
|
||||
let mut n = vec![node(agent, NodeKind::SetWanted { up: false }, Vec::new())];
|
||||
if graceful && running {
|
||||
n.push(node(agent, NodeKind::Signal, after_ok(0)));
|
||||
n.push(node(agent, NodeKind::Drain, after_ok(1)));
|
||||
n.push(node(agent, NodeKind::Reconcile, after_ok(2)));
|
||||
} else {
|
||||
n.push(node(agent, NodeKind::Reconcile, after_ok(0)));
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// One agent's **start** subgraph. `SetWanted(Up)` head; a down + stale-rev
|
||||
/// agent gets the rebuild subgraph (its tail `Reconcile` starts it on
|
||||
/// current derivations), otherwise a plain `Reconcile` (which starts a down
|
||||
/// agent and noops an already-running one).
|
||||
fn start_chain(agent: &str, running: bool, stale: bool) -> Vec<NodeSpec> {
|
||||
let mut n = vec![node(agent, NodeKind::SetWanted { up: true }, Vec::new())];
|
||||
if !running && stale {
|
||||
// Rebuild subgraph rooted at the SetWanted head (base = 1, so
|
||||
// `Prebuild` deps `after_ok(0)` = the head).
|
||||
n.extend(rebuild_nodes(agent, true, 1));
|
||||
} else {
|
||||
n.push(node(agent, NodeKind::Reconcile, after_ok(0)));
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// One agent's **restart** subgraph. `SetWanted(Up)` head + `Reconcile`
|
||||
/// tail; the stop portion (`Signal → Drain` when graceful, then
|
||||
/// `StopForUpdate`) only when the agent is running — a restart of a down
|
||||
/// agent is really a start (`SetWanted(Up) → Reconcile`).
|
||||
fn restart_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
|
||||
let mut n = vec![node(agent, NodeKind::SetWanted { up: true }, Vec::new())];
|
||||
if running {
|
||||
let mut prev = 0u32;
|
||||
if graceful {
|
||||
n.push(node(agent, NodeKind::Signal, after_ok(prev)));
|
||||
prev += 1;
|
||||
n.push(node(agent, NodeKind::Drain, after_ok(prev)));
|
||||
prev += 1;
|
||||
}
|
||||
n.push(node(agent, NodeKind::StopForUpdate, after_ok(prev)));
|
||||
prev += 1;
|
||||
n.push(node(agent, NodeKind::Reconcile, after_ok(prev)));
|
||||
} else {
|
||||
n.push(node(agent, NodeKind::Reconcile, after_ok(0)));
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// Concatenate per-agent subgraphs (each with LOCAL 0-based deps) into one
|
||||
/// node list, rebasing each subgraph's internal deps by its offset. A
|
||||
/// subgraph root (empty deps — the `SetWanted` head) stays a root, so the
|
||||
/// per-agent subgraphs are independent and run concurrently, each on its
|
||||
/// own lease.
|
||||
fn concat_subgraphs(chains: Vec<Vec<NodeSpec>>) -> Vec<NodeSpec> {
|
||||
let mut out: Vec<NodeSpec> = Vec::new();
|
||||
for chain in chains {
|
||||
let base = u32::try_from(out.len()).unwrap_or(u32::MAX);
|
||||
for spec in chain {
|
||||
let deps = spec
|
||||
.deps
|
||||
.into_iter()
|
||||
.map(|d| Dep {
|
||||
on: base + d.on,
|
||||
when: d.when,
|
||||
})
|
||||
.collect();
|
||||
out.push(NodeSpec {
|
||||
agent: spec.agent,
|
||||
kind: spec.kind,
|
||||
deps,
|
||||
});
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Wrap assembled power-op `nodes` in a `DagSpec`.
|
||||
fn power_dag(
|
||||
template: Template,
|
||||
transient: TransientKind,
|
||||
source: Source,
|
||||
reason: String,
|
||||
nodes: Vec<NodeSpec>,
|
||||
) -> DagSpec {
|
||||
DagSpec {
|
||||
template,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(transient),
|
||||
nodes,
|
||||
}
|
||||
}
|
||||
|
||||
// The `*_spec` builders below are the PURE core the async `*_many` fns call
|
||||
// after reading live state — they take the per-agent running (and stale)
|
||||
// flags explicitly, so unit tests exercise the online/offline shapes without
|
||||
// a live container. `*_many` = gather state + call `*_spec` + submit.
|
||||
|
||||
/// Assemble the stop DAG from explicit `(agent, running)` targets.
|
||||
pub(crate) fn stop_spec(
|
||||
targets: &[(String, bool)],
|
||||
graceful: bool,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> DagSpec {
|
||||
let chains = targets
|
||||
.iter()
|
||||
.map(|(agent, running)| stop_chain(agent, graceful, *running))
|
||||
.collect();
|
||||
let template = if graceful {
|
||||
Template::GracefulStop
|
||||
} else {
|
||||
Template::Stop
|
||||
};
|
||||
power_dag(
|
||||
template,
|
||||
TransientKind::Stopping,
|
||||
source,
|
||||
reason,
|
||||
concat_subgraphs(chains),
|
||||
)
|
||||
}
|
||||
|
||||
/// Assemble the start DAG from explicit `(agent, running, stale)` targets.
|
||||
/// Transient is `Rebuilding` when any down+stale agent grew a rebuild
|
||||
/// subgraph (crash-watch suppression during its Swap), else `Starting`;
|
||||
/// applied per-agent at claim time, so each agent still shows its own pill.
|
||||
pub(crate) fn start_spec(
|
||||
targets: &[(String, bool, bool)],
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> DagSpec {
|
||||
let chains = targets
|
||||
.iter()
|
||||
.map(|(agent, running, stale)| start_chain(agent, *running, *stale))
|
||||
.collect();
|
||||
let any_rebuild = targets.iter().any(|(_, running, stale)| !running && *stale);
|
||||
let transient = if any_rebuild {
|
||||
TransientKind::Rebuilding
|
||||
} else {
|
||||
TransientKind::Starting
|
||||
};
|
||||
power_dag(
|
||||
Template::Start,
|
||||
transient,
|
||||
source,
|
||||
reason,
|
||||
concat_subgraphs(chains),
|
||||
)
|
||||
}
|
||||
|
||||
/// Assemble the restart DAG from explicit `(agent, running)` targets.
|
||||
pub(crate) fn restart_spec(
|
||||
targets: &[(String, bool)],
|
||||
graceful: bool,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> DagSpec {
|
||||
let chains = targets
|
||||
.iter()
|
||||
.map(|(agent, running)| restart_chain(agent, graceful, *running))
|
||||
.collect();
|
||||
let template = if graceful {
|
||||
Template::GracefulRestart
|
||||
} else {
|
||||
Template::Restart
|
||||
};
|
||||
power_dag(
|
||||
template,
|
||||
TransientKind::Restarting,
|
||||
source,
|
||||
reason,
|
||||
concat_subgraphs(chains),
|
||||
)
|
||||
}
|
||||
|
||||
/// Restart a single agent. Thin wrapper over [`restart_many`].
|
||||
pub async fn restart(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
restart_many(coord, &[agent.to_owned()], false, source, reason).await
|
||||
}
|
||||
|
||||
/// Graceful restart of a single agent (signal → drain → stop → reconcile,
|
||||
/// when running). Thin wrapper over [`restart_many`] with `graceful = true`.
|
||||
pub async fn graceful_restart(
|
||||
coord: &Arc<Coordinator>,
|
||||
agent: &str,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> u64 {
|
||||
restart_many(coord, &[agent.to_owned()], true, source, reason).await
|
||||
}
|
||||
|
||||
/// Restart `agents` (one or many) in a **single** DAG — one per-agent
|
||||
/// subgraph each, running concurrently on their own leases. `graceful`
|
||||
/// prepends the signal→drain quiesce per agent. The whole hive-wide
|
||||
/// `hivectl restart` / `restart-all` is now one DAG instead of N.
|
||||
pub fn restart_many(
|
||||
/// subgraph each, built dynamically from live running state and run
|
||||
/// concurrently on their own leases. A running agent gets the stop→reconcile
|
||||
/// chain (`graceful` prepends signal→drain); a down agent gets just
|
||||
/// `SetWanted(Up) → Reconcile` (there's nothing to stop). The whole
|
||||
/// hive-wide `hivectl restart` / `restart-all` is one DAG.
|
||||
pub async fn restart_many(
|
||||
coord: &Arc<Coordinator>,
|
||||
agents: &[String],
|
||||
graceful: bool,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> u64 {
|
||||
submit_and_emit(coord, templates::restart(agents, graceful, source, reason))
|
||||
}
|
||||
|
||||
/// Start: `SetWanted(Up)` (a DAG node now) then reconcile. A stale rev
|
||||
/// marker upgrades the start to a rebuild-then-start (`stale_start`, whose
|
||||
/// tail `Reconcile` does the start) so the container always comes up on
|
||||
/// current derivations — the old fast-lane `run_start` upgrade, still a
|
||||
/// submit-time *shape* decision (which template), while the intent write
|
||||
/// itself is now the template's head node.
|
||||
pub fn start(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
let stored = std::fs::read_to_string(crate::paths::applied_rev_marker(agent)).ok();
|
||||
let stale = crate::auto_update::current_flake_rev(&coord.hyperhive_flake)
|
||||
.is_some_and(|rev| stored.as_deref() != Some(rev.as_str()));
|
||||
if stale {
|
||||
tracing::info!(%agent, "start: rev stale — upgrading to rebuild+start");
|
||||
return submit_and_emit(
|
||||
coord,
|
||||
templates::stale_start(agent, source, format!("{reason} (stale — rebuild+start)")),
|
||||
);
|
||||
let mut targets = Vec::with_capacity(agents.len());
|
||||
for agent in agents {
|
||||
targets.push((agent.clone(), lifecycle::is_running(agent).await));
|
||||
}
|
||||
submit_and_emit(coord, templates::start(agent, source, reason))
|
||||
submit_and_emit(coord, restart_spec(&targets, graceful, source, reason))
|
||||
}
|
||||
|
||||
/// Hard stop: `SetWanted(Offline)` (a DAG node now) then reconcile (kill +
|
||||
/// unregister + `Killed` event).
|
||||
pub fn stop(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
submit_and_emit(coord, templates::stop(agent, source, reason))
|
||||
/// Start a single agent. Thin wrapper over [`start_many`].
|
||||
pub async fn start(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
start_many(coord, &[agent.to_owned()], source, reason).await
|
||||
}
|
||||
|
||||
/// Graceful stop: signal → drain → reconcile (the actual stop). The head
|
||||
/// `SetWanted(Offline)` node writes the intent as part of the DAG.
|
||||
pub fn graceful_stop(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
submit_and_emit(coord, templates::graceful_stop(agent, source, reason))
|
||||
/// Start `agents` (one or many) in a **single** DAG — one per-agent subgraph
|
||||
/// each, built dynamically from live state and run concurrently on their own
|
||||
/// leases. A down agent gets `SetWanted(Up) → Reconcile` (or, rev stale, a
|
||||
/// rebuild-then-start so it comes up on current derivations); an already-
|
||||
/// running agent gets `SetWanted(Up) → Reconcile` (the reconcile noops). The
|
||||
/// whole hive-wide `hivectl start` is one DAG.
|
||||
pub async fn start_many(
|
||||
coord: &Arc<Coordinator>,
|
||||
agents: &[String],
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> u64 {
|
||||
let current = crate::auto_update::current_flake_rev(&coord.hyperhive_flake);
|
||||
let mut targets = Vec::with_capacity(agents.len());
|
||||
for agent in agents {
|
||||
let running = lifecycle::is_running(agent).await;
|
||||
let stored = std::fs::read_to_string(crate::paths::applied_rev_marker(agent)).ok();
|
||||
let stale = current
|
||||
.as_ref()
|
||||
.is_some_and(|rev| stored.as_deref() != Some(rev.as_str()));
|
||||
if !running && stale {
|
||||
tracing::info!(%agent, "start: rev stale + agent down — rebuild-then-start");
|
||||
}
|
||||
targets.push((agent.clone(), running, stale));
|
||||
}
|
||||
submit_and_emit(coord, start_spec(&targets, source, reason))
|
||||
}
|
||||
|
||||
/// Graceful restart of a single agent: signal → drain → mechanical stop →
|
||||
/// reconcile (starts it back up) — one atomic DAG, no client-side "await
|
||||
/// the stop DAG then submit a start DAG" split. The head `SetWanted(Up)`
|
||||
/// node writes the intent as part of the DAG. Thin wrapper over
|
||||
/// [`restart_many`] with `graceful = true`.
|
||||
pub fn graceful_restart(
|
||||
/// Hard stop a single agent. Thin wrapper over [`stop_many`].
|
||||
pub async fn stop(coord: &Arc<Coordinator>, agent: &str, source: Source, reason: String) -> u64 {
|
||||
stop_many(coord, &[agent.to_owned()], false, source, reason).await
|
||||
}
|
||||
|
||||
/// Graceful stop of a single agent (signal → drain → reconcile, when
|
||||
/// running). Thin wrapper over [`stop_many`] with `graceful = true`.
|
||||
pub async fn graceful_stop(
|
||||
coord: &Arc<Coordinator>,
|
||||
agent: &str,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> u64 {
|
||||
restart_many(coord, &[agent.to_owned()], true, source, reason)
|
||||
stop_many(coord, &[agent.to_owned()], true, source, reason).await
|
||||
}
|
||||
|
||||
/// Stop `agents` (one or many) in a **single** DAG — one per-agent subgraph
|
||||
/// each, built dynamically from live state and run concurrently on their own
|
||||
/// leases. A running agent gets `SetWanted(Off) → [Signal → Drain →](graceful)
|
||||
/// Reconcile`; a down agent gets just `SetWanted(Off) → Reconcile` (skips the
|
||||
/// pointless quiesce, keeps the Reconcile as the race-up backstop). The whole
|
||||
/// hive-wide `hivectl stop` is one DAG.
|
||||
pub async fn stop_many(
|
||||
coord: &Arc<Coordinator>,
|
||||
agents: &[String],
|
||||
graceful: bool,
|
||||
source: Source,
|
||||
reason: String,
|
||||
) -> u64 {
|
||||
let mut targets = Vec::with_capacity(agents.len());
|
||||
for agent in agents {
|
||||
targets.push((agent.clone(), lifecycle::is_running(agent).await));
|
||||
}
|
||||
submit_and_emit(coord, stop_spec(&targets, graceful, source, reason))
|
||||
}
|
||||
|
||||
/// Perm change: commit the JSON file(s) then rebuild.
|
||||
|
|
|
|||
Loading…
Reference in a new issue