vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

restart preserves wanted intent instead of forcing all agents up (#2540)

Browse source
This commit is contained in:
damocles 2026-07-16 20:05:27 +02:00 • committed by mara
commit 5bb5a88aa0
4 changed files with 77 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

58
hive-c0re/src/job_queue/submit.rs
View file

@ -9,9 +9,11 @@
//! (`templates::{node, after_ok, rebuild_nodes}`), and concatenate them into
//! ONE DAG (independent per-agent roots, concurrent on their own leases).
//!
//! Dynamic shape rule: the head `SetWanted(w)` (durable intent) and the tail
//! `Reconcile` (the convergence guarantee — cheap, noops when already
//! converged) are ALWAYS present; only the *mechanical* nodes
//! Dynamic shape rule: `stop`/`start` carry a head `SetWanted(w)` (durable
//! intent write) — `restart` does NOT (it bounces the container but leaves
//! `wanted` untouched, so a deliberately-stopped agent isn't forced up). The
//! tail `Reconcile` (the convergence guarantee — cheap, noops when already
//! converged) is ALWAYS present; only the *mechanical* nodes
//! (`Signal`/`Drain`/`StopForUpdate`) are state-conditional (skipped for a
//! down agent — nothing to quiesce/stop). Keeping `Reconcile` in every shape
//! closes the TOCTOU window: if an agent flips state between the `is_running`
@ -85,26 +87,32 @@ fn start_chain(agent: &str, running: bool, stale: bool) -> Vec<NodeSpec> {
n
}
/// One agent's **restart** subgraph. `SetWanted(Up)` head + `Reconcile`
/// tail; the stop portion (`Signal → Drain` when graceful, then
/// `StopForUpdate`) only when the agent is running — a restart of a down
/// agent is really a start (`SetWanted(Up) → Reconcile`).
/// One agent's **restart** subgraph. Restart NEVER rewrites `wanted`
/// intent (no `SetWanted` head, unlike stop/start): it bounces the
/// container and lets the tail `Reconcile` converge to the agent's
/// EXISTING intent, so a deliberately-stopped (`wanted = Off`) agent is
/// not forced back up by a hive-wide restart. A running agent gets the
/// mechanical stop (`Signal → Drain` when graceful, then `StopForUpdate`)
/// before `Reconcile`; a down agent gets just `Reconcile`, which
/// converges to intent — a stopped (`wanted = Off`) agent stays stopped,
/// a crashed (`wanted = Up`) agent comes back up.
fn restart_chain(agent: &str, graceful: bool, running: bool) -> Vec<NodeSpec> {
let mut n = vec![node(agent, NodeKind::SetWanted { up: true }, Vec::new())];
if running {
let mut prev = 0u32;
if graceful {
n.push(node(agent, NodeKind::Signal, after_ok(prev)));
prev += 1;
n.push(node(agent, NodeKind::Drain, after_ok(prev)));
prev += 1;
}
n.push(node(agent, NodeKind::StopForUpdate, after_ok(prev)));
prev += 1;
n.push(node(agent, NodeKind::Reconcile, after_ok(prev)));
} else {
n.push(node(agent, NodeKind::Reconcile, after_ok(0)));
if !running {
// Nothing to bounce — a lone Reconcile converges to intent.
return vec![node(agent, NodeKind::Reconcile, Vec::new())];
}
// Running: mechanical stop then Reconcile. The first stop node is the
// subgraph root (no SetWanted head) and acquires the agent lease.
let mut n = Vec::new();
if graceful {
n.push(node(agent, NodeKind::Signal, Vec::new()));
n.push(node(agent, NodeKind::Drain, after_ok(0)));
n.push(node(agent, NodeKind::StopForUpdate, after_ok(1)));
} else {
n.push(node(agent, NodeKind::StopForUpdate, Vec::new()));
}
let stop_idx = u32::try_from(n.len() - 1).unwrap_or(0);
n.push(node(agent, NodeKind::Reconcile, after_ok(stop_idx)));
n
}
@ -258,9 +266,11 @@ pub async fn graceful_restart(
/// Restart `agents` (one or many) in a **single** DAG — one per-agent
/// subgraph each, built dynamically from live running state and run
/// concurrently on their own leases. A running agent gets the stop→reconcile
/// chain (`graceful` prepends signal→drain); a down agent gets just
/// `SetWanted(Up) → Reconcile` (there's nothing to stop). The whole
/// hive-wide `hivectl restart` / `restart-all` is one DAG.
/// chain (`graceful` prepends signal→drain); a down agent gets just a lone
/// `Reconcile` (nothing to stop). Restart never writes `wanted`, so the
/// tail `Reconcile` converges each agent to its EXISTING intent — a
/// deliberately-stopped agent stays down. The whole hive-wide
/// `hivectl restart` / `restart-all` is one DAG.
pub async fn restart_many(
coord: &Arc<Coordinator>,
agents: &[String],