refactor(#2449): write power intent via a SetWanted DAG node, not a pre-submit side effect
The durable 'wanted' power intent was written by submit::{start,stop,
restart,graceful_restart,graceful_stop} as a synchronous pre-submit side
effect, then read by the DAG's tail Reconcile. That's not crash-safe
(a crash between the write and the enqueue loses it) and, with agent now
per-node, can't be per-agent in a DAG that spans agents.
Move it into the DAG as a head SetWanted node:
- NodeKind::SetWanted { up } + run_set_wanted executor (fails the node on
a write error, unlike the old warn-and-continue, so a stale intent
never reaches Reconcile).
- LEASE-NEEDING, not lease-exempt: it takes the agent lease so a power-op
DAG's intent-write + reconcile is atomic per-agent. If it were exempt,
two racing ops (restart vs stop) would run both intent-writes up front
and clobber each other before either reconciled — defeating the point
of moving the write into the DAG. (In stale_start the lease is thus held
across the head Prebuild, but that's a no-op there: the agent is down so
prebuild is skipped.)
- templates: explicit SetWanted node 0 on restart/graceful_restart/
graceful_stop, plus dedicated start/stop templates (SetWanted -> Reconcile)
and stale_start (SetWanted(Up) -> rebuild subgraph, reusing rebuild_nodes).
No compose helper / rebuild variant. reconcile_only is now boot-only.
- submit.rs: drop the set_wanted side effect; the stale-rev shape decision
(start vs stale_start) stays submit-side.
All 33 job_queue tests pass (shape/lease tests updated for the head node).
This commit is contained in:
atlas
mara
parent
6c654921a0
commit
5fe8008cce
5 changed files with 189 additions and 100 deletions
|
|
@ -8,14 +8,19 @@
|
|||
//! are single-agent (every node shares one agent); a future multi-agent
|
||||
//! template would stamp different agents per subgraph.
|
||||
//!
|
||||
//! The power ops write the durable `wanted` intent via a head
|
||||
//! `SetWanted(w)` node (not a pre-submit side effect); it holds the agent
|
||||
//! lease so intent+reconcile is atomic per-agent.
|
||||
//!
|
||||
//! ```text
|
||||
//! rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(any) Reconcile(a)
|
||||
//! graceful-stop(a): [wanted=Offline] Signal(a) → Drain(a) → Reconcile(a)
|
||||
//! restart(a): [wanted=Up] StopForUpdate(a) → Reconcile(a)
|
||||
//! graceful-restart(a): [wanted=Up] Signal(a) → Drain(a) → StopForUpdate(a) → Reconcile(a)
|
||||
//! start(a): [wanted=Up] Reconcile(a)
|
||||
//! stop(a): [wanted=Offline] Reconcile(a)
|
||||
//! spawn(a): [wanted=Up] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
|
||||
//! graceful-stop(a): SetWanted(a,Off) → Signal(a) → Drain(a) → Reconcile(a)
|
||||
//! restart(a): SetWanted(a,Up) → StopForUpdate(a) → Reconcile(a)
|
||||
//! graceful-restart(a): SetWanted(a,Up) → Signal(a) → Drain(a) → StopForUpdate(a) → Reconcile(a)
|
||||
//! start(a): SetWanted(a,Up) → Reconcile(a)
|
||||
//! stop(a): SetWanted(a,Off) → Reconcile(a)
|
||||
//! stale-start(a): SetWanted(a,Up) → «rebuild subgraph» (rev stale; prebuild noops, agent down)
|
||||
//! spawn(a): Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a) [wanted=Up at approve]
|
||||
//! perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
||||
//! meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected a»
|
||||
//! startup sweep: MetaLock(hyperhive, non-fatal) → «fan-out rebuild(stale a)»
|
||||
|
|
@ -117,10 +122,11 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
|
|||
|
||||
/// Graceful stop: cheap `Signal` fires immediately (no build slot), the
|
||||
/// `Drain` awaits the harness checkpoint (bounded), and the tail
|
||||
/// `Reconcile` performs the actual container stop — the caller sets
|
||||
/// `wanted = Offline` at submit time. A whole-hive graceful stop
|
||||
/// therefore signals every agent up front and overlaps every drain,
|
||||
/// replacing the old detached-watcher thread structurally.
|
||||
/// `Reconcile` performs the actual container stop. The head `SetWanted`
|
||||
/// node writes `wanted = Offline` as part of the DAG (was a pre-submit
|
||||
/// side effect). A whole-hive graceful stop therefore signals every agent
|
||||
/// up front and overlaps every drain, replacing the old detached-watcher
|
||||
/// thread structurally.
|
||||
pub fn graceful_stop(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::GracefulStop,
|
||||
|
|
@ -132,16 +138,17 @@ pub fn graceful_stop(agent: &str, source: Source, reason: String) -> DagSpec {
|
|||
perm_payload: None,
|
||||
transient: Some(TransientKind::Stopping),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::Signal, Vec::new()),
|
||||
node(agent, NodeKind::Drain, after_ok(0)),
|
||||
node(agent, NodeKind::Reconcile, after_ok(1)),
|
||||
node(agent, NodeKind::SetWanted { up: false }, Vec::new()),
|
||||
node(agent, NodeKind::Signal, after_ok(0)),
|
||||
node(agent, NodeKind::Drain, after_ok(1)),
|
||||
node(agent, NodeKind::Reconcile, after_ok(2)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Restart: mechanical stop, then converge to `wanted` — the submit
|
||||
/// layer writes `wanted = Up` first, so this is a stop + start like
|
||||
/// the old `lifecycle::restart` regardless of prior intent drift.
|
||||
/// Restart: write `wanted = Up` (head `SetWanted` node), mechanical stop,
|
||||
/// then converge — a stop + start like the old `lifecycle::restart`
|
||||
/// regardless of prior intent drift.
|
||||
pub fn restart(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::Restart,
|
||||
|
|
@ -153,20 +160,20 @@ pub fn restart(agent: &str, source: Source, reason: String) -> DagSpec {
|
|||
perm_payload: None,
|
||||
transient: Some(TransientKind::Restarting),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::StopForUpdate, Vec::new()),
|
||||
node(agent, NodeKind::Reconcile, after_ok(0)),
|
||||
node(agent, NodeKind::SetWanted { up: true }, Vec::new()),
|
||||
node(agent, NodeKind::StopForUpdate, after_ok(0)),
|
||||
node(agent, NodeKind::Reconcile, after_ok(1)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Graceful restart: signal → drain → mechanical stop → converge to
|
||||
/// `wanted` — the caller writes `wanted = Up` first, same as `restart`.
|
||||
/// One atomic DAG start to finish (no client- or server-side "submit
|
||||
/// one DAG, await it, submit the next" composition): the `Drain` node
|
||||
/// is the same bounded harness-checkpoint wait `graceful_stop` uses,
|
||||
/// then `StopForUpdate` (mechanical, ignores `wanted`) and the tail
|
||||
/// `Reconcile` (converges to `wanted = Up`, i.e. starts it back up)
|
||||
/// chain exactly like `restart`'s tail.
|
||||
/// Graceful restart: write `wanted = Up` (head `SetWanted`), signal →
|
||||
/// drain → mechanical stop → converge. One atomic DAG start to finish
|
||||
/// (no client- or server-side "submit one DAG, await it, submit the next"
|
||||
/// composition): the `Drain` node is the same bounded harness-checkpoint
|
||||
/// wait `graceful_stop` uses, then `StopForUpdate` (mechanical, ignores
|
||||
/// `wanted`) and the tail `Reconcile` (converges to `wanted = Up`, i.e.
|
||||
/// starts it back up) chain exactly like `restart`'s tail.
|
||||
pub fn graceful_restart(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::GracefulRestart,
|
||||
|
|
@ -178,16 +185,19 @@ pub fn graceful_restart(agent: &str, source: Source, reason: String) -> DagSpec
|
|||
perm_payload: None,
|
||||
transient: Some(TransientKind::Restarting),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::Signal, Vec::new()),
|
||||
node(agent, NodeKind::Drain, after_ok(0)),
|
||||
node(agent, NodeKind::StopForUpdate, after_ok(1)),
|
||||
node(agent, NodeKind::Reconcile, after_ok(2)),
|
||||
node(agent, NodeKind::SetWanted { up: true }, Vec::new()),
|
||||
node(agent, NodeKind::Signal, after_ok(0)),
|
||||
node(agent, NodeKind::Drain, after_ok(1)),
|
||||
node(agent, NodeKind::StopForUpdate, after_ok(2)),
|
||||
node(agent, NodeKind::Reconcile, after_ok(3)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Single-`Reconcile` DAG: `Start` / `Stop` (caller writes `wanted`
|
||||
/// first) and the boot-time `Reconcile` converge (wanted untouched).
|
||||
/// Boot-time reconcile: a single `Reconcile` node that converges observed
|
||||
/// power state to the persisted intent — `wanted` is untouched (no
|
||||
/// `SetWanted`), unlike the operator `start`/`stop` templates. Used only
|
||||
/// by the boot sweep now.
|
||||
pub fn reconcile_only(
|
||||
template: Template,
|
||||
agent: &str,
|
||||
|
|
@ -208,6 +218,68 @@ pub fn reconcile_only(
|
|||
}
|
||||
}
|
||||
|
||||
/// Start: write `wanted = Up` (head `SetWanted`), then reconcile (which
|
||||
/// starts the container). The intent write is a DAG node now, not a
|
||||
/// pre-submit side effect.
|
||||
pub fn start(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::Start,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Starting),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::SetWanted { up: true }, Vec::new()),
|
||||
node(agent, NodeKind::Reconcile, after_ok(0)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Stop: write `wanted = Offline` (head `SetWanted`), then reconcile
|
||||
/// (kill + unregister + `Killed` event).
|
||||
pub fn stop(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
DagSpec {
|
||||
template: Template::Stop,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Stopping),
|
||||
nodes: vec![
|
||||
node(agent, NodeKind::SetWanted { up: false }, Vec::new()),
|
||||
node(agent, NodeKind::Reconcile, after_ok(0)),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/// Stale start: a `start` whose rev marker is stale, so it rebuilds
|
||||
/// before coming up — `SetWanted(Up)` → «rebuild subgraph» → `Reconcile`
|
||||
/// (the rebuild's tail `Reconcile` starts it, since `wanted = Up`). The
|
||||
/// rebuild nodes are the same `rebuild_nodes` chain a manual rebuild uses
|
||||
/// (reused, not a variant); only the leading `SetWanted(Up)` intent
|
||||
/// differs. Shows as a `Rebuild` on the dashboard like the old
|
||||
/// submit-time upgrade did.
|
||||
pub fn stale_start(agent: &str, source: Source, reason: String) -> DagSpec {
|
||||
let mut nodes = vec![node(agent, NodeKind::SetWanted { up: true }, Vec::new())];
|
||||
nodes.extend(rebuild_nodes(agent, true, 1));
|
||||
DagSpec {
|
||||
template: Template::Rebuild,
|
||||
source,
|
||||
reason,
|
||||
parent_id: None,
|
||||
approval_id: None,
|
||||
inputs: Vec::new(),
|
||||
perm_payload: None,
|
||||
transient: Some(TransientKind::Rebuilding),
|
||||
nodes,
|
||||
}
|
||||
}
|
||||
|
||||
/// First-deploy spawn (approval-driven): `Provision` (proposed/applied
|
||||
/// repos, state subvolume, meta registration) then `Create`
|
||||
/// (`nixos-container create`), drop-in write, then `Reconcile` starts
|
||||
|
|
|
|||
Loading…
Reference in a new issue