refactor(#2439): build hive-wide stop/start/restart DAGs dynamically

Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent
subgraph each (concurrent on their own leases) instead of N DAGs — and each
subgraph is now built dynamically from the agent's live running state rather
than a fixed template shape:

- online agent: the full stop→reconcile (restart: stop-for-update→reconcile)
  chain; `graceful` prepends signal→drain.
- offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a
  restart of a down agent is really a start).

The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee)
are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate`
nodes are state-conditional. Keeping `Reconcile` in every shape closes the
TOCTOU window — a race-up between the `is_running` read and node exec is
still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no
reliance on an external reconcile sweep.

The state-aware assembly needs an async `is_running` read, so it moves out
of the pure/sync `templates.rs` into `submit.rs`, layered as pure
`*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async
`*_many` (reads live state + submits). `templates.rs` keeps only the shared
pure primitives (`node`/`after_ok`/`rebuild_nodes`).

Callers await the now-async submit fns (server, dashboard, socket_server).
Tests exercise both the online and offline shapes via the pure `*_spec`
seam. docs/coordinator.md shapes updated.
This commit is contained in:
atlas 2026-07-14 23:34:56 +02:00
commit 860484a193
8 changed files with 574 additions and 335 deletions

View file

@ -76,19 +76,32 @@ container build:
The power ops write the durable `wanted` intent via a head `SetWanted`
node (not a pre-submit side effect) — it holds the agent lease, so
intent-write + reconcile is atomic per-agent. `restart` takes an agent
*list*: a hive-wide `hivectl restart` / `restart-all` is ONE DAG with a
per-agent restart subgraph each (independent roots, run concurrently on
their own leases), not N separate DAGs.
intent-write + reconcile is atomic per-agent. The hive-wide power ops —
`restart`, `stop`, and `start` — take an agent *list*: a hive-wide `hivectl
restart` / `stop` / `start` is ONE DAG with a per-agent subgraph each
(independent roots, run concurrently on their own leases), not N separate
DAGs.
**These are built dynamically from each agent's live running state** (an
async `lifecycle::is_running` read), so they live in `job_queue/submit.rs`,
not the pure/sync `templates.rs`. Per-agent shape rule: the head `SetWanted`
(intent) and the tail `Reconcile` (convergence guarantee — cheap, noops when
already converged) are ALWAYS present; only the *mechanical* nodes
(`Signal`/`Drain`/`StopForUpdate`) are state-conditional — skipped for a
*down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
closes the TOCTOU window: if an agent flips state between the `is_running`
read and node exec, the tail `Reconcile` still converges it in-DAG (with
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
agent's subgraph is a rebuild-then-start).
```text
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-any) Reconcile(a)
graceful-stop(a): SetWanted(a,Off) → Signal(a) → Drain(a) → Reconcile(a)
restart(a..): per agent: SetWanted(a,Up) → StopForUpdate(a) → Reconcile(a) (N subgraphs, 1 DAG)
graceful-restart(a): SetWanted(a,Up) → Signal(a) → Drain(a) → StopForUpdate(a) → Reconcile(a)
start(a): SetWanted(a,Up) → Reconcile(a) (stale rev ⇒ stale-start below)
stale-start(a): SetWanted(a,Up) → «rebuild subgraph» (prebuild noops — agent is down)
stop(a): SetWanted(a,Off) → Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: SetWanted(a,Up) → [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a)
offline a: SetWanted(a,Up) → Reconcile(a) (nothing to stop — it's a start)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at approve] Create(a) → WriteDropin(a) → Reconcile(a)
perm-change(a): WritePermFile(a) → «rebuild subgraph»
meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent»