refactor(#2439): build hive-wide stop/start/restart DAGs dynamically
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent subgraph each (concurrent on their own leases) instead of N DAGs — and each subgraph is now built dynamically from the agent's live running state rather than a fixed template shape: - online agent: the full stop→reconcile (restart: stop-for-update→reconcile) chain; `graceful` prepends signal→drain. - offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a restart of a down agent is really a start). The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee) are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate` nodes are state-conditional. Keeping `Reconcile` in every shape closes the TOCTOU window — a race-up between the `is_running` read and node exec is still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no reliance on an external reconcile sweep. The state-aware assembly needs an async `is_running` read, so it moves out of the pure/sync `templates.rs` into `submit.rs`, layered as pure `*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async `*_many` (reads live state + submits). `templates.rs` keeps only the shared pure primitives (`node`/`after_ok`/`rebuild_nodes`). Callers await the now-async submit fns (server, dashboard, socket_server). Tests exercise both the online and offline shapes via the pure `*_spec` seam. docs/coordinator.md shapes updated.
This commit is contained in:
parent
3797177e7f
commit
860484a193
8 changed files with 574 additions and 335 deletions
|
|
@ -76,19 +76,32 @@ container build:
|
|||
|
||||
The power ops write the durable `wanted` intent via a head `SetWanted`
|
||||
node (not a pre-submit side effect) — it holds the agent lease, so
|
||||
intent-write + reconcile is atomic per-agent. `restart` takes an agent
|
||||
*list*: a hive-wide `hivectl restart` / `restart-all` is ONE DAG with a
|
||||
per-agent restart subgraph each (independent roots, run concurrently on
|
||||
their own leases), not N separate DAGs.
|
||||
intent-write + reconcile is atomic per-agent. The hive-wide power ops —
|
||||
`restart`, `stop`, and `start` — take an agent *list*: a hive-wide `hivectl
|
||||
restart` / `stop` / `start` is ONE DAG with a per-agent subgraph each
|
||||
(independent roots, run concurrently on their own leases), not N separate
|
||||
DAGs.
|
||||
|
||||
**These are built dynamically from each agent's live running state** (an
|
||||
async `lifecycle::is_running` read), so they live in `job_queue/submit.rs`,
|
||||
not the pure/sync `templates.rs`. Per-agent shape rule: the head `SetWanted`
|
||||
(intent) and the tail `Reconcile` (convergence guarantee — cheap, noops when
|
||||
already converged) are ALWAYS present; only the *mechanical* nodes
|
||||
(`Signal`/`Drain`/`StopForUpdate`) are state-conditional — skipped for a
|
||||
*down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
closes the TOCTOU window: if an agent flips state between the `is_running`
|
||||
read and node exec, the tail `Reconcile` still converges it in-DAG (with
|
||||
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
|
||||
agent's subgraph is a rebuild-then-start).
|
||||
|
||||
```text
|
||||
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-any) Reconcile(a)
|
||||
graceful-stop(a): SetWanted(a,Off) → Signal(a) → Drain(a) → Reconcile(a)
|
||||
restart(a..): per agent: SetWanted(a,Up) → StopForUpdate(a) → Reconcile(a) (N subgraphs, 1 DAG)
|
||||
graceful-restart(a): SetWanted(a,Up) → Signal(a) → Drain(a) → StopForUpdate(a) → Reconcile(a)
|
||||
start(a): SetWanted(a,Up) → Reconcile(a) (stale rev ⇒ stale-start below)
|
||||
stale-start(a): SetWanted(a,Up) → «rebuild subgraph» (prebuild noops — agent is down)
|
||||
stop(a): SetWanted(a,Off) → Reconcile(a)
|
||||
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
|
||||
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
|
||||
restart(a..): online a: SetWanted(a,Up) → [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a)
|
||||
offline a: SetWanted(a,Up) → Reconcile(a) (nothing to stop — it's a start)
|
||||
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
||||
spawn(a): [wanted=Up at approve] Create(a) → WriteDropin(a) → Reconcile(a)
|
||||
perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
||||
meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent»
|
||||
|
|
|
|||
Loading…
Reference in a new issue