vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

restart preserves wanted intent instead of forcing all agents up (#2540)

Browse source
This commit is contained in:
damocles 2026-07-16 20:05:27 +02:00 • committed by mara
commit 5bb5a88aa0
4 changed files with 77 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

31
docs/coordinator.md
View file

@ -75,21 +75,24 @@ container build:
### Every operation as a DAG
The power ops write the durable `wanted` intent via a head `SetWanted`
node (not a pre-submit side effect) — it holds the agent lease, so
intent-write + reconcile is atomic per-agent. The hive-wide power ops —
`restart`, `stop`, and `start` — take an agent *list*: a hive-wide `hivectl
restart` / `stop` / `start` is ONE DAG with a per-agent subgraph each
(independent roots, run concurrently on their own leases), not N separate
DAGs.
The `stop` / `start` power ops write the durable `wanted` intent via a head
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
it writes *no* intent (no `SetWanted` head) — it bounces the container and
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
deliberately-stopped agent is not forced back up by a hive-wide restart. The
hive-wide power ops — `restart`, `stop`, and `start` — take an agent *list*:
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs.
**These are built dynamically from each agent's live running state** (an
async `lifecycle::is_running` read), so they live in `job_queue/submit.rs`,
not the pure/sync `templates.rs`. Per-agent shape rule: the head `SetWanted`
(intent) and the tail `Reconcile` (convergence guarantee — cheap, noops when
already converged) are ALWAYS present; only the *mechanical* nodes
(`Signal`/`Drain`/`StopForUpdate`) are state-conditional — skipped for a
*down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the *mechanical* nodes (`Signal`/`Drain`/`StopForUpdate`) are
state-conditional — skipped for a *down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
closes the TOCTOU window: if an agent flips state between the `is_running`
read and node exec, the tail `Reconcile` still converges it in-DAG (with
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
@ -100,8 +103,8 @@ agent's subgraph is a rebuild-then-start).
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-ok) PostSwap(a) →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: SetWanted(a,Up) → [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a)
offline a: SetWanted(a,Up) → Reconcile(a) (nothing to stop — it's a start)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at approve] Create(a) → WriteDropin(a) → Reconcile(a)
perm-change(a): WritePermFile(a) → «rebuild subgraph»