vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: stop reverting power intent on a cancelled restart

Browse source 
`terminal_hook` mapped `Restart` / `GracefulRestart` to `RevertIntent`, but
a restart never writes `wanted` — `restart_chain` deliberately has no
`SetWanted` head, so the tail `Reconcile` converges to the agent's existing
intent and a deliberately-stopped agent isn't forced up by a hive-wide
restart.

`revert_intent` writes `Wanted::from_running(observed)` unconditionally on a
cancelled DAG. So for an agent that is `wanted = Up` but currently down
(crashed, or caught behind another queued op), submitting a restart and then
cancelling it writes `wanted = Offline` — reverting an intent the DAG never
touched, to a value nobody asked for. Reconcile and crash-watch both then
read the agent as deliberately-stopped and leave it down.

It's invisible for a running agent, since `from_running(true)` equals the
intent already on file, which is why it went unnoticed. `cancel` only
succeeds while every node is still `Pending`, so the reachable window is
exactly "queued restart + observed != intent" — precisely when someone
restarts and then thinks better of it.

Drop both restart templates from the `RevertIntent` arm; they fall through
to no terminal hook, which is correct for a DAG that writes no intent.
Document the invariant on `HookKind::RevertIntent` and on `revert_intent`
itself: the hook writes *observed* state, so dispatching it for a template
with no `SetWanted` head doesn't restore an intent, it invents one.

Test covers all four restart shapes (graceful x running), asserting both
that the spec carries no `SetWanted` and that a cancelled restart dispatches
no hook, with a contrast arm pinning stop's revert in place.

Fixes hyperhive/hyperhive#2710
This commit is contained in:
atlas 2026-07-26 16:05:13 +02:00 • committed by mara
commit 7589f4c06c
3 changed files with 62 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

5
hive-c0re/src/job_queue/exec.rs
View file

@ -145,6 +145,11 @@ fn emit_rebuilt(coord: &Arc<Coordinator>, terminal: &super::TerminalDag) {
/// intent to its observed state — the operator's cancel means "don't do it", so
/// the intent snaps back instead of the flip executing as a surprise side effect
/// of some later reconcile. Noop on any non-cancelled outcome.
///
/// Only valid for templates carrying a `SetWanted` head (start / stop): the
/// revert writes *observed* state, so dispatching it for a template that never
/// wrote an intent doesn't restore anything — it invents one. See
/// [`super::terminal_hook`].
async fn revert_intent(coord: &Arc<Coordinator>, terminal: &super::TerminalDag) {
if terminal.state != State::Cancelled {
return;