vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: delete the cancelled-power-op intent revert

Browse source 
The revert hook is dead by construction, so it can only ever be wrong.

DAG state `Cancelled` has exactly one producer: `JobQueue::cancel`, which
refuses unless every work node is still `Pending`. A cancel *cascade*
(some node failed, downstream cancelled) rolls up `Failed` instead —
`dag_rollup` short-circuits on any failed subtree node. So on a DAG that
reaches `Cancelled`, no node ever executed: the `SetWanted` head provably
never ran and `wanted` still reads whatever the operator last set it to.

There is therefore nothing to revert, and `revert_intent` did not revert
anything — it wrote `Wanted::from_running(observed)`, i.e. the agent's
*observed* state, over an intent the DAG never touched. Harmless when
observed already matched, silent corruption otherwise: cancel a queued
start for an agent that is down but `wanted = Up` (crashed, or caught
mid-bounce) and the intent flips to `Offline`, leaving it
deliberately-stopped as far as reconcile and crash-watch are concerned.

The hook made sense when `set_wanted` was a pre-submit side effect
written before the DAG ran; moving it into the DAG as a node left the
hook vestigial.

Drop `HookKind::RevertIntent`, `revert_intent`, and the power-op arm of
`terminal_hook` — start / stop / graceful-stop now settle with no
terminal hook, same as restart always did. The test asserts the general
statement across restart/stop/start x graceful x running: stop and start
carry a `SetWanted` head, and cancelling them still fires no hook.
This commit is contained in:
atlas 2026-07-26 16:19:26 +02:00 • committed by mara
commit 5c4a637941
4 changed files with 64 additions and 79 deletions
Download patch file Download diff file Expand all files Collapse all files

89
hive-c0re/src/job_queue/tests.rs
View file

@ -863,49 +863,64 @@ fn cancel_refuses_running_dag() {
assert_eq!(state_of(&q, id), State::Running);
}
/// A cancelled restart must fire **no** intent revert. `restart_chain` writes
/// no `SetWanted`, so a cancel has nothing to roll back — and the revert hook
/// writes the agent's *observed* state, which for a down-but-`wanted = Up`
/// agent (crashed, or caught mid-bounce) would flip the intent to `Offline`
/// and leave it deliberately-stopped as far as reconcile and crash-watch are
/// concerned. Invisible for a running agent (observed == recorded), which is
/// why it went unnoticed; the window is exactly "observed ≠ intent", which is
/// when a restart is most likely to be issued and then cancelled.
/// A cancelled power op must fire **no** compensating hook — not even one that
/// carries a `SetWanted` head.
///
/// `cancel` refuses unless every work node is still `Pending`
/// (`cancel_refuses_running_dag`) and a cancel *cascade* rolls up `Failed`
/// rather than `Cancelled`, so a `Cancelled` DAG provably never executed a
/// node: its `SetWanted` never ran and the agent's intent still reads whatever
/// the operator last set. A "revert" instead writes the agent's *observed*
/// state, which for a down-but-`wanted = Up` agent (crashed, or caught
/// mid-bounce) flips the intent to `Offline` and leaves it
/// deliberately-stopped as far as reconcile and crash-watch are concerned.
#[test]
fn cancelled_restart_reverts_no_intent() {
fn cancelled_power_op_fires_no_hook() {
for graceful in [false, true] {
for running in [false, true] {
let q = JobQueue::new(1);
let targets = vec![("agent-a".to_owned(), running)];
let spec =
submit::restart_spec(&targets, graceful, Source::Manual, "bounce".to_owned());
assert!(
!spec
.nodes
.iter()
.any(|n| matches!(n.kind, NodeKind::SetWanted { .. })),
"restart writes no intent (graceful={graceful}, running={running})"
);
let id = submit(&q, spec);
let summary = q.cancel(id).expect("cancelled while queued");
assert_eq!(summary.state, State::Cancelled);
assert_eq!(
terminal_hook(summary.template, summary.approval_id),
None,
"cancelled restart (graceful={graceful}, running={running}) must \
not revert an intent it never wrote"
);
let cases = [
(
"restart",
false,
submit::restart_spec(&targets, graceful, Source::Manual, "bounce".to_owned()),
),
(
"stop",
true,
submit::stop_spec(&targets, graceful, Source::Manual, "stop".to_owned()),
),
(
"start",
true,
submit::start_spec(
&[("agent-a".to_owned(), running, false)],
Source::Manual,
"start".to_owned(),
),
),
];
for (name, writes_intent, spec) in cases {
assert_eq!(
spec.nodes
.iter()
.any(|n| matches!(n.kind, NodeKind::SetWanted { .. })),
writes_intent,
"{name} intent head (graceful={graceful}, running={running})"
);
let q = JobQueue::new(1);
let id = submit(&q, spec);
let summary = q.cancel(id).expect("cancelled while queued");
assert_eq!(summary.state, State::Cancelled);
assert_eq!(
terminal_hook(summary.template, summary.approval_id),
None,
"cancelled {name} (graceful={graceful}, running={running}) must \
fire no hook — no node of it ever ran"
);
}
}
}
// Contrast: stop *does* carry a `SetWanted` head, so its cancel still has a
// real intent flip to undo.
let q = JobQueue::new(1);
let id = submit(&q, stop_online(&["agent-a"], false, "stop"));
let summary = q.cancel(id).expect("cancelled while queued");
assert_eq!(
terminal_hook(summary.template, summary.approval_id),
Some(HookKind::RevertIntent)
);
}
// ---- terminal reporting + lease release ----