vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(#2916): destroy submits a DAG instead of an imperative teardown

Browse source 
Destroy was a straight-line async fn with no queue node behind it, so
nothing in the graph could answer "is this container going down on
purpose?". That gap is why an imperative crash-watch suppression guard
existed: an RAII handle held for the operation's duration, a second way
to say what every other lifecycle op already says through its node.

Reuse the existing Stop node rather than teaching a new node to stop
things:

    Stop -> DestroyContainer -> (PurgeState) -> DestroyBookkeeping

Stop already declares takes_container_down honestly, so the suppression
is now derived from the graph like every other op's. It also turns the
precondition into an edge: DestroyContainer runs only under a completed
Stop, so it operates on an already-stopped container and carries
takes_container_down = false permanently. A container still alive at
that point is a real bug and stays loud instead of being absorbed by a
flag -- which matters because a wrong true silently swallows a crash
while a wrong false only costs a spurious event.

Removes suppress_crash_watch, CrashWatchSuppression, crash_suppressed,
crash_watch_suppressed and NO_NODE_LABEL. The migration call sites went
with the obsolete startup migrations, so destroy was the last caller and
intent now has exactly one home.

destroy() becomes a submit-and-return, matching every sibling endpoint
(rebuild, kill, restart, start, pause, resume) -- it was the only
lifecycle op that awaited its work. The container rescan moves into the
bookkeeping tail, so ContainerRemoved now arrives after the 200 rather
than before it.

Also drops an orphaned doc-comment in coordinator.rs: two stacked blocks
where only the second described crash_suppressed, the first documenting
a field that no longer exists. Removing the field would have re-pointed
it at recent_transient.
This commit is contained in:
atlas 2026-08-13 23:26:17 +02:00 • committed by mara
commit 6338939657
9 changed files with 326 additions and 194 deletions
Download patch file Download diff file Expand all files Collapse all files

88
hive-c0re/src/coordinator.rs
View file

@ -116,13 +116,6 @@ pub struct Coordinator {
/// is never injected into containers.
pub model_prices: crate::hive_stats::PriceTable,
agents: Mutex<HashMap<String, AgentSocket>>,
/// Agents whose lifecycle action (currently just spawn) is in flight.
/// Read by the dashboard to render a spinner; cleared when the action
/// resolves (success or failure).
/// Agents whose container is being taken down by work with **no queue node
/// behind it** (destroy, migration), so the crash watcher must not report
/// the disappearance as a crash. Not a pill — see [`CrashWatchSuppression`].
crash_suppressed: Mutex<HashSet<String>>,
/// Tombstone for transients that have JUST been cleared. The
/// crash watcher polls every 10s and would race the
/// drop-clears-immediately path of `TransientGuard`: an operator
@ -400,57 +393,6 @@ fn fold_tombstones_by_agent<'a>(
out
}
/// Tombstone label for work with **no queue node behind it** — the
/// out-of-band operations (destroy, migration) that hold a
/// [`CrashWatchGuard`] instead of appearing in the derived transient set.
///
/// [`Coordinator::recent_transient`] is keyed by `(agent, label)` so concurrent
/// pills can't overwrite each other's `deliberate_stop`; a guard has no node and
/// therefore no node label, so it needs one of its own. Angle-bracketed to keep
/// it out of the `NodeKind::as_str` namespace — no node can ever render this.
const NO_NODE_LABEL: &str = "<no-node>";
/// RAII handle returned by [`Coordinator::suppress_crash_watch`]. While held,
/// the crash watcher treats this container disappearing as **expected**.
///
/// This is *not* a dashboard pill. Transients are derived from running queue
/// nodes and nothing stores them. But destroy and migration take a container
/// down without a node behind them, so nothing in the graph says the
/// disappearance was intended — and without that, `crash_watch` fires a
/// `ContainerCrash` for every destroy and every migrated agent, and the manager
/// tries to "recover" containers that were removed on purpose.
///
/// It is held rather than stamped once because
/// [`crate::workers::crash_watch`]'s grace window is finite and these
/// operations are not: a long destroy would outlive a single tombstone. The
/// tombstone is stamped on drop, covering the poll that lands just after.
///
/// Goes away entirely once destroy + migration are real queue nodes.
#[must_use = "suppression lasts as long as the guard; bind it for the operation's duration \
(`let _guard = coord.suppress_crash_watch(...)`). An unbound call drops it \
immediately and the very next poll can report a deliberate stop as a crash."]
pub struct CrashWatchSuppression {
coord: Arc<Coordinator>,
name: String,
}
impl Drop for CrashWatchSuppression {
fn drop(&mut self) {
self.coord
.crash_suppressed
.lock()
.unwrap()
.remove(&self.name);
// Tombstone the release so the next poll — which may land in the
// window between the container going away and this guard dropping —
// still reads the stop as deliberate.
self.coord.recent_transient.lock().unwrap().insert(
(self.name.clone(), NO_NODE_LABEL.to_owned()),
(true, std::time::Instant::now()),
);
}
}
/// RAII guard for the `meta-update` in-progress flag, held for the
/// duration of a `run_meta_update` background task. Created by
/// `Coordinator::meta_update_guard`. Drop decrements the active-run
@ -586,7 +528,6 @@ impl Coordinator {
agent_io_weight,
model_prices,
agents: Mutex::new(HashMap::new()),
crash_suppressed: Mutex::new(HashSet::new()),
recent_transient: Mutex::new(HashMap::new()),
recent_crashes: Mutex::new(HashMap::new()),
graceful_stop_pending: Mutex::new(HashSet::new()),
@ -1300,35 +1241,6 @@ impl Coordinator {
map.iter().map(|(k, v)| (k.clone(), v.len())).collect()
}
/// Tell the crash watcher that `name`'s container is going down **on
/// purpose**, for the lifetime of the returned guard. See
/// [`CrashWatchSuppression`] for why this exists at all.
///
/// Only for the operations with no queue node behind them. Anything the
/// job queue runs answers this from the node itself
/// ([`crate::job_queue::NodeKind::takes_container_down`]) and must not come
/// through here.
///
/// The guard's `Drop` runs even on task cancellation, so an aborted HTTP
/// request or a panic mid-destroy can't leave a container permanently
/// exempt from crash reporting.
pub fn suppress_crash_watch(self: &Arc<Self>, name: &str) -> CrashWatchSuppression {
self.crash_suppressed
.lock()
.unwrap()
.insert(name.to_owned());
CrashWatchSuppression {
coord: self.clone(),
name: name.to_owned(),
}
}
/// Whether a no-node operation is currently taking this container down.
#[must_use]
pub fn crash_watch_suppressed(&self, name: &str) -> bool {
self.crash_suppressed.lock().unwrap().contains(name)
}
/// Every live transient, keyed by agent.
///
/// **Derived on read, stored nowhere.** Straight off the running graph, so