vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert

Browse source
This commit is contained in:
damocles 2026-10-12 22:21:04 +02:00 • committed by mara
commit 02831fb5e9
13 changed files with 214 additions and 224 deletions
Download patch file Download diff file Expand all files Collapse all files

23
hive-c0re/src/workers/crash_watch.rs
View file

@ -1,7 +1,10 @@
//! Per-container crash and login-state watcher. Polls every managed
//! container on a 10s interval. Fires `ContainerCrash`, `LoggedIn`,
//! and `NeedsLogin` helper events. Event semantics and the
//! `RECENT_TRANSIENT_GRACE` window: `docs/agent-lifecycle/config-changes.md::Helper events`.
//! container on a 10s interval. A classified crash only `tracing::warn!`s
//! (`"container crash detected"`) — a swarm-level grafana alert matches
//! that line directly; see the module doc on [`emit_crash_transitions`].
//! Login-state transitions (`needs_login`/`logged_in`) publish onto
//! `swarm_notices` instead. The `RECENT_TRANSIENT_GRACE` window:
//! `docs/agent-lifecycle/config-changes.md::Helper events`.
use std::collections::HashSet;
use std::sync::Arc;
@ -143,12 +146,12 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
if is_deliberate_stop(active, recently_cleared) {
continue;
}
// The one place this fires: a swarm-level grafana alert
// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`)
// matches this exact message against the log store and pages the
// operator directly — no agent (and no per-hive dashboard banner)
// in the loop. See `docs/scheduler/observability.md`.
tracing::warn!(agent = %stopped, "container crash detected");
coord.record_crash(stopped);
coord.notify_manager(&hive_sh4re::manager::HelperEvent::ContainerCrash {
agent: stopped.clone(),
note: Some("container stopped without an operator action".into()),
});
}
}
@ -230,7 +233,7 @@ mod tests {
/// Control: a real drop from running to absent is a *readable*, empty
/// list — it must still be processed (and so still reach
/// `emit_crash_transitions`, which fires the crash event), unlike a
/// `emit_crash_transitions`, which warns the crash), unlike a
/// read failure.
#[test]
fn readable_empty_list_still_processes() {
@ -261,7 +264,7 @@ mod tests {
#[test]
fn not_deliberate_with_no_transient_at_all() {
// The real-crash case — fires the ContainerCrash event.
// The real-crash case — warns "container crash detected".
assert!(!is_deliberate_stop(None, None));
}