vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
4 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

146
hive-c0re/src/stats/host_stats.rs
View file

@ -1,35 +1,33 @@
//! Host-system probes and the derived **server-warning** list that the
//! dashboard renders as a top-of-page banner.
//! Host-system probes that raise hive-level warnings into the push-based
//! [`crate::warnings`] registry.
//!
//! The first (and currently only) producer is a host disk-usage check:
//! the nix store filling up is what ENOSPC-failed CI before the GC
//! guardrails were documented. hive-c0re runs on the host (not inside a
//! container), so it can `statvfs` the store path directly and warn the
//! operator *before* an ENOSPC, not after.
//! The first producer is a host disk-usage check: the nix store filling up
//! is what ENOSPC-failed CI before the GC guardrails were documented.
//! hive-c0re runs on the host (not inside a container), so it can `statvfs`
//! the store path directly and warn the operator *before* an ENOSPC, not
//! after. [`refresh_disk_warning`] drives a held guard from a periodic task
//! (see `spawn` in `main.rs`). The same probe backs the
//! `hyperhive.hive.nix_store.utilization` gauge ([`disk_utilization`]), which is
//! what the early-warning Grafana alerts read.
//!
//! [`server_warnings`] is the public surface: it returns a flat list of
//! [`ServerWarning`]s for `/api/state`. It is now just a snapshot of the
//! push-based [`crate::warnings`] registry — producers raise/clear their
//! own warnings via an RAII guard rather than being re-probed on every
//! render. The disk-pressure check is reworked onto that model here:
//! [`refresh_disk_warning`] drives a held guard from a periodic task (see
//! `spawn` in `main.rs`) instead of a `statvfs` per `/api/state`.
//! The registry has no dashboard banner any more. A raised warning reaches
//! the operator as the `hyperhive.hive.warning` gauge and a Grafana alert
//! on it (see `docs/scheduler/observability.md`).
use serde::Serialize;
use utoipa::ToSchema;
use crate::container_view::ContainerView;
/// One server-level warning for the dashboard's top-of-page banner.
/// One hive-level warning in the [`crate::warnings`] registry.
#[derive(Debug, Clone, Serialize, ToSchema)]
pub struct ServerWarning {
/// Stable kind id (e.g. `"disk_pressure"`) — lets the frontend dedupe
/// or special-case without parsing the message.
/// Stable kind id (e.g. `"disk_pressure"`), the `kind` label of the
/// `hyperhive.hive.warning` gauge.
pub kind: &'static str,
/// Severity: `"warn"` (amber) or `"crit"` (red). The banner picks its
/// colour from this; everything else is just the message text.
/// Severity: `"warn"` or `"crit"`. `crit` degrades readiness.
pub level: &'static str,
/// Human-readable, already-formatted message shown in the banner.
/// Human-readable, already-formatted message.
pub message: String,
}
@ -38,17 +36,7 @@ pub struct ServerWarning {
const DISK_WARN_PCT: f64 = 85.0;
const DISK_CRIT_PCT: f64 = 95.0;
/// Snapshot the current server-level warnings for the dashboard banner —
/// a cheap read of the push-based [`crate::warnings`] registry. Producers
/// (the disk watch below, background converge sweeps, …) raise and clear
/// their own entries via [`crate::warnings::set_warning`]; nothing is
/// re-probed here.
#[must_use]
pub fn server_warnings() -> Vec<ServerWarning> {
crate::warnings::snapshot()
}
/// Pure disk-pressure decision: given usage, return the banner
/// Pure disk-pressure decision: given usage, return the warning
/// `(level, message)` or `None` when below the warn threshold. Split from
/// the probe + guard plumbing so the threshold + formatting is unit-tested
/// without a real `statvfs`.
@ -75,9 +63,8 @@ fn disk_pressure_warning(used_pct: f64, free_bytes: u64) -> Option<(&'static str
/// One tick of the disk-pressure watch: probe the store and reconcile the
/// held warning `guard` — raise or refresh it while usage is over
/// threshold, drop it (clearing the banner) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`, replacing the old
/// per-render `statvfs` in [`server_warnings`].
/// threshold, drop it (clearing the warning) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`.
pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
let decision = nix_disk_usage().and_then(|d| disk_pressure_warning(d.used_pct, d.free_bytes));
match decision {
@ -96,19 +83,44 @@ pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
}
}
/// Agent-state warnings derived from the live container snapshot the
/// dashboard already holds: agents that need a claude login. Kept
/// separate from [`server_warnings`] (host probes) because the caller
/// owns the container list; the dashboard concatenates both into one
/// banner list.
/// Fraction (0–1) of the filesystem holding the nix store that is in use,
/// the value behind the `hyperhive.hive.nix_store.utilization` gauge. `None` when
/// the probe fails, so a failing probe reports no sample rather than zero.
#[must_use]
pub fn disk_utilization() -> Option<f64> {
nix_disk_usage().map(|d| d.used_pct / 100.0)
}
/// The held `pending_logins` warning; see [`refresh_pending_logins`].
static PENDING_LOGINS: std::sync::Mutex<Option<crate::warnings::WarningGuard>> =
std::sync::Mutex::new(None);
/// Reconcile the `pending_logins` warning with the current container
/// snapshot: raise or refresh it while any agent needs a claude login,
/// drop it once none does. Driven by the container rescan in the
/// coordinator, which already builds the snapshot every poll.
pub fn refresh_pending_logins(containers: &[ContainerView]) {
let warning = agent_state_warnings(containers).into_iter().next();
let mut guard = PENDING_LOGINS
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
match (warning, guard.as_ref()) {
(Some(w), Some(g)) => g.update(w.level, w.message),
(Some(w), None) => {
*guard = Some(crate::warnings::set_warning(w.kind, w.level, w.message));
}
(None, _) => *guard = None,
}
}
/// Agent-state warnings derived from the live container snapshot: agents
/// that need a claude login. Takes the container list from its caller
/// because that is where the snapshot lives.
///
/// Container crashes used to have a sibling `agents_crashing` entry here,
/// fed by `Coordinator::recent_crash_counts` — removed in favour of a
/// swarm-level grafana alert on the crash-watcher's own log line (see
/// `nix/host-modules/swarm-grafana.nix`'s `hh-container-crashed` rule and
/// `docs/scheduler/observability.md`). That ships the same signal to the
/// operator without needing an agent (or this per-hive banner) in the
/// loop at all.
/// Container crashes are not a warning here: the crash-watcher logs
/// `container crash detected` and a swarm-level grafana alert
/// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`) matches
/// that line, with no agent in the loop.
#[must_use]
pub fn agent_state_warnings(containers: &[ContainerView]) -> Vec<ServerWarning> {
let mut out = Vec::new();
@ -263,4 +275,46 @@ mod tests {
let w = agent_state_warnings(&containers);
assert!(w[0].message.starts_with("1 agent needs claude login: solo"));
}
/// The registry entry tracks the snapshot it is refreshed from: raised
/// when an agent needs a login, rewritten when the set changes, and gone
/// when none does. This is the only test that touches `pending_logins`,
/// so reading the process-global registry by that kind stays sound under
/// the parallel test runner.
#[test]
fn pending_logins_warning_follows_the_container_snapshot() {
let current = || {
crate::warnings::snapshot()
.into_iter()
.find(|w| w.kind == "pending_logins")
};
assert!(
current().is_none(),
"nothing raised before the first refresh"
);
refresh_pending_logins(&[cv("amy", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("1 agent needs claude login: amy")),
"an agent needing a login raises the warning"
);
refresh_pending_logins(&[cv("amy", true), cv("bob", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("2 agents need claude login")),
"a second agent rewrites the same entry"
);
refresh_pending_logins(&[cv("amy", false), cv("bob", false)]);
assert!(current().is_none(), "no agent needing a login clears it");
}
/// The gauge value is a fraction, so a percent leaking through would put
/// every alert threshold 100x off.
#[test]
fn disk_utilization_is_a_fraction() {
if let Some(used) = disk_utilization() {
assert!((0.0..=1.0).contains(&used), "got {used}");
}
}
}

35
hive-c0re/src/stats/otel_metrics.rs
View file

@ -305,6 +305,41 @@ fn register_hive_instruments(meter: &opentelemetry::metrics::Meter) {
}
})
.build();
// One series per raised warning, `1` while it is raised, labelled with
// its `kind` and `level`. This is what names the cause in an alert: the
// count above says how many, this says which. A cleared warning stops
// reporting rather than reporting `0`, so an alert on it resolves when
// the series goes stale; the kinds are a fixed set in the source, so the
// label has bounded cardinality.
meter
.u64_observable_gauge("hyperhive.hive.warning")
.with_unit("1")
.with_callback(|obs| {
for w in warnings::snapshot() {
obs.observe(
1,
&[
KeyValue::new("kind", w.kind),
KeyValue::new("level", w.level),
],
);
}
})
.build();
// Fraction of the nix store's filesystem in use, by the semconv name.
// Probed in the callback (a `statvfs` is cheap and this runs once an
// export interval); a failed probe reports no sample.
meter
.f64_observable_gauge("hyperhive.hive.nix_store.utilization")
.with_unit("1")
.with_callback(|obs| {
if let Some(used) = crate::host_stats::disk_utilization() {
obs.observe(used, HIVE_ATTRS);
}
})
.build();
}
/// Attributes for the hive-scoped family: **none**.

50
hive-c0re/src/stats/sweep_health.rs
View file

@ -5,14 +5,14 @@
//! gateway nginx writes, per-agent branch protection, …) historically
//! `warn!`'d to the journal and moved on, so a persistent failure was
//! invisible to the operator. [`SweepHealth`] routes that failure to the
//! dashboard banner instead — but *debounced*, per the routing guidance in
//! hive warning instead — but *debounced*, per the routing guidance in
//! the tracker issue:
//!
//! - a transient miss (container down mid-sweep, a network blip) shouldn't
//! flap a banner on the first failure → `threshold > 1` raises only after
//! flap a warning on the first failure → `threshold > 1` raises only after
//! that many **consecutive** misses;
//! - a won't-fix-itself failure (missing team, bad config, auth rejection)
//! should banner immediately → `threshold == 1`.
//! should warning immediately → `threshold == 1`.
//!
//! The warning clears the moment the sweep next succeeds. A sweep owns one
//! `SweepHealth`, calls [`record_ok`](SweepHealth::record_ok) on success and
@ -24,11 +24,11 @@ use std::time::{Duration, Instant};
use crate::warnings::{WarningGuard, set_warning};
/// Per-sweep failure context handed to the [`record_err`](SweepHealth::record_err)
/// message builder so the banner text can include how long the sweep has been
/// message builder so the warning text can include how long the sweep has been
/// failing and since when it last worked.
#[derive(Debug, Clone, Copy)]
pub struct SweepFailure {
/// Number of consecutive failures so far (≥ `threshold` when the banner
/// Number of consecutive failures so far (≥ `threshold` when the warning
/// is raised).
pub consecutive: u32,
/// Time since the last recorded success, if the sweep has ever succeeded
@ -36,7 +36,7 @@ pub struct SweepFailure {
pub since_last_ok: Option<Duration>,
}
/// A debounced link between one background sweep and the warning banner.
/// A debounced link between one background sweep and the warning warning.
/// Not `Sync`-shared: each sweep task owns its own instance (the underlying
/// registry is the shared, thread-safe part).
pub struct SweepHealth {
@ -50,8 +50,8 @@ pub struct SweepHealth {
impl SweepHealth {
/// Track sweep `kind` at severity `level` (`"warn"` / `"crit"`), raising
/// the banner after `threshold` consecutive failures. `threshold` is
/// floored at 1 (0 would banner before any failure).
/// the warning after `threshold` consecutive failures. `threshold` is
/// floored at 1 (0 would warning before any failure).
#[must_use]
pub fn new(kind: &'static str, level: &'static str, threshold: u32) -> Self {
Self {
@ -65,7 +65,7 @@ impl SweepHealth {
}
/// Record a successful sweep: reset the consecutive-miss counter and drop
/// the guard (clearing any banner entry).
/// the guard (clearing any warning entry).
pub fn record_ok(&mut self) {
self.misses = 0;
self.last_ok = Some(Instant::now());
@ -74,9 +74,9 @@ impl SweepHealth {
}
/// Record a failed sweep. Bumps the consecutive-miss counter; once it
/// reaches `threshold`, raises (or refreshes) the banner warning built by
/// reaches `threshold`, raises (or refreshes) the warning warning built by
/// `message`. Below the threshold this is a cheap no-op — `message` is
/// only invoked when the banner is actually shown, so callers can format
/// only invoked when the warning is actually shown, so callers can format
/// lazily.
pub fn record_err(&mut self, message: impl FnOnce(SweepFailure) -> String) {
self.misses = self.misses.saturating_add(1);
@ -96,7 +96,7 @@ impl SweepHealth {
}
/// Compact human-readable age (e.g. `"45s"`, `"12m"`, `"3h"`, `"2d"`) for a
/// banner "last succeeded N ago" suffix. Coarse on purpose — the banner wants
/// warning "last succeeded N ago" suffix. Coarse on purpose — the warning wants
/// a glanceable magnitude, not precision.
#[must_use]
pub fn fmt_age(d: Duration) -> String {
@ -117,7 +117,7 @@ mod tests {
use super::*;
use crate::warnings::snapshot;
fn banner(kind: &str) -> Option<String> {
fn warning(kind: &str) -> Option<String> {
snapshot()
.into_iter()
.find(|w| w.kind == kind)
@ -125,34 +125,34 @@ mod tests {
}
#[test]
fn below_threshold_does_not_banner() {
fn below_threshold_does_not_warn() {
let mut h = SweepHealth::new("sh_below", "warn", 3);
h.record_err(|_| "boom".to_owned());
h.record_err(|_| "boom".to_owned());
assert!(h.guard.is_none());
assert!(banner("sh_below").is_none());
assert!(warning("sh_below").is_none());
}
#[test]
fn raises_at_threshold_and_clears_on_ok() {
let mut h = SweepHealth::new("sh_raise", "warn", 2);
h.record_err(|c| format!("fail #{}", c.consecutive));
assert!(banner("sh_raise").is_none(), "one miss < threshold");
assert!(warning("sh_raise").is_none(), "one miss < threshold");
h.record_err(|c| format!("fail #{}", c.consecutive));
assert_eq!(banner("sh_raise").as_deref(), Some("fail #2"));
assert_eq!(warning("sh_raise").as_deref(), Some("fail #2"));
assert!(h.guard.is_some());
h.record_ok();
assert!(h.guard.is_none());
assert!(banner("sh_raise").is_none(), "success clears the banner");
assert!(warning("sh_raise").is_none(), "success clears the warning");
}
#[test]
fn threshold_one_banners_immediately() {
fn threshold_one_warns_immediately() {
let mut h = SweepHealth::new("sh_immediate", "crit", 1);
h.record_err(|_| "gone".to_owned());
assert_eq!(banner("sh_immediate").as_deref(), Some("gone"));
assert_eq!(warning("sh_immediate").as_deref(), Some("gone"));
h.record_ok();
assert!(banner("sh_immediate").is_none());
assert!(warning("sh_immediate").is_none());
}
#[test]
@ -161,9 +161,9 @@ mod tests {
h.record_err(|_| "x".to_owned()); // miss 1
h.record_ok(); // reset
h.record_err(|c| format!("c={}", c.consecutive)); // miss 1 again, < threshold
assert!(banner("sh_reset").is_none(), "counter reset after ok");
assert!(warning("sh_reset").is_none(), "counter reset after ok");
h.record_err(|c| format!("c={}", c.consecutive)); // miss 2 → raise
assert_eq!(banner("sh_reset").as_deref(), Some("c=2"));
assert_eq!(warning("sh_reset").as_deref(), Some("c=2"));
h.record_ok();
}
@ -171,9 +171,9 @@ mod tests {
fn refresh_updates_message_while_held() {
let mut h = SweepHealth::new("sh_refresh", "warn", 1);
h.record_err(|c| format!("miss {}", c.consecutive));
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 1"));
assert_eq!(warning("sh_refresh").as_deref(), Some("miss 1"));
h.record_err(|c| format!("miss {}", c.consecutive));
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 2"));
assert_eq!(warning("sh_refresh").as_deref(), Some("miss 2"));
h.record_ok();
}

47
hive-c0re/src/stats/warnings.rs
View file

@ -1,21 +1,21 @@
//! Process-global server-warning registry backing the dashboard banner.
//! Process-global hive-warning registry.
//!
//! The dashboard shows a top-of-page banner for hive-level degradation
//! (disk pressure, a failed converge sweep, agents that can't log in, …).
//! Historically each warning was *pulled*: [`host_stats::server_warnings`]
//! recomputed the whole list on every `/api/state` by re-probing (a
//! `statvfs`, a scan of the container snapshot). Background sweeps that
//! failed had nowhere to record it, so they just `warn!`'d to the journal
//! and vanished from anywhere an operator looks.
//! Subsystems record hive-level degradation here (disk pressure, a failed
//! converge sweep, agents that can't log in, …). Three consumers read it:
//! `/health/ready` and the swarm status publisher through [`readiness`], and
//! the `hyperhive.hive.degraded` / `hyperhive.hive.warnings` /
//! `hyperhive.hive.warning` gauges in `otel_metrics`, which the Grafana
//! alerts watch. There is no dashboard banner.
//!
//! This module replaces the pull model with a **push** one: a single
//! `Mutex<BTreeMap<kind, …>>` any subsystem writes into. [`snapshot`] is a
//! cheap read of that map. The default API is RAII: [`set_warning`] returns
//! a [`WarningGuard`] whose `Drop` removes the key, so a producer holds the
//! guard exactly as long as the condition is true and lets it drop to clear
//! — no manual dismiss, no stale banner. A failing periodic sweep does
//! `let _w = set_warning(...)` in its failure branch and drops it on the
//! next success.
//! The registry is **push**-based: a single `Mutex<BTreeMap<kind, …>>` any
//! subsystem writes into. [`snapshot`] is a cheap read of that map. The
//! default API is RAII: [`set_warning`] returns a [`WarningGuard`] whose
//! `Drop` removes the key, so a producer holds the guard exactly as long as
//! the condition is true and lets it drop to clear — no manual dismiss, no
//! stale warning. A failing periodic sweep does `let _w = set_warning(...)`
//! in its failure branch and drops it on the next success. Raising a warning
//! also logs one `warn` line with the message, so the text behind a `kind`
//! label is in the log store.
//!
//! Concurrency: all mutation is behind the one mutex. Each guard carries a
//! unique id and only touches its own map entry (`Drop` / [`update`]
@ -23,7 +23,6 @@
//! so a re-`set_warning` for the same kind is never clobbered when the
//! stale guard later drops.
//!
//! [`host_stats::server_warnings`]: crate::host_stats::server_warnings
//! [`update`]: WarningGuard::update
use std::collections::BTreeMap;
@ -49,7 +48,7 @@ struct Slot {
}
/// The one process-global registry. `BTreeMap` (not `HashMap`) so
/// [`snapshot`] yields a deterministic, kind-sorted banner order across
/// [`snapshot`] yields a deterministic, kind-sorted order across
/// renders.
fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> {
static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new();
@ -81,11 +80,13 @@ pub fn set_warning(
message: impl Into<String>,
) -> WarningGuard {
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
let message = message.into();
tracing::warn!(kind, level, %message, "hive warning raised");
lock().insert(
kind,
Slot {
level,
message: message.into(),
message,
owner: id,
},
);
@ -170,24 +171,24 @@ pub fn snapshot() -> Vec<ServerWarning> {
}
/// Raise a warning for a one-shot boot-time step that has no periodic
/// retry to observe a later success and clear the banner via
/// retry to observe a later success and clear the warning via
/// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`,
/// which `tokio::spawn`s once at hive-c0re startup and never runs again
/// this process. Intentionally **leaks** the guard for the life of the
/// process: the banner clears the next time hive-c0re restarts (a fresh
/// process: the warning clears the next time hive-c0re restarts (a fresh
/// process starts with an empty registry) and re-runs the step, which is
/// exactly when a config/environment fix would take effect anyway.
///
/// Do not use this for anything that runs periodically or can be
/// retried within the same process — hold the [`WarningGuard`] (or use
/// [`crate::stats::sweep_health::SweepHealth`]) so a later success can
/// actually clear the banner instead of waiting for a restart.
/// actually clear the warning instead of waiting for a restart.
pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) {
std::mem::forget(set_warning(kind, level, message));
}
/// RAII handle for one active warning. Drop clears the `kind` from the
/// banner. Obtained from [`set_warning`].
/// registry. Obtained from [`set_warning`].
#[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"]
pub struct WarningGuard {
kind: &'static str,