vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

146
hive-c0re/src/stats/host_stats.rs
View file

@ -1,35 +1,33 @@
//! Host-system probes and the derived **server-warning** list that the
//! dashboard renders as a top-of-page banner.
//! Host-system probes that raise hive-level warnings into the push-based
//! [`crate::warnings`] registry.
//!
//! The first (and currently only) producer is a host disk-usage check:
//! the nix store filling up is what ENOSPC-failed CI before the GC
//! guardrails were documented. hive-c0re runs on the host (not inside a
//! container), so it can `statvfs` the store path directly and warn the
//! operator *before* an ENOSPC, not after.
//! The first producer is a host disk-usage check: the nix store filling up
//! is what ENOSPC-failed CI before the GC guardrails were documented.
//! hive-c0re runs on the host (not inside a container), so it can `statvfs`
//! the store path directly and warn the operator *before* an ENOSPC, not
//! after. [`refresh_disk_warning`] drives a held guard from a periodic task
//! (see `spawn` in `main.rs`). The same probe backs the
//! `hyperhive.hive.nix_store.utilization` gauge ([`disk_utilization`]), which is
//! what the early-warning Grafana alerts read.
//!
//! [`server_warnings`] is the public surface: it returns a flat list of
//! [`ServerWarning`]s for `/api/state`. It is now just a snapshot of the
//! push-based [`crate::warnings`] registry — producers raise/clear their
//! own warnings via an RAII guard rather than being re-probed on every
//! render. The disk-pressure check is reworked onto that model here:
//! [`refresh_disk_warning`] drives a held guard from a periodic task (see
//! `spawn` in `main.rs`) instead of a `statvfs` per `/api/state`.
//! The registry has no dashboard banner any more. A raised warning reaches
//! the operator as the `hyperhive.hive.warning` gauge and a Grafana alert
//! on it (see `docs/scheduler/observability.md`).
use serde::Serialize;
use utoipa::ToSchema;
use crate::container_view::ContainerView;
/// One server-level warning for the dashboard's top-of-page banner.
/// One hive-level warning in the [`crate::warnings`] registry.
#[derive(Debug, Clone, Serialize, ToSchema)]
pub struct ServerWarning {
/// Stable kind id (e.g. `"disk_pressure"`) — lets the frontend dedupe
/// or special-case without parsing the message.
/// Stable kind id (e.g. `"disk_pressure"`), the `kind` label of the
/// `hyperhive.hive.warning` gauge.
pub kind: &'static str,
/// Severity: `"warn"` (amber) or `"crit"` (red). The banner picks its
/// colour from this; everything else is just the message text.
/// Severity: `"warn"` or `"crit"`. `crit` degrades readiness.
pub level: &'static str,
/// Human-readable, already-formatted message shown in the banner.
/// Human-readable, already-formatted message.
pub message: String,
}
@ -38,17 +36,7 @@ pub struct ServerWarning {
const DISK_WARN_PCT: f64 = 85.0;
const DISK_CRIT_PCT: f64 = 95.0;
/// Snapshot the current server-level warnings for the dashboard banner —
/// a cheap read of the push-based [`crate::warnings`] registry. Producers
/// (the disk watch below, background converge sweeps, …) raise and clear
/// their own entries via [`crate::warnings::set_warning`]; nothing is
/// re-probed here.
#[must_use]
pub fn server_warnings() -> Vec<ServerWarning> {
crate::warnings::snapshot()
}
/// Pure disk-pressure decision: given usage, return the banner
/// Pure disk-pressure decision: given usage, return the warning
/// `(level, message)` or `None` when below the warn threshold. Split from
/// the probe + guard plumbing so the threshold + formatting is unit-tested
/// without a real `statvfs`.
@ -75,9 +63,8 @@ fn disk_pressure_warning(used_pct: f64, free_bytes: u64) -> Option<(&'static str
/// One tick of the disk-pressure watch: probe the store and reconcile the
/// held warning `guard` — raise or refresh it while usage is over
/// threshold, drop it (clearing the banner) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`, replacing the old
/// per-render `statvfs` in [`server_warnings`].
/// threshold, drop it (clearing the warning) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`.
pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
let decision = nix_disk_usage().and_then(|d| disk_pressure_warning(d.used_pct, d.free_bytes));
match decision {
@ -96,19 +83,44 @@ pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
}
}
/// Agent-state warnings derived from the live container snapshot the
/// dashboard already holds: agents that need a claude login. Kept
/// separate from [`server_warnings`] (host probes) because the caller
/// owns the container list; the dashboard concatenates both into one
/// banner list.
/// Fraction (0–1) of the filesystem holding the nix store that is in use,
/// the value behind the `hyperhive.hive.nix_store.utilization` gauge. `None` when
/// the probe fails, so a failing probe reports no sample rather than zero.
#[must_use]
pub fn disk_utilization() -> Option<f64> {
nix_disk_usage().map(|d| d.used_pct / 100.0)
}
/// The held `pending_logins` warning; see [`refresh_pending_logins`].
static PENDING_LOGINS: std::sync::Mutex<Option<crate::warnings::WarningGuard>> =
std::sync::Mutex::new(None);
/// Reconcile the `pending_logins` warning with the current container
/// snapshot: raise or refresh it while any agent needs a claude login,
/// drop it once none does. Driven by the container rescan in the
/// coordinator, which already builds the snapshot every poll.
pub fn refresh_pending_logins(containers: &[ContainerView]) {
let warning = agent_state_warnings(containers).into_iter().next();
let mut guard = PENDING_LOGINS
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
match (warning, guard.as_ref()) {
(Some(w), Some(g)) => g.update(w.level, w.message),
(Some(w), None) => {
*guard = Some(crate::warnings::set_warning(w.kind, w.level, w.message));
}
(None, _) => *guard = None,
}
}
/// Agent-state warnings derived from the live container snapshot: agents
/// that need a claude login. Takes the container list from its caller
/// because that is where the snapshot lives.
///
/// Container crashes used to have a sibling `agents_crashing` entry here,
/// fed by `Coordinator::recent_crash_counts` — removed in favour of a
/// swarm-level grafana alert on the crash-watcher's own log line (see
/// `nix/host-modules/swarm-grafana.nix`'s `hh-container-crashed` rule and
/// `docs/scheduler/observability.md`). That ships the same signal to the
/// operator without needing an agent (or this per-hive banner) in the
/// loop at all.
/// Container crashes are not a warning here: the crash-watcher logs
/// `container crash detected` and a swarm-level grafana alert
/// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`) matches
/// that line, with no agent in the loop.
#[must_use]
pub fn agent_state_warnings(containers: &[ContainerView]) -> Vec<ServerWarning> {
let mut out = Vec::new();
@ -263,4 +275,46 @@ mod tests {
let w = agent_state_warnings(&containers);
assert!(w[0].message.starts_with("1 agent needs claude login: solo"));
}
/// The registry entry tracks the snapshot it is refreshed from: raised
/// when an agent needs a login, rewritten when the set changes, and gone
/// when none does. This is the only test that touches `pending_logins`,
/// so reading the process-global registry by that kind stays sound under
/// the parallel test runner.
#[test]
fn pending_logins_warning_follows_the_container_snapshot() {
let current = || {
crate::warnings::snapshot()
.into_iter()
.find(|w| w.kind == "pending_logins")
};
assert!(
current().is_none(),
"nothing raised before the first refresh"
);
refresh_pending_logins(&[cv("amy", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("1 agent needs claude login: amy")),
"an agent needing a login raises the warning"
);
refresh_pending_logins(&[cv("amy", true), cv("bob", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("2 agents need claude login")),
"a second agent rewrites the same entry"
);
refresh_pending_logins(&[cv("amy", false), cv("bob", false)]);
assert!(current().is_none(), "no agent needing a login clears it");
}
/// The gauge value is a fraction, so a percent leaking through would put
/// every alert threshold 100x off.
#[test]
fn disk_utilization_is_a_fraction() {
if let Some(used) = disk_utilization() {
assert!((0.0..=1.0).contains(&used), "got {used}");
}
}
}