Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions

View file

@ -1,21 +1,21 @@
//! Process-global server-warning registry backing the dashboard banner.
//! Process-global hive-warning registry.
//!
//! The dashboard shows a top-of-page banner for hive-level degradation
//! (disk pressure, a failed converge sweep, agents that can't log in, …).
//! Historically each warning was *pulled*: [`host_stats::server_warnings`]
//! recomputed the whole list on every `/api/state` by re-probing (a
//! `statvfs`, a scan of the container snapshot). Background sweeps that
//! failed had nowhere to record it, so they just `warn!`'d to the journal
//! and vanished from anywhere an operator looks.
//! Subsystems record hive-level degradation here (disk pressure, a failed
//! converge sweep, agents that can't log in, …). Three consumers read it:
//! `/health/ready` and the swarm status publisher through [`readiness`], and
//! the `hyperhive.hive.degraded` / `hyperhive.hive.warnings` /
//! `hyperhive.hive.warning` gauges in `otel_metrics`, which the Grafana
//! alerts watch. There is no dashboard banner.
//!
//! This module replaces the pull model with a **push** one: a single
//! `Mutex<BTreeMap<kind, …>>` any subsystem writes into. [`snapshot`] is a
//! cheap read of that map. The default API is RAII: [`set_warning`] returns
//! a [`WarningGuard`] whose `Drop` removes the key, so a producer holds the
//! guard exactly as long as the condition is true and lets it drop to clear
//! — no manual dismiss, no stale banner. A failing periodic sweep does
//! `let _w = set_warning(...)` in its failure branch and drops it on the
//! next success.
//! The registry is **push**-based: a single `Mutex<BTreeMap<kind, …>>` any
//! subsystem writes into. [`snapshot`] is a cheap read of that map. The
//! default API is RAII: [`set_warning`] returns a [`WarningGuard`] whose
//! `Drop` removes the key, so a producer holds the guard exactly as long as
//! the condition is true and lets it drop to clear — no manual dismiss, no
//! stale warning. A failing periodic sweep does `let _w = set_warning(...)`
//! in its failure branch and drops it on the next success. Raising a warning
//! also logs one `warn` line with the message, so the text behind a `kind`
//! label is in the log store.
//!
//! Concurrency: all mutation is behind the one mutex. Each guard carries a
//! unique id and only touches its own map entry (`Drop` / [`update`]
@ -23,7 +23,6 @@
//! so a re-`set_warning` for the same kind is never clobbered when the
//! stale guard later drops.
//!
//! [`host_stats::server_warnings`]: crate::host_stats::server_warnings
//! [`update`]: WarningGuard::update
use std::collections::BTreeMap;
@ -49,7 +48,7 @@ struct Slot {
}
/// The one process-global registry. `BTreeMap` (not `HashMap`) so
/// [`snapshot`] yields a deterministic, kind-sorted banner order across
/// [`snapshot`] yields a deterministic, kind-sorted order across
/// renders.
fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> {
static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new();
@ -81,11 +80,13 @@ pub fn set_warning(
message: impl Into<String>,
) -> WarningGuard {
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
let message = message.into();
tracing::warn!(kind, level, %message, "hive warning raised");
lock().insert(
kind,
Slot {
level,
message: message.into(),
message,
owner: id,
},
);
@ -170,24 +171,24 @@ pub fn snapshot() -> Vec<ServerWarning> {
}
/// Raise a warning for a one-shot boot-time step that has no periodic
/// retry to observe a later success and clear the banner via
/// retry to observe a later success and clear the warning via
/// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`,
/// which `tokio::spawn`s once at hive-c0re startup and never runs again
/// this process. Intentionally **leaks** the guard for the life of the
/// process: the banner clears the next time hive-c0re restarts (a fresh
/// process: the warning clears the next time hive-c0re restarts (a fresh
/// process starts with an empty registry) and re-runs the step, which is
/// exactly when a config/environment fix would take effect anyway.
///
/// Do not use this for anything that runs periodically or can be
/// retried within the same process — hold the [`WarningGuard`] (or use
/// [`crate::stats::sweep_health::SweepHealth`]) so a later success can
/// actually clear the banner instead of waiting for a restart.
/// actually clear the warning instead of waiting for a restart.
pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) {
std::mem::forget(set_warning(kind, level, message));
}
/// RAII handle for one active warning. Drop clears the `kind` from the
/// banner. Obtained from [`set_warning`].
/// registry. Obtained from [`set_warning`].
#[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"]
pub struct WarningGuard {
kind: &'static str,