hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -1,21 +1,21 @@
|
|||
//! Process-global server-warning registry backing the dashboard banner.
|
||||
//! Process-global hive-warning registry.
|
||||
//!
|
||||
//! The dashboard shows a top-of-page banner for hive-level degradation
|
||||
//! (disk pressure, a failed converge sweep, agents that can't log in, …).
|
||||
//! Historically each warning was *pulled*: [`host_stats::server_warnings`]
|
||||
//! recomputed the whole list on every `/api/state` by re-probing (a
|
||||
//! `statvfs`, a scan of the container snapshot). Background sweeps that
|
||||
//! failed had nowhere to record it, so they just `warn!`'d to the journal
|
||||
//! and vanished from anywhere an operator looks.
|
||||
//! Subsystems record hive-level degradation here (disk pressure, a failed
|
||||
//! converge sweep, agents that can't log in, …). Three consumers read it:
|
||||
//! `/health/ready` and the swarm status publisher through [`readiness`], and
|
||||
//! the `hyperhive.hive.degraded` / `hyperhive.hive.warnings` /
|
||||
//! `hyperhive.hive.warning` gauges in `otel_metrics`, which the Grafana
|
||||
//! alerts watch. There is no dashboard banner.
|
||||
//!
|
||||
//! This module replaces the pull model with a **push** one: a single
|
||||
//! `Mutex<BTreeMap<kind, …>>` any subsystem writes into. [`snapshot`] is a
|
||||
//! cheap read of that map. The default API is RAII: [`set_warning`] returns
|
||||
//! a [`WarningGuard`] whose `Drop` removes the key, so a producer holds the
|
||||
//! guard exactly as long as the condition is true and lets it drop to clear
|
||||
//! — no manual dismiss, no stale banner. A failing periodic sweep does
|
||||
//! `let _w = set_warning(...)` in its failure branch and drops it on the
|
||||
//! next success.
|
||||
//! The registry is **push**-based: a single `Mutex<BTreeMap<kind, …>>` any
|
||||
//! subsystem writes into. [`snapshot`] is a cheap read of that map. The
|
||||
//! default API is RAII: [`set_warning`] returns a [`WarningGuard`] whose
|
||||
//! `Drop` removes the key, so a producer holds the guard exactly as long as
|
||||
//! the condition is true and lets it drop to clear — no manual dismiss, no
|
||||
//! stale warning. A failing periodic sweep does `let _w = set_warning(...)`
|
||||
//! in its failure branch and drops it on the next success. Raising a warning
|
||||
//! also logs one `warn` line with the message, so the text behind a `kind`
|
||||
//! label is in the log store.
|
||||
//!
|
||||
//! Concurrency: all mutation is behind the one mutex. Each guard carries a
|
||||
//! unique id and only touches its own map entry (`Drop` / [`update`]
|
||||
|
|
@ -23,7 +23,6 @@
|
|||
//! so a re-`set_warning` for the same kind is never clobbered when the
|
||||
//! stale guard later drops.
|
||||
//!
|
||||
//! [`host_stats::server_warnings`]: crate::host_stats::server_warnings
|
||||
//! [`update`]: WarningGuard::update
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
|
@ -49,7 +48,7 @@ struct Slot {
|
|||
}
|
||||
|
||||
/// The one process-global registry. `BTreeMap` (not `HashMap`) so
|
||||
/// [`snapshot`] yields a deterministic, kind-sorted banner order across
|
||||
/// [`snapshot`] yields a deterministic, kind-sorted order across
|
||||
/// renders.
|
||||
fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> {
|
||||
static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new();
|
||||
|
|
@ -81,11 +80,13 @@ pub fn set_warning(
|
|||
message: impl Into<String>,
|
||||
) -> WarningGuard {
|
||||
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
|
||||
let message = message.into();
|
||||
tracing::warn!(kind, level, %message, "hive warning raised");
|
||||
lock().insert(
|
||||
kind,
|
||||
Slot {
|
||||
level,
|
||||
message: message.into(),
|
||||
message,
|
||||
owner: id,
|
||||
},
|
||||
);
|
||||
|
|
@ -170,24 +171,24 @@ pub fn snapshot() -> Vec<ServerWarning> {
|
|||
}
|
||||
|
||||
/// Raise a warning for a one-shot boot-time step that has no periodic
|
||||
/// retry to observe a later success and clear the banner via
|
||||
/// retry to observe a later success and clear the warning via
|
||||
/// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`,
|
||||
/// which `tokio::spawn`s once at hive-c0re startup and never runs again
|
||||
/// this process. Intentionally **leaks** the guard for the life of the
|
||||
/// process: the banner clears the next time hive-c0re restarts (a fresh
|
||||
/// process: the warning clears the next time hive-c0re restarts (a fresh
|
||||
/// process starts with an empty registry) and re-runs the step, which is
|
||||
/// exactly when a config/environment fix would take effect anyway.
|
||||
///
|
||||
/// Do not use this for anything that runs periodically or can be
|
||||
/// retried within the same process — hold the [`WarningGuard`] (or use
|
||||
/// [`crate::stats::sweep_health::SweepHealth`]) so a later success can
|
||||
/// actually clear the banner instead of waiting for a restart.
|
||||
/// actually clear the warning instead of waiting for a restart.
|
||||
pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) {
|
||||
std::mem::forget(set_warning(kind, level, message));
|
||||
}
|
||||
|
||||
/// RAII handle for one active warning. Drop clears the `kind` from the
|
||||
/// banner. Obtained from [`set_warning`].
|
||||
/// registry. Obtained from [`set_warning`].
|
||||
#[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"]
|
||||
pub struct WarningGuard {
|
||||
kind: &'static str,
|
||||
|
|
|
|||
Loading…
Reference in a new issue