hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -14,7 +14,7 @@
|
|||
//! coordinates is a silent no-op (the ordinary case). A hive that cannot
|
||||
//! reach the queue loses the swarm's visibility of the notice, not the
|
||||
//! host's — `warn!` fires on every failed attempt regardless, and the
|
||||
//! dashboard banners only after [`FAILURES_BEFORE_BANNER`] consecutive
|
||||
//! raises a hive warning only after [`FAILURES_BEFORE_WARNING`] consecutive
|
||||
//! misses, the same debounce shape [`crate::swarm_status`] uses and for
|
||||
//! the same reason: a `warn!` that fires every call for three weeks is
|
||||
//! indistinguishable from silence in practice.
|
||||
|
|
@ -24,10 +24,10 @@ use tokio::sync::{Mutex, OnceCell};
|
|||
|
||||
use crate::stats::sweep_health::{self, SweepHealth};
|
||||
|
||||
/// Consecutive failed publishes before the dashboard banners — same
|
||||
/// Consecutive failed publishes before the sweep raises a hive warning — same
|
||||
/// value [`crate::swarm_status`] uses and for the same reason: a debounce
|
||||
/// against one blip flapping a banner an operator learns to ignore.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
/// against one blip flapping a warning an operator learns to ignore.
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
static HEALTH: OnceCell<Mutex<SweepHealth>> = OnceCell::const_new();
|
||||
|
||||
|
|
@ -37,7 +37,7 @@ async fn health() -> tokio::sync::MutexGuard<'static, SweepHealth> {
|
|||
Mutex::new(SweepHealth::new(
|
||||
"swarm_notices_publish",
|
||||
"warn",
|
||||
FAILURES_BEFORE_BANNER,
|
||||
FAILURES_BEFORE_WARNING,
|
||||
))
|
||||
})
|
||||
.await
|
||||
|
|
@ -57,7 +57,7 @@ pub async fn notify(subsystem: &str, key: Option<String>, summary: String, sourc
|
|||
return;
|
||||
};
|
||||
// Same absent-name condition `swarm_status` bails on — that module
|
||||
// already banners it under `swarm_status_config` the first time
|
||||
// already warns it under `swarm_status_config` the first time
|
||||
// either of us hits it; nothing more to add from here.
|
||||
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
|
||||
return;
|
||||
|
|
|
|||
Loading…
Reference in a new issue