hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
2 changed files with 468 additions and 322 deletions
|
|
@ -1,9 +1,8 @@
|
|||
//! Hive-wide health endpoints for external monitoring: `/health/live`
|
||||
//! (liveness) and `/health/ready` (readiness).
|
||||
//!
|
||||
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the
|
||||
//! dashboard banner already reads — as the single source of truth for
|
||||
//! "is anything degraded", rather than tracking health separately. Two
|
||||
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
|
||||
//! for "is anything degraded", rather than tracking health separately. Two
|
||||
//! systems independently deciding what counts as unhealthy is how they
|
||||
//! end up disagreeing.
|
||||
//!
|
||||
|
|
@ -12,7 +11,7 @@
|
|||
//! the *process* down over a transient dependency failure — that's what
|
||||
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
|
||||
//! warning currently set; `warn`-level entries ride along in the body for
|
||||
//! detail but don't flip the status, mirroring the banner's own severity
|
||||
//! detail but don't flip the status, mirroring the warning registry's own severity
|
||||
//! split.
|
||||
//!
|
||||
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can
|
||||
|
|
|
|||
|
|
@ -102,12 +102,6 @@ pub(super) struct StateSnapshot {
|
|||
/// var, set from `services.hyperhive.swarm.name`. `None` when
|
||||
/// unset — chrome omits the swarm segment of the breadcrumb.
|
||||
swarm_name: Option<String>,
|
||||
/// Server-level warnings for the dashboard's top-of-page banner
|
||||
/// (currently host disk-pressure; more producers can be added
|
||||
/// backend-side). Empty when all clear. Built by
|
||||
/// `host_stats::server_warnings`; the frontend renders this list
|
||||
/// generically, so new warning kinds need no frontend change.
|
||||
server_warnings: Vec<crate::host_stats::ServerWarning>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
|
|
@ -193,15 +187,6 @@ pub(super) async fn api_state(
|
|||
let tombstones = build_tombstone_views(&containers);
|
||||
let port_conflicts = build_port_conflicts(&containers);
|
||||
|
||||
// Banner warnings: host probes (disk) + agent-state (pending logins).
|
||||
// Built before the response struct because the agent-state producer
|
||||
// borrows `containers`, which moves in below.
|
||||
let server_warnings = {
|
||||
let mut w = crate::host_stats::server_warnings();
|
||||
w.extend(crate::host_stats::agent_state_warnings(&containers));
|
||||
w
|
||||
};
|
||||
|
||||
axum::Json(StateSnapshot {
|
||||
seq,
|
||||
hostname,
|
||||
|
|
@ -233,7 +218,6 @@ pub(super) async fn api_state(
|
|||
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty()),
|
||||
server_warnings,
|
||||
})
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue