vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

7
hive-c0re/src/dashboard/health.rs
View file

@ -1,9 +1,8 @@
//! Hive-wide health endpoints for external monitoring: `/health/live`
//! (liveness) and `/health/ready` (readiness).
//!
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the
//! dashboard banner already reads — as the single source of truth for
//! "is anything degraded", rather than tracking health separately. Two
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
//! for "is anything degraded", rather than tracking health separately. Two
//! systems independently deciding what counts as unhealthy is how they
//! end up disagreeing.
//!
@ -12,7 +11,7 @@
//! the *process* down over a transient dependency failure — that's what
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
//! warning currently set; `warn`-level entries ride along in the body for
//! detail but don't flip the status, mirroring the banner's own severity
//! detail but don't flip the status, mirroring the warning registry's own severity
//! split.
//!
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can