hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -148,8 +148,8 @@ swarm-services certificate.
|
|||
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
|
||||
and publishes under `hiveName`, the same name that keys `swarm.hives`.
|
||||
|
||||
If a hive stops reporting, its own dashboard is the place to look: a
|
||||
failure to publish raises a warning banner there after three consecutive
|
||||
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
|
||||
failure to publish raises a hive warning after three consecutive
|
||||
misses. It stays `warn` rather than `crit` on purpose — a hive that
|
||||
can't reach the queue isn't itself unhealthy, so it doesn't start
|
||||
calling itself degraded for being unable to say it's fine.
|
||||
|
|
|
|||
Loading…
Reference in a new issue