hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
1 changed files with 468 additions and 322 deletions
|
|
@ -305,6 +305,41 @@ fn register_hive_instruments(meter: &opentelemetry::metrics::Meter) {
|
|||
}
|
||||
})
|
||||
.build();
|
||||
|
||||
// One series per raised warning, `1` while it is raised, labelled with
|
||||
// its `kind` and `level`. This is what names the cause in an alert: the
|
||||
// count above says how many, this says which. A cleared warning stops
|
||||
// reporting rather than reporting `0`, so an alert on it resolves when
|
||||
// the series goes stale; the kinds are a fixed set in the source, so the
|
||||
// label has bounded cardinality.
|
||||
meter
|
||||
.u64_observable_gauge("hyperhive.hive.warning")
|
||||
.with_unit("1")
|
||||
.with_callback(|obs| {
|
||||
for w in warnings::snapshot() {
|
||||
obs.observe(
|
||||
1,
|
||||
&[
|
||||
KeyValue::new("kind", w.kind),
|
||||
KeyValue::new("level", w.level),
|
||||
],
|
||||
);
|
||||
}
|
||||
})
|
||||
.build();
|
||||
|
||||
// Fraction of the nix store's filesystem in use, by the semconv name.
|
||||
// Probed in the callback (a `statvfs` is cheap and this runs once an
|
||||
// export interval); a failed probe reports no sample.
|
||||
meter
|
||||
.f64_observable_gauge("hyperhive.hive.nix_store.utilization")
|
||||
.with_unit("1")
|
||||
.with_callback(|obs| {
|
||||
if let Some(used) = crate::host_stats::disk_utilization() {
|
||||
obs.observe(used, HIVE_ATTRS);
|
||||
}
|
||||
})
|
||||
.build();
|
||||
}
|
||||
|
||||
/// Attributes for the hive-scoped family: **none**.
|
||||
|
|
|
|||
Loading…
Reference in a new issue