Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions

View file

@ -498,6 +498,40 @@ let
&& lib.hasInfix ''_msg:="turn failed"'' expr
&& lib.hasInfix "stats by (hive, agent)" expr;
}
{
# The hive-health alerts query metric names that hive-c0re spells in
# Rust. A rename on either side provisions cleanly and the rule then
# reads nothing, so it never fires. Each name here is checked against
# the exporter's source rather than a copy of it, and each rule must
# query the metrics datasource with an instant query.
# 🔬 Invert-proof: rename a gauge in `otel_metrics.rs` and this fails.
name = "the hive-health alert rules query metrics hive-c0re exports";
ok =
let
groups =
grafanaHere.containers.swarm-grafana.config.services.grafana.provision.alerting.rules.settings.groups;
rules = lib.concatMap (g: g.rules) groups;
source = builtins.readFile ../../hive-c0re/src/stats/otel_metrics.rs;
expected = {
"hh-hive-degraded" = "hyperhive.hive.degraded";
"hh-hive-warning" = "hyperhive.hive.warning";
"hh-host-disk-high" = "hyperhive.hive.nix_store.utilization";
"hh-host-disk-critical" = "hyperhive.hive.nix_store.utilization";
};
check =
uid: metric:
let
rule = lib.findFirst (r: r.uid == uid) null rules;
query = (lib.head rule.data).model;
in
rule != null
&& lib.hasInfix ''{__name__="${metric}"'' query.expr
&& lib.hasInfix ''"${metric}"'' source
&& query.instant
&& (lib.head rule.data).datasourceUid == "swarm-victoriametrics";
in
lib.all (uid: check uid expected.${uid}) (builtins.attrNames expected);
}
{
# 🩸 The regression this guards: moving bao's metrics scrape into its
# own in-container collector made that collector stamp every exported