vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

39
docs/scheduler/observability.md
View file

@ -444,7 +444,7 @@ whole-swarm spend figure.
<!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these three per **hive** instead, and they carry no
runs in. It measures these five per **hive** instead, and they carry no
`agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with
@ -452,20 +452,41 @@ distinguishable from "this hive is gone." Select them with
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning |
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| Metric | Unit | Kind | Meaning |
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
visible as zeros rather than as missing series.
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
included, so a healthy hive is visible as zeros rather than as missing
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`.
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`,
`forge_ensure_core_user`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above.