Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions

View file

@ -31,26 +31,17 @@ dashboard tab strip.
K3PT ST4T3 (kept-state entries).
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
— sits below the tab strip.
- **Server-warnings banner** — a generic, sticky top-of-page strip shown
on **every** page (the dashboard + the stand-alone FL0W
and ST4TS pages), injected at the top
of `<body>` by `renderServerWarnings` in `common.js`. Driven by
`state.server_warnings` — a list of `{ kind, level, message }` — and
coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the
threshold + message, so adding a new system warning needs no frontend
change. Warnings are a push-based registry (`hive-c0re`'s
`warnings.rs`): any subsystem raises/clears its own entry via an RAII
guard, and `host_stats::server_warnings()` is just a cheap snapshot of
that registry. Producers today include the host disk-pressure check
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
provisioning/CI-runner boot failures, and an agent-state warning
(`pending_logins`) computed alongside the container snapshot. A
container crash used to have a sibling `agents_crashing` entry here —
removed in favour of a swarm-level grafana alert on the crash
watcher's own log line (`docs/scheduler/observability.md`), so a
hive with no otel export now has no operator-visible crash signal at
all. Hidden when there are no warnings.
- **No warning banner.** No page shows hive-level warnings (host disk
pressure, failing background sweeps, forge boot failures, agents that
need a claude login). They live in a push-based registry
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
entry through an RAII guard, and each raise logs one
`hive warning raised` line. The operator sees the registry as metrics and
Grafana alerts: see
[Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re).
A container crash is a swarm-level Grafana alert on the crash watcher's
own log line, so a hive with no otel export has no operator-visible
warning signal beyond `/health/ready`.
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
arrive in the state snapshot.