hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -444,7 +444,7 @@ whole-swarm spend figure.
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
hive-c0re measures everything above **per agent**, tagged with the hive it
|
||||
runs in. It measures these three per **hive** instead, and they carry no
|
||||
runs in. It measures these five per **hive** instead, and they carry no
|
||||
`agent` label — so a hive
|
||||
that hosts no agents still reports, and "this hive is quiet" is
|
||||
distinguishable from "this hive is gone." Select them with
|
||||
|
|
@ -452,20 +452,41 @@ distinguishable from "this hive is gone." Select them with
|
|||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
|
||||
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
|
||||
|
||||
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
|
||||
visible as zeros rather than as missing series.
|
||||
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
|
||||
included, so a healthy hive is visible as zeros rather than as missing
|
||||
series.
|
||||
|
||||
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
|
||||
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
|
||||
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
|
||||
detail behind it — `warn`-level entries mean "an operator should look" and do
|
||||
**not** set `degraded`.
|
||||
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
|
||||
label is one of a fixed set in the source (for example `disk_pressure`, a
|
||||
failing sweep such as `swarm_status_publish`, `pending_logins`,
|
||||
`forge_ensure_core_user`), and the message lives in the
|
||||
`hive warning raised` line hive-c0re logs when it raises the warning.
|
||||
|
||||
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
|
||||
|
||||
| Alert | Fires when |
|
||||
| ----------------------- | ----------------------------------------------------------------------------------- |
|
||||
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
|
||||
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
|
||||
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
|
||||
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
|
||||
|
||||
The disk alerts replace the old dashboard banner for the nix store, and
|
||||
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
|
||||
own pair so the thresholds sit on the metric rather than on the registry
|
||||
entry.
|
||||
|
||||
Same cadence, transport and resource labels as the container metrics above.
|
||||
|
||||
|
|
|
|||
|
|
@ -148,8 +148,8 @@ swarm-services certificate.
|
|||
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
|
||||
and publishes under `hiveName`, the same name that keys `swarm.hives`.
|
||||
|
||||
If a hive stops reporting, its own dashboard is the place to look: a
|
||||
failure to publish raises a warning banner there after three consecutive
|
||||
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
|
||||
failure to publish raises a hive warning after three consecutive
|
||||
misses. It stays `warn` rather than `crit` on purpose — a hive that
|
||||
can't reach the queue isn't itself unhealthy, so it doesn't start
|
||||
calling itself degraded for being unable to say it's fine.
|
||||
|
|
|
|||
|
|
@ -31,26 +31,17 @@ dashboard tab strip.
|
|||
K3PT ST4T3 (kept-state entries).
|
||||
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
|
||||
— sits below the tab strip.
|
||||
- **Server-warnings banner** — a generic, sticky top-of-page strip shown
|
||||
on **every** page (the dashboard + the stand-alone FL0W
|
||||
and ST4TS pages), injected at the top
|
||||
of `<body>` by `renderServerWarnings` in `common.js`. Driven by
|
||||
`state.server_warnings` — a list of `{ kind, level, message }` — and
|
||||
coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the
|
||||
threshold + message, so adding a new system warning needs no frontend
|
||||
change. Warnings are a push-based registry (`hive-c0re`'s
|
||||
`warnings.rs`): any subsystem raises/clears its own entry via an RAII
|
||||
guard, and `host_stats::server_warnings()` is just a cheap snapshot of
|
||||
that registry. Producers today include the host disk-pressure check
|
||||
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
|
||||
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
|
||||
provisioning/CI-runner boot failures, and an agent-state warning
|
||||
(`pending_logins`) computed alongside the container snapshot. A
|
||||
container crash used to have a sibling `agents_crashing` entry here —
|
||||
removed in favour of a swarm-level grafana alert on the crash
|
||||
watcher's own log line (`docs/scheduler/observability.md`), so a
|
||||
hive with no otel export now has no operator-visible crash signal at
|
||||
all. Hidden when there are no warnings.
|
||||
- **No warning banner.** No page shows hive-level warnings (host disk
|
||||
pressure, failing background sweeps, forge boot failures, agents that
|
||||
need a claude login). They live in a push-based registry
|
||||
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
|
||||
entry through an RAII guard, and each raise logs one
|
||||
`hive warning raised` line. The operator sees the registry as metrics and
|
||||
Grafana alerts: see
|
||||
[Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re).
|
||||
A container crash is a swarm-level Grafana alert on the crash watcher's
|
||||
own log line, so a hive with no otel export has no operator-visible
|
||||
warning signal beyond `/health/ready`.
|
||||
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
||||
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
||||
arrive in the state snapshot.
|
||||
|
|
|
|||
Loading…
Reference in a new issue