vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

31
docs/scheduler/observability.md
View file

@ -444,7 +444,7 @@ whole-swarm spend figure.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these three per **hive** instead, and they carry no runs in. It measures these five per **hive** instead, and they carry no
`agent` label — so a hive `agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with distinguishable from "this hive is gone." Select them with
@ -453,19 +453,40 @@ distinguishable from "this hive is gone." Select them with
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning | | Metric | Unit | Kind | Meaning |
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- | | -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 | | `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows | | `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) | | `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
visible as zeros rather than as missing series. included, so a healthy hive is visible as zeros rather than as missing
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's `hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready` `hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`. **not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`,
`forge_ensure_core_user`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above. Same cadence, transport and resource labels as the container metrics above.

4
docs/swarm/README.md
View file

@ -148,8 +148,8 @@ swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>` The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`. and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its own dashboard is the place to look: a If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a warning banner there after three consecutive failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine. calling itself degraded for being unable to say it's fine.

31
docs/web-ui/dashboard.md
View file

@ -31,26 +31,17 @@ dashboard tab strip.
K3PT ST4T3 (kept-state entries). K3PT ST4T3 (kept-state entries).
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`) - **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
— sits below the tab strip. — sits below the tab strip.
- **Server-warnings banner** — a generic, sticky top-of-page strip shown - **No warning banner.** No page shows hive-level warnings (host disk
on **every** page (the dashboard + the stand-alone FL0W pressure, failing background sweeps, forge boot failures, agents that
and ST4TS pages), injected at the top need a claude login). They live in a push-based registry
of `<body>` by `renderServerWarnings` in `common.js`. Driven by (`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
`state.server_warnings` — a list of `{ kind, level, message }` — and entry through an RAII guard, and each raise logs one
coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the `hive warning raised` line. The operator sees the registry as metrics and
threshold + message, so adding a new system warning needs no frontend Grafana alerts: see
change. Warnings are a push-based registry (`hive-c0re`'s [Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re).
`warnings.rs`): any subsystem raises/clears its own entry via an RAII A container crash is a swarm-level Grafana alert on the crash watcher's
guard, and `host_stats::server_warnings()` is just a cheap snapshot of own log line, so a hive with no otel export has no operator-visible
that registry. Producers today include the host disk-pressure check warning signal beyond `/health/ready`.
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
provisioning/CI-runner boot failures, and an agent-state warning
(`pending_logins`) computed alongside the container snapshot. A
container crash used to have a sibling `agents_crashing` entry here —
removed in favour of a swarm-level grafana alert on the crash
watcher's own log line (`docs/scheduler/observability.md`), so a
hive with no otel export now has no operator-visible crash signal at
all. Hidden when there are no warnings.
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated - **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name` to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
arrive in the state snapshot. arrive in the state snapshot.

42
frontend/packages/dashboard/src/common.css
View file

@ -564,7 +564,7 @@ hive-side-panel .md td {
Inverted-gutter layout: the dashboard keeps `<body>` full-bleed Inverted-gutter layout: the dashboard keeps `<body>` full-bleed
and wraps their padded content in a single `.page-content` div that and wraps their padded content in a single `.page-content` div that
carries the horizontal gutter. Full-width chrome (the tab strip, the carries the horizontal gutter. Full-width chrome (the tab strip, the
server-warnings banner, the footer) then spans edge-to-edge for free, footer) then spans edge-to-edge for free,
with no negative-margin breakout hacks. Only matches on pages that opt with no negative-margin breakout hacks. Only matches on pages that opt
in by adding the wrapper (index.html); FL0W / ST4TS in by adding the wrapper (index.html); FL0W / ST4TS
are full-bleed with their own `.<page>-main` padding. */ are full-bleed with their own `.<page>-main` padding. */
@ -573,50 +573,14 @@ hive-side-panel .md td {
} }
/* ─── sticky top region ─────────────────────────────────────────────── /* ─── sticky top region ───────────────────────────────────────────────
One sticky container per page holding the warning banner above the One sticky container per page holding the page chrome (tab bar / page
page chrome (tab bar / page header). common.js (ensureStickyTop) header). common.js (initStickyTop) wraps the chrome in this on load. */
wraps the page's chrome in this on first render and injects the banner
as its first child, so the banner + chrome stack in a single sticky
context — pinned together — instead of two separate `top:0` stickies
colliding (the banner used to overlay the tab bar). */
.sticky-top { .sticky-top {
position: sticky; position: sticky;
top: 0; top: 0;
z-index: 50; z-index: 50;
} }
/* ─── server warnings banner ──────────────────────────────────────────
Generic top-of-page strip, injected as the first child of `.sticky-top`
by common.js (renderServerWarnings) on every page. One row per warning;
colour comes from the per-warning `level` (warn = amber, crit = red).
Stickiness lives on the `.sticky-top` wrapper, not here. The bar spans
the full width edge-to-edge; the dashboard is full-bleed at the
body level (their 1.5em gutter lives on an inner `.page-content`
wrapper), so the banner is full-width for free. */
.server-warnings[hidden] {
display: none;
}
.server-warn {
text-align: center;
padding: 0.4em 1em;
font-size: 0.85em;
font-weight: bold;
letter-spacing: 0.02em;
}
/* Tint composited over the opaque elevated-surface grey (not transparent):
the bar is `position: sticky`, so a see-through background would let
page content scroll through behind it. */
.server-warn-warn {
color: var(--amber);
background: color-mix(in srgb, var(--amber) 16%, var(--bg-elev));
border-bottom: 1px solid var(--amber);
}
.server-warn-crit {
color: var(--red);
background: color-mix(in srgb, var(--red) 18%, var(--bg-elev));
border-bottom: 1px solid var(--red);
}
/* ─── stats table ───────────────────────────────────────────────── /* ─── stats table ─────────────────────────────────────────────────
Shared right-aligned numeric table. Used by /stats.html (busiest Shared right-aligned numeric table. Used by /stats.html (busiest
agents) — moved here from dashboard.css when ST4TS became its own agents) — moved here from dashboard.css when ST4TS became its own

70
frontend/packages/dashboard/src/common.js
View file

@ -562,22 +562,12 @@ export function appendLinkified(parent, text, refs) {
} }
} }
// ─── server warnings banner ────────────────────────────────────────── // ─── sticky top region ───────────────────────────────────────────────
// A generic top-of-page banner shown on every page (dashboard + the // One sticky container wrapping the page's chrome (tab bar / page
// stand-alone FL0W / ST4TS pages). The backend decides what to // header), built once by wrapping the existing chrome element. A page
// warn about — `/api/state.server_warnings` is a list of // without a chrome (none today) gets an empty sticky region at the top of
// `{ kind, level, message }` — and this just renders it, coloured by // <body>.
// `level` (`warn` amber / `crit` red). Adding a new system warning is a export function initStickyTop() {
// backend-only change. The bar is injected at the top of <body> so no
// page needs to add markup.
// The sticky top region — one sticky container holding the warning
// banner above the page's chrome (tab bar / page header), so the banner
// stacks with the chrome instead of being overlaid by it (two separate
// `top:0` stickies would otherwise collide). Built once by wrapping the
// page's existing chrome element; pages without a chrome (none today)
// get a banner-only sticky region at the top of <body>.
function ensureStickyTop() {
let top = document.querySelector(".sticky-top"); let top = document.querySelector(".sticky-top");
if (top) return top; if (top) return top;
top = document.createElement("div"); top = document.createElement("div");
@ -591,51 +581,3 @@ function ensureStickyTop() {
} }
return top; return top;
} }
function ensureServerWarningsBar() {
let bar = document.getElementById("server-warnings");
if (!bar) {
bar = document.createElement("div");
bar.id = "server-warnings";
bar.className = "server-warnings";
bar.setAttribute("role", "alert");
bar.hidden = true;
ensureStickyTop().prepend(bar);
}
return bar;
}
/// Render a `server_warnings` list (from /api/state) into the banner.
/// Empty / missing → the bar hides itself.
export function renderServerWarnings(warnings) {
const bar = ensureServerWarningsBar();
bar.replaceChildren();
if (!Array.isArray(warnings) || warnings.length === 0) {
bar.hidden = true;
return;
}
for (const w of warnings) {
const row = el("div", {
class:
"server-warn server-warn-" +
(w && w.level === "crit" ? "crit" : "warn"),
});
appendText(row, "⚠ " + ((w && w.message) || ""));
bar.append(row);
}
bar.hidden = false;
}
/// One-shot init for pages that don't otherwise poll /api/state: ensure
/// the bar exists, fetch the snapshot once, render. The dashboard (which
/// already polls /api/state) calls `renderServerWarnings` directly for
/// live updates instead.
export function initServerWarnings() {
ensureServerWarningsBar();
fetch("/api/state")
.then((r) => (r.ok ? r.json() : null))
.then((s) => renderServerWarnings(s && s.server_warnings))
.catch(() => {
/* non-fatal: no banner if the snapshot is unreachable */
});
}

5
frontend/packages/dashboard/src/dashboard.css
View file

@ -17,7 +17,7 @@
body.dashboard-shell { body.dashboard-shell {
/* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives /* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives
on the inner `.page-content` wrapper (see common.css), so the sticky on the inner `.page-content` wrapper (see common.css), so the sticky
chrome bar, the server-warnings banner, and the footer span the full chrome bar and the footer span the full
width edge-to-edge with no negative-margin breakout. `padding-bottom` width edge-to-edge with no negative-margin breakout. `padding-bottom`
keeps a little breathing room above the viewport edge (overridden to keeps a little breathing room above the viewport edge (overridden to
clear the selection bar via `.has-selection` below). No max-width cap clear the selection bar via `.has-selection` below). No max-width cap
@ -34,8 +34,7 @@ body.dashboard-shell {
align-items: center; align-items: center;
gap: 0.75em; gap: 0.75em;
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that /* Stickiness lives on the `.sticky-top` wrapper (common.css) that
common.js wraps this chrome in, alongside the server-warning banner, common.js wraps this chrome in. */
so they stack in one sticky context instead of colliding. */
background: color-mix(in srgb, var(--bg) 86%, transparent); background: color-mix(in srgb, var(--bg) 86%, transparent);
-webkit-backdrop-filter: blur(8px) saturate(120%); -webkit-backdrop-filter: blur(8px) saturate(120%);
backdrop-filter: blur(8px) saturate(120%); backdrop-filter: blur(8px) saturate(120%);

9
frontend/packages/dashboard/src/flow.js
View file

@ -10,17 +10,12 @@
// container list). // container list).
import { create as termCreate } from "@hive/shared/terminal.js"; import { create as termCreate } from "@hive/shared/terminal.js";
import { import { $, appendLinkified, openStream, initStickyTop } from "./common.js";
$,
appendLinkified,
openStream,
initServerWarnings,
} from "./common.js";
import { el } from "@hive/shared/dom.js"; import { el } from "@hive/shared/dom.js";
import { epochSec } from "./util.js"; import { epochSec } from "./util.js";
(() => { (() => {
initServerWarnings(); initStickyTop();
// ─── local containers cache (for compose autocomplete) ────────────────── // ─── local containers cache (for compose autocomplete) ──────────────────
// The compose box's @-mention completion suggests known agent names. // The compose box's @-mention completion suggests known agent names.

4
frontend/packages/dashboard/src/stats.js
View file

@ -5,7 +5,7 @@
// this bundle has no chart lib; per-agent trend charts live on each // this bundle has no chart lib; per-agent trend charts live on each
// agent's own /stats page. The window selector is a hash-routed // agent's own /stats page. The window selector is a hash-routed
// createTabStrip (#1h / #24h / …), matching the per-agent stats page. // createTabStrip (#1h / #24h / …), matching the per-agent stats page.
import { $, initServerWarnings } from "./common.js"; import { $, initStickyTop } from "./common.js";
import { createTabStrip } from "@hive/shared/tabs.js"; import { createTabStrip } from "@hive/shared/tabs.js";
let hiveStatsWindow = "24h"; let hiveStatsWindow = "24h";
@ -173,7 +173,7 @@ async function refreshHiveStats() {
} }
} }
initServerWarnings(); initStickyTop();
// Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes // Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes
// here (the selector swaps data, not DOM); paneFor() returns null for each // here (the selector swaps data, not DOM); paneFor() returns null for each

5
frontend/packages/dashboard/src/tabs.js
View file

@ -11,7 +11,7 @@
// 4G3NTS (containers) domain lives in `./agents.js`. // 4G3NTS (containers) domain lives in `./agents.js`.
import { marked } from "marked"; import { marked } from "marked";
import { $, openStream, renderServerWarnings } from "./common.js"; import { $, openStream, initStickyTop } from "./common.js";
import { el } from "@hive/shared/dom.js"; import { el } from "@hive/shared/dom.js";
import { bindAsyncForms } from "@hive/shared/forms.js"; import { bindAsyncForms } from "@hive/shared/forms.js";
import { createTabStrip } from "@hive/shared/tabs.js"; import { createTabStrip } from "@hive/shared/tabs.js";
@ -50,6 +50,8 @@ import { renderLinksMenu } from "./links-menu.js";
window.marked = marked; window.marked = marked;
(() => { (() => {
initStickyTop();
// ─── async forms ──────────────────────────────────────────────────────── // ─── async forms ────────────────────────────────────────────────────────
// Shared `data-async` submit interceptor (in common.js, shared with the other // Shared `data-async` submit interceptor (in common.js, shared with the other
// pages' bundles). On success it re-runs refreshState unless the form // pages' bundles). On success it re-runs refreshState unless the form
@ -135,7 +137,6 @@ window.marked = marked;
// synchronous read (e.g. the compose autocomplete pulls agent // synchronous read (e.g. the compose autocomplete pulls agent
// names from here instead of refetching on every keystroke). // names from here instead of refetching on every keystroke).
window.__hyperhive_state = s; window.__hyperhive_state = s;
renderServerWarnings(s.server_warnings);
// Hive identity: render the swarm/hive name as a headline at the // Hive identity: render the swarm/hive name as a headline at the
// top of the 4G3NTS pane + update the page title once the // top of the 4G3NTS pane + update the page title once the
// server-side display names are known. `hive_name` / `swarm_name` // server-side display names are known. `hive_name` / `swarm_name`

3
frontend/packages/shared/src/chrome.css
View file

@ -16,8 +16,7 @@
*/ */
.page-header { .page-header {
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that /* Stickiness lives on the `.sticky-top` wrapper (common.css) that
common.js wraps this header in, alongside the server-warning banner, common.js wraps this header in. */
so the two stack instead of colliding. */
background: color-mix(in srgb, var(--bg) 92%, transparent); background: color-mix(in srgb, var(--bg) 92%, transparent);
-webkit-backdrop-filter: blur(8px) saturate(120%); -webkit-backdrop-filter: blur(8px) saturate(120%);
backdrop-filter: blur(8px) saturate(120%); backdrop-filter: blur(8px) saturate(120%);

1
hive-c0re/src/coordinator.rs
View file

@ -599,6 +599,7 @@ impl Coordinator {
return; return;
} }
}; };
crate::host_stats::refresh_pending_logins(&fresh);
let mut last = self.last_containers.lock().await; let mut last = self.last_containers.lock().await;
let mut changed_or_new = Vec::new(); let mut changed_or_new = Vec::new();
let mut removed = Vec::new(); let mut removed = Vec::new();

7
hive-c0re/src/dashboard/health.rs
View file

@ -1,9 +1,8 @@
//! Hive-wide health endpoints for external monitoring: `/health/live` //! Hive-wide health endpoints for external monitoring: `/health/live`
//! (liveness) and `/health/ready` (readiness). //! (liveness) and `/health/ready` (readiness).
//! //!
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the //! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
//! dashboard banner already reads — as the single source of truth for //! for "is anything degraded", rather than tracking health separately. Two
//! "is anything degraded", rather than tracking health separately. Two
//! systems independently deciding what counts as unhealthy is how they //! systems independently deciding what counts as unhealthy is how they
//! end up disagreeing. //! end up disagreeing.
//! //!
@ -12,7 +11,7 @@
//! the *process* down over a transient dependency failure — that's what //! the *process* down over a transient dependency failure — that's what
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level //! readiness is for. Readiness derives `"degraded"` from any `crit`-level
//! warning currently set; `warn`-level entries ride along in the body for //! warning currently set; `warn`-level entries ride along in the body for
//! detail but don't flip the status, mirroring the banner's own severity //! detail but don't flip the status, mirroring the warning registry's own severity
//! split. //! split.
//! //!
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can //! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can

16
hive-c0re/src/dashboard/state_snapshot.rs
View file

@ -102,12 +102,6 @@ pub(super) struct StateSnapshot {
/// var, set from `services.hyperhive.swarm.name`. `None` when /// var, set from `services.hyperhive.swarm.name`. `None` when
/// unset — chrome omits the swarm segment of the breadcrumb. /// unset — chrome omits the swarm segment of the breadcrumb.
swarm_name: Option<String>, swarm_name: Option<String>,
/// Server-level warnings for the dashboard's top-of-page banner
/// (currently host disk-pressure; more producers can be added
/// backend-side). Empty when all clear. Built by
/// `host_stats::server_warnings`; the frontend renders this list
/// generically, so new warning kinds need no frontend change.
server_warnings: Vec<crate::host_stats::ServerWarning>,
} }
#[derive(Serialize)] #[derive(Serialize)]
@ -193,15 +187,6 @@ pub(super) async fn api_state(
let tombstones = build_tombstone_views(&containers); let tombstones = build_tombstone_views(&containers);
let port_conflicts = build_port_conflicts(&containers); let port_conflicts = build_port_conflicts(&containers);
// Banner warnings: host probes (disk) + agent-state (pending logins).
// Built before the response struct because the agent-state producer
// borrows `containers`, which moves in below.
let server_warnings = {
let mut w = crate::host_stats::server_warnings();
w.extend(crate::host_stats::agent_state_warnings(&containers));
w
};
axum::Json(StateSnapshot { axum::Json(StateSnapshot {
seq, seq,
hostname, hostname,
@ -233,7 +218,6 @@ pub(super) async fn api_state(
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME") swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
.ok() .ok()
.filter(|s| !s.is_empty()), .filter(|s| !s.is_empty()),
server_warnings,
}) })
} }

6
hive-c0re/src/forge/mod.rs
View file

@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at /// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
/// most a handful per process). [`crate::warnings::set_boot_warning`] /// most a handful per process). [`crate::warnings::set_boot_warning`]
/// requires a `'static` kind so distinct orgs/repos don't clobber each /// requires a `'static` kind so distinct orgs/repos don't clobber each
/// other's banner entry; leaking a few short strings once per boot is /// other's warning entry; leaking a few short strings once per boot is
/// cheap and bounded, unlike a per-request or per-loop-iteration leak. /// cheap and bounded, unlike a per-request or per-loop-iteration leak.
fn static_kind(s: String) -> &'static str { fn static_kind(s: String) -> &'static str {
Box::leak(s.into_boxed_str()) Box::leak(s.into_boxed_str())
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
/// confirms the container exists — not that Forgejo is *listening*. A /// confirms the container exists — not that Forgejo is *listening*. A
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races: /// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
/// without this gate every `ensure_*` fires at a refused socket and leaves a /// without this gate every `ensure_*` fires at a refused socket and leaves a
/// stale "provisioning failed" banner that never clears (the pass is one-shot). /// stale "provisioning failed" warning that never clears (the pass is one-shot).
/// Returns whether the endpoint became ready before the deadline; on timeout /// Returns whether the endpoint became ready before the deadline; on timeout
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real /// the caller proceeds anyway so a genuinely-down forge still surfaces its real
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet). /// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
@ -393,7 +393,7 @@ pub async fn ensure_all() {
} }
// The container exists, but its HTTP may not be up yet (a rebuild restarts // The container exists, but its HTTP may not be up yet (a rebuild restarts
// hive-forge + hive-c0re together). Wait for the API to answer before the // hive-forge + hive-c0re together). Wait for the API to answer before the
// provisioning pass so a boot race doesn't leave stale failure banners. // provisioning pass so a boot race doesn't leave stale failure warnings.
wait_until_ready().await; wait_until_ready().await;
let core_token = match ensure_core_user_and_token().await { let core_token = match ensure_core_user_and_token().await {
Ok(t) => Some(t), Ok(t) => Some(t),

2
hive-c0re/src/forge/repos.rs
View file

@ -125,7 +125,7 @@ pub async fn push_meta(dir: &Path) -> Result<()> {
let msg = "forge/core/meta push rejected (non-fast-forward): \ let msg = "forge/core/meta push rejected (non-fast-forward): \
remote is ahead of local; leaving remote history intact"; remote is ahead of local; leaving remote history intact";
tracing::warn!("{msg}"); tracing::warn!("{msg}");
// Raise a persistent dashboard banner; cleared on next successful push. // Raise a persistent hive warning; cleared on next successful push.
if let Ok(mut g) = META_NON_FF_GUARD.lock() { if let Ok(mut g) = META_NON_FF_GUARD.lock() {
*g = Some(crate::warnings::set_warning( *g = Some(crate::warnings::set_warning(
"forge-meta-non-ff", "forge-meta-non-ff",

2
hive-c0re/src/gateway_nginx.rs
View file

@ -32,7 +32,7 @@ static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
/// Minimum gap between retry attempts after a reload failure (30 s). /// Minimum gap between retry attempts after a reload failure (30 s).
const RELOAD_RETRY_SECS: u64 = 30; const RELOAD_RETRY_SECS: u64 = 30;
/// Dashboard-banner health tracker for the reload sweep — raises a /// Hive-warning health tracker for the reload sweep — raises a
/// `gateway_nginx_reload` warning the first time a reload fails (routing is /// `gateway_nginx_reload` warning the first time a reload fails (routing is
/// broken for whoever depends on the change *right now*, so no debounce /// broken for whoever depends on the change *right now*, so no debounce
/// window) and clears it the moment a reload succeeds again. /// window) and clears it the moment a reload succeeds again.

14
hive-c0re/src/job_queue/exec.rs
View file

@ -139,7 +139,7 @@ pub(super) async fn run_node(
/// Boot-time forge user/token sweep as a DAG node — see /// Boot-time forge user/token sweep as a DAG node — see
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own /// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
/// per-step error handling and boot-warning banners internally (it's /// per-step error handling and boot warnings internally (it's
/// best-effort by design), so this wrapper has nothing left to report; /// best-effort by design), so this wrapper has nothing left to report;
/// it exists purely to make the sweep a visible unit of work. /// it exists purely to make the sweep a visible unit of work.
async fn run_forge_sweep() -> Result<()> { async fn run_forge_sweep() -> Result<()> {
@ -149,7 +149,7 @@ async fn run_forge_sweep() -> Result<()> {
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`]. /// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
/// Every pass reports here, whoever submitted it: as the node's own error on /// Every pass reports here, whoever submitted it: as the node's own error on
/// the dashboard, and into the debounced banner. /// the dashboard, and into the debounced warning.
async fn run_matrix_sweep() -> Result<()> { async fn run_matrix_sweep() -> Result<()> {
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all()) let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
.await .await
@ -164,7 +164,7 @@ async fn run_matrix_sweep() -> Result<()> {
health.record_ok(); health.record_ok();
Ok(()) Ok(())
} else { } else {
health.record_err(matrix_sweep_banner); health.record_err(matrix_sweep_warning);
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)") anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
} }
} }
@ -177,7 +177,7 @@ async fn run_matrix_sweep() -> Result<()> {
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep /// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10); const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
/// Banner for a matrix sweep that keeps failing. Two misses in a row, so one /// Warning for a matrix sweep that keeps failing. Two misses in a row, so one
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the /// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
/// dashboard; cleared by the next clean pass. /// dashboard; cleared by the next clean pass.
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> { fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
@ -185,7 +185,7 @@ fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2))) HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
} }
fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String { fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
let age = ctx.since_last_ok.map_or_else( let age = ctx.since_last_ok.map_or_else(
|| "no success this session".to_owned(), || "no success this session".to_owned(),
|d| format!("last ok {} ago", sweep_health::fmt_age(d)), |d| format!("last ok {} ago", sweep_health::fmt_age(d)),
@ -200,7 +200,7 @@ fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every /// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
/// pass reports here, whoever submitted it: as the node's own error on the /// pass reports here, whoever submitted it: as the node's own error on the
/// dashboard, and into the debounced banner. /// dashboard, and into the debounced warning.
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> { async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
let result = tokio::time::timeout( let result = tokio::time::timeout(
KNOWLEDGE_PULL_DEADLINE, KNOWLEDGE_PULL_DEADLINE,
@ -242,7 +242,7 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree /// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5); const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
/// Banner for a `/knowledge` pull that keeps failing: three misses in a row, /// Warning for a `/knowledge` pull that keeps failing: three misses in a row,
/// cleared by the next successful pull. /// cleared by the next successful pull.
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> { fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new(); static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();

6
hive-c0re/src/lifecycle/mod.rs
View file

@ -367,11 +367,11 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
agents_for_meta(None).await agents_for_meta(None).await
} }
/// Per-agent "stop failed" banner guards, keyed by agent name. Cleared the /// Per-agent "stop failed" warning guards, keyed by agent name. Cleared the
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>` /// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
/// rather than a bare `set_boot_warning` because this condition (unlike a /// rather than a bare `set_boot_warning` because this condition (unlike a
/// one-shot boot step) genuinely resolves later, once an operator has /// one-shot boot step) genuinely resolves later, once an operator has
/// intervened, and the banner should clear with it instead of surviving /// intervened, and the warning should clear with it instead of surviving
/// until the next hive-c0re restart. /// until the next hive-c0re restart.
fn stop_failed_guards() fn stop_failed_guards()
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> { -> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
@ -402,7 +402,7 @@ pub async fn kill(name: &str) -> Result<()> {
.unwrap_or_else(std::sync::PoisonError::into_inner); .unwrap_or_else(std::sync::PoisonError::into_inner);
match result { match result {
Ok(()) => { Ok(()) => {
// A clean stop clears any earlier "this looked wedged" banner // A clean stop clears any earlier "this looked wedged" warning
// for the same agent. // for the same agent.
guards.remove(name); guards.remove(name);
Ok(()) Ok(())

4
hive-c0re/src/main.rs
View file

@ -284,12 +284,12 @@ async fn cmd_serve(
} }
} }
}); });
// Disk-pressure watch: raise a `disk_pressure` banner warning while the // Disk-pressure watch: raise a `disk_pressure` warning while the
// host nix store is over threshold and clear it when back under, via the // host nix store is over threshold and clear it when back under, via the
// push-based warnings registry. Replaces the old per-`/api/state` // push-based warnings registry. Replaces the old per-`/api/state`
// `statvfs`. ~60s cadence; first tick fires immediately. The held guard // `statvfs`. ~60s cadence; first tick fires immediately. The held guard
// lives for the task's lifetime — dropping it (on shutdown) clears the // lives for the task's lifetime — dropping it (on shutdown) clears the
// banner. // warning.
let mut disk_shutdown = coord.shutdown_rx(); let mut disk_shutdown = coord.shutdown_rx();
tokio::spawn(async move { tokio::spawn(async move {
let mut disk_guard: Option<warnings::WarningGuard> = None; let mut disk_guard: Option<warnings::WarningGuard> = None;

4
hive-c0re/src/matrix.rs
View file

@ -787,8 +787,8 @@ pub async fn invite_user(
/// Returns `true` when every step of the sweep succeeded, `false` when at /// Returns `true` when every step of the sweep succeeded, `false` when at
/// least one step failed — the caller feeds this into a /// least one step failed — the caller feeds this into a
/// [`crate::stats::sweep_health::SweepHealth`] to raise a debounced /// [`crate::stats::sweep_health::SweepHealth`] to raise a debounced
/// dashboard banner on persistent failure (this sweep re-runs every 30 /// hive warning on persistent failure (this sweep re-runs every 30
/// minutes, so a one-off blip self-heals without ever bannering). /// minutes, so a one-off blip self-heals without ever warning).
pub async fn ensure_all() -> bool { pub async fn ensure_all() -> bool {
if !is_present() { if !is_present() {
tracing::debug!("matrix: hive-matrix container absent, skipping user sweep"); tracing::debug!("matrix: hive-matrix container absent, skipping user sweep");

146
hive-c0re/src/stats/host_stats.rs
View file

@ -1,35 +1,33 @@
//! Host-system probes and the derived **server-warning** list that the //! Host-system probes that raise hive-level warnings into the push-based
//! dashboard renders as a top-of-page banner. //! [`crate::warnings`] registry.
//! //!
//! The first (and currently only) producer is a host disk-usage check: //! The first producer is a host disk-usage check: the nix store filling up
//! the nix store filling up is what ENOSPC-failed CI before the GC //! is what ENOSPC-failed CI before the GC guardrails were documented.
//! guardrails were documented. hive-c0re runs on the host (not inside a //! hive-c0re runs on the host (not inside a container), so it can `statvfs`
//! container), so it can `statvfs` the store path directly and warn the //! the store path directly and warn the operator *before* an ENOSPC, not
//! operator *before* an ENOSPC, not after. //! after. [`refresh_disk_warning`] drives a held guard from a periodic task
//! (see `spawn` in `main.rs`). The same probe backs the
//! `hyperhive.hive.nix_store.utilization` gauge ([`disk_utilization`]), which is
//! what the early-warning Grafana alerts read.
//! //!
//! [`server_warnings`] is the public surface: it returns a flat list of //! The registry has no dashboard banner any more. A raised warning reaches
//! [`ServerWarning`]s for `/api/state`. It is now just a snapshot of the //! the operator as the `hyperhive.hive.warning` gauge and a Grafana alert
//! push-based [`crate::warnings`] registry — producers raise/clear their //! on it (see `docs/scheduler/observability.md`).
//! own warnings via an RAII guard rather than being re-probed on every
//! render. The disk-pressure check is reworked onto that model here:
//! [`refresh_disk_warning`] drives a held guard from a periodic task (see
//! `spawn` in `main.rs`) instead of a `statvfs` per `/api/state`.
use serde::Serialize; use serde::Serialize;
use utoipa::ToSchema; use utoipa::ToSchema;
use crate::container_view::ContainerView; use crate::container_view::ContainerView;
/// One server-level warning for the dashboard's top-of-page banner. /// One hive-level warning in the [`crate::warnings`] registry.
#[derive(Debug, Clone, Serialize, ToSchema)] #[derive(Debug, Clone, Serialize, ToSchema)]
pub struct ServerWarning { pub struct ServerWarning {
/// Stable kind id (e.g. `"disk_pressure"`) — lets the frontend dedupe /// Stable kind id (e.g. `"disk_pressure"`), the `kind` label of the
/// or special-case without parsing the message. /// `hyperhive.hive.warning` gauge.
pub kind: &'static str, pub kind: &'static str,
/// Severity: `"warn"` (amber) or `"crit"` (red). The banner picks its /// Severity: `"warn"` or `"crit"`. `crit` degrades readiness.
/// colour from this; everything else is just the message text.
pub level: &'static str, pub level: &'static str,
/// Human-readable, already-formatted message shown in the banner. /// Human-readable, already-formatted message.
pub message: String, pub message: String,
} }
@ -38,17 +36,7 @@ pub struct ServerWarning {
const DISK_WARN_PCT: f64 = 85.0; const DISK_WARN_PCT: f64 = 85.0;
const DISK_CRIT_PCT: f64 = 95.0; const DISK_CRIT_PCT: f64 = 95.0;
/// Snapshot the current server-level warnings for the dashboard banner — /// Pure disk-pressure decision: given usage, return the warning
/// a cheap read of the push-based [`crate::warnings`] registry. Producers
/// (the disk watch below, background converge sweeps, …) raise and clear
/// their own entries via [`crate::warnings::set_warning`]; nothing is
/// re-probed here.
#[must_use]
pub fn server_warnings() -> Vec<ServerWarning> {
crate::warnings::snapshot()
}
/// Pure disk-pressure decision: given usage, return the banner
/// `(level, message)` or `None` when below the warn threshold. Split from /// `(level, message)` or `None` when below the warn threshold. Split from
/// the probe + guard plumbing so the threshold + formatting is unit-tested /// the probe + guard plumbing so the threshold + formatting is unit-tested
/// without a real `statvfs`. /// without a real `statvfs`.
@ -75,9 +63,8 @@ fn disk_pressure_warning(used_pct: f64, free_bytes: u64) -> Option<(&'static str
/// One tick of the disk-pressure watch: probe the store and reconcile the /// One tick of the disk-pressure watch: probe the store and reconcile the
/// held warning `guard` — raise or refresh it while usage is over /// held warning `guard` — raise or refresh it while usage is over
/// threshold, drop it (clearing the banner) once back under. Driven by the /// threshold, drop it (clearing the warning) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`, replacing the old /// periodic disk-watch task spawned in `main.rs`.
/// per-render `statvfs` in [`server_warnings`].
pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) { pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
let decision = nix_disk_usage().and_then(|d| disk_pressure_warning(d.used_pct, d.free_bytes)); let decision = nix_disk_usage().and_then(|d| disk_pressure_warning(d.used_pct, d.free_bytes));
match decision { match decision {
@ -96,19 +83,44 @@ pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
} }
} }
/// Agent-state warnings derived from the live container snapshot the /// Fraction (0–1) of the filesystem holding the nix store that is in use,
/// dashboard already holds: agents that need a claude login. Kept /// the value behind the `hyperhive.hive.nix_store.utilization` gauge. `None` when
/// separate from [`server_warnings`] (host probes) because the caller /// the probe fails, so a failing probe reports no sample rather than zero.
/// owns the container list; the dashboard concatenates both into one #[must_use]
/// banner list. pub fn disk_utilization() -> Option<f64> {
nix_disk_usage().map(|d| d.used_pct / 100.0)
}
/// The held `pending_logins` warning; see [`refresh_pending_logins`].
static PENDING_LOGINS: std::sync::Mutex<Option<crate::warnings::WarningGuard>> =
std::sync::Mutex::new(None);
/// Reconcile the `pending_logins` warning with the current container
/// snapshot: raise or refresh it while any agent needs a claude login,
/// drop it once none does. Driven by the container rescan in the
/// coordinator, which already builds the snapshot every poll.
pub fn refresh_pending_logins(containers: &[ContainerView]) {
let warning = agent_state_warnings(containers).into_iter().next();
let mut guard = PENDING_LOGINS
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
match (warning, guard.as_ref()) {
(Some(w), Some(g)) => g.update(w.level, w.message),
(Some(w), None) => {
*guard = Some(crate::warnings::set_warning(w.kind, w.level, w.message));
}
(None, _) => *guard = None,
}
}
/// Agent-state warnings derived from the live container snapshot: agents
/// that need a claude login. Takes the container list from its caller
/// because that is where the snapshot lives.
/// ///
/// Container crashes used to have a sibling `agents_crashing` entry here, /// Container crashes are not a warning here: the crash-watcher logs
/// fed by `Coordinator::recent_crash_counts` — removed in favour of a /// `container crash detected` and a swarm-level grafana alert
/// swarm-level grafana alert on the crash-watcher's own log line (see /// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`) matches
/// `nix/host-modules/swarm-grafana.nix`'s `hh-container-crashed` rule and /// that line, with no agent in the loop.
/// `docs/scheduler/observability.md`). That ships the same signal to the
/// operator without needing an agent (or this per-hive banner) in the
/// loop at all.
#[must_use] #[must_use]
pub fn agent_state_warnings(containers: &[ContainerView]) -> Vec<ServerWarning> { pub fn agent_state_warnings(containers: &[ContainerView]) -> Vec<ServerWarning> {
let mut out = Vec::new(); let mut out = Vec::new();
@ -263,4 +275,46 @@ mod tests {
let w = agent_state_warnings(&containers); let w = agent_state_warnings(&containers);
assert!(w[0].message.starts_with("1 agent needs claude login: solo")); assert!(w[0].message.starts_with("1 agent needs claude login: solo"));
} }
/// The registry entry tracks the snapshot it is refreshed from: raised
/// when an agent needs a login, rewritten when the set changes, and gone
/// when none does. This is the only test that touches `pending_logins`,
/// so reading the process-global registry by that kind stays sound under
/// the parallel test runner.
#[test]
fn pending_logins_warning_follows_the_container_snapshot() {
let current = || {
crate::warnings::snapshot()
.into_iter()
.find(|w| w.kind == "pending_logins")
};
assert!(
current().is_none(),
"nothing raised before the first refresh"
);
refresh_pending_logins(&[cv("amy", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("1 agent needs claude login: amy")),
"an agent needing a login raises the warning"
);
refresh_pending_logins(&[cv("amy", true), cv("bob", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("2 agents need claude login")),
"a second agent rewrites the same entry"
);
refresh_pending_logins(&[cv("amy", false), cv("bob", false)]);
assert!(current().is_none(), "no agent needing a login clears it");
}
/// The gauge value is a fraction, so a percent leaking through would put
/// every alert threshold 100x off.
#[test]
fn disk_utilization_is_a_fraction() {
if let Some(used) = disk_utilization() {
assert!((0.0..=1.0).contains(&used), "got {used}");
}
}
} }

35
hive-c0re/src/stats/otel_metrics.rs
View file

@ -305,6 +305,41 @@ fn register_hive_instruments(meter: &opentelemetry::metrics::Meter) {
} }
}) })
.build(); .build();
// One series per raised warning, `1` while it is raised, labelled with
// its `kind` and `level`. This is what names the cause in an alert: the
// count above says how many, this says which. A cleared warning stops
// reporting rather than reporting `0`, so an alert on it resolves when
// the series goes stale; the kinds are a fixed set in the source, so the
// label has bounded cardinality.
meter
.u64_observable_gauge("hyperhive.hive.warning")
.with_unit("1")
.with_callback(|obs| {
for w in warnings::snapshot() {
obs.observe(
1,
&[
KeyValue::new("kind", w.kind),
KeyValue::new("level", w.level),
],
);
}
})
.build();
// Fraction of the nix store's filesystem in use, by the semconv name.
// Probed in the callback (a `statvfs` is cheap and this runs once an
// export interval); a failed probe reports no sample.
meter
.f64_observable_gauge("hyperhive.hive.nix_store.utilization")
.with_unit("1")
.with_callback(|obs| {
if let Some(used) = crate::host_stats::disk_utilization() {
obs.observe(used, HIVE_ATTRS);
}
})
.build();
} }
/// Attributes for the hive-scoped family: **none**. /// Attributes for the hive-scoped family: **none**.

50
hive-c0re/src/stats/sweep_health.rs
View file

@ -5,14 +5,14 @@
//! gateway nginx writes, per-agent branch protection, …) historically //! gateway nginx writes, per-agent branch protection, …) historically
//! `warn!`'d to the journal and moved on, so a persistent failure was //! `warn!`'d to the journal and moved on, so a persistent failure was
//! invisible to the operator. [`SweepHealth`] routes that failure to the //! invisible to the operator. [`SweepHealth`] routes that failure to the
//! dashboard banner instead — but *debounced*, per the routing guidance in //! hive warning instead — but *debounced*, per the routing guidance in
//! the tracker issue: //! the tracker issue:
//! //!
//! - a transient miss (container down mid-sweep, a network blip) shouldn't //! - a transient miss (container down mid-sweep, a network blip) shouldn't
//! flap a banner on the first failure → `threshold > 1` raises only after //! flap a warning on the first failure → `threshold > 1` raises only after
//! that many **consecutive** misses; //! that many **consecutive** misses;
//! - a won't-fix-itself failure (missing team, bad config, auth rejection) //! - a won't-fix-itself failure (missing team, bad config, auth rejection)
//! should banner immediately → `threshold == 1`. //! should warning immediately → `threshold == 1`.
//! //!
//! The warning clears the moment the sweep next succeeds. A sweep owns one //! The warning clears the moment the sweep next succeeds. A sweep owns one
//! `SweepHealth`, calls [`record_ok`](SweepHealth::record_ok) on success and //! `SweepHealth`, calls [`record_ok`](SweepHealth::record_ok) on success and
@ -24,11 +24,11 @@ use std::time::{Duration, Instant};
use crate::warnings::{WarningGuard, set_warning}; use crate::warnings::{WarningGuard, set_warning};
/// Per-sweep failure context handed to the [`record_err`](SweepHealth::record_err) /// Per-sweep failure context handed to the [`record_err`](SweepHealth::record_err)
/// message builder so the banner text can include how long the sweep has been /// message builder so the warning text can include how long the sweep has been
/// failing and since when it last worked. /// failing and since when it last worked.
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub struct SweepFailure { pub struct SweepFailure {
/// Number of consecutive failures so far (≥ `threshold` when the banner /// Number of consecutive failures so far (≥ `threshold` when the warning
/// is raised). /// is raised).
pub consecutive: u32, pub consecutive: u32,
/// Time since the last recorded success, if the sweep has ever succeeded /// Time since the last recorded success, if the sweep has ever succeeded
@ -36,7 +36,7 @@ pub struct SweepFailure {
pub since_last_ok: Option<Duration>, pub since_last_ok: Option<Duration>,
} }
/// A debounced link between one background sweep and the warning banner. /// A debounced link between one background sweep and the warning warning.
/// Not `Sync`-shared: each sweep task owns its own instance (the underlying /// Not `Sync`-shared: each sweep task owns its own instance (the underlying
/// registry is the shared, thread-safe part). /// registry is the shared, thread-safe part).
pub struct SweepHealth { pub struct SweepHealth {
@ -50,8 +50,8 @@ pub struct SweepHealth {
impl SweepHealth { impl SweepHealth {
/// Track sweep `kind` at severity `level` (`"warn"` / `"crit"`), raising /// Track sweep `kind` at severity `level` (`"warn"` / `"crit"`), raising
/// the banner after `threshold` consecutive failures. `threshold` is /// the warning after `threshold` consecutive failures. `threshold` is
/// floored at 1 (0 would banner before any failure). /// floored at 1 (0 would warning before any failure).
#[must_use] #[must_use]
pub fn new(kind: &'static str, level: &'static str, threshold: u32) -> Self { pub fn new(kind: &'static str, level: &'static str, threshold: u32) -> Self {
Self { Self {
@ -65,7 +65,7 @@ impl SweepHealth {
} }
/// Record a successful sweep: reset the consecutive-miss counter and drop /// Record a successful sweep: reset the consecutive-miss counter and drop
/// the guard (clearing any banner entry). /// the guard (clearing any warning entry).
pub fn record_ok(&mut self) { pub fn record_ok(&mut self) {
self.misses = 0; self.misses = 0;
self.last_ok = Some(Instant::now()); self.last_ok = Some(Instant::now());
@ -74,9 +74,9 @@ impl SweepHealth {
} }
/// Record a failed sweep. Bumps the consecutive-miss counter; once it /// Record a failed sweep. Bumps the consecutive-miss counter; once it
/// reaches `threshold`, raises (or refreshes) the banner warning built by /// reaches `threshold`, raises (or refreshes) the warning warning built by
/// `message`. Below the threshold this is a cheap no-op — `message` is /// `message`. Below the threshold this is a cheap no-op — `message` is
/// only invoked when the banner is actually shown, so callers can format /// only invoked when the warning is actually shown, so callers can format
/// lazily. /// lazily.
pub fn record_err(&mut self, message: impl FnOnce(SweepFailure) -> String) { pub fn record_err(&mut self, message: impl FnOnce(SweepFailure) -> String) {
self.misses = self.misses.saturating_add(1); self.misses = self.misses.saturating_add(1);
@ -96,7 +96,7 @@ impl SweepHealth {
} }
/// Compact human-readable age (e.g. `"45s"`, `"12m"`, `"3h"`, `"2d"`) for a /// Compact human-readable age (e.g. `"45s"`, `"12m"`, `"3h"`, `"2d"`) for a
/// banner "last succeeded N ago" suffix. Coarse on purpose — the banner wants /// warning "last succeeded N ago" suffix. Coarse on purpose — the warning wants
/// a glanceable magnitude, not precision. /// a glanceable magnitude, not precision.
#[must_use] #[must_use]
pub fn fmt_age(d: Duration) -> String { pub fn fmt_age(d: Duration) -> String {
@ -117,7 +117,7 @@ mod tests {
use super::*; use super::*;
use crate::warnings::snapshot; use crate::warnings::snapshot;
fn banner(kind: &str) -> Option<String> { fn warning(kind: &str) -> Option<String> {
snapshot() snapshot()
.into_iter() .into_iter()
.find(|w| w.kind == kind) .find(|w| w.kind == kind)
@ -125,34 +125,34 @@ mod tests {
} }
#[test] #[test]
fn below_threshold_does_not_banner() { fn below_threshold_does_not_warn() {
let mut h = SweepHealth::new("sh_below", "warn", 3); let mut h = SweepHealth::new("sh_below", "warn", 3);
h.record_err(|_| "boom".to_owned()); h.record_err(|_| "boom".to_owned());
h.record_err(|_| "boom".to_owned()); h.record_err(|_| "boom".to_owned());
assert!(h.guard.is_none()); assert!(h.guard.is_none());
assert!(banner("sh_below").is_none()); assert!(warning("sh_below").is_none());
} }
#[test] #[test]
fn raises_at_threshold_and_clears_on_ok() { fn raises_at_threshold_and_clears_on_ok() {
let mut h = SweepHealth::new("sh_raise", "warn", 2); let mut h = SweepHealth::new("sh_raise", "warn", 2);
h.record_err(|c| format!("fail #{}", c.consecutive)); h.record_err(|c| format!("fail #{}", c.consecutive));
assert!(banner("sh_raise").is_none(), "one miss < threshold"); assert!(warning("sh_raise").is_none(), "one miss < threshold");
h.record_err(|c| format!("fail #{}", c.consecutive)); h.record_err(|c| format!("fail #{}", c.consecutive));
assert_eq!(banner("sh_raise").as_deref(), Some("fail #2")); assert_eq!(warning("sh_raise").as_deref(), Some("fail #2"));
assert!(h.guard.is_some()); assert!(h.guard.is_some());
h.record_ok(); h.record_ok();
assert!(h.guard.is_none()); assert!(h.guard.is_none());
assert!(banner("sh_raise").is_none(), "success clears the banner"); assert!(warning("sh_raise").is_none(), "success clears the warning");
} }
#[test] #[test]
fn threshold_one_banners_immediately() { fn threshold_one_warns_immediately() {
let mut h = SweepHealth::new("sh_immediate", "crit", 1); let mut h = SweepHealth::new("sh_immediate", "crit", 1);
h.record_err(|_| "gone".to_owned()); h.record_err(|_| "gone".to_owned());
assert_eq!(banner("sh_immediate").as_deref(), Some("gone")); assert_eq!(warning("sh_immediate").as_deref(), Some("gone"));
h.record_ok(); h.record_ok();
assert!(banner("sh_immediate").is_none()); assert!(warning("sh_immediate").is_none());
} }
#[test] #[test]
@ -161,9 +161,9 @@ mod tests {
h.record_err(|_| "x".to_owned()); // miss 1 h.record_err(|_| "x".to_owned()); // miss 1
h.record_ok(); // reset h.record_ok(); // reset
h.record_err(|c| format!("c={}", c.consecutive)); // miss 1 again, < threshold h.record_err(|c| format!("c={}", c.consecutive)); // miss 1 again, < threshold
assert!(banner("sh_reset").is_none(), "counter reset after ok"); assert!(warning("sh_reset").is_none(), "counter reset after ok");
h.record_err(|c| format!("c={}", c.consecutive)); // miss 2 → raise h.record_err(|c| format!("c={}", c.consecutive)); // miss 2 → raise
assert_eq!(banner("sh_reset").as_deref(), Some("c=2")); assert_eq!(warning("sh_reset").as_deref(), Some("c=2"));
h.record_ok(); h.record_ok();
} }
@ -171,9 +171,9 @@ mod tests {
fn refresh_updates_message_while_held() { fn refresh_updates_message_while_held() {
let mut h = SweepHealth::new("sh_refresh", "warn", 1); let mut h = SweepHealth::new("sh_refresh", "warn", 1);
h.record_err(|c| format!("miss {}", c.consecutive)); h.record_err(|c| format!("miss {}", c.consecutive));
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 1")); assert_eq!(warning("sh_refresh").as_deref(), Some("miss 1"));
h.record_err(|c| format!("miss {}", c.consecutive)); h.record_err(|c| format!("miss {}", c.consecutive));
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 2")); assert_eq!(warning("sh_refresh").as_deref(), Some("miss 2"));
h.record_ok(); h.record_ok();
} }

47
hive-c0re/src/stats/warnings.rs
View file

@ -1,21 +1,21 @@
//! Process-global server-warning registry backing the dashboard banner. //! Process-global hive-warning registry.
//! //!
//! The dashboard shows a top-of-page banner for hive-level degradation //! Subsystems record hive-level degradation here (disk pressure, a failed
//! (disk pressure, a failed converge sweep, agents that can't log in, …). //! converge sweep, agents that can't log in, …). Three consumers read it:
//! Historically each warning was *pulled*: [`host_stats::server_warnings`] //! `/health/ready` and the swarm status publisher through [`readiness`], and
//! recomputed the whole list on every `/api/state` by re-probing (a //! the `hyperhive.hive.degraded` / `hyperhive.hive.warnings` /
//! `statvfs`, a scan of the container snapshot). Background sweeps that //! `hyperhive.hive.warning` gauges in `otel_metrics`, which the Grafana
//! failed had nowhere to record it, so they just `warn!`'d to the journal //! alerts watch. There is no dashboard banner.
//! and vanished from anywhere an operator looks.
//! //!
//! This module replaces the pull model with a **push** one: a single //! The registry is **push**-based: a single `Mutex<BTreeMap<kind, …>>` any
//! `Mutex<BTreeMap<kind, …>>` any subsystem writes into. [`snapshot`] is a //! subsystem writes into. [`snapshot`] is a cheap read of that map. The
//! cheap read of that map. The default API is RAII: [`set_warning`] returns //! default API is RAII: [`set_warning`] returns a [`WarningGuard`] whose
//! a [`WarningGuard`] whose `Drop` removes the key, so a producer holds the //! `Drop` removes the key, so a producer holds the guard exactly as long as
//! guard exactly as long as the condition is true and lets it drop to clear //! the condition is true and lets it drop to clear — no manual dismiss, no
//! — no manual dismiss, no stale banner. A failing periodic sweep does //! stale warning. A failing periodic sweep does `let _w = set_warning(...)`
//! `let _w = set_warning(...)` in its failure branch and drops it on the //! in its failure branch and drops it on the next success. Raising a warning
//! next success. //! also logs one `warn` line with the message, so the text behind a `kind`
//! label is in the log store.
//! //!
//! Concurrency: all mutation is behind the one mutex. Each guard carries a //! Concurrency: all mutation is behind the one mutex. Each guard carries a
//! unique id and only touches its own map entry (`Drop` / [`update`] //! unique id and only touches its own map entry (`Drop` / [`update`]
@ -23,7 +23,6 @@
//! so a re-`set_warning` for the same kind is never clobbered when the //! so a re-`set_warning` for the same kind is never clobbered when the
//! stale guard later drops. //! stale guard later drops.
//! //!
//! [`host_stats::server_warnings`]: crate::host_stats::server_warnings
//! [`update`]: WarningGuard::update //! [`update`]: WarningGuard::update
use std::collections::BTreeMap; use std::collections::BTreeMap;
@ -49,7 +48,7 @@ struct Slot {
} }
/// The one process-global registry. `BTreeMap` (not `HashMap`) so /// The one process-global registry. `BTreeMap` (not `HashMap`) so
/// [`snapshot`] yields a deterministic, kind-sorted banner order across /// [`snapshot`] yields a deterministic, kind-sorted order across
/// renders. /// renders.
fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> { fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> {
static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new(); static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new();
@ -81,11 +80,13 @@ pub fn set_warning(
message: impl Into<String>, message: impl Into<String>,
) -> WarningGuard { ) -> WarningGuard {
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed); let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
let message = message.into();
tracing::warn!(kind, level, %message, "hive warning raised");
lock().insert( lock().insert(
kind, kind,
Slot { Slot {
level, level,
message: message.into(), message,
owner: id, owner: id,
}, },
); );
@ -170,24 +171,24 @@ pub fn snapshot() -> Vec<ServerWarning> {
} }
/// Raise a warning for a one-shot boot-time step that has no periodic /// Raise a warning for a one-shot boot-time step that has no periodic
/// retry to observe a later success and clear the banner via /// retry to observe a later success and clear the warning via
/// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`, /// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`,
/// which `tokio::spawn`s once at hive-c0re startup and never runs again /// which `tokio::spawn`s once at hive-c0re startup and never runs again
/// this process. Intentionally **leaks** the guard for the life of the /// this process. Intentionally **leaks** the guard for the life of the
/// process: the banner clears the next time hive-c0re restarts (a fresh /// process: the warning clears the next time hive-c0re restarts (a fresh
/// process starts with an empty registry) and re-runs the step, which is /// process starts with an empty registry) and re-runs the step, which is
/// exactly when a config/environment fix would take effect anyway. /// exactly when a config/environment fix would take effect anyway.
/// ///
/// Do not use this for anything that runs periodically or can be /// Do not use this for anything that runs periodically or can be
/// retried within the same process — hold the [`WarningGuard`] (or use /// retried within the same process — hold the [`WarningGuard`] (or use
/// [`crate::stats::sweep_health::SweepHealth`]) so a later success can /// [`crate::stats::sweep_health::SweepHealth`]) so a later success can
/// actually clear the banner instead of waiting for a restart. /// actually clear the warning instead of waiting for a restart.
pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) { pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) {
std::mem::forget(set_warning(kind, level, message)); std::mem::forget(set_warning(kind, level, message));
} }
/// RAII handle for one active warning. Drop clears the `kind` from the /// RAII handle for one active warning. Drop clears the `kind` from the
/// banner. Obtained from [`set_warning`]. /// registry. Obtained from [`set_warning`].
#[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"] #[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"]
pub struct WarningGuard { pub struct WarningGuard {
kind: &'static str, kind: &'static str,

15
hive-c0re/src/swarm_agent_status.rs
View file

@ -27,9 +27,9 @@ use crate::stats::sweep_health::{self, SweepHealth};
/// cadence than the hive's own. /// cadence than the hive's own.
pub const PUBLISH_INTERVAL: Duration = crate::swarm_status::PUBLISH_INTERVAL; pub const PUBLISH_INTERVAL: Duration = crate::swarm_status::PUBLISH_INTERVAL;
/// Consecutive failed sweeps before the dashboard banners. Mirrors /// Consecutive failed sweeps before the sweep raises a hive warning. Mirrors
/// [`crate::swarm_status`]'s constant of the same name and purpose. /// [`crate::swarm_status`]'s constant of the same name and purpose.
const FAILURES_BEFORE_BANNER: u32 = 3; const FAILURES_BEFORE_WARNING: u32 = 3;
/// Start the publish loop, if this deployment wired up a swarm queue. /// Start the publish loop, if this deployment wired up a swarm queue.
/// ///
@ -44,8 +44,8 @@ const FAILURES_BEFORE_BANNER: u32 = 3;
/// deploy-event listener does. /// deploy-event listener does.
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) { pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
let Some(hive) = crate::container_view::hive_swarm_names().0 else { let Some(hive) = crate::container_view::hive_swarm_names().0 else {
// Already bannered once by `swarm_status::spawn`, which runs the // Already warned once by `swarm_status::spawn`, which runs the
// identical check — a second banner for the same missing env var // identical check — a second warning for the same missing env var
// would just be the same fact told twice. // would just be the same fact told twice.
return; return;
}; };
@ -55,8 +55,11 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
return; return;
}; };
let mut health = let mut health = SweepHealth::new(
SweepHealth::new("swarm_agent_status_publish", "warn", FAILURES_BEFORE_BANNER); "swarm_agent_status_publish",
"warn",
FAILURES_BEFORE_WARNING,
);
loop { loop {
match publish_all(&client, &hive).await { match publish_all(&client, &hive).await {
Ok(published) => { Ok(published) => {

12
hive-c0re/src/swarm_notices.rs
View file

@ -14,7 +14,7 @@
//! coordinates is a silent no-op (the ordinary case). A hive that cannot //! coordinates is a silent no-op (the ordinary case). A hive that cannot
//! reach the queue loses the swarm's visibility of the notice, not the //! reach the queue loses the swarm's visibility of the notice, not the
//! host's — `warn!` fires on every failed attempt regardless, and the //! host's — `warn!` fires on every failed attempt regardless, and the
//! dashboard banners only after [`FAILURES_BEFORE_BANNER`] consecutive //! raises a hive warning only after [`FAILURES_BEFORE_WARNING`] consecutive
//! misses, the same debounce shape [`crate::swarm_status`] uses and for //! misses, the same debounce shape [`crate::swarm_status`] uses and for
//! the same reason: a `warn!` that fires every call for three weeks is //! the same reason: a `warn!` that fires every call for three weeks is
//! indistinguishable from silence in practice. //! indistinguishable from silence in practice.
@ -24,10 +24,10 @@ use tokio::sync::{Mutex, OnceCell};
use crate::stats::sweep_health::{self, SweepHealth}; use crate::stats::sweep_health::{self, SweepHealth};
/// Consecutive failed publishes before the dashboard banners — same /// Consecutive failed publishes before the sweep raises a hive warning — same
/// value [`crate::swarm_status`] uses and for the same reason: a debounce /// value [`crate::swarm_status`] uses and for the same reason: a debounce
/// against one blip flapping a banner an operator learns to ignore. /// against one blip flapping a warning an operator learns to ignore.
const FAILURES_BEFORE_BANNER: u32 = 3; const FAILURES_BEFORE_WARNING: u32 = 3;
static HEALTH: OnceCell<Mutex<SweepHealth>> = OnceCell::const_new(); static HEALTH: OnceCell<Mutex<SweepHealth>> = OnceCell::const_new();
@ -37,7 +37,7 @@ async fn health() -> tokio::sync::MutexGuard<'static, SweepHealth> {
Mutex::new(SweepHealth::new( Mutex::new(SweepHealth::new(
"swarm_notices_publish", "swarm_notices_publish",
"warn", "warn",
FAILURES_BEFORE_BANNER, FAILURES_BEFORE_WARNING,
)) ))
}) })
.await .await
@ -57,7 +57,7 @@ pub async fn notify(subsystem: &str, key: Option<String>, summary: String, sourc
return; return;
}; };
// Same absent-name condition `swarm_status` bails on — that module // Same absent-name condition `swarm_status` bails on — that module
// already banners it under `swarm_status_config` the first time // already warns it under `swarm_status_config` the first time
// either of us hits it; nothing more to add from here. // either of us hits it; nothing more to add from here.
let Some(hive) = crate::container_view::hive_swarm_names().0 else { let Some(hive) = crate::container_view::hive_swarm_names().0 else {
return; return;

30
hive-c0re/src/swarm_queue.rs
View file

@ -44,15 +44,15 @@ const FIRST_RETRY_DELAY: Duration = Duration::from_secs(5);
/// The longest wait between two attempts. /// The longest wait between two attempts.
const MAX_RETRY_DELAY: Duration = Duration::from_mins(5); const MAX_RETRY_DELAY: Duration = Duration::from_mins(5);
/// The banner kind a failed attempt raises, cleared by the next success. /// The warning kind a failed attempt raises, cleared by the next success.
const BANNER: &str = "swarm_queue_config"; const WARNING_KIND: &str = "swarm_queue_config";
static CLIENT: Retrying<async_nats::Client> = Retrying::new(FIRST_RETRY_DELAY, MAX_RETRY_DELAY); static CLIENT: Retrying<async_nats::Client> = Retrying::new(FIRST_RETRY_DELAY, MAX_RETRY_DELAY);
/// The shared swarm-queue client, or `None` when there is none right now. /// The shared swarm-queue client, or `None` when there is none right now.
/// ///
/// `None` covers "no queue configured" (the ordinary case, logged once at /// `None` covers "no queue configured" (the ordinary case, logged once at
/// `info`), a deployment bug in the coordinates (bannered, and permanent for /// `info`), a deployment bug in the coordinates (warned, and permanent for
/// this process), and an attempt that failed or is still backing off. A /// this process), and an attempt that failed or is still backing off. A
/// caller with `None` should just skip whatever it was about to publish. /// caller with `None` should just skip whatever it was about to publish.
/// Never waits out a backoff: see [`wait_for_client`] for that. /// Never waits out a backoff: see [`wait_for_client`] for that.
@ -95,12 +95,12 @@ enum Outcome<T> {
enum Slot<T> { enum Slot<T> {
Off, Off,
Ready(T), Ready(T),
/// `failures` consecutive attempts have failed. `banner` holds the /// `failures` consecutive attempts have failed. `warning` holds the
/// dashboard warning until the slot leaves this state. /// dashboard warning until the slot leaves this state.
Failing { Failing {
failures: u32, failures: u32,
not_before: Instant, not_before: Instant,
banner: Option<crate::warnings::WarningGuard>, warning: Option<crate::warnings::WarningGuard>,
}, },
} }
@ -163,22 +163,22 @@ impl<T: Clone> Retrying<T> {
let not_before = Instant::now() + delay; let not_before = Instant::now() + delay;
let message = format!("swarm queue is off, retrying in {delay:?}: {e:#}"); let message = format!("swarm queue is off, retrying in {delay:?}: {e:#}");
tracing::warn!(failures, "{message}"); tracing::warn!(failures, "{message}");
// Updating the held guard keeps the banner up without a // Updating the held guard keeps the warning up without a
// moment where it is cleared and re-raised. // moment where it is cleared and re-raised.
let banner = match slot.take() { let warning = match slot.take() {
Some(Slot::Failing { Some(Slot::Failing {
banner: Some(banner), warning: Some(warning),
.. ..
}) => { }) => {
banner.update("warn", message); warning.update("warn", message);
banner warning
} }
_ => crate::warnings::set_warning(BANNER, "warn", message), _ => crate::warnings::set_warning(WARNING_KIND, "warn", message),
}; };
*slot = Some(Slot::Failing { *slot = Some(Slot::Failing {
failures, failures,
not_before, not_before,
banner: Some(banner), warning: Some(warning),
}); });
Outcome::NotBefore(not_before) Outcome::NotBefore(not_before)
} }
@ -204,7 +204,11 @@ async fn connect_once() -> Attempt<async_nats::Client> {
} }
Err(e) => { Err(e) => {
// A deployment bug, which no retry clears. // A deployment bug, which no retry clears.
crate::warnings::set_boot_warning(BANNER, "warn", format!("swarm queue is off: {e:#}")); crate::warnings::set_boot_warning(
WARNING_KIND,
"warn",
format!("swarm queue is off: {e:#}"),
);
return Attempt::Off; return Attempt::Off;
} }
}; };

14
hive-c0re/src/swarm_status.rs
View file

@ -47,18 +47,18 @@ use crate::stats::sweep_health::{self, SweepHealth};
/// other silently re-tunes the swarm's definition of "quiet". /// other silently re-tunes the swarm's definition of "quiet".
pub const PUBLISH_INTERVAL: Duration = Duration::from_mins(1); pub const PUBLISH_INTERVAL: Duration = Duration::from_mins(1);
/// Consecutive failed publishes before the dashboard banners. /// Consecutive failed publishes before the sweep raises a hive warning.
/// ///
/// At [`PUBLISH_INTERVAL`] this is ~3 minutes of genuine failure, so a /// At [`PUBLISH_INTERVAL`] this is ~3 minutes of genuine failure, so a
/// blip does not flap a banner an operator learns to ignore. /// blip does not flap a warning an operator learns to ignore.
const FAILURES_BEFORE_BANNER: u32 = 3; const FAILURES_BEFORE_WARNING: u32 = 3;
/// Start the publish loop, if this deployment wired up a swarm queue. /// Start the publish loop, if this deployment wired up a swarm queue.
/// ///
/// The connect itself is shared with every other swarm-queue consumer in /// The connect itself is shared with every other swarm-queue consumer in
/// this process — see [`crate::swarm_queue`] for why one connection and /// this process — see [`crate::swarm_queue`] for why one connection and
/// not one per consumer, and for where "no queue configured" vs. "queue /// not one per consumer, and for where "no queue configured" vs. "queue
/// configured but unreachable" gets bannered. This function only decides /// configured but unreachable" gets warned. This function only decides
/// whether *status* has anything to offer once a client exists. /// whether *status* has anything to offer once a client exists.
pub fn spawn( pub fn spawn(
coord: std::sync::Arc<crate::coordinator::Coordinator>, coord: std::sync::Arc<crate::coordinator::Coordinator>,
@ -76,7 +76,7 @@ pub fn spawn(
tokio::spawn(async move { tokio::spawn(async move {
let Some(client) = crate::swarm_queue::wait_for_client().await else { let Some(client) = crate::swarm_queue::wait_for_client().await else {
// No queue configured, already logged or bannered by the // No queue configured, already logged or warned by the
// shared connector; nothing left to report here. // shared connector; nothing left to report here.
return; return;
}; };
@ -117,7 +117,7 @@ pub fn spawn(
shutdown.clone(), shutdown.clone(),
)); ));
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER); let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_WARNING);
loop { loop {
match publish(&client, &hive).await { match publish(&client, &hive).await {
Ok(()) => health.record_ok(), Ok(()) => health.record_ok(),
@ -197,7 +197,7 @@ async fn drain_swarm_events(
let mut sub = match client.subscribe(subject).await { let mut sub = match client.subscribe(subject).await {
Ok(sub) => sub, Ok(sub) => sub,
Err(e) => { Err(e) => {
// Warn rather than a boot banner: the hive is fully functional // Warn rather than a boot warning: the hive is fully functional
// without this, it just falls back to learning about knowledge // without this, it just falls back to learning about knowledge
// changes at its next boot. // changes at its next boot.
tracing::warn!( tracing::warn!(

2
hive-c0re/src/workers/crash_watch.rs
View file

@ -149,7 +149,7 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
// The one place this fires: a swarm-level grafana alert // The one place this fires: a swarm-level grafana alert
// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`) // (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`)
// matches this exact message against the log store and pages the // matches this exact message against the log store and pages the
// operator directly — no agent (and no per-hive dashboard banner) // operator directly — no agent (and no per-hive warning)
// in the loop. See `docs/scheduler/observability.md`. // in the loop. See `docs/scheduler/observability.md`.
tracing::warn!(agent = %stopped, "container crash detected"); tracing::warn!(agent = %stopped, "container crash detected");
} }

2
hive-c0re/src/workers/permissions_import.rs
View file

@ -181,7 +181,7 @@ impl ImportOutcome {
matches!(self, Self::NothingPending | Self::Imported) matches!(self, Self::NothingPending | Self::Imported)
} }
/// Why reading is not allowed, for the caller's log and banner. /// Why reading is not allowed, for the caller's log and warning.
#[must_use] #[must_use]
pub fn into_error(self) -> Option<anyhow::Error> { pub fn into_error(self) -> Option<anyhow::Error> {
match self { match self {

4
hive-c0re/src/workers/swarm_limits.rs
View file

@ -24,11 +24,11 @@ use crate::stats::sweep_health::{self, SweepHealth};
pub const READ_INTERVAL: Duration = Duration::from_secs(30); pub const READ_INTERVAL: Duration = Duration::from_secs(30);
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads. /// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
const FAILURES_BEFORE_BANNER: u32 = 3; const FAILURES_BEFORE_WARNING: u32 = 3;
/// Read the caps now and every [`READ_INTERVAL`] until shutdown. /// Read the caps now and every [`READ_INTERVAL`] until shutdown.
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) { pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER); let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_WARNING);
loop { loop {
match sync(&client).await { match sync(&client).await {
Ok(()) => health.record_ok(), Ok(()) => health.record_ok(),

4
hive-c0re/src/workers/swarm_permissions.rs
View file

@ -30,7 +30,7 @@ use crate::stats::sweep_health::{self, SweepHealth};
pub const READ_INTERVAL: Duration = Duration::from_secs(30); pub const READ_INTERVAL: Duration = Duration::from_secs(30);
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads. /// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
const FAILURES_BEFORE_BANNER: u32 = 3; const FAILURES_BEFORE_WARNING: u32 = 3;
/// Send the meta repo's grants once, then read the permissions now and every /// Send the meta repo's grants once, then read the permissions now and every
/// [`READ_INTERVAL`] until shutdown. /// [`READ_INTERVAL`] until shutdown.
@ -40,7 +40,7 @@ pub async fn follow(
hive: String, hive: String,
mut shutdown: tokio::sync::watch::Receiver<bool>, mut shutdown: tokio::sync::watch::Receiver<bool>,
) { ) {
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_BANNER); let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_WARNING);
loop { loop {
let import = super::permissions_import::send_pending(&client, &hive).await; let import = super::permissions_import::send_pending(&client, &hive).await;
let pass = if import.read_allowed() { let pass = if import.read_allowed() {

2
hive-c0re/src/workers/wanted.rs
View file

@ -76,7 +76,7 @@ pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
let client = let client =
match tokio::time::timeout_at(deadline, crate::swarm_queue::wait_for_client()).await { match tokio::time::timeout_at(deadline, crate::swarm_queue::wait_for_client()).await {
// Absent — already reported by the shared connector (an `info` log or // Absent — already reported by the shared connector (an `info` log or
// a `swarm_queue_config` banner). // a `swarm_queue_config` warning).
Ok(None) => return Ok(()), Ok(None) => return Ok(()),
Ok(Some(client)) => client, Ok(Some(client)) => client,
Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()), Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()),

119
nix/host-modules/swarm-grafana.nix
View file

@ -142,6 +142,78 @@ let
annotations = { inherit summary; }; annotations = { inherit summary; };
}; };
# An alert rule that fires while a metric query's latest value is above
# `threshold`. `expr` is an instant MetricsQL query; names with dots are
# selected as `{__name__="a.b"}`, the way the dashboards do.
#
# No data is `OK`: a series that is absent means the condition is not
# raised (hive-c0re stops reporting a warning once it clears), and a hive
# that stopped exporting at all is the log-absence rules' job.
metricThresholdRule =
{
uid,
title,
windowSeconds ? 600,
expr,
threshold,
for,
summary,
}:
{
inherit uid title for;
condition = "C";
data = [
{
refId = "A";
relativeTimeRange = {
from = windowSeconds;
to = 0;
};
inherit datasourceUid;
model = {
refId = "A";
datasource = {
type = "prometheus";
uid = datasourceUid;
};
inherit expr;
instant = true;
range = false;
};
}
{
refId = "B";
datasourceUid = "__expr__";
model = {
refId = "B";
type = "reduce";
expression = "A";
reducer = "last";
};
}
{
refId = "C";
datasourceUid = "__expr__";
model = {
refId = "C";
type = "threshold";
expression = "B";
conditions = [
{
evaluator = {
type = "gt";
params = [ threshold ];
};
}
];
};
}
];
noDataState = "OK";
execErrState = "Error";
annotations = { inherit summary; };
};
# `logCountRule` inverted: fires while the count is below one. `expr` must # `logCountRule` inverted: fires while the count is below one. `expr` must
# be an ungrouped `stats count()`, which answers 0 rather than no row when # be an ungrouped `stats count()`, which answers 0 rather than no row when
# nothing matched; a grouped count drops the group instead, and a dropped # nothing matched; a grouped count drops the group instead, and a dropped
@ -1600,6 +1672,53 @@ in
for = "0s"; for = "0s";
summary = "{{ $labels._HOSTNAME }} detected agent {{ $labels.F_AGENT }}'s container crash (not an operator-initiated stop)"; summary = "{{ $labels._HOSTNAME }} detected agent {{ $labels.F_AGENT }}'s container crash (not an operator-initiated stop)";
}) })
# hive-c0re's own readiness verdict (`hyperhive.hive.degraded`
# in otel_metrics.rs), the same one `/health/ready` gives:
# `1` while any `crit`-level warning is raised. The
# per-kind rule below says which one. `for` rides out a
# restart-time blip.
(metricThresholdRule {
uid = "hh-hive-degraded";
title = "hive degraded";
expr = ''max by (hive) ({__name__="hyperhive.hive.degraded"})'';
threshold = 0;
for = "2m";
summary = "hive {{ $labels.hive }} reports itself degraded: a critical warning is raised; the `hh-hive-warning` alert names it";
})
# One series per raised warning (`hyperhive.hive.warning`),
# so each `kind` is its own alert instance. The text is the
# `hive warning raised` line in the hive-c0re log. Disk
# pressure is left out: it has its own thresholds below, on
# the filesystem metric rather than the registry entry.
# Sweeps raise their warning only after several misses, so
# `for` is short.
(metricThresholdRule {
uid = "hh-hive-warning";
title = "hive warning raised";
expr = ''max by (hive, kind, level) ({__name__="hyperhive.hive.warning", kind!="disk_pressure"})'';
threshold = 0;
for = "2m";
summary = "hive {{ $labels.hive }} has a {{ $labels.level }} warning raised: {{ $labels.kind }}; the message is the `hive warning raised` line in the hive-c0re log";
})
# `hyperhive.hive.nix_store.utilization` is the used fraction of the
# filesystem holding the nix store, the one that failed CI
# with ENOSPC before the GC guardrails.
(metricThresholdRule {
uid = "hh-host-disk-high";
title = "host disk filling";
expr = ''max by (hive) ({__name__="hyperhive.hive.nix_store.utilization"})'';
threshold = 0.85;
for = "10m";
summary = "the nix store's filesystem on hive {{ $labels.hive }} is over 85% full; garbage-collect before it runs out of space";
})
(metricThresholdRule {
uid = "hh-host-disk-critical";
title = "host disk almost full";
expr = ''max by (hive) ({__name__="hyperhive.hive.nix_store.utilization"})'';
threshold = 0.95;
for = "2m";
summary = "the nix store's filesystem on hive {{ $labels.hive }} is over 95% full; garbage-collect now";
})
] ]
# One rule per declared hive rather than one grouped by # One rule per declared hive rather than one grouped by
# `hive`, so a hive that sends nothing at all still has # `hive`, so a hive that sends nothing at all still has

34
nix/module-eval/grafana.nix
View file

@ -498,6 +498,40 @@ let
&& lib.hasInfix ''_msg:="turn failed"'' expr && lib.hasInfix ''_msg:="turn failed"'' expr
&& lib.hasInfix "stats by (hive, agent)" expr; && lib.hasInfix "stats by (hive, agent)" expr;
} }
{
# The hive-health alerts query metric names that hive-c0re spells in
# Rust. A rename on either side provisions cleanly and the rule then
# reads nothing, so it never fires. Each name here is checked against
# the exporter's source rather than a copy of it, and each rule must
# query the metrics datasource with an instant query.
# 🔬 Invert-proof: rename a gauge in `otel_metrics.rs` and this fails.
name = "the hive-health alert rules query metrics hive-c0re exports";
ok =
let
groups =
grafanaHere.containers.swarm-grafana.config.services.grafana.provision.alerting.rules.settings.groups;
rules = lib.concatMap (g: g.rules) groups;
source = builtins.readFile ../../hive-c0re/src/stats/otel_metrics.rs;
expected = {
"hh-hive-degraded" = "hyperhive.hive.degraded";
"hh-hive-warning" = "hyperhive.hive.warning";
"hh-host-disk-high" = "hyperhive.hive.nix_store.utilization";
"hh-host-disk-critical" = "hyperhive.hive.nix_store.utilization";
};
check =
uid: metric:
let
rule = lib.findFirst (r: r.uid == uid) null rules;
query = (lib.head rule.data).model;
in
rule != null
&& lib.hasInfix ''{__name__="${metric}"'' query.expr
&& lib.hasInfix ''"${metric}"'' source
&& query.instant
&& (lib.head rule.data).datasourceUid == "swarm-victoriametrics";
in
lib.all (uid: check uid expected.${uid}) (builtins.attrNames expected);
}
{ {
# 🩸 The regression this guards: moving bao's metrics scrape into its # 🩸 The regression this guards: moving bao's metrics scrape into its
# own in-container collector made that collector stamp every exported # own in-container collector made that collector stamp every exported