hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -444,7 +444,7 @@ whole-swarm spend figure.
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
hive-c0re measures everything above **per agent**, tagged with the hive it
|
||||
runs in. It measures these three per **hive** instead, and they carry no
|
||||
runs in. It measures these five per **hive** instead, and they carry no
|
||||
`agent` label — so a hive
|
||||
that hosts no agents still reports, and "this hive is quiet" is
|
||||
distinguishable from "this hive is gone." Select them with
|
||||
|
|
@ -452,20 +452,41 @@ distinguishable from "this hive is gone." Select them with
|
|||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
|
||||
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
|
||||
|
||||
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
|
||||
visible as zeros rather than as missing series.
|
||||
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
|
||||
included, so a healthy hive is visible as zeros rather than as missing
|
||||
series.
|
||||
|
||||
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
|
||||
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
|
||||
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
|
||||
detail behind it — `warn`-level entries mean "an operator should look" and do
|
||||
**not** set `degraded`.
|
||||
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
|
||||
label is one of a fixed set in the source (for example `disk_pressure`, a
|
||||
failing sweep such as `swarm_status_publish`, `pending_logins`,
|
||||
`forge_ensure_core_user`), and the message lives in the
|
||||
`hive warning raised` line hive-c0re logs when it raises the warning.
|
||||
|
||||
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
|
||||
|
||||
| Alert | Fires when |
|
||||
| ----------------------- | ----------------------------------------------------------------------------------- |
|
||||
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
|
||||
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
|
||||
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
|
||||
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
|
||||
|
||||
The disk alerts replace the old dashboard banner for the nix store, and
|
||||
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
|
||||
own pair so the thresholds sit on the metric rather than on the registry
|
||||
entry.
|
||||
|
||||
Same cadence, transport and resource labels as the container metrics above.
|
||||
|
||||
|
|
|
|||
|
|
@ -148,8 +148,8 @@ swarm-services certificate.
|
|||
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
|
||||
and publishes under `hiveName`, the same name that keys `swarm.hives`.
|
||||
|
||||
If a hive stops reporting, its own dashboard is the place to look: a
|
||||
failure to publish raises a warning banner there after three consecutive
|
||||
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
|
||||
failure to publish raises a hive warning after three consecutive
|
||||
misses. It stays `warn` rather than `crit` on purpose — a hive that
|
||||
can't reach the queue isn't itself unhealthy, so it doesn't start
|
||||
calling itself degraded for being unable to say it's fine.
|
||||
|
|
|
|||
|
|
@ -31,26 +31,17 @@ dashboard tab strip.
|
|||
K3PT ST4T3 (kept-state entries).
|
||||
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
|
||||
— sits below the tab strip.
|
||||
- **Server-warnings banner** — a generic, sticky top-of-page strip shown
|
||||
on **every** page (the dashboard + the stand-alone FL0W
|
||||
and ST4TS pages), injected at the top
|
||||
of `<body>` by `renderServerWarnings` in `common.js`. Driven by
|
||||
`state.server_warnings` — a list of `{ kind, level, message }` — and
|
||||
coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the
|
||||
threshold + message, so adding a new system warning needs no frontend
|
||||
change. Warnings are a push-based registry (`hive-c0re`'s
|
||||
`warnings.rs`): any subsystem raises/clears its own entry via an RAII
|
||||
guard, and `host_stats::server_warnings()` is just a cheap snapshot of
|
||||
that registry. Producers today include the host disk-pressure check
|
||||
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
|
||||
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
|
||||
provisioning/CI-runner boot failures, and an agent-state warning
|
||||
(`pending_logins`) computed alongside the container snapshot. A
|
||||
container crash used to have a sibling `agents_crashing` entry here —
|
||||
removed in favour of a swarm-level grafana alert on the crash
|
||||
watcher's own log line (`docs/scheduler/observability.md`), so a
|
||||
hive with no otel export now has no operator-visible crash signal at
|
||||
all. Hidden when there are no warnings.
|
||||
- **No warning banner.** No page shows hive-level warnings (host disk
|
||||
pressure, failing background sweeps, forge boot failures, agents that
|
||||
need a claude login). They live in a push-based registry
|
||||
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
|
||||
entry through an RAII guard, and each raise logs one
|
||||
`hive warning raised` line. The operator sees the registry as metrics and
|
||||
Grafana alerts: see
|
||||
[Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re).
|
||||
A container crash is a swarm-level Grafana alert on the crash watcher's
|
||||
own log line, so a hive with no otel export has no operator-visible
|
||||
warning signal beyond `/health/ready`.
|
||||
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
||||
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
||||
arrive in the state snapshot.
|
||||
|
|
|
|||
|
|
@ -564,7 +564,7 @@ hive-side-panel .md td {
|
|||
Inverted-gutter layout: the dashboard keeps `<body>` full-bleed
|
||||
and wraps their padded content in a single `.page-content` div that
|
||||
carries the horizontal gutter. Full-width chrome (the tab strip, the
|
||||
server-warnings banner, the footer) then spans edge-to-edge for free,
|
||||
footer) then spans edge-to-edge for free,
|
||||
with no negative-margin breakout hacks. Only matches on pages that opt
|
||||
in by adding the wrapper (index.html); FL0W / ST4TS
|
||||
are full-bleed with their own `.<page>-main` padding. */
|
||||
|
|
@ -573,50 +573,14 @@ hive-side-panel .md td {
|
|||
}
|
||||
|
||||
/* ─── sticky top region ───────────────────────────────────────────────
|
||||
One sticky container per page holding the warning banner above the
|
||||
page chrome (tab bar / page header). common.js (ensureStickyTop)
|
||||
wraps the page's chrome in this on first render and injects the banner
|
||||
as its first child, so the banner + chrome stack in a single sticky
|
||||
context — pinned together — instead of two separate `top:0` stickies
|
||||
colliding (the banner used to overlay the tab bar). */
|
||||
One sticky container per page holding the page chrome (tab bar / page
|
||||
header). common.js (initStickyTop) wraps the chrome in this on load. */
|
||||
.sticky-top {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 50;
|
||||
}
|
||||
|
||||
/* ─── server warnings banner ──────────────────────────────────────────
|
||||
Generic top-of-page strip, injected as the first child of `.sticky-top`
|
||||
by common.js (renderServerWarnings) on every page. One row per warning;
|
||||
colour comes from the per-warning `level` (warn = amber, crit = red).
|
||||
Stickiness lives on the `.sticky-top` wrapper, not here. The bar spans
|
||||
the full width edge-to-edge; the dashboard is full-bleed at the
|
||||
body level (their 1.5em gutter lives on an inner `.page-content`
|
||||
wrapper), so the banner is full-width for free. */
|
||||
.server-warnings[hidden] {
|
||||
display: none;
|
||||
}
|
||||
.server-warn {
|
||||
text-align: center;
|
||||
padding: 0.4em 1em;
|
||||
font-size: 0.85em;
|
||||
font-weight: bold;
|
||||
letter-spacing: 0.02em;
|
||||
}
|
||||
/* Tint composited over the opaque elevated-surface grey (not transparent):
|
||||
the bar is `position: sticky`, so a see-through background would let
|
||||
page content scroll through behind it. */
|
||||
.server-warn-warn {
|
||||
color: var(--amber);
|
||||
background: color-mix(in srgb, var(--amber) 16%, var(--bg-elev));
|
||||
border-bottom: 1px solid var(--amber);
|
||||
}
|
||||
.server-warn-crit {
|
||||
color: var(--red);
|
||||
background: color-mix(in srgb, var(--red) 18%, var(--bg-elev));
|
||||
border-bottom: 1px solid var(--red);
|
||||
}
|
||||
|
||||
/* ─── stats table ─────────────────────────────────────────────────
|
||||
Shared right-aligned numeric table. Used by /stats.html (busiest
|
||||
agents) — moved here from dashboard.css when ST4TS became its own
|
||||
|
|
|
|||
|
|
@ -562,22 +562,12 @@ export function appendLinkified(parent, text, refs) {
|
|||
}
|
||||
}
|
||||
|
||||
// ─── server warnings banner ──────────────────────────────────────────
|
||||
// A generic top-of-page banner shown on every page (dashboard + the
|
||||
// stand-alone FL0W / ST4TS pages). The backend decides what to
|
||||
// warn about — `/api/state.server_warnings` is a list of
|
||||
// `{ kind, level, message }` — and this just renders it, coloured by
|
||||
// `level` (`warn` amber / `crit` red). Adding a new system warning is a
|
||||
// backend-only change. The bar is injected at the top of <body> so no
|
||||
// page needs to add markup.
|
||||
|
||||
// The sticky top region — one sticky container holding the warning
|
||||
// banner above the page's chrome (tab bar / page header), so the banner
|
||||
// stacks with the chrome instead of being overlaid by it (two separate
|
||||
// `top:0` stickies would otherwise collide). Built once by wrapping the
|
||||
// page's existing chrome element; pages without a chrome (none today)
|
||||
// get a banner-only sticky region at the top of <body>.
|
||||
function ensureStickyTop() {
|
||||
// ─── sticky top region ───────────────────────────────────────────────
|
||||
// One sticky container wrapping the page's chrome (tab bar / page
|
||||
// header), built once by wrapping the existing chrome element. A page
|
||||
// without a chrome (none today) gets an empty sticky region at the top of
|
||||
// <body>.
|
||||
export function initStickyTop() {
|
||||
let top = document.querySelector(".sticky-top");
|
||||
if (top) return top;
|
||||
top = document.createElement("div");
|
||||
|
|
@ -591,51 +581,3 @@ function ensureStickyTop() {
|
|||
}
|
||||
return top;
|
||||
}
|
||||
|
||||
function ensureServerWarningsBar() {
|
||||
let bar = document.getElementById("server-warnings");
|
||||
if (!bar) {
|
||||
bar = document.createElement("div");
|
||||
bar.id = "server-warnings";
|
||||
bar.className = "server-warnings";
|
||||
bar.setAttribute("role", "alert");
|
||||
bar.hidden = true;
|
||||
ensureStickyTop().prepend(bar);
|
||||
}
|
||||
return bar;
|
||||
}
|
||||
|
||||
/// Render a `server_warnings` list (from /api/state) into the banner.
|
||||
/// Empty / missing → the bar hides itself.
|
||||
export function renderServerWarnings(warnings) {
|
||||
const bar = ensureServerWarningsBar();
|
||||
bar.replaceChildren();
|
||||
if (!Array.isArray(warnings) || warnings.length === 0) {
|
||||
bar.hidden = true;
|
||||
return;
|
||||
}
|
||||
for (const w of warnings) {
|
||||
const row = el("div", {
|
||||
class:
|
||||
"server-warn server-warn-" +
|
||||
(w && w.level === "crit" ? "crit" : "warn"),
|
||||
});
|
||||
appendText(row, "⚠ " + ((w && w.message) || ""));
|
||||
bar.append(row);
|
||||
}
|
||||
bar.hidden = false;
|
||||
}
|
||||
|
||||
/// One-shot init for pages that don't otherwise poll /api/state: ensure
|
||||
/// the bar exists, fetch the snapshot once, render. The dashboard (which
|
||||
/// already polls /api/state) calls `renderServerWarnings` directly for
|
||||
/// live updates instead.
|
||||
export function initServerWarnings() {
|
||||
ensureServerWarningsBar();
|
||||
fetch("/api/state")
|
||||
.then((r) => (r.ok ? r.json() : null))
|
||||
.then((s) => renderServerWarnings(s && s.server_warnings))
|
||||
.catch(() => {
|
||||
/* non-fatal: no banner if the snapshot is unreachable */
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
body.dashboard-shell {
|
||||
/* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives
|
||||
on the inner `.page-content` wrapper (see common.css), so the sticky
|
||||
chrome bar, the server-warnings banner, and the footer span the full
|
||||
chrome bar and the footer span the full
|
||||
width edge-to-edge with no negative-margin breakout. `padding-bottom`
|
||||
keeps a little breathing room above the viewport edge (overridden to
|
||||
clear the selection bar via `.has-selection` below). No max-width cap
|
||||
|
|
@ -34,8 +34,7 @@ body.dashboard-shell {
|
|||
align-items: center;
|
||||
gap: 0.75em;
|
||||
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
|
||||
common.js wraps this chrome in, alongside the server-warning banner,
|
||||
so they stack in one sticky context instead of colliding. */
|
||||
common.js wraps this chrome in. */
|
||||
background: color-mix(in srgb, var(--bg) 86%, transparent);
|
||||
-webkit-backdrop-filter: blur(8px) saturate(120%);
|
||||
backdrop-filter: blur(8px) saturate(120%);
|
||||
|
|
|
|||
|
|
@ -10,17 +10,12 @@
|
|||
// container list).
|
||||
|
||||
import { create as termCreate } from "@hive/shared/terminal.js";
|
||||
import {
|
||||
$,
|
||||
appendLinkified,
|
||||
openStream,
|
||||
initServerWarnings,
|
||||
} from "./common.js";
|
||||
import { $, appendLinkified, openStream, initStickyTop } from "./common.js";
|
||||
import { el } from "@hive/shared/dom.js";
|
||||
import { epochSec } from "./util.js";
|
||||
|
||||
(() => {
|
||||
initServerWarnings();
|
||||
initStickyTop();
|
||||
|
||||
// ─── local containers cache (for compose autocomplete) ──────────────────
|
||||
// The compose box's @-mention completion suggests known agent names.
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
// this bundle has no chart lib; per-agent trend charts live on each
|
||||
// agent's own /stats page. The window selector is a hash-routed
|
||||
// createTabStrip (#1h / #24h / …), matching the per-agent stats page.
|
||||
import { $, initServerWarnings } from "./common.js";
|
||||
import { $, initStickyTop } from "./common.js";
|
||||
import { createTabStrip } from "@hive/shared/tabs.js";
|
||||
|
||||
let hiveStatsWindow = "24h";
|
||||
|
|
@ -173,7 +173,7 @@ async function refreshHiveStats() {
|
|||
}
|
||||
}
|
||||
|
||||
initServerWarnings();
|
||||
initStickyTop();
|
||||
|
||||
// Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes
|
||||
// here (the selector swaps data, not DOM); paneFor() returns null for each
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@
|
|||
// 4G3NTS (containers) domain lives in `./agents.js`.
|
||||
|
||||
import { marked } from "marked";
|
||||
import { $, openStream, renderServerWarnings } from "./common.js";
|
||||
import { $, openStream, initStickyTop } from "./common.js";
|
||||
import { el } from "@hive/shared/dom.js";
|
||||
import { bindAsyncForms } from "@hive/shared/forms.js";
|
||||
import { createTabStrip } from "@hive/shared/tabs.js";
|
||||
|
|
@ -50,6 +50,8 @@ import { renderLinksMenu } from "./links-menu.js";
|
|||
window.marked = marked;
|
||||
|
||||
(() => {
|
||||
initStickyTop();
|
||||
|
||||
// ─── async forms ────────────────────────────────────────────────────────
|
||||
// Shared `data-async` submit interceptor (in common.js, shared with the other
|
||||
// pages' bundles). On success it re-runs refreshState unless the form
|
||||
|
|
@ -135,7 +137,6 @@ window.marked = marked;
|
|||
// synchronous read (e.g. the compose autocomplete pulls agent
|
||||
// names from here instead of refetching on every keystroke).
|
||||
window.__hyperhive_state = s;
|
||||
renderServerWarnings(s.server_warnings);
|
||||
// Hive identity: render the swarm/hive name as a headline at the
|
||||
// top of the 4G3NTS pane + update the page title once the
|
||||
// server-side display names are known. `hive_name` / `swarm_name`
|
||||
|
|
|
|||
|
|
@ -16,8 +16,7 @@
|
|||
*/
|
||||
.page-header {
|
||||
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
|
||||
common.js wraps this header in, alongside the server-warning banner,
|
||||
so the two stack instead of colliding. */
|
||||
common.js wraps this header in. */
|
||||
background: color-mix(in srgb, var(--bg) 92%, transparent);
|
||||
-webkit-backdrop-filter: blur(8px) saturate(120%);
|
||||
backdrop-filter: blur(8px) saturate(120%);
|
||||
|
|
|
|||
|
|
@ -599,6 +599,7 @@ impl Coordinator {
|
|||
return;
|
||||
}
|
||||
};
|
||||
crate::host_stats::refresh_pending_logins(&fresh);
|
||||
let mut last = self.last_containers.lock().await;
|
||||
let mut changed_or_new = Vec::new();
|
||||
let mut removed = Vec::new();
|
||||
|
|
|
|||
|
|
@ -1,9 +1,8 @@
|
|||
//! Hive-wide health endpoints for external monitoring: `/health/live`
|
||||
//! (liveness) and `/health/ready` (readiness).
|
||||
//!
|
||||
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the
|
||||
//! dashboard banner already reads — as the single source of truth for
|
||||
//! "is anything degraded", rather than tracking health separately. Two
|
||||
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
|
||||
//! for "is anything degraded", rather than tracking health separately. Two
|
||||
//! systems independently deciding what counts as unhealthy is how they
|
||||
//! end up disagreeing.
|
||||
//!
|
||||
|
|
@ -12,7 +11,7 @@
|
|||
//! the *process* down over a transient dependency failure — that's what
|
||||
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
|
||||
//! warning currently set; `warn`-level entries ride along in the body for
|
||||
//! detail but don't flip the status, mirroring the banner's own severity
|
||||
//! detail but don't flip the status, mirroring the warning registry's own severity
|
||||
//! split.
|
||||
//!
|
||||
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can
|
||||
|
|
|
|||
|
|
@ -102,12 +102,6 @@ pub(super) struct StateSnapshot {
|
|||
/// var, set from `services.hyperhive.swarm.name`. `None` when
|
||||
/// unset — chrome omits the swarm segment of the breadcrumb.
|
||||
swarm_name: Option<String>,
|
||||
/// Server-level warnings for the dashboard's top-of-page banner
|
||||
/// (currently host disk-pressure; more producers can be added
|
||||
/// backend-side). Empty when all clear. Built by
|
||||
/// `host_stats::server_warnings`; the frontend renders this list
|
||||
/// generically, so new warning kinds need no frontend change.
|
||||
server_warnings: Vec<crate::host_stats::ServerWarning>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
|
|
@ -193,15 +187,6 @@ pub(super) async fn api_state(
|
|||
let tombstones = build_tombstone_views(&containers);
|
||||
let port_conflicts = build_port_conflicts(&containers);
|
||||
|
||||
// Banner warnings: host probes (disk) + agent-state (pending logins).
|
||||
// Built before the response struct because the agent-state producer
|
||||
// borrows `containers`, which moves in below.
|
||||
let server_warnings = {
|
||||
let mut w = crate::host_stats::server_warnings();
|
||||
w.extend(crate::host_stats::agent_state_warnings(&containers));
|
||||
w
|
||||
};
|
||||
|
||||
axum::Json(StateSnapshot {
|
||||
seq,
|
||||
hostname,
|
||||
|
|
@ -233,7 +218,6 @@ pub(super) async fn api_state(
|
|||
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty()),
|
||||
server_warnings,
|
||||
})
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
|
|||
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
|
||||
/// most a handful per process). [`crate::warnings::set_boot_warning`]
|
||||
/// requires a `'static` kind so distinct orgs/repos don't clobber each
|
||||
/// other's banner entry; leaking a few short strings once per boot is
|
||||
/// other's warning entry; leaking a few short strings once per boot is
|
||||
/// cheap and bounded, unlike a per-request or per-loop-iteration leak.
|
||||
fn static_kind(s: String) -> &'static str {
|
||||
Box::leak(s.into_boxed_str())
|
||||
|
|
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
|
|||
/// confirms the container exists — not that Forgejo is *listening*. A
|
||||
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
|
||||
/// without this gate every `ensure_*` fires at a refused socket and leaves a
|
||||
/// stale "provisioning failed" banner that never clears (the pass is one-shot).
|
||||
/// stale "provisioning failed" warning that never clears (the pass is one-shot).
|
||||
/// Returns whether the endpoint became ready before the deadline; on timeout
|
||||
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real
|
||||
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
|
||||
|
|
@ -393,7 +393,7 @@ pub async fn ensure_all() {
|
|||
}
|
||||
// The container exists, but its HTTP may not be up yet (a rebuild restarts
|
||||
// hive-forge + hive-c0re together). Wait for the API to answer before the
|
||||
// provisioning pass so a boot race doesn't leave stale failure banners.
|
||||
// provisioning pass so a boot race doesn't leave stale failure warnings.
|
||||
wait_until_ready().await;
|
||||
let core_token = match ensure_core_user_and_token().await {
|
||||
Ok(t) => Some(t),
|
||||
|
|
|
|||
|
|
@ -125,7 +125,7 @@ pub async fn push_meta(dir: &Path) -> Result<()> {
|
|||
let msg = "forge/core/meta push rejected (non-fast-forward): \
|
||||
remote is ahead of local; leaving remote history intact";
|
||||
tracing::warn!("{msg}");
|
||||
// Raise a persistent dashboard banner; cleared on next successful push.
|
||||
// Raise a persistent hive warning; cleared on next successful push.
|
||||
if let Ok(mut g) = META_NON_FF_GUARD.lock() {
|
||||
*g = Some(crate::warnings::set_warning(
|
||||
"forge-meta-non-ff",
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
|
|||
/// Minimum gap between retry attempts after a reload failure (30 s).
|
||||
const RELOAD_RETRY_SECS: u64 = 30;
|
||||
|
||||
/// Dashboard-banner health tracker for the reload sweep — raises a
|
||||
/// Hive-warning health tracker for the reload sweep — raises a
|
||||
/// `gateway_nginx_reload` warning the first time a reload fails (routing is
|
||||
/// broken for whoever depends on the change *right now*, so no debounce
|
||||
/// window) and clears it the moment a reload succeeds again.
|
||||
|
|
|
|||
|
|
@ -139,7 +139,7 @@ pub(super) async fn run_node(
|
|||
|
||||
/// Boot-time forge user/token sweep as a DAG node — see
|
||||
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
|
||||
/// per-step error handling and boot-warning banners internally (it's
|
||||
/// per-step error handling and boot warnings internally (it's
|
||||
/// best-effort by design), so this wrapper has nothing left to report;
|
||||
/// it exists purely to make the sweep a visible unit of work.
|
||||
async fn run_forge_sweep() -> Result<()> {
|
||||
|
|
@ -149,7 +149,7 @@ async fn run_forge_sweep() -> Result<()> {
|
|||
|
||||
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
|
||||
/// Every pass reports here, whoever submitted it: as the node's own error on
|
||||
/// the dashboard, and into the debounced banner.
|
||||
/// the dashboard, and into the debounced warning.
|
||||
async fn run_matrix_sweep() -> Result<()> {
|
||||
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
|
||||
.await
|
||||
|
|
@ -164,7 +164,7 @@ async fn run_matrix_sweep() -> Result<()> {
|
|||
health.record_ok();
|
||||
Ok(())
|
||||
} else {
|
||||
health.record_err(matrix_sweep_banner);
|
||||
health.record_err(matrix_sweep_warning);
|
||||
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
|
||||
}
|
||||
}
|
||||
|
|
@ -177,7 +177,7 @@ async fn run_matrix_sweep() -> Result<()> {
|
|||
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
|
||||
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
|
||||
|
||||
/// Banner for a matrix sweep that keeps failing. Two misses in a row, so one
|
||||
/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one
|
||||
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
|
||||
/// dashboard; cleared by the next clean pass.
|
||||
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
|
||||
|
|
@ -185,7 +185,7 @@ fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
|
|||
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
|
||||
}
|
||||
|
||||
fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
|
||||
fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
|
||||
let age = ctx.since_last_ok.map_or_else(
|
||||
|| "no success this session".to_owned(),
|
||||
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
|
||||
|
|
@ -200,7 +200,7 @@ fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
|
|||
|
||||
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
|
||||
/// pass reports here, whoever submitted it: as the node's own error on the
|
||||
/// dashboard, and into the debounced banner.
|
||||
/// dashboard, and into the debounced warning.
|
||||
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||
let result = tokio::time::timeout(
|
||||
KNOWLEDGE_PULL_DEADLINE,
|
||||
|
|
@ -242,7 +242,7 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
|||
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
|
||||
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
|
||||
|
||||
/// Banner for a `/knowledge` pull that keeps failing: three misses in a row,
|
||||
/// Warning for a `/knowledge` pull that keeps failing: three misses in a row,
|
||||
/// cleared by the next successful pull.
|
||||
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
|
||||
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();
|
||||
|
|
|
|||
|
|
@ -367,11 +367,11 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
|
|||
agents_for_meta(None).await
|
||||
}
|
||||
|
||||
/// Per-agent "stop failed" banner guards, keyed by agent name. Cleared the
|
||||
/// Per-agent "stop failed" warning guards, keyed by agent name. Cleared the
|
||||
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
|
||||
/// rather than a bare `set_boot_warning` because this condition (unlike a
|
||||
/// one-shot boot step) genuinely resolves later, once an operator has
|
||||
/// intervened, and the banner should clear with it instead of surviving
|
||||
/// intervened, and the warning should clear with it instead of surviving
|
||||
/// until the next hive-c0re restart.
|
||||
fn stop_failed_guards()
|
||||
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
|
||||
|
|
@ -402,7 +402,7 @@ pub async fn kill(name: &str) -> Result<()> {
|
|||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
match result {
|
||||
Ok(()) => {
|
||||
// A clean stop clears any earlier "this looked wedged" banner
|
||||
// A clean stop clears any earlier "this looked wedged" warning
|
||||
// for the same agent.
|
||||
guards.remove(name);
|
||||
Ok(())
|
||||
|
|
|
|||
|
|
@ -284,12 +284,12 @@ async fn cmd_serve(
|
|||
}
|
||||
}
|
||||
});
|
||||
// Disk-pressure watch: raise a `disk_pressure` banner warning while the
|
||||
// Disk-pressure watch: raise a `disk_pressure` warning while the
|
||||
// host nix store is over threshold and clear it when back under, via the
|
||||
// push-based warnings registry. Replaces the old per-`/api/state`
|
||||
// `statvfs`. ~60s cadence; first tick fires immediately. The held guard
|
||||
// lives for the task's lifetime — dropping it (on shutdown) clears the
|
||||
// banner.
|
||||
// warning.
|
||||
let mut disk_shutdown = coord.shutdown_rx();
|
||||
tokio::spawn(async move {
|
||||
let mut disk_guard: Option<warnings::WarningGuard> = None;
|
||||
|
|
|
|||
|
|
@ -787,8 +787,8 @@ pub async fn invite_user(
|
|||
/// Returns `true` when every step of the sweep succeeded, `false` when at
|
||||
/// least one step failed — the caller feeds this into a
|
||||
/// [`crate::stats::sweep_health::SweepHealth`] to raise a debounced
|
||||
/// dashboard banner on persistent failure (this sweep re-runs every 30
|
||||
/// minutes, so a one-off blip self-heals without ever bannering).
|
||||
/// hive warning on persistent failure (this sweep re-runs every 30
|
||||
/// minutes, so a one-off blip self-heals without ever warning).
|
||||
pub async fn ensure_all() -> bool {
|
||||
if !is_present() {
|
||||
tracing::debug!("matrix: hive-matrix container absent, skipping user sweep");
|
||||
|
|
|
|||
|
|
@ -1,35 +1,33 @@
|
|||
//! Host-system probes and the derived **server-warning** list that the
|
||||
//! dashboard renders as a top-of-page banner.
|
||||
//! Host-system probes that raise hive-level warnings into the push-based
|
||||
//! [`crate::warnings`] registry.
|
||||
//!
|
||||
//! The first (and currently only) producer is a host disk-usage check:
|
||||
//! the nix store filling up is what ENOSPC-failed CI before the GC
|
||||
//! guardrails were documented. hive-c0re runs on the host (not inside a
|
||||
//! container), so it can `statvfs` the store path directly and warn the
|
||||
//! operator *before* an ENOSPC, not after.
|
||||
//! The first producer is a host disk-usage check: the nix store filling up
|
||||
//! is what ENOSPC-failed CI before the GC guardrails were documented.
|
||||
//! hive-c0re runs on the host (not inside a container), so it can `statvfs`
|
||||
//! the store path directly and warn the operator *before* an ENOSPC, not
|
||||
//! after. [`refresh_disk_warning`] drives a held guard from a periodic task
|
||||
//! (see `spawn` in `main.rs`). The same probe backs the
|
||||
//! `hyperhive.hive.nix_store.utilization` gauge ([`disk_utilization`]), which is
|
||||
//! what the early-warning Grafana alerts read.
|
||||
//!
|
||||
//! [`server_warnings`] is the public surface: it returns a flat list of
|
||||
//! [`ServerWarning`]s for `/api/state`. It is now just a snapshot of the
|
||||
//! push-based [`crate::warnings`] registry — producers raise/clear their
|
||||
//! own warnings via an RAII guard rather than being re-probed on every
|
||||
//! render. The disk-pressure check is reworked onto that model here:
|
||||
//! [`refresh_disk_warning`] drives a held guard from a periodic task (see
|
||||
//! `spawn` in `main.rs`) instead of a `statvfs` per `/api/state`.
|
||||
//! The registry has no dashboard banner any more. A raised warning reaches
|
||||
//! the operator as the `hyperhive.hive.warning` gauge and a Grafana alert
|
||||
//! on it (see `docs/scheduler/observability.md`).
|
||||
|
||||
use serde::Serialize;
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use crate::container_view::ContainerView;
|
||||
|
||||
/// One server-level warning for the dashboard's top-of-page banner.
|
||||
/// One hive-level warning in the [`crate::warnings`] registry.
|
||||
#[derive(Debug, Clone, Serialize, ToSchema)]
|
||||
pub struct ServerWarning {
|
||||
/// Stable kind id (e.g. `"disk_pressure"`) — lets the frontend dedupe
|
||||
/// or special-case without parsing the message.
|
||||
/// Stable kind id (e.g. `"disk_pressure"`), the `kind` label of the
|
||||
/// `hyperhive.hive.warning` gauge.
|
||||
pub kind: &'static str,
|
||||
/// Severity: `"warn"` (amber) or `"crit"` (red). The banner picks its
|
||||
/// colour from this; everything else is just the message text.
|
||||
/// Severity: `"warn"` or `"crit"`. `crit` degrades readiness.
|
||||
pub level: &'static str,
|
||||
/// Human-readable, already-formatted message shown in the banner.
|
||||
/// Human-readable, already-formatted message.
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
|
|
@ -38,17 +36,7 @@ pub struct ServerWarning {
|
|||
const DISK_WARN_PCT: f64 = 85.0;
|
||||
const DISK_CRIT_PCT: f64 = 95.0;
|
||||
|
||||
/// Snapshot the current server-level warnings for the dashboard banner —
|
||||
/// a cheap read of the push-based [`crate::warnings`] registry. Producers
|
||||
/// (the disk watch below, background converge sweeps, …) raise and clear
|
||||
/// their own entries via [`crate::warnings::set_warning`]; nothing is
|
||||
/// re-probed here.
|
||||
#[must_use]
|
||||
pub fn server_warnings() -> Vec<ServerWarning> {
|
||||
crate::warnings::snapshot()
|
||||
}
|
||||
|
||||
/// Pure disk-pressure decision: given usage, return the banner
|
||||
/// Pure disk-pressure decision: given usage, return the warning
|
||||
/// `(level, message)` or `None` when below the warn threshold. Split from
|
||||
/// the probe + guard plumbing so the threshold + formatting is unit-tested
|
||||
/// without a real `statvfs`.
|
||||
|
|
@ -75,9 +63,8 @@ fn disk_pressure_warning(used_pct: f64, free_bytes: u64) -> Option<(&'static str
|
|||
|
||||
/// One tick of the disk-pressure watch: probe the store and reconcile the
|
||||
/// held warning `guard` — raise or refresh it while usage is over
|
||||
/// threshold, drop it (clearing the banner) once back under. Driven by the
|
||||
/// periodic disk-watch task spawned in `main.rs`, replacing the old
|
||||
/// per-render `statvfs` in [`server_warnings`].
|
||||
/// threshold, drop it (clearing the warning) once back under. Driven by the
|
||||
/// periodic disk-watch task spawned in `main.rs`.
|
||||
pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
|
||||
let decision = nix_disk_usage().and_then(|d| disk_pressure_warning(d.used_pct, d.free_bytes));
|
||||
match decision {
|
||||
|
|
@ -96,19 +83,44 @@ pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
|
|||
}
|
||||
}
|
||||
|
||||
/// Agent-state warnings derived from the live container snapshot the
|
||||
/// dashboard already holds: agents that need a claude login. Kept
|
||||
/// separate from [`server_warnings`] (host probes) because the caller
|
||||
/// owns the container list; the dashboard concatenates both into one
|
||||
/// banner list.
|
||||
/// Fraction (0–1) of the filesystem holding the nix store that is in use,
|
||||
/// the value behind the `hyperhive.hive.nix_store.utilization` gauge. `None` when
|
||||
/// the probe fails, so a failing probe reports no sample rather than zero.
|
||||
#[must_use]
|
||||
pub fn disk_utilization() -> Option<f64> {
|
||||
nix_disk_usage().map(|d| d.used_pct / 100.0)
|
||||
}
|
||||
|
||||
/// The held `pending_logins` warning; see [`refresh_pending_logins`].
|
||||
static PENDING_LOGINS: std::sync::Mutex<Option<crate::warnings::WarningGuard>> =
|
||||
std::sync::Mutex::new(None);
|
||||
|
||||
/// Reconcile the `pending_logins` warning with the current container
|
||||
/// snapshot: raise or refresh it while any agent needs a claude login,
|
||||
/// drop it once none does. Driven by the container rescan in the
|
||||
/// coordinator, which already builds the snapshot every poll.
|
||||
pub fn refresh_pending_logins(containers: &[ContainerView]) {
|
||||
let warning = agent_state_warnings(containers).into_iter().next();
|
||||
let mut guard = PENDING_LOGINS
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
match (warning, guard.as_ref()) {
|
||||
(Some(w), Some(g)) => g.update(w.level, w.message),
|
||||
(Some(w), None) => {
|
||||
*guard = Some(crate::warnings::set_warning(w.kind, w.level, w.message));
|
||||
}
|
||||
(None, _) => *guard = None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Agent-state warnings derived from the live container snapshot: agents
|
||||
/// that need a claude login. Takes the container list from its caller
|
||||
/// because that is where the snapshot lives.
|
||||
///
|
||||
/// Container crashes used to have a sibling `agents_crashing` entry here,
|
||||
/// fed by `Coordinator::recent_crash_counts` — removed in favour of a
|
||||
/// swarm-level grafana alert on the crash-watcher's own log line (see
|
||||
/// `nix/host-modules/swarm-grafana.nix`'s `hh-container-crashed` rule and
|
||||
/// `docs/scheduler/observability.md`). That ships the same signal to the
|
||||
/// operator without needing an agent (or this per-hive banner) in the
|
||||
/// loop at all.
|
||||
/// Container crashes are not a warning here: the crash-watcher logs
|
||||
/// `container crash detected` and a swarm-level grafana alert
|
||||
/// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`) matches
|
||||
/// that line, with no agent in the loop.
|
||||
#[must_use]
|
||||
pub fn agent_state_warnings(containers: &[ContainerView]) -> Vec<ServerWarning> {
|
||||
let mut out = Vec::new();
|
||||
|
|
@ -263,4 +275,46 @@ mod tests {
|
|||
let w = agent_state_warnings(&containers);
|
||||
assert!(w[0].message.starts_with("1 agent needs claude login: solo"));
|
||||
}
|
||||
|
||||
/// The registry entry tracks the snapshot it is refreshed from: raised
|
||||
/// when an agent needs a login, rewritten when the set changes, and gone
|
||||
/// when none does. This is the only test that touches `pending_logins`,
|
||||
/// so reading the process-global registry by that kind stays sound under
|
||||
/// the parallel test runner.
|
||||
#[test]
|
||||
fn pending_logins_warning_follows_the_container_snapshot() {
|
||||
let current = || {
|
||||
crate::warnings::snapshot()
|
||||
.into_iter()
|
||||
.find(|w| w.kind == "pending_logins")
|
||||
};
|
||||
assert!(
|
||||
current().is_none(),
|
||||
"nothing raised before the first refresh"
|
||||
);
|
||||
|
||||
refresh_pending_logins(&[cv("amy", true)]);
|
||||
assert!(
|
||||
current().is_some_and(|w| w.message.starts_with("1 agent needs claude login: amy")),
|
||||
"an agent needing a login raises the warning"
|
||||
);
|
||||
|
||||
refresh_pending_logins(&[cv("amy", true), cv("bob", true)]);
|
||||
assert!(
|
||||
current().is_some_and(|w| w.message.starts_with("2 agents need claude login")),
|
||||
"a second agent rewrites the same entry"
|
||||
);
|
||||
|
||||
refresh_pending_logins(&[cv("amy", false), cv("bob", false)]);
|
||||
assert!(current().is_none(), "no agent needing a login clears it");
|
||||
}
|
||||
|
||||
/// The gauge value is a fraction, so a percent leaking through would put
|
||||
/// every alert threshold 100x off.
|
||||
#[test]
|
||||
fn disk_utilization_is_a_fraction() {
|
||||
if let Some(used) = disk_utilization() {
|
||||
assert!((0.0..=1.0).contains(&used), "got {used}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -305,6 +305,41 @@ fn register_hive_instruments(meter: &opentelemetry::metrics::Meter) {
|
|||
}
|
||||
})
|
||||
.build();
|
||||
|
||||
// One series per raised warning, `1` while it is raised, labelled with
|
||||
// its `kind` and `level`. This is what names the cause in an alert: the
|
||||
// count above says how many, this says which. A cleared warning stops
|
||||
// reporting rather than reporting `0`, so an alert on it resolves when
|
||||
// the series goes stale; the kinds are a fixed set in the source, so the
|
||||
// label has bounded cardinality.
|
||||
meter
|
||||
.u64_observable_gauge("hyperhive.hive.warning")
|
||||
.with_unit("1")
|
||||
.with_callback(|obs| {
|
||||
for w in warnings::snapshot() {
|
||||
obs.observe(
|
||||
1,
|
||||
&[
|
||||
KeyValue::new("kind", w.kind),
|
||||
KeyValue::new("level", w.level),
|
||||
],
|
||||
);
|
||||
}
|
||||
})
|
||||
.build();
|
||||
|
||||
// Fraction of the nix store's filesystem in use, by the semconv name.
|
||||
// Probed in the callback (a `statvfs` is cheap and this runs once an
|
||||
// export interval); a failed probe reports no sample.
|
||||
meter
|
||||
.f64_observable_gauge("hyperhive.hive.nix_store.utilization")
|
||||
.with_unit("1")
|
||||
.with_callback(|obs| {
|
||||
if let Some(used) = crate::host_stats::disk_utilization() {
|
||||
obs.observe(used, HIVE_ATTRS);
|
||||
}
|
||||
})
|
||||
.build();
|
||||
}
|
||||
|
||||
/// Attributes for the hive-scoped family: **none**.
|
||||
|
|
|
|||
|
|
@ -5,14 +5,14 @@
|
|||
//! gateway nginx writes, per-agent branch protection, …) historically
|
||||
//! `warn!`'d to the journal and moved on, so a persistent failure was
|
||||
//! invisible to the operator. [`SweepHealth`] routes that failure to the
|
||||
//! dashboard banner instead — but *debounced*, per the routing guidance in
|
||||
//! hive warning instead — but *debounced*, per the routing guidance in
|
||||
//! the tracker issue:
|
||||
//!
|
||||
//! - a transient miss (container down mid-sweep, a network blip) shouldn't
|
||||
//! flap a banner on the first failure → `threshold > 1` raises only after
|
||||
//! flap a warning on the first failure → `threshold > 1` raises only after
|
||||
//! that many **consecutive** misses;
|
||||
//! - a won't-fix-itself failure (missing team, bad config, auth rejection)
|
||||
//! should banner immediately → `threshold == 1`.
|
||||
//! should warning immediately → `threshold == 1`.
|
||||
//!
|
||||
//! The warning clears the moment the sweep next succeeds. A sweep owns one
|
||||
//! `SweepHealth`, calls [`record_ok`](SweepHealth::record_ok) on success and
|
||||
|
|
@ -24,11 +24,11 @@ use std::time::{Duration, Instant};
|
|||
use crate::warnings::{WarningGuard, set_warning};
|
||||
|
||||
/// Per-sweep failure context handed to the [`record_err`](SweepHealth::record_err)
|
||||
/// message builder so the banner text can include how long the sweep has been
|
||||
/// message builder so the warning text can include how long the sweep has been
|
||||
/// failing and since when it last worked.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct SweepFailure {
|
||||
/// Number of consecutive failures so far (≥ `threshold` when the banner
|
||||
/// Number of consecutive failures so far (≥ `threshold` when the warning
|
||||
/// is raised).
|
||||
pub consecutive: u32,
|
||||
/// Time since the last recorded success, if the sweep has ever succeeded
|
||||
|
|
@ -36,7 +36,7 @@ pub struct SweepFailure {
|
|||
pub since_last_ok: Option<Duration>,
|
||||
}
|
||||
|
||||
/// A debounced link between one background sweep and the warning banner.
|
||||
/// A debounced link between one background sweep and the warning warning.
|
||||
/// Not `Sync`-shared: each sweep task owns its own instance (the underlying
|
||||
/// registry is the shared, thread-safe part).
|
||||
pub struct SweepHealth {
|
||||
|
|
@ -50,8 +50,8 @@ pub struct SweepHealth {
|
|||
|
||||
impl SweepHealth {
|
||||
/// Track sweep `kind` at severity `level` (`"warn"` / `"crit"`), raising
|
||||
/// the banner after `threshold` consecutive failures. `threshold` is
|
||||
/// floored at 1 (0 would banner before any failure).
|
||||
/// the warning after `threshold` consecutive failures. `threshold` is
|
||||
/// floored at 1 (0 would warning before any failure).
|
||||
#[must_use]
|
||||
pub fn new(kind: &'static str, level: &'static str, threshold: u32) -> Self {
|
||||
Self {
|
||||
|
|
@ -65,7 +65,7 @@ impl SweepHealth {
|
|||
}
|
||||
|
||||
/// Record a successful sweep: reset the consecutive-miss counter and drop
|
||||
/// the guard (clearing any banner entry).
|
||||
/// the guard (clearing any warning entry).
|
||||
pub fn record_ok(&mut self) {
|
||||
self.misses = 0;
|
||||
self.last_ok = Some(Instant::now());
|
||||
|
|
@ -74,9 +74,9 @@ impl SweepHealth {
|
|||
}
|
||||
|
||||
/// Record a failed sweep. Bumps the consecutive-miss counter; once it
|
||||
/// reaches `threshold`, raises (or refreshes) the banner warning built by
|
||||
/// reaches `threshold`, raises (or refreshes) the warning warning built by
|
||||
/// `message`. Below the threshold this is a cheap no-op — `message` is
|
||||
/// only invoked when the banner is actually shown, so callers can format
|
||||
/// only invoked when the warning is actually shown, so callers can format
|
||||
/// lazily.
|
||||
pub fn record_err(&mut self, message: impl FnOnce(SweepFailure) -> String) {
|
||||
self.misses = self.misses.saturating_add(1);
|
||||
|
|
@ -96,7 +96,7 @@ impl SweepHealth {
|
|||
}
|
||||
|
||||
/// Compact human-readable age (e.g. `"45s"`, `"12m"`, `"3h"`, `"2d"`) for a
|
||||
/// banner "last succeeded N ago" suffix. Coarse on purpose — the banner wants
|
||||
/// warning "last succeeded N ago" suffix. Coarse on purpose — the warning wants
|
||||
/// a glanceable magnitude, not precision.
|
||||
#[must_use]
|
||||
pub fn fmt_age(d: Duration) -> String {
|
||||
|
|
@ -117,7 +117,7 @@ mod tests {
|
|||
use super::*;
|
||||
use crate::warnings::snapshot;
|
||||
|
||||
fn banner(kind: &str) -> Option<String> {
|
||||
fn warning(kind: &str) -> Option<String> {
|
||||
snapshot()
|
||||
.into_iter()
|
||||
.find(|w| w.kind == kind)
|
||||
|
|
@ -125,34 +125,34 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn below_threshold_does_not_banner() {
|
||||
fn below_threshold_does_not_warn() {
|
||||
let mut h = SweepHealth::new("sh_below", "warn", 3);
|
||||
h.record_err(|_| "boom".to_owned());
|
||||
h.record_err(|_| "boom".to_owned());
|
||||
assert!(h.guard.is_none());
|
||||
assert!(banner("sh_below").is_none());
|
||||
assert!(warning("sh_below").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn raises_at_threshold_and_clears_on_ok() {
|
||||
let mut h = SweepHealth::new("sh_raise", "warn", 2);
|
||||
h.record_err(|c| format!("fail #{}", c.consecutive));
|
||||
assert!(banner("sh_raise").is_none(), "one miss < threshold");
|
||||
assert!(warning("sh_raise").is_none(), "one miss < threshold");
|
||||
h.record_err(|c| format!("fail #{}", c.consecutive));
|
||||
assert_eq!(banner("sh_raise").as_deref(), Some("fail #2"));
|
||||
assert_eq!(warning("sh_raise").as_deref(), Some("fail #2"));
|
||||
assert!(h.guard.is_some());
|
||||
h.record_ok();
|
||||
assert!(h.guard.is_none());
|
||||
assert!(banner("sh_raise").is_none(), "success clears the banner");
|
||||
assert!(warning("sh_raise").is_none(), "success clears the warning");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn threshold_one_banners_immediately() {
|
||||
fn threshold_one_warns_immediately() {
|
||||
let mut h = SweepHealth::new("sh_immediate", "crit", 1);
|
||||
h.record_err(|_| "gone".to_owned());
|
||||
assert_eq!(banner("sh_immediate").as_deref(), Some("gone"));
|
||||
assert_eq!(warning("sh_immediate").as_deref(), Some("gone"));
|
||||
h.record_ok();
|
||||
assert!(banner("sh_immediate").is_none());
|
||||
assert!(warning("sh_immediate").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -161,9 +161,9 @@ mod tests {
|
|||
h.record_err(|_| "x".to_owned()); // miss 1
|
||||
h.record_ok(); // reset
|
||||
h.record_err(|c| format!("c={}", c.consecutive)); // miss 1 again, < threshold
|
||||
assert!(banner("sh_reset").is_none(), "counter reset after ok");
|
||||
assert!(warning("sh_reset").is_none(), "counter reset after ok");
|
||||
h.record_err(|c| format!("c={}", c.consecutive)); // miss 2 → raise
|
||||
assert_eq!(banner("sh_reset").as_deref(), Some("c=2"));
|
||||
assert_eq!(warning("sh_reset").as_deref(), Some("c=2"));
|
||||
h.record_ok();
|
||||
}
|
||||
|
||||
|
|
@ -171,9 +171,9 @@ mod tests {
|
|||
fn refresh_updates_message_while_held() {
|
||||
let mut h = SweepHealth::new("sh_refresh", "warn", 1);
|
||||
h.record_err(|c| format!("miss {}", c.consecutive));
|
||||
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 1"));
|
||||
assert_eq!(warning("sh_refresh").as_deref(), Some("miss 1"));
|
||||
h.record_err(|c| format!("miss {}", c.consecutive));
|
||||
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 2"));
|
||||
assert_eq!(warning("sh_refresh").as_deref(), Some("miss 2"));
|
||||
h.record_ok();
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,21 +1,21 @@
|
|||
//! Process-global server-warning registry backing the dashboard banner.
|
||||
//! Process-global hive-warning registry.
|
||||
//!
|
||||
//! The dashboard shows a top-of-page banner for hive-level degradation
|
||||
//! (disk pressure, a failed converge sweep, agents that can't log in, …).
|
||||
//! Historically each warning was *pulled*: [`host_stats::server_warnings`]
|
||||
//! recomputed the whole list on every `/api/state` by re-probing (a
|
||||
//! `statvfs`, a scan of the container snapshot). Background sweeps that
|
||||
//! failed had nowhere to record it, so they just `warn!`'d to the journal
|
||||
//! and vanished from anywhere an operator looks.
|
||||
//! Subsystems record hive-level degradation here (disk pressure, a failed
|
||||
//! converge sweep, agents that can't log in, …). Three consumers read it:
|
||||
//! `/health/ready` and the swarm status publisher through [`readiness`], and
|
||||
//! the `hyperhive.hive.degraded` / `hyperhive.hive.warnings` /
|
||||
//! `hyperhive.hive.warning` gauges in `otel_metrics`, which the Grafana
|
||||
//! alerts watch. There is no dashboard banner.
|
||||
//!
|
||||
//! This module replaces the pull model with a **push** one: a single
|
||||
//! `Mutex<BTreeMap<kind, …>>` any subsystem writes into. [`snapshot`] is a
|
||||
//! cheap read of that map. The default API is RAII: [`set_warning`] returns
|
||||
//! a [`WarningGuard`] whose `Drop` removes the key, so a producer holds the
|
||||
//! guard exactly as long as the condition is true and lets it drop to clear
|
||||
//! — no manual dismiss, no stale banner. A failing periodic sweep does
|
||||
//! `let _w = set_warning(...)` in its failure branch and drops it on the
|
||||
//! next success.
|
||||
//! The registry is **push**-based: a single `Mutex<BTreeMap<kind, …>>` any
|
||||
//! subsystem writes into. [`snapshot`] is a cheap read of that map. The
|
||||
//! default API is RAII: [`set_warning`] returns a [`WarningGuard`] whose
|
||||
//! `Drop` removes the key, so a producer holds the guard exactly as long as
|
||||
//! the condition is true and lets it drop to clear — no manual dismiss, no
|
||||
//! stale warning. A failing periodic sweep does `let _w = set_warning(...)`
|
||||
//! in its failure branch and drops it on the next success. Raising a warning
|
||||
//! also logs one `warn` line with the message, so the text behind a `kind`
|
||||
//! label is in the log store.
|
||||
//!
|
||||
//! Concurrency: all mutation is behind the one mutex. Each guard carries a
|
||||
//! unique id and only touches its own map entry (`Drop` / [`update`]
|
||||
|
|
@ -23,7 +23,6 @@
|
|||
//! so a re-`set_warning` for the same kind is never clobbered when the
|
||||
//! stale guard later drops.
|
||||
//!
|
||||
//! [`host_stats::server_warnings`]: crate::host_stats::server_warnings
|
||||
//! [`update`]: WarningGuard::update
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
|
@ -49,7 +48,7 @@ struct Slot {
|
|||
}
|
||||
|
||||
/// The one process-global registry. `BTreeMap` (not `HashMap`) so
|
||||
/// [`snapshot`] yields a deterministic, kind-sorted banner order across
|
||||
/// [`snapshot`] yields a deterministic, kind-sorted order across
|
||||
/// renders.
|
||||
fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> {
|
||||
static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new();
|
||||
|
|
@ -81,11 +80,13 @@ pub fn set_warning(
|
|||
message: impl Into<String>,
|
||||
) -> WarningGuard {
|
||||
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
|
||||
let message = message.into();
|
||||
tracing::warn!(kind, level, %message, "hive warning raised");
|
||||
lock().insert(
|
||||
kind,
|
||||
Slot {
|
||||
level,
|
||||
message: message.into(),
|
||||
message,
|
||||
owner: id,
|
||||
},
|
||||
);
|
||||
|
|
@ -170,24 +171,24 @@ pub fn snapshot() -> Vec<ServerWarning> {
|
|||
}
|
||||
|
||||
/// Raise a warning for a one-shot boot-time step that has no periodic
|
||||
/// retry to observe a later success and clear the banner via
|
||||
/// retry to observe a later success and clear the warning via
|
||||
/// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`,
|
||||
/// which `tokio::spawn`s once at hive-c0re startup and never runs again
|
||||
/// this process. Intentionally **leaks** the guard for the life of the
|
||||
/// process: the banner clears the next time hive-c0re restarts (a fresh
|
||||
/// process: the warning clears the next time hive-c0re restarts (a fresh
|
||||
/// process starts with an empty registry) and re-runs the step, which is
|
||||
/// exactly when a config/environment fix would take effect anyway.
|
||||
///
|
||||
/// Do not use this for anything that runs periodically or can be
|
||||
/// retried within the same process — hold the [`WarningGuard`] (or use
|
||||
/// [`crate::stats::sweep_health::SweepHealth`]) so a later success can
|
||||
/// actually clear the banner instead of waiting for a restart.
|
||||
/// actually clear the warning instead of waiting for a restart.
|
||||
pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) {
|
||||
std::mem::forget(set_warning(kind, level, message));
|
||||
}
|
||||
|
||||
/// RAII handle for one active warning. Drop clears the `kind` from the
|
||||
/// banner. Obtained from [`set_warning`].
|
||||
/// registry. Obtained from [`set_warning`].
|
||||
#[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"]
|
||||
pub struct WarningGuard {
|
||||
kind: &'static str,
|
||||
|
|
|
|||
|
|
@ -27,9 +27,9 @@ use crate::stats::sweep_health::{self, SweepHealth};
|
|||
/// cadence than the hive's own.
|
||||
pub const PUBLISH_INTERVAL: Duration = crate::swarm_status::PUBLISH_INTERVAL;
|
||||
|
||||
/// Consecutive failed sweeps before the dashboard banners. Mirrors
|
||||
/// Consecutive failed sweeps before the sweep raises a hive warning. Mirrors
|
||||
/// [`crate::swarm_status`]'s constant of the same name and purpose.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
/// Start the publish loop, if this deployment wired up a swarm queue.
|
||||
///
|
||||
|
|
@ -44,8 +44,8 @@ const FAILURES_BEFORE_BANNER: u32 = 3;
|
|||
/// deploy-event listener does.
|
||||
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
||||
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
|
||||
// Already bannered once by `swarm_status::spawn`, which runs the
|
||||
// identical check — a second banner for the same missing env var
|
||||
// Already warned once by `swarm_status::spawn`, which runs the
|
||||
// identical check — a second warning for the same missing env var
|
||||
// would just be the same fact told twice.
|
||||
return;
|
||||
};
|
||||
|
|
@ -55,8 +55,11 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
|||
return;
|
||||
};
|
||||
|
||||
let mut health =
|
||||
SweepHealth::new("swarm_agent_status_publish", "warn", FAILURES_BEFORE_BANNER);
|
||||
let mut health = SweepHealth::new(
|
||||
"swarm_agent_status_publish",
|
||||
"warn",
|
||||
FAILURES_BEFORE_WARNING,
|
||||
);
|
||||
loop {
|
||||
match publish_all(&client, &hive).await {
|
||||
Ok(published) => {
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@
|
|||
//! coordinates is a silent no-op (the ordinary case). A hive that cannot
|
||||
//! reach the queue loses the swarm's visibility of the notice, not the
|
||||
//! host's — `warn!` fires on every failed attempt regardless, and the
|
||||
//! dashboard banners only after [`FAILURES_BEFORE_BANNER`] consecutive
|
||||
//! raises a hive warning only after [`FAILURES_BEFORE_WARNING`] consecutive
|
||||
//! misses, the same debounce shape [`crate::swarm_status`] uses and for
|
||||
//! the same reason: a `warn!` that fires every call for three weeks is
|
||||
//! indistinguishable from silence in practice.
|
||||
|
|
@ -24,10 +24,10 @@ use tokio::sync::{Mutex, OnceCell};
|
|||
|
||||
use crate::stats::sweep_health::{self, SweepHealth};
|
||||
|
||||
/// Consecutive failed publishes before the dashboard banners — same
|
||||
/// Consecutive failed publishes before the sweep raises a hive warning — same
|
||||
/// value [`crate::swarm_status`] uses and for the same reason: a debounce
|
||||
/// against one blip flapping a banner an operator learns to ignore.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
/// against one blip flapping a warning an operator learns to ignore.
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
static HEALTH: OnceCell<Mutex<SweepHealth>> = OnceCell::const_new();
|
||||
|
||||
|
|
@ -37,7 +37,7 @@ async fn health() -> tokio::sync::MutexGuard<'static, SweepHealth> {
|
|||
Mutex::new(SweepHealth::new(
|
||||
"swarm_notices_publish",
|
||||
"warn",
|
||||
FAILURES_BEFORE_BANNER,
|
||||
FAILURES_BEFORE_WARNING,
|
||||
))
|
||||
})
|
||||
.await
|
||||
|
|
@ -57,7 +57,7 @@ pub async fn notify(subsystem: &str, key: Option<String>, summary: String, sourc
|
|||
return;
|
||||
};
|
||||
// Same absent-name condition `swarm_status` bails on — that module
|
||||
// already banners it under `swarm_status_config` the first time
|
||||
// already warns it under `swarm_status_config` the first time
|
||||
// either of us hits it; nothing more to add from here.
|
||||
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
|
||||
return;
|
||||
|
|
|
|||
|
|
@ -44,15 +44,15 @@ const FIRST_RETRY_DELAY: Duration = Duration::from_secs(5);
|
|||
/// The longest wait between two attempts.
|
||||
const MAX_RETRY_DELAY: Duration = Duration::from_mins(5);
|
||||
|
||||
/// The banner kind a failed attempt raises, cleared by the next success.
|
||||
const BANNER: &str = "swarm_queue_config";
|
||||
/// The warning kind a failed attempt raises, cleared by the next success.
|
||||
const WARNING_KIND: &str = "swarm_queue_config";
|
||||
|
||||
static CLIENT: Retrying<async_nats::Client> = Retrying::new(FIRST_RETRY_DELAY, MAX_RETRY_DELAY);
|
||||
|
||||
/// The shared swarm-queue client, or `None` when there is none right now.
|
||||
///
|
||||
/// `None` covers "no queue configured" (the ordinary case, logged once at
|
||||
/// `info`), a deployment bug in the coordinates (bannered, and permanent for
|
||||
/// `info`), a deployment bug in the coordinates (warned, and permanent for
|
||||
/// this process), and an attempt that failed or is still backing off. A
|
||||
/// caller with `None` should just skip whatever it was about to publish.
|
||||
/// Never waits out a backoff: see [`wait_for_client`] for that.
|
||||
|
|
@ -95,12 +95,12 @@ enum Outcome<T> {
|
|||
enum Slot<T> {
|
||||
Off,
|
||||
Ready(T),
|
||||
/// `failures` consecutive attempts have failed. `banner` holds the
|
||||
/// `failures` consecutive attempts have failed. `warning` holds the
|
||||
/// dashboard warning until the slot leaves this state.
|
||||
Failing {
|
||||
failures: u32,
|
||||
not_before: Instant,
|
||||
banner: Option<crate::warnings::WarningGuard>,
|
||||
warning: Option<crate::warnings::WarningGuard>,
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -163,22 +163,22 @@ impl<T: Clone> Retrying<T> {
|
|||
let not_before = Instant::now() + delay;
|
||||
let message = format!("swarm queue is off, retrying in {delay:?}: {e:#}");
|
||||
tracing::warn!(failures, "{message}");
|
||||
// Updating the held guard keeps the banner up without a
|
||||
// Updating the held guard keeps the warning up without a
|
||||
// moment where it is cleared and re-raised.
|
||||
let banner = match slot.take() {
|
||||
let warning = match slot.take() {
|
||||
Some(Slot::Failing {
|
||||
banner: Some(banner),
|
||||
warning: Some(warning),
|
||||
..
|
||||
}) => {
|
||||
banner.update("warn", message);
|
||||
banner
|
||||
warning.update("warn", message);
|
||||
warning
|
||||
}
|
||||
_ => crate::warnings::set_warning(BANNER, "warn", message),
|
||||
_ => crate::warnings::set_warning(WARNING_KIND, "warn", message),
|
||||
};
|
||||
*slot = Some(Slot::Failing {
|
||||
failures,
|
||||
not_before,
|
||||
banner: Some(banner),
|
||||
warning: Some(warning),
|
||||
});
|
||||
Outcome::NotBefore(not_before)
|
||||
}
|
||||
|
|
@ -204,7 +204,11 @@ async fn connect_once() -> Attempt<async_nats::Client> {
|
|||
}
|
||||
Err(e) => {
|
||||
// A deployment bug, which no retry clears.
|
||||
crate::warnings::set_boot_warning(BANNER, "warn", format!("swarm queue is off: {e:#}"));
|
||||
crate::warnings::set_boot_warning(
|
||||
WARNING_KIND,
|
||||
"warn",
|
||||
format!("swarm queue is off: {e:#}"),
|
||||
);
|
||||
return Attempt::Off;
|
||||
}
|
||||
};
|
||||
|
|
|
|||
|
|
@ -47,18 +47,18 @@ use crate::stats::sweep_health::{self, SweepHealth};
|
|||
/// other silently re-tunes the swarm's definition of "quiet".
|
||||
pub const PUBLISH_INTERVAL: Duration = Duration::from_mins(1);
|
||||
|
||||
/// Consecutive failed publishes before the dashboard banners.
|
||||
/// Consecutive failed publishes before the sweep raises a hive warning.
|
||||
///
|
||||
/// At [`PUBLISH_INTERVAL`] this is ~3 minutes of genuine failure, so a
|
||||
/// blip does not flap a banner an operator learns to ignore.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
/// blip does not flap a warning an operator learns to ignore.
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
/// Start the publish loop, if this deployment wired up a swarm queue.
|
||||
///
|
||||
/// The connect itself is shared with every other swarm-queue consumer in
|
||||
/// this process — see [`crate::swarm_queue`] for why one connection and
|
||||
/// not one per consumer, and for where "no queue configured" vs. "queue
|
||||
/// configured but unreachable" gets bannered. This function only decides
|
||||
/// configured but unreachable" gets warned. This function only decides
|
||||
/// whether *status* has anything to offer once a client exists.
|
||||
pub fn spawn(
|
||||
coord: std::sync::Arc<crate::coordinator::Coordinator>,
|
||||
|
|
@ -76,7 +76,7 @@ pub fn spawn(
|
|||
|
||||
tokio::spawn(async move {
|
||||
let Some(client) = crate::swarm_queue::wait_for_client().await else {
|
||||
// No queue configured, already logged or bannered by the
|
||||
// No queue configured, already logged or warned by the
|
||||
// shared connector; nothing left to report here.
|
||||
return;
|
||||
};
|
||||
|
|
@ -117,7 +117,7 @@ pub fn spawn(
|
|||
shutdown.clone(),
|
||||
));
|
||||
|
||||
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER);
|
||||
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_WARNING);
|
||||
loop {
|
||||
match publish(&client, &hive).await {
|
||||
Ok(()) => health.record_ok(),
|
||||
|
|
@ -197,7 +197,7 @@ async fn drain_swarm_events(
|
|||
let mut sub = match client.subscribe(subject).await {
|
||||
Ok(sub) => sub,
|
||||
Err(e) => {
|
||||
// Warn rather than a boot banner: the hive is fully functional
|
||||
// Warn rather than a boot warning: the hive is fully functional
|
||||
// without this, it just falls back to learning about knowledge
|
||||
// changes at its next boot.
|
||||
tracing::warn!(
|
||||
|
|
|
|||
|
|
@ -149,7 +149,7 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
|
|||
// The one place this fires: a swarm-level grafana alert
|
||||
// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`)
|
||||
// matches this exact message against the log store and pages the
|
||||
// operator directly — no agent (and no per-hive dashboard banner)
|
||||
// operator directly — no agent (and no per-hive warning)
|
||||
// in the loop. See `docs/scheduler/observability.md`.
|
||||
tracing::warn!(agent = %stopped, "container crash detected");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -181,7 +181,7 @@ impl ImportOutcome {
|
|||
matches!(self, Self::NothingPending | Self::Imported)
|
||||
}
|
||||
|
||||
/// Why reading is not allowed, for the caller's log and banner.
|
||||
/// Why reading is not allowed, for the caller's log and warning.
|
||||
#[must_use]
|
||||
pub fn into_error(self) -> Option<anyhow::Error> {
|
||||
match self {
|
||||
|
|
|
|||
|
|
@ -24,11 +24,11 @@ use crate::stats::sweep_health::{self, SweepHealth};
|
|||
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
/// Read the caps now and every [`READ_INTERVAL`] until shutdown.
|
||||
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
||||
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER);
|
||||
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_WARNING);
|
||||
loop {
|
||||
match sync(&client).await {
|
||||
Ok(()) => health.record_ok(),
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ use crate::stats::sweep_health::{self, SweepHealth};
|
|||
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
/// Send the meta repo's grants once, then read the permissions now and every
|
||||
/// [`READ_INTERVAL`] until shutdown.
|
||||
|
|
@ -40,7 +40,7 @@ pub async fn follow(
|
|||
hive: String,
|
||||
mut shutdown: tokio::sync::watch::Receiver<bool>,
|
||||
) {
|
||||
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_BANNER);
|
||||
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_WARNING);
|
||||
loop {
|
||||
let import = super::permissions_import::send_pending(&client, &hive).await;
|
||||
let pass = if import.read_allowed() {
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
|
|||
let client =
|
||||
match tokio::time::timeout_at(deadline, crate::swarm_queue::wait_for_client()).await {
|
||||
// Absent — already reported by the shared connector (an `info` log or
|
||||
// a `swarm_queue_config` banner).
|
||||
// a `swarm_queue_config` warning).
|
||||
Ok(None) => return Ok(()),
|
||||
Ok(Some(client)) => client,
|
||||
Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()),
|
||||
|
|
|
|||
|
|
@ -142,6 +142,78 @@ let
|
|||
annotations = { inherit summary; };
|
||||
};
|
||||
|
||||
# An alert rule that fires while a metric query's latest value is above
|
||||
# `threshold`. `expr` is an instant MetricsQL query; names with dots are
|
||||
# selected as `{__name__="a.b"}`, the way the dashboards do.
|
||||
#
|
||||
# No data is `OK`: a series that is absent means the condition is not
|
||||
# raised (hive-c0re stops reporting a warning once it clears), and a hive
|
||||
# that stopped exporting at all is the log-absence rules' job.
|
||||
metricThresholdRule =
|
||||
{
|
||||
uid,
|
||||
title,
|
||||
windowSeconds ? 600,
|
||||
expr,
|
||||
threshold,
|
||||
for,
|
||||
summary,
|
||||
}:
|
||||
{
|
||||
inherit uid title for;
|
||||
condition = "C";
|
||||
data = [
|
||||
{
|
||||
refId = "A";
|
||||
relativeTimeRange = {
|
||||
from = windowSeconds;
|
||||
to = 0;
|
||||
};
|
||||
inherit datasourceUid;
|
||||
model = {
|
||||
refId = "A";
|
||||
datasource = {
|
||||
type = "prometheus";
|
||||
uid = datasourceUid;
|
||||
};
|
||||
inherit expr;
|
||||
instant = true;
|
||||
range = false;
|
||||
};
|
||||
}
|
||||
{
|
||||
refId = "B";
|
||||
datasourceUid = "__expr__";
|
||||
model = {
|
||||
refId = "B";
|
||||
type = "reduce";
|
||||
expression = "A";
|
||||
reducer = "last";
|
||||
};
|
||||
}
|
||||
{
|
||||
refId = "C";
|
||||
datasourceUid = "__expr__";
|
||||
model = {
|
||||
refId = "C";
|
||||
type = "threshold";
|
||||
expression = "B";
|
||||
conditions = [
|
||||
{
|
||||
evaluator = {
|
||||
type = "gt";
|
||||
params = [ threshold ];
|
||||
};
|
||||
}
|
||||
];
|
||||
};
|
||||
}
|
||||
];
|
||||
noDataState = "OK";
|
||||
execErrState = "Error";
|
||||
annotations = { inherit summary; };
|
||||
};
|
||||
|
||||
# `logCountRule` inverted: fires while the count is below one. `expr` must
|
||||
# be an ungrouped `stats count()`, which answers 0 rather than no row when
|
||||
# nothing matched; a grouped count drops the group instead, and a dropped
|
||||
|
|
@ -1600,6 +1672,53 @@ in
|
|||
for = "0s";
|
||||
summary = "{{ $labels._HOSTNAME }} detected agent {{ $labels.F_AGENT }}'s container crash (not an operator-initiated stop)";
|
||||
})
|
||||
# hive-c0re's own readiness verdict (`hyperhive.hive.degraded`
|
||||
# in otel_metrics.rs), the same one `/health/ready` gives:
|
||||
# `1` while any `crit`-level warning is raised. The
|
||||
# per-kind rule below says which one. `for` rides out a
|
||||
# restart-time blip.
|
||||
(metricThresholdRule {
|
||||
uid = "hh-hive-degraded";
|
||||
title = "hive degraded";
|
||||
expr = ''max by (hive) ({__name__="hyperhive.hive.degraded"})'';
|
||||
threshold = 0;
|
||||
for = "2m";
|
||||
summary = "hive {{ $labels.hive }} reports itself degraded: a critical warning is raised; the `hh-hive-warning` alert names it";
|
||||
})
|
||||
# One series per raised warning (`hyperhive.hive.warning`),
|
||||
# so each `kind` is its own alert instance. The text is the
|
||||
# `hive warning raised` line in the hive-c0re log. Disk
|
||||
# pressure is left out: it has its own thresholds below, on
|
||||
# the filesystem metric rather than the registry entry.
|
||||
# Sweeps raise their warning only after several misses, so
|
||||
# `for` is short.
|
||||
(metricThresholdRule {
|
||||
uid = "hh-hive-warning";
|
||||
title = "hive warning raised";
|
||||
expr = ''max by (hive, kind, level) ({__name__="hyperhive.hive.warning", kind!="disk_pressure"})'';
|
||||
threshold = 0;
|
||||
for = "2m";
|
||||
summary = "hive {{ $labels.hive }} has a {{ $labels.level }} warning raised: {{ $labels.kind }}; the message is the `hive warning raised` line in the hive-c0re log";
|
||||
})
|
||||
# `hyperhive.hive.nix_store.utilization` is the used fraction of the
|
||||
# filesystem holding the nix store, the one that failed CI
|
||||
# with ENOSPC before the GC guardrails.
|
||||
(metricThresholdRule {
|
||||
uid = "hh-host-disk-high";
|
||||
title = "host disk filling";
|
||||
expr = ''max by (hive) ({__name__="hyperhive.hive.nix_store.utilization"})'';
|
||||
threshold = 0.85;
|
||||
for = "10m";
|
||||
summary = "the nix store's filesystem on hive {{ $labels.hive }} is over 85% full; garbage-collect before it runs out of space";
|
||||
})
|
||||
(metricThresholdRule {
|
||||
uid = "hh-host-disk-critical";
|
||||
title = "host disk almost full";
|
||||
expr = ''max by (hive) ({__name__="hyperhive.hive.nix_store.utilization"})'';
|
||||
threshold = 0.95;
|
||||
for = "2m";
|
||||
summary = "the nix store's filesystem on hive {{ $labels.hive }} is over 95% full; garbage-collect now";
|
||||
})
|
||||
]
|
||||
# One rule per declared hive rather than one grouped by
|
||||
# `hive`, so a hive that sends nothing at all still has
|
||||
|
|
|
|||
|
|
@ -498,6 +498,40 @@ let
|
|||
&& lib.hasInfix ''_msg:="turn failed"'' expr
|
||||
&& lib.hasInfix "stats by (hive, agent)" expr;
|
||||
}
|
||||
{
|
||||
# The hive-health alerts query metric names that hive-c0re spells in
|
||||
# Rust. A rename on either side provisions cleanly and the rule then
|
||||
# reads nothing, so it never fires. Each name here is checked against
|
||||
# the exporter's source rather than a copy of it, and each rule must
|
||||
# query the metrics datasource with an instant query.
|
||||
# 🔬 Invert-proof: rename a gauge in `otel_metrics.rs` and this fails.
|
||||
name = "the hive-health alert rules query metrics hive-c0re exports";
|
||||
ok =
|
||||
let
|
||||
groups =
|
||||
grafanaHere.containers.swarm-grafana.config.services.grafana.provision.alerting.rules.settings.groups;
|
||||
rules = lib.concatMap (g: g.rules) groups;
|
||||
source = builtins.readFile ../../hive-c0re/src/stats/otel_metrics.rs;
|
||||
expected = {
|
||||
"hh-hive-degraded" = "hyperhive.hive.degraded";
|
||||
"hh-hive-warning" = "hyperhive.hive.warning";
|
||||
"hh-host-disk-high" = "hyperhive.hive.nix_store.utilization";
|
||||
"hh-host-disk-critical" = "hyperhive.hive.nix_store.utilization";
|
||||
};
|
||||
check =
|
||||
uid: metric:
|
||||
let
|
||||
rule = lib.findFirst (r: r.uid == uid) null rules;
|
||||
query = (lib.head rule.data).model;
|
||||
in
|
||||
rule != null
|
||||
&& lib.hasInfix ''{__name__="${metric}"'' query.expr
|
||||
&& lib.hasInfix ''"${metric}"'' source
|
||||
&& query.instant
|
||||
&& (lib.head rule.data).datasourceUid == "swarm-victoriametrics";
|
||||
in
|
||||
lib.all (uid: check uid expected.${uid}) (builtins.attrNames expected);
|
||||
}
|
||||
{
|
||||
# 🩸 The regression this guards: moving bao's metrics scrape into its
|
||||
# own in-container collector made that collector stamp every exported
|
||||
|
|
|
|||
Loading…
Reference in a new issue