vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

1
hive-c0re/src/coordinator.rs
View file

@ -599,6 +599,7 @@ impl Coordinator {
return;
}
};
crate::host_stats::refresh_pending_logins(&fresh);
let mut last = self.last_containers.lock().await;
let mut changed_or_new = Vec::new();
let mut removed = Vec::new();

7
hive-c0re/src/dashboard/health.rs
View file

@ -1,9 +1,8 @@
//! Hive-wide health endpoints for external monitoring: `/health/live`
//! (liveness) and `/health/ready` (readiness).
//!
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the
//! dashboard banner already reads — as the single source of truth for
//! "is anything degraded", rather than tracking health separately. Two
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
//! for "is anything degraded", rather than tracking health separately. Two
//! systems independently deciding what counts as unhealthy is how they
//! end up disagreeing.
//!
@ -12,7 +11,7 @@
//! the *process* down over a transient dependency failure — that's what
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
//! warning currently set; `warn`-level entries ride along in the body for
//! detail but don't flip the status, mirroring the banner's own severity
//! detail but don't flip the status, mirroring the warning registry's own severity
//! split.
//!
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can

16
hive-c0re/src/dashboard/state_snapshot.rs
View file

@ -102,12 +102,6 @@ pub(super) struct StateSnapshot {
/// var, set from `services.hyperhive.swarm.name`. `None` when
/// unset — chrome omits the swarm segment of the breadcrumb.
swarm_name: Option<String>,
/// Server-level warnings for the dashboard's top-of-page banner
/// (currently host disk-pressure; more producers can be added
/// backend-side). Empty when all clear. Built by
/// `host_stats::server_warnings`; the frontend renders this list
/// generically, so new warning kinds need no frontend change.
server_warnings: Vec<crate::host_stats::ServerWarning>,
}
#[derive(Serialize)]
@ -193,15 +187,6 @@ pub(super) async fn api_state(
let tombstones = build_tombstone_views(&containers);
let port_conflicts = build_port_conflicts(&containers);
// Banner warnings: host probes (disk) + agent-state (pending logins).
// Built before the response struct because the agent-state producer
// borrows `containers`, which moves in below.
let server_warnings = {
let mut w = crate::host_stats::server_warnings();
w.extend(crate::host_stats::agent_state_warnings(&containers));
w
};
axum::Json(StateSnapshot {
seq,
hostname,
@ -233,7 +218,6 @@ pub(super) async fn api_state(
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
.ok()
.filter(|s| !s.is_empty()),
server_warnings,
})
}

6
hive-c0re/src/forge/mod.rs
View file

@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
/// most a handful per process). [`crate::warnings::set_boot_warning`]
/// requires a `'static` kind so distinct orgs/repos don't clobber each
/// other's banner entry; leaking a few short strings once per boot is
/// other's warning entry; leaking a few short strings once per boot is
/// cheap and bounded, unlike a per-request or per-loop-iteration leak.
fn static_kind(s: String) -> &'static str {
Box::leak(s.into_boxed_str())
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
/// confirms the container exists — not that Forgejo is *listening*. A
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
/// without this gate every `ensure_*` fires at a refused socket and leaves a
/// stale "provisioning failed" banner that never clears (the pass is one-shot).
/// stale "provisioning failed" warning that never clears (the pass is one-shot).
/// Returns whether the endpoint became ready before the deadline; on timeout
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
@ -393,7 +393,7 @@ pub async fn ensure_all() {
}
// The container exists, but its HTTP may not be up yet (a rebuild restarts
// hive-forge + hive-c0re together). Wait for the API to answer before the
// provisioning pass so a boot race doesn't leave stale failure banners.
// provisioning pass so a boot race doesn't leave stale failure warnings.
wait_until_ready().await;
let core_token = match ensure_core_user_and_token().await {
Ok(t) => Some(t),

2
hive-c0re/src/forge/repos.rs
View file

@ -125,7 +125,7 @@ pub async fn push_meta(dir: &Path) -> Result<()> {
let msg = "forge/core/meta push rejected (non-fast-forward): \
remote is ahead of local; leaving remote history intact";
tracing::warn!("{msg}");
// Raise a persistent dashboard banner; cleared on next successful push.
// Raise a persistent hive warning; cleared on next successful push.
if let Ok(mut g) = META_NON_FF_GUARD.lock() {
*g = Some(crate::warnings::set_warning(
"forge-meta-non-ff",

2
hive-c0re/src/gateway_nginx.rs
View file

@ -32,7 +32,7 @@ static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
/// Minimum gap between retry attempts after a reload failure (30 s).
const RELOAD_RETRY_SECS: u64 = 30;
/// Dashboard-banner health tracker for the reload sweep — raises a
/// Hive-warning health tracker for the reload sweep — raises a
/// `gateway_nginx_reload` warning the first time a reload fails (routing is
/// broken for whoever depends on the change *right now*, so no debounce
/// window) and clears it the moment a reload succeeds again.

14
hive-c0re/src/job_queue/exec.rs
View file

@ -139,7 +139,7 @@ pub(super) async fn run_node(
/// Boot-time forge user/token sweep as a DAG node — see
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
/// per-step error handling and boot-warning banners internally (it's
/// per-step error handling and boot warnings internally (it's
/// best-effort by design), so this wrapper has nothing left to report;
/// it exists purely to make the sweep a visible unit of work.
async fn run_forge_sweep() -> Result<()> {
@ -149,7 +149,7 @@ async fn run_forge_sweep() -> Result<()> {
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
/// Every pass reports here, whoever submitted it: as the node's own error on
/// the dashboard, and into the debounced banner.
/// the dashboard, and into the debounced warning.
async fn run_matrix_sweep() -> Result<()> {
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
.await
@ -164,7 +164,7 @@ async fn run_matrix_sweep() -> Result<()> {
health.record_ok();
Ok(())
} else {
health.record_err(matrix_sweep_banner);
health.record_err(matrix_sweep_warning);
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
}
}
@ -177,7 +177,7 @@ async fn run_matrix_sweep() -> Result<()> {
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
/// Banner for a matrix sweep that keeps failing. Two misses in a row, so one
/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
/// dashboard; cleared by the next clean pass.
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
@ -185,7 +185,7 @@ fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
}
fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
let age = ctx.since_last_ok.map_or_else(
|| "no success this session".to_owned(),
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
@ -200,7 +200,7 @@ fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
/// pass reports here, whoever submitted it: as the node's own error on the
/// dashboard, and into the debounced banner.
/// dashboard, and into the debounced warning.
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
let result = tokio::time::timeout(
KNOWLEDGE_PULL_DEADLINE,
@ -242,7 +242,7 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
/// Banner for a `/knowledge` pull that keeps failing: three misses in a row,
/// Warning for a `/knowledge` pull that keeps failing: three misses in a row,
/// cleared by the next successful pull.
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();

6
hive-c0re/src/lifecycle/mod.rs
View file

@ -367,11 +367,11 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
agents_for_meta(None).await
}
/// Per-agent "stop failed" banner guards, keyed by agent name. Cleared the
/// Per-agent "stop failed" warning guards, keyed by agent name. Cleared the
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
/// rather than a bare `set_boot_warning` because this condition (unlike a
/// one-shot boot step) genuinely resolves later, once an operator has
/// intervened, and the banner should clear with it instead of surviving
/// intervened, and the warning should clear with it instead of surviving
/// until the next hive-c0re restart.
fn stop_failed_guards()
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
@ -402,7 +402,7 @@ pub async fn kill(name: &str) -> Result<()> {
.unwrap_or_else(std::sync::PoisonError::into_inner);
match result {
Ok(()) => {
// A clean stop clears any earlier "this looked wedged" banner
// A clean stop clears any earlier "this looked wedged" warning
// for the same agent.
guards.remove(name);
Ok(())

4
hive-c0re/src/main.rs
View file

@ -284,12 +284,12 @@ async fn cmd_serve(
}
}
});
// Disk-pressure watch: raise a `disk_pressure` banner warning while the
// Disk-pressure watch: raise a `disk_pressure` warning while the
// host nix store is over threshold and clear it when back under, via the
// push-based warnings registry. Replaces the old per-`/api/state`
// `statvfs`. ~60s cadence; first tick fires immediately. The held guard
// lives for the task's lifetime — dropping it (on shutdown) clears the
// banner.
// warning.
let mut disk_shutdown = coord.shutdown_rx();
tokio::spawn(async move {
let mut disk_guard: Option<warnings::WarningGuard> = None;

4
hive-c0re/src/matrix.rs
View file

@ -787,8 +787,8 @@ pub async fn invite_user(
/// Returns `true` when every step of the sweep succeeded, `false` when at
/// least one step failed — the caller feeds this into a
/// [`crate::stats::sweep_health::SweepHealth`] to raise a debounced
/// dashboard banner on persistent failure (this sweep re-runs every 30
/// minutes, so a one-off blip self-heals without ever bannering).
/// hive warning on persistent failure (this sweep re-runs every 30
/// minutes, so a one-off blip self-heals without ever warning).
pub async fn ensure_all() -> bool {
if !is_present() {
tracing::debug!("matrix: hive-matrix container absent, skipping user sweep");

146
hive-c0re/src/stats/host_stats.rs
View file

@ -1,35 +1,33 @@
//! Host-system probes and the derived **server-warning** list that the
//! dashboard renders as a top-of-page banner.
//! Host-system probes that raise hive-level warnings into the push-based
//! [`crate::warnings`] registry.
//!
//! The first (and currently only) producer is a host disk-usage check:
//! the nix store filling up is what ENOSPC-failed CI before the GC
//! guardrails were documented. hive-c0re runs on the host (not inside a
//! container), so it can `statvfs` the store path directly and warn the
//! operator *before* an ENOSPC, not after.
//! The first producer is a host disk-usage check: the nix store filling up
//! is what ENOSPC-failed CI before the GC guardrails were documented.
//! hive-c0re runs on the host (not inside a container), so it can `statvfs`
//! the store path directly and warn the operator *before* an ENOSPC, not
//! after. [`refresh_disk_warning`] drives a held guard from a periodic task
//! (see `spawn` in `main.rs`). The same probe backs the
//! `hyperhive.hive.nix_store.utilization` gauge ([`disk_utilization`]), which is
//! what the early-warning Grafana alerts read.
//!
//! [`server_warnings`] is the public surface: it returns a flat list of
//! [`ServerWarning`]s for `/api/state`. It is now just a snapshot of the
//! push-based [`crate::warnings`] registry — producers raise/clear their
//! own warnings via an RAII guard rather than being re-probed on every
//! render. The disk-pressure check is reworked onto that model here:
//! [`refresh_disk_warning`] drives a held guard from a periodic task (see
//! `spawn` in `main.rs`) instead of a `statvfs` per `/api/state`.
//! The registry has no dashboard banner any more. A raised warning reaches
//! the operator as the `hyperhive.hive.warning` gauge and a Grafana alert
//! on it (see `docs/scheduler/observability.md`).
use serde::Serialize;
use utoipa::ToSchema;
use crate::container_view::ContainerView;
/// One server-level warning for the dashboard's top-of-page banner.
/// One hive-level warning in the [`crate::warnings`] registry.
#[derive(Debug, Clone, Serialize, ToSchema)]
pub struct ServerWarning {
/// Stable kind id (e.g. `"disk_pressure"`) — lets the frontend dedupe
/// or special-case without parsing the message.
/// Stable kind id (e.g. `"disk_pressure"`), the `kind` label of the
/// `hyperhive.hive.warning` gauge.
pub kind: &'static str,
/// Severity: `"warn"` (amber) or `"crit"` (red). The banner picks its
/// colour from this; everything else is just the message text.
/// Severity: `"warn"` or `"crit"`. `crit` degrades readiness.
pub level: &'static str,
/// Human-readable, already-formatted message shown in the banner.
/// Human-readable, already-formatted message.
pub message: String,
}
@ -38,17 +36,7 @@ pub struct ServerWarning {
const DISK_WARN_PCT: f64 = 85.0;
const DISK_CRIT_PCT: f64 = 95.0;
/// Snapshot the current server-level warnings for the dashboard banner —
/// a cheap read of the push-based [`crate::warnings`] registry. Producers
/// (the disk watch below, background converge sweeps, …) raise and clear
/// their own entries via [`crate::warnings::set_warning`]; nothing is
/// re-probed here.
#[must_use]
pub fn server_warnings() -> Vec<ServerWarning> {
crate::warnings::snapshot()
}
/// Pure disk-pressure decision: given usage, return the banner
/// Pure disk-pressure decision: given usage, return the warning
/// `(level, message)` or `None` when below the warn threshold. Split from
/// the probe + guard plumbing so the threshold + formatting is unit-tested
/// without a real `statvfs`.
@ -75,9 +63,8 @@ fn disk_pressure_warning(used_pct: f64, free_bytes: u64) -> Option<(&'static str
/// One tick of the disk-pressure watch: probe the store and reconcile the
/// held warning `guard` — raise or refresh it while usage is over
/// threshold, drop it (clearing the banner) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`, replacing the old
/// per-render `statvfs` in [`server_warnings`].
/// threshold, drop it (clearing the warning) once back under. Driven by the
/// periodic disk-watch task spawned in `main.rs`.
pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
let decision = nix_disk_usage().and_then(|d| disk_pressure_warning(d.used_pct, d.free_bytes));
match decision {
@ -96,19 +83,44 @@ pub fn refresh_disk_warning(guard: &mut Option<crate::warnings::WarningGuard>) {
}
}
/// Agent-state warnings derived from the live container snapshot the
/// dashboard already holds: agents that need a claude login. Kept
/// separate from [`server_warnings`] (host probes) because the caller
/// owns the container list; the dashboard concatenates both into one
/// banner list.
/// Fraction (0–1) of the filesystem holding the nix store that is in use,
/// the value behind the `hyperhive.hive.nix_store.utilization` gauge. `None` when
/// the probe fails, so a failing probe reports no sample rather than zero.
#[must_use]
pub fn disk_utilization() -> Option<f64> {
nix_disk_usage().map(|d| d.used_pct / 100.0)
}
/// The held `pending_logins` warning; see [`refresh_pending_logins`].
static PENDING_LOGINS: std::sync::Mutex<Option<crate::warnings::WarningGuard>> =
std::sync::Mutex::new(None);
/// Reconcile the `pending_logins` warning with the current container
/// snapshot: raise or refresh it while any agent needs a claude login,
/// drop it once none does. Driven by the container rescan in the
/// coordinator, which already builds the snapshot every poll.
pub fn refresh_pending_logins(containers: &[ContainerView]) {
let warning = agent_state_warnings(containers).into_iter().next();
let mut guard = PENDING_LOGINS
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
match (warning, guard.as_ref()) {
(Some(w), Some(g)) => g.update(w.level, w.message),
(Some(w), None) => {
*guard = Some(crate::warnings::set_warning(w.kind, w.level, w.message));
}
(None, _) => *guard = None,
}
}
/// Agent-state warnings derived from the live container snapshot: agents
/// that need a claude login. Takes the container list from its caller
/// because that is where the snapshot lives.
///
/// Container crashes used to have a sibling `agents_crashing` entry here,
/// fed by `Coordinator::recent_crash_counts` — removed in favour of a
/// swarm-level grafana alert on the crash-watcher's own log line (see
/// `nix/host-modules/swarm-grafana.nix`'s `hh-container-crashed` rule and
/// `docs/scheduler/observability.md`). That ships the same signal to the
/// operator without needing an agent (or this per-hive banner) in the
/// loop at all.
/// Container crashes are not a warning here: the crash-watcher logs
/// `container crash detected` and a swarm-level grafana alert
/// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`) matches
/// that line, with no agent in the loop.
#[must_use]
pub fn agent_state_warnings(containers: &[ContainerView]) -> Vec<ServerWarning> {
let mut out = Vec::new();
@ -263,4 +275,46 @@ mod tests {
let w = agent_state_warnings(&containers);
assert!(w[0].message.starts_with("1 agent needs claude login: solo"));
}
/// The registry entry tracks the snapshot it is refreshed from: raised
/// when an agent needs a login, rewritten when the set changes, and gone
/// when none does. This is the only test that touches `pending_logins`,
/// so reading the process-global registry by that kind stays sound under
/// the parallel test runner.
#[test]
fn pending_logins_warning_follows_the_container_snapshot() {
let current = || {
crate::warnings::snapshot()
.into_iter()
.find(|w| w.kind == "pending_logins")
};
assert!(
current().is_none(),
"nothing raised before the first refresh"
);
refresh_pending_logins(&[cv("amy", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("1 agent needs claude login: amy")),
"an agent needing a login raises the warning"
);
refresh_pending_logins(&[cv("amy", true), cv("bob", true)]);
assert!(
current().is_some_and(|w| w.message.starts_with("2 agents need claude login")),
"a second agent rewrites the same entry"
);
refresh_pending_logins(&[cv("amy", false), cv("bob", false)]);
assert!(current().is_none(), "no agent needing a login clears it");
}
/// The gauge value is a fraction, so a percent leaking through would put
/// every alert threshold 100x off.
#[test]
fn disk_utilization_is_a_fraction() {
if let Some(used) = disk_utilization() {
assert!((0.0..=1.0).contains(&used), "got {used}");
}
}
}

35
hive-c0re/src/stats/otel_metrics.rs
View file

@ -305,6 +305,41 @@ fn register_hive_instruments(meter: &opentelemetry::metrics::Meter) {
}
})
.build();
// One series per raised warning, `1` while it is raised, labelled with
// its `kind` and `level`. This is what names the cause in an alert: the
// count above says how many, this says which. A cleared warning stops
// reporting rather than reporting `0`, so an alert on it resolves when
// the series goes stale; the kinds are a fixed set in the source, so the
// label has bounded cardinality.
meter
.u64_observable_gauge("hyperhive.hive.warning")
.with_unit("1")
.with_callback(|obs| {
for w in warnings::snapshot() {
obs.observe(
1,
&[
KeyValue::new("kind", w.kind),
KeyValue::new("level", w.level),
],
);
}
})
.build();
// Fraction of the nix store's filesystem in use, by the semconv name.
// Probed in the callback (a `statvfs` is cheap and this runs once an
// export interval); a failed probe reports no sample.
meter
.f64_observable_gauge("hyperhive.hive.nix_store.utilization")
.with_unit("1")
.with_callback(|obs| {
if let Some(used) = crate::host_stats::disk_utilization() {
obs.observe(used, HIVE_ATTRS);
}
})
.build();
}
/// Attributes for the hive-scoped family: **none**.

50
hive-c0re/src/stats/sweep_health.rs
View file

@ -5,14 +5,14 @@
//! gateway nginx writes, per-agent branch protection, …) historically
//! `warn!`'d to the journal and moved on, so a persistent failure was
//! invisible to the operator. [`SweepHealth`] routes that failure to the
//! dashboard banner instead — but *debounced*, per the routing guidance in
//! hive warning instead — but *debounced*, per the routing guidance in
//! the tracker issue:
//!
//! - a transient miss (container down mid-sweep, a network blip) shouldn't
//! flap a banner on the first failure → `threshold > 1` raises only after
//! flap a warning on the first failure → `threshold > 1` raises only after
//! that many **consecutive** misses;
//! - a won't-fix-itself failure (missing team, bad config, auth rejection)
//! should banner immediately → `threshold == 1`.
//! should warning immediately → `threshold == 1`.
//!
//! The warning clears the moment the sweep next succeeds. A sweep owns one
//! `SweepHealth`, calls [`record_ok`](SweepHealth::record_ok) on success and
@ -24,11 +24,11 @@ use std::time::{Duration, Instant};
use crate::warnings::{WarningGuard, set_warning};
/// Per-sweep failure context handed to the [`record_err`](SweepHealth::record_err)
/// message builder so the banner text can include how long the sweep has been
/// message builder so the warning text can include how long the sweep has been
/// failing and since when it last worked.
#[derive(Debug, Clone, Copy)]
pub struct SweepFailure {
/// Number of consecutive failures so far (≥ `threshold` when the banner
/// Number of consecutive failures so far (≥ `threshold` when the warning
/// is raised).
pub consecutive: u32,
/// Time since the last recorded success, if the sweep has ever succeeded
@ -36,7 +36,7 @@ pub struct SweepFailure {
pub since_last_ok: Option<Duration>,
}
/// A debounced link between one background sweep and the warning banner.
/// A debounced link between one background sweep and the warning warning.
/// Not `Sync`-shared: each sweep task owns its own instance (the underlying
/// registry is the shared, thread-safe part).
pub struct SweepHealth {
@ -50,8 +50,8 @@ pub struct SweepHealth {
impl SweepHealth {
/// Track sweep `kind` at severity `level` (`"warn"` / `"crit"`), raising
/// the banner after `threshold` consecutive failures. `threshold` is
/// floored at 1 (0 would banner before any failure).
/// the warning after `threshold` consecutive failures. `threshold` is
/// floored at 1 (0 would warning before any failure).
#[must_use]
pub fn new(kind: &'static str, level: &'static str, threshold: u32) -> Self {
Self {
@ -65,7 +65,7 @@ impl SweepHealth {
}
/// Record a successful sweep: reset the consecutive-miss counter and drop
/// the guard (clearing any banner entry).
/// the guard (clearing any warning entry).
pub fn record_ok(&mut self) {
self.misses = 0;
self.last_ok = Some(Instant::now());
@ -74,9 +74,9 @@ impl SweepHealth {
}
/// Record a failed sweep. Bumps the consecutive-miss counter; once it
/// reaches `threshold`, raises (or refreshes) the banner warning built by
/// reaches `threshold`, raises (or refreshes) the warning warning built by
/// `message`. Below the threshold this is a cheap no-op — `message` is
/// only invoked when the banner is actually shown, so callers can format
/// only invoked when the warning is actually shown, so callers can format
/// lazily.
pub fn record_err(&mut self, message: impl FnOnce(SweepFailure) -> String) {
self.misses = self.misses.saturating_add(1);
@ -96,7 +96,7 @@ impl SweepHealth {
}
/// Compact human-readable age (e.g. `"45s"`, `"12m"`, `"3h"`, `"2d"`) for a
/// banner "last succeeded N ago" suffix. Coarse on purpose — the banner wants
/// warning "last succeeded N ago" suffix. Coarse on purpose — the warning wants
/// a glanceable magnitude, not precision.
#[must_use]
pub fn fmt_age(d: Duration) -> String {
@ -117,7 +117,7 @@ mod tests {
use super::*;
use crate::warnings::snapshot;
fn banner(kind: &str) -> Option<String> {
fn warning(kind: &str) -> Option<String> {
snapshot()
.into_iter()
.find(|w| w.kind == kind)
@ -125,34 +125,34 @@ mod tests {
}
#[test]
fn below_threshold_does_not_banner() {
fn below_threshold_does_not_warn() {
let mut h = SweepHealth::new("sh_below", "warn", 3);
h.record_err(|_| "boom".to_owned());
h.record_err(|_| "boom".to_owned());
assert!(h.guard.is_none());
assert!(banner("sh_below").is_none());
assert!(warning("sh_below").is_none());
}
#[test]
fn raises_at_threshold_and_clears_on_ok() {
let mut h = SweepHealth::new("sh_raise", "warn", 2);
h.record_err(|c| format!("fail #{}", c.consecutive));
assert!(banner("sh_raise").is_none(), "one miss < threshold");
assert!(warning("sh_raise").is_none(), "one miss < threshold");
h.record_err(|c| format!("fail #{}", c.consecutive));
assert_eq!(banner("sh_raise").as_deref(), Some("fail #2"));
assert_eq!(warning("sh_raise").as_deref(), Some("fail #2"));
assert!(h.guard.is_some());
h.record_ok();
assert!(h.guard.is_none());
assert!(banner("sh_raise").is_none(), "success clears the banner");
assert!(warning("sh_raise").is_none(), "success clears the warning");
}
#[test]
fn threshold_one_banners_immediately() {
fn threshold_one_warns_immediately() {
let mut h = SweepHealth::new("sh_immediate", "crit", 1);
h.record_err(|_| "gone".to_owned());
assert_eq!(banner("sh_immediate").as_deref(), Some("gone"));
assert_eq!(warning("sh_immediate").as_deref(), Some("gone"));
h.record_ok();
assert!(banner("sh_immediate").is_none());
assert!(warning("sh_immediate").is_none());
}
#[test]
@ -161,9 +161,9 @@ mod tests {
h.record_err(|_| "x".to_owned()); // miss 1
h.record_ok(); // reset
h.record_err(|c| format!("c={}", c.consecutive)); // miss 1 again, < threshold
assert!(banner("sh_reset").is_none(), "counter reset after ok");
assert!(warning("sh_reset").is_none(), "counter reset after ok");
h.record_err(|c| format!("c={}", c.consecutive)); // miss 2 → raise
assert_eq!(banner("sh_reset").as_deref(), Some("c=2"));
assert_eq!(warning("sh_reset").as_deref(), Some("c=2"));
h.record_ok();
}
@ -171,9 +171,9 @@ mod tests {
fn refresh_updates_message_while_held() {
let mut h = SweepHealth::new("sh_refresh", "warn", 1);
h.record_err(|c| format!("miss {}", c.consecutive));
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 1"));
assert_eq!(warning("sh_refresh").as_deref(), Some("miss 1"));
h.record_err(|c| format!("miss {}", c.consecutive));
assert_eq!(banner("sh_refresh").as_deref(), Some("miss 2"));
assert_eq!(warning("sh_refresh").as_deref(), Some("miss 2"));
h.record_ok();
}

47
hive-c0re/src/stats/warnings.rs
View file

@ -1,21 +1,21 @@
//! Process-global server-warning registry backing the dashboard banner.
//! Process-global hive-warning registry.
//!
//! The dashboard shows a top-of-page banner for hive-level degradation
//! (disk pressure, a failed converge sweep, agents that can't log in, …).
//! Historically each warning was *pulled*: [`host_stats::server_warnings`]
//! recomputed the whole list on every `/api/state` by re-probing (a
//! `statvfs`, a scan of the container snapshot). Background sweeps that
//! failed had nowhere to record it, so they just `warn!`'d to the journal
//! and vanished from anywhere an operator looks.
//! Subsystems record hive-level degradation here (disk pressure, a failed
//! converge sweep, agents that can't log in, …). Three consumers read it:
//! `/health/ready` and the swarm status publisher through [`readiness`], and
//! the `hyperhive.hive.degraded` / `hyperhive.hive.warnings` /
//! `hyperhive.hive.warning` gauges in `otel_metrics`, which the Grafana
//! alerts watch. There is no dashboard banner.
//!
//! This module replaces the pull model with a **push** one: a single
//! `Mutex<BTreeMap<kind, …>>` any subsystem writes into. [`snapshot`] is a
//! cheap read of that map. The default API is RAII: [`set_warning`] returns
//! a [`WarningGuard`] whose `Drop` removes the key, so a producer holds the
//! guard exactly as long as the condition is true and lets it drop to clear
//! — no manual dismiss, no stale banner. A failing periodic sweep does
//! `let _w = set_warning(...)` in its failure branch and drops it on the
//! next success.
//! The registry is **push**-based: a single `Mutex<BTreeMap<kind, …>>` any
//! subsystem writes into. [`snapshot`] is a cheap read of that map. The
//! default API is RAII: [`set_warning`] returns a [`WarningGuard`] whose
//! `Drop` removes the key, so a producer holds the guard exactly as long as
//! the condition is true and lets it drop to clear — no manual dismiss, no
//! stale warning. A failing periodic sweep does `let _w = set_warning(...)`
//! in its failure branch and drops it on the next success. Raising a warning
//! also logs one `warn` line with the message, so the text behind a `kind`
//! label is in the log store.
//!
//! Concurrency: all mutation is behind the one mutex. Each guard carries a
//! unique id and only touches its own map entry (`Drop` / [`update`]
@ -23,7 +23,6 @@
//! so a re-`set_warning` for the same kind is never clobbered when the
//! stale guard later drops.
//!
//! [`host_stats::server_warnings`]: crate::host_stats::server_warnings
//! [`update`]: WarningGuard::update
use std::collections::BTreeMap;
@ -49,7 +48,7 @@ struct Slot {
}
/// The one process-global registry. `BTreeMap` (not `HashMap`) so
/// [`snapshot`] yields a deterministic, kind-sorted banner order across
/// [`snapshot`] yields a deterministic, kind-sorted order across
/// renders.
fn registry() -> &'static Mutex<BTreeMap<&'static str, Slot>> {
static REG: OnceLock<Mutex<BTreeMap<&'static str, Slot>>> = OnceLock::new();
@ -81,11 +80,13 @@ pub fn set_warning(
message: impl Into<String>,
) -> WarningGuard {
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
let message = message.into();
tracing::warn!(kind, level, %message, "hive warning raised");
lock().insert(
kind,
Slot {
level,
message: message.into(),
message,
owner: id,
},
);
@ -170,24 +171,24 @@ pub fn snapshot() -> Vec<ServerWarning> {
}
/// Raise a warning for a one-shot boot-time step that has no periodic
/// retry to observe a later success and clear the banner via
/// retry to observe a later success and clear the warning via
/// [`WarningGuard`] drop — e.g. a step inside `forge::ensure_all()`,
/// which `tokio::spawn`s once at hive-c0re startup and never runs again
/// this process. Intentionally **leaks** the guard for the life of the
/// process: the banner clears the next time hive-c0re restarts (a fresh
/// process: the warning clears the next time hive-c0re restarts (a fresh
/// process starts with an empty registry) and re-runs the step, which is
/// exactly when a config/environment fix would take effect anyway.
///
/// Do not use this for anything that runs periodically or can be
/// retried within the same process — hold the [`WarningGuard`] (or use
/// [`crate::stats::sweep_health::SweepHealth`]) so a later success can
/// actually clear the banner instead of waiting for a restart.
/// actually clear the warning instead of waiting for a restart.
pub fn set_boot_warning(kind: &'static str, level: &'static str, message: impl Into<String>) {
std::mem::forget(set_warning(kind, level, message));
}
/// RAII handle for one active warning. Drop clears the `kind` from the
/// banner. Obtained from [`set_warning`].
/// registry. Obtained from [`set_warning`].
#[must_use = "dropping the guard immediately clears the warning; bind it for as long as the condition holds"]
pub struct WarningGuard {
kind: &'static str,

15
hive-c0re/src/swarm_agent_status.rs
View file

@ -27,9 +27,9 @@ use crate::stats::sweep_health::{self, SweepHealth};
/// cadence than the hive's own.
pub const PUBLISH_INTERVAL: Duration = crate::swarm_status::PUBLISH_INTERVAL;
/// Consecutive failed sweeps before the dashboard banners. Mirrors
/// Consecutive failed sweeps before the sweep raises a hive warning. Mirrors
/// [`crate::swarm_status`]'s constant of the same name and purpose.
const FAILURES_BEFORE_BANNER: u32 = 3;
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Start the publish loop, if this deployment wired up a swarm queue.
///
@ -44,8 +44,8 @@ const FAILURES_BEFORE_BANNER: u32 = 3;
/// deploy-event listener does.
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
// Already bannered once by `swarm_status::spawn`, which runs the
// identical check — a second banner for the same missing env var
// Already warned once by `swarm_status::spawn`, which runs the
// identical check — a second warning for the same missing env var
// would just be the same fact told twice.
return;
};
@ -55,8 +55,11 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
return;
};
let mut health =
SweepHealth::new("swarm_agent_status_publish", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new(
"swarm_agent_status_publish",
"warn",
FAILURES_BEFORE_WARNING,
);
loop {
match publish_all(&client, &hive).await {
Ok(published) => {

12
hive-c0re/src/swarm_notices.rs
View file

@ -14,7 +14,7 @@
//! coordinates is a silent no-op (the ordinary case). A hive that cannot
//! reach the queue loses the swarm's visibility of the notice, not the
//! host's — `warn!` fires on every failed attempt regardless, and the
//! dashboard banners only after [`FAILURES_BEFORE_BANNER`] consecutive
//! raises a hive warning only after [`FAILURES_BEFORE_WARNING`] consecutive
//! misses, the same debounce shape [`crate::swarm_status`] uses and for
//! the same reason: a `warn!` that fires every call for three weeks is
//! indistinguishable from silence in practice.
@ -24,10 +24,10 @@ use tokio::sync::{Mutex, OnceCell};
use crate::stats::sweep_health::{self, SweepHealth};
/// Consecutive failed publishes before the dashboard banners — same
/// Consecutive failed publishes before the sweep raises a hive warning — same
/// value [`crate::swarm_status`] uses and for the same reason: a debounce
/// against one blip flapping a banner an operator learns to ignore.
const FAILURES_BEFORE_BANNER: u32 = 3;
/// against one blip flapping a warning an operator learns to ignore.
const FAILURES_BEFORE_WARNING: u32 = 3;
static HEALTH: OnceCell<Mutex<SweepHealth>> = OnceCell::const_new();
@ -37,7 +37,7 @@ async fn health() -> tokio::sync::MutexGuard<'static, SweepHealth> {
Mutex::new(SweepHealth::new(
"swarm_notices_publish",
"warn",
FAILURES_BEFORE_BANNER,
FAILURES_BEFORE_WARNING,
))
})
.await
@ -57,7 +57,7 @@ pub async fn notify(subsystem: &str, key: Option<String>, summary: String, sourc
return;
};
// Same absent-name condition `swarm_status` bails on — that module
// already banners it under `swarm_status_config` the first time
// already warns it under `swarm_status_config` the first time
// either of us hits it; nothing more to add from here.
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
return;

30
hive-c0re/src/swarm_queue.rs
View file

@ -44,15 +44,15 @@ const FIRST_RETRY_DELAY: Duration = Duration::from_secs(5);
/// The longest wait between two attempts.
const MAX_RETRY_DELAY: Duration = Duration::from_mins(5);
/// The banner kind a failed attempt raises, cleared by the next success.
const BANNER: &str = "swarm_queue_config";
/// The warning kind a failed attempt raises, cleared by the next success.
const WARNING_KIND: &str = "swarm_queue_config";
static CLIENT: Retrying<async_nats::Client> = Retrying::new(FIRST_RETRY_DELAY, MAX_RETRY_DELAY);
/// The shared swarm-queue client, or `None` when there is none right now.
///
/// `None` covers "no queue configured" (the ordinary case, logged once at
/// `info`), a deployment bug in the coordinates (bannered, and permanent for
/// `info`), a deployment bug in the coordinates (warned, and permanent for
/// this process), and an attempt that failed or is still backing off. A
/// caller with `None` should just skip whatever it was about to publish.
/// Never waits out a backoff: see [`wait_for_client`] for that.
@ -95,12 +95,12 @@ enum Outcome<T> {
enum Slot<T> {
Off,
Ready(T),
/// `failures` consecutive attempts have failed. `banner` holds the
/// `failures` consecutive attempts have failed. `warning` holds the
/// dashboard warning until the slot leaves this state.
Failing {
failures: u32,
not_before: Instant,
banner: Option<crate::warnings::WarningGuard>,
warning: Option<crate::warnings::WarningGuard>,
},
}
@ -163,22 +163,22 @@ impl<T: Clone> Retrying<T> {
let not_before = Instant::now() + delay;
let message = format!("swarm queue is off, retrying in {delay:?}: {e:#}");
tracing::warn!(failures, "{message}");
// Updating the held guard keeps the banner up without a
// Updating the held guard keeps the warning up without a
// moment where it is cleared and re-raised.
let banner = match slot.take() {
let warning = match slot.take() {
Some(Slot::Failing {
banner: Some(banner),
warning: Some(warning),
..
}) => {
banner.update("warn", message);
banner
warning.update("warn", message);
warning
}
_ => crate::warnings::set_warning(BANNER, "warn", message),
_ => crate::warnings::set_warning(WARNING_KIND, "warn", message),
};
*slot = Some(Slot::Failing {
failures,
not_before,
banner: Some(banner),
warning: Some(warning),
});
Outcome::NotBefore(not_before)
}
@ -204,7 +204,11 @@ async fn connect_once() -> Attempt<async_nats::Client> {
}
Err(e) => {
// A deployment bug, which no retry clears.
crate::warnings::set_boot_warning(BANNER, "warn", format!("swarm queue is off: {e:#}"));
crate::warnings::set_boot_warning(
WARNING_KIND,
"warn",
format!("swarm queue is off: {e:#}"),
);
return Attempt::Off;
}
};

14
hive-c0re/src/swarm_status.rs
View file

@ -47,18 +47,18 @@ use crate::stats::sweep_health::{self, SweepHealth};
/// other silently re-tunes the swarm's definition of "quiet".
pub const PUBLISH_INTERVAL: Duration = Duration::from_mins(1);
/// Consecutive failed publishes before the dashboard banners.
/// Consecutive failed publishes before the sweep raises a hive warning.
///
/// At [`PUBLISH_INTERVAL`] this is ~3 minutes of genuine failure, so a
/// blip does not flap a banner an operator learns to ignore.
const FAILURES_BEFORE_BANNER: u32 = 3;
/// blip does not flap a warning an operator learns to ignore.
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Start the publish loop, if this deployment wired up a swarm queue.
///
/// The connect itself is shared with every other swarm-queue consumer in
/// this process — see [`crate::swarm_queue`] for why one connection and
/// not one per consumer, and for where "no queue configured" vs. "queue
/// configured but unreachable" gets bannered. This function only decides
/// configured but unreachable" gets warned. This function only decides
/// whether *status* has anything to offer once a client exists.
pub fn spawn(
coord: std::sync::Arc<crate::coordinator::Coordinator>,
@ -76,7 +76,7 @@ pub fn spawn(
tokio::spawn(async move {
let Some(client) = crate::swarm_queue::wait_for_client().await else {
// No queue configured, already logged or bannered by the
// No queue configured, already logged or warned by the
// shared connector; nothing left to report here.
return;
};
@ -117,7 +117,7 @@ pub fn spawn(
shutdown.clone(),
));
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_WARNING);
loop {
match publish(&client, &hive).await {
Ok(()) => health.record_ok(),
@ -197,7 +197,7 @@ async fn drain_swarm_events(
let mut sub = match client.subscribe(subject).await {
Ok(sub) => sub,
Err(e) => {
// Warn rather than a boot banner: the hive is fully functional
// Warn rather than a boot warning: the hive is fully functional
// without this, it just falls back to learning about knowledge
// changes at its next boot.
tracing::warn!(

2
hive-c0re/src/workers/crash_watch.rs
View file

@ -149,7 +149,7 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
// The one place this fires: a swarm-level grafana alert
// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`)
// matches this exact message against the log store and pages the
// operator directly — no agent (and no per-hive dashboard banner)
// operator directly — no agent (and no per-hive warning)
// in the loop. See `docs/scheduler/observability.md`.
tracing::warn!(agent = %stopped, "container crash detected");
}

2
hive-c0re/src/workers/permissions_import.rs
View file

@ -181,7 +181,7 @@ impl ImportOutcome {
matches!(self, Self::NothingPending | Self::Imported)
}
/// Why reading is not allowed, for the caller's log and banner.
/// Why reading is not allowed, for the caller's log and warning.
#[must_use]
pub fn into_error(self) -> Option<anyhow::Error> {
match self {

4
hive-c0re/src/workers/swarm_limits.rs
View file

@ -24,11 +24,11 @@ use crate::stats::sweep_health::{self, SweepHealth};
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
const FAILURES_BEFORE_BANNER: u32 = 3;
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Read the caps now and every [`READ_INTERVAL`] until shutdown.
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_WARNING);
loop {
match sync(&client).await {
Ok(()) => health.record_ok(),

4
hive-c0re/src/workers/swarm_permissions.rs
View file

@ -30,7 +30,7 @@ use crate::stats::sweep_health::{self, SweepHealth};
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
const FAILURES_BEFORE_BANNER: u32 = 3;
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Send the meta repo's grants once, then read the permissions now and every
/// [`READ_INTERVAL`] until shutdown.
@ -40,7 +40,7 @@ pub async fn follow(
hive: String,
mut shutdown: tokio::sync::watch::Receiver<bool>,
) {
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_WARNING);
loop {
let import = super::permissions_import::send_pending(&client, &hive).await;
let pass = if import.read_allowed() {

2
hive-c0re/src/workers/wanted.rs
View file

@ -76,7 +76,7 @@ pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
let client =
match tokio::time::timeout_at(deadline, crate::swarm_queue::wait_for_client()).await {
// Absent — already reported by the shared connector (an `info` log or
// a `swarm_queue_config` banner).
// a `swarm_queue_config` warning).
Ok(None) => return Ok(()),
Ok(Some(client)) => client,
Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()),