Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
5 changed files with 468 additions and 322 deletions

View file

@ -149,7 +149,7 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
// The one place this fires: a swarm-level grafana alert
// (`hh-container-crashed` in `nix/host-modules/swarm-grafana.nix`)
// matches this exact message against the log store and pages the
// operator directly — no agent (and no per-hive dashboard banner)
// operator directly — no agent (and no per-hive warning)
// in the loop. See `docs/scheduler/observability.md`.
tracing::warn!(agent = %stopped, "container crash detected");
}

View file

@ -181,7 +181,7 @@ impl ImportOutcome {
matches!(self, Self::NothingPending | Self::Imported)
}
/// Why reading is not allowed, for the caller's log and banner.
/// Why reading is not allowed, for the caller's log and warning.
#[must_use]
pub fn into_error(self) -> Option<anyhow::Error> {
match self {

View file

@ -24,11 +24,11 @@ use crate::stats::sweep_health::{self, SweepHealth};
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
const FAILURES_BEFORE_BANNER: u32 = 3;
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Read the caps now and every [`READ_INTERVAL`] until shutdown.
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_WARNING);
loop {
match sync(&client).await {
Ok(()) => health.record_ok(),

View file

@ -30,7 +30,7 @@ use crate::stats::sweep_health::{self, SweepHealth};
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
const FAILURES_BEFORE_BANNER: u32 = 3;
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Send the meta repo's grants once, then read the permissions now and every
/// [`READ_INTERVAL`] until shutdown.
@ -40,7 +40,7 @@ pub async fn follow(
hive: String,
mut shutdown: tokio::sync::watch::Receiver<bool>,
) {
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new("swarm_permissions_read", "warn", FAILURES_BEFORE_WARNING);
loop {
let import = super::permissions_import::send_pending(&client, &hive).await;
let pass = if import.read_allowed() {

View file

@ -76,7 +76,7 @@ pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
let client =
match tokio::time::timeout_at(deadline, crate::swarm_queue::wait_for_client()).await {
// Absent — already reported by the shared connector (an `info` log or
// a `swarm_queue_config` banner).
// a `swarm_queue_config` warning).
Ok(None) => return Ok(()),
Ok(Some(client)) => client,
Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()),