vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

14
hive-c0re/src/swarm_status.rs
View file

@ -47,18 +47,18 @@ use crate::stats::sweep_health::{self, SweepHealth};
/// other silently re-tunes the swarm's definition of "quiet".
pub const PUBLISH_INTERVAL: Duration = Duration::from_mins(1);
/// Consecutive failed publishes before the dashboard banners.
/// Consecutive failed publishes before the sweep raises a hive warning.
///
/// At [`PUBLISH_INTERVAL`] this is ~3 minutes of genuine failure, so a
/// blip does not flap a banner an operator learns to ignore.
const FAILURES_BEFORE_BANNER: u32 = 3;
/// blip does not flap a warning an operator learns to ignore.
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Start the publish loop, if this deployment wired up a swarm queue.
///
/// The connect itself is shared with every other swarm-queue consumer in
/// this process — see [`crate::swarm_queue`] for why one connection and
/// not one per consumer, and for where "no queue configured" vs. "queue
/// configured but unreachable" gets bannered. This function only decides
/// configured but unreachable" gets warned. This function only decides
/// whether *status* has anything to offer once a client exists.
pub fn spawn(
coord: std::sync::Arc<crate::coordinator::Coordinator>,
@ -76,7 +76,7 @@ pub fn spawn(
tokio::spawn(async move {
let Some(client) = crate::swarm_queue::wait_for_client().await else {
// No queue configured, already logged or bannered by the
// No queue configured, already logged or warned by the
// shared connector; nothing left to report here.
return;
};
@ -117,7 +117,7 @@ pub fn spawn(
shutdown.clone(),
));
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_WARNING);
loop {
match publish(&client, &hive).await {
Ok(()) => health.record_ok(),
@ -197,7 +197,7 @@ async fn drain_swarm_events(
let mut sub = match client.subscribe(subject).await {
Ok(sub) => sub,
Err(e) => {
// Warn rather than a boot banner: the hive is fully functional
// Warn rather than a boot warning: the hive is fully functional
// without this, it just falls back to learning about knowledge
// changes at its next boot.
tracing::warn!(