vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner

Browse source 
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
iris 2026-10-21 17:52:53 +02:00
commit 4dbcf6fa80
35 changed files with 468 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

15
hive-c0re/src/swarm_agent_status.rs
View file

@ -27,9 +27,9 @@ use crate::stats::sweep_health::{self, SweepHealth};
/// cadence than the hive's own.
pub const PUBLISH_INTERVAL: Duration = crate::swarm_status::PUBLISH_INTERVAL;
/// Consecutive failed sweeps before the dashboard banners. Mirrors
/// Consecutive failed sweeps before the sweep raises a hive warning. Mirrors
/// [`crate::swarm_status`]'s constant of the same name and purpose.
const FAILURES_BEFORE_BANNER: u32 = 3;
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Start the publish loop, if this deployment wired up a swarm queue.
///
@ -44,8 +44,8 @@ const FAILURES_BEFORE_BANNER: u32 = 3;
/// deploy-event listener does.
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
// Already bannered once by `swarm_status::spawn`, which runs the
// identical check — a second banner for the same missing env var
// Already warned once by `swarm_status::spawn`, which runs the
// identical check — a second warning for the same missing env var
// would just be the same fact told twice.
return;
};
@ -55,8 +55,11 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
return;
};
let mut health =
SweepHealth::new("swarm_agent_status_publish", "warn", FAILURES_BEFORE_BANNER);
let mut health = SweepHealth::new(
"swarm_agent_status_publish",
"warn",
FAILURES_BEFORE_WARNING,
);
loop {
match publish_all(&client, &hive).await {
Ok(published) => {