hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -44,15 +44,15 @@ const FIRST_RETRY_DELAY: Duration = Duration::from_secs(5);
|
|||
/// The longest wait between two attempts.
|
||||
const MAX_RETRY_DELAY: Duration = Duration::from_mins(5);
|
||||
|
||||
/// The banner kind a failed attempt raises, cleared by the next success.
|
||||
const BANNER: &str = "swarm_queue_config";
|
||||
/// The warning kind a failed attempt raises, cleared by the next success.
|
||||
const WARNING_KIND: &str = "swarm_queue_config";
|
||||
|
||||
static CLIENT: Retrying<async_nats::Client> = Retrying::new(FIRST_RETRY_DELAY, MAX_RETRY_DELAY);
|
||||
|
||||
/// The shared swarm-queue client, or `None` when there is none right now.
|
||||
///
|
||||
/// `None` covers "no queue configured" (the ordinary case, logged once at
|
||||
/// `info`), a deployment bug in the coordinates (bannered, and permanent for
|
||||
/// `info`), a deployment bug in the coordinates (warned, and permanent for
|
||||
/// this process), and an attempt that failed or is still backing off. A
|
||||
/// caller with `None` should just skip whatever it was about to publish.
|
||||
/// Never waits out a backoff: see [`wait_for_client`] for that.
|
||||
|
|
@ -95,12 +95,12 @@ enum Outcome<T> {
|
|||
enum Slot<T> {
|
||||
Off,
|
||||
Ready(T),
|
||||
/// `failures` consecutive attempts have failed. `banner` holds the
|
||||
/// `failures` consecutive attempts have failed. `warning` holds the
|
||||
/// dashboard warning until the slot leaves this state.
|
||||
Failing {
|
||||
failures: u32,
|
||||
not_before: Instant,
|
||||
banner: Option<crate::warnings::WarningGuard>,
|
||||
warning: Option<crate::warnings::WarningGuard>,
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -163,22 +163,22 @@ impl<T: Clone> Retrying<T> {
|
|||
let not_before = Instant::now() + delay;
|
||||
let message = format!("swarm queue is off, retrying in {delay:?}: {e:#}");
|
||||
tracing::warn!(failures, "{message}");
|
||||
// Updating the held guard keeps the banner up without a
|
||||
// Updating the held guard keeps the warning up without a
|
||||
// moment where it is cleared and re-raised.
|
||||
let banner = match slot.take() {
|
||||
let warning = match slot.take() {
|
||||
Some(Slot::Failing {
|
||||
banner: Some(banner),
|
||||
warning: Some(warning),
|
||||
..
|
||||
}) => {
|
||||
banner.update("warn", message);
|
||||
banner
|
||||
warning.update("warn", message);
|
||||
warning
|
||||
}
|
||||
_ => crate::warnings::set_warning(BANNER, "warn", message),
|
||||
_ => crate::warnings::set_warning(WARNING_KIND, "warn", message),
|
||||
};
|
||||
*slot = Some(Slot::Failing {
|
||||
failures,
|
||||
not_before,
|
||||
banner: Some(banner),
|
||||
warning: Some(warning),
|
||||
});
|
||||
Outcome::NotBefore(not_before)
|
||||
}
|
||||
|
|
@ -204,7 +204,11 @@ async fn connect_once() -> Attempt<async_nats::Client> {
|
|||
}
|
||||
Err(e) => {
|
||||
// A deployment bug, which no retry clears.
|
||||
crate::warnings::set_boot_warning(BANNER, "warn", format!("swarm queue is off: {e:#}"));
|
||||
crate::warnings::set_boot_warning(
|
||||
WARNING_KIND,
|
||||
"warn",
|
||||
format!("swarm queue is off: {e:#}"),
|
||||
);
|
||||
return Attempt::Off;
|
||||
}
|
||||
};
|
||||
|
|
|
|||
Loading…
Reference in a new issue