hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
This commit is contained in:
parent
013bbe5adc
commit
4dbcf6fa80
35 changed files with 468 additions and 322 deletions
|
|
@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
|
|||
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
|
||||
/// most a handful per process). [`crate::warnings::set_boot_warning`]
|
||||
/// requires a `'static` kind so distinct orgs/repos don't clobber each
|
||||
/// other's banner entry; leaking a few short strings once per boot is
|
||||
/// other's warning entry; leaking a few short strings once per boot is
|
||||
/// cheap and bounded, unlike a per-request or per-loop-iteration leak.
|
||||
fn static_kind(s: String) -> &'static str {
|
||||
Box::leak(s.into_boxed_str())
|
||||
|
|
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
|
|||
/// confirms the container exists — not that Forgejo is *listening*. A
|
||||
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
|
||||
/// without this gate every `ensure_*` fires at a refused socket and leaves a
|
||||
/// stale "provisioning failed" banner that never clears (the pass is one-shot).
|
||||
/// stale "provisioning failed" warning that never clears (the pass is one-shot).
|
||||
/// Returns whether the endpoint became ready before the deadline; on timeout
|
||||
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real
|
||||
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
|
||||
|
|
@ -393,7 +393,7 @@ pub async fn ensure_all() {
|
|||
}
|
||||
// The container exists, but its HTTP may not be up yet (a rebuild restarts
|
||||
// hive-forge + hive-c0re together). Wait for the API to answer before the
|
||||
// provisioning pass so a boot race doesn't leave stale failure banners.
|
||||
// provisioning pass so a boot race doesn't leave stale failure warnings.
|
||||
wait_until_ready().await;
|
||||
let core_token = match ensure_core_user_and_token().await {
|
||||
Ok(t) => Some(t),
|
||||
|
|
|
|||
Loading…
Reference in a new issue