Per operator request: instead of a disk-specific alert, surface a generic
server-warnings banner at the very top of every page, so new system
warnings can be added backend-side with no frontend change.
- hive-c0re `host_stats`: `server_warnings() -> Vec<ServerWarning>`
(`{ kind, level, message }`). The threshold logic lives server-side; the
host disk-pressure check (a `statvfs` probe of `/nix`: ≥85% used → warn,
≥95% → crit) is the first and only producer today. No new deps (libc).
- `/api/state` carries `server_warnings` (replaces the disk-specific
field). Empty when all clear.
- frontend: `renderServerWarnings` / `initServerWarnings` in `common.js`
inject a sticky top-of-<body> banner and render the list, coloured by
`level`. Wired on every page — dashboard (live, via refreshState),
FL0W, L0GS, H0M3. No per-warning frontend code; adding a warning kind
is a pure backend change.
cargo check/clippy/fmt + npm run build green. Closes #1518.
124 lines
5 KiB
Rust
124 lines
5 KiB
Rust
//! Host-system probes and the derived **server-warning** list that the
|
||
//! dashboard renders as a top-of-page banner.
|
||
//!
|
||
//! The first (and currently only) producer is a host disk-usage check:
|
||
//! the nix store filling up is what ENOSPC-failed CI before the GC
|
||
//! guardrails were documented. hive-c0re runs on the host (not inside a
|
||
//! container), so it can `statvfs` the store path directly and warn the
|
||
//! operator *before* an ENOSPC, not after.
|
||
//!
|
||
//! [`server_warnings`] is the public surface: it returns a flat list of
|
||
//! [`ServerWarning`]s for `/api/state`. The dashboard renders whatever it
|
||
//! returns, coloured by `level`, so adding a new system warning (memory
|
||
//! pressure, a failed unit, …) is a backend-only change — no frontend
|
||
//! edit. Keep producers cheap; this runs on every `/api/state` assembly.
|
||
|
||
use serde::Serialize;
|
||
|
||
/// One server-level warning for the dashboard's top-of-page banner.
|
||
#[derive(Debug, Clone, Serialize)]
|
||
pub struct ServerWarning {
|
||
/// Stable kind id (e.g. `"disk_pressure"`) — lets the frontend dedupe
|
||
/// or special-case without parsing the message.
|
||
pub kind: &'static str,
|
||
/// Severity: `"warn"` (amber) or `"crit"` (red). The banner picks its
|
||
/// colour from this; everything else is just the message text.
|
||
pub level: &'static str,
|
||
/// Human-readable, already-formatted message shown in the banner.
|
||
pub message: String,
|
||
}
|
||
|
||
/// Percent-used past which the host nix store earns a disk-pressure
|
||
/// warning; above [`DISK_CRIT_PCT`] it escalates to `crit`.
|
||
const DISK_WARN_PCT: f64 = 85.0;
|
||
const DISK_CRIT_PCT: f64 = 95.0;
|
||
|
||
/// Collect the current server-level warnings for the dashboard banner.
|
||
/// Each producer pushes zero or more [`ServerWarning`]s; the frontend
|
||
/// renders whatever this returns. Cheap to call on every `/api/state`
|
||
/// assembly (currently a single `statvfs`).
|
||
#[must_use]
|
||
pub fn server_warnings() -> Vec<ServerWarning> {
|
||
let mut out = Vec::new();
|
||
if let Some(d) = nix_disk_usage()
|
||
&& d.used_pct >= DISK_WARN_PCT
|
||
{
|
||
#[allow(
|
||
clippy::cast_precision_loss,
|
||
reason = "byte counts stay well under f64's 2^53 exact-integer range, so this GiB conversion loses no precision"
|
||
)]
|
||
let free_gib = d.free_bytes as f64 / (1024.0 * 1024.0 * 1024.0);
|
||
out.push(ServerWarning {
|
||
kind: "disk_pressure",
|
||
level: if d.used_pct >= DISK_CRIT_PCT {
|
||
"crit"
|
||
} else {
|
||
"warn"
|
||
},
|
||
message: format!(
|
||
"host nix store {:.0}% full ({free_gib:.1} GiB free) \
|
||
— garbage-collect the store before it runs out of space",
|
||
d.used_pct
|
||
),
|
||
});
|
||
}
|
||
out
|
||
}
|
||
|
||
/// Disk usage for the filesystem backing the host nix store — internal to
|
||
/// the disk-pressure producer above.
|
||
struct DiskUsage {
|
||
/// Space available to unprivileged writers, in bytes.
|
||
free_bytes: u64,
|
||
/// Percentage used, 0–100. Mirrors `df`'s use% — `used / (used +
|
||
/// available)` — so the threshold means what the operator sees in `df`.
|
||
used_pct: f64,
|
||
}
|
||
|
||
/// Probe disk usage for the filesystem containing the nix store (`/nix`),
|
||
/// falling back to `/` when `/nix` isn't its own mount. Returns `None` if
|
||
/// the `statvfs` syscall fails (path missing, permission, etc.).
|
||
fn nix_disk_usage() -> Option<DiskUsage> {
|
||
disk_usage("/nix").or_else(|| disk_usage("/"))
|
||
}
|
||
|
||
fn disk_usage(path: &str) -> Option<DiskUsage> {
|
||
let c_path = std::ffi::CString::new(path).ok()?;
|
||
// SAFETY: `statvfs` reads only through the valid NUL-terminated
|
||
// `c_path` pointer and writes into the zeroed `stat` we own. We check
|
||
// the return code before reading any field.
|
||
let mut stat: libc::statvfs = unsafe { std::mem::zeroed() };
|
||
let rc = unsafe { libc::statvfs(c_path.as_ptr(), &raw mut stat) };
|
||
if rc != 0 {
|
||
return None;
|
||
}
|
||
// statvfs fields are `c_ulong` (== u64 on the x86_64-linux host this
|
||
// runs on); the arithmetic below stays in that native width.
|
||
let frsize = stat.f_frsize;
|
||
let total_blocks = stat.f_blocks;
|
||
let free_blocks = stat.f_bfree;
|
||
let avail_blocks = stat.f_bavail;
|
||
if total_blocks == 0 || frsize == 0 {
|
||
return None;
|
||
}
|
||
let free_bytes = avail_blocks.saturating_mul(frsize);
|
||
// df's use%: used / (used + available). `used` counts root-reserved
|
||
// blocks (total - bfree); `available` is the unprivileged free
|
||
// (bavail), so the percentage matches what `df` reports.
|
||
let used_blocks = total_blocks.saturating_sub(free_blocks);
|
||
let capacity = used_blocks.saturating_add(avail_blocks);
|
||
let used_pct = if capacity == 0 {
|
||
0.0
|
||
} else {
|
||
#[allow(
|
||
clippy::cast_precision_loss,
|
||
reason = "block counts stay well under f64's 2^53 exact-integer range, so this percentage computation loses no precision"
|
||
)]
|
||
let raw = used_blocks as f64 / capacity as f64 * 100.0;
|
||
raw
|
||
};
|
||
Some(DiskUsage {
|
||
free_bytes,
|
||
used_pct,
|
||
})
|
||
}
|