vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

container stats: measure agent disk usage through hive-priv, unknown on failure

Browse source 
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.

Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.

hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.

`Coordinator::agent_work_dir` had no other caller and is removed.

Closes #4956
This commit is contained in:
atlas 2026-10-05 16:12:33 +02:00 • committed by mara
commit 98ccc31e2b
10 changed files with 244 additions and 179 deletions
Download patch file Download diff file Expand all files Collapse all files

8
hive-c0re/src/coordinator.rs
View file

@ -1130,14 +1130,6 @@ impl Coordinator {
crate::paths::agent_state_dir(name).join("state")
}
/// Per-agent working dir for worktrees and build dirs. Bind-mounted RW
/// into the agent container at `/agents/{name}/work`. Survives
/// destroy/recreate. A sibling of `state/` rather than inside it, so a
/// claude started under `work/` never walks up into `state/CLAUDE.md`.
pub fn agent_work_dir(name: &hive_types::Ident) -> PathBuf {
crate::paths::agent_state_dir(name).join(crate::paths::WORK_SUBDIR)
}
/// Per-agent harness-internal state dir. Bind-mounted RW into the
/// agent container at `/agents/{name}/harness`. Holds sqlite dbs
/// and config files owned by the harness (`hyperhive-events.sqlite`,

4
hive-c0re/src/main.rs
View file

@ -399,8 +399,8 @@ async fn cmd_serve(
// (turn-stats.sqlite has no vacuum — it's one tiny row per turn,
// ~hundreds of KB, and the /stats + hive-stats views read it
// directly; pruning it would just lose trend history for no gain.)
// Slow per-container disk sampler: a `du` of each agent's state/ + work/
// + writable rootfs every ~5 min, cached so the 5s container-load
// Slow per-container disk sampler: hive-priv measures each agent's state/
// + work/ + writable rootfs every 30 min, cached so the 5s container-load
// poll stays cheap cgroup-only reads. Feeds `disk_bytes` on the LOAD
// tab. See container_stats::disk_sampler_loop.
crate::container_stats::spawn_disk_sampler();

19
hive-c0re/src/priv_client.rs
View file

@ -429,6 +429,25 @@ pub async fn read_subvolume_usage(agent_name: &str) -> Result<(u64, u64)> {
parse_qgroup_usage(&stdout).with_context(|| format!("parse qgroup usage for {agent_name}"))
}
/// Measure an agent's disk usage in bytes (`state/` + `work/` + container
/// rootfs, one bounded `du` run by hive-priv as root).
///
/// # Errors
/// Returns an error if the hive-priv call fails, the `du` walk times out or
/// exits non-zero, or the returned total isn't a number.
pub async fn measure_agent_disk(agent_name: &str) -> Result<u64> {
let (stdout, _) = check(
call(&PrivRequest::MeasureAgentDisk {
agent_name: agent_name.to_owned(),
})
.await?,
)?;
stdout
.trim()
.parse()
.with_context(|| format!("parse disk usage for {agent_name}: {stdout:?}"))
}
/// Set or clear a btrfs qgroup size limit on an agent's state subvolume.
/// `limit_bytes = None` clears it. Requires quota enabled. Via hive-priv.
///

187
hive-c0re/src/stats/container_stats.rs
View file

@ -1,4 +1,5 @@
//! Live per-agent-container resource load (CPU + memory) for the
//! Live per-agent-container resource load (CPU + memory, plus the slow
//! disk sample from hive-priv in [`disk_sampler_loop`]) for the
//! dashboard. Reads cgroup v2 stat files for each running agent
//! machine directly — same privsep-clean posture as the turn-stats
//! sqlite reads (`cpu.stat` / `memory.*` are world-readable; no
@ -57,121 +58,47 @@ pub struct ContainerResource {
/// Memory ceiling (`memory.max`), bytes. `None` when unlimited
/// (the file reads `max`).
pub mem_max_bytes: Option<u64>,
/// Last-sampled total disk usage — the agent's `state/` and `work/` dirs plus the
/// container's writable rootfs, excluding the shared read-only nix
/// store — in bytes. `None` until the slow background sampler
/// ([`disk_sampler_loop`]) has populated the cache at least once.
/// Decoupled from this hot per-poll path because a `du` tree-walk is
/// far too expensive at the 5s `/api/container-load` cadence.
/// Last-sampled total disk usage — the agent's `state/` and `work/` dirs
/// plus the container's writable rootfs, excluding the shared read-only
/// nix store — in bytes. `None` when it is unknown: the slow background
/// sampler ([`disk_sampler_loop`]) has not measured this agent yet, or
/// its last measurement failed. Never 0 for a failed measurement.
pub disk_bytes: Option<u64>,
}
/// Cadence for the slow disk sampler. Disk size changes slowly relative
/// to CPU/mem, and a `du` walk is expensive, so this runs far less often
/// than the per-poll cgroup reads — the row just carries the last value.
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(5);
/// Cadence for the slow disk sampler. Each pass has hive-priv walk every
/// agent's full tree as root, which is real I/O on the disk the agents
/// work on, and the sizes move over hours, not minutes.
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(30);
/// Root of the per-container writable rootfs trees that nixos-containers
/// manages (`/var/lib/nixos-containers/<machine>`).
const NIXOS_CONTAINERS_ROOT: &str = "/var/lib/nixos-containers";
/// Last-sampled per-agent disk usage (bytes), keyed by agent name.
/// Written by [`disk_sampler_loop`] (~every 5 min), read by [`gather`]
/// so the hot poll never runs `du`.
fn disk_cache() -> &'static RwLock<HashMap<String, u64>> {
static CACHE: OnceLock<RwLock<HashMap<String, u64>>> = OnceLock::new();
/// Last per-agent disk measurement, keyed by agent name; `None` is a failed
/// one. Written by [`disk_sampler_loop`], read by [`gather`] so the hot
/// poll never measures.
fn disk_cache() -> &'static RwLock<HashMap<String, Option<u64>>> {
static CACHE: OnceLock<RwLock<HashMap<String, Option<u64>>>> = OnceLock::new();
CACHE.get_or_init(|| RwLock::new(HashMap::new()))
}
/// `du -sxb <path>` → apparent total bytes. `-s` summary, `-b` apparent
/// size, `-x` stay on one filesystem so the walk never descends into the
/// bind-mounted read-only shared nix store (or any other bind mount) —
/// exactly the per-container-only measure we want. Best-effort: the
/// caller treats `None` as a 0 contribution either way, but the two
/// `None` causes are logged differently. A path that plain doesn't exist
/// (`kept_state_names` includes destroyed-but-kept tombstones with no
/// rootfs, and a fresh agent has no state dir yet) is the expected,
/// silent case — every sampler pass would otherwise warn on it forever.
/// A path that exists but `du` still can't read — permission, I/O error,
/// unparseable output — is warn!'d, since that 0 is not "nothing there",
/// it's a read we should have gotten.
async fn du_bytes(path: &std::path::Path) -> Option<u64> {
match std::fs::symlink_metadata(path) {
Ok(_) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return None,
Err(e) => {
tracing::warn!(path = %path.display(), error = %e, "du: path stat failed");
return None;
}
}
let out = match tokio::process::Command::new("du")
.arg("-sxb")
.arg(path)
.output()
.await
{
Ok(out) => out,
Err(e) => {
tracing::warn!(path = %path.display(), error = %e, "du: failed to spawn");
return None;
}
};
if !out.status.success() {
tracing::warn!(
path = %path.display(),
status = %out.status,
stderr = %String::from_utf8_lossy(&out.stderr),
"du: exited non-zero"
);
return None;
}
let stdout = String::from_utf8_lossy(&out.stdout);
let Some(first) = stdout.split_whitespace().next() else {
tracing::warn!(path = %path.display(), stdout = %stdout, "du: empty output");
return None;
};
match first.parse::<u64>() {
Ok(bytes) => Some(bytes),
Err(e) => {
tracing::warn!(path = %path.display(), stdout = %stdout, error = %e, "du: unparseable output");
None
}
}
/// Total disk usage of `name`'s `state/` + `work/` + container rootfs,
/// measured by hive-priv as root (`hive-core` can read none of the rootfs
/// and not all of the agent-owned dirs). `None` on any failure.
async fn measure_agent_disk(name: &hive_types::Ident) -> Option<u64> {
disk_reading(
name,
crate::priv_client::measure_agent_disk(name.as_str()).await,
)
}
/// Host paths summed into an agent's disk usage: the agent's host-side
/// state and work dirs (`/var/lib/hyperhive/agents/<name>/{state,work}`) plus
/// the container's writable rootfs (`/var/lib/nixos-containers/h-<name>`).
fn agent_disk_paths(name: &hive_types::Ident) -> [PathBuf; 3] {
[
Coordinator::agent_notes_dir(name),
Coordinator::agent_work_dir(name),
PathBuf::from(format!("{NIXOS_CONTAINERS_ROOT}/h-{name}")),
]
/// A failed measurement is unknown, never a 0 contribution.
fn disk_reading(name: &hive_types::Ident, measured: anyhow::Result<u64>) -> Option<u64> {
measured
.inspect_err(|e| {
tracing::warn!(agent = %name, error = %format!("{e:#}"), "disk usage measurement failed");
})
.ok()
}
/// Total per-agent disk over [`agent_disk_paths`]. Each path is measured
/// with `du -sxb` so the shared nix store and other bind mounts are
/// excluded. A path that doesn't exist contributes 0.
///
/// The state dir is resolved via [`Coordinator::agent_notes_dir`] — the same
/// host path the dashboard reads agent state from. Hardcoding the
/// container-internal bind-mount path (`/agents/<name>/state`) instead made
/// `du` fail host-side (that path only exists inside the container), so every
/// agent's state-dir contribution was 0; with the writable rootfs nearly empty
/// (almost everything is bind-mounted), that surfaced as all agents reporting
/// 0 disk.
async fn measure_agent_disk(name: &hive_types::Ident) -> u64 {
let mut total = 0u64;
for path in agent_disk_paths(name) {
if let Some(bytes) = du_bytes(&path).await {
total = total.saturating_add(bytes);
}
}
total
}
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], recompute disk usage
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], remeasure disk usage
/// for every kept-state agent and refresh [`disk_cache`]. The first pass
/// runs immediately so the dashboard shows disk shortly after boot.
/// Runs forever; spawn once at hive-c0re startup via [`spawn_disk_sampler`].
@ -308,7 +235,7 @@ pub async fn gather() -> Vec<ContainerResource> {
// Snapshot the slow disk cache once (cheap clone of a small map) so
// the per-agent loop below is a plain map lookup — no `du` on this
// hot path. Agents not yet sampled simply carry `disk_bytes = None`.
// hot path. Agents not yet measured carry `disk_bytes = None` too.
let disk = disk_cache().read().map(|c| c.clone()).unwrap_or_default();
let mut out: Vec<ContainerResource> = Vec::with_capacity(candidates.len());
@ -335,7 +262,7 @@ pub async fn gather() -> Vec<ContainerResource> {
mem_current_bytes: read_u64(&dir.join("memory.current")).unwrap_or(0),
mem_peak_bytes: read_u64(&dir.join("memory.peak")),
mem_max_bytes: read_mem_max(&dir.join("memory.max")),
disk_bytes: disk.get(name).copied(),
disk_bytes: disk.get(name).copied().flatten(),
});
}
out
@ -343,8 +270,6 @@ pub async fn gather() -> Vec<ContainerResource> {
#[cfg(test)]
mod tests {
use std::os::unix::fs::PermissionsExt;
use super::*;
#[test]
@ -364,44 +289,16 @@ mod tests {
}
#[test]
fn disk_paths_cover_state_work_and_rootfs() {
fn failed_disk_measurement_is_unknown_not_zero() {
let name = hive_types::Ident::parse("iris").expect("valid ident");
assert_eq!(
agent_disk_paths(&name),
[
Coordinator::agent_notes_dir(&name),
Coordinator::agent_work_dir(&name),
PathBuf::from(format!("{NIXOS_CONTAINERS_ROOT}/h-iris")),
]
disk_reading(&name, Err(anyhow::anyhow!("du for iris timed out"))),
None
);
assert_eq!(disk_reading(&name, Ok(0)), Some(0));
assert_eq!(
disk_reading(&name, Ok(13_000_000_000)),
Some(13_000_000_000)
);
}
#[tokio::test]
async fn du_bytes_missing_path_is_none() {
let dir = tempfile::tempdir().expect("tempdir");
assert_eq!(du_bytes(&dir.path().join("does-not-exist")).await, None);
}
#[tokio::test]
async fn du_bytes_reads_a_real_dir() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join("f"), b"hello").expect("write");
assert!(du_bytes(dir.path()).await.is_some_and(|b| b > 0));
}
#[tokio::test]
async fn du_bytes_unreadable_dir_is_none() {
// A dir that exists but can't be entered (mode 000) is the case
// that must be distinguished from "doesn't exist" — both currently
// become a 0 contribution, but only this one is worth a warn.
let dir = tempfile::tempdir().expect("tempdir");
let unreadable = dir.path().join("locked");
std::fs::create_dir(&unreadable).expect("mkdir");
std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o000))
.expect("chmod");
let result = du_bytes(&unreadable).await;
std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o755))
.expect("restore perms so tempdir cleanup can remove it");
assert_eq!(result, None);
}
}