diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index 83106e6a..28a982e9 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -292,14 +292,14 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource `service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name` (= the `h-` machine) and the hive `agent` label: -| Metric | Unit | Kind | Source | -| ----------------------------------- | ---- | ------- | -------------------------------------------------------------------------------------------------------------------- | -| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds | -| `container.memory.usage` | `By` | gauge | `memory.current` | -| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) | -| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) | -| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) | -| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) | +| Metric | Unit | Kind | Source | +| ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds | +| `container.memory.usage` | `By` | gauge | `memory.current` | +| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) | +| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) | +| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) | +| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) | The `hyperhive.`-prefixed metrics have no semconv equivalent (memory limit + peak, on-disk footprint, and an instantaneous cpu percent kept diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index 91578cf3..14d1bfae 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -292,6 +292,7 @@ known operations; there is no arbitrary command pass-through: | `EnsureBtrfsQuota` | `btrfs quota enable `. Operator opt-in — enabling forces a full rescan | | `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw /` | | `SetSubvolumeQuota` | `btrfs qgroup limit /` | +| `MeasureAgentDisk` | `du -s -x -b -c -P` over `//state`, `//work` and `/var/lib/nixos-containers/h-`, killed after 10 min. hive-priv derives the paths from the name; the request carries none | | `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r ` — the frozen point-in-time copy a migration streams from | | `DeleteAgentSnapshot` | `btrfs subvolume delete ` | | `SendAgentSnapshotToFile` | `btrfs send [-p ] ` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export | diff --git a/docs/web-ui/dashboard.md b/docs/web-ui/dashboard.md index d316ec74..32381ec4 100644 --- a/docs/web-ui/dashboard.md +++ b/docs/web-ui/dashboard.md @@ -156,10 +156,12 @@ files read-only (world-readable; no `hive-priv`) and skips agents whose scope dir is absent (= not running). Pull-only: `core.js` polls every 5 s **only while the C0NT41N3R L04D sub-tab is active** (CPU needs a fresh sample each refresh), and stops on sub-tab change. Disk size (`disk_bytes`) -rides the same row, but a separate ~5 min background `du` sampler feeds -it (`state/` + `work/` + container writable rootfs, shared nix store excluded via -`du -x`), so the 5 s poll stays cheap cgroup-only reads; the row carries the -last-sampled value (`null` until the first sample). Network isn't tracked +rides the same row, but a separate background sampler feeds it every +30 min: `hive-priv` runs one bounded `du -x` as root over the agent's +`state/` + `work/` + container writable rootfs (shared nix store excluded), +so the 5 s poll stays cheap cgroup-only reads. The row carries the last +measured value; the column shows "unknown" (`null`) before the first +measurement and whenever the last one failed, never 0. Network isn't tracked per container — see `docs/networking/network.md` for the current per-agent netns model. @@ -922,9 +924,10 @@ below — some endpoints aren't in it yet. ~200 ms so the call briefly awaits. Skips non-running agents (no scope dir). No network field — agents share the host netns. `disk_bytes` is the last value from a **separate slow sampler** (not this hot path): - a background `du -sxb` of the agent's `state/` + `work/` + container writable - rootfs every ~5 min, `-x` excluding the shared read-only nix store. - `null` until the first sample lands. + every 30 min `hive-priv` runs a bounded `du -sxb` as root over the agent's + `state/` + `work/` + container writable rootfs, `-x` excluding the shared + read-only nix store (`PrivRequest::MeasureAgentDisk`). `null` means + unknown: not measured yet, or the last measurement failed. - `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder. - `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery failed (clears the failure state so the scheduler retries). diff --git a/frontend/packages/dashboard/src/core.js b/frontend/packages/dashboard/src/core.js index 848626a2..b4be8b03 100644 --- a/frontend/packages/dashboard/src/core.js +++ b/frontend/packages/dashboard/src/core.js @@ -286,14 +286,14 @@ function renderContainerLoad(rows) { : "∞"; tr.append(lim); // Disk: on-disk footprint (state/ + work/ + container writable rootfs, shared - // nix store excluded). Sampled out-of-band every few minutes server-side, - // so it's null until the first sample lands — show an em-dash then. + // nix store excluded). Measured out-of-band every 30 min server-side; + // null means unknown (not measured yet, or the measurement failed). const disk = document.createElement("td"); disk.className = "num"; disk.title = - "state/ + work/ + container writable rootfs (shared nix store excluded); sampled every few minutes"; + "state/ + work/ + container writable rootfs (shared nix store excluded); measured every 30 min"; disk.textContent = - r.disk_bytes != null ? cloadFmtBytes(Number(r.disk_bytes)) : "—"; + r.disk_bytes != null ? cloadFmtBytes(Number(r.disk_bytes)) : "unknown"; tr.append(disk); // Configured CPU / memory caps from ContainerView (resolved effective diff --git a/hive-c0re/src/coordinator.rs b/hive-c0re/src/coordinator.rs index a22e1528..d06f5969 100644 --- a/hive-c0re/src/coordinator.rs +++ b/hive-c0re/src/coordinator.rs @@ -1130,14 +1130,6 @@ impl Coordinator { crate::paths::agent_state_dir(name).join("state") } - /// Per-agent working dir for worktrees and build dirs. Bind-mounted RW - /// into the agent container at `/agents/{name}/work`. Survives - /// destroy/recreate. A sibling of `state/` rather than inside it, so a - /// claude started under `work/` never walks up into `state/CLAUDE.md`. - pub fn agent_work_dir(name: &hive_types::Ident) -> PathBuf { - crate::paths::agent_state_dir(name).join(crate::paths::WORK_SUBDIR) - } - /// Per-agent harness-internal state dir. Bind-mounted RW into the /// agent container at `/agents/{name}/harness`. Holds sqlite dbs /// and config files owned by the harness (`hyperhive-events.sqlite`, diff --git a/hive-c0re/src/main.rs b/hive-c0re/src/main.rs index d8c4ef0f..51dbc06d 100644 --- a/hive-c0re/src/main.rs +++ b/hive-c0re/src/main.rs @@ -399,8 +399,8 @@ async fn cmd_serve( // (turn-stats.sqlite has no vacuum — it's one tiny row per turn, // ~hundreds of KB, and the /stats + hive-stats views read it // directly; pruning it would just lose trend history for no gain.) - // Slow per-container disk sampler: a `du` of each agent's state/ + work/ - // + writable rootfs every ~5 min, cached so the 5s container-load + // Slow per-container disk sampler: hive-priv measures each agent's state/ + // + work/ + writable rootfs every 30 min, cached so the 5s container-load // poll stays cheap cgroup-only reads. Feeds `disk_bytes` on the LOAD // tab. See container_stats::disk_sampler_loop. crate::container_stats::spawn_disk_sampler(); diff --git a/hive-c0re/src/priv_client.rs b/hive-c0re/src/priv_client.rs index 287a5c2e..791782ef 100644 --- a/hive-c0re/src/priv_client.rs +++ b/hive-c0re/src/priv_client.rs @@ -429,6 +429,25 @@ pub async fn read_subvolume_usage(agent_name: &str) -> Result<(u64, u64)> { parse_qgroup_usage(&stdout).with_context(|| format!("parse qgroup usage for {agent_name}")) } +/// Measure an agent's disk usage in bytes (`state/` + `work/` + container +/// rootfs, one bounded `du` run by hive-priv as root). +/// +/// # Errors +/// Returns an error if the hive-priv call fails, the `du` walk times out or +/// exits non-zero, or the returned total isn't a number. +pub async fn measure_agent_disk(agent_name: &str) -> Result { + let (stdout, _) = check( + call(&PrivRequest::MeasureAgentDisk { + agent_name: agent_name.to_owned(), + }) + .await?, + )?; + stdout + .trim() + .parse() + .with_context(|| format!("parse disk usage for {agent_name}: {stdout:?}")) +} + /// Set or clear a btrfs qgroup size limit on an agent's state subvolume. /// `limit_bytes = None` clears it. Requires quota enabled. Via hive-priv. /// diff --git a/hive-c0re/src/stats/container_stats.rs b/hive-c0re/src/stats/container_stats.rs index 37329e4b..b3137e9b 100644 --- a/hive-c0re/src/stats/container_stats.rs +++ b/hive-c0re/src/stats/container_stats.rs @@ -1,4 +1,5 @@ -//! Live per-agent-container resource load (CPU + memory) for the +//! Live per-agent-container resource load (CPU + memory, plus the slow +//! disk sample from hive-priv in [`disk_sampler_loop`]) for the //! dashboard. Reads cgroup v2 stat files for each running agent //! machine directly — same privsep-clean posture as the turn-stats //! sqlite reads (`cpu.stat` / `memory.*` are world-readable; no @@ -57,121 +58,47 @@ pub struct ContainerResource { /// Memory ceiling (`memory.max`), bytes. `None` when unlimited /// (the file reads `max`). pub mem_max_bytes: Option, - /// Last-sampled total disk usage — the agent's `state/` and `work/` dirs plus the - /// container's writable rootfs, excluding the shared read-only nix - /// store — in bytes. `None` until the slow background sampler - /// ([`disk_sampler_loop`]) has populated the cache at least once. - /// Decoupled from this hot per-poll path because a `du` tree-walk is - /// far too expensive at the 5s `/api/container-load` cadence. + /// Last-sampled total disk usage — the agent's `state/` and `work/` dirs + /// plus the container's writable rootfs, excluding the shared read-only + /// nix store — in bytes. `None` when it is unknown: the slow background + /// sampler ([`disk_sampler_loop`]) has not measured this agent yet, or + /// its last measurement failed. Never 0 for a failed measurement. pub disk_bytes: Option, } -/// Cadence for the slow disk sampler. Disk size changes slowly relative -/// to CPU/mem, and a `du` walk is expensive, so this runs far less often -/// than the per-poll cgroup reads — the row just carries the last value. -const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(5); +/// Cadence for the slow disk sampler. Each pass has hive-priv walk every +/// agent's full tree as root, which is real I/O on the disk the agents +/// work on, and the sizes move over hours, not minutes. +const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(30); -/// Root of the per-container writable rootfs trees that nixos-containers -/// manages (`/var/lib/nixos-containers/`). -const NIXOS_CONTAINERS_ROOT: &str = "/var/lib/nixos-containers"; - -/// Last-sampled per-agent disk usage (bytes), keyed by agent name. -/// Written by [`disk_sampler_loop`] (~every 5 min), read by [`gather`] -/// so the hot poll never runs `du`. -fn disk_cache() -> &'static RwLock> { - static CACHE: OnceLock>> = OnceLock::new(); +/// Last per-agent disk measurement, keyed by agent name; `None` is a failed +/// one. Written by [`disk_sampler_loop`], read by [`gather`] so the hot +/// poll never measures. +fn disk_cache() -> &'static RwLock>> { + static CACHE: OnceLock>>> = OnceLock::new(); CACHE.get_or_init(|| RwLock::new(HashMap::new())) } -/// `du -sxb ` → apparent total bytes. `-s` summary, `-b` apparent -/// size, `-x` stay on one filesystem so the walk never descends into the -/// bind-mounted read-only shared nix store (or any other bind mount) — -/// exactly the per-container-only measure we want. Best-effort: the -/// caller treats `None` as a 0 contribution either way, but the two -/// `None` causes are logged differently. A path that plain doesn't exist -/// (`kept_state_names` includes destroyed-but-kept tombstones with no -/// rootfs, and a fresh agent has no state dir yet) is the expected, -/// silent case — every sampler pass would otherwise warn on it forever. -/// A path that exists but `du` still can't read — permission, I/O error, -/// unparseable output — is warn!'d, since that 0 is not "nothing there", -/// it's a read we should have gotten. -async fn du_bytes(path: &std::path::Path) -> Option { - match std::fs::symlink_metadata(path) { - Ok(_) => {} - Err(e) if e.kind() == std::io::ErrorKind::NotFound => return None, - Err(e) => { - tracing::warn!(path = %path.display(), error = %e, "du: path stat failed"); - return None; - } - } - let out = match tokio::process::Command::new("du") - .arg("-sxb") - .arg(path) - .output() - .await - { - Ok(out) => out, - Err(e) => { - tracing::warn!(path = %path.display(), error = %e, "du: failed to spawn"); - return None; - } - }; - if !out.status.success() { - tracing::warn!( - path = %path.display(), - status = %out.status, - stderr = %String::from_utf8_lossy(&out.stderr), - "du: exited non-zero" - ); - return None; - } - let stdout = String::from_utf8_lossy(&out.stdout); - let Some(first) = stdout.split_whitespace().next() else { - tracing::warn!(path = %path.display(), stdout = %stdout, "du: empty output"); - return None; - }; - match first.parse::() { - Ok(bytes) => Some(bytes), - Err(e) => { - tracing::warn!(path = %path.display(), stdout = %stdout, error = %e, "du: unparseable output"); - None - } - } +/// Total disk usage of `name`'s `state/` + `work/` + container rootfs, +/// measured by hive-priv as root (`hive-core` can read none of the rootfs +/// and not all of the agent-owned dirs). `None` on any failure. +async fn measure_agent_disk(name: &hive_types::Ident) -> Option { + disk_reading( + name, + crate::priv_client::measure_agent_disk(name.as_str()).await, + ) } -/// Host paths summed into an agent's disk usage: the agent's host-side -/// state and work dirs (`/var/lib/hyperhive/agents//{state,work}`) plus -/// the container's writable rootfs (`/var/lib/nixos-containers/h-`). -fn agent_disk_paths(name: &hive_types::Ident) -> [PathBuf; 3] { - [ - Coordinator::agent_notes_dir(name), - Coordinator::agent_work_dir(name), - PathBuf::from(format!("{NIXOS_CONTAINERS_ROOT}/h-{name}")), - ] +/// A failed measurement is unknown, never a 0 contribution. +fn disk_reading(name: &hive_types::Ident, measured: anyhow::Result) -> Option { + measured + .inspect_err(|e| { + tracing::warn!(agent = %name, error = %format!("{e:#}"), "disk usage measurement failed"); + }) + .ok() } -/// Total per-agent disk over [`agent_disk_paths`]. Each path is measured -/// with `du -sxb` so the shared nix store and other bind mounts are -/// excluded. A path that doesn't exist contributes 0. -/// -/// The state dir is resolved via [`Coordinator::agent_notes_dir`] — the same -/// host path the dashboard reads agent state from. Hardcoding the -/// container-internal bind-mount path (`/agents//state`) instead made -/// `du` fail host-side (that path only exists inside the container), so every -/// agent's state-dir contribution was 0; with the writable rootfs nearly empty -/// (almost everything is bind-mounted), that surfaced as all agents reporting -/// 0 disk. -async fn measure_agent_disk(name: &hive_types::Ident) -> u64 { - let mut total = 0u64; - for path in agent_disk_paths(name) { - if let Some(bytes) = du_bytes(&path).await { - total = total.saturating_add(bytes); - } - } - total -} - -/// Background loop: every [`DISK_SAMPLE_INTERVAL`], recompute disk usage +/// Background loop: every [`DISK_SAMPLE_INTERVAL`], remeasure disk usage /// for every kept-state agent and refresh [`disk_cache`]. The first pass /// runs immediately so the dashboard shows disk shortly after boot. /// Runs forever; spawn once at hive-c0re startup via [`spawn_disk_sampler`]. @@ -308,7 +235,7 @@ pub async fn gather() -> Vec { // Snapshot the slow disk cache once (cheap clone of a small map) so // the per-agent loop below is a plain map lookup — no `du` on this - // hot path. Agents not yet sampled simply carry `disk_bytes = None`. + // hot path. Agents not yet measured carry `disk_bytes = None` too. let disk = disk_cache().read().map(|c| c.clone()).unwrap_or_default(); let mut out: Vec = Vec::with_capacity(candidates.len()); @@ -335,7 +262,7 @@ pub async fn gather() -> Vec { mem_current_bytes: read_u64(&dir.join("memory.current")).unwrap_or(0), mem_peak_bytes: read_u64(&dir.join("memory.peak")), mem_max_bytes: read_mem_max(&dir.join("memory.max")), - disk_bytes: disk.get(name).copied(), + disk_bytes: disk.get(name).copied().flatten(), }); } out @@ -343,8 +270,6 @@ pub async fn gather() -> Vec { #[cfg(test)] mod tests { - use std::os::unix::fs::PermissionsExt; - use super::*; #[test] @@ -364,44 +289,16 @@ mod tests { } #[test] - fn disk_paths_cover_state_work_and_rootfs() { + fn failed_disk_measurement_is_unknown_not_zero() { let name = hive_types::Ident::parse("iris").expect("valid ident"); assert_eq!( - agent_disk_paths(&name), - [ - Coordinator::agent_notes_dir(&name), - Coordinator::agent_work_dir(&name), - PathBuf::from(format!("{NIXOS_CONTAINERS_ROOT}/h-iris")), - ] + disk_reading(&name, Err(anyhow::anyhow!("du for iris timed out"))), + None + ); + assert_eq!(disk_reading(&name, Ok(0)), Some(0)); + assert_eq!( + disk_reading(&name, Ok(13_000_000_000)), + Some(13_000_000_000) ); } - - #[tokio::test] - async fn du_bytes_missing_path_is_none() { - let dir = tempfile::tempdir().expect("tempdir"); - assert_eq!(du_bytes(&dir.path().join("does-not-exist")).await, None); - } - - #[tokio::test] - async fn du_bytes_reads_a_real_dir() { - let dir = tempfile::tempdir().expect("tempdir"); - std::fs::write(dir.path().join("f"), b"hello").expect("write"); - assert!(du_bytes(dir.path()).await.is_some_and(|b| b > 0)); - } - - #[tokio::test] - async fn du_bytes_unreadable_dir_is_none() { - // A dir that exists but can't be entered (mode 000) is the case - // that must be distinguished from "doesn't exist" — both currently - // become a 0 contribution, but only this one is worth a warn. - let dir = tempfile::tempdir().expect("tempdir"); - let unreadable = dir.path().join("locked"); - std::fs::create_dir(&unreadable).expect("mkdir"); - std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o000)) - .expect("chmod"); - let result = du_bytes(&unreadable).await; - std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o755)) - .expect("restore perms so tempdir cleanup can remove it"); - assert_eq!(result, None); - } } diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index ad2b5418..c8e61b8b 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -557,6 +557,23 @@ pub enum PrivRequest { limit_bytes: Option, }, + // --- Agent disk usage --- + /// Measure an agent's on-disk footprint with one bounded + /// `du -s -x -b -c -P` over its `//state` + /// and `…/work` dirs and its container rootfs + /// (`/var/lib/nixos-containers/h-`). Returns the total in + /// bytes as a decimal string in `stdout`. + /// + /// hive-priv derives all three paths from `agent_name`; the request + /// carries no path. A path that doesn't exist is skipped. A timeout, a + /// non-zero `du` exit or unparseable output is an error, never a partial + /// or zero total. Requires root: `hive-core` cannot read the rootfs or + /// the agent-owned files under `state/` and `work/`. + MeasureAgentDisk { + /// Logical agent name (validated by `validate_agent_name`). + agent_name: String, + }, + /// Convert an existing **plain-directory** agent state root into a btrfs /// subvolume in place. The operator opt-in counterpart to the progressive /// `EnsureAgentSubvolume` (which only ever makes *new* agents subvolumes): diff --git a/hive-priv/src/main.rs b/hive-priv/src/main.rs index f8202673..5c8dd607 100644 --- a/hive-priv/src/main.rs +++ b/hive-priv/src/main.rs @@ -430,6 +430,10 @@ async fn exec( limit_bytes, } => exec_set_subvolume_quota(agent_name, limit_bytes).await, + PrivRequest::MeasureAgentDisk { ref agent_name } => { + exec_measure_agent_disk(agent_name).await + } + PrivRequest::UpgradeAgentSubvolume { ref agent_name } => { exec_upgrade_agent_subvolume(agent_name).await } @@ -558,6 +562,12 @@ async fn exec_read_subvolume_usage(agent_name: &str) -> Result<(String, String)> read_subvolume_usage(agent_name).await } +/// `MeasureAgentDisk`. +async fn exec_measure_agent_disk(agent_name: &str) -> Result<(String, String)> { + validate_agent_name(agent_name)?; + measure_agent_disk(agent_name).await +} + /// `SetSubvolumeQuota`. async fn exec_set_subvolume_quota( agent_name: &str, @@ -1674,6 +1684,82 @@ async fn read_subvolume_usage(agent_name: &str) -> Result<(String, String)> { )) } +/// Wall-clock bound on one [`measure_agent_disk`] walk. It only stops a walk +/// wedged on I/O from holding the caller's sampler forever. +const DISK_USAGE_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(10); + +/// The host paths `MeasureAgentDisk` sums for `agent_name`: its `state/` +/// and `work/` dirs under `AGENT_STATE_ROOT` and its container rootfs. +/// These are the only paths that op ever walks. +// `state` / `work` must match hive-c0re's `agent_notes_dir` and +// `paths::WORK_SUBDIR`. +fn agent_disk_paths(agent_name: &str) -> [PathBuf; 3] { + let agent_root = PathBuf::from(AGENT_STATE_ROOT).join(agent_name); + [ + agent_root.join("state"), + agent_root.join("work"), + PathBuf::from("/var/lib/nixos-containers").join(container_system_name(agent_name)), + ] +} + +/// `MeasureAgentDisk` — one `du` over the [`agent_disk_paths`] that exist, +/// returning the grand total in bytes. `-x` keeps each walk on its +/// operand's filesystem (so the bind-mounted nix store is never counted), +/// `-P` never follows a symlink, `-b` counts apparent size, `-c` adds the +/// `total` line [`parse_du_total`] reads. +async fn measure_agent_disk(agent_name: &str) -> Result<(String, String)> { + let mut paths = Vec::new(); + for path in agent_disk_paths(agent_name) { + match std::fs::symlink_metadata(&path) { + Ok(_) => paths.push(path), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => return Err(e).with_context(|| format!("stat {}", path.display())), + } + } + // `du` with no operand would measure its working directory instead. + if paths.is_empty() { + return Ok(("0".to_owned(), String::new())); + } + let mut cmd = Command::new("du"); + cmd.args(["-s", "-x", "-b", "-c", "-P", "--"]) + .args(&paths) + .kill_on_drop(true); + let Ok(out) = tokio::time::timeout(DISK_USAGE_TIMEOUT, cmd.output()).await else { + bail!( + "du for {agent_name} timed out after {}s; killed it", + DISK_USAGE_TIMEOUT.as_secs() + ); + }; + let out = out.with_context(|| format!("spawn du for {agent_name}"))?; + if !out.status.success() { + bail!( + "du for {agent_name} failed ({}): {}", + out.status, + String::from_utf8_lossy(&out.stderr) + .lines() + .last() + .unwrap_or("") + .trim() + ); + } + let stdout = String::from_utf8_lossy(&out.stdout); + let total = parse_du_total(&stdout) + .ok_or_else(|| anyhow!("no `total` line in du output for {agent_name}: {stdout:?}"))?; + Ok((total.to_string(), String::new())) +} + +/// The byte count on `du -c`'s final `\ttotal` line. `None` when the +/// last line isn't that shape, so truncated or unexpected output is never +/// read as a smaller total. +fn parse_du_total(stdout: &str) -> Option { + let last = stdout.lines().rev().find(|l| !l.trim().is_empty())?; + let (bytes, label) = last.split_once('\t')?; + if label != "total" { + return None; + } + bytes.parse().ok() +} + /// Best-effort removal of a leftover migration path from a prior aborted /// upgrade: try `btrfs subvolume delete` (in case it's a half-created /// subvolume) then a plain recursive remove. Both failures are ignored — @@ -3093,11 +3179,12 @@ fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> { mod tests { use super::{ BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile, - check_fd_agreement, contains_secret_shaped_run, describe_forge_admin, - ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags, limits_dropin_body, - open_dir, open_export_dest, partial_name, publish_file, redact_secret_line, - remove_marker_in, run_bounded, single_output_path, toplevel_attr, validate_credential_name, - validate_snapshot_name, write_agent_dir_file, write_bridge_dns_marker_in, + agent_disk_paths, check_fd_agreement, contains_secret_shaped_run, describe_forge_admin, + ensure_plain_filename, ensure_socket_dir_in, exec_measure_agent_disk, git_overlay_flags, + limits_dropin_body, open_dir, open_export_dest, parse_du_total, partial_name, publish_file, + redact_secret_line, remove_marker_in, run_bounded, single_output_path, toplevel_attr, + validate_credential_name, validate_snapshot_name, write_agent_dir_file, + write_bridge_dns_marker_in, }; use std::path::PathBuf; use std::sync::atomic::{AtomicU32, Ordering}; @@ -4104,4 +4191,53 @@ mod tests { "must not have touched the existing file" ); } + + #[test] + fn disk_paths_are_derived_from_the_name_only() { + assert_eq!( + agent_disk_paths("iris"), + [ + PathBuf::from("/var/lib/hyperhive/agents/iris/state"), + PathBuf::from("/var/lib/hyperhive/agents/iris/work"), + PathBuf::from("/var/lib/nixos-containers/h-iris"), + ] + ); + } + + /// Every name that could steer the walk out of the agent's own paths is + /// refused before any path is built, let alone `du` spawned. + #[tokio::test] + async fn measure_rejects_names_that_are_not_agent_names() { + for bad in [ + "", "..", "../etc", "a/b", "/etc", "Iris", "a b", "a\0b", "foo_bar", + ] { + let err = exec_measure_agent_disk(bad) + .await + .expect_err(&format!("{bad:?} must be rejected")); + assert!( + err.to_string().contains("invalid name"), + "{bad:?}: wrong error: {err}" + ); + } + } + + #[test] + fn du_total_reads_the_total_line() { + let out = "100\t/var/lib/hyperhive/agents/iris/state\n\ + 2000\t/var/lib/nixos-containers/h-iris\n\ + 2100\ttotal\n"; + assert_eq!(parse_du_total(out), Some(2100)); + assert_eq!(parse_du_total("0\ttotal"), Some(0)); + } + + #[test] + fn du_total_rejects_output_without_a_total_line() { + assert_eq!(parse_du_total(""), None); + assert_eq!(parse_du_total("\n"), None); + // A per-path line is not the total, even when it is the last one. + assert_eq!(parse_du_total("100\t/var/lib/x\n"), None); + assert_eq!(parse_du_total("abc\ttotal\n"), None); + assert_eq!(parse_du_total("-1\ttotal\n"), None); + assert_eq!(parse_du_total("100 total\n"), None); + } }