vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

container stats: measure agent disk usage through hive-priv, unknown on failure

Browse source 
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.

Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.

hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.

`Coordinator::agent_work_dir` had no other caller and is removed.

Closes #4956
This commit is contained in:
atlas 2026-10-05 16:12:33 +02:00 • committed by mara
commit 98ccc31e2b
10 changed files with 244 additions and 179 deletions
Download patch file Download diff file Expand all files Collapse all files

16
docs/scheduler/observability.md
View file

@ -292,14 +292,14 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source |
| ----------------------------------- | ---- | ------- | -------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
| Metric | Unit | Kind | Source |
| ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept

1
docs/trust-boundary/security.md
View file

@ -292,6 +292,7 @@ known operations; there is no arbitrary command pass-through:
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |

17
docs/web-ui/dashboard.md
View file

@ -156,10 +156,12 @@ files read-only (world-readable; no `hive-priv`) and skips agents whose
scope dir is absent (= not running). Pull-only: `core.js` polls every
5 s **only while the C0NT41N3R L04D sub-tab is active** (CPU needs a fresh
sample each refresh), and stops on sub-tab change. Disk size (`disk_bytes`)
rides the same row, but a separate ~5 min background `du` sampler feeds
it (`state/` + `work/` + container writable rootfs, shared nix store excluded via
`du -x`), so the 5 s poll stays cheap cgroup-only reads; the row carries the
last-sampled value (`null` until the first sample). Network isn't tracked
rides the same row, but a separate background sampler feeds it every
30 min: `hive-priv` runs one bounded `du -x` as root over the agent's
`state/` + `work/` + container writable rootfs (shared nix store excluded),
so the 5 s poll stays cheap cgroup-only reads. The row carries the last
measured value; the column shows "unknown" (`null`) before the first
measurement and whenever the last one failed, never 0. Network isn't tracked
per container — see `docs/networking/network.md` for the current
per-agent netns model.
@ -922,9 +924,10 @@ below — some endpoints aren't in it yet.
~200 ms so the call briefly awaits. Skips non-running agents (no scope
dir). No network field — agents share the host netns. `disk_bytes` is
the last value from a **separate slow sampler** (not this hot path):
a background `du -sxb` of the agent's `state/` + `work/` + container writable
rootfs every ~5 min, `-x` excluding the shared read-only nix store.
`null` until the first sample lands.
every 30 min `hive-priv` runs a bounded `du -sxb` as root over the agent's
`state/` + `work/` + container writable rootfs, `-x` excluding the shared
read-only nix store (`PrivRequest::MeasureAgentDisk`). `null` means
unknown: not measured yet, or the last measurement failed.
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
failed (clears the failure state so the scheduler retries).

8
frontend/packages/dashboard/src/core.js
View file

@ -286,14 +286,14 @@ function renderContainerLoad(rows) {
: "∞";
tr.append(lim);
// Disk: on-disk footprint (state/ + work/ + container writable rootfs, shared
// nix store excluded). Sampled out-of-band every few minutes server-side,
// so it's null until the first sample lands — show an em-dash then.
// nix store excluded). Measured out-of-band every 30 min server-side;
// null means unknown (not measured yet, or the measurement failed).
const disk = document.createElement("td");
disk.className = "num";
disk.title =
"state/ + work/ + container writable rootfs (shared nix store excluded); sampled every few minutes";
"state/ + work/ + container writable rootfs (shared nix store excluded); measured every 30 min";
disk.textContent =
r.disk_bytes != null ? cloadFmtBytes(Number(r.disk_bytes)) : "—";
r.disk_bytes != null ? cloadFmtBytes(Number(r.disk_bytes)) : "unknown";
tr.append(disk);
// Configured CPU / memory caps from ContainerView (resolved effective

8
hive-c0re/src/coordinator.rs
View file

@ -1130,14 +1130,6 @@ impl Coordinator {
crate::paths::agent_state_dir(name).join("state")
}
/// Per-agent working dir for worktrees and build dirs. Bind-mounted RW
/// into the agent container at `/agents/{name}/work`. Survives
/// destroy/recreate. A sibling of `state/` rather than inside it, so a
/// claude started under `work/` never walks up into `state/CLAUDE.md`.
pub fn agent_work_dir(name: &hive_types::Ident) -> PathBuf {
crate::paths::agent_state_dir(name).join(crate::paths::WORK_SUBDIR)
}
/// Per-agent harness-internal state dir. Bind-mounted RW into the
/// agent container at `/agents/{name}/harness`. Holds sqlite dbs
/// and config files owned by the harness (`hyperhive-events.sqlite`,

4
hive-c0re/src/main.rs
View file

@ -399,8 +399,8 @@ async fn cmd_serve(
// (turn-stats.sqlite has no vacuum — it's one tiny row per turn,
// ~hundreds of KB, and the /stats + hive-stats views read it
// directly; pruning it would just lose trend history for no gain.)
// Slow per-container disk sampler: a `du` of each agent's state/ + work/
// + writable rootfs every ~5 min, cached so the 5s container-load
// Slow per-container disk sampler: hive-priv measures each agent's state/
// + work/ + writable rootfs every 30 min, cached so the 5s container-load
// poll stays cheap cgroup-only reads. Feeds `disk_bytes` on the LOAD
// tab. See container_stats::disk_sampler_loop.
crate::container_stats::spawn_disk_sampler();

19
hive-c0re/src/priv_client.rs
View file

@ -429,6 +429,25 @@ pub async fn read_subvolume_usage(agent_name: &str) -> Result<(u64, u64)> {
parse_qgroup_usage(&stdout).with_context(|| format!("parse qgroup usage for {agent_name}"))
}
/// Measure an agent's disk usage in bytes (`state/` + `work/` + container
/// rootfs, one bounded `du` run by hive-priv as root).
///
/// # Errors
/// Returns an error if the hive-priv call fails, the `du` walk times out or
/// exits non-zero, or the returned total isn't a number.
pub async fn measure_agent_disk(agent_name: &str) -> Result<u64> {
let (stdout, _) = check(
call(&PrivRequest::MeasureAgentDisk {
agent_name: agent_name.to_owned(),
})
.await?,
)?;
stdout
.trim()
.parse()
.with_context(|| format!("parse disk usage for {agent_name}: {stdout:?}"))
}
/// Set or clear a btrfs qgroup size limit on an agent's state subvolume.
/// `limit_bytes = None` clears it. Requires quota enabled. Via hive-priv.
///

187
hive-c0re/src/stats/container_stats.rs
View file

@ -1,4 +1,5 @@
//! Live per-agent-container resource load (CPU + memory) for the
//! Live per-agent-container resource load (CPU + memory, plus the slow
//! disk sample from hive-priv in [`disk_sampler_loop`]) for the
//! dashboard. Reads cgroup v2 stat files for each running agent
//! machine directly — same privsep-clean posture as the turn-stats
//! sqlite reads (`cpu.stat` / `memory.*` are world-readable; no
@ -57,121 +58,47 @@ pub struct ContainerResource {
/// Memory ceiling (`memory.max`), bytes. `None` when unlimited
/// (the file reads `max`).
pub mem_max_bytes: Option<u64>,
/// Last-sampled total disk usage — the agent's `state/` and `work/` dirs plus the
/// container's writable rootfs, excluding the shared read-only nix
/// store — in bytes. `None` until the slow background sampler
/// ([`disk_sampler_loop`]) has populated the cache at least once.
/// Decoupled from this hot per-poll path because a `du` tree-walk is
/// far too expensive at the 5s `/api/container-load` cadence.
/// Last-sampled total disk usage — the agent's `state/` and `work/` dirs
/// plus the container's writable rootfs, excluding the shared read-only
/// nix store — in bytes. `None` when it is unknown: the slow background
/// sampler ([`disk_sampler_loop`]) has not measured this agent yet, or
/// its last measurement failed. Never 0 for a failed measurement.
pub disk_bytes: Option<u64>,
}
/// Cadence for the slow disk sampler. Disk size changes slowly relative
/// to CPU/mem, and a `du` walk is expensive, so this runs far less often
/// than the per-poll cgroup reads — the row just carries the last value.
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(5);
/// Cadence for the slow disk sampler. Each pass has hive-priv walk every
/// agent's full tree as root, which is real I/O on the disk the agents
/// work on, and the sizes move over hours, not minutes.
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(30);
/// Root of the per-container writable rootfs trees that nixos-containers
/// manages (`/var/lib/nixos-containers/<machine>`).
const NIXOS_CONTAINERS_ROOT: &str = "/var/lib/nixos-containers";
/// Last-sampled per-agent disk usage (bytes), keyed by agent name.
/// Written by [`disk_sampler_loop`] (~every 5 min), read by [`gather`]
/// so the hot poll never runs `du`.
fn disk_cache() -> &'static RwLock<HashMap<String, u64>> {
static CACHE: OnceLock<RwLock<HashMap<String, u64>>> = OnceLock::new();
/// Last per-agent disk measurement, keyed by agent name; `None` is a failed
/// one. Written by [`disk_sampler_loop`], read by [`gather`] so the hot
/// poll never measures.
fn disk_cache() -> &'static RwLock<HashMap<String, Option<u64>>> {
static CACHE: OnceLock<RwLock<HashMap<String, Option<u64>>>> = OnceLock::new();
CACHE.get_or_init(|| RwLock::new(HashMap::new()))
}
/// `du -sxb <path>` → apparent total bytes. `-s` summary, `-b` apparent
/// size, `-x` stay on one filesystem so the walk never descends into the
/// bind-mounted read-only shared nix store (or any other bind mount) —
/// exactly the per-container-only measure we want. Best-effort: the
/// caller treats `None` as a 0 contribution either way, but the two
/// `None` causes are logged differently. A path that plain doesn't exist
/// (`kept_state_names` includes destroyed-but-kept tombstones with no
/// rootfs, and a fresh agent has no state dir yet) is the expected,
/// silent case — every sampler pass would otherwise warn on it forever.
/// A path that exists but `du` still can't read — permission, I/O error,
/// unparseable output — is warn!'d, since that 0 is not "nothing there",
/// it's a read we should have gotten.
async fn du_bytes(path: &std::path::Path) -> Option<u64> {
match std::fs::symlink_metadata(path) {
Ok(_) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return None,
Err(e) => {
tracing::warn!(path = %path.display(), error = %e, "du: path stat failed");
return None;
}
}
let out = match tokio::process::Command::new("du")
.arg("-sxb")
.arg(path)
.output()
.await
{
Ok(out) => out,
Err(e) => {
tracing::warn!(path = %path.display(), error = %e, "du: failed to spawn");
return None;
}
};
if !out.status.success() {
tracing::warn!(
path = %path.display(),
status = %out.status,
stderr = %String::from_utf8_lossy(&out.stderr),
"du: exited non-zero"
);
return None;
}
let stdout = String::from_utf8_lossy(&out.stdout);
let Some(first) = stdout.split_whitespace().next() else {
tracing::warn!(path = %path.display(), stdout = %stdout, "du: empty output");
return None;
};
match first.parse::<u64>() {
Ok(bytes) => Some(bytes),
Err(e) => {
tracing::warn!(path = %path.display(), stdout = %stdout, error = %e, "du: unparseable output");
None
}
}
/// Total disk usage of `name`'s `state/` + `work/` + container rootfs,
/// measured by hive-priv as root (`hive-core` can read none of the rootfs
/// and not all of the agent-owned dirs). `None` on any failure.
async fn measure_agent_disk(name: &hive_types::Ident) -> Option<u64> {
disk_reading(
name,
crate::priv_client::measure_agent_disk(name.as_str()).await,
)
}
/// Host paths summed into an agent's disk usage: the agent's host-side
/// state and work dirs (`/var/lib/hyperhive/agents/<name>/{state,work}`) plus
/// the container's writable rootfs (`/var/lib/nixos-containers/h-<name>`).
fn agent_disk_paths(name: &hive_types::Ident) -> [PathBuf; 3] {
[
Coordinator::agent_notes_dir(name),
Coordinator::agent_work_dir(name),
PathBuf::from(format!("{NIXOS_CONTAINERS_ROOT}/h-{name}")),
]
/// A failed measurement is unknown, never a 0 contribution.
fn disk_reading(name: &hive_types::Ident, measured: anyhow::Result<u64>) -> Option<u64> {
measured
.inspect_err(|e| {
tracing::warn!(agent = %name, error = %format!("{e:#}"), "disk usage measurement failed");
})
.ok()
}
/// Total per-agent disk over [`agent_disk_paths`]. Each path is measured
/// with `du -sxb` so the shared nix store and other bind mounts are
/// excluded. A path that doesn't exist contributes 0.
///
/// The state dir is resolved via [`Coordinator::agent_notes_dir`] — the same
/// host path the dashboard reads agent state from. Hardcoding the
/// container-internal bind-mount path (`/agents/<name>/state`) instead made
/// `du` fail host-side (that path only exists inside the container), so every
/// agent's state-dir contribution was 0; with the writable rootfs nearly empty
/// (almost everything is bind-mounted), that surfaced as all agents reporting
/// 0 disk.
async fn measure_agent_disk(name: &hive_types::Ident) -> u64 {
let mut total = 0u64;
for path in agent_disk_paths(name) {
if let Some(bytes) = du_bytes(&path).await {
total = total.saturating_add(bytes);
}
}
total
}
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], recompute disk usage
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], remeasure disk usage
/// for every kept-state agent and refresh [`disk_cache`]. The first pass
/// runs immediately so the dashboard shows disk shortly after boot.
/// Runs forever; spawn once at hive-c0re startup via [`spawn_disk_sampler`].
@ -308,7 +235,7 @@ pub async fn gather() -> Vec<ContainerResource> {
// Snapshot the slow disk cache once (cheap clone of a small map) so
// the per-agent loop below is a plain map lookup — no `du` on this
// hot path. Agents not yet sampled simply carry `disk_bytes = None`.
// hot path. Agents not yet measured carry `disk_bytes = None` too.
let disk = disk_cache().read().map(|c| c.clone()).unwrap_or_default();
let mut out: Vec<ContainerResource> = Vec::with_capacity(candidates.len());
@ -335,7 +262,7 @@ pub async fn gather() -> Vec<ContainerResource> {
mem_current_bytes: read_u64(&dir.join("memory.current")).unwrap_or(0),
mem_peak_bytes: read_u64(&dir.join("memory.peak")),
mem_max_bytes: read_mem_max(&dir.join("memory.max")),
disk_bytes: disk.get(name).copied(),
disk_bytes: disk.get(name).copied().flatten(),
});
}
out
@ -343,8 +270,6 @@ pub async fn gather() -> Vec<ContainerResource> {
#[cfg(test)]
mod tests {
use std::os::unix::fs::PermissionsExt;
use super::*;
#[test]
@ -364,44 +289,16 @@ mod tests {
}
#[test]
fn disk_paths_cover_state_work_and_rootfs() {
fn failed_disk_measurement_is_unknown_not_zero() {
let name = hive_types::Ident::parse("iris").expect("valid ident");
assert_eq!(
agent_disk_paths(&name),
[
Coordinator::agent_notes_dir(&name),
Coordinator::agent_work_dir(&name),
PathBuf::from(format!("{NIXOS_CONTAINERS_ROOT}/h-iris")),
]
disk_reading(&name, Err(anyhow::anyhow!("du for iris timed out"))),
None
);
assert_eq!(disk_reading(&name, Ok(0)), Some(0));
assert_eq!(
disk_reading(&name, Ok(13_000_000_000)),
Some(13_000_000_000)
);
}
#[tokio::test]
async fn du_bytes_missing_path_is_none() {
let dir = tempfile::tempdir().expect("tempdir");
assert_eq!(du_bytes(&dir.path().join("does-not-exist")).await, None);
}
#[tokio::test]
async fn du_bytes_reads_a_real_dir() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join("f"), b"hello").expect("write");
assert!(du_bytes(dir.path()).await.is_some_and(|b| b > 0));
}
#[tokio::test]
async fn du_bytes_unreadable_dir_is_none() {
// A dir that exists but can't be entered (mode 000) is the case
// that must be distinguished from "doesn't exist" — both currently
// become a 0 contribution, but only this one is worth a warn.
let dir = tempfile::tempdir().expect("tempdir");
let unreadable = dir.path().join("locked");
std::fs::create_dir(&unreadable).expect("mkdir");
std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o000))
.expect("chmod");
let result = du_bytes(&unreadable).await;
std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o755))
.expect("restore perms so tempdir cleanup can remove it");
assert_eq!(result, None);
}
}

17
hive-priv-sock/src/lib.rs
View file

@ -557,6 +557,23 @@ pub enum PrivRequest {
limit_bytes: Option<u64>,
},
// --- Agent disk usage ---
/// Measure an agent's on-disk footprint with one bounded
/// `du -s -x -b -c -P` over its `<AGENT_STATE_ROOT>/<agent_name>/state`
/// and `…/work` dirs and its container rootfs
/// (`/var/lib/nixos-containers/h-<agent_name>`). Returns the total in
/// bytes as a decimal string in `stdout`.
///
/// hive-priv derives all three paths from `agent_name`; the request
/// carries no path. A path that doesn't exist is skipped. A timeout, a
/// non-zero `du` exit or unparseable output is an error, never a partial
/// or zero total. Requires root: `hive-core` cannot read the rootfs or
/// the agent-owned files under `state/` and `work/`.
MeasureAgentDisk {
/// Logical agent name (validated by `validate_agent_name`).
agent_name: String,
},
/// Convert an existing **plain-directory** agent state root into a btrfs
/// subvolume in place. The operator opt-in counterpart to the progressive
/// `EnsureAgentSubvolume` (which only ever makes *new* agents subvolumes):

146
hive-priv/src/main.rs
View file

@ -430,6 +430,10 @@ async fn exec(
limit_bytes,
} => exec_set_subvolume_quota(agent_name, limit_bytes).await,
PrivRequest::MeasureAgentDisk { ref agent_name } => {
exec_measure_agent_disk(agent_name).await
}
PrivRequest::UpgradeAgentSubvolume { ref agent_name } => {
exec_upgrade_agent_subvolume(agent_name).await
}
@ -558,6 +562,12 @@ async fn exec_read_subvolume_usage(agent_name: &str) -> Result<(String, String)>
read_subvolume_usage(agent_name).await
}
/// `MeasureAgentDisk`.
async fn exec_measure_agent_disk(agent_name: &str) -> Result<(String, String)> {
validate_agent_name(agent_name)?;
measure_agent_disk(agent_name).await
}
/// `SetSubvolumeQuota`.
async fn exec_set_subvolume_quota(
agent_name: &str,
@ -1674,6 +1684,82 @@ async fn read_subvolume_usage(agent_name: &str) -> Result<(String, String)> {
))
}
/// Wall-clock bound on one [`measure_agent_disk`] walk. It only stops a walk
/// wedged on I/O from holding the caller's sampler forever.
const DISK_USAGE_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(10);
/// The host paths `MeasureAgentDisk` sums for `agent_name`: its `state/`
/// and `work/` dirs under `AGENT_STATE_ROOT` and its container rootfs.
/// These are the only paths that op ever walks.
// `state` / `work` must match hive-c0re's `agent_notes_dir` and
// `paths::WORK_SUBDIR`.
fn agent_disk_paths(agent_name: &str) -> [PathBuf; 3] {
let agent_root = PathBuf::from(AGENT_STATE_ROOT).join(agent_name);
[
agent_root.join("state"),
agent_root.join("work"),
PathBuf::from("/var/lib/nixos-containers").join(container_system_name(agent_name)),
]
}
/// `MeasureAgentDisk` — one `du` over the [`agent_disk_paths`] that exist,
/// returning the grand total in bytes. `-x` keeps each walk on its
/// operand's filesystem (so the bind-mounted nix store is never counted),
/// `-P` never follows a symlink, `-b` counts apparent size, `-c` adds the
/// `total` line [`parse_du_total`] reads.
async fn measure_agent_disk(agent_name: &str) -> Result<(String, String)> {
let mut paths = Vec::new();
for path in agent_disk_paths(agent_name) {
match std::fs::symlink_metadata(&path) {
Ok(_) => paths.push(path),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(e).with_context(|| format!("stat {}", path.display())),
}
}
// `du` with no operand would measure its working directory instead.
if paths.is_empty() {
return Ok(("0".to_owned(), String::new()));
}
let mut cmd = Command::new("du");
cmd.args(["-s", "-x", "-b", "-c", "-P", "--"])
.args(&paths)
.kill_on_drop(true);
let Ok(out) = tokio::time::timeout(DISK_USAGE_TIMEOUT, cmd.output()).await else {
bail!(
"du for {agent_name} timed out after {}s; killed it",
DISK_USAGE_TIMEOUT.as_secs()
);
};
let out = out.with_context(|| format!("spawn du for {agent_name}"))?;
if !out.status.success() {
bail!(
"du for {agent_name} failed ({}): {}",
out.status,
String::from_utf8_lossy(&out.stderr)
.lines()
.last()
.unwrap_or("")
.trim()
);
}
let stdout = String::from_utf8_lossy(&out.stdout);
let total = parse_du_total(&stdout)
.ok_or_else(|| anyhow!("no `total` line in du output for {agent_name}: {stdout:?}"))?;
Ok((total.to_string(), String::new()))
}
/// The byte count on `du -c`'s final `<bytes>\ttotal` line. `None` when the
/// last line isn't that shape, so truncated or unexpected output is never
/// read as a smaller total.
fn parse_du_total(stdout: &str) -> Option<u64> {
let last = stdout.lines().rev().find(|l| !l.trim().is_empty())?;
let (bytes, label) = last.split_once('\t')?;
if label != "total" {
return None;
}
bytes.parse().ok()
}
/// Best-effort removal of a leftover migration path from a prior aborted
/// upgrade: try `btrfs subvolume delete` (in case it's a half-created
/// subvolume) then a plain recursive remove. Both failures are ignored —
@ -3093,11 +3179,12 @@ fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> {
mod tests {
use super::{
BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
check_fd_agreement, contains_secret_shaped_run, describe_forge_admin,
ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags, limits_dropin_body,
open_dir, open_export_dest, partial_name, publish_file, redact_secret_line,
remove_marker_in, run_bounded, single_output_path, toplevel_attr, validate_credential_name,
validate_snapshot_name, write_agent_dir_file, write_bridge_dns_marker_in,
agent_disk_paths, check_fd_agreement, contains_secret_shaped_run, describe_forge_admin,
ensure_plain_filename, ensure_socket_dir_in, exec_measure_agent_disk, git_overlay_flags,
limits_dropin_body, open_dir, open_export_dest, parse_du_total, partial_name, publish_file,
redact_secret_line, remove_marker_in, run_bounded, single_output_path, toplevel_attr,
validate_credential_name, validate_snapshot_name, write_agent_dir_file,
write_bridge_dns_marker_in,
};
use std::path::PathBuf;
use std::sync::atomic::{AtomicU32, Ordering};
@ -4104,4 +4191,53 @@ mod tests {
"must not have touched the existing file"
);
}
#[test]
fn disk_paths_are_derived_from_the_name_only() {
assert_eq!(
agent_disk_paths("iris"),
[
PathBuf::from("/var/lib/hyperhive/agents/iris/state"),
PathBuf::from("/var/lib/hyperhive/agents/iris/work"),
PathBuf::from("/var/lib/nixos-containers/h-iris"),
]
);
}
/// Every name that could steer the walk out of the agent's own paths is
/// refused before any path is built, let alone `du` spawned.
#[tokio::test]
async fn measure_rejects_names_that_are_not_agent_names() {
for bad in [
"", "..", "../etc", "a/b", "/etc", "Iris", "a b", "a\0b", "foo_bar",
] {
let err = exec_measure_agent_disk(bad)
.await
.expect_err(&format!("{bad:?} must be rejected"));
assert!(
err.to_string().contains("invalid name"),
"{bad:?}: wrong error: {err}"
);
}
}
#[test]
fn du_total_reads_the_total_line() {
let out = "100\t/var/lib/hyperhive/agents/iris/state\n\
2000\t/var/lib/nixos-containers/h-iris\n\
2100\ttotal\n";
assert_eq!(parse_du_total(out), Some(2100));
assert_eq!(parse_du_total("0\ttotal"), Some(0));
}
#[test]
fn du_total_rejects_output_without_a_total_line() {
assert_eq!(parse_du_total(""), None);
assert_eq!(parse_du_total("\n"), None);
// A per-path line is not the total, even when it is the last one.
assert_eq!(parse_du_total("100\t/var/lib/x\n"), None);
assert_eq!(parse_du_total("abc\ttotal\n"), None);
assert_eq!(parse_du_total("-1\ttotal\n"), None);
assert_eq!(parse_du_total("100 total\n"), None);
}
}