container stats: measure agent disk usage through hive-priv, unknown on failure
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.
Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.
hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.
`Coordinator::agent_work_dir` had no other caller and is removed.
Closes #4956
This commit is contained in:
parent
8ffceea8e3
commit
98ccc31e2b
10 changed files with 244 additions and 179 deletions
|
|
@ -399,8 +399,8 @@ async fn cmd_serve(
|
|||
// (turn-stats.sqlite has no vacuum — it's one tiny row per turn,
|
||||
// ~hundreds of KB, and the /stats + hive-stats views read it
|
||||
// directly; pruning it would just lose trend history for no gain.)
|
||||
// Slow per-container disk sampler: a `du` of each agent's state/ + work/
|
||||
// + writable rootfs every ~5 min, cached so the 5s container-load
|
||||
// Slow per-container disk sampler: hive-priv measures each agent's state/
|
||||
// + work/ + writable rootfs every 30 min, cached so the 5s container-load
|
||||
// poll stays cheap cgroup-only reads. Feeds `disk_bytes` on the LOAD
|
||||
// tab. See container_stats::disk_sampler_loop.
|
||||
crate::container_stats::spawn_disk_sampler();
|
||||
|
|
|
|||
Loading…
Reference in a new issue