container stats: measure agent disk usage through hive-priv, unknown on failure
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.
Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.
hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.
`Coordinator::agent_work_dir` had no other caller and is removed.
Closes #4956
This commit is contained in:
atlas
mara
parent
8ffceea8e3
commit
98ccc31e2b
10 changed files with 244 additions and 179 deletions
|
|
@ -292,14 +292,14 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
| ----------------------------------- | ---- | ------- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
| Metric | Unit | Kind | Source |
|
||||
| ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
|
|
|
|||
|
|
@ -292,6 +292,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
||||
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
||||
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
||||
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
||||
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
||||
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
||||
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
||||
|
|
|
|||
|
|
@ -156,10 +156,12 @@ files read-only (world-readable; no `hive-priv`) and skips agents whose
|
|||
scope dir is absent (= not running). Pull-only: `core.js` polls every
|
||||
5 s **only while the C0NT41N3R L04D sub-tab is active** (CPU needs a fresh
|
||||
sample each refresh), and stops on sub-tab change. Disk size (`disk_bytes`)
|
||||
rides the same row, but a separate ~5 min background `du` sampler feeds
|
||||
it (`state/` + `work/` + container writable rootfs, shared nix store excluded via
|
||||
`du -x`), so the 5 s poll stays cheap cgroup-only reads; the row carries the
|
||||
last-sampled value (`null` until the first sample). Network isn't tracked
|
||||
rides the same row, but a separate background sampler feeds it every
|
||||
30 min: `hive-priv` runs one bounded `du -x` as root over the agent's
|
||||
`state/` + `work/` + container writable rootfs (shared nix store excluded),
|
||||
so the 5 s poll stays cheap cgroup-only reads. The row carries the last
|
||||
measured value; the column shows "unknown" (`null`) before the first
|
||||
measurement and whenever the last one failed, never 0. Network isn't tracked
|
||||
per container — see `docs/networking/network.md` for the current
|
||||
per-agent netns model.
|
||||
|
||||
|
|
@ -922,9 +924,10 @@ below — some endpoints aren't in it yet.
|
|||
~200 ms so the call briefly awaits. Skips non-running agents (no scope
|
||||
dir). No network field — agents share the host netns. `disk_bytes` is
|
||||
the last value from a **separate slow sampler** (not this hot path):
|
||||
a background `du -sxb` of the agent's `state/` + `work/` + container writable
|
||||
rootfs every ~5 min, `-x` excluding the shared read-only nix store.
|
||||
`null` until the first sample lands.
|
||||
every 30 min `hive-priv` runs a bounded `du -sxb` as root over the agent's
|
||||
`state/` + `work/` + container writable rootfs, `-x` excluding the shared
|
||||
read-only nix store (`PrivRequest::MeasureAgentDisk`). `null` means
|
||||
unknown: not measured yet, or the last measurement failed.
|
||||
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
||||
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
||||
failed (clears the failure state so the scheduler retries).
|
||||
|
|
|
|||
Loading…
Reference in a new issue