feat(#2007): export per-agent container cpu/mem/disk via otel
hive-c0re already samples each agent container's cgroup load for the dashboard (stats/container_stats.rs); this rides those gauges out to the configured OTLP endpoint, reusing the existing services.hyperhive.otel config (endpoint + auth header) — no new toggle. - New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same crates as hive-metric) with the semconv container.* metric names + container.name attribute so off-the-shelf OTel/Grafana dashboards work, plus the hive agent label. container.cpu.time (counter, s, from cumulative cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit; memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.* custom (no semconv equivalent). Observable instruments read a shared snapshot an async task refreshes (gather() is async; SDK callbacks sync). - container_stats: expose cpu_time_usec (cumulative) on ContainerResource. - The OTLP auth header is loaded onto hive-c0re's own unit via systemd LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers. - docs/observability.md documents the host-emitted semconv metrics. Host-side export, so it covers containers even when their agent is idle.
This commit is contained in:
parent
c238ffe1ff
commit
419c9659a3
9 changed files with 396 additions and 2 deletions
|
|
@ -127,6 +127,42 @@ Additional labels can be appended via `extraResourceAttributes` (see option
|
|||
reference above); custom per-data-point labels can be passed with
|
||||
`hive-metric --labels` (see below).
|
||||
|
||||
## Host-emitted container-resource metrics (hive-c0re)
|
||||
|
||||
When OTEL is enabled, **hive-c0re itself** also exports each agent
|
||||
container's resource load — the same cgroup gauges shown on the dashboard
|
||||
LOAD tab — to the configured `endpoint`, reusing the same
|
||||
`services.hyperhive.otel` config (no separate toggle). These come from the
|
||||
host, not the in-container Claude SDK, so they cover containers even when
|
||||
their agent is idle.
|
||||
|
||||
Emitted via the OpenTelemetry Rust SDK, using the
|
||||
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
|
||||
metric names + the standard `container.name` attribute where a spec metric
|
||||
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
||||
`service.name = hyperhive-c0re`; each data point is tagged `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
|--------|------|------|--------|
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
alongside the spec `container.cpu.time` counter for convenience). Hive
|
||||
labels (`hive`, `swarm`, …) ride on the resource via
|
||||
`extraResourceAttributes`.
|
||||
|
||||
Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP
|
||||
(JSON) to `<endpoint>`; the auth header is loaded onto hive-c0re's own unit
|
||||
via systemd `LoadCredential` (from the same `headersCredential` file) and
|
||||
sent as `Authorization`.
|
||||
|
||||
## Agent-emitted custom metrics (`hive-metric`)
|
||||
|
||||
Agents can push arbitrary labeled metrics to the same OTEL collector via the
|
||||
|
|
|
|||
Loading…
Reference in a new issue