mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

document budgeted-agent swap and oom metrics; add oom kills panel

Browse source
This commit is contained in:
atlas 2026-11-11 01:58:30 +01:00
commit c014b48c22
1 changed files with 138 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

24
docs/4-implementation/scheduler/observability.md
View file

@ -344,17 +344,19 @@ CI runner, the same gauges also cover its `swarm-ci-runner` container, with
filters those series out and the host dashboard's top-10 panels group by
`container.name`. Its storage usage isn't measured, so that series is absent.
| Metric | Unit | Kind | Source |
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
| `hyperhive.agent.needs_login` | `1` | gauge | `1` when a running agent needs a claude login (the dashboard's needs-login flag), `0` when it has a login. Omitted for a stopped agent and before the first container scan |
| Metric | Unit | Kind | Source |
| --------------------------------------- | ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.memory.oom_kills` | `{process}` | counter | `memory.events` `oom_kill`: processes the kernel killed for memory in the container and below it (custom; omitted if unavailable) |
| `hyperhive.container.memory.swap.usage` | `By` | gauge | `memory.swap.current` (custom; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
| `hyperhive.agent.needs_login` | `1` | gauge | `1` when a running agent needs a claude login (the dashboard's needs-login flag), `0` when it has a login. Omitted for a stopped agent and before the first container scan |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept