mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

budgeted agents throttle at 90% of the cap; alert on repeated oom kills

Browse source
This commit is contained in:
atlas 2026-11-11 02:38:53 +01:00
commit 11393bc538
2 changed files with 201 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/4-implementation/scheduler/coordinator.md
View file

@ -446,11 +446,12 @@ changed cap reaches them at the next container start without a rebuild.
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
A container with a finite `MemoryMax=` also gets `MemorySwapMax=0` in the same
drop-in, so a budgeted agent never swaps: paging it out to host swap would
move the pressure onto the host. `MemoryMax=infinity` leaves swap alone. Like
the rest of the drop-in, it reaches a running agent at its next spawn or
rebuild.
A container with a finite `MemoryMax=` also gets, in the same drop-in,
`MemorySwapMax=0` and `MemoryHigh=` at 90% of the cap (bytes, rounded down).
A budgeted agent never swaps: paging it out to host swap would move the
pressure onto the host. Past 90% the kernel slows the container and reclaims
from it, and at 100% it kills. `MemoryMax=infinity` sets neither. Like the
rest of the drop-in, they reach a running agent at its next spawn or rebuild.
When the container reaches its cap, the kernel kills inside it, in this order:
@ -474,6 +475,8 @@ is the memory the container has swapped out: nonzero only for an agent with
no budget, or one whose container started before its drop-in carried
`MemorySwapMax=0`.
More than 3 kills of one agent in an hour raise the `hh-agent-oom-kills` alert.
<details><summary>Implementation: swarm caps</summary>
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per

1
docs/4-implementation/scheduler/observability.md
View file

@ -560,6 +560,7 @@ The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists (`pending_logins` only for a hive that doesn't export `hyperhive.agent.needs_login`) for 2 minutes |
| `hh-agent-needs-login` | `hyperhive.agent.needs_login` is `1` for an agent for 5 minutes, one alert per agent |
| `hh-agent-oom-kills` | `hyperhive.container.memory.oom_kills` grew by more than 3 in the last hour for an agent, one alert per agent, no delay |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |