mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

budgeted agents throttle at 90% of the cap; alert on repeated oom kills

Browse source
This commit is contained in:
atlas 2026-11-11 02:38:53 +01:00
commit 11393bc538
1 changed files with 201 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/4-implementation/scheduler/observability.md
View file

@ -560,6 +560,7 @@ The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists (`pending_logins` only for a hive that doesn't export `hyperhive.agent.needs_login`) for 2 minutes |
| `hh-agent-needs-login` | `hyperhive.agent.needs_login` is `1` for an agent for 5 minutes, one alert per agent |
| `hh-agent-oom-kills` | `hyperhive.container.memory.oom_kills` grew by more than 3 in the last hour for an agent, one alert per agent, no delay |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |