mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

budgeted agents throttle at 90% of the cap; alert on repeated oom kills

Browse source
This commit is contained in:
atlas 2026-11-11 02:38:53 +01:00
commit 11393bc538
1 changed files with 201 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

15
nix/host-modules/swarm-grafana.nix
View file

@ -1767,6 +1767,21 @@ in
for = "5m";
summary = "agent {{ $labels.agent }} on hive {{ $labels.hive }} has needed a claude login for 5 minutes; open its page in the dashboard to complete it";
})
# Processes the kernel OOM-killed in an agent's container, from
# the counter hive-c0re exports (`otel_metrics.rs`). A budgeted
# agent has no swap, so each kill is a subagent run, bash task or
# the agent's claude that hit its RAM budget. More than three in
# an hour is a budget that is too small for the work, not a
# one-off peak. One alert instance per `(hive, agent)`.
(metricThresholdRule {
uid = "hh-agent-oom-kills";
title = "agent OOM-killed repeatedly";
windowSeconds = 3600;
expr = ''sum by (hive, agent) (increase({__name__="hyperhive.container.memory.oom_kills", agent!=""}[1h]))'';
threshold = 3;
for = "0m";
summary = "agent {{ $labels.agent }} on hive {{ $labels.hive }} had more than 3 processes OOM-killed in the last hour: it keeps hitting its RAM budget; see \"OOM kills by agent\" on the agents dashboard, and raise its memory cap if the work needs it";
})
# `hyperhive.hive.nix_store.utilization` is the used fraction of the
# filesystem holding the nix store, the one that failed CI
# with ENOSPC before the GC guardrails.