Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: read each agent's swarm caps from agent-limits

Every 30 seconds `workers::swarm_limits` reads the `agent-limits` key of
each agent this hive has and stores the result in
swarm-resource-limits.json, where the drop-in, the heap ceiling and the
dashboard read it. A cap therefore follows its agent to whichever hive
runs it.

Each value goes through the hive's own validate_cpu_quota /
validate_memory_max before it is stored, since hive-priv renders it
verbatim into the container's drop-in, and one bad value refuses the whole
set (argus on #4268, carried over from the wanted-state consumer this
replaces). A failed read, or a bucket that does not answer, leaves the
stored caps as they are; three failures in a row raise a banner.

swarm-nats-auth grants every hive `$JS.API.DIRECT.GET.KV_agent-limits.>`:
read-only, and every agent's key, not just the hive's own. Scoping it to
the hive's agents would need a roster in the auth responder, which
policy.rs argues against. mara chose this on #4268 (c95293); narrowing it
is a possible low-priority follow-up, not planned. A test pins that the
grant reads every key, writes none, and reaches no other bucket.

Contributes to #3865.
This commit is contained in:
atlas 2026-10-12 11:26:18 +02:00 • committed by mara
commit fccf2ab892
8 changed files with 320 additions and 4 deletions

View file

@ -452,14 +452,24 @@ regardless.
rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild.
The answer per agent and per field is, first match wins:
1. the swarm's cap for the agent (see below);
2. the operator's per-agent override in `meta/resource-limits.json`;
3. `agentCpuQuota` / `agentMemoryMax`.
A swarm cap replaces the hive's answer, so a hive can't raise an agent
above it. Like the rest of the drop-in, a changed swarm cap reaches a
running agent at its next spawn or rebuild.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the overrides file, the file carries
that drop-in's `MemoryMax=`.
the last drop-in because it can't read the overrides or the swarm caps file,
the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
@ -469,6 +479,34 @@ changed cap reaches them at the next container start without a rebuild.
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
<details><summary>Implementation: swarm caps</summary>
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
agent and no hive in it, so an agent that moves to another hive keeps its
caps. swarm-controller alone writes it:
| method | path | body |
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
`PUT` refuses an agent no hive's wanted state declares, unknown fields, zero
and anything systemd doesn't take, and stores memory the way systemd spells
it (`8gb` as `8G`).
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the same validators as the operator's overrides first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a
dashboard banner.
A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
hive's own agents would need a roster in the auth responder.
</details>
The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle