Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: read each agent's swarm caps from agent-limits

Every 30 seconds `workers::swarm_limits` reads the `agent-limits` key of
each agent this hive has and stores the result in
swarm-resource-limits.json, where the drop-in, the heap ceiling and the
dashboard read it. A cap therefore follows its agent to whichever hive
runs it.

Each value goes through the hive's own validate_cpu_quota /
validate_memory_max before it is stored, since hive-priv renders it
verbatim into the container's drop-in, and one bad value refuses the whole
set (argus on #4268, carried over from the wanted-state consumer this
replaces). A failed read, or a bucket that does not answer, leaves the
stored caps as they are; three failures in a row raise a banner.

swarm-nats-auth grants every hive `$JS.API.DIRECT.GET.KV_agent-limits.>`:
read-only, and every agent's key, not just the hive's own. Scoping it to
the hive's agents would need a roster in the auth responder, which
policy.rs argues against. mara chose this on #4268 (c95293); narrowing it
is a possible low-priority follow-up, not planned. A test pins that the
grant reads every key, writes none, and reaches no other bucket.

Contributes to #3865.
This commit is contained in:
atlas 2026-10-12 11:26:18 +02:00 • committed by mara
commit fccf2ab892
8 changed files with 320 additions and 4 deletions

View file

@ -411,6 +411,13 @@ Contents:
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
drop-in have **no** per-agent override — they're hive-wide only and
come straight off `HiveEnv`, so this file has no field for them.
- `swarm-resource-limits.json` — the caps the swarm sets for this
hive's agents, same shape as `resource-limits.json`. Written only by
`workers::swarm_limits`, which copies each agent's key of the swarm's
`agent-limits` bucket here every 30 seconds. A field set here replaces
that field's answer from `resource-limits.json` and the hive-wide
default. Every value passes the same validators as the operator's
overrides first, and one bad value makes the worker change nothing here.
The root agent has the meta dir RO-mounted at `/meta/`.

View file

@ -452,14 +452,24 @@ regardless.
rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild.
The answer per agent and per field is, first match wins:
1. the swarm's cap for the agent (see below);
2. the operator's per-agent override in `meta/resource-limits.json`;
3. `agentCpuQuota` / `agentMemoryMax`.
A swarm cap replaces the hive's answer, so a hive can't raise an agent
above it. Like the rest of the drop-in, a changed swarm cap reaches a
running agent at its next spawn or rebuild.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the overrides file, the file carries
that drop-in's `MemoryMax=`.
the last drop-in because it can't read the overrides or the swarm caps file,
the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
@ -469,6 +479,34 @@ changed cap reaches them at the next container start without a rebuild.
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
<details><summary>Implementation: swarm caps</summary>
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
agent and no hive in it, so an agent that moves to another hive keeps its
caps. swarm-controller alone writes it:
| method | path | body |
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
`PUT` refuses an agent no hive's wanted state declares, unknown fields, zero
and anything systemd doesn't take, and stores memory the way systemd spells
it (`8gb` as `8G`).
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the same validators as the operator's overrides first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a
dashboard banner.
A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
hive's own agents would need a roster in the auth responder.
</details>
The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle

View file

@ -25,6 +25,9 @@ grafana, the controller, authelia, …) →
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
over its own. → [`inference.md`](inference.md)
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
over its hive's. →
[`coordinator.md`](../scheduler/coordinator.md#container-resource-limits)
- **review classifier** — planned, not implemented yet: checks a PR against
its repo's own yes/no rules and posts the result as a review. →
[`review-classifier.md`](review-classifier.md)