hive-c0re: read each agent's swarm caps from agent-limits
Every 30 seconds `workers::swarm_limits` reads the `agent-limits` key of each agent this hive has and stores the result in swarm-resource-limits.json, where the drop-in, the heap ceiling and the dashboard read it. A cap therefore follows its agent to whichever hive runs it. Each value goes through the hive's own validate_cpu_quota / validate_memory_max before it is stored, since hive-priv renders it verbatim into the container's drop-in, and one bad value refuses the whole set (argus on #4268, carried over from the wanted-state consumer this replaces). A failed read, or a bucket that does not answer, leaves the stored caps as they are; three failures in a row raise a banner. swarm-nats-auth grants every hive `$JS.API.DIRECT.GET.KV_agent-limits.>`: read-only, and every agent's key, not just the hive's own. Scoping it to the hive's agents would need a roster in the auth responder, which policy.rs argues against. mara chose this on #4268 (c95293); narrowing it is a possible low-priority follow-up, not planned. A test pins that the grant reads every key, writes none, and reaches no other bucket. Contributes to #3865.
This commit is contained in:
parent
ebeef4b84d
commit
fccf2ab892
8 changed files with 320 additions and 4 deletions
|
|
@ -452,14 +452,24 @@ regardless.
|
|||
rebuild, so changes take effect on the next lifecycle op without
|
||||
requiring a host rebuild.
|
||||
|
||||
The answer per agent and per field is, first match wins:
|
||||
|
||||
1. the swarm's cap for the agent (see below);
|
||||
2. the operator's per-agent override in `meta/resource-limits.json`;
|
||||
3. `agentCpuQuota` / `agentMemoryMax`.
|
||||
|
||||
A swarm cap replaces the hive's answer, so a hive can't raise an agent
|
||||
above it. Like the rest of the drop-in, a changed swarm cap reaches a
|
||||
running agent at its next spawn or rebuild.
|
||||
|
||||
The container can't read its own cap, so `write_dropins` also writes the
|
||||
effective `MemoryMax=` into the agent's runtime dir
|
||||
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
|
||||
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
|
||||
it, as that share of the host's physical memory rounded down to a page. A
|
||||
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
|
||||
the last drop-in because it can't read the overrides file, the file carries
|
||||
that drop-in's `MemoryMax=`.
|
||||
the last drop-in because it can't read the overrides or the swarm caps file,
|
||||
the file carries that drop-in's `MemoryMax=`.
|
||||
|
||||
At boot the container derives two values from a byte count:
|
||||
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
|
||||
|
|
@ -469,6 +479,34 @@ changed cap reaches them at the next container start without a rebuild.
|
|||
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
|
||||
container's journal. The container's `MemoryMax=` still applies.
|
||||
|
||||
<details><summary>Implementation: swarm caps</summary>
|
||||
|
||||
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
|
||||
agent and no hive in it, so an agent that moves to another hive keeps its
|
||||
caps. swarm-controller alone writes it:
|
||||
|
||||
| method | path | body |
|
||||
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
|
||||
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
|
||||
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
|
||||
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
|
||||
|
||||
`PUT` refuses an agent no hive's wanted state declares, unknown fields, zero
|
||||
and anything systemd doesn't take, and stores memory the way systemd spells
|
||||
it (`8gb` as `8G`).
|
||||
|
||||
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
||||
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
||||
value passes the same validators as the operator's overrides first. One bad
|
||||
value or a failed read leaves the file unchanged, and repeated failures raise a
|
||||
dashboard banner.
|
||||
|
||||
A hive's queue grant reads every key of the bucket
|
||||
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
|
||||
hive's own agents would need a roster in the auth responder.
|
||||
|
||||
</details>
|
||||
|
||||
The same drop-in carries `CPUWeight=` / `IOWeight=` from
|
||||
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
|
||||
setting: the quota and the memory max are **hard caps** that throttle
|
||||
|
|
|
|||
Loading…
Reference in a new issue