budgeted agents throttle at 90% of the cap; alert on repeated oom kills
This commit is contained in:
parent
c014b48c22
commit
11393bc538
1 changed files with 201 additions and 27 deletions
|
|
@ -446,11 +446,12 @@ changed cap reaches them at the next container start without a rebuild.
|
|||
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
|
||||
container's journal. The container's `MemoryMax=` still applies.
|
||||
|
||||
A container with a finite `MemoryMax=` also gets `MemorySwapMax=0` in the same
|
||||
drop-in, so a budgeted agent never swaps: paging it out to host swap would
|
||||
move the pressure onto the host. `MemoryMax=infinity` leaves swap alone. Like
|
||||
the rest of the drop-in, it reaches a running agent at its next spawn or
|
||||
rebuild.
|
||||
A container with a finite `MemoryMax=` also gets, in the same drop-in,
|
||||
`MemorySwapMax=0` and `MemoryHigh=` at 90% of the cap (bytes, rounded down).
|
||||
A budgeted agent never swaps: paging it out to host swap would move the
|
||||
pressure onto the host. Past 90% the kernel slows the container and reclaims
|
||||
from it, and at 100% it kills. `MemoryMax=infinity` sets neither. Like the
|
||||
rest of the drop-in, they reach a running agent at its next spawn or rebuild.
|
||||
|
||||
When the container reaches its cap, the kernel kills inside it, in this order:
|
||||
|
||||
|
|
@ -474,6 +475,8 @@ is the memory the container has swapped out: nonzero only for an agent with
|
|||
no budget, or one whose container started before its drop-in carried
|
||||
`MemorySwapMax=0`.
|
||||
|
||||
More than 3 kills of one agent in an hour raise the `hh-agent-oom-kills` alert.
|
||||
|
||||
<details><summary>Implementation: swarm caps</summary>
|
||||
|
||||
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
|
||||
|
|
|
|||
Loading…
Reference in a new issue