Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
6f2455f84a | ||
|
|
1a86304f92 | ||
|
|
c875223db7 | ||
|
|
23cedf3fa8 | ||
|
|
90e33cbd19 | ||
|
|
a2540be98b |
34 changed files with 232 additions and 1364 deletions
|
|
@ -395,26 +395,26 @@ Contents:
|
|||
(`{ "ruth": ["manage_root_agent"] }`). Written by
|
||||
`capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
|
||||
var. Absent agents have no extra capabilities.
|
||||
- `resource-limits.json` — per-agent container resource overrides
|
||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||
hive's agents
|
||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||
Written by `resource_limits::set_limits`; read where
|
||||
Written only by `workers::swarm_limits`, which copies each agent's key of
|
||||
the swarm's `agent-limits` bucket here every 30 seconds. Read where
|
||||
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
|
||||
into the container — these are host-side caps on the container, so
|
||||
the capped party never sets them. It sees only the resulting memory cap,
|
||||
as a byte count or `infinity` in its runtime dir (below). Fallback is per *field*:
|
||||
an absent file, absent agent, or absent field falls back to the
|
||||
as a byte count or `infinity` in its runtime dir (below). Fallback is per
|
||||
*field*: an absent file, absent agent, or absent field falls back to the
|
||||
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can override only its memory and still track the hive
|
||||
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
|
||||
drop-in have **no** per-agent override — they're hive-wide only and
|
||||
come straight off `HiveEnv`, so this file has no field for them.
|
||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||
hive's agents, same shape as `resource-limits.json`. Written only by
|
||||
`workers::swarm_limits`, which copies each agent's key of the swarm's
|
||||
`agent-limits` bucket here every 30 seconds. A field set here replaces
|
||||
that field's answer from `resource-limits.json` and the hive-wide
|
||||
default. Every value passes the same validators as the operator's
|
||||
overrides first, and one bad value makes the worker change nothing here.
|
||||
so an agent can have only its memory capped and still track the hive
|
||||
default for CPU. Every value passes the validators first, and one bad
|
||||
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
|
||||
shares in the same drop-in have **no** per-agent setting — they're
|
||||
hive-wide only and come straight off `HiveEnv`, so this file has no field
|
||||
for them.
|
||||
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
|
||||
to live here; nothing reads or writes it now. An old copy in the meta repo
|
||||
is harmless and you can delete it.
|
||||
|
||||
The root agent has the meta dir RO-mounted at `/meta/`.
|
||||
|
||||
|
|
|
|||
|
|
@ -455,12 +455,12 @@ requiring a host rebuild.
|
|||
The answer per agent and per field is, first match wins:
|
||||
|
||||
1. the swarm's cap for the agent (see below);
|
||||
2. the operator's per-agent override in `meta/resource-limits.json`;
|
||||
3. `agentCpuQuota` / `agentMemoryMax`.
|
||||
2. `agentCpuQuota` / `agentMemoryMax`.
|
||||
|
||||
A swarm cap replaces the hive's answer, so a hive can't raise an agent
|
||||
above it. Like the rest of the drop-in, a changed swarm cap reaches a
|
||||
running agent at its next spawn or rebuild.
|
||||
running agent at its next spawn or rebuild. A hive has no way to set a cap
|
||||
for one agent: you set caps per agent in the swarm UI.
|
||||
|
||||
The container can't read its own cap, so `write_dropins` also writes the
|
||||
effective `MemoryMax=` into the agent's runtime dir
|
||||
|
|
@ -468,7 +468,7 @@ effective `MemoryMax=` into the agent's runtime dir
|
|||
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
|
||||
it, as that share of the host's physical memory rounded down to a page. A
|
||||
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
|
||||
the last drop-in because it can't read the overrides or the swarm caps file,
|
||||
the last drop-in because it can't read the swarm caps file,
|
||||
the file carries that drop-in's `MemoryMax=`.
|
||||
|
||||
At boot the container derives two values from a byte count:
|
||||
|
|
@ -520,7 +520,7 @@ given.
|
|||
|
||||
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
||||
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
||||
value passes the same validators as the operator's overrides first. One bad
|
||||
value passes the validators first. One bad
|
||||
value or a failed read leaves the file unchanged, and repeated failures raise a
|
||||
dashboard banner.
|
||||
|
||||
|
|
@ -548,8 +548,8 @@ wants it.
|
|||
Two things to know about the weights:
|
||||
|
||||
- They're **hive-wide** — unlike the caps there is no per-agent
|
||||
override in `meta/resource-limits.json`, so every agent carries the
|
||||
same value and the weight does _not_ rank agents against each other.
|
||||
setting, so every agent carries the same value and the weight does _not_
|
||||
rank agents against each other.
|
||||
What `80` buys is that agents yield to everything **not** on this
|
||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
|
||||
|
|
|
|||
|
|
@ -291,8 +291,8 @@ cleanly and the affected level button goes on returning zero rows:
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
When OTEL is enabled, **hive-c0re itself** also exports each agent
|
||||
container's resource load — the same cgroup gauges shown on the dashboard
|
||||
LOAD tab — to this hive's own collector, exactly like an agent does and with
|
||||
container's resource load — the same cgroup gauges —
|
||||
to this hive's own collector, exactly like an agent does and with
|
||||
no separate toggle. These come from the host, not the in-container Claude SDK,
|
||||
so they cover containers even when their agent is idle.
|
||||
|
||||
|
|
@ -312,7 +312,7 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
|
|
|
|||
|
|
@ -22,7 +22,6 @@ This document contains the help content for the `hivectl` command-line program.
|
|||
* [`hivectl agent kill`↴](#hivectl-agent-kill)
|
||||
* [`hivectl agent destroy`↴](#hivectl-agent-destroy)
|
||||
* [`hivectl agent rebuild`↴](#hivectl-agent-rebuild)
|
||||
* [`hivectl agent set-limits`↴](#hivectl-agent-set-limits)
|
||||
* [`hivectl agent choom`↴](#hivectl-agent-choom)
|
||||
* [`hivectl agent watch`↴](#hivectl-agent-watch)
|
||||
* [`hivectl agent quota`↴](#hivectl-agent-quota)
|
||||
|
|
@ -221,7 +220,6 @@ Everything here targets a single named agent (`hivectl agent foo restart`, `hive
|
|||
* `kill` — Hard-stop this managed container
|
||||
* `destroy` — Tear down this sub-agent container, keeping its state by default. No undo
|
||||
* `rebuild` — Apply pending config to this managed container
|
||||
* `set-limits` — Declare this agent's CPU/memory limits, overriding the hive-wide defaults
|
||||
* `choom` — Open an interactive Claude session inside this agent's container
|
||||
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI
|
||||
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups
|
||||
|
|
@ -315,22 +313,6 @@ Apply pending config to this managed container
|
|||
|
||||
|
||||
|
||||
## `hivectl agent set-limits`
|
||||
|
||||
Declare this agent's CPU/memory limits, overriding the hive-wide defaults.
|
||||
|
||||
Replaces the agent's whole override entry rather than merging into it: any limit you don't pass returns to the hive-wide default. To change one and keep the other, pass both. Disk is a separate resource with its own group — see `quota`.
|
||||
|
||||
**Usage:** `hivectl agent set-limits [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--cpu-quota <CPU_QUOTA>` — systemd `CPUQuota=` value, for example `400%` (100% = one full core)
|
||||
* `--memory-max <MEMORY_MAX>` — systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`
|
||||
* `--reset` — Drop all overrides — the agent returns to the hive-wide defaults. Required to clear limits, so that a `set-limits` with a forgotten value can't silently reset the agent
|
||||
|
||||
|
||||
|
||||
## `hivectl agent choom`
|
||||
|
||||
Open an interactive Claude session inside this agent's container.
|
||||
|
|
|
|||
|
|
@ -136,34 +136,6 @@ resume drains the backlog rather than dropping it. Points worth knowing:
|
|||
- Visible as ` paused` in `list-agents`'s STATUS column, as a `paused`
|
||||
field on the JSON rows, and as a badge on the dashboard card.
|
||||
|
||||
### Per-agent resource limits
|
||||
|
||||
```bash
|
||||
hivectl agent sock set-limits --cpu-quota 400% --memory-max 8G
|
||||
hivectl agent sock set-limits --memory-max 8G # CPU falls back to the hive default
|
||||
hivectl agent sock set-limits --reset # drop all overrides
|
||||
```
|
||||
|
||||
Overrides the hive-wide `services.hyperhive.c0re.agentCpuQuota` /
|
||||
`agentMemoryMax` for one agent, persisted to
|
||||
`meta/resource-limits.json` (see
|
||||
[`persistence.md`](../agent-lifecycle/persistence.md)). Values are systemd's
|
||||
`CPUQuota=` / `MemoryMax=` syntax: a percentage (`400%` = four full
|
||||
cores) for CPU; a size (`8G`), a percentage of physical RAM, or
|
||||
`infinity` for memory. `set-limits` validates both before persisting them —
|
||||
they go into a systemd drop-in verbatim, and a typo there makes the
|
||||
unit fail to start.
|
||||
|
||||
**Declarative, not incremental**: each invocation replaces the agent's
|
||||
whole entry. `hivectl agent sock set-limits --memory-max 8G` leaves `sock`
|
||||
with _only_ a memory override, reverting any previously set CPU quota to
|
||||
the hive default. To avoid a forgotten flag silently wiping an override,
|
||||
hivectl rejects `set-limits` with no flags at all — clearing requires the
|
||||
explicit `--reset`.
|
||||
|
||||
The command rewrites the container's drop-in and reloads systemd, so
|
||||
new containers and restarts pick the values up immediately.
|
||||
|
||||
## Choom
|
||||
|
||||
Drop into an interactive Claude session inside an agent container.
|
||||
|
|
|
|||
|
|
@ -74,13 +74,17 @@ agent is stale.
|
|||
Passive / rare-interaction state, reached from the **Core** tile on the
|
||||
H0M3 hub (served at
|
||||
`/core.html`), with the same minimal chrome as `/builds.html`: a `← home`
|
||||
back-link + a `createTabStrip` sub-tab nav (**K3PT ST4T3** default,
|
||||
then **C0NT41N3R L04D**). The page is its own esbuild bundle (`core.js`)
|
||||
that cold-loads `/api/state` and subscribes to `/api/dashboard/stream` for
|
||||
`tombstones_changed`, `capabilities_changed`, and `tool_groups_changed`
|
||||
(the latter two re-render the stale-perms sub-section when permission
|
||||
data changes). (Rebuild queue and meta inputs live on the BU1LDS
|
||||
page — see below.)
|
||||
back-link and a single pane, **K3PT ST4T3**. The page is its own esbuild
|
||||
bundle (`core.js`) that cold-loads `/api/state` and subscribes to
|
||||
`/api/dashboard/stream` for `tombstones_changed`, `capabilities_changed`,
|
||||
and `tool_groups_changed` (the latter two re-render the stale-perms
|
||||
sub-section when permission data changes). (Rebuild queue and meta inputs
|
||||
live on the BU1LDS page — see below.)
|
||||
|
||||
Per-container CPU, memory and disk load isn't shown in the hive UI. The
|
||||
same figures go out as OTEL metrics and appear in Grafana (see
|
||||
[observability](../scheduler/observability.md)). An agent's CPU and memory
|
||||
caps live per agent in the swarm UI.
|
||||
|
||||
**K3PT ST4T3** — two sub-sections on one pane:
|
||||
|
||||
|
|
@ -102,32 +106,11 @@ snapshot-storage rework where the problem changes shape.
|
|||
*Stale permission entries*: agents with explicit capability or
|
||||
tool-group JSON entries but no live container — typically renamed or
|
||||
deleted agents whose entries persisted in `capabilities.json` /
|
||||
`tool-groups.json`. Lazy-loaded on first K3PT ST4T3 tab activation;
|
||||
`tool-groups.json`. Loaded when the page opens;
|
||||
autorefreshes on `capabilities_changed` + `tool_groups_changed` SSE
|
||||
events. Each ghost agent gets a `✕ clear perms` button
|
||||
(`DELETE /api/permissions/{name}`).
|
||||
|
||||
**C0NT41N3R L04D** — live CPU + memory per agent container, read
|
||||
straight from cgroup v2 on the host (`cpu.stat`, `memory.current`,
|
||||
`memory.peak`, `memory.max` under
|
||||
`/sys/fs/cgroup/machine.slice/machine-h\x2d<name>.scope/`). CPU is a
|
||||
host-normalised percentage (0..100 across all cores) sampled over a
|
||||
short (~200 ms) two-read interval; memory shows current + peak with a
|
||||
bar against the `memory.max` quota. Backed by
|
||||
`GET /api/container-resources` (`container_stats.rs`), which reads the
|
||||
files read-only (world-readable; no `hive-priv`) and skips agents whose
|
||||
scope dir is absent (= not running). Pull-only: `core.js` polls every
|
||||
5 s **only while the C0NT41N3R L04D sub-tab is active** (CPU needs a fresh
|
||||
sample each refresh), and stops on sub-tab change. Disk size (`disk_bytes`)
|
||||
rides the same row, but a separate background sampler feeds it every
|
||||
30 min: `hive-priv` runs one bounded `du -x` as root over the agent's
|
||||
`state/` + `work/` + container writable rootfs (shared nix store excluded),
|
||||
so the 5 s poll stays cheap cgroup-only reads. The row carries the last
|
||||
measured value; the column shows "unknown" (`null`) before the first
|
||||
measurement and whenever the last one failed, never 0. Network isn't tracked
|
||||
per container — see `docs/networking/network.md` for the current
|
||||
per-agent netns model.
|
||||
|
||||
## BU1LDS page (`/builds.html`)
|
||||
|
||||
The build lifecycle hub — rebuild queue, live build log, meta inputs,
|
||||
|
|
@ -765,17 +748,6 @@ below — some endpoints aren't in it yet.
|
|||
zero-turn dbs); returns swarm totals, a busiest-first per-agent
|
||||
rollup, swarm model mix, and a labelled `est_cost_usd`. Window
|
||||
defaults to `24h`.
|
||||
- `GET /api/container-resources` — live per-agent-container CPU +
|
||||
memory from cgroup v2 (C0R3 › C0NT41N3R L04D panel). Returns one
|
||||
row per running agent (`name`, `cpu_pct`, `mem_current_bytes`,
|
||||
`mem_peak_bytes`, `mem_max_bytes`, `disk_bytes`); samples CPU over
|
||||
~200 ms so the call briefly awaits. Skips non-running agents (no scope
|
||||
dir). No network field — agents share the host netns. `disk_bytes` is
|
||||
the last value from a **separate slow sampler** (not this hot path):
|
||||
every 30 min `hive-priv` runs a bounded `du -sxb` as root over the agent's
|
||||
`state/` + `work/` + container writable rootfs, `-x` excluding the shared
|
||||
read-only nix store (`PrivRequest::MeasureAgentDisk`). `null` means
|
||||
unknown: not measured yet, or the last measurement failed.
|
||||
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
||||
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
||||
failed (clears the failure state so the scheduler retries).
|
||||
|
|
|
|||
|
|
@ -619,8 +619,8 @@ hive-side-panel .md td {
|
|||
|
||||
/* ─── stats table ─────────────────────────────────────────────────
|
||||
Shared right-aligned numeric table. Used by /stats.html (busiest
|
||||
agents) and the dashboard SYST3M › C0NT41N3R L04D table — moved here
|
||||
from dashboard.css when ST4TS became its own page. */
|
||||
agents) — moved here from dashboard.css when ST4TS became its own
|
||||
page. */
|
||||
.hive-stats-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
/* ─── /core.html — C0R3 page ───────────────────────────────────────
|
||||
Kept state (tombstones) and container load. Section rules imported
|
||||
from system-sections.css (shared with dashboard.css + builds.css). */
|
||||
Kept state (tombstones) and stale permission entries. Section rules
|
||||
imported from system-sections.css (shared with dashboard.css +
|
||||
builds.css). */
|
||||
@import "./system-sections.css";
|
||||
|
||||
body.core-shell {
|
||||
|
|
@ -8,22 +9,10 @@ body.core-shell {
|
|||
padding: 0;
|
||||
}
|
||||
|
||||
/* Fill the header row next to the ← home back-link, same as the
|
||||
builds sub-tab strip. */
|
||||
.core-tabbar {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.core-main {
|
||||
padding: 1.2em 1.5em 2em;
|
||||
}
|
||||
|
||||
/* createTabStrip toggles the native `hidden` attribute on inactive
|
||||
panes; ensure it wins over any inherited display. */
|
||||
.core-pane[hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
/* ─── K3PT ST4T3 container cards ────────────────────────────────────
|
||||
core.html doesn't load dashboard.css (that's the operator SPA),
|
||||
so the .container-row card styles aren't inherited. Redefine them
|
||||
|
|
|
|||
|
|
@ -13,70 +13,31 @@
|
|||
<body class="core-shell">
|
||||
<!-- C0R3: the host/coordinator surface. Standalone page (served at
|
||||
/core.html) carved out of the dashboard's old SYST3M tab so the
|
||||
dashboard tab strip stays lean. Same minimal chrome as
|
||||
/builds.html — a `← home` back-link to the H0M3 hub + a
|
||||
<hive-tab-strip> sub-tab nav. Two sub-tabs: kept state (tombstones)
|
||||
and container load. Rebuild queue + meta inputs have moved to
|
||||
/builds.html (the build lifecycle hub). Default tab: K3PT ST4T3.
|
||||
The section <div> ids (tombstones-section, container-load-section)
|
||||
match what core.js's renderers target. Hive infra containers
|
||||
(`hive-forge`, `hive-gateway`, `hive-matrix`) have no
|
||||
dashboard panel — `hivectl stop`/`start`/`restart` is the only
|
||||
dashboard tab strip stays lean: kept state (tombstones) and the
|
||||
stale permission entries. The section <div> ids (tombstones-section,
|
||||
tombstones-stale-perms) match what core.js's renderers target. Hive
|
||||
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`) have
|
||||
no dashboard panel — `hivectl stop`/`start`/`restart` is the only
|
||||
control surface. -->
|
||||
<header class="page-header">
|
||||
<a class="page-back" href="/">← home</a>
|
||||
<hive-tab-strip
|
||||
class="hive-tabbar core-tabbar"
|
||||
id="core-tabbar"
|
||||
prefix="core"
|
||||
role="tablist"
|
||||
></hive-tab-strip>
|
||||
<span class="page-title">K3PT ST4T3</span>
|
||||
</header>
|
||||
|
||||
<main class="core-main">
|
||||
<!-- K3PT ST4T3: tombstoned-agent kept state + stale permission entries.
|
||||
tombstones-section shows destroyed agents (purge button).
|
||||
tombstones-stale-perms is lazy-loaded on tab activation and shows
|
||||
agents with explicit capability/tool-group entries but no live
|
||||
container (typically renamed/deleted agents whose JSON entries
|
||||
persisted). Each stale entry gets a "✕ clear perms" button. -->
|
||||
<section
|
||||
class="core-pane"
|
||||
id="core-pane-kept"
|
||||
data-tab-pane="kept"
|
||||
role="tabpanel"
|
||||
aria-labelledby="core-tab-kept"
|
||||
>
|
||||
<p class="meta">
|
||||
kept state from previously tombstoned agents — recreating an agent
|
||||
with the same name reuses it.
|
||||
</p>
|
||||
<div id="tombstones-section">
|
||||
<p class="meta">loading…</p>
|
||||
</div>
|
||||
<div id="tombstones-stale-perms"></div>
|
||||
</section>
|
||||
|
||||
<!-- C0NT41N3R L04D: live cpu + memory per agent container, from
|
||||
cgroup v2 on the host. Polled only while this sub-tab is active. -->
|
||||
<section
|
||||
class="core-pane"
|
||||
id="core-pane-load"
|
||||
data-tab-pane="load"
|
||||
role="tabpanel"
|
||||
aria-labelledby="core-tab-load"
|
||||
>
|
||||
<p class="meta">
|
||||
live cpu + memory per agent container, from cgroup v2 on the host. cpu
|
||||
is % of total host capacity (all cores), sampled over ~200ms each
|
||||
refresh; polled every 5s while this tab is open. network is omitted on
|
||||
purpose — agents share the host netns, so there is no per-container
|
||||
counter (see <code>docs/networking/network.md</code>).
|
||||
</p>
|
||||
<div id="container-load-section">
|
||||
<p class="meta">loading…</p>
|
||||
</div>
|
||||
</section>
|
||||
<!-- tombstones-section shows destroyed agents (purge button).
|
||||
tombstones-stale-perms shows agents with explicit capability/
|
||||
tool-group entries but no live container (typically renamed/deleted
|
||||
agents whose JSON entries persisted). Each stale entry gets a
|
||||
"✕ clear perms" button. -->
|
||||
<p class="meta">
|
||||
kept state from previously tombstoned agents — recreating an agent with
|
||||
the same name reuses it.
|
||||
</p>
|
||||
<div id="tombstones-section">
|
||||
<p class="meta">loading…</p>
|
||||
</div>
|
||||
<div id="tombstones-stale-perms"></div>
|
||||
</main>
|
||||
|
||||
<script type="module" src="/static/core.js" defer></script>
|
||||
|
|
|
|||
|
|
@ -1,27 +1,20 @@
|
|||
// C0R3 page entry (/core.html). The host/coordinator surface, its own
|
||||
// standalone page so the dashboard tab strip stays lean. Two sub-tabs
|
||||
// via <hive-tab-strip> (default = K3PT ST4T3): kept state (tombstones)
|
||||
// and container load.
|
||||
// standalone page so the dashboard tab strip stays lean: kept state
|
||||
// (tombstones) and the stale permission entries.
|
||||
//
|
||||
// Rebuild queue + meta inputs live at /builds.html (the build lifecycle
|
||||
// hub), not here. This page is its own esbuild bundle with its own
|
||||
// runtime: it cold-loads /api/state and subscribes to /dashboard/stream
|
||||
// (the same broker event channel the dashboard uses), maintaining its
|
||||
// own copy of the tombstones state.
|
||||
// This page is its own esbuild bundle with its own runtime: it cold-loads
|
||||
// /api/state and subscribes to /dashboard/stream (the same broker event
|
||||
// channel the dashboard uses), maintaining its own copy of the tombstones
|
||||
// state.
|
||||
|
||||
import { $, form, openStream, initServerWarnings } from "./common.js";
|
||||
import { el } from "@hive/shared/dom.js";
|
||||
import { asyncBtn, bindAsyncForms } from "@hive/shared/forms.js";
|
||||
import "@hive/shared/hive-tab-strip.js";
|
||||
|
||||
// ─── derived state (own copies; this bundle has its own runtime) ──────────
|
||||
let tombstonesState = [];
|
||||
// ContainerView list from the last /api/state snapshot — used by renderContainerLoad
|
||||
// to splice in the configured cpu_quota/memory_max limits alongside live cgroup data.
|
||||
let containersState = [];
|
||||
function syncFromSnapshot(s) {
|
||||
tombstonesState = (s.tombstones || []).slice();
|
||||
containersState = (s.containers || []).slice();
|
||||
}
|
||||
|
||||
// ─── kept state (tombstones) ──────────────────────────────────────────────
|
||||
|
|
@ -187,273 +180,6 @@ async function fetchAndRenderStalePerms() {
|
|||
stalePermsLoaded = true;
|
||||
}
|
||||
|
||||
// ─── container load (live cgroup poll while the LOAD sub-tab is open) ──────
|
||||
let containerLoadTimer = null;
|
||||
|
||||
function cloadFmtBytes(n) {
|
||||
if (!Number.isFinite(n) || n <= 0) return "0";
|
||||
const u = ["B", "KiB", "MiB", "GiB", "TiB"];
|
||||
let i = 0;
|
||||
let v = n;
|
||||
while (v >= 1024 && i < u.length - 1) {
|
||||
v /= 1024;
|
||||
i += 1;
|
||||
}
|
||||
return v.toFixed(v < 10 && i > 0 ? 1 : 0) + " " + u[i];
|
||||
}
|
||||
function cloadMeter(pct) {
|
||||
const p = Math.max(0, Math.min(100, pct));
|
||||
const cls = p >= 90 ? "hot" : p >= 70 ? "warn" : "";
|
||||
const m = document.createElement("span");
|
||||
m.className = "cload-meter";
|
||||
m.title = p.toFixed(0) + "%";
|
||||
const fill = document.createElement("span");
|
||||
fill.className = "fill" + (cls ? " " + cls : "");
|
||||
fill.style.width = p + "%";
|
||||
m.append(fill);
|
||||
return m;
|
||||
}
|
||||
|
||||
// Stash the last load rows so re-renders triggered by SSE state updates can
|
||||
// call renderContainerLoad(lastLoadRows) without waiting for the next poll.
|
||||
let lastLoadRows = [];
|
||||
|
||||
function renderContainerLoad(rows) {
|
||||
lastLoadRows = rows;
|
||||
const root = $("container-load-section");
|
||||
if (!root) return;
|
||||
if (!Array.isArray(rows) || rows.length === 0) {
|
||||
root.replaceChildren();
|
||||
const p = document.createElement("p");
|
||||
p.className = "meta";
|
||||
p.textContent = "no running agent containers";
|
||||
root.append(p);
|
||||
return;
|
||||
}
|
||||
|
||||
// Build a name→ContainerView lookup so we can splice in the configured
|
||||
// cpu_quota / memory_max limits (set in meta/resource-limits.json and
|
||||
// resolved by the server) next to the live cgroup readings.
|
||||
const cvByName = new Map(containersState.map((c) => [c.name, c]));
|
||||
|
||||
const table = document.createElement("table");
|
||||
table.className = "hive-stats-table";
|
||||
// "cpu cap" / "mem cap" show the configured ceilings from ContainerView
|
||||
// (effective drop-in values, take effect on next start/restart).
|
||||
// "limit" remains the live cgroup memory ceiling from /api/container-resources.
|
||||
table.innerHTML =
|
||||
"<thead><tr><th>agent</th><th>cpu</th><th>memory</th>" +
|
||||
"<th>peak</th><th>limit</th><th>disk</th>" +
|
||||
'<th class="cload-cap-th" title="configured ceiling — takes effect on next start">cpu cap</th>' +
|
||||
'<th class="cload-cap-th" title="configured ceiling — takes effect on next start">mem cap</th>' +
|
||||
"<th></th></tr></thead>";
|
||||
const tb = document.createElement("tbody");
|
||||
for (const r of rows) {
|
||||
const cv = cvByName.get(r.name);
|
||||
const tr = document.createElement("tr");
|
||||
const name = document.createElement("td");
|
||||
name.textContent = r.name;
|
||||
tr.append(name);
|
||||
const cpu = document.createElement("td");
|
||||
cpu.className = "num";
|
||||
cpu.append(
|
||||
(Number(r.cpu_pct) || 0).toFixed(1) + "%",
|
||||
cloadMeter(Number(r.cpu_pct) || 0),
|
||||
);
|
||||
tr.append(cpu);
|
||||
const mem = document.createElement("td");
|
||||
mem.className = "num";
|
||||
const memCur = Number(r.mem_current_bytes) || 0;
|
||||
if (r.mem_max_bytes) {
|
||||
mem.append(
|
||||
cloadFmtBytes(memCur),
|
||||
cloadMeter((100 * memCur) / r.mem_max_bytes),
|
||||
);
|
||||
} else {
|
||||
mem.textContent = cloadFmtBytes(memCur);
|
||||
}
|
||||
tr.append(mem);
|
||||
const peak = document.createElement("td");
|
||||
peak.className = "num";
|
||||
peak.textContent = r.mem_peak_bytes
|
||||
? cloadFmtBytes(Number(r.mem_peak_bytes))
|
||||
: "—";
|
||||
tr.append(peak);
|
||||
const lim = document.createElement("td");
|
||||
lim.className = "num";
|
||||
lim.textContent = r.mem_max_bytes
|
||||
? cloadFmtBytes(Number(r.mem_max_bytes))
|
||||
: "∞";
|
||||
tr.append(lim);
|
||||
// Disk: on-disk footprint (state/ + work/ + container writable rootfs, shared
|
||||
// nix store excluded). Measured out-of-band every 30 min server-side;
|
||||
// null means unknown (not measured yet, or the measurement failed).
|
||||
const disk = document.createElement("td");
|
||||
disk.className = "num";
|
||||
disk.title =
|
||||
"state/ + work/ + container writable rootfs (shared nix store excluded); measured every 30 min";
|
||||
disk.textContent =
|
||||
r.disk_bytes != null ? cloadFmtBytes(Number(r.disk_bytes)) : "unknown";
|
||||
tr.append(disk);
|
||||
|
||||
// Configured CPU / memory caps from ContainerView (resolved effective
|
||||
// values: per-agent override when set, hive-wide default otherwise).
|
||||
const cpuCap = document.createElement("td");
|
||||
cpuCap.className = "num cload-cap";
|
||||
cpuCap.title = "configured ceiling — takes effect on next start";
|
||||
cpuCap.textContent = cv?.cpu_quota || "—";
|
||||
tr.append(cpuCap);
|
||||
const memCap = document.createElement("td");
|
||||
memCap.className = "num cload-cap";
|
||||
memCap.title = "configured ceiling — takes effect on next start";
|
||||
memCap.textContent = cv?.memory_max || "—";
|
||||
tr.append(memCap);
|
||||
|
||||
// S3T button toggles the inline edit row for this agent.
|
||||
const actTd = document.createElement("td");
|
||||
const setBtn = document.createElement("button");
|
||||
setBtn.type = "button";
|
||||
setBtn.className = "btn btn-sm cload-set-btn";
|
||||
setBtn.textContent = "S3T";
|
||||
setBtn.title = "set CPU / memory cap for " + r.name;
|
||||
tr.append(actTd);
|
||||
actTd.append(setBtn);
|
||||
tb.append(tr);
|
||||
|
||||
// Inline edit row (hidden by default, toggled by the S3T button).
|
||||
const editRow = document.createElement("tr");
|
||||
editRow.className = "cload-edit-row";
|
||||
editRow.hidden = true;
|
||||
const editTd = document.createElement("td");
|
||||
editTd.colSpan = 9;
|
||||
editTd.className = "cload-edit-cell";
|
||||
|
||||
const editForm = document.createElement("form");
|
||||
editForm.className = "cload-edit-form";
|
||||
editForm.addEventListener("submit", async (e) => {
|
||||
e.preventDefault();
|
||||
const cpuInput = editForm.querySelector(".cload-cpu-input");
|
||||
const memInput = editForm.querySelector(".cload-mem-input");
|
||||
const errSpan = editForm.querySelector(".cload-edit-err");
|
||||
const submitBtn = editForm.querySelector('[type="submit"]');
|
||||
errSpan.hidden = true;
|
||||
submitBtn.disabled = true;
|
||||
try {
|
||||
const body = new URLSearchParams({
|
||||
cpu_quota: cpuInput.value.trim(),
|
||||
memory_max: memInput.value.trim(),
|
||||
});
|
||||
const resp = await fetch(
|
||||
"/api/resource-limits/" + encodeURIComponent(r.name),
|
||||
{ method: "POST", body },
|
||||
);
|
||||
if (!resp.ok) {
|
||||
errSpan.textContent = await resp
|
||||
.text()
|
||||
.catch(() => "error " + resp.status);
|
||||
errSpan.hidden = false;
|
||||
return;
|
||||
}
|
||||
// Success: collapse the edit row. The SSE rescan will push updated
|
||||
// ContainerView data (cpu_quota/memory_max) to containersState,
|
||||
// triggering a re-render of the cap columns via refreshContainerLoad.
|
||||
editRow.hidden = true;
|
||||
setBtn.textContent = "S3T";
|
||||
} catch (err) {
|
||||
errSpan.textContent = String(err);
|
||||
errSpan.hidden = false;
|
||||
} finally {
|
||||
submitBtn.disabled = false;
|
||||
}
|
||||
});
|
||||
|
||||
const cpuLabel = document.createElement("label");
|
||||
cpuLabel.className = "cload-edit-label";
|
||||
cpuLabel.textContent = "cpu quota";
|
||||
const cpuInput = document.createElement("input");
|
||||
cpuInput.type = "text";
|
||||
cpuInput.className = "cload-cpu-input";
|
||||
// Pre-fill with the current *effective* value (not just a placeholder)
|
||||
// so submitting after editing only the other field round-trips this one
|
||||
// unchanged. The form always posts both fields, and the server treats
|
||||
// an empty field as "clear this override" — a placeholder-only hint
|
||||
// reads as filled to the eye but posts as empty, silently resetting
|
||||
// whichever field the user didn't touch.
|
||||
cpuInput.value = cv ? cv.cpu_quota : "";
|
||||
cpuInput.placeholder = "e.g. 200%";
|
||||
cpuInput.title =
|
||||
'systemd CPUQuota= value (e.g. "400%"). empty = use hive default';
|
||||
cpuLabel.append(cpuInput);
|
||||
|
||||
const memLabel = document.createElement("label");
|
||||
memLabel.className = "cload-edit-label";
|
||||
memLabel.textContent = "mem max";
|
||||
const memInput = document.createElement("input");
|
||||
memInput.type = "text";
|
||||
memInput.className = "cload-mem-input";
|
||||
// Same reasoning as cpuInput above.
|
||||
memInput.value = cv ? cv.memory_max : "";
|
||||
memInput.placeholder = "e.g. 8G";
|
||||
memInput.title =
|
||||
'systemd MemoryMax= value (e.g. "8G", "50%", "infinity"). empty = use hive default';
|
||||
memLabel.append(memInput);
|
||||
|
||||
const submitBtn = document.createElement("button");
|
||||
submitBtn.type = "submit";
|
||||
submitBtn.className = "btn btn-restart cload-save-btn";
|
||||
submitBtn.textContent = "S4V3";
|
||||
|
||||
const hintSpan = document.createElement("span");
|
||||
hintSpan.className = "meta cload-edit-hint";
|
||||
hintSpan.textContent = "↺ restart to apply to a running container";
|
||||
|
||||
const errSpan = document.createElement("span");
|
||||
errSpan.className = "cload-edit-err";
|
||||
errSpan.hidden = true;
|
||||
|
||||
editForm.append(cpuLabel, memLabel, submitBtn, hintSpan, errSpan);
|
||||
editTd.append(editForm);
|
||||
editRow.append(editTd);
|
||||
tb.append(editRow);
|
||||
|
||||
setBtn.addEventListener("click", () => {
|
||||
const open = !editRow.hidden;
|
||||
editRow.hidden = open;
|
||||
setBtn.textContent = open ? "S3T" : "✕";
|
||||
});
|
||||
}
|
||||
table.append(tb);
|
||||
root.replaceChildren(table);
|
||||
}
|
||||
|
||||
async function refreshContainerLoad() {
|
||||
try {
|
||||
const resp = await fetch("/api/container-resources");
|
||||
if (!resp.ok) throw new Error("http " + resp.status);
|
||||
renderContainerLoad(await resp.json());
|
||||
} catch (e) {
|
||||
const root = $("container-load-section");
|
||||
if (root) {
|
||||
root.replaceChildren();
|
||||
const p = document.createElement("p");
|
||||
p.className = "meta";
|
||||
p.textContent = "container load fetch failed: " + e;
|
||||
root.append(p);
|
||||
}
|
||||
}
|
||||
}
|
||||
function startContainerLoadPolling() {
|
||||
refreshContainerLoad();
|
||||
if (containerLoadTimer) return;
|
||||
containerLoadTimer = setInterval(refreshContainerLoad, 5000);
|
||||
}
|
||||
function stopContainerLoadPolling() {
|
||||
if (containerLoadTimer) {
|
||||
clearInterval(containerLoadTimer);
|
||||
containerLoadTimer = null;
|
||||
}
|
||||
}
|
||||
|
||||
// ─── render-all (cold load + any full re-render) ──────────────────────────
|
||||
function renderAll() {
|
||||
renderTombstones({ tombstones: tombstonesState });
|
||||
|
|
@ -465,19 +191,6 @@ const SSE_HANDLERS = {
|
|||
tombstonesState = (ev.tombstones || []).slice();
|
||||
renderTombstones({ tombstones: tombstonesState });
|
||||
},
|
||||
// When a container's state changes (e.g. after resource-limits update
|
||||
// triggers rescan_containers_and_emit), update containersState in place
|
||||
// so the cap columns in the LOAD table reflect the new configured values.
|
||||
container_state_changed(ev) {
|
||||
if (!ev.container) return;
|
||||
const idx = containersState.findIndex((c) => c.name === ev.container.name);
|
||||
if (idx >= 0) {
|
||||
containersState[idx] = ev.container;
|
||||
} else {
|
||||
containersState.push(ev.container);
|
||||
}
|
||||
if (lastLoadRows.length) renderContainerLoad(lastLoadRows);
|
||||
},
|
||||
// Refresh the stale-perms sub-section when perm data changes (a ghost
|
||||
// was cleared, or perms were saved for an agent whose name collides).
|
||||
capabilities_changed(_ev) {
|
||||
|
|
@ -511,33 +224,16 @@ async function init() {
|
|||
// to the bare `ok` response page.
|
||||
bindAsyncForms(() => refreshState());
|
||||
|
||||
// Hash-routed sub-tab strip; default K3PT ST4T3. Container-load
|
||||
// polling runs only while the LOAD sub-tab is open (cpu is a short
|
||||
// two-sample read each refresh on the server).
|
||||
document.getElementById("core-tabbar").configure({
|
||||
tabs: [
|
||||
{ id: "kept", label: "K3PT ST4T3" },
|
||||
{ id: "load", label: "C0NT41N3R L04D" },
|
||||
],
|
||||
defaultId: "kept",
|
||||
onShow: (id) => {
|
||||
if (id === "load") startContainerLoadPolling();
|
||||
else stopContainerLoadPolling();
|
||||
// Lazy-load stale-perms on first K3PT ST4T3 activation; always
|
||||
// re-fetch on subsequent visits in case perms changed.
|
||||
if (id === "kept") fetchAndRenderStalePerms();
|
||||
},
|
||||
});
|
||||
|
||||
await refreshState();
|
||||
fetchAndRenderStalePerms();
|
||||
|
||||
// kinds= matches SSE_HANDLERS below verbatim — narrows this from all
|
||||
// 13 wire kinds down to the 4 this page acts on. This page was one
|
||||
// 13 wire kinds down to the 3 this page acts on. This page was one
|
||||
// of 4 unfiltered `/api/dashboard/stream` subscribers before this
|
||||
// (subscription discipline, part 1 of the dashboard-event-stream-
|
||||
// split issue).
|
||||
const es = openStream(
|
||||
"/api/dashboard/stream?kinds=tombstones_changed,container_state_changed," +
|
||||
"/api/dashboard/stream?kinds=tombstones_changed," +
|
||||
"capabilities_changed,tool_groups_changed",
|
||||
);
|
||||
if (es) {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
it defines the CSS custom properties, typography, and shared
|
||||
component rules that this file depends on. */
|
||||
|
||||
/* SYST3M panel rules (rebuild queue / meta inputs / container load)
|
||||
/* SYST3M panel rules (rebuild queue / meta inputs / kept state)
|
||||
live in system-sections.css, shared with /core.html. A few are still
|
||||
referenced from the dashboard (`.rebuild-queue`), so the dashboard
|
||||
pulls them in here. */
|
||||
|
|
@ -771,6 +771,4 @@ body.dashboard-shell.has-selection {
|
|||
|
||||
/* ST4TS moved to its own page (`/stats.html`): the
|
||||
window selector / summary chips / bars live in stats.css, and the
|
||||
shared `.hive-stats-table` moved to common.css (the SYST3M ›
|
||||
C0NT41N3R L04D table in system-sections.css / on /core.html still
|
||||
uses it). */
|
||||
shared `.hive-stats-table` moved to common.css. */
|
||||
|
|
|
|||
|
|
@ -103,7 +103,7 @@
|
|||
</section>
|
||||
|
||||
<!-- SYST3M pane moved to /core.html (the standalone "C0R3" page):
|
||||
meta inputs, rebuild queue, kept state, container load. Reached
|
||||
kept state and stale permissions. Reached
|
||||
from the H0M3 hub. -->
|
||||
|
||||
<!-- P3RM1SS10NS: per-agent capability grants + tool-group
|
||||
|
|
|
|||
|
|
@ -79,7 +79,7 @@
|
|||
<span class="home-tile-icon" aria-hidden="true">⬡</span>
|
||||
<span class="home-tile-label">Core</span>
|
||||
</span>
|
||||
<span class="home-tile-desc">kept state · container load</span>
|
||||
<span class="home-tile-desc">kept state · stale permissions</span>
|
||||
</a>
|
||||
|
||||
<!-- API tile: the OpenAPI spec + Swagger UI are always served by
|
||||
|
|
|
|||
|
|
@ -2,8 +2,7 @@
|
|||
// dashboard SPA: the live agent roster.
|
||||
//
|
||||
// The roster is read by nearly every tab (the container tree, the
|
||||
// capabilities + tool-group matrices, the
|
||||
// container-load poll, and the selection bar), so
|
||||
// capabilities + tool-group matrices, and the selection bar), so
|
||||
// it lives here as a single source of truth that each domain module
|
||||
// imports rather than threading through call signatures. Everything
|
||||
// else in the dashboard is domain-local and lives with its own module.
|
||||
|
|
|
|||
|
|
@ -3,8 +3,7 @@
|
|||
chrome pattern as /flow.html: the back-link header +
|
||||
title (.page-header / .page-back / .page-title) live in
|
||||
@hive/shared/chrome.css.
|
||||
`.hive-stats-table` also lives in common.css — it's shared with the
|
||||
dashboard SYST3M › C0NT41N3R L04D table. Only the ST4TS-specific
|
||||
`.hive-stats-table` lives in common.css. Only the ST4TS-specific
|
||||
window selector, summary chips, and bars are page-specific. */
|
||||
|
||||
body.stats-shell {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
/* ─── SYST3M/BU1LDS section panels — shared by dashboard, /core.html,
|
||||
and /builds.html ──────────────────────────────────────────────────
|
||||
Rebuild queue, meta inputs, kept state, and container-load panels.
|
||||
Rebuild queue, meta inputs and kept-state panels.
|
||||
Imported by core.css, dashboard.css, and builds.css. */
|
||||
|
||||
/* ─── meta inputs panel ────────────────────────────────────────────── */
|
||||
|
|
@ -198,83 +198,6 @@
|
|||
color: var(--fg);
|
||||
}
|
||||
|
||||
/* ─── SYST3M › C0NT41N3R L04D: live cgroup cpu/mem ─────────────────────────────
|
||||
Reuses the shared `.hive-stats-table` styling (in common.css); only
|
||||
the inline meter bar is new. */
|
||||
.cload-meter {
|
||||
display: inline-block;
|
||||
width: 6em;
|
||||
height: 10px;
|
||||
vertical-align: middle;
|
||||
margin-left: 6px;
|
||||
background: var(--bg-elev);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 3px;
|
||||
overflow: hidden;
|
||||
}
|
||||
.cload-meter .fill {
|
||||
display: block;
|
||||
height: 100%;
|
||||
background: var(--green);
|
||||
}
|
||||
.cload-meter .fill.warn {
|
||||
background: var(--amber);
|
||||
}
|
||||
.cload-meter .fill.hot {
|
||||
background: var(--red);
|
||||
}
|
||||
|
||||
/* Configured-cap columns: muted header so they read apart from live metrics.
|
||||
The S3T button sits in its own column; keep it compact. */
|
||||
.cload-cap-th {
|
||||
color: var(--muted);
|
||||
}
|
||||
.cload-cap {
|
||||
color: var(--muted);
|
||||
}
|
||||
.cload-set-btn {
|
||||
padding: 0.1em 0.5em;
|
||||
font-size: 0.8em;
|
||||
}
|
||||
|
||||
/* Inline edit row — spans all columns, contains a mini-form. */
|
||||
.cload-edit-row td {
|
||||
border-top: none;
|
||||
padding-top: 0;
|
||||
}
|
||||
.cload-edit-cell {
|
||||
padding: 0.2em 0.4em 0.6em 1.2em;
|
||||
}
|
||||
.cload-edit-form {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: center;
|
||||
gap: 0.5em;
|
||||
}
|
||||
.cload-edit-label {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.3em;
|
||||
font-size: 0.85em;
|
||||
color: var(--muted);
|
||||
}
|
||||
.cload-cpu-input,
|
||||
.cload-mem-input {
|
||||
width: 7em;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
.cload-save-btn {
|
||||
padding: 0.15em 0.6em;
|
||||
font-size: 0.85em;
|
||||
}
|
||||
.cload-edit-hint {
|
||||
font-size: 0.8em;
|
||||
}
|
||||
.cload-edit-err {
|
||||
color: var(--red);
|
||||
font-size: 0.85em;
|
||||
}
|
||||
|
||||
/* Remove the now-dead `.rqe-step` rule (the `step` sub-step label was
|
||||
retired from the wire in `feat(job_queue): retire the now-off-wire step
|
||||
sub-step label` — the rule has had no JS referents since then). */
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
//! Per-agent CPU/memory limit overrides. Stored at
|
||||
//! `/var/lib/hyperhive/meta/resource-limits.json` alongside
|
||||
//! CPU/memory limits for an agent's container: the caps the swarm controller
|
||||
//! declared, and the validators for the values the systemd drop-in takes.
|
||||
//!
|
||||
//! The swarm caps are stored at
|
||||
//! `/var/lib/hyperhive/meta/swarm-resource-limits.json` alongside
|
||||
//! `topology.json`, `tool-groups.json` and `capabilities.json`.
|
||||
//!
|
||||
//! Format: a JSON object mapping agent name to an object with optional
|
||||
|
|
@ -14,26 +17,22 @@
|
|||
//!
|
||||
//! Fallback is **per field**: an absent file, an absent agent, or an
|
||||
//! absent field all fall back to the hive-wide
|
||||
//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So an agent
|
||||
//! can raise only its memory cap and keep tracking the hive default for
|
||||
//! CPU — see [`effective`].
|
||||
//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So a swarm cap on
|
||||
//! memory alone leaves CPU on the hive default — see
|
||||
//! [`effective_from`].
|
||||
//!
|
||||
//! Read path: `lifecycle::host_config::write_dropins`, on every spawn
|
||||
//! and every rebuild.
|
||||
//!
|
||||
//! Why host-side JSON and not an option in the agent's own `agent.nix`:
|
||||
//! the drop-in lands on the *host's* `container@h-<name>.service`, so
|
||||
//! c0re would have to `nix eval` the agent's whole nixosConfiguration
|
||||
//! just to read two strings. It is also the wrong trust boundary —
|
||||
//! a resource *cap* should not be sourced from the capped party.
|
||||
//! the drop-in lands on the *host's* `container@h-<name>.service`, and a
|
||||
//! resource *cap* should not be sourced from the capped party.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
const RESOURCE_LIMITS_FILE: &str = "resource-limits.json";
|
||||
|
||||
/// One agent's overrides. Both fields optional and independent; `None`
|
||||
/// means "use the hive-wide default for this field".
|
||||
/// One agent's caps. Both fields optional and independent; `None` means
|
||||
/// "use the hive-wide default for this field".
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct AgentLimits {
|
||||
/// systemd `CPUQuota=` value, e.g. `"400%"`.
|
||||
|
|
@ -44,84 +43,6 @@ pub struct AgentLimits {
|
|||
pub memory_max: Option<String>,
|
||||
}
|
||||
|
||||
impl AgentLimits {
|
||||
/// True when neither field is set — such an entry is dropped rather
|
||||
/// than persisted as an empty object.
|
||||
#[must_use]
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.cpu_quota.is_none() && self.memory_max.is_none()
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn resource_limits_path() -> PathBuf {
|
||||
crate::paths::meta_root().join(RESOURCE_LIMITS_FILE)
|
||||
}
|
||||
|
||||
/// Read the per-agent limit map. An absent file is the empty map —
|
||||
/// callers treat a missing entry as "hive-wide defaults". A file that
|
||||
/// exists but can't be read or parsed is an error: an override can be
|
||||
/// tighter than the hive default, so reading it as empty is not a safe
|
||||
/// fallback.
|
||||
pub fn read() -> std::io::Result<BTreeMap<String, AgentLimits>> {
|
||||
super::read_map(&resource_limits_path())
|
||||
}
|
||||
|
||||
/// Resolve the effective values for an agent from the override file at
|
||||
/// `path` (normally [`resource_limits_path`]), filling each unset field
|
||||
/// from the hive-wide default. This is the single place the fallback
|
||||
/// rule lives; `write_dropins` calls it and passes the result straight
|
||||
/// to systemd. Errors as [`read`] does.
|
||||
///
|
||||
/// Use [`effective_from`] when resolving more than one agent in a row.
|
||||
pub fn effective(
|
||||
path: &Path,
|
||||
name: &str,
|
||||
hive_cpu_quota: &str,
|
||||
hive_memory_max: &str,
|
||||
) -> std::io::Result<(String, String)> {
|
||||
let limits = super::read_map(path)?;
|
||||
Ok(effective_from(
|
||||
&limits,
|
||||
name,
|
||||
hive_cpu_quota,
|
||||
hive_memory_max,
|
||||
))
|
||||
}
|
||||
|
||||
/// [`effective`] against an already-loaded map — the multi-agent form.
|
||||
///
|
||||
/// `container_view::build_all` renders every agent on each SSE scan, so
|
||||
/// it loads the map once and calls this per agent rather than re-reading
|
||||
/// the same small file N times per scan.
|
||||
#[must_use]
|
||||
pub fn effective_from(
|
||||
limits: &BTreeMap<String, AgentLimits>,
|
||||
name: &str,
|
||||
hive_cpu_quota: &str,
|
||||
hive_memory_max: &str,
|
||||
) -> (String, String) {
|
||||
match limits.get(name) {
|
||||
Some(l) => resolve(l, hive_cpu_quota, hive_memory_max),
|
||||
None => (hive_cpu_quota.to_owned(), hive_memory_max.to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Pure core of [`effective`], split out so the fallback matrix is
|
||||
/// testable without touching the filesystem.
|
||||
#[must_use]
|
||||
fn resolve(limits: &AgentLimits, hive_cpu_quota: &str, hive_memory_max: &str) -> (String, String) {
|
||||
let cpu = limits
|
||||
.cpu_quota
|
||||
.clone()
|
||||
.unwrap_or_else(|| hive_cpu_quota.to_owned());
|
||||
let mem = limits
|
||||
.memory_max
|
||||
.clone()
|
||||
.unwrap_or_else(|| hive_memory_max.to_owned());
|
||||
(cpu, mem)
|
||||
}
|
||||
|
||||
/// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`,
|
||||
/// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`)
|
||||
/// into a raw byte count, accepting exactly the single-term forms systemd's
|
||||
|
|
@ -164,35 +85,7 @@ pub fn parse_bytes(value: &str) -> Option<u64> {
|
|||
u64::try_from(bytes).ok()
|
||||
}
|
||||
|
||||
/// Set one agent's overrides and persist the map atomically. An entry
|
||||
/// with both fields unset is removed rather than stored, so "reset to
|
||||
/// hive defaults" and "never configured" are the same state on disk.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Fails without writing when the existing file can't be read or parsed
|
||||
/// (`InvalidData` for a parse failure), and otherwise when the meta dir
|
||||
/// or the file can't be written.
|
||||
pub fn set_limits(name: &str, limits: &AgentLimits) -> std::io::Result<()> {
|
||||
set_limits_at(&resource_limits_path(), name, limits)
|
||||
}
|
||||
|
||||
fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Result<()> {
|
||||
let mut current: BTreeMap<String, AgentLimits> = super::read_map(path)?;
|
||||
if limits.is_empty() {
|
||||
current.remove(name);
|
||||
} else {
|
||||
current.insert(name.to_owned(), limits.clone());
|
||||
}
|
||||
super::write_map(path, ¤t)
|
||||
}
|
||||
|
||||
/// Caps the swarm controller declared for this hive's agents.
|
||||
///
|
||||
/// A separate file from the operator's overrides above, not a merge into it:
|
||||
/// the two have different writers and different lifetimes, and a swarm
|
||||
/// declaration landing in the file an operator edits would make "reset to
|
||||
/// hive defaults" destroy something the operator never set.
|
||||
const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json";
|
||||
|
||||
#[must_use]
|
||||
|
|
@ -201,9 +94,9 @@ pub fn swarm_limits_path() -> PathBuf {
|
|||
}
|
||||
|
||||
/// Read the swarm-declared caps. An absent file is the empty map — no swarm
|
||||
/// opinion, so the hive's own chain decides. A file that exists but can't be
|
||||
/// read or parsed is an error, for the reason [`read`] gives: reading it as
|
||||
/// empty would silently drop a cap.
|
||||
/// opinion, so the hive-wide defaults decide. A file that exists but can't be
|
||||
/// read or parsed is an error: a cap can be tighter than the hive default, so
|
||||
/// reading it as empty would silently drop it.
|
||||
pub fn read_swarm() -> std::io::Result<BTreeMap<String, AgentLimits>> {
|
||||
super::read_map(&swarm_limits_path())
|
||||
}
|
||||
|
|
@ -216,55 +109,43 @@ pub fn write_swarm(map: &BTreeMap<String, AgentLimits>) -> std::io::Result<()> {
|
|||
super::write_map(&swarm_limits_path(), map)
|
||||
}
|
||||
|
||||
/// [`effective_from`] with the swarm's declaration on top.
|
||||
/// Resolve the effective values for an agent from the swarm caps in `swarm`:
|
||||
/// each field the swarm declared a cap for, and the hive-wide default for
|
||||
/// each it did not. This is the single place the fallback rule lives;
|
||||
/// `write_dropins` calls it and passes the result straight to systemd.
|
||||
///
|
||||
/// A swarm cap **replaces** the hive's answer for that field rather than
|
||||
/// joining the fallback chain as another tier: a tier a hive can out-set is
|
||||
/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's
|
||||
/// chain.
|
||||
///
|
||||
/// The chain itself is untouched — [`effective_from`] is called exactly as
|
||||
/// before and its result is what a missing swarm value falls back to.
|
||||
/// Use this form when resolving more than one agent in a row.
|
||||
#[must_use]
|
||||
pub fn effective_from_with_swarm(
|
||||
pub fn effective_from(
|
||||
swarm: &BTreeMap<String, AgentLimits>,
|
||||
limits: &BTreeMap<String, AgentLimits>,
|
||||
name: &str,
|
||||
hive_cpu_quota: &str,
|
||||
hive_memory_max: &str,
|
||||
) -> (String, String) {
|
||||
let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max);
|
||||
match swarm.get(name) {
|
||||
Some(s) => (
|
||||
s.cpu_quota.clone().unwrap_or(cpu),
|
||||
s.memory_max.clone().unwrap_or(mem),
|
||||
),
|
||||
None => (cpu, mem),
|
||||
}
|
||||
let caps = swarm.get(name);
|
||||
(
|
||||
caps.and_then(|c| c.cpu_quota.clone())
|
||||
.unwrap_or_else(|| hive_cpu_quota.to_owned()),
|
||||
caps.and_then(|c| c.memory_max.clone())
|
||||
.unwrap_or_else(|| hive_memory_max.to_owned()),
|
||||
)
|
||||
}
|
||||
|
||||
/// [`effective_from_with_swarm`] for a caller resolving a single agent, from
|
||||
/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`.
|
||||
///
|
||||
/// Delegates to [`effective`] rather than re-deriving the hive's answer, so
|
||||
/// there stays exactly one place the fallback chain lives. Errors as [`read`]
|
||||
/// does, for either file.
|
||||
pub fn effective_with_swarm(
|
||||
limits_path: &Path,
|
||||
/// [`effective_from`] for a caller resolving a single agent, from the swarm
|
||||
/// caps at `swarm_path`. Errors as [`read_swarm`] does.
|
||||
pub fn effective(
|
||||
swarm_path: &Path,
|
||||
name: &str,
|
||||
hive_cpu_quota: &str,
|
||||
hive_memory_max: &str,
|
||||
) -> std::io::Result<(String, String)> {
|
||||
let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?;
|
||||
let swarm: BTreeMap<String, AgentLimits> = super::read_map(swarm_path)?;
|
||||
Ok(match swarm.get(name) {
|
||||
Some(s) => (
|
||||
s.cpu_quota.clone().unwrap_or(cpu),
|
||||
s.memory_max.clone().unwrap_or(mem),
|
||||
),
|
||||
None => (cpu, mem),
|
||||
})
|
||||
Ok(effective_from(
|
||||
&swarm,
|
||||
name,
|
||||
hive_cpu_quota,
|
||||
hive_memory_max,
|
||||
))
|
||||
}
|
||||
|
||||
/// Validate a systemd `CPUQuota=` value. Percentages only, and values
|
||||
|
|
@ -386,96 +267,66 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn unset_agent_gets_hive_defaults() {
|
||||
let (cpu, mem) = resolve(&AgentLimits::default(), HIVE_CPU, HIVE_MEM);
|
||||
assert_eq!(cpu, "200%");
|
||||
assert_eq!(mem, "4G");
|
||||
fn an_agent_the_swarm_says_nothing_about_gets_the_hive_defaults() {
|
||||
let swarm = BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]);
|
||||
for map in [BTreeMap::new(), swarm] {
|
||||
assert_eq!(
|
||||
effective_from(&map, "sock", HIVE_CPU, HIVE_MEM),
|
||||
(HIVE_CPU.to_owned(), HIVE_MEM.to_owned())
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn both_fields_override() {
|
||||
let (cpu, mem) = resolve(&limits(Some("400%"), Some("8G")), HIVE_CPU, HIVE_MEM);
|
||||
assert_eq!(cpu, "400%");
|
||||
assert_eq!(mem, "8G");
|
||||
fn a_swarm_cap_replaces_the_hive_default() {
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
|
||||
assert_eq!(
|
||||
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
|
||||
("150%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
/// The point of per-field fallback: overriding memory must not drag
|
||||
/// CPU along with it.
|
||||
/// The point of per-field fallback: capping memory must not drag CPU
|
||||
/// along with it.
|
||||
#[test]
|
||||
fn partial_override_keeps_other_field_on_hive_default() {
|
||||
let (cpu, mem) = resolve(&limits(None, Some("8G")), HIVE_CPU, HIVE_MEM);
|
||||
assert_eq!(cpu, "200%", "cpu should still track the hive default");
|
||||
assert_eq!(mem, "8G");
|
||||
|
||||
let (cpu, mem) = resolve(&limits(Some("400%"), None), HIVE_CPU, HIVE_MEM);
|
||||
assert_eq!(cpu, "400%");
|
||||
assert_eq!(mem, "4G", "memory should still track the hive default");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_entry_is_reported_empty() {
|
||||
assert!(AgentLimits::default().is_empty());
|
||||
assert!(!limits(None, Some("8G")).is_empty());
|
||||
assert!(!limits(Some("400%"), None).is_empty());
|
||||
fn a_swarm_cap_is_per_field() {
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
|
||||
assert_eq!(
|
||||
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
|
||||
(HIVE_CPU.to_owned(), "2G".to_owned()),
|
||||
"cpu should still track the hive default"
|
||||
);
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), None))]);
|
||||
assert_eq!(
|
||||
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
|
||||
("150%".to_owned(), HIVE_MEM.to_owned()),
|
||||
"memory should still track the hive default"
|
||||
);
|
||||
}
|
||||
|
||||
const TRUNCATED: &str =
|
||||
"{\n \"sock\": { \"cpu_quota\": \"50%\", \"memory_max\": \"1G\" },\n \"iris\": { \"mem";
|
||||
|
||||
fn corrupt_file() -> (tempfile::TempDir, PathBuf) {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join(RESOURCE_LIMITS_FILE);
|
||||
std::fs::write(&path, TRUNCATED).expect("seed");
|
||||
(dir, path)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn effective_of_a_corrupt_file_is_an_error() {
|
||||
let (_dir, path) = corrupt_file();
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join(SWARM_LIMITS_FILE);
|
||||
std::fs::write(&path, TRUNCATED).expect("seed");
|
||||
let err = effective(&path, "sock", HIVE_CPU, HIVE_MEM)
|
||||
.expect_err("a corrupt file must not resolve to hive defaults");
|
||||
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn set_limits_leaves_a_corrupt_file_untouched() {
|
||||
let (_dir, path) = corrupt_file();
|
||||
let err = set_limits_at(&path, "ruth", &limits(Some("400%"), None))
|
||||
.expect_err("a corrupt file must not be overwritten");
|
||||
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
||||
assert_eq!(std::fs::read(&path).expect("read"), TRUNCATED.as_bytes());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_file_resolves_to_hive_defaults() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join(RESOURCE_LIMITS_FILE);
|
||||
let path = dir.path().join(SWARM_LIMITS_FILE);
|
||||
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("missing file");
|
||||
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, HIVE_MEM));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn set_limits_keeps_other_agents_and_leaves_no_temp_file() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join(RESOURCE_LIMITS_FILE);
|
||||
set_limits_at(&path, "sock", &limits(None, Some("8G"))).expect("set on missing file");
|
||||
set_limits_at(&path, "iris", &limits(Some("50%"), None)).expect("set");
|
||||
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("read");
|
||||
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, "8G"));
|
||||
let (cpu, _) = effective(&path, "iris", HIVE_CPU, HIVE_MEM).expect("read");
|
||||
assert_eq!(cpu, "50%");
|
||||
let entries: Vec<_> = std::fs::read_dir(dir.path())
|
||||
.expect("read_dir")
|
||||
.map(|e| e.expect("entry").file_name())
|
||||
.collect();
|
||||
assert_eq!(
|
||||
entries,
|
||||
vec![std::ffi::OsString::from(RESOURCE_LIMITS_FILE)]
|
||||
);
|
||||
}
|
||||
|
||||
/// Absent fields must deserialize to `None`, not fail — an entry
|
||||
/// written by an older version with only one field must still load.
|
||||
/// with only one field must still load.
|
||||
#[test]
|
||||
fn partial_entry_deserializes() {
|
||||
let map: BTreeMap<String, AgentLimits> =
|
||||
|
|
@ -605,52 +456,4 @@ mod tests {
|
|||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() {
|
||||
// The whole point of the ruling: a tier a hive can out-set is not a
|
||||
// cap. The agent override below is the strongest hive-side value
|
||||
// there is, and the swarm still wins.
|
||||
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]);
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
|
||||
assert_eq!(
|
||||
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
("150%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swarm_cap_is_per_field_like_every_other_tier() {
|
||||
// A swarm that caps memory and says nothing about CPU leaves CPU on
|
||||
// the hive's chain — here the per-agent override, not the hive default.
|
||||
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]);
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
|
||||
assert_eq!(
|
||||
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
("800%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() {
|
||||
// The control that makes the two tests above mean something: the
|
||||
// wrapper must be a no-op when the swarm has said nothing, for every
|
||||
// shape the old chain distinguishes. If this drifts, the swarm tier
|
||||
// changed behaviour it was explicitly told not to change.
|
||||
let none = BTreeMap::new();
|
||||
let cases = [
|
||||
BTreeMap::new(),
|
||||
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]),
|
||||
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]),
|
||||
BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]),
|
||||
BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]),
|
||||
];
|
||||
for hive_side in &cases {
|
||||
assert_eq!(
|
||||
effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
"the wrapper changed the answer with no swarm value present"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -76,16 +76,6 @@ pub struct ContainerView {
|
|||
/// a legitimate way to keep it idle when it next boots.
|
||||
#[serde(default)]
|
||||
pub paused: bool,
|
||||
/// Effective systemd `CPUQuota=` for this container (e.g. `"400%"`) —
|
||||
/// the per-agent override from `meta/resource-limits.json` when set,
|
||||
/// otherwise the hive-wide `agentCpuQuota`. Always populated: there
|
||||
/// is no "unset" state to render, only "same as everyone else".
|
||||
/// Reflects what the *drop-in says*, which is what the next start
|
||||
/// will enforce — not a live cgroup reading.
|
||||
pub cpu_quota: String,
|
||||
/// Effective systemd `MemoryMax=` for this container (e.g. `"8G"`).
|
||||
/// Same resolution + caveat as [`ContainerView::cpu_quota`].
|
||||
pub memory_max: String,
|
||||
}
|
||||
|
||||
impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
|
||||
|
|
@ -99,10 +89,9 @@ impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
|
|||
/// anymore. Stubbed to `0` rather than dropping the wire field
|
||||
/// outright — leaves `hivectl status`/the dashboard column intact
|
||||
/// syntactically, just always empty, until iris's frontend follow-up
|
||||
/// decides whether to drop the column entirely. `port`, `cpu_quota`,
|
||||
/// `memory_max` have no equivalent on the row at all and are simply
|
||||
/// not carried over. `url` has no source on `ContainerView` at all —
|
||||
/// see [`agent_url`].
|
||||
/// decides whether to drop the column entirely. `port` has no
|
||||
/// equivalent on the row at all and is simply not carried over. `url`
|
||||
/// has no source on `ContainerView` at all — see [`agent_url`].
|
||||
fn from(v: ContainerView) -> Self {
|
||||
Self {
|
||||
url: agent_url(&v.name),
|
||||
|
|
@ -148,37 +137,12 @@ fn agent_url_for_domain(domain: &str, name: &str) -> String {
|
|||
/// Build the full container list. Wraps `lifecycle::list()` and
|
||||
/// resolves every per-agent attribute the dashboard surfaces.
|
||||
///
|
||||
/// Takes `hive` because the effective resource limits are a per-field
|
||||
/// fallback onto the hive-wide `agent_cpu_quota` / `agent_memory_max`,
|
||||
/// and those live on [`crate::coordinator::HiveEnv`], not on disk. Both callers already
|
||||
/// hold a `Coordinator`, so this is a parameter rather than a global.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// The container list could not be read.
|
||||
pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec<ContainerView>> {
|
||||
pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
|
||||
let raw = lifecycle::list().await?;
|
||||
let locked = read_meta_locked_revs();
|
||||
// Read once per scan rather than re-read for every container on every
|
||||
// SSE scan. An unreadable file shows every row at the hive defaults;
|
||||
// the error log is the only signal, as `ContainerView` has no field
|
||||
// to carry it.
|
||||
let limits = crate::resource_limits::read().unwrap_or_else(|e| {
|
||||
tracing::error!(
|
||||
path = %crate::resource_limits::resource_limits_path().display(),
|
||||
error = ?e,
|
||||
"resource limits unreadable — container rows show hive defaults"
|
||||
);
|
||||
std::collections::BTreeMap::new()
|
||||
});
|
||||
let swarm_limits = crate::resource_limits::read_swarm().unwrap_or_else(|e| {
|
||||
tracing::error!(
|
||||
path = %crate::resource_limits::swarm_limits_path().display(),
|
||||
error = ?e,
|
||||
"swarm resource caps unreadable — container rows ignore them"
|
||||
);
|
||||
std::collections::BTreeMap::new()
|
||||
});
|
||||
let mut out = Vec::new();
|
||||
for c in &raw {
|
||||
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
|
||||
|
|
@ -243,13 +207,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
|
|||
(None, None)
|
||||
};
|
||||
let paused = Coordinator::is_paused(&logical);
|
||||
let (cpu_quota, memory_max) = crate::resource_limits::effective_from_with_swarm(
|
||||
&swarm_limits,
|
||||
&limits,
|
||||
logical.as_str(),
|
||||
&hive.agent_cpu_quota,
|
||||
&hive.agent_memory_max,
|
||||
);
|
||||
out.push(ContainerView {
|
||||
port: lifecycle::agent_web_port(logical.as_str()),
|
||||
running,
|
||||
|
|
@ -263,8 +220,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
|
|||
status_text,
|
||||
status_set_at,
|
||||
paused,
|
||||
cpu_quota,
|
||||
memory_max,
|
||||
});
|
||||
}
|
||||
Ok(out)
|
||||
|
|
@ -570,9 +525,8 @@ mod tests {
|
|||
|
||||
/// The `From` impl both `handle_agent_status`'s poll path and
|
||||
/// `stream_agent_status`'s push path go through — pins the field
|
||||
/// mapping so the two can't quietly diverge, and that `port` /
|
||||
/// `cpu_quota` / `memory_max` (no equivalent on the row) are dropped
|
||||
/// on purpose rather than by omission.
|
||||
/// mapping so the two can't quietly diverge, and that `port` (no
|
||||
/// equivalent on the row) is dropped on purpose rather than by omission.
|
||||
#[test]
|
||||
fn agent_status_row_carries_every_field_the_row_has_and_drops_the_rest() {
|
||||
use super::ContainerView;
|
||||
|
|
@ -592,8 +546,6 @@ mod tests {
|
|||
status_text: Some("shipping".to_owned()),
|
||||
status_set_at: Some(Utc.timestamp_opt(1_700_000_000, 0).unwrap()),
|
||||
paused: true,
|
||||
cpu_quota: "400%".to_owned(),
|
||||
memory_max: "8G".to_owned(),
|
||||
};
|
||||
|
||||
let row = AgentStatusRow::from(view);
|
||||
|
|
|
|||
|
|
@ -720,7 +720,7 @@ impl Coordinator {
|
|||
pub async fn rescan_containers_and_emit(self: &Arc<Self>) {
|
||||
// An unreadable list says nothing about which containers exist, so
|
||||
// the last snapshot stands rather than every row being reported gone.
|
||||
let fresh = match container_view::build_all(&self.hive_env()).await {
|
||||
let fresh = match container_view::build_all().await {
|
||||
Ok(fresh) => fresh,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "container rescan: list failed; keeping last snapshot");
|
||||
|
|
|
|||
|
|
@ -311,95 +311,6 @@ fn set_paused_error_response(
|
|||
}
|
||||
}
|
||||
|
||||
/// Form fields for `post_resource_limits`. Both fields are optional strings;
|
||||
/// an empty value clears the per-agent override for that field, falling back
|
||||
/// to the hive-wide default.
|
||||
#[derive(Deserialize, Default, ToSchema)]
|
||||
pub(super) struct ResourceLimitsForm {
|
||||
#[serde(default)]
|
||||
cpu_quota: String,
|
||||
#[serde(default)]
|
||||
memory_max: String,
|
||||
}
|
||||
|
||||
/// Write per-agent CPU/memory limit
|
||||
/// overrides for `name`.
|
||||
///
|
||||
/// An empty `cpu_quota` or `memory_max` field clears that field's override,
|
||||
/// falling back to the hive-wide default. Both empty together removes the
|
||||
/// agent's entry entirely. The new drop-in is written immediately — the
|
||||
/// limits take effect on the next container start or restart. Triggers an
|
||||
/// immediate rescan so `ContainerView.cpu_quota`/`memory_max` update on
|
||||
/// the dashboard via SSE without waiting for the next periodic sweep.
|
||||
#[utoipa::path(
|
||||
post,
|
||||
path = "/api/resource-limits/{name}",
|
||||
params(("name" = String, Path, description = "agent name")),
|
||||
request_body(content = ResourceLimitsForm, content_type = "application/x-www-form-urlencoded"),
|
||||
responses(
|
||||
(status = 200, description = "limits written", body = String),
|
||||
(status = 400, description = "bad agent name"),
|
||||
(status = 404, description = "no such agent"),
|
||||
(status = 422, description = "invalid cpu_quota/memory_max value"),
|
||||
(status = 500, description = "commit or drop-in write failed"),
|
||||
),
|
||||
tag = "lifecycle_ops"
|
||||
)]
|
||||
pub(super) async fn post_resource_limits(
|
||||
State(state): State<AppState>,
|
||||
AxumPath(name): AxumPath<String>,
|
||||
Form(form): Form<ResourceLimitsForm>,
|
||||
) -> Response {
|
||||
let logical = strip_container_prefix(&name);
|
||||
if let Some(reject) = guard_agent_name(&state, &logical).await {
|
||||
return reject;
|
||||
}
|
||||
let ident = match Ident::parse(&logical) {
|
||||
Ok(i) => i,
|
||||
Err(e) => return (StatusCode::BAD_REQUEST, format!("bad agent name: {e}")).into_response(),
|
||||
};
|
||||
let cpu_quota = if form.cpu_quota.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(form.cpu_quota.as_str())
|
||||
};
|
||||
let memory_max = if form.memory_max.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(form.memory_max.as_str())
|
||||
};
|
||||
if let Some(v) = cpu_quota
|
||||
&& let Err(e) = crate::resource_limits::validate_cpu_quota(v)
|
||||
{
|
||||
return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response();
|
||||
}
|
||||
// Returns the value to store, not just an ok/err verdict — see
|
||||
// `validate_memory_max`'s own doc for why that isn't always `v`.
|
||||
let memory_max = match memory_max.map(crate::resource_limits::validate_memory_max) {
|
||||
Some(Err(e)) => return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response(),
|
||||
Some(Ok(v)) => Some(v),
|
||||
None => None,
|
||||
};
|
||||
let limits = crate::resource_limits::AgentLimits {
|
||||
cpu_quota: cpu_quota.map(str::to_owned),
|
||||
memory_max,
|
||||
};
|
||||
if let Err(e) = crate::meta::commit_resource_limits(ident.as_str(), &limits).await {
|
||||
return error_response(&format!("set limits {logical}: {e:#}"));
|
||||
}
|
||||
let agent_dir = crate::paths::agent_runtime_dir(ident.as_str());
|
||||
let hive = state.coord.hive_env();
|
||||
let paths = match crate::coordinator::Coordinator::agent_paths(ident.as_str(), agent_dir) {
|
||||
Ok(p) => p,
|
||||
Err(e) => return error_response(&format!("write_dropins {logical}: {e:#}")),
|
||||
};
|
||||
if let Err(e) = crate::lifecycle::write_dropins(ident.as_str(), &hive, &paths).await {
|
||||
return error_response(&format!("write_dropins {logical}: {e:#}"));
|
||||
}
|
||||
state.coord.rescan_containers_and_emit().await;
|
||||
(StatusCode::OK, "ok").into_response()
|
||||
}
|
||||
|
||||
/// Queue a rebuild DAG for every live agent
|
||||
/// container.
|
||||
#[utoipa::path(
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
//! Remaining single-endpoint dashboard handlers: mark-all-read, operator
|
||||
//! compose (`op-send`), spawn-request, hive-wide turn stats, and container
|
||||
//! resources.
|
||||
//! compose (`op-send`), spawn-request and hive-wide turn stats.
|
||||
|
||||
use axum::{
|
||||
extract::{Form, Path as AxumPath, State},
|
||||
|
|
@ -11,7 +10,6 @@ use serde::{Deserialize, Serialize};
|
|||
use utoipa::{IntoParams, ToSchema};
|
||||
|
||||
use super::{AppState, Ident, error_response};
|
||||
use crate::container_stats::ContainerResource;
|
||||
use crate::hive_stats::HiveStats;
|
||||
|
||||
#[derive(Deserialize, IntoParams)]
|
||||
|
|
@ -43,20 +41,6 @@ pub(super) async fn api_stats_hive(
|
|||
.into_response()
|
||||
}
|
||||
|
||||
/// Live per-agent-container CPU + memory load from cgroup v2.
|
||||
///
|
||||
/// Samples CPU over a short interval (~200 ms), so this call briefly
|
||||
/// awaits.
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/container-resources",
|
||||
responses((status = 200, description = "live per-container CPU + memory load", body = Vec<ContainerResource>)),
|
||||
tag = "misc_api"
|
||||
)]
|
||||
pub(super) async fn api_container_resources() -> Response {
|
||||
axum::Json(crate::container_stats::gather().await).into_response()
|
||||
}
|
||||
|
||||
#[derive(Serialize, ToSchema)]
|
||||
pub(super) struct MarkAllReadBody {
|
||||
marked: u64,
|
||||
|
|
|
|||
|
|
@ -116,7 +116,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
|
|||
.routes(routes!(state_snapshot::api_state))
|
||||
.routes(routes!(state_files::get_state_file))
|
||||
.routes(routes!(misc_api::api_stats_hive))
|
||||
.routes(routes!(misc_api::api_container_resources))
|
||||
.routes(routes!(misc_api::post_mark_all_read))
|
||||
.routes(routes!(misc_api::post_op_send))
|
||||
.routes(routes!(build_logs::get_build_logs_all))
|
||||
|
|
@ -140,7 +139,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
|
|||
.routes(routes!(lifecycle_ops::post_rebuild))
|
||||
.routes(routes!(lifecycle_ops::post_pause))
|
||||
.routes(routes!(lifecycle_ops::post_resume))
|
||||
.routes(routes!(lifecycle_ops::post_resource_limits))
|
||||
.routes(routes!(lifecycle_ops::post_update_all))
|
||||
.routes(routes!(tombstones::post_purge_tombstone))
|
||||
.routes(routes!(meta_inputs::post_meta_update))
|
||||
|
|
@ -372,7 +370,6 @@ mod router_build_probe {
|
|||
.routes(routes!(state_snapshot::api_state))
|
||||
.routes(routes!(state_files::get_state_file))
|
||||
.routes(routes!(misc_api::api_stats_hive))
|
||||
.routes(routes!(misc_api::api_container_resources))
|
||||
.routes(routes!(misc_api::post_mark_all_read))
|
||||
.routes(routes!(misc_api::post_op_send))
|
||||
.routes(routes!(build_logs::get_build_logs_all))
|
||||
|
|
@ -396,7 +393,6 @@ mod router_build_probe {
|
|||
.routes(routes!(lifecycle_ops::post_rebuild))
|
||||
.routes(routes!(lifecycle_ops::post_pause))
|
||||
.routes(routes!(lifecycle_ops::post_resume))
|
||||
.routes(routes!(lifecycle_ops::post_resource_limits))
|
||||
.routes(routes!(lifecycle_ops::post_update_all))
|
||||
.routes(routes!(tombstones::post_purge_tombstone))
|
||||
.routes(routes!(meta_inputs::post_meta_update))
|
||||
|
|
|
|||
|
|
@ -603,8 +603,6 @@ mod tests {
|
|||
status_text: None,
|
||||
status_set_at: None,
|
||||
paused: false,
|
||||
cpu_quota: "200%".to_owned(),
|
||||
memory_max: "4G".to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,6 @@ pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Re
|
|||
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
|
||||
if let Some((cpu_quota, memory_max)) = stage_limits(
|
||||
name,
|
||||
&crate::resource_limits::resource_limits_path(),
|
||||
&crate::resource_limits::swarm_limits_path(),
|
||||
&limits_dropin_path(&container),
|
||||
&paths.agent,
|
||||
|
|
@ -97,14 +96,13 @@ impl HostMemory {
|
|||
/// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's.
|
||||
fn stage_limits(
|
||||
agent_name: &str,
|
||||
limits_path: &Path,
|
||||
swarm_path: &Path,
|
||||
dropin_path: &Path,
|
||||
runtime_dir: &Path,
|
||||
hive: &HiveEnv,
|
||||
host: Option<HostMemory>,
|
||||
) -> Result<Option<(String, String)>> {
|
||||
let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive);
|
||||
let limits = limits_to_apply(agent_name, swarm_path, dropin_path, hive);
|
||||
let memory_max = match &limits {
|
||||
Some((_, memory_max)) => Some(memory_max.clone()),
|
||||
None => dropin_memory_max(agent_name, dropin_path),
|
||||
|
|
@ -226,35 +224,27 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
|
|||
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
|
||||
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
|
||||
///
|
||||
/// When the overrides file at `limits_path` or the swarm caps at
|
||||
/// `swarm_path` can't be read, the error is logged and the agent still starts.
|
||||
/// The last-applied drop-in is kept, because the hive defaults can be looser
|
||||
/// than the agent's own override.
|
||||
/// When the swarm caps at `swarm_path` can't be read, the error is logged and
|
||||
/// the agent still starts. The last-applied drop-in is kept, because the hive
|
||||
/// defaults can be looser than the swarm's cap.
|
||||
/// With no drop-in yet, the agent gets the hive defaults: without one it
|
||||
/// would run uncapped.
|
||||
fn limits_to_apply(
|
||||
agent_name: &str,
|
||||
limits_path: &Path,
|
||||
swarm_path: &Path,
|
||||
dropin_path: &Path,
|
||||
hive: &HiveEnv,
|
||||
) -> Option<(String, String)> {
|
||||
let hive_cpu = &hive.agent_cpu_quota;
|
||||
let hive_mem = &hive.agent_memory_max;
|
||||
let e = match crate::resource_limits::effective_with_swarm(
|
||||
limits_path,
|
||||
swarm_path,
|
||||
agent_name,
|
||||
hive_cpu,
|
||||
hive_mem,
|
||||
) {
|
||||
let e = match crate::resource_limits::effective(swarm_path, agent_name, hive_cpu, hive_mem) {
|
||||
Ok(limits) => return Some(limits),
|
||||
Err(e) => e,
|
||||
};
|
||||
if dropin_path.exists() {
|
||||
tracing::error!(
|
||||
agent = %agent_name,
|
||||
path = %limits_path.display(),
|
||||
path = %swarm_path.display(),
|
||||
error = ?e,
|
||||
"resource limits unreadable — keeping the last-applied limits drop-in"
|
||||
);
|
||||
|
|
@ -262,7 +252,7 @@ fn limits_to_apply(
|
|||
} else {
|
||||
tracing::error!(
|
||||
agent = %agent_name,
|
||||
path = %limits_path.display(),
|
||||
path = %swarm_path.display(),
|
||||
error = ?e,
|
||||
"resource limits unreadable and no limits drop-in yet — applying hive defaults"
|
||||
);
|
||||
|
|
@ -271,14 +261,14 @@ fn limits_to_apply(
|
|||
}
|
||||
|
||||
/// Write a systemd drop-in for `container@<container>.service` that applies
|
||||
/// the agent's effective resource caps — its per-agent overrides from
|
||||
/// `meta/resource-limits.json` where set, the hive-wide defaults
|
||||
/// the agent's effective resource caps — the swarm's cap from
|
||||
/// `meta/swarm-resource-limits.json` where set, the hive-wide defaults
|
||||
/// otherwise. Goes under `/run/systemd/system/...` so it's ephemeral
|
||||
/// (regenerated on every spawn / rebuild, except as [`limits_to_apply`] says).
|
||||
///
|
||||
/// The weights are hive-wide (`services.hyperhive.agentCpuWeight` /
|
||||
/// `agentIoWeight`) — unlike the caps they have no per-agent override in
|
||||
/// `meta/resource-limits.json`, so they come straight off [`HiveEnv`].
|
||||
/// `agentIoWeight`) — unlike the caps they have no swarm cap, so they come
|
||||
/// straight off [`HiveEnv`].
|
||||
/// `None` (the nix option set to `null`) means the weight line is left out
|
||||
/// and the container keeps the kernel default.
|
||||
async fn set_resource_limits(
|
||||
|
|
@ -848,42 +838,35 @@ mod tests {
|
|||
|
||||
const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem";
|
||||
|
||||
fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf {
|
||||
dir.path().join("absent-swarm-resource-limits.json")
|
||||
fn swarm_file(dir: &tempfile::TempDir) -> std::path::PathBuf {
|
||||
dir.path().join("swarm-resource-limits.json")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_limits_keep_an_existing_dropin() {
|
||||
fn corrupt_swarm_caps_keep_an_existing_dropin() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = swarm_file(&dir);
|
||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
||||
assert_eq!(
|
||||
limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&absent_swarm(&dir),
|
||||
&dropin,
|
||||
&HiveEnv::default()
|
||||
),
|
||||
limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default()),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(&limits).expect("read"),
|
||||
std::fs::read(&swarm).expect("read"),
|
||||
TRUNCATED_LIMITS.as_bytes()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_limits_without_a_dropin_apply_hive_defaults() {
|
||||
fn corrupt_swarm_caps_without_a_dropin_apply_hive_defaults() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
||||
let swarm = swarm_file(&dir);
|
||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&absent_swarm(&dir),
|
||||
&swarm,
|
||||
&dir.path().join("absent.conf"),
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
|
|
@ -891,32 +874,12 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_swarm_caps_keep_an_existing_dropin() {
|
||||
fn a_swarm_cap_is_applied_over_the_hive_default() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let swarm = dir.path().join("swarm-resource-limits.json");
|
||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&dir.path().join("resource-limits.json"),
|
||||
&swarm,
|
||||
&dropin,
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
assert_eq!(applied, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swarm_cap_is_applied_over_the_operator_override() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = dir.path().join("swarm-resource-limits.json");
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
|
||||
let swarm = swarm_file(&dir);
|
||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&swarm,
|
||||
&dir.path().join("absent.conf"),
|
||||
&HiveEnv::default(),
|
||||
|
|
@ -925,19 +888,13 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn readable_limits_are_applied_over_an_existing_dropin() {
|
||||
fn readable_swarm_caps_are_applied_over_an_existing_dropin() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = swarm_file(&dir);
|
||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
|
||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps");
|
||||
std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&absent_swarm(&dir),
|
||||
&dropin,
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
let applied = limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default());
|
||||
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
|
||||
}
|
||||
|
||||
|
|
@ -953,8 +910,7 @@ mod tests {
|
|||
let runtime = dir.path().join("runtime");
|
||||
let applied = stage_limits(
|
||||
"sock",
|
||||
&dir.path().join("resource-limits.json"),
|
||||
&dir.path().join("swarm-resource-limits.json"),
|
||||
&swarm_file(dir),
|
||||
&dir.path().join("hyperhive-limits.conf"),
|
||||
&runtime,
|
||||
&HiveEnv::default(),
|
||||
|
|
@ -965,40 +921,18 @@ mod tests {
|
|||
(applied.map(|(_, mem)| mem), copy)
|
||||
}
|
||||
|
||||
/// The agent's copy follows the swarm's cap, the one systemd enforces, over
|
||||
/// the operator's override: a heap sized against the override would
|
||||
/// outgrow the cgroup.
|
||||
#[test]
|
||||
fn the_agent_reads_the_swarm_cap_over_the_operator_override() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
std::fs::write(
|
||||
dir.path().join("resource-limits.json"),
|
||||
r#"{"sock": {"memory_max": "16G"}}"#,
|
||||
)
|
||||
.expect("seed limits");
|
||||
std::fs::write(
|
||||
dir.path().join("swarm-resource-limits.json"),
|
||||
r#"{"sock": {"memory_max": "2G"}}"#,
|
||||
)
|
||||
.expect("seed swarm caps");
|
||||
assert_eq!(
|
||||
stage(&dir),
|
||||
(Some("2G".to_owned()), Some("2147483648\n".to_owned()))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_agent_reads_the_cap_its_dropin_carries() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = swarm_file(&dir);
|
||||
|
||||
// No overrides anywhere: the hive-wide default, 4G.
|
||||
// No swarm cap anywhere: the hive-wide default, 4G.
|
||||
assert_eq!(
|
||||
stage(&dir),
|
||||
(Some("4G".to_owned()), Some("4294967296\n".to_owned()))
|
||||
);
|
||||
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
|
||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed swarm caps");
|
||||
assert_eq!(
|
||||
stage(&dir),
|
||||
(Some("16G".to_owned()), Some("17179869184\n".to_owned()))
|
||||
|
|
@ -1008,17 +942,17 @@ mod tests {
|
|||
#[test]
|
||||
fn every_kind_of_cap_is_written_for_the_agent() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = swarm_file(&dir);
|
||||
for (memory_max, copy) in [
|
||||
("infinity", "infinity\n"),
|
||||
("50%", "8192004096\n"),
|
||||
("8G", "8589934592\n"),
|
||||
] {
|
||||
std::fs::write(
|
||||
&limits,
|
||||
&swarm,
|
||||
format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#),
|
||||
)
|
||||
.expect("seed limits");
|
||||
.expect("seed swarm caps");
|
||||
assert_eq!(
|
||||
stage(&dir),
|
||||
(Some(memory_max.to_owned()), Some(copy.to_owned())),
|
||||
|
|
@ -1072,14 +1006,14 @@ mod tests {
|
|||
#[test]
|
||||
fn a_kept_dropin_hands_the_agent_its_cap() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = swarm_file(&dir);
|
||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE);
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
|
||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps");
|
||||
assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned()));
|
||||
|
||||
// Unreadable limits keep the drop-in; the copy follows the drop-in.
|
||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
||||
// Unreadable caps keep the drop-in; the copy follows the drop-in.
|
||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||
std::fs::write(
|
||||
&dropin,
|
||||
"[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n",
|
||||
|
|
@ -1103,29 +1037,21 @@ mod tests {
|
|||
#[test]
|
||||
fn an_unknown_cap_leaves_no_copy() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = swarm_file(&dir);
|
||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned()));
|
||||
|
||||
// Unreadable limits and an unreadable drop-in: nothing says what the
|
||||
// Unreadable caps and an unreadable drop-in: nothing says what the
|
||||
// cgroup enforces.
|
||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||
std::fs::create_dir(&dropin).expect("seed unreadable drop-in");
|
||||
assert_eq!(stage(&dir), (None, None));
|
||||
|
||||
// A percentage with no host memory to resolve it against.
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed limits");
|
||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed swarm caps");
|
||||
let runtime = dir.path().join("runtime");
|
||||
let applied = stage_limits(
|
||||
"sock",
|
||||
&limits,
|
||||
&dir.path().join("swarm-resource-limits.json"),
|
||||
&dropin,
|
||||
&runtime,
|
||||
&HiveEnv::default(),
|
||||
None,
|
||||
)
|
||||
.expect("stage limits");
|
||||
let applied = stage_limits("sock", &swarm, &dropin, &runtime, &HiveEnv::default(), None)
|
||||
.expect("stage limits");
|
||||
assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned()));
|
||||
assert!(!runtime.join(MEMORY_CAP_FILE).exists());
|
||||
}
|
||||
|
|
|
|||
|
|
@ -404,9 +404,9 @@ async fn cmd_serve(
|
|||
// ~hundreds of KB, and the /stats + hive-stats views read it
|
||||
// directly; pruning it would just lose trend history for no gain.)
|
||||
// Slow per-container disk sampler: hive-priv measures each agent's state/
|
||||
// + work/ + writable rootfs every 30 min, cached so the 5s container-load
|
||||
// poll stays cheap cgroup-only reads. Feeds `disk_bytes` on the LOAD
|
||||
// tab. See container_stats::disk_sampler_loop.
|
||||
// + work/ + writable rootfs every 30 min, cached so a sample only reads
|
||||
// cgroup files. Feeds `disk_bytes` to the OTEL exporter. See
|
||||
// container_stats::disk_sampler_loop.
|
||||
crate::container_stats::spawn_disk_sampler();
|
||||
// Per-agent container-resource OTEL export: rides the same
|
||||
// cgroup gauges out to the configured OTLP endpoint, reusing the hive
|
||||
|
|
|
|||
|
|
@ -47,8 +47,8 @@ pub struct AgentSpec {
|
|||
|
||||
/// Stage every generated meta JSON file that exists: topology.json is
|
||||
/// regenerated by `reconcile` whenever the agent set changed;
|
||||
/// tool-groups/capabilities/resource-limits are created lazily on
|
||||
/// first write (`set_groups`/`set_caps`/`set_limits`) — absent means
|
||||
/// tool-groups/capabilities are created lazily on
|
||||
/// first write (`set_groups`/`set_caps`) — absent means
|
||||
/// every agent is on defaults, no file needed. Without staging, an
|
||||
/// existing-but-untracked file shows up as untracked in the meta repo,
|
||||
/// which can confuse nix's dirty-tree fetch. `git add` is a no-op when
|
||||
|
|
@ -61,10 +61,6 @@ async fn stage_generated_meta_files(dir: &std::path::Path) -> Result<()> {
|
|||
crate::capabilities::capabilities_path(),
|
||||
"capabilities.json",
|
||||
),
|
||||
(
|
||||
crate::resource_limits::resource_limits_path(),
|
||||
"resource-limits.json",
|
||||
),
|
||||
] {
|
||||
if path.exists() {
|
||||
git(dir, &["add", name]).await?;
|
||||
|
|
@ -202,7 +198,6 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> {
|
|||
f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"),
|
||||
"topology.json" => Some("topology"),
|
||||
"capabilities.json" => Some("capabilities"),
|
||||
"resource-limits.json" => Some("resource-limits"),
|
||||
"tool-groups.json" => Some("tool-groups"),
|
||||
_ => None,
|
||||
})
|
||||
|
|
@ -352,41 +347,6 @@ pub async fn commit_capabilities(agent: &str, caps: &[String]) -> Result<()> {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Write the resource-limits file for `agent` and commit it atomically
|
||||
/// under `META_LOCK`. Same rationale as `commit_tool_groups`: the
|
||||
/// working tree must never be left dirty for the next `sync_agents` to trip
|
||||
/// over.
|
||||
///
|
||||
/// Unlike the perm files this one is never injected into the container —
|
||||
/// it's a host-side cap *on* the agent — but it lives in the same repo
|
||||
/// so a limit change gets the same auditable one-commit-per-change trail.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns an error if writing the JSON file fails or a git stage/commit
|
||||
/// step fails.
|
||||
pub async fn commit_resource_limits(
|
||||
agent: &str,
|
||||
limits: &crate::resource_limits::AgentLimits,
|
||||
) -> Result<()> {
|
||||
let _guard = META_LOCK.lock().await;
|
||||
crate::resource_limits::set_limits(agent, limits)
|
||||
.map_err(|e| anyhow::anyhow!("set resource limits for {agent}: {e}"))?;
|
||||
let dir = crate::paths::meta_root();
|
||||
if crate::resource_limits::resource_limits_path().exists() {
|
||||
git(&dir, &["add", "resource-limits.json"]).await?;
|
||||
}
|
||||
if paths_dirty(&dir, &["resource-limits.json"]).await? {
|
||||
git_commit_paths(
|
||||
&dir,
|
||||
&format!("set resource limits for {agent}"),
|
||||
&["resource-limits.json"],
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Write both perm files for `agent` (whichever are `Some`) and commit
|
||||
/// them in a SINGLE git commit under `META_LOCK` — the batch
|
||||
/// `POST /api/permissions` path. A `None` field leaves that file
|
||||
|
|
|
|||
|
|
@ -170,7 +170,7 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
|
|||
// question is only answerable on this side of the socket —
|
||||
// see the request's doc comment for why the client asks.
|
||||
HostRequest::AgentExists { name } => HostResponse::agent_exists(agent_exists(name)?),
|
||||
HostRequest::AgentStatus => handle_agent_status(&coord).await,
|
||||
HostRequest::AgentStatus => handle_agent_status().await,
|
||||
// Intercepted in `handle()` before a request ever reaches
|
||||
// `dispatch` — see `stream_agent_status`'s doc comment. This
|
||||
// arm exists only so the match stays exhaustive; reaching it
|
||||
|
|
@ -184,19 +184,6 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
|
|||
// federation peer-config block) and open the web surfaces
|
||||
// (`hivectl open`).
|
||||
HostRequest::Urls => HostResponse::urls(hive_urls()),
|
||||
HostRequest::SetResourceLimits {
|
||||
name,
|
||||
cpu_quota,
|
||||
memory_max,
|
||||
} => {
|
||||
handle_set_resource_limits(
|
||||
&coord,
|
||||
name,
|
||||
cpu_quota.as_deref(),
|
||||
memory_max.as_deref(),
|
||||
)
|
||||
.await?
|
||||
}
|
||||
HostRequest::MatrixInvite { user, room } => {
|
||||
handle_matrix_invite(user, room.as_deref()).await?
|
||||
}
|
||||
|
|
@ -293,8 +280,8 @@ async fn handle_set_paused(
|
|||
}
|
||||
|
||||
/// Collect per-agent status rows for `hivectl status` and the dashboard.
|
||||
async fn handle_agent_status(coord: &Arc<Coordinator>) -> HostResponse {
|
||||
match crate::container_view::build_all(&coord.hive_env()).await {
|
||||
async fn handle_agent_status() -> HostResponse {
|
||||
match crate::container_view::build_all().await {
|
||||
Ok(views) => HostResponse::agent_statuses(
|
||||
views
|
||||
.into_iter()
|
||||
|
|
@ -380,66 +367,6 @@ fn agent_exists(name: &hive_types::Ident) -> Result<bool> {
|
|||
.with_context(|| format!("check agent state dir for {name}"))
|
||||
}
|
||||
|
||||
/// Validate + persist an agent's CPU/memory overrides, then re-apply the
|
||||
/// drop-in so the change lands without waiting for a rebuild.
|
||||
///
|
||||
/// Validation is here rather than only in `hivectl` because the values
|
||||
/// are written verbatim into the systemd drop-in: a malformed
|
||||
/// `CPUQuota=` makes systemd reject the unit, and the container stops
|
||||
/// starting. Every client (CLI, dashboard, anything later) goes through
|
||||
/// this path, so the guard belongs on this side of the socket.
|
||||
///
|
||||
/// `None`/`None` removes the agent's entry, returning it to the
|
||||
/// hive-wide defaults.
|
||||
async fn handle_set_resource_limits(
|
||||
coord: &Arc<Coordinator>,
|
||||
name: &hive_types::Ident,
|
||||
cpu_quota: Option<&str>,
|
||||
memory_max: Option<&str>,
|
||||
) -> Result<HostResponse> {
|
||||
if let Some(value) = cpu_quota {
|
||||
crate::resource_limits::validate_cpu_quota(value).map_err(anyhow::Error::msg)?;
|
||||
}
|
||||
// `validate_memory_max` returns the value to store, not just an
|
||||
// ok/err verdict: a friendly spelling like "8GB" is accepted but
|
||||
// normalized to the "8G" systemd's own parser actually takes.
|
||||
let memory_max = memory_max
|
||||
.map(crate::resource_limits::validate_memory_max)
|
||||
.transpose()
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
tracing::info!(%name, ?cpu_quota, ?memory_max, "set_resource_limits");
|
||||
let limits = crate::resource_limits::AgentLimits {
|
||||
cpu_quota: cpu_quota.map(ToOwned::to_owned),
|
||||
memory_max,
|
||||
};
|
||||
// Goes through `meta::commit_resource_limits`, not the bare
|
||||
// `resource_limits::set_limits`: the write has to be staged +
|
||||
// committed under `META_LOCK` or it leaves the meta working tree
|
||||
// dirty for the next `sync_agents` to trip over.
|
||||
crate::meta::commit_resource_limits(name.as_str(), &limits).await?;
|
||||
|
||||
// Re-apply the drop-in straight away — same three lines as the job
|
||||
// queue's `WriteDropin` node. Without this the new values would sit
|
||||
// in the JSON until the agent's next spawn or rebuild.
|
||||
let agent_dir = crate::paths::agent_runtime_dir(name.as_str());
|
||||
let hive = coord.hive_env();
|
||||
let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?;
|
||||
crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?;
|
||||
|
||||
let (cpu, mem) = crate::resource_limits::effective_with_swarm(
|
||||
&crate::resource_limits::resource_limits_path(),
|
||||
&crate::resource_limits::swarm_limits_path(),
|
||||
name.as_str(),
|
||||
&hive.agent_cpu_quota,
|
||||
&hive.agent_memory_max,
|
||||
)?;
|
||||
Ok(HostResponse::messages(vec![format!(
|
||||
"{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now; the \
|
||||
Claude/JSC heap ceiling and the subagent MemoryHigh= derived from it follow at the \
|
||||
container's next start (restart it if it's running and needs them now)"
|
||||
)]))
|
||||
}
|
||||
|
||||
/// Guard: matrix provisioning needs a homeserver to provision against.
|
||||
///
|
||||
/// Answers "is one configured", not "is one running here": the address is
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
//! Live per-agent-container resource load (CPU + memory, plus the slow
|
||||
//! disk sample from hive-priv in [`disk_sampler_loop`]) for the
|
||||
//! dashboard. Reads cgroup v2 stat files for each running agent
|
||||
//! disk sample from hive-priv in [`disk_sampler_loop`]) for the OTEL
|
||||
//! exporter. Reads cgroup v2 stat files for each running agent
|
||||
//! machine directly — same privsep-clean posture as the turn-stats
|
||||
//! sqlite reads (`cpu.stat` / `memory.*` are world-readable; no
|
||||
//! `hive-priv` needed). Runs host-side in hive-c0re, where
|
||||
|
|
@ -31,8 +31,8 @@ use utoipa::ToSchema;
|
|||
|
||||
use crate::coordinator::Coordinator;
|
||||
|
||||
/// Sampling interval for the two CPU reads. Short enough to keep the
|
||||
/// endpoint snappy, long enough that the delta isn't dominated by read
|
||||
/// Sampling interval for the two CPU reads. Short enough to keep a sample
|
||||
/// quick, long enough that the delta isn't dominated by read
|
||||
/// jitter.
|
||||
const CPU_SAMPLE: Duration = Duration::from_millis(200);
|
||||
|
||||
|
|
@ -72,8 +72,8 @@ pub struct ContainerResource {
|
|||
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(30);
|
||||
|
||||
/// Last per-agent disk measurement, keyed by agent name; `None` is a failed
|
||||
/// one. Written by [`disk_sampler_loop`], read by [`gather`] so the hot
|
||||
/// poll never measures.
|
||||
/// one. Written by [`disk_sampler_loop`], read by [`gather`], so a sample
|
||||
/// never measures.
|
||||
fn disk_cache() -> &'static RwLock<HashMap<String, Option<u64>>> {
|
||||
static CACHE: OnceLock<RwLock<HashMap<String, Option<u64>>>> = OnceLock::new();
|
||||
CACHE.get_or_init(|| RwLock::new(HashMap::new()))
|
||||
|
|
@ -100,7 +100,7 @@ fn disk_reading(name: &hive_types::Ident, measured: anyhow::Result<u64>) -> Opti
|
|||
|
||||
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], remeasure disk usage
|
||||
/// for every kept-state agent and refresh [`disk_cache`]. The first pass
|
||||
/// runs immediately so the dashboard shows disk shortly after boot.
|
||||
/// runs immediately so the exporter has disk shortly after boot.
|
||||
/// Runs forever; spawn once at hive-c0re startup via [`spawn_disk_sampler`].
|
||||
pub async fn disk_sampler_loop() {
|
||||
loop {
|
||||
|
|
|
|||
|
|
@ -234,8 +234,6 @@ mod tests {
|
|||
status_text: None,
|
||||
status_set_at: None,
|
||||
paused: false,
|
||||
cpu_quota: "200%".to_owned(),
|
||||
memory_max: "4G".to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
//! Per-agent container-resource OTEL export. hive-c0re already samples each
|
||||
//! agent container's cgroup load for the dashboard
|
||||
//! ([`super::container_stats`]); this rides those same gauges out to the
|
||||
//! Per-agent container-resource OTEL export. This is what the cgroup load
|
||||
//! sampled in [`super::container_stats`] feeds: it rides those gauges out to the
|
||||
//! hive's own collector — the same first hop the agents use, named by the
|
||||
//! standard `OTEL_EXPORTER_OTLP_ENDPOINT` (see `nix/host-modules/hive-c0re`,
|
||||
//! which sets it from the same binding it hands agents). No toggle of
|
||||
|
|
@ -194,7 +193,7 @@ fn register_instruments(provider: &SdkMeterProvider, snapshot: Arc<Mutex<Vec<Con
|
|||
.build();
|
||||
|
||||
// Custom (`hyperhive.`) metrics semconv doesn't standardise: memory peak,
|
||||
// on-disk footprint, and the instantaneous cpu percent the dashboard shows.
|
||||
// on-disk footprint, and the instantaneous cpu percent.
|
||||
let snap = snapshot.clone();
|
||||
meter
|
||||
.u64_observable_gauge("hyperhive.container.memory.peak")
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
//! swarm-controller keeps one key per agent in the `agent-limits` bucket
|
||||
//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads
|
||||
//! the key of each agent this hive has and stores what it finds in
|
||||
//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it.
|
||||
//! `swarm-resource-limits.json`, where the drop-in reads it.
|
||||
//! A running agent gets a changed cap at its first spawn or rebuild after
|
||||
//! that read.
|
||||
//!
|
||||
|
|
@ -92,9 +92,9 @@ async fn sync(client: &async_nats::Client) -> Result<()> {
|
|||
/// The caps to store for `declared`, by agent. An agent with no field set
|
||||
/// gets no entry.
|
||||
///
|
||||
/// Every value goes through the validators the operator's own input path
|
||||
/// uses, because it is rendered verbatim into the container's systemd
|
||||
/// drop-in. One bad value refuses the whole set.
|
||||
/// Every value goes through the `resource_limits` validators, because it is
|
||||
/// rendered verbatim into the container's systemd drop-in. One bad value
|
||||
/// refuses the whole set.
|
||||
fn validated(declared: &BTreeMap<String, AgentCaps>) -> Result<BTreeMap<String, AgentLimits>> {
|
||||
let mut out = BTreeMap::new();
|
||||
for (agent, caps) in declared {
|
||||
|
|
@ -127,48 +127,38 @@ mod tests {
|
|||
use std::collections::BTreeMap;
|
||||
|
||||
use super::validated;
|
||||
use crate::resource_limits::AgentLimits;
|
||||
use swarm_queue_client::agent_limits::AgentCaps;
|
||||
|
||||
/// `atlas`'s answer with `atlas_key` as the value read for it, over a hive
|
||||
/// chain that caps it at 800% / 16G.
|
||||
/// `atlas`'s answer with `atlas_key` as the value read for it, over the
|
||||
/// hive-wide defaults of 200% / 4G.
|
||||
fn atlas_with(atlas_key: &str) -> (String, String) {
|
||||
let declared = BTreeMap::from([(
|
||||
"atlas".to_owned(),
|
||||
serde_json::from_str::<AgentCaps>(atlas_key).expect("decodes"),
|
||||
)]);
|
||||
let stored = validated(&declared).expect("valid caps");
|
||||
let hive_side = BTreeMap::from([(
|
||||
"atlas".to_owned(),
|
||||
AgentLimits {
|
||||
cpu_quota: Some("800%".to_owned()),
|
||||
memory_max: Some("16G".to_owned()),
|
||||
},
|
||||
)]);
|
||||
crate::resource_limits::effective_from_with_swarm(
|
||||
&stored, &hive_side, "atlas", "200%", "4G",
|
||||
)
|
||||
crate::resource_limits::effective_from(&stored, "atlas", "200%", "4G")
|
||||
}
|
||||
|
||||
/// The other half of swarm-controller's
|
||||
/// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`:
|
||||
/// the first value is the bytes that route stores. The cap replaces the
|
||||
/// hive's chain; a deleted key reads as `{}` and gives the chain back.
|
||||
/// hive default; a deleted key reads as `{}` and gives the default back.
|
||||
#[test]
|
||||
fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() {
|
||||
fn the_caps_the_route_stores_replace_the_hive_default_and_a_cleared_key_restores_it() {
|
||||
assert_eq!(
|
||||
atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#),
|
||||
("400%".to_owned(), "8G".to_owned())
|
||||
);
|
||||
assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned()));
|
||||
assert_eq!(atlas_with("{}"), ("200%".to_owned(), "4G".to_owned()));
|
||||
}
|
||||
|
||||
/// Per field: a swarm memory cap leaves CPU on the hive's chain.
|
||||
/// Per field: a swarm memory cap leaves CPU on the hive default.
|
||||
#[test]
|
||||
fn a_field_the_swarm_leaves_unset_stays_with_the_hive() {
|
||||
assert_eq!(
|
||||
atlas_with(r#"{"memory_max":"2G"}"#),
|
||||
("800%".to_owned(), "2G".to_owned())
|
||||
("200%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -211,23 +211,6 @@ pub enum HostRequest {
|
|||
/// this request just answers whatever it's asked. Result:
|
||||
/// [`HostResponse::nodes`].
|
||||
QueueNodes { ids: Vec<u64> },
|
||||
/// Declare an agent's CPU/memory overrides for the per-container
|
||||
/// systemd drop-in, persisted to `meta/resource-limits.json`.
|
||||
///
|
||||
/// **Replace, not merge** — the pair given here becomes the agent's
|
||||
/// entire entry, matching how tool-groups/capabilities are set. A
|
||||
/// `None` field falls back to the hive-wide
|
||||
/// `agentCpuQuota` / `agentMemoryMax`, so passing both as `None`
|
||||
/// removes the entry entirely (reset to hive defaults).
|
||||
///
|
||||
/// Values are passed verbatim to systemd, so the server validates
|
||||
/// their shape before persisting: a malformed `CPUQuota=` makes
|
||||
/// systemd reject the unit, which would stop the container starting.
|
||||
SetResourceLimits {
|
||||
name: Ident,
|
||||
cpu_quota: Option<String>,
|
||||
memory_max: Option<String>,
|
||||
},
|
||||
/// Stop managed containers hive-wide in one operator action
|
||||
/// (`hivectl stop`): agents plus the selected infra containers. `scope`
|
||||
/// selects which classes; an all-false scope means **everything** (the
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
//! `hivectl agent <name>` — everything scoped to ONE managed agent:
|
||||
//! container lifecycle over the host admin socket (restart/pause/resume/
|
||||
//! spawn/kill/destroy/rebuild/set-limits/choom/watch), plus the
|
||||
//! spawn/kill/destroy/rebuild/choom/watch), plus the
|
||||
//! `quota` and `subvol` groups, whose handlers live in their own modules.
|
||||
//! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide
|
||||
//! read that lives in this module too since it shares the same daemon
|
||||
|
|
@ -282,32 +282,6 @@ pub(crate) async fn run_agent(socket: &Path, name: &str, cmd: AgentCmd) -> Resul
|
|||
let name = crate::util::parse_ident(name)?;
|
||||
render(crate::client::request(socket, HostRequest::Rebuild { name }).await?)
|
||||
}
|
||||
AgentCmd::SetLimits {
|
||||
cpu_quota,
|
||||
memory_max,
|
||||
reset,
|
||||
} => {
|
||||
let name = crate::util::parse_ident(name)?;
|
||||
// `--reset` is the only way to reach an all-`None` request;
|
||||
// clap rejects a bare `set-limits` with neither flag, so a
|
||||
// forgotten value can't silently clear the overrides.
|
||||
let (cpu_quota, memory_max) = if reset {
|
||||
(None, None)
|
||||
} else {
|
||||
(cpu_quota, memory_max)
|
||||
};
|
||||
render(
|
||||
crate::client::request(
|
||||
socket,
|
||||
HostRequest::SetResourceLimits {
|
||||
name,
|
||||
cpu_quota,
|
||||
memory_max,
|
||||
},
|
||||
)
|
||||
.await?,
|
||||
)
|
||||
}
|
||||
AgentCmd::Choom { resume_session } => {
|
||||
crate::choom::choom(socket, name, resume_session.as_deref()).await
|
||||
}
|
||||
|
|
|
|||
|
|
@ -393,30 +393,6 @@ pub enum AgentCmd {
|
|||
},
|
||||
/// Apply pending config to this managed container.
|
||||
Rebuild,
|
||||
/// Declare this agent's CPU/memory limits, overriding the hive-wide
|
||||
/// defaults.
|
||||
///
|
||||
/// Replaces the agent's whole override entry rather than merging into
|
||||
/// it: any limit you don't pass returns to the hive-wide default. To
|
||||
/// change one and keep the other, pass both. Disk is a separate
|
||||
/// resource with its own group — see `quota`.
|
||||
SetLimits {
|
||||
/// systemd `CPUQuota=` value, for example `400%` (100% = one full core).
|
||||
#[arg(long, conflicts_with = "reset")]
|
||||
cpu_quota: Option<String>,
|
||||
/// systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`.
|
||||
#[arg(long, conflicts_with = "reset")]
|
||||
memory_max: Option<String>,
|
||||
/// Drop all overrides — the agent returns to the hive-wide
|
||||
/// defaults. Required to clear limits, so that a `set-limits`
|
||||
/// with a forgotten value can't silently reset the agent.
|
||||
#[arg(
|
||||
long,
|
||||
conflicts_with_all = ["cpu_quota", "memory_max"],
|
||||
required_unless_present_any = ["cpu_quota", "memory_max"],
|
||||
)]
|
||||
reset: bool,
|
||||
},
|
||||
/// Open an interactive Claude session inside this agent's container.
|
||||
///
|
||||
/// A fresh session by default, or resume a prior one. Requires root
|
||||
|
|
|
|||
Loading…
Reference in a new issue