Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6f2455f84a | ||
|
|
1a86304f92 | ||
|
|
c875223db7 | ||
|
|
23cedf3fa8 | ||
|
|
90e33cbd19 | ||
|
|
a2540be98b |
34 changed files with 232 additions and 1364 deletions
|
|
@ -395,26 +395,26 @@ Contents:
|
||||||
(`{ "ruth": ["manage_root_agent"] }`). Written by
|
(`{ "ruth": ["manage_root_agent"] }`). Written by
|
||||||
`capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
|
`capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
|
||||||
var. Absent agents have no extra capabilities.
|
var. Absent agents have no extra capabilities.
|
||||||
- `resource-limits.json` — per-agent container resource overrides
|
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||||
|
hive's agents
|
||||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||||
Written by `resource_limits::set_limits`; read where
|
Written only by `workers::swarm_limits`, which copies each agent's key of
|
||||||
|
the swarm's `agent-limits` bucket here every 30 seconds. Read where
|
||||||
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
|
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
|
||||||
into the container — these are host-side caps on the container, so
|
into the container — these are host-side caps on the container, so
|
||||||
the capped party never sets them. It sees only the resulting memory cap,
|
the capped party never sets them. It sees only the resulting memory cap,
|
||||||
as a byte count or `infinity` in its runtime dir (below). Fallback is per *field*:
|
as a byte count or `infinity` in its runtime dir (below). Fallback is per
|
||||||
an absent file, absent agent, or absent field falls back to the
|
*field*: an absent file, absent agent, or absent field falls back to the
|
||||||
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
|
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
|
||||||
so an agent can override only its memory and still track the hive
|
so an agent can have only its memory capped and still track the hive
|
||||||
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
|
default for CPU. Every value passes the validators first, and one bad
|
||||||
drop-in have **no** per-agent override — they're hive-wide only and
|
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
|
||||||
come straight off `HiveEnv`, so this file has no field for them.
|
shares in the same drop-in have **no** per-agent setting — they're
|
||||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
hive-wide only and come straight off `HiveEnv`, so this file has no field
|
||||||
hive's agents, same shape as `resource-limits.json`. Written only by
|
for them.
|
||||||
`workers::swarm_limits`, which copies each agent's key of the swarm's
|
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
|
||||||
`agent-limits` bucket here every 30 seconds. A field set here replaces
|
to live here; nothing reads or writes it now. An old copy in the meta repo
|
||||||
that field's answer from `resource-limits.json` and the hive-wide
|
is harmless and you can delete it.
|
||||||
default. Every value passes the same validators as the operator's
|
|
||||||
overrides first, and one bad value makes the worker change nothing here.
|
|
||||||
|
|
||||||
The root agent has the meta dir RO-mounted at `/meta/`.
|
The root agent has the meta dir RO-mounted at `/meta/`.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -455,12 +455,12 @@ requiring a host rebuild.
|
||||||
The answer per agent and per field is, first match wins:
|
The answer per agent and per field is, first match wins:
|
||||||
|
|
||||||
1. the swarm's cap for the agent (see below);
|
1. the swarm's cap for the agent (see below);
|
||||||
2. the operator's per-agent override in `meta/resource-limits.json`;
|
2. `agentCpuQuota` / `agentMemoryMax`.
|
||||||
3. `agentCpuQuota` / `agentMemoryMax`.
|
|
||||||
|
|
||||||
A swarm cap replaces the hive's answer, so a hive can't raise an agent
|
A swarm cap replaces the hive's answer, so a hive can't raise an agent
|
||||||
above it. Like the rest of the drop-in, a changed swarm cap reaches a
|
above it. Like the rest of the drop-in, a changed swarm cap reaches a
|
||||||
running agent at its next spawn or rebuild.
|
running agent at its next spawn or rebuild. A hive has no way to set a cap
|
||||||
|
for one agent: you set caps per agent in the swarm UI.
|
||||||
|
|
||||||
The container can't read its own cap, so `write_dropins` also writes the
|
The container can't read its own cap, so `write_dropins` also writes the
|
||||||
effective `MemoryMax=` into the agent's runtime dir
|
effective `MemoryMax=` into the agent's runtime dir
|
||||||
|
|
@ -468,7 +468,7 @@ effective `MemoryMax=` into the agent's runtime dir
|
||||||
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
|
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
|
||||||
it, as that share of the host's physical memory rounded down to a page. A
|
it, as that share of the host's physical memory rounded down to a page. A
|
||||||
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
|
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
|
||||||
the last drop-in because it can't read the overrides or the swarm caps file,
|
the last drop-in because it can't read the swarm caps file,
|
||||||
the file carries that drop-in's `MemoryMax=`.
|
the file carries that drop-in's `MemoryMax=`.
|
||||||
|
|
||||||
At boot the container derives two values from a byte count:
|
At boot the container derives two values from a byte count:
|
||||||
|
|
@ -520,7 +520,7 @@ given.
|
||||||
|
|
||||||
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
||||||
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
||||||
value passes the same validators as the operator's overrides first. One bad
|
value passes the validators first. One bad
|
||||||
value or a failed read leaves the file unchanged, and repeated failures raise a
|
value or a failed read leaves the file unchanged, and repeated failures raise a
|
||||||
dashboard banner.
|
dashboard banner.
|
||||||
|
|
||||||
|
|
@ -548,8 +548,8 @@ wants it.
|
||||||
Two things to know about the weights:
|
Two things to know about the weights:
|
||||||
|
|
||||||
- They're **hive-wide** — unlike the caps there is no per-agent
|
- They're **hive-wide** — unlike the caps there is no per-agent
|
||||||
override in `meta/resource-limits.json`, so every agent carries the
|
setting, so every agent carries the same value and the weight does _not_
|
||||||
same value and the weight does _not_ rank agents against each other.
|
rank agents against each other.
|
||||||
What `80` buys is that agents yield to everything **not** on this
|
What `80` buys is that agents yield to everything **not** on this
|
||||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||||
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
|
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
|
||||||
|
|
|
||||||
|
|
@ -291,8 +291,8 @@ cleanly and the affected level button goes on returning zero rows:
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
|
||||||
When OTEL is enabled, **hive-c0re itself** also exports each agent
|
When OTEL is enabled, **hive-c0re itself** also exports each agent
|
||||||
container's resource load — the same cgroup gauges shown on the dashboard
|
container's resource load — the same cgroup gauges —
|
||||||
LOAD tab — to this hive's own collector, exactly like an agent does and with
|
to this hive's own collector, exactly like an agent does and with
|
||||||
no separate toggle. These come from the host, not the in-container Claude SDK,
|
no separate toggle. These come from the host, not the in-container Claude SDK,
|
||||||
so they cover containers even when their agent is idle.
|
so they cover containers even when their agent is idle.
|
||||||
|
|
||||||
|
|
@ -312,7 +312,7 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
||||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||||
|
|
||||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,6 @@ This document contains the help content for the `hivectl` command-line program.
|
||||||
* [`hivectl agent kill`↴](#hivectl-agent-kill)
|
* [`hivectl agent kill`↴](#hivectl-agent-kill)
|
||||||
* [`hivectl agent destroy`↴](#hivectl-agent-destroy)
|
* [`hivectl agent destroy`↴](#hivectl-agent-destroy)
|
||||||
* [`hivectl agent rebuild`↴](#hivectl-agent-rebuild)
|
* [`hivectl agent rebuild`↴](#hivectl-agent-rebuild)
|
||||||
* [`hivectl agent set-limits`↴](#hivectl-agent-set-limits)
|
|
||||||
* [`hivectl agent choom`↴](#hivectl-agent-choom)
|
* [`hivectl agent choom`↴](#hivectl-agent-choom)
|
||||||
* [`hivectl agent watch`↴](#hivectl-agent-watch)
|
* [`hivectl agent watch`↴](#hivectl-agent-watch)
|
||||||
* [`hivectl agent quota`↴](#hivectl-agent-quota)
|
* [`hivectl agent quota`↴](#hivectl-agent-quota)
|
||||||
|
|
@ -221,7 +220,6 @@ Everything here targets a single named agent (`hivectl agent foo restart`, `hive
|
||||||
* `kill` — Hard-stop this managed container
|
* `kill` — Hard-stop this managed container
|
||||||
* `destroy` — Tear down this sub-agent container, keeping its state by default. No undo
|
* `destroy` — Tear down this sub-agent container, keeping its state by default. No undo
|
||||||
* `rebuild` — Apply pending config to this managed container
|
* `rebuild` — Apply pending config to this managed container
|
||||||
* `set-limits` — Declare this agent's CPU/memory limits, overriding the hive-wide defaults
|
|
||||||
* `choom` — Open an interactive Claude session inside this agent's container
|
* `choom` — Open an interactive Claude session inside this agent's container
|
||||||
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI
|
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI
|
||||||
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups
|
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups
|
||||||
|
|
@ -315,22 +313,6 @@ Apply pending config to this managed container
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl agent set-limits`
|
|
||||||
|
|
||||||
Declare this agent's CPU/memory limits, overriding the hive-wide defaults.
|
|
||||||
|
|
||||||
Replaces the agent's whole override entry rather than merging into it: any limit you don't pass returns to the hive-wide default. To change one and keep the other, pass both. Disk is a separate resource with its own group — see `quota`.
|
|
||||||
|
|
||||||
**Usage:** `hivectl agent set-limits [OPTIONS]`
|
|
||||||
|
|
||||||
###### **Options:**
|
|
||||||
|
|
||||||
* `--cpu-quota <CPU_QUOTA>` — systemd `CPUQuota=` value, for example `400%` (100% = one full core)
|
|
||||||
* `--memory-max <MEMORY_MAX>` — systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`
|
|
||||||
* `--reset` — Drop all overrides — the agent returns to the hive-wide defaults. Required to clear limits, so that a `set-limits` with a forgotten value can't silently reset the agent
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl agent choom`
|
## `hivectl agent choom`
|
||||||
|
|
||||||
Open an interactive Claude session inside this agent's container.
|
Open an interactive Claude session inside this agent's container.
|
||||||
|
|
|
||||||
|
|
@ -136,34 +136,6 @@ resume drains the backlog rather than dropping it. Points worth knowing:
|
||||||
- Visible as ` paused` in `list-agents`'s STATUS column, as a `paused`
|
- Visible as ` paused` in `list-agents`'s STATUS column, as a `paused`
|
||||||
field on the JSON rows, and as a badge on the dashboard card.
|
field on the JSON rows, and as a badge on the dashboard card.
|
||||||
|
|
||||||
### Per-agent resource limits
|
|
||||||
|
|
||||||
```bash
|
|
||||||
hivectl agent sock set-limits --cpu-quota 400% --memory-max 8G
|
|
||||||
hivectl agent sock set-limits --memory-max 8G # CPU falls back to the hive default
|
|
||||||
hivectl agent sock set-limits --reset # drop all overrides
|
|
||||||
```
|
|
||||||
|
|
||||||
Overrides the hive-wide `services.hyperhive.c0re.agentCpuQuota` /
|
|
||||||
`agentMemoryMax` for one agent, persisted to
|
|
||||||
`meta/resource-limits.json` (see
|
|
||||||
[`persistence.md`](../agent-lifecycle/persistence.md)). Values are systemd's
|
|
||||||
`CPUQuota=` / `MemoryMax=` syntax: a percentage (`400%` = four full
|
|
||||||
cores) for CPU; a size (`8G`), a percentage of physical RAM, or
|
|
||||||
`infinity` for memory. `set-limits` validates both before persisting them —
|
|
||||||
they go into a systemd drop-in verbatim, and a typo there makes the
|
|
||||||
unit fail to start.
|
|
||||||
|
|
||||||
**Declarative, not incremental**: each invocation replaces the agent's
|
|
||||||
whole entry. `hivectl agent sock set-limits --memory-max 8G` leaves `sock`
|
|
||||||
with _only_ a memory override, reverting any previously set CPU quota to
|
|
||||||
the hive default. To avoid a forgotten flag silently wiping an override,
|
|
||||||
hivectl rejects `set-limits` with no flags at all — clearing requires the
|
|
||||||
explicit `--reset`.
|
|
||||||
|
|
||||||
The command rewrites the container's drop-in and reloads systemd, so
|
|
||||||
new containers and restarts pick the values up immediately.
|
|
||||||
|
|
||||||
## Choom
|
## Choom
|
||||||
|
|
||||||
Drop into an interactive Claude session inside an agent container.
|
Drop into an interactive Claude session inside an agent container.
|
||||||
|
|
|
||||||
|
|
@ -74,13 +74,17 @@ agent is stale.
|
||||||
Passive / rare-interaction state, reached from the **Core** tile on the
|
Passive / rare-interaction state, reached from the **Core** tile on the
|
||||||
H0M3 hub (served at
|
H0M3 hub (served at
|
||||||
`/core.html`), with the same minimal chrome as `/builds.html`: a `← home`
|
`/core.html`), with the same minimal chrome as `/builds.html`: a `← home`
|
||||||
back-link + a `createTabStrip` sub-tab nav (**K3PT ST4T3** default,
|
back-link and a single pane, **K3PT ST4T3**. The page is its own esbuild
|
||||||
then **C0NT41N3R L04D**). The page is its own esbuild bundle (`core.js`)
|
bundle (`core.js`) that cold-loads `/api/state` and subscribes to
|
||||||
that cold-loads `/api/state` and subscribes to `/api/dashboard/stream` for
|
`/api/dashboard/stream` for `tombstones_changed`, `capabilities_changed`,
|
||||||
`tombstones_changed`, `capabilities_changed`, and `tool_groups_changed`
|
and `tool_groups_changed` (the latter two re-render the stale-perms
|
||||||
(the latter two re-render the stale-perms sub-section when permission
|
sub-section when permission data changes). (Rebuild queue and meta inputs
|
||||||
data changes). (Rebuild queue and meta inputs live on the BU1LDS
|
live on the BU1LDS page — see below.)
|
||||||
page — see below.)
|
|
||||||
|
Per-container CPU, memory and disk load isn't shown in the hive UI. The
|
||||||
|
same figures go out as OTEL metrics and appear in Grafana (see
|
||||||
|
[observability](../scheduler/observability.md)). An agent's CPU and memory
|
||||||
|
caps live per agent in the swarm UI.
|
||||||
|
|
||||||
**K3PT ST4T3** — two sub-sections on one pane:
|
**K3PT ST4T3** — two sub-sections on one pane:
|
||||||
|
|
||||||
|
|
@ -102,32 +106,11 @@ snapshot-storage rework where the problem changes shape.
|
||||||
*Stale permission entries*: agents with explicit capability or
|
*Stale permission entries*: agents with explicit capability or
|
||||||
tool-group JSON entries but no live container — typically renamed or
|
tool-group JSON entries but no live container — typically renamed or
|
||||||
deleted agents whose entries persisted in `capabilities.json` /
|
deleted agents whose entries persisted in `capabilities.json` /
|
||||||
`tool-groups.json`. Lazy-loaded on first K3PT ST4T3 tab activation;
|
`tool-groups.json`. Loaded when the page opens;
|
||||||
autorefreshes on `capabilities_changed` + `tool_groups_changed` SSE
|
autorefreshes on `capabilities_changed` + `tool_groups_changed` SSE
|
||||||
events. Each ghost agent gets a `✕ clear perms` button
|
events. Each ghost agent gets a `✕ clear perms` button
|
||||||
(`DELETE /api/permissions/{name}`).
|
(`DELETE /api/permissions/{name}`).
|
||||||
|
|
||||||
**C0NT41N3R L04D** — live CPU + memory per agent container, read
|
|
||||||
straight from cgroup v2 on the host (`cpu.stat`, `memory.current`,
|
|
||||||
`memory.peak`, `memory.max` under
|
|
||||||
`/sys/fs/cgroup/machine.slice/machine-h\x2d<name>.scope/`). CPU is a
|
|
||||||
host-normalised percentage (0..100 across all cores) sampled over a
|
|
||||||
short (~200 ms) two-read interval; memory shows current + peak with a
|
|
||||||
bar against the `memory.max` quota. Backed by
|
|
||||||
`GET /api/container-resources` (`container_stats.rs`), which reads the
|
|
||||||
files read-only (world-readable; no `hive-priv`) and skips agents whose
|
|
||||||
scope dir is absent (= not running). Pull-only: `core.js` polls every
|
|
||||||
5 s **only while the C0NT41N3R L04D sub-tab is active** (CPU needs a fresh
|
|
||||||
sample each refresh), and stops on sub-tab change. Disk size (`disk_bytes`)
|
|
||||||
rides the same row, but a separate background sampler feeds it every
|
|
||||||
30 min: `hive-priv` runs one bounded `du -x` as root over the agent's
|
|
||||||
`state/` + `work/` + container writable rootfs (shared nix store excluded),
|
|
||||||
so the 5 s poll stays cheap cgroup-only reads. The row carries the last
|
|
||||||
measured value; the column shows "unknown" (`null`) before the first
|
|
||||||
measurement and whenever the last one failed, never 0. Network isn't tracked
|
|
||||||
per container — see `docs/networking/network.md` for the current
|
|
||||||
per-agent netns model.
|
|
||||||
|
|
||||||
## BU1LDS page (`/builds.html`)
|
## BU1LDS page (`/builds.html`)
|
||||||
|
|
||||||
The build lifecycle hub — rebuild queue, live build log, meta inputs,
|
The build lifecycle hub — rebuild queue, live build log, meta inputs,
|
||||||
|
|
@ -765,17 +748,6 @@ below — some endpoints aren't in it yet.
|
||||||
zero-turn dbs); returns swarm totals, a busiest-first per-agent
|
zero-turn dbs); returns swarm totals, a busiest-first per-agent
|
||||||
rollup, swarm model mix, and a labelled `est_cost_usd`. Window
|
rollup, swarm model mix, and a labelled `est_cost_usd`. Window
|
||||||
defaults to `24h`.
|
defaults to `24h`.
|
||||||
- `GET /api/container-resources` — live per-agent-container CPU +
|
|
||||||
memory from cgroup v2 (C0R3 › C0NT41N3R L04D panel). Returns one
|
|
||||||
row per running agent (`name`, `cpu_pct`, `mem_current_bytes`,
|
|
||||||
`mem_peak_bytes`, `mem_max_bytes`, `disk_bytes`); samples CPU over
|
|
||||||
~200 ms so the call briefly awaits. Skips non-running agents (no scope
|
|
||||||
dir). No network field — agents share the host netns. `disk_bytes` is
|
|
||||||
the last value from a **separate slow sampler** (not this hot path):
|
|
||||||
every 30 min `hive-priv` runs a bounded `du -sxb` as root over the agent's
|
|
||||||
`state/` + `work/` + container writable rootfs, `-x` excluding the shared
|
|
||||||
read-only nix store (`PrivRequest::MeasureAgentDisk`). `null` means
|
|
||||||
unknown: not measured yet, or the last measurement failed.
|
|
||||||
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
||||||
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
||||||
failed (clears the failure state so the scheduler retries).
|
failed (clears the failure state so the scheduler retries).
|
||||||
|
|
|
||||||
|
|
@ -619,8 +619,8 @@ hive-side-panel .md td {
|
||||||
|
|
||||||
/* ─── stats table ─────────────────────────────────────────────────
|
/* ─── stats table ─────────────────────────────────────────────────
|
||||||
Shared right-aligned numeric table. Used by /stats.html (busiest
|
Shared right-aligned numeric table. Used by /stats.html (busiest
|
||||||
agents) and the dashboard SYST3M › C0NT41N3R L04D table — moved here
|
agents) — moved here from dashboard.css when ST4TS became its own
|
||||||
from dashboard.css when ST4TS became its own page. */
|
page. */
|
||||||
.hive-stats-table {
|
.hive-stats-table {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
border-collapse: collapse;
|
border-collapse: collapse;
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
/* ─── /core.html — C0R3 page ───────────────────────────────────────
|
/* ─── /core.html — C0R3 page ───────────────────────────────────────
|
||||||
Kept state (tombstones) and container load. Section rules imported
|
Kept state (tombstones) and stale permission entries. Section rules
|
||||||
from system-sections.css (shared with dashboard.css + builds.css). */
|
imported from system-sections.css (shared with dashboard.css +
|
||||||
|
builds.css). */
|
||||||
@import "./system-sections.css";
|
@import "./system-sections.css";
|
||||||
|
|
||||||
body.core-shell {
|
body.core-shell {
|
||||||
|
|
@ -8,22 +9,10 @@ body.core-shell {
|
||||||
padding: 0;
|
padding: 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Fill the header row next to the ← home back-link, same as the
|
|
||||||
builds sub-tab strip. */
|
|
||||||
.core-tabbar {
|
|
||||||
flex: 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
.core-main {
|
.core-main {
|
||||||
padding: 1.2em 1.5em 2em;
|
padding: 1.2em 1.5em 2em;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* createTabStrip toggles the native `hidden` attribute on inactive
|
|
||||||
panes; ensure it wins over any inherited display. */
|
|
||||||
.core-pane[hidden] {
|
|
||||||
display: none;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ─── K3PT ST4T3 container cards ────────────────────────────────────
|
/* ─── K3PT ST4T3 container cards ────────────────────────────────────
|
||||||
core.html doesn't load dashboard.css (that's the operator SPA),
|
core.html doesn't load dashboard.css (that's the operator SPA),
|
||||||
so the .container-row card styles aren't inherited. Redefine them
|
so the .container-row card styles aren't inherited. Redefine them
|
||||||
|
|
|
||||||
|
|
@ -13,70 +13,31 @@
|
||||||
<body class="core-shell">
|
<body class="core-shell">
|
||||||
<!-- C0R3: the host/coordinator surface. Standalone page (served at
|
<!-- C0R3: the host/coordinator surface. Standalone page (served at
|
||||||
/core.html) carved out of the dashboard's old SYST3M tab so the
|
/core.html) carved out of the dashboard's old SYST3M tab so the
|
||||||
dashboard tab strip stays lean. Same minimal chrome as
|
dashboard tab strip stays lean: kept state (tombstones) and the
|
||||||
/builds.html — a `← home` back-link to the H0M3 hub + a
|
stale permission entries. The section <div> ids (tombstones-section,
|
||||||
<hive-tab-strip> sub-tab nav. Two sub-tabs: kept state (tombstones)
|
tombstones-stale-perms) match what core.js's renderers target. Hive
|
||||||
and container load. Rebuild queue + meta inputs have moved to
|
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`) have
|
||||||
/builds.html (the build lifecycle hub). Default tab: K3PT ST4T3.
|
no dashboard panel — `hivectl stop`/`start`/`restart` is the only
|
||||||
The section <div> ids (tombstones-section, container-load-section)
|
|
||||||
match what core.js's renderers target. Hive infra containers
|
|
||||||
(`hive-forge`, `hive-gateway`, `hive-matrix`) have no
|
|
||||||
dashboard panel — `hivectl stop`/`start`/`restart` is the only
|
|
||||||
control surface. -->
|
control surface. -->
|
||||||
<header class="page-header">
|
<header class="page-header">
|
||||||
<a class="page-back" href="/">← home</a>
|
<a class="page-back" href="/">← home</a>
|
||||||
<hive-tab-strip
|
<span class="page-title">K3PT ST4T3</span>
|
||||||
class="hive-tabbar core-tabbar"
|
|
||||||
id="core-tabbar"
|
|
||||||
prefix="core"
|
|
||||||
role="tablist"
|
|
||||||
></hive-tab-strip>
|
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
<main class="core-main">
|
<main class="core-main">
|
||||||
<!-- K3PT ST4T3: tombstoned-agent kept state + stale permission entries.
|
<!-- tombstones-section shows destroyed agents (purge button).
|
||||||
tombstones-section shows destroyed agents (purge button).
|
tombstones-stale-perms shows agents with explicit capability/
|
||||||
tombstones-stale-perms is lazy-loaded on tab activation and shows
|
tool-group entries but no live container (typically renamed/deleted
|
||||||
agents with explicit capability/tool-group entries but no live
|
agents whose JSON entries persisted). Each stale entry gets a
|
||||||
container (typically renamed/deleted agents whose JSON entries
|
"✕ clear perms" button. -->
|
||||||
persisted). Each stale entry gets a "✕ clear perms" button. -->
|
<p class="meta">
|
||||||
<section
|
kept state from previously tombstoned agents — recreating an agent with
|
||||||
class="core-pane"
|
the same name reuses it.
|
||||||
id="core-pane-kept"
|
</p>
|
||||||
data-tab-pane="kept"
|
<div id="tombstones-section">
|
||||||
role="tabpanel"
|
<p class="meta">loading…</p>
|
||||||
aria-labelledby="core-tab-kept"
|
</div>
|
||||||
>
|
<div id="tombstones-stale-perms"></div>
|
||||||
<p class="meta">
|
|
||||||
kept state from previously tombstoned agents — recreating an agent
|
|
||||||
with the same name reuses it.
|
|
||||||
</p>
|
|
||||||
<div id="tombstones-section">
|
|
||||||
<p class="meta">loading…</p>
|
|
||||||
</div>
|
|
||||||
<div id="tombstones-stale-perms"></div>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- C0NT41N3R L04D: live cpu + memory per agent container, from
|
|
||||||
cgroup v2 on the host. Polled only while this sub-tab is active. -->
|
|
||||||
<section
|
|
||||||
class="core-pane"
|
|
||||||
id="core-pane-load"
|
|
||||||
data-tab-pane="load"
|
|
||||||
role="tabpanel"
|
|
||||||
aria-labelledby="core-tab-load"
|
|
||||||
>
|
|
||||||
<p class="meta">
|
|
||||||
live cpu + memory per agent container, from cgroup v2 on the host. cpu
|
|
||||||
is % of total host capacity (all cores), sampled over ~200ms each
|
|
||||||
refresh; polled every 5s while this tab is open. network is omitted on
|
|
||||||
purpose — agents share the host netns, so there is no per-container
|
|
||||||
counter (see <code>docs/networking/network.md</code>).
|
|
||||||
</p>
|
|
||||||
<div id="container-load-section">
|
|
||||||
<p class="meta">loading…</p>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<script type="module" src="/static/core.js" defer></script>
|
<script type="module" src="/static/core.js" defer></script>
|
||||||
|
|
|
||||||
|
|
@ -1,27 +1,20 @@
|
||||||
// C0R3 page entry (/core.html). The host/coordinator surface, its own
|
// C0R3 page entry (/core.html). The host/coordinator surface, its own
|
||||||
// standalone page so the dashboard tab strip stays lean. Two sub-tabs
|
// standalone page so the dashboard tab strip stays lean: kept state
|
||||||
// via <hive-tab-strip> (default = K3PT ST4T3): kept state (tombstones)
|
// (tombstones) and the stale permission entries.
|
||||||
// and container load.
|
|
||||||
//
|
//
|
||||||
// Rebuild queue + meta inputs live at /builds.html (the build lifecycle
|
// This page is its own esbuild bundle with its own runtime: it cold-loads
|
||||||
// hub), not here. This page is its own esbuild bundle with its own
|
// /api/state and subscribes to /dashboard/stream (the same broker event
|
||||||
// runtime: it cold-loads /api/state and subscribes to /dashboard/stream
|
// channel the dashboard uses), maintaining its own copy of the tombstones
|
||||||
// (the same broker event channel the dashboard uses), maintaining its
|
// state.
|
||||||
// own copy of the tombstones state.
|
|
||||||
|
|
||||||
import { $, form, openStream, initServerWarnings } from "./common.js";
|
import { $, form, openStream, initServerWarnings } from "./common.js";
|
||||||
import { el } from "@hive/shared/dom.js";
|
import { el } from "@hive/shared/dom.js";
|
||||||
import { asyncBtn, bindAsyncForms } from "@hive/shared/forms.js";
|
import { asyncBtn, bindAsyncForms } from "@hive/shared/forms.js";
|
||||||
import "@hive/shared/hive-tab-strip.js";
|
|
||||||
|
|
||||||
// ─── derived state (own copies; this bundle has its own runtime) ──────────
|
// ─── derived state (own copies; this bundle has its own runtime) ──────────
|
||||||
let tombstonesState = [];
|
let tombstonesState = [];
|
||||||
// ContainerView list from the last /api/state snapshot — used by renderContainerLoad
|
|
||||||
// to splice in the configured cpu_quota/memory_max limits alongside live cgroup data.
|
|
||||||
let containersState = [];
|
|
||||||
function syncFromSnapshot(s) {
|
function syncFromSnapshot(s) {
|
||||||
tombstonesState = (s.tombstones || []).slice();
|
tombstonesState = (s.tombstones || []).slice();
|
||||||
containersState = (s.containers || []).slice();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── kept state (tombstones) ──────────────────────────────────────────────
|
// ─── kept state (tombstones) ──────────────────────────────────────────────
|
||||||
|
|
@ -187,273 +180,6 @@ async function fetchAndRenderStalePerms() {
|
||||||
stalePermsLoaded = true;
|
stalePermsLoaded = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── container load (live cgroup poll while the LOAD sub-tab is open) ──────
|
|
||||||
let containerLoadTimer = null;
|
|
||||||
|
|
||||||
function cloadFmtBytes(n) {
|
|
||||||
if (!Number.isFinite(n) || n <= 0) return "0";
|
|
||||||
const u = ["B", "KiB", "MiB", "GiB", "TiB"];
|
|
||||||
let i = 0;
|
|
||||||
let v = n;
|
|
||||||
while (v >= 1024 && i < u.length - 1) {
|
|
||||||
v /= 1024;
|
|
||||||
i += 1;
|
|
||||||
}
|
|
||||||
return v.toFixed(v < 10 && i > 0 ? 1 : 0) + " " + u[i];
|
|
||||||
}
|
|
||||||
function cloadMeter(pct) {
|
|
||||||
const p = Math.max(0, Math.min(100, pct));
|
|
||||||
const cls = p >= 90 ? "hot" : p >= 70 ? "warn" : "";
|
|
||||||
const m = document.createElement("span");
|
|
||||||
m.className = "cload-meter";
|
|
||||||
m.title = p.toFixed(0) + "%";
|
|
||||||
const fill = document.createElement("span");
|
|
||||||
fill.className = "fill" + (cls ? " " + cls : "");
|
|
||||||
fill.style.width = p + "%";
|
|
||||||
m.append(fill);
|
|
||||||
return m;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stash the last load rows so re-renders triggered by SSE state updates can
|
|
||||||
// call renderContainerLoad(lastLoadRows) without waiting for the next poll.
|
|
||||||
let lastLoadRows = [];
|
|
||||||
|
|
||||||
function renderContainerLoad(rows) {
|
|
||||||
lastLoadRows = rows;
|
|
||||||
const root = $("container-load-section");
|
|
||||||
if (!root) return;
|
|
||||||
if (!Array.isArray(rows) || rows.length === 0) {
|
|
||||||
root.replaceChildren();
|
|
||||||
const p = document.createElement("p");
|
|
||||||
p.className = "meta";
|
|
||||||
p.textContent = "no running agent containers";
|
|
||||||
root.append(p);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Build a name→ContainerView lookup so we can splice in the configured
|
|
||||||
// cpu_quota / memory_max limits (set in meta/resource-limits.json and
|
|
||||||
// resolved by the server) next to the live cgroup readings.
|
|
||||||
const cvByName = new Map(containersState.map((c) => [c.name, c]));
|
|
||||||
|
|
||||||
const table = document.createElement("table");
|
|
||||||
table.className = "hive-stats-table";
|
|
||||||
// "cpu cap" / "mem cap" show the configured ceilings from ContainerView
|
|
||||||
// (effective drop-in values, take effect on next start/restart).
|
|
||||||
// "limit" remains the live cgroup memory ceiling from /api/container-resources.
|
|
||||||
table.innerHTML =
|
|
||||||
"<thead><tr><th>agent</th><th>cpu</th><th>memory</th>" +
|
|
||||||
"<th>peak</th><th>limit</th><th>disk</th>" +
|
|
||||||
'<th class="cload-cap-th" title="configured ceiling — takes effect on next start">cpu cap</th>' +
|
|
||||||
'<th class="cload-cap-th" title="configured ceiling — takes effect on next start">mem cap</th>' +
|
|
||||||
"<th></th></tr></thead>";
|
|
||||||
const tb = document.createElement("tbody");
|
|
||||||
for (const r of rows) {
|
|
||||||
const cv = cvByName.get(r.name);
|
|
||||||
const tr = document.createElement("tr");
|
|
||||||
const name = document.createElement("td");
|
|
||||||
name.textContent = r.name;
|
|
||||||
tr.append(name);
|
|
||||||
const cpu = document.createElement("td");
|
|
||||||
cpu.className = "num";
|
|
||||||
cpu.append(
|
|
||||||
(Number(r.cpu_pct) || 0).toFixed(1) + "%",
|
|
||||||
cloadMeter(Number(r.cpu_pct) || 0),
|
|
||||||
);
|
|
||||||
tr.append(cpu);
|
|
||||||
const mem = document.createElement("td");
|
|
||||||
mem.className = "num";
|
|
||||||
const memCur = Number(r.mem_current_bytes) || 0;
|
|
||||||
if (r.mem_max_bytes) {
|
|
||||||
mem.append(
|
|
||||||
cloadFmtBytes(memCur),
|
|
||||||
cloadMeter((100 * memCur) / r.mem_max_bytes),
|
|
||||||
);
|
|
||||||
} else {
|
|
||||||
mem.textContent = cloadFmtBytes(memCur);
|
|
||||||
}
|
|
||||||
tr.append(mem);
|
|
||||||
const peak = document.createElement("td");
|
|
||||||
peak.className = "num";
|
|
||||||
peak.textContent = r.mem_peak_bytes
|
|
||||||
? cloadFmtBytes(Number(r.mem_peak_bytes))
|
|
||||||
: "—";
|
|
||||||
tr.append(peak);
|
|
||||||
const lim = document.createElement("td");
|
|
||||||
lim.className = "num";
|
|
||||||
lim.textContent = r.mem_max_bytes
|
|
||||||
? cloadFmtBytes(Number(r.mem_max_bytes))
|
|
||||||
: "∞";
|
|
||||||
tr.append(lim);
|
|
||||||
// Disk: on-disk footprint (state/ + work/ + container writable rootfs, shared
|
|
||||||
// nix store excluded). Measured out-of-band every 30 min server-side;
|
|
||||||
// null means unknown (not measured yet, or the measurement failed).
|
|
||||||
const disk = document.createElement("td");
|
|
||||||
disk.className = "num";
|
|
||||||
disk.title =
|
|
||||||
"state/ + work/ + container writable rootfs (shared nix store excluded); measured every 30 min";
|
|
||||||
disk.textContent =
|
|
||||||
r.disk_bytes != null ? cloadFmtBytes(Number(r.disk_bytes)) : "unknown";
|
|
||||||
tr.append(disk);
|
|
||||||
|
|
||||||
// Configured CPU / memory caps from ContainerView (resolved effective
|
|
||||||
// values: per-agent override when set, hive-wide default otherwise).
|
|
||||||
const cpuCap = document.createElement("td");
|
|
||||||
cpuCap.className = "num cload-cap";
|
|
||||||
cpuCap.title = "configured ceiling — takes effect on next start";
|
|
||||||
cpuCap.textContent = cv?.cpu_quota || "—";
|
|
||||||
tr.append(cpuCap);
|
|
||||||
const memCap = document.createElement("td");
|
|
||||||
memCap.className = "num cload-cap";
|
|
||||||
memCap.title = "configured ceiling — takes effect on next start";
|
|
||||||
memCap.textContent = cv?.memory_max || "—";
|
|
||||||
tr.append(memCap);
|
|
||||||
|
|
||||||
// S3T button toggles the inline edit row for this agent.
|
|
||||||
const actTd = document.createElement("td");
|
|
||||||
const setBtn = document.createElement("button");
|
|
||||||
setBtn.type = "button";
|
|
||||||
setBtn.className = "btn btn-sm cload-set-btn";
|
|
||||||
setBtn.textContent = "S3T";
|
|
||||||
setBtn.title = "set CPU / memory cap for " + r.name;
|
|
||||||
tr.append(actTd);
|
|
||||||
actTd.append(setBtn);
|
|
||||||
tb.append(tr);
|
|
||||||
|
|
||||||
// Inline edit row (hidden by default, toggled by the S3T button).
|
|
||||||
const editRow = document.createElement("tr");
|
|
||||||
editRow.className = "cload-edit-row";
|
|
||||||
editRow.hidden = true;
|
|
||||||
const editTd = document.createElement("td");
|
|
||||||
editTd.colSpan = 9;
|
|
||||||
editTd.className = "cload-edit-cell";
|
|
||||||
|
|
||||||
const editForm = document.createElement("form");
|
|
||||||
editForm.className = "cload-edit-form";
|
|
||||||
editForm.addEventListener("submit", async (e) => {
|
|
||||||
e.preventDefault();
|
|
||||||
const cpuInput = editForm.querySelector(".cload-cpu-input");
|
|
||||||
const memInput = editForm.querySelector(".cload-mem-input");
|
|
||||||
const errSpan = editForm.querySelector(".cload-edit-err");
|
|
||||||
const submitBtn = editForm.querySelector('[type="submit"]');
|
|
||||||
errSpan.hidden = true;
|
|
||||||
submitBtn.disabled = true;
|
|
||||||
try {
|
|
||||||
const body = new URLSearchParams({
|
|
||||||
cpu_quota: cpuInput.value.trim(),
|
|
||||||
memory_max: memInput.value.trim(),
|
|
||||||
});
|
|
||||||
const resp = await fetch(
|
|
||||||
"/api/resource-limits/" + encodeURIComponent(r.name),
|
|
||||||
{ method: "POST", body },
|
|
||||||
);
|
|
||||||
if (!resp.ok) {
|
|
||||||
errSpan.textContent = await resp
|
|
||||||
.text()
|
|
||||||
.catch(() => "error " + resp.status);
|
|
||||||
errSpan.hidden = false;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
// Success: collapse the edit row. The SSE rescan will push updated
|
|
||||||
// ContainerView data (cpu_quota/memory_max) to containersState,
|
|
||||||
// triggering a re-render of the cap columns via refreshContainerLoad.
|
|
||||||
editRow.hidden = true;
|
|
||||||
setBtn.textContent = "S3T";
|
|
||||||
} catch (err) {
|
|
||||||
errSpan.textContent = String(err);
|
|
||||||
errSpan.hidden = false;
|
|
||||||
} finally {
|
|
||||||
submitBtn.disabled = false;
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
const cpuLabel = document.createElement("label");
|
|
||||||
cpuLabel.className = "cload-edit-label";
|
|
||||||
cpuLabel.textContent = "cpu quota";
|
|
||||||
const cpuInput = document.createElement("input");
|
|
||||||
cpuInput.type = "text";
|
|
||||||
cpuInput.className = "cload-cpu-input";
|
|
||||||
// Pre-fill with the current *effective* value (not just a placeholder)
|
|
||||||
// so submitting after editing only the other field round-trips this one
|
|
||||||
// unchanged. The form always posts both fields, and the server treats
|
|
||||||
// an empty field as "clear this override" — a placeholder-only hint
|
|
||||||
// reads as filled to the eye but posts as empty, silently resetting
|
|
||||||
// whichever field the user didn't touch.
|
|
||||||
cpuInput.value = cv ? cv.cpu_quota : "";
|
|
||||||
cpuInput.placeholder = "e.g. 200%";
|
|
||||||
cpuInput.title =
|
|
||||||
'systemd CPUQuota= value (e.g. "400%"). empty = use hive default';
|
|
||||||
cpuLabel.append(cpuInput);
|
|
||||||
|
|
||||||
const memLabel = document.createElement("label");
|
|
||||||
memLabel.className = "cload-edit-label";
|
|
||||||
memLabel.textContent = "mem max";
|
|
||||||
const memInput = document.createElement("input");
|
|
||||||
memInput.type = "text";
|
|
||||||
memInput.className = "cload-mem-input";
|
|
||||||
// Same reasoning as cpuInput above.
|
|
||||||
memInput.value = cv ? cv.memory_max : "";
|
|
||||||
memInput.placeholder = "e.g. 8G";
|
|
||||||
memInput.title =
|
|
||||||
'systemd MemoryMax= value (e.g. "8G", "50%", "infinity"). empty = use hive default';
|
|
||||||
memLabel.append(memInput);
|
|
||||||
|
|
||||||
const submitBtn = document.createElement("button");
|
|
||||||
submitBtn.type = "submit";
|
|
||||||
submitBtn.className = "btn btn-restart cload-save-btn";
|
|
||||||
submitBtn.textContent = "S4V3";
|
|
||||||
|
|
||||||
const hintSpan = document.createElement("span");
|
|
||||||
hintSpan.className = "meta cload-edit-hint";
|
|
||||||
hintSpan.textContent = "↺ restart to apply to a running container";
|
|
||||||
|
|
||||||
const errSpan = document.createElement("span");
|
|
||||||
errSpan.className = "cload-edit-err";
|
|
||||||
errSpan.hidden = true;
|
|
||||||
|
|
||||||
editForm.append(cpuLabel, memLabel, submitBtn, hintSpan, errSpan);
|
|
||||||
editTd.append(editForm);
|
|
||||||
editRow.append(editTd);
|
|
||||||
tb.append(editRow);
|
|
||||||
|
|
||||||
setBtn.addEventListener("click", () => {
|
|
||||||
const open = !editRow.hidden;
|
|
||||||
editRow.hidden = open;
|
|
||||||
setBtn.textContent = open ? "S3T" : "✕";
|
|
||||||
});
|
|
||||||
}
|
|
||||||
table.append(tb);
|
|
||||||
root.replaceChildren(table);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function refreshContainerLoad() {
|
|
||||||
try {
|
|
||||||
const resp = await fetch("/api/container-resources");
|
|
||||||
if (!resp.ok) throw new Error("http " + resp.status);
|
|
||||||
renderContainerLoad(await resp.json());
|
|
||||||
} catch (e) {
|
|
||||||
const root = $("container-load-section");
|
|
||||||
if (root) {
|
|
||||||
root.replaceChildren();
|
|
||||||
const p = document.createElement("p");
|
|
||||||
p.className = "meta";
|
|
||||||
p.textContent = "container load fetch failed: " + e;
|
|
||||||
root.append(p);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function startContainerLoadPolling() {
|
|
||||||
refreshContainerLoad();
|
|
||||||
if (containerLoadTimer) return;
|
|
||||||
containerLoadTimer = setInterval(refreshContainerLoad, 5000);
|
|
||||||
}
|
|
||||||
function stopContainerLoadPolling() {
|
|
||||||
if (containerLoadTimer) {
|
|
||||||
clearInterval(containerLoadTimer);
|
|
||||||
containerLoadTimer = null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── render-all (cold load + any full re-render) ──────────────────────────
|
// ─── render-all (cold load + any full re-render) ──────────────────────────
|
||||||
function renderAll() {
|
function renderAll() {
|
||||||
renderTombstones({ tombstones: tombstonesState });
|
renderTombstones({ tombstones: tombstonesState });
|
||||||
|
|
@ -465,19 +191,6 @@ const SSE_HANDLERS = {
|
||||||
tombstonesState = (ev.tombstones || []).slice();
|
tombstonesState = (ev.tombstones || []).slice();
|
||||||
renderTombstones({ tombstones: tombstonesState });
|
renderTombstones({ tombstones: tombstonesState });
|
||||||
},
|
},
|
||||||
// When a container's state changes (e.g. after resource-limits update
|
|
||||||
// triggers rescan_containers_and_emit), update containersState in place
|
|
||||||
// so the cap columns in the LOAD table reflect the new configured values.
|
|
||||||
container_state_changed(ev) {
|
|
||||||
if (!ev.container) return;
|
|
||||||
const idx = containersState.findIndex((c) => c.name === ev.container.name);
|
|
||||||
if (idx >= 0) {
|
|
||||||
containersState[idx] = ev.container;
|
|
||||||
} else {
|
|
||||||
containersState.push(ev.container);
|
|
||||||
}
|
|
||||||
if (lastLoadRows.length) renderContainerLoad(lastLoadRows);
|
|
||||||
},
|
|
||||||
// Refresh the stale-perms sub-section when perm data changes (a ghost
|
// Refresh the stale-perms sub-section when perm data changes (a ghost
|
||||||
// was cleared, or perms were saved for an agent whose name collides).
|
// was cleared, or perms were saved for an agent whose name collides).
|
||||||
capabilities_changed(_ev) {
|
capabilities_changed(_ev) {
|
||||||
|
|
@ -511,33 +224,16 @@ async function init() {
|
||||||
// to the bare `ok` response page.
|
// to the bare `ok` response page.
|
||||||
bindAsyncForms(() => refreshState());
|
bindAsyncForms(() => refreshState());
|
||||||
|
|
||||||
// Hash-routed sub-tab strip; default K3PT ST4T3. Container-load
|
|
||||||
// polling runs only while the LOAD sub-tab is open (cpu is a short
|
|
||||||
// two-sample read each refresh on the server).
|
|
||||||
document.getElementById("core-tabbar").configure({
|
|
||||||
tabs: [
|
|
||||||
{ id: "kept", label: "K3PT ST4T3" },
|
|
||||||
{ id: "load", label: "C0NT41N3R L04D" },
|
|
||||||
],
|
|
||||||
defaultId: "kept",
|
|
||||||
onShow: (id) => {
|
|
||||||
if (id === "load") startContainerLoadPolling();
|
|
||||||
else stopContainerLoadPolling();
|
|
||||||
// Lazy-load stale-perms on first K3PT ST4T3 activation; always
|
|
||||||
// re-fetch on subsequent visits in case perms changed.
|
|
||||||
if (id === "kept") fetchAndRenderStalePerms();
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
await refreshState();
|
await refreshState();
|
||||||
|
fetchAndRenderStalePerms();
|
||||||
|
|
||||||
// kinds= matches SSE_HANDLERS below verbatim — narrows this from all
|
// kinds= matches SSE_HANDLERS below verbatim — narrows this from all
|
||||||
// 13 wire kinds down to the 4 this page acts on. This page was one
|
// 13 wire kinds down to the 3 this page acts on. This page was one
|
||||||
// of 4 unfiltered `/api/dashboard/stream` subscribers before this
|
// of 4 unfiltered `/api/dashboard/stream` subscribers before this
|
||||||
// (subscription discipline, part 1 of the dashboard-event-stream-
|
// (subscription discipline, part 1 of the dashboard-event-stream-
|
||||||
// split issue).
|
// split issue).
|
||||||
const es = openStream(
|
const es = openStream(
|
||||||
"/api/dashboard/stream?kinds=tombstones_changed,container_state_changed," +
|
"/api/dashboard/stream?kinds=tombstones_changed," +
|
||||||
"capabilities_changed,tool_groups_changed",
|
"capabilities_changed,tool_groups_changed",
|
||||||
);
|
);
|
||||||
if (es) {
|
if (es) {
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
it defines the CSS custom properties, typography, and shared
|
it defines the CSS custom properties, typography, and shared
|
||||||
component rules that this file depends on. */
|
component rules that this file depends on. */
|
||||||
|
|
||||||
/* SYST3M panel rules (rebuild queue / meta inputs / container load)
|
/* SYST3M panel rules (rebuild queue / meta inputs / kept state)
|
||||||
live in system-sections.css, shared with /core.html. A few are still
|
live in system-sections.css, shared with /core.html. A few are still
|
||||||
referenced from the dashboard (`.rebuild-queue`), so the dashboard
|
referenced from the dashboard (`.rebuild-queue`), so the dashboard
|
||||||
pulls them in here. */
|
pulls them in here. */
|
||||||
|
|
@ -771,6 +771,4 @@ body.dashboard-shell.has-selection {
|
||||||
|
|
||||||
/* ST4TS moved to its own page (`/stats.html`): the
|
/* ST4TS moved to its own page (`/stats.html`): the
|
||||||
window selector / summary chips / bars live in stats.css, and the
|
window selector / summary chips / bars live in stats.css, and the
|
||||||
shared `.hive-stats-table` moved to common.css (the SYST3M ›
|
shared `.hive-stats-table` moved to common.css. */
|
||||||
C0NT41N3R L04D table in system-sections.css / on /core.html still
|
|
||||||
uses it). */
|
|
||||||
|
|
|
||||||
|
|
@ -103,7 +103,7 @@
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<!-- SYST3M pane moved to /core.html (the standalone "C0R3" page):
|
<!-- SYST3M pane moved to /core.html (the standalone "C0R3" page):
|
||||||
meta inputs, rebuild queue, kept state, container load. Reached
|
kept state and stale permissions. Reached
|
||||||
from the H0M3 hub. -->
|
from the H0M3 hub. -->
|
||||||
|
|
||||||
<!-- P3RM1SS10NS: per-agent capability grants + tool-group
|
<!-- P3RM1SS10NS: per-agent capability grants + tool-group
|
||||||
|
|
|
||||||
|
|
@ -79,7 +79,7 @@
|
||||||
<span class="home-tile-icon" aria-hidden="true">⬡</span>
|
<span class="home-tile-icon" aria-hidden="true">⬡</span>
|
||||||
<span class="home-tile-label">Core</span>
|
<span class="home-tile-label">Core</span>
|
||||||
</span>
|
</span>
|
||||||
<span class="home-tile-desc">kept state · container load</span>
|
<span class="home-tile-desc">kept state · stale permissions</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
<!-- API tile: the OpenAPI spec + Swagger UI are always served by
|
<!-- API tile: the OpenAPI spec + Swagger UI are always served by
|
||||||
|
|
|
||||||
|
|
@ -2,8 +2,7 @@
|
||||||
// dashboard SPA: the live agent roster.
|
// dashboard SPA: the live agent roster.
|
||||||
//
|
//
|
||||||
// The roster is read by nearly every tab (the container tree, the
|
// The roster is read by nearly every tab (the container tree, the
|
||||||
// capabilities + tool-group matrices, the
|
// capabilities + tool-group matrices, and the selection bar), so
|
||||||
// container-load poll, and the selection bar), so
|
|
||||||
// it lives here as a single source of truth that each domain module
|
// it lives here as a single source of truth that each domain module
|
||||||
// imports rather than threading through call signatures. Everything
|
// imports rather than threading through call signatures. Everything
|
||||||
// else in the dashboard is domain-local and lives with its own module.
|
// else in the dashboard is domain-local and lives with its own module.
|
||||||
|
|
|
||||||
|
|
@ -3,8 +3,7 @@
|
||||||
chrome pattern as /flow.html: the back-link header +
|
chrome pattern as /flow.html: the back-link header +
|
||||||
title (.page-header / .page-back / .page-title) live in
|
title (.page-header / .page-back / .page-title) live in
|
||||||
@hive/shared/chrome.css.
|
@hive/shared/chrome.css.
|
||||||
`.hive-stats-table` also lives in common.css — it's shared with the
|
`.hive-stats-table` lives in common.css. Only the ST4TS-specific
|
||||||
dashboard SYST3M › C0NT41N3R L04D table. Only the ST4TS-specific
|
|
||||||
window selector, summary chips, and bars are page-specific. */
|
window selector, summary chips, and bars are page-specific. */
|
||||||
|
|
||||||
body.stats-shell {
|
body.stats-shell {
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
/* ─── SYST3M/BU1LDS section panels — shared by dashboard, /core.html,
|
/* ─── SYST3M/BU1LDS section panels — shared by dashboard, /core.html,
|
||||||
and /builds.html ──────────────────────────────────────────────────
|
and /builds.html ──────────────────────────────────────────────────
|
||||||
Rebuild queue, meta inputs, kept state, and container-load panels.
|
Rebuild queue, meta inputs and kept-state panels.
|
||||||
Imported by core.css, dashboard.css, and builds.css. */
|
Imported by core.css, dashboard.css, and builds.css. */
|
||||||
|
|
||||||
/* ─── meta inputs panel ────────────────────────────────────────────── */
|
/* ─── meta inputs panel ────────────────────────────────────────────── */
|
||||||
|
|
@ -198,83 +198,6 @@
|
||||||
color: var(--fg);
|
color: var(--fg);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ─── SYST3M › C0NT41N3R L04D: live cgroup cpu/mem ─────────────────────────────
|
|
||||||
Reuses the shared `.hive-stats-table` styling (in common.css); only
|
|
||||||
the inline meter bar is new. */
|
|
||||||
.cload-meter {
|
|
||||||
display: inline-block;
|
|
||||||
width: 6em;
|
|
||||||
height: 10px;
|
|
||||||
vertical-align: middle;
|
|
||||||
margin-left: 6px;
|
|
||||||
background: var(--bg-elev);
|
|
||||||
border: 1px solid var(--border);
|
|
||||||
border-radius: 3px;
|
|
||||||
overflow: hidden;
|
|
||||||
}
|
|
||||||
.cload-meter .fill {
|
|
||||||
display: block;
|
|
||||||
height: 100%;
|
|
||||||
background: var(--green);
|
|
||||||
}
|
|
||||||
.cload-meter .fill.warn {
|
|
||||||
background: var(--amber);
|
|
||||||
}
|
|
||||||
.cload-meter .fill.hot {
|
|
||||||
background: var(--red);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Configured-cap columns: muted header so they read apart from live metrics.
|
|
||||||
The S3T button sits in its own column; keep it compact. */
|
|
||||||
.cload-cap-th {
|
|
||||||
color: var(--muted);
|
|
||||||
}
|
|
||||||
.cload-cap {
|
|
||||||
color: var(--muted);
|
|
||||||
}
|
|
||||||
.cload-set-btn {
|
|
||||||
padding: 0.1em 0.5em;
|
|
||||||
font-size: 0.8em;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Inline edit row — spans all columns, contains a mini-form. */
|
|
||||||
.cload-edit-row td {
|
|
||||||
border-top: none;
|
|
||||||
padding-top: 0;
|
|
||||||
}
|
|
||||||
.cload-edit-cell {
|
|
||||||
padding: 0.2em 0.4em 0.6em 1.2em;
|
|
||||||
}
|
|
||||||
.cload-edit-form {
|
|
||||||
display: flex;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
align-items: center;
|
|
||||||
gap: 0.5em;
|
|
||||||
}
|
|
||||||
.cload-edit-label {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 0.3em;
|
|
||||||
font-size: 0.85em;
|
|
||||||
color: var(--muted);
|
|
||||||
}
|
|
||||||
.cload-cpu-input,
|
|
||||||
.cload-mem-input {
|
|
||||||
width: 7em;
|
|
||||||
font-size: 0.9em;
|
|
||||||
}
|
|
||||||
.cload-save-btn {
|
|
||||||
padding: 0.15em 0.6em;
|
|
||||||
font-size: 0.85em;
|
|
||||||
}
|
|
||||||
.cload-edit-hint {
|
|
||||||
font-size: 0.8em;
|
|
||||||
}
|
|
||||||
.cload-edit-err {
|
|
||||||
color: var(--red);
|
|
||||||
font-size: 0.85em;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Remove the now-dead `.rqe-step` rule (the `step` sub-step label was
|
/* Remove the now-dead `.rqe-step` rule (the `step` sub-step label was
|
||||||
retired from the wire in `feat(job_queue): retire the now-off-wire step
|
retired from the wire in `feat(job_queue): retire the now-off-wire step
|
||||||
sub-step label` — the rule has had no JS referents since then). */
|
sub-step label` — the rule has had no JS referents since then). */
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,8 @@
|
||||||
//! Per-agent CPU/memory limit overrides. Stored at
|
//! CPU/memory limits for an agent's container: the caps the swarm controller
|
||||||
//! `/var/lib/hyperhive/meta/resource-limits.json` alongside
|
//! declared, and the validators for the values the systemd drop-in takes.
|
||||||
|
//!
|
||||||
|
//! The swarm caps are stored at
|
||||||
|
//! `/var/lib/hyperhive/meta/swarm-resource-limits.json` alongside
|
||||||
//! `topology.json`, `tool-groups.json` and `capabilities.json`.
|
//! `topology.json`, `tool-groups.json` and `capabilities.json`.
|
||||||
//!
|
//!
|
||||||
//! Format: a JSON object mapping agent name to an object with optional
|
//! Format: a JSON object mapping agent name to an object with optional
|
||||||
|
|
@ -14,26 +17,22 @@
|
||||||
//!
|
//!
|
||||||
//! Fallback is **per field**: an absent file, an absent agent, or an
|
//! Fallback is **per field**: an absent file, an absent agent, or an
|
||||||
//! absent field all fall back to the hive-wide
|
//! absent field all fall back to the hive-wide
|
||||||
//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So an agent
|
//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So a swarm cap on
|
||||||
//! can raise only its memory cap and keep tracking the hive default for
|
//! memory alone leaves CPU on the hive default — see
|
||||||
//! CPU — see [`effective`].
|
//! [`effective_from`].
|
||||||
//!
|
//!
|
||||||
//! Read path: `lifecycle::host_config::write_dropins`, on every spawn
|
//! Read path: `lifecycle::host_config::write_dropins`, on every spawn
|
||||||
//! and every rebuild.
|
//! and every rebuild.
|
||||||
//!
|
//!
|
||||||
//! Why host-side JSON and not an option in the agent's own `agent.nix`:
|
//! Why host-side JSON and not an option in the agent's own `agent.nix`:
|
||||||
//! the drop-in lands on the *host's* `container@h-<name>.service`, so
|
//! the drop-in lands on the *host's* `container@h-<name>.service`, and a
|
||||||
//! c0re would have to `nix eval` the agent's whole nixosConfiguration
|
//! resource *cap* should not be sourced from the capped party.
|
||||||
//! just to read two strings. It is also the wrong trust boundary —
|
|
||||||
//! a resource *cap* should not be sourced from the capped party.
|
|
||||||
|
|
||||||
use std::collections::BTreeMap;
|
use std::collections::BTreeMap;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
const RESOURCE_LIMITS_FILE: &str = "resource-limits.json";
|
/// One agent's caps. Both fields optional and independent; `None` means
|
||||||
|
/// "use the hive-wide default for this field".
|
||||||
/// One agent's overrides. Both fields optional and independent; `None`
|
|
||||||
/// means "use the hive-wide default for this field".
|
|
||||||
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||||
pub struct AgentLimits {
|
pub struct AgentLimits {
|
||||||
/// systemd `CPUQuota=` value, e.g. `"400%"`.
|
/// systemd `CPUQuota=` value, e.g. `"400%"`.
|
||||||
|
|
@ -44,84 +43,6 @@ pub struct AgentLimits {
|
||||||
pub memory_max: Option<String>,
|
pub memory_max: Option<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AgentLimits {
|
|
||||||
/// True when neither field is set — such an entry is dropped rather
|
|
||||||
/// than persisted as an empty object.
|
|
||||||
#[must_use]
|
|
||||||
pub fn is_empty(&self) -> bool {
|
|
||||||
self.cpu_quota.is_none() && self.memory_max.is_none()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[must_use]
|
|
||||||
pub fn resource_limits_path() -> PathBuf {
|
|
||||||
crate::paths::meta_root().join(RESOURCE_LIMITS_FILE)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read the per-agent limit map. An absent file is the empty map —
|
|
||||||
/// callers treat a missing entry as "hive-wide defaults". A file that
|
|
||||||
/// exists but can't be read or parsed is an error: an override can be
|
|
||||||
/// tighter than the hive default, so reading it as empty is not a safe
|
|
||||||
/// fallback.
|
|
||||||
pub fn read() -> std::io::Result<BTreeMap<String, AgentLimits>> {
|
|
||||||
super::read_map(&resource_limits_path())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Resolve the effective values for an agent from the override file at
|
|
||||||
/// `path` (normally [`resource_limits_path`]), filling each unset field
|
|
||||||
/// from the hive-wide default. This is the single place the fallback
|
|
||||||
/// rule lives; `write_dropins` calls it and passes the result straight
|
|
||||||
/// to systemd. Errors as [`read`] does.
|
|
||||||
///
|
|
||||||
/// Use [`effective_from`] when resolving more than one agent in a row.
|
|
||||||
pub fn effective(
|
|
||||||
path: &Path,
|
|
||||||
name: &str,
|
|
||||||
hive_cpu_quota: &str,
|
|
||||||
hive_memory_max: &str,
|
|
||||||
) -> std::io::Result<(String, String)> {
|
|
||||||
let limits = super::read_map(path)?;
|
|
||||||
Ok(effective_from(
|
|
||||||
&limits,
|
|
||||||
name,
|
|
||||||
hive_cpu_quota,
|
|
||||||
hive_memory_max,
|
|
||||||
))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// [`effective`] against an already-loaded map — the multi-agent form.
|
|
||||||
///
|
|
||||||
/// `container_view::build_all` renders every agent on each SSE scan, so
|
|
||||||
/// it loads the map once and calls this per agent rather than re-reading
|
|
||||||
/// the same small file N times per scan.
|
|
||||||
#[must_use]
|
|
||||||
pub fn effective_from(
|
|
||||||
limits: &BTreeMap<String, AgentLimits>,
|
|
||||||
name: &str,
|
|
||||||
hive_cpu_quota: &str,
|
|
||||||
hive_memory_max: &str,
|
|
||||||
) -> (String, String) {
|
|
||||||
match limits.get(name) {
|
|
||||||
Some(l) => resolve(l, hive_cpu_quota, hive_memory_max),
|
|
||||||
None => (hive_cpu_quota.to_owned(), hive_memory_max.to_owned()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Pure core of [`effective`], split out so the fallback matrix is
|
|
||||||
/// testable without touching the filesystem.
|
|
||||||
#[must_use]
|
|
||||||
fn resolve(limits: &AgentLimits, hive_cpu_quota: &str, hive_memory_max: &str) -> (String, String) {
|
|
||||||
let cpu = limits
|
|
||||||
.cpu_quota
|
|
||||||
.clone()
|
|
||||||
.unwrap_or_else(|| hive_cpu_quota.to_owned());
|
|
||||||
let mem = limits
|
|
||||||
.memory_max
|
|
||||||
.clone()
|
|
||||||
.unwrap_or_else(|| hive_memory_max.to_owned());
|
|
||||||
(cpu, mem)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`,
|
/// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`,
|
||||||
/// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`)
|
/// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`)
|
||||||
/// into a raw byte count, accepting exactly the single-term forms systemd's
|
/// into a raw byte count, accepting exactly the single-term forms systemd's
|
||||||
|
|
@ -164,35 +85,7 @@ pub fn parse_bytes(value: &str) -> Option<u64> {
|
||||||
u64::try_from(bytes).ok()
|
u64::try_from(bytes).ok()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Set one agent's overrides and persist the map atomically. An entry
|
|
||||||
/// with both fields unset is removed rather than stored, so "reset to
|
|
||||||
/// hive defaults" and "never configured" are the same state on disk.
|
|
||||||
///
|
|
||||||
/// # Errors
|
|
||||||
///
|
|
||||||
/// Fails without writing when the existing file can't be read or parsed
|
|
||||||
/// (`InvalidData` for a parse failure), and otherwise when the meta dir
|
|
||||||
/// or the file can't be written.
|
|
||||||
pub fn set_limits(name: &str, limits: &AgentLimits) -> std::io::Result<()> {
|
|
||||||
set_limits_at(&resource_limits_path(), name, limits)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Result<()> {
|
|
||||||
let mut current: BTreeMap<String, AgentLimits> = super::read_map(path)?;
|
|
||||||
if limits.is_empty() {
|
|
||||||
current.remove(name);
|
|
||||||
} else {
|
|
||||||
current.insert(name.to_owned(), limits.clone());
|
|
||||||
}
|
|
||||||
super::write_map(path, ¤t)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Caps the swarm controller declared for this hive's agents.
|
/// Caps the swarm controller declared for this hive's agents.
|
||||||
///
|
|
||||||
/// A separate file from the operator's overrides above, not a merge into it:
|
|
||||||
/// the two have different writers and different lifetimes, and a swarm
|
|
||||||
/// declaration landing in the file an operator edits would make "reset to
|
|
||||||
/// hive defaults" destroy something the operator never set.
|
|
||||||
const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json";
|
const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json";
|
||||||
|
|
||||||
#[must_use]
|
#[must_use]
|
||||||
|
|
@ -201,9 +94,9 @@ pub fn swarm_limits_path() -> PathBuf {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Read the swarm-declared caps. An absent file is the empty map — no swarm
|
/// Read the swarm-declared caps. An absent file is the empty map — no swarm
|
||||||
/// opinion, so the hive's own chain decides. A file that exists but can't be
|
/// opinion, so the hive-wide defaults decide. A file that exists but can't be
|
||||||
/// read or parsed is an error, for the reason [`read`] gives: reading it as
|
/// read or parsed is an error: a cap can be tighter than the hive default, so
|
||||||
/// empty would silently drop a cap.
|
/// reading it as empty would silently drop it.
|
||||||
pub fn read_swarm() -> std::io::Result<BTreeMap<String, AgentLimits>> {
|
pub fn read_swarm() -> std::io::Result<BTreeMap<String, AgentLimits>> {
|
||||||
super::read_map(&swarm_limits_path())
|
super::read_map(&swarm_limits_path())
|
||||||
}
|
}
|
||||||
|
|
@ -216,55 +109,43 @@ pub fn write_swarm(map: &BTreeMap<String, AgentLimits>) -> std::io::Result<()> {
|
||||||
super::write_map(&swarm_limits_path(), map)
|
super::write_map(&swarm_limits_path(), map)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// [`effective_from`] with the swarm's declaration on top.
|
/// Resolve the effective values for an agent from the swarm caps in `swarm`:
|
||||||
|
/// each field the swarm declared a cap for, and the hive-wide default for
|
||||||
|
/// each it did not. This is the single place the fallback rule lives;
|
||||||
|
/// `write_dropins` calls it and passes the result straight to systemd.
|
||||||
///
|
///
|
||||||
/// A swarm cap **replaces** the hive's answer for that field rather than
|
/// Use this form when resolving more than one agent in a row.
|
||||||
/// joining the fallback chain as another tier: a tier a hive can out-set is
|
|
||||||
/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's
|
|
||||||
/// chain.
|
|
||||||
///
|
|
||||||
/// The chain itself is untouched — [`effective_from`] is called exactly as
|
|
||||||
/// before and its result is what a missing swarm value falls back to.
|
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn effective_from_with_swarm(
|
pub fn effective_from(
|
||||||
swarm: &BTreeMap<String, AgentLimits>,
|
swarm: &BTreeMap<String, AgentLimits>,
|
||||||
limits: &BTreeMap<String, AgentLimits>,
|
|
||||||
name: &str,
|
name: &str,
|
||||||
hive_cpu_quota: &str,
|
hive_cpu_quota: &str,
|
||||||
hive_memory_max: &str,
|
hive_memory_max: &str,
|
||||||
) -> (String, String) {
|
) -> (String, String) {
|
||||||
let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max);
|
let caps = swarm.get(name);
|
||||||
match swarm.get(name) {
|
(
|
||||||
Some(s) => (
|
caps.and_then(|c| c.cpu_quota.clone())
|
||||||
s.cpu_quota.clone().unwrap_or(cpu),
|
.unwrap_or_else(|| hive_cpu_quota.to_owned()),
|
||||||
s.memory_max.clone().unwrap_or(mem),
|
caps.and_then(|c| c.memory_max.clone())
|
||||||
),
|
.unwrap_or_else(|| hive_memory_max.to_owned()),
|
||||||
None => (cpu, mem),
|
)
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// [`effective_from_with_swarm`] for a caller resolving a single agent, from
|
/// [`effective_from`] for a caller resolving a single agent, from the swarm
|
||||||
/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`.
|
/// caps at `swarm_path`. Errors as [`read_swarm`] does.
|
||||||
///
|
pub fn effective(
|
||||||
/// Delegates to [`effective`] rather than re-deriving the hive's answer, so
|
|
||||||
/// there stays exactly one place the fallback chain lives. Errors as [`read`]
|
|
||||||
/// does, for either file.
|
|
||||||
pub fn effective_with_swarm(
|
|
||||||
limits_path: &Path,
|
|
||||||
swarm_path: &Path,
|
swarm_path: &Path,
|
||||||
name: &str,
|
name: &str,
|
||||||
hive_cpu_quota: &str,
|
hive_cpu_quota: &str,
|
||||||
hive_memory_max: &str,
|
hive_memory_max: &str,
|
||||||
) -> std::io::Result<(String, String)> {
|
) -> std::io::Result<(String, String)> {
|
||||||
let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?;
|
|
||||||
let swarm: BTreeMap<String, AgentLimits> = super::read_map(swarm_path)?;
|
let swarm: BTreeMap<String, AgentLimits> = super::read_map(swarm_path)?;
|
||||||
Ok(match swarm.get(name) {
|
Ok(effective_from(
|
||||||
Some(s) => (
|
&swarm,
|
||||||
s.cpu_quota.clone().unwrap_or(cpu),
|
name,
|
||||||
s.memory_max.clone().unwrap_or(mem),
|
hive_cpu_quota,
|
||||||
),
|
hive_memory_max,
|
||||||
None => (cpu, mem),
|
))
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Validate a systemd `CPUQuota=` value. Percentages only, and values
|
/// Validate a systemd `CPUQuota=` value. Percentages only, and values
|
||||||
|
|
@ -386,96 +267,66 @@ mod tests {
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn unset_agent_gets_hive_defaults() {
|
fn an_agent_the_swarm_says_nothing_about_gets_the_hive_defaults() {
|
||||||
let (cpu, mem) = resolve(&AgentLimits::default(), HIVE_CPU, HIVE_MEM);
|
let swarm = BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]);
|
||||||
assert_eq!(cpu, "200%");
|
for map in [BTreeMap::new(), swarm] {
|
||||||
assert_eq!(mem, "4G");
|
assert_eq!(
|
||||||
|
effective_from(&map, "sock", HIVE_CPU, HIVE_MEM),
|
||||||
|
(HIVE_CPU.to_owned(), HIVE_MEM.to_owned())
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn both_fields_override() {
|
fn a_swarm_cap_replaces_the_hive_default() {
|
||||||
let (cpu, mem) = resolve(&limits(Some("400%"), Some("8G")), HIVE_CPU, HIVE_MEM);
|
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
|
||||||
assert_eq!(cpu, "400%");
|
assert_eq!(
|
||||||
assert_eq!(mem, "8G");
|
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
|
||||||
|
("150%".to_owned(), "2G".to_owned())
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The point of per-field fallback: overriding memory must not drag
|
/// The point of per-field fallback: capping memory must not drag CPU
|
||||||
/// CPU along with it.
|
/// along with it.
|
||||||
#[test]
|
#[test]
|
||||||
fn partial_override_keeps_other_field_on_hive_default() {
|
fn a_swarm_cap_is_per_field() {
|
||||||
let (cpu, mem) = resolve(&limits(None, Some("8G")), HIVE_CPU, HIVE_MEM);
|
let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
|
||||||
assert_eq!(cpu, "200%", "cpu should still track the hive default");
|
assert_eq!(
|
||||||
assert_eq!(mem, "8G");
|
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
|
||||||
|
(HIVE_CPU.to_owned(), "2G".to_owned()),
|
||||||
let (cpu, mem) = resolve(&limits(Some("400%"), None), HIVE_CPU, HIVE_MEM);
|
"cpu should still track the hive default"
|
||||||
assert_eq!(cpu, "400%");
|
);
|
||||||
assert_eq!(mem, "4G", "memory should still track the hive default");
|
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), None))]);
|
||||||
}
|
assert_eq!(
|
||||||
|
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
|
||||||
#[test]
|
("150%".to_owned(), HIVE_MEM.to_owned()),
|
||||||
fn empty_entry_is_reported_empty() {
|
"memory should still track the hive default"
|
||||||
assert!(AgentLimits::default().is_empty());
|
);
|
||||||
assert!(!limits(None, Some("8G")).is_empty());
|
|
||||||
assert!(!limits(Some("400%"), None).is_empty());
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const TRUNCATED: &str =
|
const TRUNCATED: &str =
|
||||||
"{\n \"sock\": { \"cpu_quota\": \"50%\", \"memory_max\": \"1G\" },\n \"iris\": { \"mem";
|
"{\n \"sock\": { \"cpu_quota\": \"50%\", \"memory_max\": \"1G\" },\n \"iris\": { \"mem";
|
||||||
|
|
||||||
fn corrupt_file() -> (tempfile::TempDir, PathBuf) {
|
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
|
||||||
let path = dir.path().join(RESOURCE_LIMITS_FILE);
|
|
||||||
std::fs::write(&path, TRUNCATED).expect("seed");
|
|
||||||
(dir, path)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn effective_of_a_corrupt_file_is_an_error() {
|
fn effective_of_a_corrupt_file_is_an_error() {
|
||||||
let (_dir, path) = corrupt_file();
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
let path = dir.path().join(SWARM_LIMITS_FILE);
|
||||||
|
std::fs::write(&path, TRUNCATED).expect("seed");
|
||||||
let err = effective(&path, "sock", HIVE_CPU, HIVE_MEM)
|
let err = effective(&path, "sock", HIVE_CPU, HIVE_MEM)
|
||||||
.expect_err("a corrupt file must not resolve to hive defaults");
|
.expect_err("a corrupt file must not resolve to hive defaults");
|
||||||
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn set_limits_leaves_a_corrupt_file_untouched() {
|
|
||||||
let (_dir, path) = corrupt_file();
|
|
||||||
let err = set_limits_at(&path, "ruth", &limits(Some("400%"), None))
|
|
||||||
.expect_err("a corrupt file must not be overwritten");
|
|
||||||
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
|
||||||
assert_eq!(std::fs::read(&path).expect("read"), TRUNCATED.as_bytes());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn missing_file_resolves_to_hive_defaults() {
|
fn missing_file_resolves_to_hive_defaults() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let path = dir.path().join(RESOURCE_LIMITS_FILE);
|
let path = dir.path().join(SWARM_LIMITS_FILE);
|
||||||
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("missing file");
|
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("missing file");
|
||||||
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, HIVE_MEM));
|
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, HIVE_MEM));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn set_limits_keeps_other_agents_and_leaves_no_temp_file() {
|
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
|
||||||
let path = dir.path().join(RESOURCE_LIMITS_FILE);
|
|
||||||
set_limits_at(&path, "sock", &limits(None, Some("8G"))).expect("set on missing file");
|
|
||||||
set_limits_at(&path, "iris", &limits(Some("50%"), None)).expect("set");
|
|
||||||
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("read");
|
|
||||||
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, "8G"));
|
|
||||||
let (cpu, _) = effective(&path, "iris", HIVE_CPU, HIVE_MEM).expect("read");
|
|
||||||
assert_eq!(cpu, "50%");
|
|
||||||
let entries: Vec<_> = std::fs::read_dir(dir.path())
|
|
||||||
.expect("read_dir")
|
|
||||||
.map(|e| e.expect("entry").file_name())
|
|
||||||
.collect();
|
|
||||||
assert_eq!(
|
|
||||||
entries,
|
|
||||||
vec![std::ffi::OsString::from(RESOURCE_LIMITS_FILE)]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Absent fields must deserialize to `None`, not fail — an entry
|
/// Absent fields must deserialize to `None`, not fail — an entry
|
||||||
/// written by an older version with only one field must still load.
|
/// with only one field must still load.
|
||||||
#[test]
|
#[test]
|
||||||
fn partial_entry_deserializes() {
|
fn partial_entry_deserializes() {
|
||||||
let map: BTreeMap<String, AgentLimits> =
|
let map: BTreeMap<String, AgentLimits> =
|
||||||
|
|
@ -605,52 +456,4 @@ mod tests {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() {
|
|
||||||
// The whole point of the ruling: a tier a hive can out-set is not a
|
|
||||||
// cap. The agent override below is the strongest hive-side value
|
|
||||||
// there is, and the swarm still wins.
|
|
||||||
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]);
|
|
||||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
|
|
||||||
assert_eq!(
|
|
||||||
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
|
||||||
("150%".to_owned(), "2G".to_owned())
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_swarm_cap_is_per_field_like_every_other_tier() {
|
|
||||||
// A swarm that caps memory and says nothing about CPU leaves CPU on
|
|
||||||
// the hive's chain — here the per-agent override, not the hive default.
|
|
||||||
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]);
|
|
||||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
|
|
||||||
assert_eq!(
|
|
||||||
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
|
||||||
("800%".to_owned(), "2G".to_owned())
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() {
|
|
||||||
// The control that makes the two tests above mean something: the
|
|
||||||
// wrapper must be a no-op when the swarm has said nothing, for every
|
|
||||||
// shape the old chain distinguishes. If this drifts, the swarm tier
|
|
||||||
// changed behaviour it was explicitly told not to change.
|
|
||||||
let none = BTreeMap::new();
|
|
||||||
let cases = [
|
|
||||||
BTreeMap::new(),
|
|
||||||
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]),
|
|
||||||
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]),
|
|
||||||
BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]),
|
|
||||||
BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]),
|
|
||||||
];
|
|
||||||
for hive_side in &cases {
|
|
||||||
assert_eq!(
|
|
||||||
effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
|
||||||
effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
|
||||||
"the wrapper changed the answer with no swarm value present"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -76,16 +76,6 @@ pub struct ContainerView {
|
||||||
/// a legitimate way to keep it idle when it next boots.
|
/// a legitimate way to keep it idle when it next boots.
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub paused: bool,
|
pub paused: bool,
|
||||||
/// Effective systemd `CPUQuota=` for this container (e.g. `"400%"`) —
|
|
||||||
/// the per-agent override from `meta/resource-limits.json` when set,
|
|
||||||
/// otherwise the hive-wide `agentCpuQuota`. Always populated: there
|
|
||||||
/// is no "unset" state to render, only "same as everyone else".
|
|
||||||
/// Reflects what the *drop-in says*, which is what the next start
|
|
||||||
/// will enforce — not a live cgroup reading.
|
|
||||||
pub cpu_quota: String,
|
|
||||||
/// Effective systemd `MemoryMax=` for this container (e.g. `"8G"`).
|
|
||||||
/// Same resolution + caveat as [`ContainerView::cpu_quota`].
|
|
||||||
pub memory_max: String,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
|
impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
|
||||||
|
|
@ -99,10 +89,9 @@ impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
|
||||||
/// anymore. Stubbed to `0` rather than dropping the wire field
|
/// anymore. Stubbed to `0` rather than dropping the wire field
|
||||||
/// outright — leaves `hivectl status`/the dashboard column intact
|
/// outright — leaves `hivectl status`/the dashboard column intact
|
||||||
/// syntactically, just always empty, until iris's frontend follow-up
|
/// syntactically, just always empty, until iris's frontend follow-up
|
||||||
/// decides whether to drop the column entirely. `port`, `cpu_quota`,
|
/// decides whether to drop the column entirely. `port` has no
|
||||||
/// `memory_max` have no equivalent on the row at all and are simply
|
/// equivalent on the row at all and is simply not carried over. `url`
|
||||||
/// not carried over. `url` has no source on `ContainerView` at all —
|
/// has no source on `ContainerView` at all — see [`agent_url`].
|
||||||
/// see [`agent_url`].
|
|
||||||
fn from(v: ContainerView) -> Self {
|
fn from(v: ContainerView) -> Self {
|
||||||
Self {
|
Self {
|
||||||
url: agent_url(&v.name),
|
url: agent_url(&v.name),
|
||||||
|
|
@ -148,37 +137,12 @@ fn agent_url_for_domain(domain: &str, name: &str) -> String {
|
||||||
/// Build the full container list. Wraps `lifecycle::list()` and
|
/// Build the full container list. Wraps `lifecycle::list()` and
|
||||||
/// resolves every per-agent attribute the dashboard surfaces.
|
/// resolves every per-agent attribute the dashboard surfaces.
|
||||||
///
|
///
|
||||||
/// Takes `hive` because the effective resource limits are a per-field
|
|
||||||
/// fallback onto the hive-wide `agent_cpu_quota` / `agent_memory_max`,
|
|
||||||
/// and those live on [`crate::coordinator::HiveEnv`], not on disk. Both callers already
|
|
||||||
/// hold a `Coordinator`, so this is a parameter rather than a global.
|
|
||||||
///
|
|
||||||
/// # Errors
|
/// # Errors
|
||||||
///
|
///
|
||||||
/// The container list could not be read.
|
/// The container list could not be read.
|
||||||
pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec<ContainerView>> {
|
pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
|
||||||
let raw = lifecycle::list().await?;
|
let raw = lifecycle::list().await?;
|
||||||
let locked = read_meta_locked_revs();
|
let locked = read_meta_locked_revs();
|
||||||
// Read once per scan rather than re-read for every container on every
|
|
||||||
// SSE scan. An unreadable file shows every row at the hive defaults;
|
|
||||||
// the error log is the only signal, as `ContainerView` has no field
|
|
||||||
// to carry it.
|
|
||||||
let limits = crate::resource_limits::read().unwrap_or_else(|e| {
|
|
||||||
tracing::error!(
|
|
||||||
path = %crate::resource_limits::resource_limits_path().display(),
|
|
||||||
error = ?e,
|
|
||||||
"resource limits unreadable — container rows show hive defaults"
|
|
||||||
);
|
|
||||||
std::collections::BTreeMap::new()
|
|
||||||
});
|
|
||||||
let swarm_limits = crate::resource_limits::read_swarm().unwrap_or_else(|e| {
|
|
||||||
tracing::error!(
|
|
||||||
path = %crate::resource_limits::swarm_limits_path().display(),
|
|
||||||
error = ?e,
|
|
||||||
"swarm resource caps unreadable — container rows ignore them"
|
|
||||||
);
|
|
||||||
std::collections::BTreeMap::new()
|
|
||||||
});
|
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
for c in &raw {
|
for c in &raw {
|
||||||
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
|
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
|
||||||
|
|
@ -243,13 +207,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
|
||||||
(None, None)
|
(None, None)
|
||||||
};
|
};
|
||||||
let paused = Coordinator::is_paused(&logical);
|
let paused = Coordinator::is_paused(&logical);
|
||||||
let (cpu_quota, memory_max) = crate::resource_limits::effective_from_with_swarm(
|
|
||||||
&swarm_limits,
|
|
||||||
&limits,
|
|
||||||
logical.as_str(),
|
|
||||||
&hive.agent_cpu_quota,
|
|
||||||
&hive.agent_memory_max,
|
|
||||||
);
|
|
||||||
out.push(ContainerView {
|
out.push(ContainerView {
|
||||||
port: lifecycle::agent_web_port(logical.as_str()),
|
port: lifecycle::agent_web_port(logical.as_str()),
|
||||||
running,
|
running,
|
||||||
|
|
@ -263,8 +220,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
|
||||||
status_text,
|
status_text,
|
||||||
status_set_at,
|
status_set_at,
|
||||||
paused,
|
paused,
|
||||||
cpu_quota,
|
|
||||||
memory_max,
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Ok(out)
|
Ok(out)
|
||||||
|
|
@ -570,9 +525,8 @@ mod tests {
|
||||||
|
|
||||||
/// The `From` impl both `handle_agent_status`'s poll path and
|
/// The `From` impl both `handle_agent_status`'s poll path and
|
||||||
/// `stream_agent_status`'s push path go through — pins the field
|
/// `stream_agent_status`'s push path go through — pins the field
|
||||||
/// mapping so the two can't quietly diverge, and that `port` /
|
/// mapping so the two can't quietly diverge, and that `port` (no
|
||||||
/// `cpu_quota` / `memory_max` (no equivalent on the row) are dropped
|
/// equivalent on the row) is dropped on purpose rather than by omission.
|
||||||
/// on purpose rather than by omission.
|
|
||||||
#[test]
|
#[test]
|
||||||
fn agent_status_row_carries_every_field_the_row_has_and_drops_the_rest() {
|
fn agent_status_row_carries_every_field_the_row_has_and_drops_the_rest() {
|
||||||
use super::ContainerView;
|
use super::ContainerView;
|
||||||
|
|
@ -592,8 +546,6 @@ mod tests {
|
||||||
status_text: Some("shipping".to_owned()),
|
status_text: Some("shipping".to_owned()),
|
||||||
status_set_at: Some(Utc.timestamp_opt(1_700_000_000, 0).unwrap()),
|
status_set_at: Some(Utc.timestamp_opt(1_700_000_000, 0).unwrap()),
|
||||||
paused: true,
|
paused: true,
|
||||||
cpu_quota: "400%".to_owned(),
|
|
||||||
memory_max: "8G".to_owned(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let row = AgentStatusRow::from(view);
|
let row = AgentStatusRow::from(view);
|
||||||
|
|
|
||||||
|
|
@ -720,7 +720,7 @@ impl Coordinator {
|
||||||
pub async fn rescan_containers_and_emit(self: &Arc<Self>) {
|
pub async fn rescan_containers_and_emit(self: &Arc<Self>) {
|
||||||
// An unreadable list says nothing about which containers exist, so
|
// An unreadable list says nothing about which containers exist, so
|
||||||
// the last snapshot stands rather than every row being reported gone.
|
// the last snapshot stands rather than every row being reported gone.
|
||||||
let fresh = match container_view::build_all(&self.hive_env()).await {
|
let fresh = match container_view::build_all().await {
|
||||||
Ok(fresh) => fresh,
|
Ok(fresh) => fresh,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::warn!(error = ?e, "container rescan: list failed; keeping last snapshot");
|
tracing::warn!(error = ?e, "container rescan: list failed; keeping last snapshot");
|
||||||
|
|
|
||||||
|
|
@ -311,95 +311,6 @@ fn set_paused_error_response(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Form fields for `post_resource_limits`. Both fields are optional strings;
|
|
||||||
/// an empty value clears the per-agent override for that field, falling back
|
|
||||||
/// to the hive-wide default.
|
|
||||||
#[derive(Deserialize, Default, ToSchema)]
|
|
||||||
pub(super) struct ResourceLimitsForm {
|
|
||||||
#[serde(default)]
|
|
||||||
cpu_quota: String,
|
|
||||||
#[serde(default)]
|
|
||||||
memory_max: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Write per-agent CPU/memory limit
|
|
||||||
/// overrides for `name`.
|
|
||||||
///
|
|
||||||
/// An empty `cpu_quota` or `memory_max` field clears that field's override,
|
|
||||||
/// falling back to the hive-wide default. Both empty together removes the
|
|
||||||
/// agent's entry entirely. The new drop-in is written immediately — the
|
|
||||||
/// limits take effect on the next container start or restart. Triggers an
|
|
||||||
/// immediate rescan so `ContainerView.cpu_quota`/`memory_max` update on
|
|
||||||
/// the dashboard via SSE without waiting for the next periodic sweep.
|
|
||||||
#[utoipa::path(
|
|
||||||
post,
|
|
||||||
path = "/api/resource-limits/{name}",
|
|
||||||
params(("name" = String, Path, description = "agent name")),
|
|
||||||
request_body(content = ResourceLimitsForm, content_type = "application/x-www-form-urlencoded"),
|
|
||||||
responses(
|
|
||||||
(status = 200, description = "limits written", body = String),
|
|
||||||
(status = 400, description = "bad agent name"),
|
|
||||||
(status = 404, description = "no such agent"),
|
|
||||||
(status = 422, description = "invalid cpu_quota/memory_max value"),
|
|
||||||
(status = 500, description = "commit or drop-in write failed"),
|
|
||||||
),
|
|
||||||
tag = "lifecycle_ops"
|
|
||||||
)]
|
|
||||||
pub(super) async fn post_resource_limits(
|
|
||||||
State(state): State<AppState>,
|
|
||||||
AxumPath(name): AxumPath<String>,
|
|
||||||
Form(form): Form<ResourceLimitsForm>,
|
|
||||||
) -> Response {
|
|
||||||
let logical = strip_container_prefix(&name);
|
|
||||||
if let Some(reject) = guard_agent_name(&state, &logical).await {
|
|
||||||
return reject;
|
|
||||||
}
|
|
||||||
let ident = match Ident::parse(&logical) {
|
|
||||||
Ok(i) => i,
|
|
||||||
Err(e) => return (StatusCode::BAD_REQUEST, format!("bad agent name: {e}")).into_response(),
|
|
||||||
};
|
|
||||||
let cpu_quota = if form.cpu_quota.is_empty() {
|
|
||||||
None
|
|
||||||
} else {
|
|
||||||
Some(form.cpu_quota.as_str())
|
|
||||||
};
|
|
||||||
let memory_max = if form.memory_max.is_empty() {
|
|
||||||
None
|
|
||||||
} else {
|
|
||||||
Some(form.memory_max.as_str())
|
|
||||||
};
|
|
||||||
if let Some(v) = cpu_quota
|
|
||||||
&& let Err(e) = crate::resource_limits::validate_cpu_quota(v)
|
|
||||||
{
|
|
||||||
return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response();
|
|
||||||
}
|
|
||||||
// Returns the value to store, not just an ok/err verdict — see
|
|
||||||
// `validate_memory_max`'s own doc for why that isn't always `v`.
|
|
||||||
let memory_max = match memory_max.map(crate::resource_limits::validate_memory_max) {
|
|
||||||
Some(Err(e)) => return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response(),
|
|
||||||
Some(Ok(v)) => Some(v),
|
|
||||||
None => None,
|
|
||||||
};
|
|
||||||
let limits = crate::resource_limits::AgentLimits {
|
|
||||||
cpu_quota: cpu_quota.map(str::to_owned),
|
|
||||||
memory_max,
|
|
||||||
};
|
|
||||||
if let Err(e) = crate::meta::commit_resource_limits(ident.as_str(), &limits).await {
|
|
||||||
return error_response(&format!("set limits {logical}: {e:#}"));
|
|
||||||
}
|
|
||||||
let agent_dir = crate::paths::agent_runtime_dir(ident.as_str());
|
|
||||||
let hive = state.coord.hive_env();
|
|
||||||
let paths = match crate::coordinator::Coordinator::agent_paths(ident.as_str(), agent_dir) {
|
|
||||||
Ok(p) => p,
|
|
||||||
Err(e) => return error_response(&format!("write_dropins {logical}: {e:#}")),
|
|
||||||
};
|
|
||||||
if let Err(e) = crate::lifecycle::write_dropins(ident.as_str(), &hive, &paths).await {
|
|
||||||
return error_response(&format!("write_dropins {logical}: {e:#}"));
|
|
||||||
}
|
|
||||||
state.coord.rescan_containers_and_emit().await;
|
|
||||||
(StatusCode::OK, "ok").into_response()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Queue a rebuild DAG for every live agent
|
/// Queue a rebuild DAG for every live agent
|
||||||
/// container.
|
/// container.
|
||||||
#[utoipa::path(
|
#[utoipa::path(
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,5 @@
|
||||||
//! Remaining single-endpoint dashboard handlers: mark-all-read, operator
|
//! Remaining single-endpoint dashboard handlers: mark-all-read, operator
|
||||||
//! compose (`op-send`), spawn-request, hive-wide turn stats, and container
|
//! compose (`op-send`), spawn-request and hive-wide turn stats.
|
||||||
//! resources.
|
|
||||||
|
|
||||||
use axum::{
|
use axum::{
|
||||||
extract::{Form, Path as AxumPath, State},
|
extract::{Form, Path as AxumPath, State},
|
||||||
|
|
@ -11,7 +10,6 @@ use serde::{Deserialize, Serialize};
|
||||||
use utoipa::{IntoParams, ToSchema};
|
use utoipa::{IntoParams, ToSchema};
|
||||||
|
|
||||||
use super::{AppState, Ident, error_response};
|
use super::{AppState, Ident, error_response};
|
||||||
use crate::container_stats::ContainerResource;
|
|
||||||
use crate::hive_stats::HiveStats;
|
use crate::hive_stats::HiveStats;
|
||||||
|
|
||||||
#[derive(Deserialize, IntoParams)]
|
#[derive(Deserialize, IntoParams)]
|
||||||
|
|
@ -43,20 +41,6 @@ pub(super) async fn api_stats_hive(
|
||||||
.into_response()
|
.into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Live per-agent-container CPU + memory load from cgroup v2.
|
|
||||||
///
|
|
||||||
/// Samples CPU over a short interval (~200 ms), so this call briefly
|
|
||||||
/// awaits.
|
|
||||||
#[utoipa::path(
|
|
||||||
get,
|
|
||||||
path = "/api/container-resources",
|
|
||||||
responses((status = 200, description = "live per-container CPU + memory load", body = Vec<ContainerResource>)),
|
|
||||||
tag = "misc_api"
|
|
||||||
)]
|
|
||||||
pub(super) async fn api_container_resources() -> Response {
|
|
||||||
axum::Json(crate::container_stats::gather().await).into_response()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Serialize, ToSchema)]
|
#[derive(Serialize, ToSchema)]
|
||||||
pub(super) struct MarkAllReadBody {
|
pub(super) struct MarkAllReadBody {
|
||||||
marked: u64,
|
marked: u64,
|
||||||
|
|
|
||||||
|
|
@ -116,7 +116,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
|
||||||
.routes(routes!(state_snapshot::api_state))
|
.routes(routes!(state_snapshot::api_state))
|
||||||
.routes(routes!(state_files::get_state_file))
|
.routes(routes!(state_files::get_state_file))
|
||||||
.routes(routes!(misc_api::api_stats_hive))
|
.routes(routes!(misc_api::api_stats_hive))
|
||||||
.routes(routes!(misc_api::api_container_resources))
|
|
||||||
.routes(routes!(misc_api::post_mark_all_read))
|
.routes(routes!(misc_api::post_mark_all_read))
|
||||||
.routes(routes!(misc_api::post_op_send))
|
.routes(routes!(misc_api::post_op_send))
|
||||||
.routes(routes!(build_logs::get_build_logs_all))
|
.routes(routes!(build_logs::get_build_logs_all))
|
||||||
|
|
@ -140,7 +139,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
|
||||||
.routes(routes!(lifecycle_ops::post_rebuild))
|
.routes(routes!(lifecycle_ops::post_rebuild))
|
||||||
.routes(routes!(lifecycle_ops::post_pause))
|
.routes(routes!(lifecycle_ops::post_pause))
|
||||||
.routes(routes!(lifecycle_ops::post_resume))
|
.routes(routes!(lifecycle_ops::post_resume))
|
||||||
.routes(routes!(lifecycle_ops::post_resource_limits))
|
|
||||||
.routes(routes!(lifecycle_ops::post_update_all))
|
.routes(routes!(lifecycle_ops::post_update_all))
|
||||||
.routes(routes!(tombstones::post_purge_tombstone))
|
.routes(routes!(tombstones::post_purge_tombstone))
|
||||||
.routes(routes!(meta_inputs::post_meta_update))
|
.routes(routes!(meta_inputs::post_meta_update))
|
||||||
|
|
@ -372,7 +370,6 @@ mod router_build_probe {
|
||||||
.routes(routes!(state_snapshot::api_state))
|
.routes(routes!(state_snapshot::api_state))
|
||||||
.routes(routes!(state_files::get_state_file))
|
.routes(routes!(state_files::get_state_file))
|
||||||
.routes(routes!(misc_api::api_stats_hive))
|
.routes(routes!(misc_api::api_stats_hive))
|
||||||
.routes(routes!(misc_api::api_container_resources))
|
|
||||||
.routes(routes!(misc_api::post_mark_all_read))
|
.routes(routes!(misc_api::post_mark_all_read))
|
||||||
.routes(routes!(misc_api::post_op_send))
|
.routes(routes!(misc_api::post_op_send))
|
||||||
.routes(routes!(build_logs::get_build_logs_all))
|
.routes(routes!(build_logs::get_build_logs_all))
|
||||||
|
|
@ -396,7 +393,6 @@ mod router_build_probe {
|
||||||
.routes(routes!(lifecycle_ops::post_rebuild))
|
.routes(routes!(lifecycle_ops::post_rebuild))
|
||||||
.routes(routes!(lifecycle_ops::post_pause))
|
.routes(routes!(lifecycle_ops::post_pause))
|
||||||
.routes(routes!(lifecycle_ops::post_resume))
|
.routes(routes!(lifecycle_ops::post_resume))
|
||||||
.routes(routes!(lifecycle_ops::post_resource_limits))
|
|
||||||
.routes(routes!(lifecycle_ops::post_update_all))
|
.routes(routes!(lifecycle_ops::post_update_all))
|
||||||
.routes(routes!(tombstones::post_purge_tombstone))
|
.routes(routes!(tombstones::post_purge_tombstone))
|
||||||
.routes(routes!(meta_inputs::post_meta_update))
|
.routes(routes!(meta_inputs::post_meta_update))
|
||||||
|
|
|
||||||
|
|
@ -603,8 +603,6 @@ mod tests {
|
||||||
status_text: None,
|
status_text: None,
|
||||||
status_set_at: None,
|
status_set_at: None,
|
||||||
paused: false,
|
paused: false,
|
||||||
cpu_quota: "200%".to_owned(),
|
|
||||||
memory_max: "4G".to_owned(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,6 @@ pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Re
|
||||||
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
|
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
|
||||||
if let Some((cpu_quota, memory_max)) = stage_limits(
|
if let Some((cpu_quota, memory_max)) = stage_limits(
|
||||||
name,
|
name,
|
||||||
&crate::resource_limits::resource_limits_path(),
|
|
||||||
&crate::resource_limits::swarm_limits_path(),
|
&crate::resource_limits::swarm_limits_path(),
|
||||||
&limits_dropin_path(&container),
|
&limits_dropin_path(&container),
|
||||||
&paths.agent,
|
&paths.agent,
|
||||||
|
|
@ -97,14 +96,13 @@ impl HostMemory {
|
||||||
/// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's.
|
/// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's.
|
||||||
fn stage_limits(
|
fn stage_limits(
|
||||||
agent_name: &str,
|
agent_name: &str,
|
||||||
limits_path: &Path,
|
|
||||||
swarm_path: &Path,
|
swarm_path: &Path,
|
||||||
dropin_path: &Path,
|
dropin_path: &Path,
|
||||||
runtime_dir: &Path,
|
runtime_dir: &Path,
|
||||||
hive: &HiveEnv,
|
hive: &HiveEnv,
|
||||||
host: Option<HostMemory>,
|
host: Option<HostMemory>,
|
||||||
) -> Result<Option<(String, String)>> {
|
) -> Result<Option<(String, String)>> {
|
||||||
let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive);
|
let limits = limits_to_apply(agent_name, swarm_path, dropin_path, hive);
|
||||||
let memory_max = match &limits {
|
let memory_max = match &limits {
|
||||||
Some((_, memory_max)) => Some(memory_max.clone()),
|
Some((_, memory_max)) => Some(memory_max.clone()),
|
||||||
None => dropin_memory_max(agent_name, dropin_path),
|
None => dropin_memory_max(agent_name, dropin_path),
|
||||||
|
|
@ -226,35 +224,27 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
|
||||||
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
|
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
|
||||||
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
|
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
|
||||||
///
|
///
|
||||||
/// When the overrides file at `limits_path` or the swarm caps at
|
/// When the swarm caps at `swarm_path` can't be read, the error is logged and
|
||||||
/// `swarm_path` can't be read, the error is logged and the agent still starts.
|
/// the agent still starts. The last-applied drop-in is kept, because the hive
|
||||||
/// The last-applied drop-in is kept, because the hive defaults can be looser
|
/// defaults can be looser than the swarm's cap.
|
||||||
/// than the agent's own override.
|
|
||||||
/// With no drop-in yet, the agent gets the hive defaults: without one it
|
/// With no drop-in yet, the agent gets the hive defaults: without one it
|
||||||
/// would run uncapped.
|
/// would run uncapped.
|
||||||
fn limits_to_apply(
|
fn limits_to_apply(
|
||||||
agent_name: &str,
|
agent_name: &str,
|
||||||
limits_path: &Path,
|
|
||||||
swarm_path: &Path,
|
swarm_path: &Path,
|
||||||
dropin_path: &Path,
|
dropin_path: &Path,
|
||||||
hive: &HiveEnv,
|
hive: &HiveEnv,
|
||||||
) -> Option<(String, String)> {
|
) -> Option<(String, String)> {
|
||||||
let hive_cpu = &hive.agent_cpu_quota;
|
let hive_cpu = &hive.agent_cpu_quota;
|
||||||
let hive_mem = &hive.agent_memory_max;
|
let hive_mem = &hive.agent_memory_max;
|
||||||
let e = match crate::resource_limits::effective_with_swarm(
|
let e = match crate::resource_limits::effective(swarm_path, agent_name, hive_cpu, hive_mem) {
|
||||||
limits_path,
|
|
||||||
swarm_path,
|
|
||||||
agent_name,
|
|
||||||
hive_cpu,
|
|
||||||
hive_mem,
|
|
||||||
) {
|
|
||||||
Ok(limits) => return Some(limits),
|
Ok(limits) => return Some(limits),
|
||||||
Err(e) => e,
|
Err(e) => e,
|
||||||
};
|
};
|
||||||
if dropin_path.exists() {
|
if dropin_path.exists() {
|
||||||
tracing::error!(
|
tracing::error!(
|
||||||
agent = %agent_name,
|
agent = %agent_name,
|
||||||
path = %limits_path.display(),
|
path = %swarm_path.display(),
|
||||||
error = ?e,
|
error = ?e,
|
||||||
"resource limits unreadable — keeping the last-applied limits drop-in"
|
"resource limits unreadable — keeping the last-applied limits drop-in"
|
||||||
);
|
);
|
||||||
|
|
@ -262,7 +252,7 @@ fn limits_to_apply(
|
||||||
} else {
|
} else {
|
||||||
tracing::error!(
|
tracing::error!(
|
||||||
agent = %agent_name,
|
agent = %agent_name,
|
||||||
path = %limits_path.display(),
|
path = %swarm_path.display(),
|
||||||
error = ?e,
|
error = ?e,
|
||||||
"resource limits unreadable and no limits drop-in yet — applying hive defaults"
|
"resource limits unreadable and no limits drop-in yet — applying hive defaults"
|
||||||
);
|
);
|
||||||
|
|
@ -271,14 +261,14 @@ fn limits_to_apply(
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Write a systemd drop-in for `container@<container>.service` that applies
|
/// Write a systemd drop-in for `container@<container>.service` that applies
|
||||||
/// the agent's effective resource caps — its per-agent overrides from
|
/// the agent's effective resource caps — the swarm's cap from
|
||||||
/// `meta/resource-limits.json` where set, the hive-wide defaults
|
/// `meta/swarm-resource-limits.json` where set, the hive-wide defaults
|
||||||
/// otherwise. Goes under `/run/systemd/system/...` so it's ephemeral
|
/// otherwise. Goes under `/run/systemd/system/...` so it's ephemeral
|
||||||
/// (regenerated on every spawn / rebuild, except as [`limits_to_apply`] says).
|
/// (regenerated on every spawn / rebuild, except as [`limits_to_apply`] says).
|
||||||
///
|
///
|
||||||
/// The weights are hive-wide (`services.hyperhive.agentCpuWeight` /
|
/// The weights are hive-wide (`services.hyperhive.agentCpuWeight` /
|
||||||
/// `agentIoWeight`) — unlike the caps they have no per-agent override in
|
/// `agentIoWeight`) — unlike the caps they have no swarm cap, so they come
|
||||||
/// `meta/resource-limits.json`, so they come straight off [`HiveEnv`].
|
/// straight off [`HiveEnv`].
|
||||||
/// `None` (the nix option set to `null`) means the weight line is left out
|
/// `None` (the nix option set to `null`) means the weight line is left out
|
||||||
/// and the container keeps the kernel default.
|
/// and the container keeps the kernel default.
|
||||||
async fn set_resource_limits(
|
async fn set_resource_limits(
|
||||||
|
|
@ -848,42 +838,35 @@ mod tests {
|
||||||
|
|
||||||
const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem";
|
const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem";
|
||||||
|
|
||||||
fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf {
|
fn swarm_file(dir: &tempfile::TempDir) -> std::path::PathBuf {
|
||||||
dir.path().join("absent-swarm-resource-limits.json")
|
dir.path().join("swarm-resource-limits.json")
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn corrupt_limits_keep_an_existing_dropin() {
|
fn corrupt_swarm_caps_keep_an_existing_dropin() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||||
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
limits_to_apply(
|
limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default()),
|
||||||
"sock",
|
|
||||||
&limits,
|
|
||||||
&absent_swarm(&dir),
|
|
||||||
&dropin,
|
|
||||||
&HiveEnv::default()
|
|
||||||
),
|
|
||||||
None
|
None
|
||||||
);
|
);
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
std::fs::read(&limits).expect("read"),
|
std::fs::read(&swarm).expect("read"),
|
||||||
TRUNCATED_LIMITS.as_bytes()
|
TRUNCATED_LIMITS.as_bytes()
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn corrupt_limits_without_a_dropin_apply_hive_defaults() {
|
fn corrupt_swarm_caps_without_a_dropin_apply_hive_defaults() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||||
let applied = limits_to_apply(
|
let applied = limits_to_apply(
|
||||||
"sock",
|
"sock",
|
||||||
&limits,
|
&swarm,
|
||||||
&absent_swarm(&dir),
|
|
||||||
&dir.path().join("absent.conf"),
|
&dir.path().join("absent.conf"),
|
||||||
&HiveEnv::default(),
|
&HiveEnv::default(),
|
||||||
);
|
);
|
||||||
|
|
@ -891,32 +874,12 @@ mod tests {
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn corrupt_swarm_caps_keep_an_existing_dropin() {
|
fn a_swarm_cap_is_applied_over_the_hive_default() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let swarm = dir.path().join("swarm-resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
|
||||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
|
||||||
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
|
||||||
let applied = limits_to_apply(
|
|
||||||
"sock",
|
|
||||||
&dir.path().join("resource-limits.json"),
|
|
||||||
&swarm,
|
|
||||||
&dropin,
|
|
||||||
&HiveEnv::default(),
|
|
||||||
);
|
|
||||||
assert_eq!(applied, None);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_swarm_cap_is_applied_over_the_operator_override() {
|
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
|
||||||
let limits = dir.path().join("resource-limits.json");
|
|
||||||
let swarm = dir.path().join("swarm-resource-limits.json");
|
|
||||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
|
|
||||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps");
|
std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps");
|
||||||
let applied = limits_to_apply(
|
let applied = limits_to_apply(
|
||||||
"sock",
|
"sock",
|
||||||
&limits,
|
|
||||||
&swarm,
|
&swarm,
|
||||||
&dir.path().join("absent.conf"),
|
&dir.path().join("absent.conf"),
|
||||||
&HiveEnv::default(),
|
&HiveEnv::default(),
|
||||||
|
|
@ -925,19 +888,13 @@ mod tests {
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn readable_limits_are_applied_over_an_existing_dropin() {
|
fn readable_swarm_caps_are_applied_over_an_existing_dropin() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
|
std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps");
|
||||||
std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in");
|
std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in");
|
||||||
let applied = limits_to_apply(
|
let applied = limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default());
|
||||||
"sock",
|
|
||||||
&limits,
|
|
||||||
&absent_swarm(&dir),
|
|
||||||
&dropin,
|
|
||||||
&HiveEnv::default(),
|
|
||||||
);
|
|
||||||
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
|
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -953,8 +910,7 @@ mod tests {
|
||||||
let runtime = dir.path().join("runtime");
|
let runtime = dir.path().join("runtime");
|
||||||
let applied = stage_limits(
|
let applied = stage_limits(
|
||||||
"sock",
|
"sock",
|
||||||
&dir.path().join("resource-limits.json"),
|
&swarm_file(dir),
|
||||||
&dir.path().join("swarm-resource-limits.json"),
|
|
||||||
&dir.path().join("hyperhive-limits.conf"),
|
&dir.path().join("hyperhive-limits.conf"),
|
||||||
&runtime,
|
&runtime,
|
||||||
&HiveEnv::default(),
|
&HiveEnv::default(),
|
||||||
|
|
@ -965,40 +921,18 @@ mod tests {
|
||||||
(applied.map(|(_, mem)| mem), copy)
|
(applied.map(|(_, mem)| mem), copy)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The agent's copy follows the swarm's cap, the one systemd enforces, over
|
|
||||||
/// the operator's override: a heap sized against the override would
|
|
||||||
/// outgrow the cgroup.
|
|
||||||
#[test]
|
|
||||||
fn the_agent_reads_the_swarm_cap_over_the_operator_override() {
|
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
|
||||||
std::fs::write(
|
|
||||||
dir.path().join("resource-limits.json"),
|
|
||||||
r#"{"sock": {"memory_max": "16G"}}"#,
|
|
||||||
)
|
|
||||||
.expect("seed limits");
|
|
||||||
std::fs::write(
|
|
||||||
dir.path().join("swarm-resource-limits.json"),
|
|
||||||
r#"{"sock": {"memory_max": "2G"}}"#,
|
|
||||||
)
|
|
||||||
.expect("seed swarm caps");
|
|
||||||
assert_eq!(
|
|
||||||
stage(&dir),
|
|
||||||
(Some("2G".to_owned()), Some("2147483648\n".to_owned()))
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn the_agent_reads_the_cap_its_dropin_carries() {
|
fn the_agent_reads_the_cap_its_dropin_carries() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
|
|
||||||
// No overrides anywhere: the hive-wide default, 4G.
|
// No swarm cap anywhere: the hive-wide default, 4G.
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
stage(&dir),
|
stage(&dir),
|
||||||
(Some("4G".to_owned()), Some("4294967296\n".to_owned()))
|
(Some("4G".to_owned()), Some("4294967296\n".to_owned()))
|
||||||
);
|
);
|
||||||
|
|
||||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
|
std::fs::write(&swarm, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed swarm caps");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
stage(&dir),
|
stage(&dir),
|
||||||
(Some("16G".to_owned()), Some("17179869184\n".to_owned()))
|
(Some("16G".to_owned()), Some("17179869184\n".to_owned()))
|
||||||
|
|
@ -1008,17 +942,17 @@ mod tests {
|
||||||
#[test]
|
#[test]
|
||||||
fn every_kind_of_cap_is_written_for_the_agent() {
|
fn every_kind_of_cap_is_written_for_the_agent() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
for (memory_max, copy) in [
|
for (memory_max, copy) in [
|
||||||
("infinity", "infinity\n"),
|
("infinity", "infinity\n"),
|
||||||
("50%", "8192004096\n"),
|
("50%", "8192004096\n"),
|
||||||
("8G", "8589934592\n"),
|
("8G", "8589934592\n"),
|
||||||
] {
|
] {
|
||||||
std::fs::write(
|
std::fs::write(
|
||||||
&limits,
|
&swarm,
|
||||||
format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#),
|
format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#),
|
||||||
)
|
)
|
||||||
.expect("seed limits");
|
.expect("seed swarm caps");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
stage(&dir),
|
stage(&dir),
|
||||||
(Some(memory_max.to_owned()), Some(copy.to_owned())),
|
(Some(memory_max.to_owned()), Some(copy.to_owned())),
|
||||||
|
|
@ -1072,14 +1006,14 @@ mod tests {
|
||||||
#[test]
|
#[test]
|
||||||
fn a_kept_dropin_hands_the_agent_its_cap() {
|
fn a_kept_dropin_hands_the_agent_its_cap() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||||
let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE);
|
let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE);
|
||||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
|
std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps");
|
||||||
assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned()));
|
assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned()));
|
||||||
|
|
||||||
// Unreadable limits keep the drop-in; the copy follows the drop-in.
|
// Unreadable caps keep the drop-in; the copy follows the drop-in.
|
||||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||||
std::fs::write(
|
std::fs::write(
|
||||||
&dropin,
|
&dropin,
|
||||||
"[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n",
|
"[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n",
|
||||||
|
|
@ -1103,29 +1037,21 @@ mod tests {
|
||||||
#[test]
|
#[test]
|
||||||
fn an_unknown_cap_leaves_no_copy() {
|
fn an_unknown_cap_leaves_no_copy() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let limits = dir.path().join("resource-limits.json");
|
let swarm = swarm_file(&dir);
|
||||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||||
assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned()));
|
assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned()));
|
||||||
|
|
||||||
// Unreadable limits and an unreadable drop-in: nothing says what the
|
// Unreadable caps and an unreadable drop-in: nothing says what the
|
||||||
// cgroup enforces.
|
// cgroup enforces.
|
||||||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||||
std::fs::create_dir(&dropin).expect("seed unreadable drop-in");
|
std::fs::create_dir(&dropin).expect("seed unreadable drop-in");
|
||||||
assert_eq!(stage(&dir), (None, None));
|
assert_eq!(stage(&dir), (None, None));
|
||||||
|
|
||||||
// A percentage with no host memory to resolve it against.
|
// A percentage with no host memory to resolve it against.
|
||||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed limits");
|
std::fs::write(&swarm, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed swarm caps");
|
||||||
let runtime = dir.path().join("runtime");
|
let runtime = dir.path().join("runtime");
|
||||||
let applied = stage_limits(
|
let applied = stage_limits("sock", &swarm, &dropin, &runtime, &HiveEnv::default(), None)
|
||||||
"sock",
|
.expect("stage limits");
|
||||||
&limits,
|
|
||||||
&dir.path().join("swarm-resource-limits.json"),
|
|
||||||
&dropin,
|
|
||||||
&runtime,
|
|
||||||
&HiveEnv::default(),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
.expect("stage limits");
|
|
||||||
assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned()));
|
assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned()));
|
||||||
assert!(!runtime.join(MEMORY_CAP_FILE).exists());
|
assert!(!runtime.join(MEMORY_CAP_FILE).exists());
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -404,9 +404,9 @@ async fn cmd_serve(
|
||||||
// ~hundreds of KB, and the /stats + hive-stats views read it
|
// ~hundreds of KB, and the /stats + hive-stats views read it
|
||||||
// directly; pruning it would just lose trend history for no gain.)
|
// directly; pruning it would just lose trend history for no gain.)
|
||||||
// Slow per-container disk sampler: hive-priv measures each agent's state/
|
// Slow per-container disk sampler: hive-priv measures each agent's state/
|
||||||
// + work/ + writable rootfs every 30 min, cached so the 5s container-load
|
// + work/ + writable rootfs every 30 min, cached so a sample only reads
|
||||||
// poll stays cheap cgroup-only reads. Feeds `disk_bytes` on the LOAD
|
// cgroup files. Feeds `disk_bytes` to the OTEL exporter. See
|
||||||
// tab. See container_stats::disk_sampler_loop.
|
// container_stats::disk_sampler_loop.
|
||||||
crate::container_stats::spawn_disk_sampler();
|
crate::container_stats::spawn_disk_sampler();
|
||||||
// Per-agent container-resource OTEL export: rides the same
|
// Per-agent container-resource OTEL export: rides the same
|
||||||
// cgroup gauges out to the configured OTLP endpoint, reusing the hive
|
// cgroup gauges out to the configured OTLP endpoint, reusing the hive
|
||||||
|
|
|
||||||
|
|
@ -47,8 +47,8 @@ pub struct AgentSpec {
|
||||||
|
|
||||||
/// Stage every generated meta JSON file that exists: topology.json is
|
/// Stage every generated meta JSON file that exists: topology.json is
|
||||||
/// regenerated by `reconcile` whenever the agent set changed;
|
/// regenerated by `reconcile` whenever the agent set changed;
|
||||||
/// tool-groups/capabilities/resource-limits are created lazily on
|
/// tool-groups/capabilities are created lazily on
|
||||||
/// first write (`set_groups`/`set_caps`/`set_limits`) — absent means
|
/// first write (`set_groups`/`set_caps`) — absent means
|
||||||
/// every agent is on defaults, no file needed. Without staging, an
|
/// every agent is on defaults, no file needed. Without staging, an
|
||||||
/// existing-but-untracked file shows up as untracked in the meta repo,
|
/// existing-but-untracked file shows up as untracked in the meta repo,
|
||||||
/// which can confuse nix's dirty-tree fetch. `git add` is a no-op when
|
/// which can confuse nix's dirty-tree fetch. `git add` is a no-op when
|
||||||
|
|
@ -61,10 +61,6 @@ async fn stage_generated_meta_files(dir: &std::path::Path) -> Result<()> {
|
||||||
crate::capabilities::capabilities_path(),
|
crate::capabilities::capabilities_path(),
|
||||||
"capabilities.json",
|
"capabilities.json",
|
||||||
),
|
),
|
||||||
(
|
|
||||||
crate::resource_limits::resource_limits_path(),
|
|
||||||
"resource-limits.json",
|
|
||||||
),
|
|
||||||
] {
|
] {
|
||||||
if path.exists() {
|
if path.exists() {
|
||||||
git(dir, &["add", name]).await?;
|
git(dir, &["add", name]).await?;
|
||||||
|
|
@ -202,7 +198,6 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> {
|
||||||
f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"),
|
f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"),
|
||||||
"topology.json" => Some("topology"),
|
"topology.json" => Some("topology"),
|
||||||
"capabilities.json" => Some("capabilities"),
|
"capabilities.json" => Some("capabilities"),
|
||||||
"resource-limits.json" => Some("resource-limits"),
|
|
||||||
"tool-groups.json" => Some("tool-groups"),
|
"tool-groups.json" => Some("tool-groups"),
|
||||||
_ => None,
|
_ => None,
|
||||||
})
|
})
|
||||||
|
|
@ -352,41 +347,6 @@ pub async fn commit_capabilities(agent: &str, caps: &[String]) -> Result<()> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Write the resource-limits file for `agent` and commit it atomically
|
|
||||||
/// under `META_LOCK`. Same rationale as `commit_tool_groups`: the
|
|
||||||
/// working tree must never be left dirty for the next `sync_agents` to trip
|
|
||||||
/// over.
|
|
||||||
///
|
|
||||||
/// Unlike the perm files this one is never injected into the container —
|
|
||||||
/// it's a host-side cap *on* the agent — but it lives in the same repo
|
|
||||||
/// so a limit change gets the same auditable one-commit-per-change trail.
|
|
||||||
///
|
|
||||||
/// # Errors
|
|
||||||
///
|
|
||||||
/// Returns an error if writing the JSON file fails or a git stage/commit
|
|
||||||
/// step fails.
|
|
||||||
pub async fn commit_resource_limits(
|
|
||||||
agent: &str,
|
|
||||||
limits: &crate::resource_limits::AgentLimits,
|
|
||||||
) -> Result<()> {
|
|
||||||
let _guard = META_LOCK.lock().await;
|
|
||||||
crate::resource_limits::set_limits(agent, limits)
|
|
||||||
.map_err(|e| anyhow::anyhow!("set resource limits for {agent}: {e}"))?;
|
|
||||||
let dir = crate::paths::meta_root();
|
|
||||||
if crate::resource_limits::resource_limits_path().exists() {
|
|
||||||
git(&dir, &["add", "resource-limits.json"]).await?;
|
|
||||||
}
|
|
||||||
if paths_dirty(&dir, &["resource-limits.json"]).await? {
|
|
||||||
git_commit_paths(
|
|
||||||
&dir,
|
|
||||||
&format!("set resource limits for {agent}"),
|
|
||||||
&["resource-limits.json"],
|
|
||||||
)
|
|
||||||
.await?;
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Write both perm files for `agent` (whichever are `Some`) and commit
|
/// Write both perm files for `agent` (whichever are `Some`) and commit
|
||||||
/// them in a SINGLE git commit under `META_LOCK` — the batch
|
/// them in a SINGLE git commit under `META_LOCK` — the batch
|
||||||
/// `POST /api/permissions` path. A `None` field leaves that file
|
/// `POST /api/permissions` path. A `None` field leaves that file
|
||||||
|
|
|
||||||
|
|
@ -170,7 +170,7 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
|
||||||
// question is only answerable on this side of the socket —
|
// question is only answerable on this side of the socket —
|
||||||
// see the request's doc comment for why the client asks.
|
// see the request's doc comment for why the client asks.
|
||||||
HostRequest::AgentExists { name } => HostResponse::agent_exists(agent_exists(name)?),
|
HostRequest::AgentExists { name } => HostResponse::agent_exists(agent_exists(name)?),
|
||||||
HostRequest::AgentStatus => handle_agent_status(&coord).await,
|
HostRequest::AgentStatus => handle_agent_status().await,
|
||||||
// Intercepted in `handle()` before a request ever reaches
|
// Intercepted in `handle()` before a request ever reaches
|
||||||
// `dispatch` — see `stream_agent_status`'s doc comment. This
|
// `dispatch` — see `stream_agent_status`'s doc comment. This
|
||||||
// arm exists only so the match stays exhaustive; reaching it
|
// arm exists only so the match stays exhaustive; reaching it
|
||||||
|
|
@ -184,19 +184,6 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
|
||||||
// federation peer-config block) and open the web surfaces
|
// federation peer-config block) and open the web surfaces
|
||||||
// (`hivectl open`).
|
// (`hivectl open`).
|
||||||
HostRequest::Urls => HostResponse::urls(hive_urls()),
|
HostRequest::Urls => HostResponse::urls(hive_urls()),
|
||||||
HostRequest::SetResourceLimits {
|
|
||||||
name,
|
|
||||||
cpu_quota,
|
|
||||||
memory_max,
|
|
||||||
} => {
|
|
||||||
handle_set_resource_limits(
|
|
||||||
&coord,
|
|
||||||
name,
|
|
||||||
cpu_quota.as_deref(),
|
|
||||||
memory_max.as_deref(),
|
|
||||||
)
|
|
||||||
.await?
|
|
||||||
}
|
|
||||||
HostRequest::MatrixInvite { user, room } => {
|
HostRequest::MatrixInvite { user, room } => {
|
||||||
handle_matrix_invite(user, room.as_deref()).await?
|
handle_matrix_invite(user, room.as_deref()).await?
|
||||||
}
|
}
|
||||||
|
|
@ -293,8 +280,8 @@ async fn handle_set_paused(
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Collect per-agent status rows for `hivectl status` and the dashboard.
|
/// Collect per-agent status rows for `hivectl status` and the dashboard.
|
||||||
async fn handle_agent_status(coord: &Arc<Coordinator>) -> HostResponse {
|
async fn handle_agent_status() -> HostResponse {
|
||||||
match crate::container_view::build_all(&coord.hive_env()).await {
|
match crate::container_view::build_all().await {
|
||||||
Ok(views) => HostResponse::agent_statuses(
|
Ok(views) => HostResponse::agent_statuses(
|
||||||
views
|
views
|
||||||
.into_iter()
|
.into_iter()
|
||||||
|
|
@ -380,66 +367,6 @@ fn agent_exists(name: &hive_types::Ident) -> Result<bool> {
|
||||||
.with_context(|| format!("check agent state dir for {name}"))
|
.with_context(|| format!("check agent state dir for {name}"))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Validate + persist an agent's CPU/memory overrides, then re-apply the
|
|
||||||
/// drop-in so the change lands without waiting for a rebuild.
|
|
||||||
///
|
|
||||||
/// Validation is here rather than only in `hivectl` because the values
|
|
||||||
/// are written verbatim into the systemd drop-in: a malformed
|
|
||||||
/// `CPUQuota=` makes systemd reject the unit, and the container stops
|
|
||||||
/// starting. Every client (CLI, dashboard, anything later) goes through
|
|
||||||
/// this path, so the guard belongs on this side of the socket.
|
|
||||||
///
|
|
||||||
/// `None`/`None` removes the agent's entry, returning it to the
|
|
||||||
/// hive-wide defaults.
|
|
||||||
async fn handle_set_resource_limits(
|
|
||||||
coord: &Arc<Coordinator>,
|
|
||||||
name: &hive_types::Ident,
|
|
||||||
cpu_quota: Option<&str>,
|
|
||||||
memory_max: Option<&str>,
|
|
||||||
) -> Result<HostResponse> {
|
|
||||||
if let Some(value) = cpu_quota {
|
|
||||||
crate::resource_limits::validate_cpu_quota(value).map_err(anyhow::Error::msg)?;
|
|
||||||
}
|
|
||||||
// `validate_memory_max` returns the value to store, not just an
|
|
||||||
// ok/err verdict: a friendly spelling like "8GB" is accepted but
|
|
||||||
// normalized to the "8G" systemd's own parser actually takes.
|
|
||||||
let memory_max = memory_max
|
|
||||||
.map(crate::resource_limits::validate_memory_max)
|
|
||||||
.transpose()
|
|
||||||
.map_err(anyhow::Error::msg)?;
|
|
||||||
tracing::info!(%name, ?cpu_quota, ?memory_max, "set_resource_limits");
|
|
||||||
let limits = crate::resource_limits::AgentLimits {
|
|
||||||
cpu_quota: cpu_quota.map(ToOwned::to_owned),
|
|
||||||
memory_max,
|
|
||||||
};
|
|
||||||
// Goes through `meta::commit_resource_limits`, not the bare
|
|
||||||
// `resource_limits::set_limits`: the write has to be staged +
|
|
||||||
// committed under `META_LOCK` or it leaves the meta working tree
|
|
||||||
// dirty for the next `sync_agents` to trip over.
|
|
||||||
crate::meta::commit_resource_limits(name.as_str(), &limits).await?;
|
|
||||||
|
|
||||||
// Re-apply the drop-in straight away — same three lines as the job
|
|
||||||
// queue's `WriteDropin` node. Without this the new values would sit
|
|
||||||
// in the JSON until the agent's next spawn or rebuild.
|
|
||||||
let agent_dir = crate::paths::agent_runtime_dir(name.as_str());
|
|
||||||
let hive = coord.hive_env();
|
|
||||||
let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?;
|
|
||||||
crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?;
|
|
||||||
|
|
||||||
let (cpu, mem) = crate::resource_limits::effective_with_swarm(
|
|
||||||
&crate::resource_limits::resource_limits_path(),
|
|
||||||
&crate::resource_limits::swarm_limits_path(),
|
|
||||||
name.as_str(),
|
|
||||||
&hive.agent_cpu_quota,
|
|
||||||
&hive.agent_memory_max,
|
|
||||||
)?;
|
|
||||||
Ok(HostResponse::messages(vec![format!(
|
|
||||||
"{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now; the \
|
|
||||||
Claude/JSC heap ceiling and the subagent MemoryHigh= derived from it follow at the \
|
|
||||||
container's next start (restart it if it's running and needs them now)"
|
|
||||||
)]))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Guard: matrix provisioning needs a homeserver to provision against.
|
/// Guard: matrix provisioning needs a homeserver to provision against.
|
||||||
///
|
///
|
||||||
/// Answers "is one configured", not "is one running here": the address is
|
/// Answers "is one configured", not "is one running here": the address is
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
//! Live per-agent-container resource load (CPU + memory, plus the slow
|
//! Live per-agent-container resource load (CPU + memory, plus the slow
|
||||||
//! disk sample from hive-priv in [`disk_sampler_loop`]) for the
|
//! disk sample from hive-priv in [`disk_sampler_loop`]) for the OTEL
|
||||||
//! dashboard. Reads cgroup v2 stat files for each running agent
|
//! exporter. Reads cgroup v2 stat files for each running agent
|
||||||
//! machine directly — same privsep-clean posture as the turn-stats
|
//! machine directly — same privsep-clean posture as the turn-stats
|
||||||
//! sqlite reads (`cpu.stat` / `memory.*` are world-readable; no
|
//! sqlite reads (`cpu.stat` / `memory.*` are world-readable; no
|
||||||
//! `hive-priv` needed). Runs host-side in hive-c0re, where
|
//! `hive-priv` needed). Runs host-side in hive-c0re, where
|
||||||
|
|
@ -31,8 +31,8 @@ use utoipa::ToSchema;
|
||||||
|
|
||||||
use crate::coordinator::Coordinator;
|
use crate::coordinator::Coordinator;
|
||||||
|
|
||||||
/// Sampling interval for the two CPU reads. Short enough to keep the
|
/// Sampling interval for the two CPU reads. Short enough to keep a sample
|
||||||
/// endpoint snappy, long enough that the delta isn't dominated by read
|
/// quick, long enough that the delta isn't dominated by read
|
||||||
/// jitter.
|
/// jitter.
|
||||||
const CPU_SAMPLE: Duration = Duration::from_millis(200);
|
const CPU_SAMPLE: Duration = Duration::from_millis(200);
|
||||||
|
|
||||||
|
|
@ -72,8 +72,8 @@ pub struct ContainerResource {
|
||||||
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(30);
|
const DISK_SAMPLE_INTERVAL: Duration = Duration::from_mins(30);
|
||||||
|
|
||||||
/// Last per-agent disk measurement, keyed by agent name; `None` is a failed
|
/// Last per-agent disk measurement, keyed by agent name; `None` is a failed
|
||||||
/// one. Written by [`disk_sampler_loop`], read by [`gather`] so the hot
|
/// one. Written by [`disk_sampler_loop`], read by [`gather`], so a sample
|
||||||
/// poll never measures.
|
/// never measures.
|
||||||
fn disk_cache() -> &'static RwLock<HashMap<String, Option<u64>>> {
|
fn disk_cache() -> &'static RwLock<HashMap<String, Option<u64>>> {
|
||||||
static CACHE: OnceLock<RwLock<HashMap<String, Option<u64>>>> = OnceLock::new();
|
static CACHE: OnceLock<RwLock<HashMap<String, Option<u64>>>> = OnceLock::new();
|
||||||
CACHE.get_or_init(|| RwLock::new(HashMap::new()))
|
CACHE.get_or_init(|| RwLock::new(HashMap::new()))
|
||||||
|
|
@ -100,7 +100,7 @@ fn disk_reading(name: &hive_types::Ident, measured: anyhow::Result<u64>) -> Opti
|
||||||
|
|
||||||
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], remeasure disk usage
|
/// Background loop: every [`DISK_SAMPLE_INTERVAL`], remeasure disk usage
|
||||||
/// for every kept-state agent and refresh [`disk_cache`]. The first pass
|
/// for every kept-state agent and refresh [`disk_cache`]. The first pass
|
||||||
/// runs immediately so the dashboard shows disk shortly after boot.
|
/// runs immediately so the exporter has disk shortly after boot.
|
||||||
/// Runs forever; spawn once at hive-c0re startup via [`spawn_disk_sampler`].
|
/// Runs forever; spawn once at hive-c0re startup via [`spawn_disk_sampler`].
|
||||||
pub async fn disk_sampler_loop() {
|
pub async fn disk_sampler_loop() {
|
||||||
loop {
|
loop {
|
||||||
|
|
|
||||||
|
|
@ -234,8 +234,6 @@ mod tests {
|
||||||
status_text: None,
|
status_text: None,
|
||||||
status_set_at: None,
|
status_set_at: None,
|
||||||
paused: false,
|
paused: false,
|
||||||
cpu_quota: "200%".to_owned(),
|
|
||||||
memory_max: "4G".to_owned(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,5 @@
|
||||||
//! Per-agent container-resource OTEL export. hive-c0re already samples each
|
//! Per-agent container-resource OTEL export. This is what the cgroup load
|
||||||
//! agent container's cgroup load for the dashboard
|
//! sampled in [`super::container_stats`] feeds: it rides those gauges out to the
|
||||||
//! ([`super::container_stats`]); this rides those same gauges out to the
|
|
||||||
//! hive's own collector — the same first hop the agents use, named by the
|
//! hive's own collector — the same first hop the agents use, named by the
|
||||||
//! standard `OTEL_EXPORTER_OTLP_ENDPOINT` (see `nix/host-modules/hive-c0re`,
|
//! standard `OTEL_EXPORTER_OTLP_ENDPOINT` (see `nix/host-modules/hive-c0re`,
|
||||||
//! which sets it from the same binding it hands agents). No toggle of
|
//! which sets it from the same binding it hands agents). No toggle of
|
||||||
|
|
@ -194,7 +193,7 @@ fn register_instruments(provider: &SdkMeterProvider, snapshot: Arc<Mutex<Vec<Con
|
||||||
.build();
|
.build();
|
||||||
|
|
||||||
// Custom (`hyperhive.`) metrics semconv doesn't standardise: memory peak,
|
// Custom (`hyperhive.`) metrics semconv doesn't standardise: memory peak,
|
||||||
// on-disk footprint, and the instantaneous cpu percent the dashboard shows.
|
// on-disk footprint, and the instantaneous cpu percent.
|
||||||
let snap = snapshot.clone();
|
let snap = snapshot.clone();
|
||||||
meter
|
meter
|
||||||
.u64_observable_gauge("hyperhive.container.memory.peak")
|
.u64_observable_gauge("hyperhive.container.memory.peak")
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
//! swarm-controller keeps one key per agent in the `agent-limits` bucket
|
//! swarm-controller keeps one key per agent in the `agent-limits` bucket
|
||||||
//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads
|
//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads
|
||||||
//! the key of each agent this hive has and stores what it finds in
|
//! the key of each agent this hive has and stores what it finds in
|
||||||
//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it.
|
//! `swarm-resource-limits.json`, where the drop-in reads it.
|
||||||
//! A running agent gets a changed cap at its first spawn or rebuild after
|
//! A running agent gets a changed cap at its first spawn or rebuild after
|
||||||
//! that read.
|
//! that read.
|
||||||
//!
|
//!
|
||||||
|
|
@ -92,9 +92,9 @@ async fn sync(client: &async_nats::Client) -> Result<()> {
|
||||||
/// The caps to store for `declared`, by agent. An agent with no field set
|
/// The caps to store for `declared`, by agent. An agent with no field set
|
||||||
/// gets no entry.
|
/// gets no entry.
|
||||||
///
|
///
|
||||||
/// Every value goes through the validators the operator's own input path
|
/// Every value goes through the `resource_limits` validators, because it is
|
||||||
/// uses, because it is rendered verbatim into the container's systemd
|
/// rendered verbatim into the container's systemd drop-in. One bad value
|
||||||
/// drop-in. One bad value refuses the whole set.
|
/// refuses the whole set.
|
||||||
fn validated(declared: &BTreeMap<String, AgentCaps>) -> Result<BTreeMap<String, AgentLimits>> {
|
fn validated(declared: &BTreeMap<String, AgentCaps>) -> Result<BTreeMap<String, AgentLimits>> {
|
||||||
let mut out = BTreeMap::new();
|
let mut out = BTreeMap::new();
|
||||||
for (agent, caps) in declared {
|
for (agent, caps) in declared {
|
||||||
|
|
@ -127,48 +127,38 @@ mod tests {
|
||||||
use std::collections::BTreeMap;
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
use super::validated;
|
use super::validated;
|
||||||
use crate::resource_limits::AgentLimits;
|
|
||||||
use swarm_queue_client::agent_limits::AgentCaps;
|
use swarm_queue_client::agent_limits::AgentCaps;
|
||||||
|
|
||||||
/// `atlas`'s answer with `atlas_key` as the value read for it, over a hive
|
/// `atlas`'s answer with `atlas_key` as the value read for it, over the
|
||||||
/// chain that caps it at 800% / 16G.
|
/// hive-wide defaults of 200% / 4G.
|
||||||
fn atlas_with(atlas_key: &str) -> (String, String) {
|
fn atlas_with(atlas_key: &str) -> (String, String) {
|
||||||
let declared = BTreeMap::from([(
|
let declared = BTreeMap::from([(
|
||||||
"atlas".to_owned(),
|
"atlas".to_owned(),
|
||||||
serde_json::from_str::<AgentCaps>(atlas_key).expect("decodes"),
|
serde_json::from_str::<AgentCaps>(atlas_key).expect("decodes"),
|
||||||
)]);
|
)]);
|
||||||
let stored = validated(&declared).expect("valid caps");
|
let stored = validated(&declared).expect("valid caps");
|
||||||
let hive_side = BTreeMap::from([(
|
crate::resource_limits::effective_from(&stored, "atlas", "200%", "4G")
|
||||||
"atlas".to_owned(),
|
|
||||||
AgentLimits {
|
|
||||||
cpu_quota: Some("800%".to_owned()),
|
|
||||||
memory_max: Some("16G".to_owned()),
|
|
||||||
},
|
|
||||||
)]);
|
|
||||||
crate::resource_limits::effective_from_with_swarm(
|
|
||||||
&stored, &hive_side, "atlas", "200%", "4G",
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The other half of swarm-controller's
|
/// The other half of swarm-controller's
|
||||||
/// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`:
|
/// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`:
|
||||||
/// the first value is the bytes that route stores. The cap replaces the
|
/// the first value is the bytes that route stores. The cap replaces the
|
||||||
/// hive's chain; a deleted key reads as `{}` and gives the chain back.
|
/// hive default; a deleted key reads as `{}` and gives the default back.
|
||||||
#[test]
|
#[test]
|
||||||
fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() {
|
fn the_caps_the_route_stores_replace_the_hive_default_and_a_cleared_key_restores_it() {
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#),
|
atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#),
|
||||||
("400%".to_owned(), "8G".to_owned())
|
("400%".to_owned(), "8G".to_owned())
|
||||||
);
|
);
|
||||||
assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned()));
|
assert_eq!(atlas_with("{}"), ("200%".to_owned(), "4G".to_owned()));
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Per field: a swarm memory cap leaves CPU on the hive's chain.
|
/// Per field: a swarm memory cap leaves CPU on the hive default.
|
||||||
#[test]
|
#[test]
|
||||||
fn a_field_the_swarm_leaves_unset_stays_with_the_hive() {
|
fn a_field_the_swarm_leaves_unset_stays_with_the_hive() {
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
atlas_with(r#"{"memory_max":"2G"}"#),
|
atlas_with(r#"{"memory_max":"2G"}"#),
|
||||||
("800%".to_owned(), "2G".to_owned())
|
("200%".to_owned(), "2G".to_owned())
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -211,23 +211,6 @@ pub enum HostRequest {
|
||||||
/// this request just answers whatever it's asked. Result:
|
/// this request just answers whatever it's asked. Result:
|
||||||
/// [`HostResponse::nodes`].
|
/// [`HostResponse::nodes`].
|
||||||
QueueNodes { ids: Vec<u64> },
|
QueueNodes { ids: Vec<u64> },
|
||||||
/// Declare an agent's CPU/memory overrides for the per-container
|
|
||||||
/// systemd drop-in, persisted to `meta/resource-limits.json`.
|
|
||||||
///
|
|
||||||
/// **Replace, not merge** — the pair given here becomes the agent's
|
|
||||||
/// entire entry, matching how tool-groups/capabilities are set. A
|
|
||||||
/// `None` field falls back to the hive-wide
|
|
||||||
/// `agentCpuQuota` / `agentMemoryMax`, so passing both as `None`
|
|
||||||
/// removes the entry entirely (reset to hive defaults).
|
|
||||||
///
|
|
||||||
/// Values are passed verbatim to systemd, so the server validates
|
|
||||||
/// their shape before persisting: a malformed `CPUQuota=` makes
|
|
||||||
/// systemd reject the unit, which would stop the container starting.
|
|
||||||
SetResourceLimits {
|
|
||||||
name: Ident,
|
|
||||||
cpu_quota: Option<String>,
|
|
||||||
memory_max: Option<String>,
|
|
||||||
},
|
|
||||||
/// Stop managed containers hive-wide in one operator action
|
/// Stop managed containers hive-wide in one operator action
|
||||||
/// (`hivectl stop`): agents plus the selected infra containers. `scope`
|
/// (`hivectl stop`): agents plus the selected infra containers. `scope`
|
||||||
/// selects which classes; an all-false scope means **everything** (the
|
/// selects which classes; an all-false scope means **everything** (the
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
//! `hivectl agent <name>` — everything scoped to ONE managed agent:
|
//! `hivectl agent <name>` — everything scoped to ONE managed agent:
|
||||||
//! container lifecycle over the host admin socket (restart/pause/resume/
|
//! container lifecycle over the host admin socket (restart/pause/resume/
|
||||||
//! spawn/kill/destroy/rebuild/set-limits/choom/watch), plus the
|
//! spawn/kill/destroy/rebuild/choom/watch), plus the
|
||||||
//! `quota` and `subvol` groups, whose handlers live in their own modules.
|
//! `quota` and `subvol` groups, whose handlers live in their own modules.
|
||||||
//! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide
|
//! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide
|
||||||
//! read that lives in this module too since it shares the same daemon
|
//! read that lives in this module too since it shares the same daemon
|
||||||
|
|
@ -282,32 +282,6 @@ pub(crate) async fn run_agent(socket: &Path, name: &str, cmd: AgentCmd) -> Resul
|
||||||
let name = crate::util::parse_ident(name)?;
|
let name = crate::util::parse_ident(name)?;
|
||||||
render(crate::client::request(socket, HostRequest::Rebuild { name }).await?)
|
render(crate::client::request(socket, HostRequest::Rebuild { name }).await?)
|
||||||
}
|
}
|
||||||
AgentCmd::SetLimits {
|
|
||||||
cpu_quota,
|
|
||||||
memory_max,
|
|
||||||
reset,
|
|
||||||
} => {
|
|
||||||
let name = crate::util::parse_ident(name)?;
|
|
||||||
// `--reset` is the only way to reach an all-`None` request;
|
|
||||||
// clap rejects a bare `set-limits` with neither flag, so a
|
|
||||||
// forgotten value can't silently clear the overrides.
|
|
||||||
let (cpu_quota, memory_max) = if reset {
|
|
||||||
(None, None)
|
|
||||||
} else {
|
|
||||||
(cpu_quota, memory_max)
|
|
||||||
};
|
|
||||||
render(
|
|
||||||
crate::client::request(
|
|
||||||
socket,
|
|
||||||
HostRequest::SetResourceLimits {
|
|
||||||
name,
|
|
||||||
cpu_quota,
|
|
||||||
memory_max,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
.await?,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
AgentCmd::Choom { resume_session } => {
|
AgentCmd::Choom { resume_session } => {
|
||||||
crate::choom::choom(socket, name, resume_session.as_deref()).await
|
crate::choom::choom(socket, name, resume_session.as_deref()).await
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -393,30 +393,6 @@ pub enum AgentCmd {
|
||||||
},
|
},
|
||||||
/// Apply pending config to this managed container.
|
/// Apply pending config to this managed container.
|
||||||
Rebuild,
|
Rebuild,
|
||||||
/// Declare this agent's CPU/memory limits, overriding the hive-wide
|
|
||||||
/// defaults.
|
|
||||||
///
|
|
||||||
/// Replaces the agent's whole override entry rather than merging into
|
|
||||||
/// it: any limit you don't pass returns to the hive-wide default. To
|
|
||||||
/// change one and keep the other, pass both. Disk is a separate
|
|
||||||
/// resource with its own group — see `quota`.
|
|
||||||
SetLimits {
|
|
||||||
/// systemd `CPUQuota=` value, for example `400%` (100% = one full core).
|
|
||||||
#[arg(long, conflicts_with = "reset")]
|
|
||||||
cpu_quota: Option<String>,
|
|
||||||
/// systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`.
|
|
||||||
#[arg(long, conflicts_with = "reset")]
|
|
||||||
memory_max: Option<String>,
|
|
||||||
/// Drop all overrides — the agent returns to the hive-wide
|
|
||||||
/// defaults. Required to clear limits, so that a `set-limits`
|
|
||||||
/// with a forgotten value can't silently reset the agent.
|
|
||||||
#[arg(
|
|
||||||
long,
|
|
||||||
conflicts_with_all = ["cpu_quota", "memory_max"],
|
|
||||||
required_unless_present_any = ["cpu_quota", "memory_max"],
|
|
||||||
)]
|
|
||||||
reset: bool,
|
|
||||||
},
|
|
||||||
/// Open an interactive Claude session inside this agent's container.
|
/// Open an interactive Claude session inside this agent's container.
|
||||||
///
|
///
|
||||||
/// A fresh session by default, or resume a prior one. Requires root
|
/// A fresh session by default, or resume a prior one. Requires root
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue