Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: the per-agent resource limit override is gone

The hive no longer sets an agent's CPU or memory cap: caps come from the swarm, else the hive-wide defaults. The container load tab was the only hive UI that set them, and its figures are already exported as OTEL metrics, so with the tab gone nothing wrote meta/resource-limits.json.

Removed: the file's reader and writer, the `hivectl agent set-limits` verb with its host-socket request, the `GET /api/container-resources` and `POST /api/resource-limits/{name}` dashboard endpoints, and the `cpu_quota`/`memory_max` fields of the container view that only the tab read. A leftover resource-limits.json is ignored.
This commit is contained in:
iris 2026-10-14 23:57:55 +02:00 • committed by mara
commit 90e33cbd19
19 changed files with 161 additions and 828 deletions

View file

@ -395,26 +395,26 @@ Contents:
(`{ "ruth": ["manage_root_agent"] }`). Written by (`{ "ruth": ["manage_root_agent"] }`). Written by
`capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env `capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
var. Absent agents have no extra capabilities. var. Absent agents have no extra capabilities.
- `resource-limits.json` — per-agent container resource overrides - `swarm-resource-limits.json` — the caps the swarm sets for this
hive's agents
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`). (`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
Written by `resource_limits::set_limits`; read where Written only by `workers::swarm_limits`, which copies each agent's key of
the swarm's `agent-limits` bucket here every 30 seconds. Read where
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected `lifecycle::write_dropins` generates the systemd drop-in, **not** injected
into the container — these are host-side caps on the container, so into the container — these are host-side caps on the container, so
the capped party never sets them. It sees only the resulting memory cap, the capped party never sets them. It sees only the resulting memory cap,
as a byte count or `infinity` in its runtime dir (below). Fallback is per *field*: as a byte count or `infinity` in its runtime dir (below). Fallback is per
an absent file, absent agent, or absent field falls back to the *field*: an absent file, absent agent, or absent field falls back to the
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`, hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
so an agent can override only its memory and still track the hive so an agent can have only its memory capped and still track the hive
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same default for CPU. Every value passes the validators first, and one bad
drop-in have **no** per-agent override — they're hive-wide only and value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
come straight off `HiveEnv`, so this file has no field for them. shares in the same drop-in have **no** per-agent setting — they're
- `swarm-resource-limits.json` — the caps the swarm sets for this hive-wide only and come straight off `HiveEnv`, so this file has no field
hive's agents, same shape as `resource-limits.json`. Written only by for them.
`workers::swarm_limits`, which copies each agent's key of the swarm's - `resource-limits.json` — retired. Per-agent overrides set on the hive used
`agent-limits` bucket here every 30 seconds. A field set here replaces to live here; nothing reads or writes it now. An old copy in the meta repo
that field's answer from `resource-limits.json` and the hive-wide is ignored and can be deleted.
default. Every value passes the same validators as the operator's
overrides first, and one bad value makes the worker change nothing here.
The root agent has the meta dir RO-mounted at `/meta/`. The root agent has the meta dir RO-mounted at `/meta/`.

View file

@ -455,12 +455,12 @@ requiring a host rebuild.
The answer per agent and per field is, first match wins: The answer per agent and per field is, first match wins:
1. the swarm's cap for the agent (see below); 1. the swarm's cap for the agent (see below);
2. the operator's per-agent override in `meta/resource-limits.json`; 2. `agentCpuQuota` / `agentMemoryMax`.
3. `agentCpuQuota` / `agentMemoryMax`.
A swarm cap replaces the hive's answer, so a hive can't raise an agent A swarm cap replaces the hive's answer, so a hive can't raise an agent
above it. Like the rest of the drop-in, a changed swarm cap reaches a above it. Like the rest of the drop-in, a changed swarm cap reaches a
running agent at its next spawn or rebuild. running agent at its next spawn or rebuild. A hive has no way to set a cap
for one agent: caps are set per agent in the swarm UI.
The container can't read its own cap, so `write_dropins` also writes the The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir effective `MemoryMax=` into the agent's runtime dir
@ -468,7 +468,7 @@ effective `MemoryMax=` into the agent's runtime dir
`infinity` for no cap. A percentage becomes bytes the way systemd resolves `infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the overrides or the swarm caps file, the last drop-in because it can't read the swarm caps file,
the file carries that drop-in's `MemoryMax=`. the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count: At boot the container derives two values from a byte count:
@ -520,7 +520,7 @@ given.
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the same validators as the operator's overrides first. One bad value passes the validators first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a value or a failed read leaves the file unchanged, and repeated failures raise a
dashboard banner. dashboard banner.
@ -548,8 +548,7 @@ wants it.
Two things to know about the weights: Two things to know about the weights:
- They're **hive-wide** — unlike the caps there is no per-agent - They're **hive-wide** — unlike the caps there is no per-agent
override in `meta/resource-limits.json`, so every agent carries the setting, so every agent carries the same value and the weight does _not_ rank agents against each other.
same value and the weight does _not_ rank agents against each other.
What `80` buys is that agents yield to everything **not** on this What `80` buys is that agents yield to everything **not** on this
drop-in path: host services (nginx and dnsmasq among them) and the drop-in path: host services (nginx and dnsmasq among them) and the
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which

View file

@ -22,7 +22,6 @@ This document contains the help content for the `hivectl` command-line program.
* [`hivectl agent kill`↴](#hivectl-agent-kill) * [`hivectl agent kill`↴](#hivectl-agent-kill)
* [`hivectl agent destroy`↴](#hivectl-agent-destroy) * [`hivectl agent destroy`↴](#hivectl-agent-destroy)
* [`hivectl agent rebuild`↴](#hivectl-agent-rebuild) * [`hivectl agent rebuild`↴](#hivectl-agent-rebuild)
* [`hivectl agent set-limits`↴](#hivectl-agent-set-limits)
* [`hivectl agent choom`↴](#hivectl-agent-choom) * [`hivectl agent choom`↴](#hivectl-agent-choom)
* [`hivectl agent watch`↴](#hivectl-agent-watch) * [`hivectl agent watch`↴](#hivectl-agent-watch)
* [`hivectl agent quota`↴](#hivectl-agent-quota) * [`hivectl agent quota`↴](#hivectl-agent-quota)
@ -221,7 +220,6 @@ Everything here targets a single named agent (`hivectl agent foo restart`, `hive
* `kill` — Hard-stop this managed container * `kill` — Hard-stop this managed container
* `destroy` — Tear down this sub-agent container, keeping its state by default. No undo * `destroy` — Tear down this sub-agent container, keeping its state by default. No undo
* `rebuild` — Apply pending config to this managed container * `rebuild` — Apply pending config to this managed container
* `set-limits` — Declare this agent's CPU/memory limits, overriding the hive-wide defaults
* `choom` — Open an interactive Claude session inside this agent's container * `choom` — Open an interactive Claude session inside this agent's container
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI * `watch` — Follow this agent's live turn/tool-call event stream from the CLI
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups * `quota` — This agent's disk accounting + optional quota via btrfs qgroups
@ -315,22 +313,6 @@ Apply pending config to this managed container
## `hivectl agent set-limits`
Declare this agent's CPU/memory limits, overriding the hive-wide defaults.
Replaces the agent's whole override entry rather than merging into it: any limit you don't pass returns to the hive-wide default. To change one and keep the other, pass both. Disk is a separate resource with its own group — see `quota`.
**Usage:** `hivectl agent set-limits [OPTIONS]`
###### **Options:**
* `--cpu-quota <CPU_QUOTA>` — systemd `CPUQuota=` value, for example `400%` (100% = one full core)
* `--memory-max <MEMORY_MAX>` — systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`
* `--reset` — Drop all overrides — the agent returns to the hive-wide defaults. Required to clear limits, so that a `set-limits` with a forgotten value can't silently reset the agent
## `hivectl agent choom` ## `hivectl agent choom`
Open an interactive Claude session inside this agent's container. Open an interactive Claude session inside this agent's container.

View file

@ -136,34 +136,6 @@ resume drains the backlog rather than dropping it. Points worth knowing:
- Visible as ` paused` in `list-agents`'s STATUS column, as a `paused` - Visible as ` paused` in `list-agents`'s STATUS column, as a `paused`
field on the JSON rows, and as a badge on the dashboard card. field on the JSON rows, and as a badge on the dashboard card.
### Per-agent resource limits
```bash
hivectl agent sock set-limits --cpu-quota 400% --memory-max 8G
hivectl agent sock set-limits --memory-max 8G # CPU falls back to the hive default
hivectl agent sock set-limits --reset # drop all overrides
```
Overrides the hive-wide `services.hyperhive.c0re.agentCpuQuota` /
`agentMemoryMax` for one agent, persisted to
`meta/resource-limits.json` (see
[`persistence.md`](../agent-lifecycle/persistence.md)). Values are systemd's
`CPUQuota=` / `MemoryMax=` syntax: a percentage (`400%` = four full
cores) for CPU; a size (`8G`), a percentage of physical RAM, or
`infinity` for memory. `set-limits` validates both before persisting them —
they go into a systemd drop-in verbatim, and a typo there makes the
unit fail to start.
**Declarative, not incremental**: each invocation replaces the agent's
whole entry. `hivectl agent sock set-limits --memory-max 8G` leaves `sock`
with _only_ a memory override, reverting any previously set CPU quota to
the hive default. To avoid a forgotten flag silently wiping an override,
hivectl rejects `set-limits` with no flags at all — clearing requires the
explicit `--reset`.
The command rewrites the container's drop-in and reloads systemd, so
new containers and restarts pick the values up immediately.
## Choom ## Choom
Drop into an interactive Claude session inside an agent container. Drop into an interactive Claude session inside an agent container.

View file

@ -1,5 +1,8 @@
//! Per-agent CPU/memory limit overrides. Stored at //! CPU/memory limits for an agent's container: the caps the swarm controller
//! `/var/lib/hyperhive/meta/resource-limits.json` alongside //! declared, and the validators for the values the systemd drop-in takes.
//!
//! The swarm caps are stored at
//! `/var/lib/hyperhive/meta/swarm-resource-limits.json` alongside
//! `topology.json`, `tool-groups.json` and `capabilities.json`. //! `topology.json`, `tool-groups.json` and `capabilities.json`.
//! //!
//! Format: a JSON object mapping agent name to an object with optional //! Format: a JSON object mapping agent name to an object with optional
@ -14,9 +17,9 @@
//! //!
//! Fallback is **per field**: an absent file, an absent agent, or an //! Fallback is **per field**: an absent file, an absent agent, or an
//! absent field all fall back to the hive-wide //! absent field all fall back to the hive-wide
//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So an agent //! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So a swarm cap on
//! can raise only its memory cap and keep tracking the hive default for //! memory alone leaves CPU on the hive default — see
//! CPU — see [`effective`]. //! [`effective_from_with_swarm`].
//! //!
//! Read path: `lifecycle::host_config::write_dropins`, on every spawn //! Read path: `lifecycle::host_config::write_dropins`, on every spawn
//! and every rebuild. //! and every rebuild.
@ -30,10 +33,8 @@
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
const RESOURCE_LIMITS_FILE: &str = "resource-limits.json"; /// One agent's caps. Both fields optional and independent; `None` means
/// "use the hive-wide default for this field".
/// One agent's overrides. Both fields optional and independent; `None`
/// means "use the hive-wide default for this field".
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] #[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct AgentLimits { pub struct AgentLimits {
/// systemd `CPUQuota=` value, e.g. `"400%"`. /// systemd `CPUQuota=` value, e.g. `"400%"`.
@ -44,84 +45,6 @@ pub struct AgentLimits {
pub memory_max: Option<String>, pub memory_max: Option<String>,
} }
impl AgentLimits {
/// True when neither field is set — such an entry is dropped rather
/// than persisted as an empty object.
#[must_use]
pub fn is_empty(&self) -> bool {
self.cpu_quota.is_none() && self.memory_max.is_none()
}
}
#[must_use]
pub fn resource_limits_path() -> PathBuf {
crate::paths::meta_root().join(RESOURCE_LIMITS_FILE)
}
/// Read the per-agent limit map. An absent file is the empty map —
/// callers treat a missing entry as "hive-wide defaults". A file that
/// exists but can't be read or parsed is an error: an override can be
/// tighter than the hive default, so reading it as empty is not a safe
/// fallback.
pub fn read() -> std::io::Result<BTreeMap<String, AgentLimits>> {
super::read_map(&resource_limits_path())
}
/// Resolve the effective values for an agent from the override file at
/// `path` (normally [`resource_limits_path`]), filling each unset field
/// from the hive-wide default. This is the single place the fallback
/// rule lives; `write_dropins` calls it and passes the result straight
/// to systemd. Errors as [`read`] does.
///
/// Use [`effective_from`] when resolving more than one agent in a row.
pub fn effective(
path: &Path,
name: &str,
hive_cpu_quota: &str,
hive_memory_max: &str,
) -> std::io::Result<(String, String)> {
let limits = super::read_map(path)?;
Ok(effective_from(
&limits,
name,
hive_cpu_quota,
hive_memory_max,
))
}
/// [`effective`] against an already-loaded map — the multi-agent form.
///
/// `container_view::build_all` renders every agent on each SSE scan, so
/// it loads the map once and calls this per agent rather than re-reading
/// the same small file N times per scan.
#[must_use]
pub fn effective_from(
limits: &BTreeMap<String, AgentLimits>,
name: &str,
hive_cpu_quota: &str,
hive_memory_max: &str,
) -> (String, String) {
match limits.get(name) {
Some(l) => resolve(l, hive_cpu_quota, hive_memory_max),
None => (hive_cpu_quota.to_owned(), hive_memory_max.to_owned()),
}
}
/// Pure core of [`effective`], split out so the fallback matrix is
/// testable without touching the filesystem.
#[must_use]
fn resolve(limits: &AgentLimits, hive_cpu_quota: &str, hive_memory_max: &str) -> (String, String) {
let cpu = limits
.cpu_quota
.clone()
.unwrap_or_else(|| hive_cpu_quota.to_owned());
let mem = limits
.memory_max
.clone()
.unwrap_or_else(|| hive_memory_max.to_owned());
(cpu, mem)
}
/// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`, /// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`,
/// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`) /// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`)
/// into a raw byte count, accepting exactly the single-term forms systemd's /// into a raw byte count, accepting exactly the single-term forms systemd's
@ -164,35 +87,7 @@ pub fn parse_bytes(value: &str) -> Option<u64> {
u64::try_from(bytes).ok() u64::try_from(bytes).ok()
} }
/// Set one agent's overrides and persist the map atomically. An entry
/// with both fields unset is removed rather than stored, so "reset to
/// hive defaults" and "never configured" are the same state on disk.
///
/// # Errors
///
/// Fails without writing when the existing file can't be read or parsed
/// (`InvalidData` for a parse failure), and otherwise when the meta dir
/// or the file can't be written.
pub fn set_limits(name: &str, limits: &AgentLimits) -> std::io::Result<()> {
set_limits_at(&resource_limits_path(), name, limits)
}
fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Result<()> {
let mut current: BTreeMap<String, AgentLimits> = super::read_map(path)?;
if limits.is_empty() {
current.remove(name);
} else {
current.insert(name.to_owned(), limits.clone());
}
super::write_map(path, &current)
}
/// Caps the swarm controller declared for this hive's agents. /// Caps the swarm controller declared for this hive's agents.
///
/// A separate file from the operator's overrides above, not a merge into it:
/// the two have different writers and different lifetimes, and a swarm
/// declaration landing in the file an operator edits would make "reset to
/// hive defaults" destroy something the operator never set.
const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json"; const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json";
#[must_use] #[must_use]
@ -201,9 +96,9 @@ pub fn swarm_limits_path() -> PathBuf {
} }
/// Read the swarm-declared caps. An absent file is the empty map — no swarm /// Read the swarm-declared caps. An absent file is the empty map — no swarm
/// opinion, so the hive's own chain decides. A file that exists but can't be /// opinion, so the hive-wide defaults decide. A file that exists but can't be
/// read or parsed is an error, for the reason [`read`] gives: reading it as /// read or parsed is an error: a cap can be tighter than the hive default, so
/// empty would silently drop a cap. /// reading it as empty would silently drop it.
pub fn read_swarm() -> std::io::Result<BTreeMap<String, AgentLimits>> { pub fn read_swarm() -> std::io::Result<BTreeMap<String, AgentLimits>> {
super::read_map(&swarm_limits_path()) super::read_map(&swarm_limits_path())
} }
@ -216,55 +111,43 @@ pub fn write_swarm(map: &BTreeMap<String, AgentLimits>) -> std::io::Result<()> {
super::write_map(&swarm_limits_path(), map) super::write_map(&swarm_limits_path(), map)
} }
/// [`effective_from`] with the swarm's declaration on top. /// Resolve the effective values for an agent from the swarm caps in `swarm`:
/// each field the swarm declared a cap for, and the hive-wide default for
/// each it did not. This is the single place the fallback rule lives;
/// `write_dropins` calls it and passes the result straight to systemd.
/// ///
/// A swarm cap **replaces** the hive's answer for that field rather than /// Use this form when resolving more than one agent in a row.
/// joining the fallback chain as another tier: a tier a hive can out-set is
/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's
/// chain.
///
/// The chain itself is untouched — [`effective_from`] is called exactly as
/// before and its result is what a missing swarm value falls back to.
#[must_use] #[must_use]
pub fn effective_from_with_swarm( pub fn effective_from(
swarm: &BTreeMap<String, AgentLimits>, swarm: &BTreeMap<String, AgentLimits>,
limits: &BTreeMap<String, AgentLimits>,
name: &str, name: &str,
hive_cpu_quota: &str, hive_cpu_quota: &str,
hive_memory_max: &str, hive_memory_max: &str,
) -> (String, String) { ) -> (String, String) {
let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max); let caps = swarm.get(name);
match swarm.get(name) { (
Some(s) => ( caps.and_then(|c| c.cpu_quota.clone())
s.cpu_quota.clone().unwrap_or(cpu), .unwrap_or_else(|| hive_cpu_quota.to_owned()),
s.memory_max.clone().unwrap_or(mem), caps.and_then(|c| c.memory_max.clone())
), .unwrap_or_else(|| hive_memory_max.to_owned()),
None => (cpu, mem), )
}
} }
/// [`effective_from_with_swarm`] for a caller resolving a single agent, from /// [`effective_from`] for a caller resolving a single agent, from the swarm
/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`. /// caps at `swarm_path`. Errors as [`read_swarm`] does.
/// pub fn effective(
/// Delegates to [`effective`] rather than re-deriving the hive's answer, so
/// there stays exactly one place the fallback chain lives. Errors as [`read`]
/// does, for either file.
pub fn effective_with_swarm(
limits_path: &Path,
swarm_path: &Path, swarm_path: &Path,
name: &str, name: &str,
hive_cpu_quota: &str, hive_cpu_quota: &str,
hive_memory_max: &str, hive_memory_max: &str,
) -> std::io::Result<(String, String)> { ) -> std::io::Result<(String, String)> {
let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?;
let swarm: BTreeMap<String, AgentLimits> = super::read_map(swarm_path)?; let swarm: BTreeMap<String, AgentLimits> = super::read_map(swarm_path)?;
Ok(match swarm.get(name) { Ok(effective_from(
Some(s) => ( &swarm,
s.cpu_quota.clone().unwrap_or(cpu), name,
s.memory_max.clone().unwrap_or(mem), hive_cpu_quota,
), hive_memory_max,
None => (cpu, mem), ))
})
} }
/// Validate a systemd `CPUQuota=` value. Percentages only, and values /// Validate a systemd `CPUQuota=` value. Percentages only, and values
@ -386,96 +269,66 @@ mod tests {
} }
#[test] #[test]
fn unset_agent_gets_hive_defaults() { fn an_agent_the_swarm_says_nothing_about_gets_the_hive_defaults() {
let (cpu, mem) = resolve(&AgentLimits::default(), HIVE_CPU, HIVE_MEM); let swarm = BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]);
assert_eq!(cpu, "200%"); for map in [BTreeMap::new(), swarm] {
assert_eq!(mem, "4G"); assert_eq!(
effective_from(&map, "sock", HIVE_CPU, HIVE_MEM),
(HIVE_CPU.to_owned(), HIVE_MEM.to_owned())
);
}
} }
#[test] #[test]
fn both_fields_override() { fn a_swarm_cap_replaces_the_hive_default() {
let (cpu, mem) = resolve(&limits(Some("400%"), Some("8G")), HIVE_CPU, HIVE_MEM); let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
assert_eq!(cpu, "400%"); assert_eq!(
assert_eq!(mem, "8G"); effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
("150%".to_owned(), "2G".to_owned())
);
} }
/// The point of per-field fallback: overriding memory must not drag /// The point of per-field fallback: capping memory must not drag CPU
/// CPU along with it. /// along with it.
#[test] #[test]
fn partial_override_keeps_other_field_on_hive_default() { fn a_swarm_cap_is_per_field() {
let (cpu, mem) = resolve(&limits(None, Some("8G")), HIVE_CPU, HIVE_MEM); let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
assert_eq!(cpu, "200%", "cpu should still track the hive default"); assert_eq!(
assert_eq!(mem, "8G"); effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
(HIVE_CPU.to_owned(), "2G".to_owned()),
let (cpu, mem) = resolve(&limits(Some("400%"), None), HIVE_CPU, HIVE_MEM); "cpu should still track the hive default"
assert_eq!(cpu, "400%"); );
assert_eq!(mem, "4G", "memory should still track the hive default"); let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), None))]);
} assert_eq!(
effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM),
#[test] ("150%".to_owned(), HIVE_MEM.to_owned()),
fn empty_entry_is_reported_empty() { "memory should still track the hive default"
assert!(AgentLimits::default().is_empty()); );
assert!(!limits(None, Some("8G")).is_empty());
assert!(!limits(Some("400%"), None).is_empty());
} }
const TRUNCATED: &str = const TRUNCATED: &str =
"{\n \"sock\": { \"cpu_quota\": \"50%\", \"memory_max\": \"1G\" },\n \"iris\": { \"mem"; "{\n \"sock\": { \"cpu_quota\": \"50%\", \"memory_max\": \"1G\" },\n \"iris\": { \"mem";
fn corrupt_file() -> (tempfile::TempDir, PathBuf) {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(RESOURCE_LIMITS_FILE);
std::fs::write(&path, TRUNCATED).expect("seed");
(dir, path)
}
#[test] #[test]
fn effective_of_a_corrupt_file_is_an_error() { fn effective_of_a_corrupt_file_is_an_error() {
let (_dir, path) = corrupt_file(); let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(SWARM_LIMITS_FILE);
std::fs::write(&path, TRUNCATED).expect("seed");
let err = effective(&path, "sock", HIVE_CPU, HIVE_MEM) let err = effective(&path, "sock", HIVE_CPU, HIVE_MEM)
.expect_err("a corrupt file must not resolve to hive defaults"); .expect_err("a corrupt file must not resolve to hive defaults");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData); assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
} }
#[test]
fn set_limits_leaves_a_corrupt_file_untouched() {
let (_dir, path) = corrupt_file();
let err = set_limits_at(&path, "ruth", &limits(Some("400%"), None))
.expect_err("a corrupt file must not be overwritten");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert_eq!(std::fs::read(&path).expect("read"), TRUNCATED.as_bytes());
}
#[test] #[test]
fn missing_file_resolves_to_hive_defaults() { fn missing_file_resolves_to_hive_defaults() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(RESOURCE_LIMITS_FILE); let path = dir.path().join(SWARM_LIMITS_FILE);
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("missing file"); let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("missing file");
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, HIVE_MEM)); assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, HIVE_MEM));
} }
#[test]
fn set_limits_keeps_other_agents_and_leaves_no_temp_file() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(RESOURCE_LIMITS_FILE);
set_limits_at(&path, "sock", &limits(None, Some("8G"))).expect("set on missing file");
set_limits_at(&path, "iris", &limits(Some("50%"), None)).expect("set");
let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("read");
assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, "8G"));
let (cpu, _) = effective(&path, "iris", HIVE_CPU, HIVE_MEM).expect("read");
assert_eq!(cpu, "50%");
let entries: Vec<_> = std::fs::read_dir(dir.path())
.expect("read_dir")
.map(|e| e.expect("entry").file_name())
.collect();
assert_eq!(
entries,
vec![std::ffi::OsString::from(RESOURCE_LIMITS_FILE)]
);
}
/// Absent fields must deserialize to `None`, not fail — an entry /// Absent fields must deserialize to `None`, not fail — an entry
/// written by an older version with only one field must still load. /// with only one field must still load.
#[test] #[test]
fn partial_entry_deserializes() { fn partial_entry_deserializes() {
let map: BTreeMap<String, AgentLimits> = let map: BTreeMap<String, AgentLimits> =
@ -605,52 +458,4 @@ mod tests {
); );
} }
} }
#[test]
fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() {
// The whole point of the ruling: a tier a hive can out-set is not a
// cap. The agent override below is the strongest hive-side value
// there is, and the swarm still wins.
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]);
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
assert_eq!(
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
("150%".to_owned(), "2G".to_owned())
);
}
#[test]
fn a_swarm_cap_is_per_field_like_every_other_tier() {
// A swarm that caps memory and says nothing about CPU leaves CPU on
// the hive's chain — here the per-agent override, not the hive default.
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]);
let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
assert_eq!(
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
("800%".to_owned(), "2G".to_owned())
);
}
#[test]
fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() {
// The control that makes the two tests above mean something: the
// wrapper must be a no-op when the swarm has said nothing, for every
// shape the old chain distinguishes. If this drifts, the swarm tier
// changed behaviour it was explicitly told not to change.
let none = BTreeMap::new();
let cases = [
BTreeMap::new(),
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]),
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]),
BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]),
BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]),
];
for hive_side in &cases {
assert_eq!(
effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM),
effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM),
"the wrapper changed the answer with no swarm value present"
);
}
}
} }

View file

@ -76,16 +76,6 @@ pub struct ContainerView {
/// a legitimate way to keep it idle when it next boots. /// a legitimate way to keep it idle when it next boots.
#[serde(default)] #[serde(default)]
pub paused: bool, pub paused: bool,
/// Effective systemd `CPUQuota=` for this container (e.g. `"400%"`) —
/// the per-agent override from `meta/resource-limits.json` when set,
/// otherwise the hive-wide `agentCpuQuota`. Always populated: there
/// is no "unset" state to render, only "same as everyone else".
/// Reflects what the *drop-in says*, which is what the next start
/// will enforce — not a live cgroup reading.
pub cpu_quota: String,
/// Effective systemd `MemoryMax=` for this container (e.g. `"8G"`).
/// Same resolution + caveat as [`ContainerView::cpu_quota`].
pub memory_max: String,
} }
impl From<ContainerView> for hive_sh4re::container::AgentStatusRow { impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
@ -99,10 +89,9 @@ impl From<ContainerView> for hive_sh4re::container::AgentStatusRow {
/// anymore. Stubbed to `0` rather than dropping the wire field /// anymore. Stubbed to `0` rather than dropping the wire field
/// outright — leaves `hivectl status`/the dashboard column intact /// outright — leaves `hivectl status`/the dashboard column intact
/// syntactically, just always empty, until iris's frontend follow-up /// syntactically, just always empty, until iris's frontend follow-up
/// decides whether to drop the column entirely. `port`, `cpu_quota`, /// decides whether to drop the column entirely. `port` has no
/// `memory_max` have no equivalent on the row at all and are simply /// equivalent on the row at all and is simply not carried over. `url`
/// not carried over. `url` has no source on `ContainerView` at all — /// has no source on `ContainerView` at all — see [`agent_url`].
/// see [`agent_url`].
fn from(v: ContainerView) -> Self { fn from(v: ContainerView) -> Self {
Self { Self {
url: agent_url(&v.name), url: agent_url(&v.name),
@ -148,37 +137,12 @@ fn agent_url_for_domain(domain: &str, name: &str) -> String {
/// Build the full container list. Wraps `lifecycle::list()` and /// Build the full container list. Wraps `lifecycle::list()` and
/// resolves every per-agent attribute the dashboard surfaces. /// resolves every per-agent attribute the dashboard surfaces.
/// ///
/// Takes `hive` because the effective resource limits are a per-field
/// fallback onto the hive-wide `agent_cpu_quota` / `agent_memory_max`,
/// and those live on [`crate::coordinator::HiveEnv`], not on disk. Both callers already
/// hold a `Coordinator`, so this is a parameter rather than a global.
///
/// # Errors /// # Errors
/// ///
/// The container list could not be read. /// The container list could not be read.
pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec<ContainerView>> { pub async fn build_all() -> anyhow::Result<Vec<ContainerView>> {
let raw = lifecycle::list().await?; let raw = lifecycle::list().await?;
let locked = read_meta_locked_revs(); let locked = read_meta_locked_revs();
// Read once per scan rather than re-read for every container on every
// SSE scan. An unreadable file shows every row at the hive defaults;
// the error log is the only signal, as `ContainerView` has no field
// to carry it.
let limits = crate::resource_limits::read().unwrap_or_else(|e| {
tracing::error!(
path = %crate::resource_limits::resource_limits_path().display(),
error = ?e,
"resource limits unreadable — container rows show hive defaults"
);
std::collections::BTreeMap::new()
});
let swarm_limits = crate::resource_limits::read_swarm().unwrap_or_else(|e| {
tracing::error!(
path = %crate::resource_limits::swarm_limits_path().display(),
error = ?e,
"swarm resource caps unreadable — container rows ignore them"
);
std::collections::BTreeMap::new()
});
let mut out = Vec::new(); let mut out = Vec::new();
for c in &raw { for c in &raw {
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else { let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
@ -243,13 +207,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
(None, None) (None, None)
}; };
let paused = Coordinator::is_paused(&logical); let paused = Coordinator::is_paused(&logical);
let (cpu_quota, memory_max) = crate::resource_limits::effective_from_with_swarm(
&swarm_limits,
&limits,
logical.as_str(),
&hive.agent_cpu_quota,
&hive.agent_memory_max,
);
out.push(ContainerView { out.push(ContainerView {
port: lifecycle::agent_web_port(logical.as_str()), port: lifecycle::agent_web_port(logical.as_str()),
running, running,
@ -263,8 +220,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
status_text, status_text,
status_set_at, status_set_at,
paused, paused,
cpu_quota,
memory_max,
}); });
} }
Ok(out) Ok(out)
@ -570,9 +525,8 @@ mod tests {
/// The `From` impl both `handle_agent_status`'s poll path and /// The `From` impl both `handle_agent_status`'s poll path and
/// `stream_agent_status`'s push path go through — pins the field /// `stream_agent_status`'s push path go through — pins the field
/// mapping so the two can't quietly diverge, and that `port` / /// mapping so the two can't quietly diverge, and that `port` (no
/// `cpu_quota` / `memory_max` (no equivalent on the row) are dropped /// equivalent on the row) is dropped on purpose rather than by omission.
/// on purpose rather than by omission.
#[test] #[test]
fn agent_status_row_carries_every_field_the_row_has_and_drops_the_rest() { fn agent_status_row_carries_every_field_the_row_has_and_drops_the_rest() {
use super::ContainerView; use super::ContainerView;
@ -592,8 +546,6 @@ mod tests {
status_text: Some("shipping".to_owned()), status_text: Some("shipping".to_owned()),
status_set_at: Some(Utc.timestamp_opt(1_700_000_000, 0).unwrap()), status_set_at: Some(Utc.timestamp_opt(1_700_000_000, 0).unwrap()),
paused: true, paused: true,
cpu_quota: "400%".to_owned(),
memory_max: "8G".to_owned(),
}; };
let row = AgentStatusRow::from(view); let row = AgentStatusRow::from(view);

View file

@ -720,7 +720,7 @@ impl Coordinator {
pub async fn rescan_containers_and_emit(self: &Arc<Self>) { pub async fn rescan_containers_and_emit(self: &Arc<Self>) {
// An unreadable list says nothing about which containers exist, so // An unreadable list says nothing about which containers exist, so
// the last snapshot stands rather than every row being reported gone. // the last snapshot stands rather than every row being reported gone.
let fresh = match container_view::build_all(&self.hive_env()).await { let fresh = match container_view::build_all().await {
Ok(fresh) => fresh, Ok(fresh) => fresh,
Err(e) => { Err(e) => {
tracing::warn!(error = ?e, "container rescan: list failed; keeping last snapshot"); tracing::warn!(error = ?e, "container rescan: list failed; keeping last snapshot");

View file

@ -311,95 +311,6 @@ fn set_paused_error_response(
} }
} }
/// Form fields for `post_resource_limits`. Both fields are optional strings;
/// an empty value clears the per-agent override for that field, falling back
/// to the hive-wide default.
#[derive(Deserialize, Default, ToSchema)]
pub(super) struct ResourceLimitsForm {
#[serde(default)]
cpu_quota: String,
#[serde(default)]
memory_max: String,
}
/// Write per-agent CPU/memory limit
/// overrides for `name`.
///
/// An empty `cpu_quota` or `memory_max` field clears that field's override,
/// falling back to the hive-wide default. Both empty together removes the
/// agent's entry entirely. The new drop-in is written immediately — the
/// limits take effect on the next container start or restart. Triggers an
/// immediate rescan so `ContainerView.cpu_quota`/`memory_max` update on
/// the dashboard via SSE without waiting for the next periodic sweep.
#[utoipa::path(
post,
path = "/api/resource-limits/{name}",
params(("name" = String, Path, description = "agent name")),
request_body(content = ResourceLimitsForm, content_type = "application/x-www-form-urlencoded"),
responses(
(status = 200, description = "limits written", body = String),
(status = 400, description = "bad agent name"),
(status = 404, description = "no such agent"),
(status = 422, description = "invalid cpu_quota/memory_max value"),
(status = 500, description = "commit or drop-in write failed"),
),
tag = "lifecycle_ops"
)]
pub(super) async fn post_resource_limits(
State(state): State<AppState>,
AxumPath(name): AxumPath<String>,
Form(form): Form<ResourceLimitsForm>,
) -> Response {
let logical = strip_container_prefix(&name);
if let Some(reject) = guard_agent_name(&state, &logical).await {
return reject;
}
let ident = match Ident::parse(&logical) {
Ok(i) => i,
Err(e) => return (StatusCode::BAD_REQUEST, format!("bad agent name: {e}")).into_response(),
};
let cpu_quota = if form.cpu_quota.is_empty() {
None
} else {
Some(form.cpu_quota.as_str())
};
let memory_max = if form.memory_max.is_empty() {
None
} else {
Some(form.memory_max.as_str())
};
if let Some(v) = cpu_quota
&& let Err(e) = crate::resource_limits::validate_cpu_quota(v)
{
return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response();
}
// Returns the value to store, not just an ok/err verdict — see
// `validate_memory_max`'s own doc for why that isn't always `v`.
let memory_max = match memory_max.map(crate::resource_limits::validate_memory_max) {
Some(Err(e)) => return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response(),
Some(Ok(v)) => Some(v),
None => None,
};
let limits = crate::resource_limits::AgentLimits {
cpu_quota: cpu_quota.map(str::to_owned),
memory_max,
};
if let Err(e) = crate::meta::commit_resource_limits(ident.as_str(), &limits).await {
return error_response(&format!("set limits {logical}: {e:#}"));
}
let agent_dir = crate::paths::agent_runtime_dir(ident.as_str());
let hive = state.coord.hive_env();
let paths = match crate::coordinator::Coordinator::agent_paths(ident.as_str(), agent_dir) {
Ok(p) => p,
Err(e) => return error_response(&format!("write_dropins {logical}: {e:#}")),
};
if let Err(e) = crate::lifecycle::write_dropins(ident.as_str(), &hive, &paths).await {
return error_response(&format!("write_dropins {logical}: {e:#}"));
}
state.coord.rescan_containers_and_emit().await;
(StatusCode::OK, "ok").into_response()
}
/// Queue a rebuild DAG for every live agent /// Queue a rebuild DAG for every live agent
/// container. /// container.
#[utoipa::path( #[utoipa::path(

View file

@ -1,6 +1,5 @@
//! Remaining single-endpoint dashboard handlers: mark-all-read, operator //! Remaining single-endpoint dashboard handlers: mark-all-read, operator
//! compose (`op-send`), spawn-request, hive-wide turn stats, and container //! compose (`op-send`), spawn-request and hive-wide turn stats.
//! resources.
use axum::{ use axum::{
extract::{Form, Path as AxumPath, State}, extract::{Form, Path as AxumPath, State},
@ -11,7 +10,6 @@ use serde::{Deserialize, Serialize};
use utoipa::{IntoParams, ToSchema}; use utoipa::{IntoParams, ToSchema};
use super::{AppState, Ident, error_response}; use super::{AppState, Ident, error_response};
use crate::container_stats::ContainerResource;
use crate::hive_stats::HiveStats; use crate::hive_stats::HiveStats;
#[derive(Deserialize, IntoParams)] #[derive(Deserialize, IntoParams)]
@ -43,20 +41,6 @@ pub(super) async fn api_stats_hive(
.into_response() .into_response()
} }
/// Live per-agent-container CPU + memory load from cgroup v2.
///
/// Samples CPU over a short interval (~200 ms), so this call briefly
/// awaits.
#[utoipa::path(
get,
path = "/api/container-resources",
responses((status = 200, description = "live per-container CPU + memory load", body = Vec<ContainerResource>)),
tag = "misc_api"
)]
pub(super) async fn api_container_resources() -> Response {
axum::Json(crate::container_stats::gather().await).into_response()
}
#[derive(Serialize, ToSchema)] #[derive(Serialize, ToSchema)]
pub(super) struct MarkAllReadBody { pub(super) struct MarkAllReadBody {
marked: u64, marked: u64,

View file

@ -116,7 +116,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
.routes(routes!(state_snapshot::api_state)) .routes(routes!(state_snapshot::api_state))
.routes(routes!(state_files::get_state_file)) .routes(routes!(state_files::get_state_file))
.routes(routes!(misc_api::api_stats_hive)) .routes(routes!(misc_api::api_stats_hive))
.routes(routes!(misc_api::api_container_resources))
.routes(routes!(misc_api::post_mark_all_read)) .routes(routes!(misc_api::post_mark_all_read))
.routes(routes!(misc_api::post_op_send)) .routes(routes!(misc_api::post_op_send))
.routes(routes!(build_logs::get_build_logs_all)) .routes(routes!(build_logs::get_build_logs_all))
@ -140,7 +139,6 @@ fn app(coord: Arc<Coordinator>) -> axum::Router {
.routes(routes!(lifecycle_ops::post_rebuild)) .routes(routes!(lifecycle_ops::post_rebuild))
.routes(routes!(lifecycle_ops::post_pause)) .routes(routes!(lifecycle_ops::post_pause))
.routes(routes!(lifecycle_ops::post_resume)) .routes(routes!(lifecycle_ops::post_resume))
.routes(routes!(lifecycle_ops::post_resource_limits))
.routes(routes!(lifecycle_ops::post_update_all)) .routes(routes!(lifecycle_ops::post_update_all))
.routes(routes!(tombstones::post_purge_tombstone)) .routes(routes!(tombstones::post_purge_tombstone))
.routes(routes!(meta_inputs::post_meta_update)) .routes(routes!(meta_inputs::post_meta_update))
@ -372,7 +370,6 @@ mod router_build_probe {
.routes(routes!(state_snapshot::api_state)) .routes(routes!(state_snapshot::api_state))
.routes(routes!(state_files::get_state_file)) .routes(routes!(state_files::get_state_file))
.routes(routes!(misc_api::api_stats_hive)) .routes(routes!(misc_api::api_stats_hive))
.routes(routes!(misc_api::api_container_resources))
.routes(routes!(misc_api::post_mark_all_read)) .routes(routes!(misc_api::post_mark_all_read))
.routes(routes!(misc_api::post_op_send)) .routes(routes!(misc_api::post_op_send))
.routes(routes!(build_logs::get_build_logs_all)) .routes(routes!(build_logs::get_build_logs_all))
@ -396,7 +393,6 @@ mod router_build_probe {
.routes(routes!(lifecycle_ops::post_rebuild)) .routes(routes!(lifecycle_ops::post_rebuild))
.routes(routes!(lifecycle_ops::post_pause)) .routes(routes!(lifecycle_ops::post_pause))
.routes(routes!(lifecycle_ops::post_resume)) .routes(routes!(lifecycle_ops::post_resume))
.routes(routes!(lifecycle_ops::post_resource_limits))
.routes(routes!(lifecycle_ops::post_update_all)) .routes(routes!(lifecycle_ops::post_update_all))
.routes(routes!(tombstones::post_purge_tombstone)) .routes(routes!(tombstones::post_purge_tombstone))
.routes(routes!(meta_inputs::post_meta_update)) .routes(routes!(meta_inputs::post_meta_update))

View file

@ -603,8 +603,6 @@ mod tests {
status_text: None, status_text: None,
status_set_at: None, status_set_at: None,
paused: false, paused: false,
cpu_quota: "200%".to_owned(),
memory_max: "4G".to_owned(),
} }
} }

View file

@ -27,7 +27,6 @@ pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Re
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?; set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
if let Some((cpu_quota, memory_max)) = stage_limits( if let Some((cpu_quota, memory_max)) = stage_limits(
name, name,
&crate::resource_limits::resource_limits_path(),
&crate::resource_limits::swarm_limits_path(), &crate::resource_limits::swarm_limits_path(),
&limits_dropin_path(&container), &limits_dropin_path(&container),
&paths.agent, &paths.agent,
@ -97,14 +96,13 @@ impl HostMemory {
/// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's. /// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's.
fn stage_limits( fn stage_limits(
agent_name: &str, agent_name: &str,
limits_path: &Path,
swarm_path: &Path, swarm_path: &Path,
dropin_path: &Path, dropin_path: &Path,
runtime_dir: &Path, runtime_dir: &Path,
hive: &HiveEnv, hive: &HiveEnv,
host: Option<HostMemory>, host: Option<HostMemory>,
) -> Result<Option<(String, String)>> { ) -> Result<Option<(String, String)>> {
let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive); let limits = limits_to_apply(agent_name, swarm_path, dropin_path, hive);
let memory_max = match &limits { let memory_max = match &limits {
Some((_, memory_max)) => Some(memory_max.clone()), Some((_, memory_max)) => Some(memory_max.clone()),
None => dropin_memory_max(agent_name, dropin_path), None => dropin_memory_max(agent_name, dropin_path),
@ -226,35 +224,27 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits /// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is. /// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
/// ///
/// When the overrides file at `limits_path` or the swarm caps at /// When the swarm caps at `swarm_path` can't be read, the error is logged and
/// `swarm_path` can't be read, the error is logged and the agent still starts. /// the agent still starts. The last-applied drop-in is kept, because the hive
/// The last-applied drop-in is kept, because the hive defaults can be looser /// defaults can be looser than the swarm's cap.
/// than the agent's own override.
/// With no drop-in yet, the agent gets the hive defaults: without one it /// With no drop-in yet, the agent gets the hive defaults: without one it
/// would run uncapped. /// would run uncapped.
fn limits_to_apply( fn limits_to_apply(
agent_name: &str, agent_name: &str,
limits_path: &Path,
swarm_path: &Path, swarm_path: &Path,
dropin_path: &Path, dropin_path: &Path,
hive: &HiveEnv, hive: &HiveEnv,
) -> Option<(String, String)> { ) -> Option<(String, String)> {
let hive_cpu = &hive.agent_cpu_quota; let hive_cpu = &hive.agent_cpu_quota;
let hive_mem = &hive.agent_memory_max; let hive_mem = &hive.agent_memory_max;
let e = match crate::resource_limits::effective_with_swarm( let e = match crate::resource_limits::effective(swarm_path, agent_name, hive_cpu, hive_mem) {
limits_path,
swarm_path,
agent_name,
hive_cpu,
hive_mem,
) {
Ok(limits) => return Some(limits), Ok(limits) => return Some(limits),
Err(e) => e, Err(e) => e,
}; };
if dropin_path.exists() { if dropin_path.exists() {
tracing::error!( tracing::error!(
agent = %agent_name, agent = %agent_name,
path = %limits_path.display(), path = %swarm_path.display(),
error = ?e, error = ?e,
"resource limits unreadable — keeping the last-applied limits drop-in" "resource limits unreadable — keeping the last-applied limits drop-in"
); );
@ -262,7 +252,7 @@ fn limits_to_apply(
} else { } else {
tracing::error!( tracing::error!(
agent = %agent_name, agent = %agent_name,
path = %limits_path.display(), path = %swarm_path.display(),
error = ?e, error = ?e,
"resource limits unreadable and no limits drop-in yet — applying hive defaults" "resource limits unreadable and no limits drop-in yet — applying hive defaults"
); );
@ -271,14 +261,14 @@ fn limits_to_apply(
} }
/// Write a systemd drop-in for `container@<container>.service` that applies /// Write a systemd drop-in for `container@<container>.service` that applies
/// the agent's effective resource caps — its per-agent overrides from /// the agent's effective resource caps — the swarm's cap from
/// `meta/resource-limits.json` where set, the hive-wide defaults /// `meta/swarm-resource-limits.json` where set, the hive-wide defaults
/// otherwise. Goes under `/run/systemd/system/...` so it's ephemeral /// otherwise. Goes under `/run/systemd/system/...` so it's ephemeral
/// (regenerated on every spawn / rebuild, except as [`limits_to_apply`] says). /// (regenerated on every spawn / rebuild, except as [`limits_to_apply`] says).
/// ///
/// The weights are hive-wide (`services.hyperhive.agentCpuWeight` / /// The weights are hive-wide (`services.hyperhive.agentCpuWeight` /
/// `agentIoWeight`) — unlike the caps they have no per-agent override in /// `agentIoWeight`) — unlike the caps they have no swarm cap, so they come
/// `meta/resource-limits.json`, so they come straight off [`HiveEnv`]. /// straight off [`HiveEnv`].
/// `None` (the nix option set to `null`) means the weight line is left out /// `None` (the nix option set to `null`) means the weight line is left out
/// and the container keeps the kernel default. /// and the container keeps the kernel default.
async fn set_resource_limits( async fn set_resource_limits(
@ -848,42 +838,35 @@ mod tests {
const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem"; const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem";
fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf { fn swarm_file(dir: &tempfile::TempDir) -> std::path::PathBuf {
dir.path().join("absent-swarm-resource-limits.json") dir.path().join("swarm-resource-limits.json")
} }
#[test] #[test]
fn corrupt_limits_keep_an_existing_dropin() { fn corrupt_swarm_caps_keep_an_existing_dropin() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
let dropin = dir.path().join("hyperhive-limits.conf"); let dropin = dir.path().join("hyperhive-limits.conf");
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
assert_eq!( assert_eq!(
limits_to_apply( limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default()),
"sock",
&limits,
&absent_swarm(&dir),
&dropin,
&HiveEnv::default()
),
None None
); );
assert_eq!( assert_eq!(
std::fs::read(&limits).expect("read"), std::fs::read(&swarm).expect("read"),
TRUNCATED_LIMITS.as_bytes() TRUNCATED_LIMITS.as_bytes()
); );
} }
#[test] #[test]
fn corrupt_limits_without_a_dropin_apply_hive_defaults() { fn corrupt_swarm_caps_without_a_dropin_apply_hive_defaults() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
let applied = limits_to_apply( let applied = limits_to_apply(
"sock", "sock",
&limits, &swarm,
&absent_swarm(&dir),
&dir.path().join("absent.conf"), &dir.path().join("absent.conf"),
&HiveEnv::default(), &HiveEnv::default(),
); );
@ -891,32 +874,12 @@ mod tests {
} }
#[test] #[test]
fn corrupt_swarm_caps_keep_an_existing_dropin() { fn a_swarm_cap_is_applied_over_the_hive_default() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let swarm = dir.path().join("swarm-resource-limits.json"); let swarm = swarm_file(&dir);
let dropin = dir.path().join("hyperhive-limits.conf");
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
let applied = limits_to_apply(
"sock",
&dir.path().join("resource-limits.json"),
&swarm,
&dropin,
&HiveEnv::default(),
);
assert_eq!(applied, None);
}
#[test]
fn a_swarm_cap_is_applied_over_the_operator_override() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
let swarm = dir.path().join("swarm-resource-limits.json");
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps"); std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps");
let applied = limits_to_apply( let applied = limits_to_apply(
"sock", "sock",
&limits,
&swarm, &swarm,
&dir.path().join("absent.conf"), &dir.path().join("absent.conf"),
&HiveEnv::default(), &HiveEnv::default(),
@ -925,19 +888,13 @@ mod tests {
} }
#[test] #[test]
fn readable_limits_are_applied_over_an_existing_dropin() { fn readable_swarm_caps_are_applied_over_an_existing_dropin() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
let dropin = dir.path().join("hyperhive-limits.conf"); let dropin = dir.path().join("hyperhive-limits.conf");
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits"); std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps");
std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in"); std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in");
let applied = limits_to_apply( let applied = limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default());
"sock",
&limits,
&absent_swarm(&dir),
&dropin,
&HiveEnv::default(),
);
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned()))); assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
} }
@ -953,8 +910,7 @@ mod tests {
let runtime = dir.path().join("runtime"); let runtime = dir.path().join("runtime");
let applied = stage_limits( let applied = stage_limits(
"sock", "sock",
&dir.path().join("resource-limits.json"), &swarm_file(dir),
&dir.path().join("swarm-resource-limits.json"),
&dir.path().join("hyperhive-limits.conf"), &dir.path().join("hyperhive-limits.conf"),
&runtime, &runtime,
&HiveEnv::default(), &HiveEnv::default(),
@ -965,40 +921,18 @@ mod tests {
(applied.map(|(_, mem)| mem), copy) (applied.map(|(_, mem)| mem), copy)
} }
/// The agent's copy follows the swarm's cap, the one systemd enforces, over
/// the operator's override: a heap sized against the override would
/// outgrow the cgroup.
#[test]
fn the_agent_reads_the_swarm_cap_over_the_operator_override() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(
dir.path().join("resource-limits.json"),
r#"{"sock": {"memory_max": "16G"}}"#,
)
.expect("seed limits");
std::fs::write(
dir.path().join("swarm-resource-limits.json"),
r#"{"sock": {"memory_max": "2G"}}"#,
)
.expect("seed swarm caps");
assert_eq!(
stage(&dir),
(Some("2G".to_owned()), Some("2147483648\n".to_owned()))
);
}
#[test] #[test]
fn the_agent_reads_the_cap_its_dropin_carries() { fn the_agent_reads_the_cap_its_dropin_carries() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
// No overrides anywhere: the hive-wide default, 4G. // No swarm cap anywhere: the hive-wide default, 4G.
assert_eq!( assert_eq!(
stage(&dir), stage(&dir),
(Some("4G".to_owned()), Some("4294967296\n".to_owned())) (Some("4G".to_owned()), Some("4294967296\n".to_owned()))
); );
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits"); std::fs::write(&swarm, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed swarm caps");
assert_eq!( assert_eq!(
stage(&dir), stage(&dir),
(Some("16G".to_owned()), Some("17179869184\n".to_owned())) (Some("16G".to_owned()), Some("17179869184\n".to_owned()))
@ -1008,17 +942,17 @@ mod tests {
#[test] #[test]
fn every_kind_of_cap_is_written_for_the_agent() { fn every_kind_of_cap_is_written_for_the_agent() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
for (memory_max, copy) in [ for (memory_max, copy) in [
("infinity", "infinity\n"), ("infinity", "infinity\n"),
("50%", "8192004096\n"), ("50%", "8192004096\n"),
("8G", "8589934592\n"), ("8G", "8589934592\n"),
] { ] {
std::fs::write( std::fs::write(
&limits, &swarm,
format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#), format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#),
) )
.expect("seed limits"); .expect("seed swarm caps");
assert_eq!( assert_eq!(
stage(&dir), stage(&dir),
(Some(memory_max.to_owned()), Some(copy.to_owned())), (Some(memory_max.to_owned()), Some(copy.to_owned())),
@ -1072,14 +1006,14 @@ mod tests {
#[test] #[test]
fn a_kept_dropin_hands_the_agent_its_cap() { fn a_kept_dropin_hands_the_agent_its_cap() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
let dropin = dir.path().join("hyperhive-limits.conf"); let dropin = dir.path().join("hyperhive-limits.conf");
let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE); let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE);
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits"); std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps");
assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned())); assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned()));
// Unreadable limits keep the drop-in; the copy follows the drop-in. // Unreadable caps keep the drop-in; the copy follows the drop-in.
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
std::fs::write( std::fs::write(
&dropin, &dropin,
"[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n", "[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n",
@ -1103,28 +1037,20 @@ mod tests {
#[test] #[test]
fn an_unknown_cap_leaves_no_copy() { fn an_unknown_cap_leaves_no_copy() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json"); let swarm = swarm_file(&dir);
let dropin = dir.path().join("hyperhive-limits.conf"); let dropin = dir.path().join("hyperhive-limits.conf");
assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned())); assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned()));
// Unreadable limits and an unreadable drop-in: nothing says what the // Unreadable caps and an unreadable drop-in: nothing says what the
// cgroup enforces. // cgroup enforces.
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
std::fs::create_dir(&dropin).expect("seed unreadable drop-in"); std::fs::create_dir(&dropin).expect("seed unreadable drop-in");
assert_eq!(stage(&dir), (None, None)); assert_eq!(stage(&dir), (None, None));
// A percentage with no host memory to resolve it against. // A percentage with no host memory to resolve it against.
std::fs::write(&limits, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed limits"); std::fs::write(&swarm, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed swarm caps");
let runtime = dir.path().join("runtime"); let runtime = dir.path().join("runtime");
let applied = stage_limits( let applied = stage_limits("sock", &swarm, &dropin, &runtime, &HiveEnv::default(), None)
"sock",
&limits,
&dir.path().join("swarm-resource-limits.json"),
&dropin,
&runtime,
&HiveEnv::default(),
None,
)
.expect("stage limits"); .expect("stage limits");
assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned())); assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned()));
assert!(!runtime.join(MEMORY_CAP_FILE).exists()); assert!(!runtime.join(MEMORY_CAP_FILE).exists());

View file

@ -47,8 +47,8 @@ pub struct AgentSpec {
/// Stage every generated meta JSON file that exists: topology.json is /// Stage every generated meta JSON file that exists: topology.json is
/// regenerated by `reconcile` whenever the agent set changed; /// regenerated by `reconcile` whenever the agent set changed;
/// tool-groups/capabilities/resource-limits are created lazily on /// tool-groups/capabilities are created lazily on
/// first write (`set_groups`/`set_caps`/`set_limits`) — absent means /// first write (`set_groups`/`set_caps`) — absent means
/// every agent is on defaults, no file needed. Without staging, an /// every agent is on defaults, no file needed. Without staging, an
/// existing-but-untracked file shows up as untracked in the meta repo, /// existing-but-untracked file shows up as untracked in the meta repo,
/// which can confuse nix's dirty-tree fetch. `git add` is a no-op when /// which can confuse nix's dirty-tree fetch. `git add` is a no-op when
@ -61,10 +61,6 @@ async fn stage_generated_meta_files(dir: &std::path::Path) -> Result<()> {
crate::capabilities::capabilities_path(), crate::capabilities::capabilities_path(),
"capabilities.json", "capabilities.json",
), ),
(
crate::resource_limits::resource_limits_path(),
"resource-limits.json",
),
] { ] {
if path.exists() { if path.exists() {
git(dir, &["add", name]).await?; git(dir, &["add", name]).await?;
@ -202,7 +198,6 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> {
f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"), f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"),
"topology.json" => Some("topology"), "topology.json" => Some("topology"),
"capabilities.json" => Some("capabilities"), "capabilities.json" => Some("capabilities"),
"resource-limits.json" => Some("resource-limits"),
"tool-groups.json" => Some("tool-groups"), "tool-groups.json" => Some("tool-groups"),
_ => None, _ => None,
}) })
@ -352,41 +347,6 @@ pub async fn commit_capabilities(agent: &str, caps: &[String]) -> Result<()> {
Ok(()) Ok(())
} }
/// Write the resource-limits file for `agent` and commit it atomically
/// under `META_LOCK`. Same rationale as `commit_tool_groups`: the
/// working tree must never be left dirty for the next `sync_agents` to trip
/// over.
///
/// Unlike the perm files this one is never injected into the container —
/// it's a host-side cap *on* the agent — but it lives in the same repo
/// so a limit change gets the same auditable one-commit-per-change trail.
///
/// # Errors
///
/// Returns an error if writing the JSON file fails or a git stage/commit
/// step fails.
pub async fn commit_resource_limits(
agent: &str,
limits: &crate::resource_limits::AgentLimits,
) -> Result<()> {
let _guard = META_LOCK.lock().await;
crate::resource_limits::set_limits(agent, limits)
.map_err(|e| anyhow::anyhow!("set resource limits for {agent}: {e}"))?;
let dir = crate::paths::meta_root();
if crate::resource_limits::resource_limits_path().exists() {
git(&dir, &["add", "resource-limits.json"]).await?;
}
if paths_dirty(&dir, &["resource-limits.json"]).await? {
git_commit_paths(
&dir,
&format!("set resource limits for {agent}"),
&["resource-limits.json"],
)
.await?;
}
Ok(())
}
/// Write both perm files for `agent` (whichever are `Some`) and commit /// Write both perm files for `agent` (whichever are `Some`) and commit
/// them in a SINGLE git commit under `META_LOCK` — the batch /// them in a SINGLE git commit under `META_LOCK` — the batch
/// `POST /api/permissions` path. A `None` field leaves that file /// `POST /api/permissions` path. A `None` field leaves that file

View file

@ -170,7 +170,7 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
// question is only answerable on this side of the socket — // question is only answerable on this side of the socket —
// see the request's doc comment for why the client asks. // see the request's doc comment for why the client asks.
HostRequest::AgentExists { name } => HostResponse::agent_exists(agent_exists(name)?), HostRequest::AgentExists { name } => HostResponse::agent_exists(agent_exists(name)?),
HostRequest::AgentStatus => handle_agent_status(&coord).await, HostRequest::AgentStatus => handle_agent_status().await,
// Intercepted in `handle()` before a request ever reaches // Intercepted in `handle()` before a request ever reaches
// `dispatch` — see `stream_agent_status`'s doc comment. This // `dispatch` — see `stream_agent_status`'s doc comment. This
// arm exists only so the match stays exhaustive; reaching it // arm exists only so the match stays exhaustive; reaching it
@ -184,19 +184,6 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
// federation peer-config block) and open the web surfaces // federation peer-config block) and open the web surfaces
// (`hivectl open`). // (`hivectl open`).
HostRequest::Urls => HostResponse::urls(hive_urls()), HostRequest::Urls => HostResponse::urls(hive_urls()),
HostRequest::SetResourceLimits {
name,
cpu_quota,
memory_max,
} => {
handle_set_resource_limits(
&coord,
name,
cpu_quota.as_deref(),
memory_max.as_deref(),
)
.await?
}
HostRequest::MatrixInvite { user, room } => { HostRequest::MatrixInvite { user, room } => {
handle_matrix_invite(user, room.as_deref()).await? handle_matrix_invite(user, room.as_deref()).await?
} }
@ -293,8 +280,8 @@ async fn handle_set_paused(
} }
/// Collect per-agent status rows for `hivectl status` and the dashboard. /// Collect per-agent status rows for `hivectl status` and the dashboard.
async fn handle_agent_status(coord: &Arc<Coordinator>) -> HostResponse { async fn handle_agent_status() -> HostResponse {
match crate::container_view::build_all(&coord.hive_env()).await { match crate::container_view::build_all().await {
Ok(views) => HostResponse::agent_statuses( Ok(views) => HostResponse::agent_statuses(
views views
.into_iter() .into_iter()
@ -380,66 +367,6 @@ fn agent_exists(name: &hive_types::Ident) -> Result<bool> {
.with_context(|| format!("check agent state dir for {name}")) .with_context(|| format!("check agent state dir for {name}"))
} }
/// Validate + persist an agent's CPU/memory overrides, then re-apply the
/// drop-in so the change lands without waiting for a rebuild.
///
/// Validation is here rather than only in `hivectl` because the values
/// are written verbatim into the systemd drop-in: a malformed
/// `CPUQuota=` makes systemd reject the unit, and the container stops
/// starting. Every client (CLI, dashboard, anything later) goes through
/// this path, so the guard belongs on this side of the socket.
///
/// `None`/`None` removes the agent's entry, returning it to the
/// hive-wide defaults.
async fn handle_set_resource_limits(
coord: &Arc<Coordinator>,
name: &hive_types::Ident,
cpu_quota: Option<&str>,
memory_max: Option<&str>,
) -> Result<HostResponse> {
if let Some(value) = cpu_quota {
crate::resource_limits::validate_cpu_quota(value).map_err(anyhow::Error::msg)?;
}
// `validate_memory_max` returns the value to store, not just an
// ok/err verdict: a friendly spelling like "8GB" is accepted but
// normalized to the "8G" systemd's own parser actually takes.
let memory_max = memory_max
.map(crate::resource_limits::validate_memory_max)
.transpose()
.map_err(anyhow::Error::msg)?;
tracing::info!(%name, ?cpu_quota, ?memory_max, "set_resource_limits");
let limits = crate::resource_limits::AgentLimits {
cpu_quota: cpu_quota.map(ToOwned::to_owned),
memory_max,
};
// Goes through `meta::commit_resource_limits`, not the bare
// `resource_limits::set_limits`: the write has to be staged +
// committed under `META_LOCK` or it leaves the meta working tree
// dirty for the next `sync_agents` to trip over.
crate::meta::commit_resource_limits(name.as_str(), &limits).await?;
// Re-apply the drop-in straight away — same three lines as the job
// queue's `WriteDropin` node. Without this the new values would sit
// in the JSON until the agent's next spawn or rebuild.
let agent_dir = crate::paths::agent_runtime_dir(name.as_str());
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?;
crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?;
let (cpu, mem) = crate::resource_limits::effective_with_swarm(
&crate::resource_limits::resource_limits_path(),
&crate::resource_limits::swarm_limits_path(),
name.as_str(),
&hive.agent_cpu_quota,
&hive.agent_memory_max,
)?;
Ok(HostResponse::messages(vec![format!(
"{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now; the \
Claude/JSC heap ceiling and the subagent MemoryHigh= derived from it follow at the \
container's next start (restart it if it's running and needs them now)"
)]))
}
/// Guard: matrix provisioning needs a homeserver to provision against. /// Guard: matrix provisioning needs a homeserver to provision against.
/// ///
/// Answers "is one configured", not "is one running here": the address is /// Answers "is one configured", not "is one running here": the address is

View file

@ -234,8 +234,6 @@ mod tests {
status_text: None, status_text: None,
status_set_at: None, status_set_at: None,
paused: false, paused: false,
cpu_quota: "200%".to_owned(),
memory_max: "4G".to_owned(),
} }
} }

View file

@ -3,7 +3,7 @@
//! swarm-controller keeps one key per agent in the `agent-limits` bucket //! swarm-controller keeps one key per agent in the `agent-limits` bucket
//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads //! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads
//! the key of each agent this hive has and stores what it finds in //! the key of each agent this hive has and stores what it finds in
//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it. //! `swarm-resource-limits.json`, where the drop-in reads it.
//! A running agent gets a changed cap at its first spawn or rebuild after //! A running agent gets a changed cap at its first spawn or rebuild after
//! that read. //! that read.
//! //!
@ -92,9 +92,9 @@ async fn sync(client: &async_nats::Client) -> Result<()> {
/// The caps to store for `declared`, by agent. An agent with no field set /// The caps to store for `declared`, by agent. An agent with no field set
/// gets no entry. /// gets no entry.
/// ///
/// Every value goes through the validators the operator's own input path /// Every value goes through the `resource_limits` validators, because it is
/// uses, because it is rendered verbatim into the container's systemd /// rendered verbatim into the container's systemd drop-in. One bad value
/// drop-in. One bad value refuses the whole set. /// refuses the whole set.
fn validated(declared: &BTreeMap<String, AgentCaps>) -> Result<BTreeMap<String, AgentLimits>> { fn validated(declared: &BTreeMap<String, AgentCaps>) -> Result<BTreeMap<String, AgentLimits>> {
let mut out = BTreeMap::new(); let mut out = BTreeMap::new();
for (agent, caps) in declared { for (agent, caps) in declared {
@ -127,48 +127,38 @@ mod tests {
use std::collections::BTreeMap; use std::collections::BTreeMap;
use super::validated; use super::validated;
use crate::resource_limits::AgentLimits;
use swarm_queue_client::agent_limits::AgentCaps; use swarm_queue_client::agent_limits::AgentCaps;
/// `atlas`'s answer with `atlas_key` as the value read for it, over a hive /// `atlas`'s answer with `atlas_key` as the value read for it, over the
/// chain that caps it at 800% / 16G. /// hive-wide defaults of 200% / 4G.
fn atlas_with(atlas_key: &str) -> (String, String) { fn atlas_with(atlas_key: &str) -> (String, String) {
let declared = BTreeMap::from([( let declared = BTreeMap::from([(
"atlas".to_owned(), "atlas".to_owned(),
serde_json::from_str::<AgentCaps>(atlas_key).expect("decodes"), serde_json::from_str::<AgentCaps>(atlas_key).expect("decodes"),
)]); )]);
let stored = validated(&declared).expect("valid caps"); let stored = validated(&declared).expect("valid caps");
let hive_side = BTreeMap::from([( crate::resource_limits::effective_from(&stored, "atlas", "200%", "4G")
"atlas".to_owned(),
AgentLimits {
cpu_quota: Some("800%".to_owned()),
memory_max: Some("16G".to_owned()),
},
)]);
crate::resource_limits::effective_from_with_swarm(
&stored, &hive_side, "atlas", "200%", "4G",
)
} }
/// The other half of swarm-controller's /// The other half of swarm-controller's
/// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`: /// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`:
/// the first value is the bytes that route stores. The cap replaces the /// the first value is the bytes that route stores. The cap replaces the
/// hive's chain; a deleted key reads as `{}` and gives the chain back. /// hive default; a deleted key reads as `{}` and gives the default back.
#[test] #[test]
fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() { fn the_caps_the_route_stores_replace_the_hive_default_and_a_cleared_key_restores_it() {
assert_eq!( assert_eq!(
atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#), atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#),
("400%".to_owned(), "8G".to_owned()) ("400%".to_owned(), "8G".to_owned())
); );
assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned())); assert_eq!(atlas_with("{}"), ("200%".to_owned(), "4G".to_owned()));
} }
/// Per field: a swarm memory cap leaves CPU on the hive's chain. /// Per field: a swarm memory cap leaves CPU on the hive default.
#[test] #[test]
fn a_field_the_swarm_leaves_unset_stays_with_the_hive() { fn a_field_the_swarm_leaves_unset_stays_with_the_hive() {
assert_eq!( assert_eq!(
atlas_with(r#"{"memory_max":"2G"}"#), atlas_with(r#"{"memory_max":"2G"}"#),
("800%".to_owned(), "2G".to_owned()) ("200%".to_owned(), "2G".to_owned())
); );
} }

View file

@ -211,23 +211,6 @@ pub enum HostRequest {
/// this request just answers whatever it's asked. Result: /// this request just answers whatever it's asked. Result:
/// [`HostResponse::nodes`]. /// [`HostResponse::nodes`].
QueueNodes { ids: Vec<u64> }, QueueNodes { ids: Vec<u64> },
/// Declare an agent's CPU/memory overrides for the per-container
/// systemd drop-in, persisted to `meta/resource-limits.json`.
///
/// **Replace, not merge** — the pair given here becomes the agent's
/// entire entry, matching how tool-groups/capabilities are set. A
/// `None` field falls back to the hive-wide
/// `agentCpuQuota` / `agentMemoryMax`, so passing both as `None`
/// removes the entry entirely (reset to hive defaults).
///
/// Values are passed verbatim to systemd, so the server validates
/// their shape before persisting: a malformed `CPUQuota=` makes
/// systemd reject the unit, which would stop the container starting.
SetResourceLimits {
name: Ident,
cpu_quota: Option<String>,
memory_max: Option<String>,
},
/// Stop managed containers hive-wide in one operator action /// Stop managed containers hive-wide in one operator action
/// (`hivectl stop`): agents plus the selected infra containers. `scope` /// (`hivectl stop`): agents plus the selected infra containers. `scope`
/// selects which classes; an all-false scope means **everything** (the /// selects which classes; an all-false scope means **everything** (the

View file

@ -1,6 +1,6 @@
//! `hivectl agent <name>` — everything scoped to ONE managed agent: //! `hivectl agent <name>` — everything scoped to ONE managed agent:
//! container lifecycle over the host admin socket (restart/pause/resume/ //! container lifecycle over the host admin socket (restart/pause/resume/
//! spawn/kill/destroy/rebuild/set-limits/choom/watch), plus the //! spawn/kill/destroy/rebuild/choom/watch), plus the
//! `quota` and `subvol` groups, whose handlers live in their own modules. //! `quota` and `subvol` groups, whose handlers live in their own modules.
//! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide //! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide
//! read that lives in this module too since it shares the same daemon //! read that lives in this module too since it shares the same daemon
@ -282,32 +282,6 @@ pub(crate) async fn run_agent(socket: &Path, name: &str, cmd: AgentCmd) -> Resul
let name = crate::util::parse_ident(name)?; let name = crate::util::parse_ident(name)?;
render(crate::client::request(socket, HostRequest::Rebuild { name }).await?) render(crate::client::request(socket, HostRequest::Rebuild { name }).await?)
} }
AgentCmd::SetLimits {
cpu_quota,
memory_max,
reset,
} => {
let name = crate::util::parse_ident(name)?;
// `--reset` is the only way to reach an all-`None` request;
// clap rejects a bare `set-limits` with neither flag, so a
// forgotten value can't silently clear the overrides.
let (cpu_quota, memory_max) = if reset {
(None, None)
} else {
(cpu_quota, memory_max)
};
render(
crate::client::request(
socket,
HostRequest::SetResourceLimits {
name,
cpu_quota,
memory_max,
},
)
.await?,
)
}
AgentCmd::Choom { resume_session } => { AgentCmd::Choom { resume_session } => {
crate::choom::choom(socket, name, resume_session.as_deref()).await crate::choom::choom(socket, name, resume_session.as_deref()).await
} }

View file

@ -393,30 +393,6 @@ pub enum AgentCmd {
}, },
/// Apply pending config to this managed container. /// Apply pending config to this managed container.
Rebuild, Rebuild,
/// Declare this agent's CPU/memory limits, overriding the hive-wide
/// defaults.
///
/// Replaces the agent's whole override entry rather than merging into
/// it: any limit you don't pass returns to the hive-wide default. To
/// change one and keep the other, pass both. Disk is a separate
/// resource with its own group — see `quota`.
SetLimits {
/// systemd `CPUQuota=` value, for example `400%` (100% = one full core).
#[arg(long, conflicts_with = "reset")]
cpu_quota: Option<String>,
/// systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`.
#[arg(long, conflicts_with = "reset")]
memory_max: Option<String>,
/// Drop all overrides — the agent returns to the hive-wide
/// defaults. Required to clear limits, so that a `set-limits`
/// with a forgotten value can't silently reset the agent.
#[arg(
long,
conflicts_with_all = ["cpu_quota", "memory_max"],
required_unless_present_any = ["cpu_quota", "memory_max"],
)]
reset: bool,
},
/// Open an interactive Claude session inside this agent's container. /// Open an interactive Claude session inside this agent's container.
/// ///
/// A fresh session by default, or resume a prior one. Requires root /// A fresh session by default, or resume a prior one. Requires root