diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index a6e8d25b..7c0a3efe 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -395,26 +395,26 @@ Contents: (`{ "ruth": ["manage_root_agent"] }`). Written by `capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env var. Absent agents have no extra capabilities. -- `resource-limits.json` — per-agent container resource overrides +- `swarm-resource-limits.json` — the caps the swarm sets for this + hive's agents (`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`). - Written by `resource_limits::set_limits`; read where + Written only by `workers::swarm_limits`, which copies each agent's key of + the swarm's `agent-limits` bucket here every 30 seconds. Read where `lifecycle::write_dropins` generates the systemd drop-in, **not** injected into the container — these are host-side caps on the container, so the capped party never sets them. It sees only the resulting memory cap, - as a byte count or `infinity` in its runtime dir (below). Fallback is per *field*: - an absent file, absent agent, or absent field falls back to the + as a byte count or `infinity` in its runtime dir (below). Fallback is per + *field*: an absent file, absent agent, or absent field falls back to the hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`, - so an agent can override only its memory and still track the hive - default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same - drop-in have **no** per-agent override — they're hive-wide only and - come straight off `HiveEnv`, so this file has no field for them. -- `swarm-resource-limits.json` — the caps the swarm sets for this - hive's agents, same shape as `resource-limits.json`. Written only by - `workers::swarm_limits`, which copies each agent's key of the swarm's - `agent-limits` bucket here every 30 seconds. A field set here replaces - that field's answer from `resource-limits.json` and the hive-wide - default. Every value passes the same validators as the operator's - overrides first, and one bad value makes the worker change nothing here. + so an agent can have only its memory capped and still track the hive + default for CPU. Every value passes the validators first, and one bad + value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=` + shares in the same drop-in have **no** per-agent setting — they're + hive-wide only and come straight off `HiveEnv`, so this file has no field + for them. +- `resource-limits.json` — retired. Per-agent overrides set on the hive used + to live here; nothing reads or writes it now. An old copy in the meta repo + is ignored and can be deleted. The root agent has the meta dir RO-mounted at `/meta/`. diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index e7fb554e..f1c3cbc7 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -455,12 +455,12 @@ requiring a host rebuild. The answer per agent and per field is, first match wins: 1. the swarm's cap for the agent (see below); -2. the operator's per-agent override in `meta/resource-limits.json`; -3. `agentCpuQuota` / `agentMemoryMax`. +2. `agentCpuQuota` / `agentMemoryMax`. A swarm cap replaces the hive's answer, so a hive can't raise an agent above it. Like the rest of the drop-in, a changed swarm cap reaches a -running agent at its next spawn or rebuild. +running agent at its next spawn or rebuild. A hive has no way to set a cap +for one agent: caps are set per agent in the swarm UI. The container can't read its own cap, so `write_dropins` also writes the effective `MemoryMax=` into the agent's runtime dir @@ -468,7 +468,7 @@ effective `MemoryMax=` into the agent's runtime dir `infinity` for no cap. A percentage becomes bytes the way systemd resolves it, as that share of the host's physical memory rounded down to a page. A value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps -the last drop-in because it can't read the overrides or the swarm caps file, +the last drop-in because it can't read the swarm caps file, the file carries that drop-in's `MemoryMax=`. At boot the container derives two values from a byte count: @@ -520,7 +520,7 @@ given. Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each agent it has and stores the result in `meta/swarm-resource-limits.json`. Each -value passes the same validators as the operator's overrides first. One bad +value passes the validators first. One bad value or a failed read leaves the file unchanged, and repeated failures raise a dashboard banner. @@ -548,8 +548,7 @@ wants it. Two things to know about the weights: - They're **hive-wide** — unlike the caps there is no per-agent - override in `meta/resource-limits.json`, so every agent carries the - same value and the weight does _not_ rank agents against each other. + setting, so every agent carries the same value and the weight does _not_ rank agents against each other. What `80` buys is that agents yield to everything **not** on this drop-in path: host services (nginx and dnsmasq among them) and the infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which diff --git a/docs/tools/hivectl-cli.md b/docs/tools/hivectl-cli.md index 99bb2b1e..480bf2d9 100644 --- a/docs/tools/hivectl-cli.md +++ b/docs/tools/hivectl-cli.md @@ -22,7 +22,6 @@ This document contains the help content for the `hivectl` command-line program. * [`hivectl agent kill`↴](#hivectl-agent-kill) * [`hivectl agent destroy`↴](#hivectl-agent-destroy) * [`hivectl agent rebuild`↴](#hivectl-agent-rebuild) -* [`hivectl agent set-limits`↴](#hivectl-agent-set-limits) * [`hivectl agent choom`↴](#hivectl-agent-choom) * [`hivectl agent watch`↴](#hivectl-agent-watch) * [`hivectl agent quota`↴](#hivectl-agent-quota) @@ -221,7 +220,6 @@ Everything here targets a single named agent (`hivectl agent foo restart`, `hive * `kill` — Hard-stop this managed container * `destroy` — Tear down this sub-agent container, keeping its state by default. No undo * `rebuild` — Apply pending config to this managed container -* `set-limits` — Declare this agent's CPU/memory limits, overriding the hive-wide defaults * `choom` — Open an interactive Claude session inside this agent's container * `watch` — Follow this agent's live turn/tool-call event stream from the CLI * `quota` — This agent's disk accounting + optional quota via btrfs qgroups @@ -315,22 +313,6 @@ Apply pending config to this managed container -## `hivectl agent set-limits` - -Declare this agent's CPU/memory limits, overriding the hive-wide defaults. - -Replaces the agent's whole override entry rather than merging into it: any limit you don't pass returns to the hive-wide default. To change one and keep the other, pass both. Disk is a separate resource with its own group — see `quota`. - -**Usage:** `hivectl agent set-limits [OPTIONS]` - -###### **Options:** - -* `--cpu-quota ` — systemd `CPUQuota=` value, for example `400%` (100% = one full core) -* `--memory-max ` — systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity` -* `--reset` — Drop all overrides — the agent returns to the hive-wide defaults. Required to clear limits, so that a `set-limits` with a forgotten value can't silently reset the agent - - - ## `hivectl agent choom` Open an interactive Claude session inside this agent's container. diff --git a/docs/tools/hivectl.md b/docs/tools/hivectl.md index 293ed79e..7629c072 100644 --- a/docs/tools/hivectl.md +++ b/docs/tools/hivectl.md @@ -136,34 +136,6 @@ resume drains the backlog rather than dropping it. Points worth knowing: - Visible as ` paused` in `list-agents`'s STATUS column, as a `paused` field on the JSON rows, and as a badge on the dashboard card. -### Per-agent resource limits - -```bash -hivectl agent sock set-limits --cpu-quota 400% --memory-max 8G -hivectl agent sock set-limits --memory-max 8G # CPU falls back to the hive default -hivectl agent sock set-limits --reset # drop all overrides -``` - -Overrides the hive-wide `services.hyperhive.c0re.agentCpuQuota` / -`agentMemoryMax` for one agent, persisted to -`meta/resource-limits.json` (see -[`persistence.md`](../agent-lifecycle/persistence.md)). Values are systemd's -`CPUQuota=` / `MemoryMax=` syntax: a percentage (`400%` = four full -cores) for CPU; a size (`8G`), a percentage of physical RAM, or -`infinity` for memory. `set-limits` validates both before persisting them — -they go into a systemd drop-in verbatim, and a typo there makes the -unit fail to start. - -**Declarative, not incremental**: each invocation replaces the agent's -whole entry. `hivectl agent sock set-limits --memory-max 8G` leaves `sock` -with _only_ a memory override, reverting any previously set CPU quota to -the hive default. To avoid a forgotten flag silently wiping an override, -hivectl rejects `set-limits` with no flags at all — clearing requires the -explicit `--reset`. - -The command rewrites the container's drop-in and reloads systemd, so -new containers and restarts pick the values up immediately. - ## Choom Drop into an interactive Claude session inside an agent container. diff --git a/hive-c0re/src/agent_config/resource_limits.rs b/hive-c0re/src/agent_config/resource_limits.rs index 880c87da..c0da2272 100644 --- a/hive-c0re/src/agent_config/resource_limits.rs +++ b/hive-c0re/src/agent_config/resource_limits.rs @@ -1,5 +1,8 @@ -//! Per-agent CPU/memory limit overrides. Stored at -//! `/var/lib/hyperhive/meta/resource-limits.json` alongside +//! CPU/memory limits for an agent's container: the caps the swarm controller +//! declared, and the validators for the values the systemd drop-in takes. +//! +//! The swarm caps are stored at +//! `/var/lib/hyperhive/meta/swarm-resource-limits.json` alongside //! `topology.json`, `tool-groups.json` and `capabilities.json`. //! //! Format: a JSON object mapping agent name to an object with optional @@ -14,9 +17,9 @@ //! //! Fallback is **per field**: an absent file, an absent agent, or an //! absent field all fall back to the hive-wide -//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So an agent -//! can raise only its memory cap and keep tracking the hive default for -//! CPU — see [`effective`]. +//! `services.hyperhive.agentCpuQuota` / `agentMemoryMax`. So a swarm cap on +//! memory alone leaves CPU on the hive default — see +//! [`effective_from_with_swarm`]. //! //! Read path: `lifecycle::host_config::write_dropins`, on every spawn //! and every rebuild. @@ -30,10 +33,8 @@ use std::collections::BTreeMap; use std::path::{Path, PathBuf}; -const RESOURCE_LIMITS_FILE: &str = "resource-limits.json"; - -/// One agent's overrides. Both fields optional and independent; `None` -/// means "use the hive-wide default for this field". +/// One agent's caps. Both fields optional and independent; `None` means +/// "use the hive-wide default for this field". #[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub struct AgentLimits { /// systemd `CPUQuota=` value, e.g. `"400%"`. @@ -44,84 +45,6 @@ pub struct AgentLimits { pub memory_max: Option, } -impl AgentLimits { - /// True when neither field is set — such an entry is dropped rather - /// than persisted as an empty object. - #[must_use] - pub fn is_empty(&self) -> bool { - self.cpu_quota.is_none() && self.memory_max.is_none() - } -} - -#[must_use] -pub fn resource_limits_path() -> PathBuf { - crate::paths::meta_root().join(RESOURCE_LIMITS_FILE) -} - -/// Read the per-agent limit map. An absent file is the empty map — -/// callers treat a missing entry as "hive-wide defaults". A file that -/// exists but can't be read or parsed is an error: an override can be -/// tighter than the hive default, so reading it as empty is not a safe -/// fallback. -pub fn read() -> std::io::Result> { - super::read_map(&resource_limits_path()) -} - -/// Resolve the effective values for an agent from the override file at -/// `path` (normally [`resource_limits_path`]), filling each unset field -/// from the hive-wide default. This is the single place the fallback -/// rule lives; `write_dropins` calls it and passes the result straight -/// to systemd. Errors as [`read`] does. -/// -/// Use [`effective_from`] when resolving more than one agent in a row. -pub fn effective( - path: &Path, - name: &str, - hive_cpu_quota: &str, - hive_memory_max: &str, -) -> std::io::Result<(String, String)> { - let limits = super::read_map(path)?; - Ok(effective_from( - &limits, - name, - hive_cpu_quota, - hive_memory_max, - )) -} - -/// [`effective`] against an already-loaded map — the multi-agent form. -/// -/// `container_view::build_all` renders every agent on each SSE scan, so -/// it loads the map once and calls this per agent rather than re-reading -/// the same small file N times per scan. -#[must_use] -pub fn effective_from( - limits: &BTreeMap, - name: &str, - hive_cpu_quota: &str, - hive_memory_max: &str, -) -> (String, String) { - match limits.get(name) { - Some(l) => resolve(l, hive_cpu_quota, hive_memory_max), - None => (hive_cpu_quota.to_owned(), hive_memory_max.to_owned()), - } -} - -/// Pure core of [`effective`], split out so the fallback matrix is -/// testable without touching the filesystem. -#[must_use] -fn resolve(limits: &AgentLimits, hive_cpu_quota: &str, hive_memory_max: &str) -> (String, String) { - let cpu = limits - .cpu_quota - .clone() - .unwrap_or_else(|| hive_cpu_quota.to_owned()); - let mem = limits - .memory_max - .clone() - .unwrap_or_else(|| hive_memory_max.to_owned()); - (cpu, mem) -} - /// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`, /// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`) /// into a raw byte count, accepting exactly the single-term forms systemd's @@ -164,35 +87,7 @@ pub fn parse_bytes(value: &str) -> Option { u64::try_from(bytes).ok() } -/// Set one agent's overrides and persist the map atomically. An entry -/// with both fields unset is removed rather than stored, so "reset to -/// hive defaults" and "never configured" are the same state on disk. -/// -/// # Errors -/// -/// Fails without writing when the existing file can't be read or parsed -/// (`InvalidData` for a parse failure), and otherwise when the meta dir -/// or the file can't be written. -pub fn set_limits(name: &str, limits: &AgentLimits) -> std::io::Result<()> { - set_limits_at(&resource_limits_path(), name, limits) -} - -fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Result<()> { - let mut current: BTreeMap = super::read_map(path)?; - if limits.is_empty() { - current.remove(name); - } else { - current.insert(name.to_owned(), limits.clone()); - } - super::write_map(path, ¤t) -} - /// Caps the swarm controller declared for this hive's agents. -/// -/// A separate file from the operator's overrides above, not a merge into it: -/// the two have different writers and different lifetimes, and a swarm -/// declaration landing in the file an operator edits would make "reset to -/// hive defaults" destroy something the operator never set. const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json"; #[must_use] @@ -201,9 +96,9 @@ pub fn swarm_limits_path() -> PathBuf { } /// Read the swarm-declared caps. An absent file is the empty map — no swarm -/// opinion, so the hive's own chain decides. A file that exists but can't be -/// read or parsed is an error, for the reason [`read`] gives: reading it as -/// empty would silently drop a cap. +/// opinion, so the hive-wide defaults decide. A file that exists but can't be +/// read or parsed is an error: a cap can be tighter than the hive default, so +/// reading it as empty would silently drop it. pub fn read_swarm() -> std::io::Result> { super::read_map(&swarm_limits_path()) } @@ -216,55 +111,43 @@ pub fn write_swarm(map: &BTreeMap) -> std::io::Result<()> { super::write_map(&swarm_limits_path(), map) } -/// [`effective_from`] with the swarm's declaration on top. +/// Resolve the effective values for an agent from the swarm caps in `swarm`: +/// each field the swarm declared a cap for, and the hive-wide default for +/// each it did not. This is the single place the fallback rule lives; +/// `write_dropins` calls it and passes the result straight to systemd. /// -/// A swarm cap **replaces** the hive's answer for that field rather than -/// joining the fallback chain as another tier: a tier a hive can out-set is -/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's -/// chain. -/// -/// The chain itself is untouched — [`effective_from`] is called exactly as -/// before and its result is what a missing swarm value falls back to. +/// Use this form when resolving more than one agent in a row. #[must_use] -pub fn effective_from_with_swarm( +pub fn effective_from( swarm: &BTreeMap, - limits: &BTreeMap, name: &str, hive_cpu_quota: &str, hive_memory_max: &str, ) -> (String, String) { - let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max); - match swarm.get(name) { - Some(s) => ( - s.cpu_quota.clone().unwrap_or(cpu), - s.memory_max.clone().unwrap_or(mem), - ), - None => (cpu, mem), - } + let caps = swarm.get(name); + ( + caps.and_then(|c| c.cpu_quota.clone()) + .unwrap_or_else(|| hive_cpu_quota.to_owned()), + caps.and_then(|c| c.memory_max.clone()) + .unwrap_or_else(|| hive_memory_max.to_owned()), + ) } -/// [`effective_from_with_swarm`] for a caller resolving a single agent, from -/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`. -/// -/// Delegates to [`effective`] rather than re-deriving the hive's answer, so -/// there stays exactly one place the fallback chain lives. Errors as [`read`] -/// does, for either file. -pub fn effective_with_swarm( - limits_path: &Path, +/// [`effective_from`] for a caller resolving a single agent, from the swarm +/// caps at `swarm_path`. Errors as [`read_swarm`] does. +pub fn effective( swarm_path: &Path, name: &str, hive_cpu_quota: &str, hive_memory_max: &str, ) -> std::io::Result<(String, String)> { - let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?; let swarm: BTreeMap = super::read_map(swarm_path)?; - Ok(match swarm.get(name) { - Some(s) => ( - s.cpu_quota.clone().unwrap_or(cpu), - s.memory_max.clone().unwrap_or(mem), - ), - None => (cpu, mem), - }) + Ok(effective_from( + &swarm, + name, + hive_cpu_quota, + hive_memory_max, + )) } /// Validate a systemd `CPUQuota=` value. Percentages only, and values @@ -386,96 +269,66 @@ mod tests { } #[test] - fn unset_agent_gets_hive_defaults() { - let (cpu, mem) = resolve(&AgentLimits::default(), HIVE_CPU, HIVE_MEM); - assert_eq!(cpu, "200%"); - assert_eq!(mem, "4G"); + fn an_agent_the_swarm_says_nothing_about_gets_the_hive_defaults() { + let swarm = BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]); + for map in [BTreeMap::new(), swarm] { + assert_eq!( + effective_from(&map, "sock", HIVE_CPU, HIVE_MEM), + (HIVE_CPU.to_owned(), HIVE_MEM.to_owned()) + ); + } } #[test] - fn both_fields_override() { - let (cpu, mem) = resolve(&limits(Some("400%"), Some("8G")), HIVE_CPU, HIVE_MEM); - assert_eq!(cpu, "400%"); - assert_eq!(mem, "8G"); + fn a_swarm_cap_replaces_the_hive_default() { + let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]); + assert_eq!( + effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM), + ("150%".to_owned(), "2G".to_owned()) + ); } - /// The point of per-field fallback: overriding memory must not drag - /// CPU along with it. + /// The point of per-field fallback: capping memory must not drag CPU + /// along with it. #[test] - fn partial_override_keeps_other_field_on_hive_default() { - let (cpu, mem) = resolve(&limits(None, Some("8G")), HIVE_CPU, HIVE_MEM); - assert_eq!(cpu, "200%", "cpu should still track the hive default"); - assert_eq!(mem, "8G"); - - let (cpu, mem) = resolve(&limits(Some("400%"), None), HIVE_CPU, HIVE_MEM); - assert_eq!(cpu, "400%"); - assert_eq!(mem, "4G", "memory should still track the hive default"); - } - - #[test] - fn empty_entry_is_reported_empty() { - assert!(AgentLimits::default().is_empty()); - assert!(!limits(None, Some("8G")).is_empty()); - assert!(!limits(Some("400%"), None).is_empty()); + fn a_swarm_cap_is_per_field() { + let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]); + assert_eq!( + effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM), + (HIVE_CPU.to_owned(), "2G".to_owned()), + "cpu should still track the hive default" + ); + let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), None))]); + assert_eq!( + effective_from(&swarm, "sock", HIVE_CPU, HIVE_MEM), + ("150%".to_owned(), HIVE_MEM.to_owned()), + "memory should still track the hive default" + ); } const TRUNCATED: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"memory_max\": \"1G\" },\n \"iris\": { \"mem"; - fn corrupt_file() -> (tempfile::TempDir, PathBuf) { - let dir = tempfile::tempdir().expect("tempdir"); - let path = dir.path().join(RESOURCE_LIMITS_FILE); - std::fs::write(&path, TRUNCATED).expect("seed"); - (dir, path) - } - #[test] fn effective_of_a_corrupt_file_is_an_error() { - let (_dir, path) = corrupt_file(); + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join(SWARM_LIMITS_FILE); + std::fs::write(&path, TRUNCATED).expect("seed"); let err = effective(&path, "sock", HIVE_CPU, HIVE_MEM) .expect_err("a corrupt file must not resolve to hive defaults"); assert_eq!(err.kind(), std::io::ErrorKind::InvalidData); } - #[test] - fn set_limits_leaves_a_corrupt_file_untouched() { - let (_dir, path) = corrupt_file(); - let err = set_limits_at(&path, "ruth", &limits(Some("400%"), None)) - .expect_err("a corrupt file must not be overwritten"); - assert_eq!(err.kind(), std::io::ErrorKind::InvalidData); - assert_eq!(std::fs::read(&path).expect("read"), TRUNCATED.as_bytes()); - } - #[test] fn missing_file_resolves_to_hive_defaults() { let dir = tempfile::tempdir().expect("tempdir"); - let path = dir.path().join(RESOURCE_LIMITS_FILE); + let path = dir.path().join(SWARM_LIMITS_FILE); let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("missing file"); assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, HIVE_MEM)); } - #[test] - fn set_limits_keeps_other_agents_and_leaves_no_temp_file() { - let dir = tempfile::tempdir().expect("tempdir"); - let path = dir.path().join(RESOURCE_LIMITS_FILE); - set_limits_at(&path, "sock", &limits(None, Some("8G"))).expect("set on missing file"); - set_limits_at(&path, "iris", &limits(Some("50%"), None)).expect("set"); - let (cpu, mem) = effective(&path, "sock", HIVE_CPU, HIVE_MEM).expect("read"); - assert_eq!((cpu.as_str(), mem.as_str()), (HIVE_CPU, "8G")); - let (cpu, _) = effective(&path, "iris", HIVE_CPU, HIVE_MEM).expect("read"); - assert_eq!(cpu, "50%"); - let entries: Vec<_> = std::fs::read_dir(dir.path()) - .expect("read_dir") - .map(|e| e.expect("entry").file_name()) - .collect(); - assert_eq!( - entries, - vec![std::ffi::OsString::from(RESOURCE_LIMITS_FILE)] - ); - } - /// Absent fields must deserialize to `None`, not fail — an entry - /// written by an older version with only one field must still load. + /// with only one field must still load. #[test] fn partial_entry_deserializes() { let map: BTreeMap = @@ -605,52 +458,4 @@ mod tests { ); } } - - #[test] - fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() { - // The whole point of the ruling: a tier a hive can out-set is not a - // cap. The agent override below is the strongest hive-side value - // there is, and the swarm still wins. - let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]); - let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]); - assert_eq!( - effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM), - ("150%".to_owned(), "2G".to_owned()) - ); - } - - #[test] - fn a_swarm_cap_is_per_field_like_every_other_tier() { - // A swarm that caps memory and says nothing about CPU leaves CPU on - // the hive's chain — here the per-agent override, not the hive default. - let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]); - let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]); - assert_eq!( - effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM), - ("800%".to_owned(), "2G".to_owned()) - ); - } - - #[test] - fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() { - // The control that makes the two tests above mean something: the - // wrapper must be a no-op when the swarm has said nothing, for every - // shape the old chain distinguishes. If this drifts, the swarm tier - // changed behaviour it was explicitly told not to change. - let none = BTreeMap::new(); - let cases = [ - BTreeMap::new(), - BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]), - BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]), - BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]), - BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]), - ]; - for hive_side in &cases { - assert_eq!( - effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM), - effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM), - "the wrapper changed the answer with no swarm value present" - ); - } - } } diff --git a/hive-c0re/src/container_view.rs b/hive-c0re/src/container_view.rs index d55686cc..67e98753 100644 --- a/hive-c0re/src/container_view.rs +++ b/hive-c0re/src/container_view.rs @@ -76,16 +76,6 @@ pub struct ContainerView { /// a legitimate way to keep it idle when it next boots. #[serde(default)] pub paused: bool, - /// Effective systemd `CPUQuota=` for this container (e.g. `"400%"`) — - /// the per-agent override from `meta/resource-limits.json` when set, - /// otherwise the hive-wide `agentCpuQuota`. Always populated: there - /// is no "unset" state to render, only "same as everyone else". - /// Reflects what the *drop-in says*, which is what the next start - /// will enforce — not a live cgroup reading. - pub cpu_quota: String, - /// Effective systemd `MemoryMax=` for this container (e.g. `"8G"`). - /// Same resolution + caveat as [`ContainerView::cpu_quota`]. - pub memory_max: String, } impl From for hive_sh4re::container::AgentStatusRow { @@ -99,10 +89,9 @@ impl From for hive_sh4re::container::AgentStatusRow { /// anymore. Stubbed to `0` rather than dropping the wire field /// outright — leaves `hivectl status`/the dashboard column intact /// syntactically, just always empty, until iris's frontend follow-up - /// decides whether to drop the column entirely. `port`, `cpu_quota`, - /// `memory_max` have no equivalent on the row at all and are simply - /// not carried over. `url` has no source on `ContainerView` at all — - /// see [`agent_url`]. + /// decides whether to drop the column entirely. `port` has no + /// equivalent on the row at all and is simply not carried over. `url` + /// has no source on `ContainerView` at all — see [`agent_url`]. fn from(v: ContainerView) -> Self { Self { url: agent_url(&v.name), @@ -148,37 +137,12 @@ fn agent_url_for_domain(domain: &str, name: &str) -> String { /// Build the full container list. Wraps `lifecycle::list()` and /// resolves every per-agent attribute the dashboard surfaces. /// -/// Takes `hive` because the effective resource limits are a per-field -/// fallback onto the hive-wide `agent_cpu_quota` / `agent_memory_max`, -/// and those live on [`crate::coordinator::HiveEnv`], not on disk. Both callers already -/// hold a `Coordinator`, so this is a parameter rather than a global. -/// /// # Errors /// /// The container list could not be read. -pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result> { +pub async fn build_all() -> anyhow::Result> { let raw = lifecycle::list().await?; let locked = read_meta_locked_revs(); - // Read once per scan rather than re-read for every container on every - // SSE scan. An unreadable file shows every row at the hive defaults; - // the error log is the only signal, as `ContainerView` has no field - // to carry it. - let limits = crate::resource_limits::read().unwrap_or_else(|e| { - tracing::error!( - path = %crate::resource_limits::resource_limits_path().display(), - error = ?e, - "resource limits unreadable — container rows show hive defaults" - ); - std::collections::BTreeMap::new() - }); - let swarm_limits = crate::resource_limits::read_swarm().unwrap_or_else(|e| { - tracing::error!( - path = %crate::resource_limits::swarm_limits_path().display(), - error = ?e, - "swarm resource caps unreadable — container rows ignore them" - ); - std::collections::BTreeMap::new() - }); let mut out = Vec::new(); for c in &raw { let Some(logical) = c.strip_prefix(AGENT_PREFIX) else { @@ -243,13 +207,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result anyhow::Result) { // An unreadable list says nothing about which containers exist, so // the last snapshot stands rather than every row being reported gone. - let fresh = match container_view::build_all(&self.hive_env()).await { + let fresh = match container_view::build_all().await { Ok(fresh) => fresh, Err(e) => { tracing::warn!(error = ?e, "container rescan: list failed; keeping last snapshot"); diff --git a/hive-c0re/src/dashboard/lifecycle_ops.rs b/hive-c0re/src/dashboard/lifecycle_ops.rs index c26995e6..d93be152 100644 --- a/hive-c0re/src/dashboard/lifecycle_ops.rs +++ b/hive-c0re/src/dashboard/lifecycle_ops.rs @@ -311,95 +311,6 @@ fn set_paused_error_response( } } -/// Form fields for `post_resource_limits`. Both fields are optional strings; -/// an empty value clears the per-agent override for that field, falling back -/// to the hive-wide default. -#[derive(Deserialize, Default, ToSchema)] -pub(super) struct ResourceLimitsForm { - #[serde(default)] - cpu_quota: String, - #[serde(default)] - memory_max: String, -} - -/// Write per-agent CPU/memory limit -/// overrides for `name`. -/// -/// An empty `cpu_quota` or `memory_max` field clears that field's override, -/// falling back to the hive-wide default. Both empty together removes the -/// agent's entry entirely. The new drop-in is written immediately — the -/// limits take effect on the next container start or restart. Triggers an -/// immediate rescan so `ContainerView.cpu_quota`/`memory_max` update on -/// the dashboard via SSE without waiting for the next periodic sweep. -#[utoipa::path( - post, - path = "/api/resource-limits/{name}", - params(("name" = String, Path, description = "agent name")), - request_body(content = ResourceLimitsForm, content_type = "application/x-www-form-urlencoded"), - responses( - (status = 200, description = "limits written", body = String), - (status = 400, description = "bad agent name"), - (status = 404, description = "no such agent"), - (status = 422, description = "invalid cpu_quota/memory_max value"), - (status = 500, description = "commit or drop-in write failed"), - ), - tag = "lifecycle_ops" -)] -pub(super) async fn post_resource_limits( - State(state): State, - AxumPath(name): AxumPath, - Form(form): Form, -) -> Response { - let logical = strip_container_prefix(&name); - if let Some(reject) = guard_agent_name(&state, &logical).await { - return reject; - } - let ident = match Ident::parse(&logical) { - Ok(i) => i, - Err(e) => return (StatusCode::BAD_REQUEST, format!("bad agent name: {e}")).into_response(), - }; - let cpu_quota = if form.cpu_quota.is_empty() { - None - } else { - Some(form.cpu_quota.as_str()) - }; - let memory_max = if form.memory_max.is_empty() { - None - } else { - Some(form.memory_max.as_str()) - }; - if let Some(v) = cpu_quota - && let Err(e) = crate::resource_limits::validate_cpu_quota(v) - { - return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response(); - } - // Returns the value to store, not just an ok/err verdict — see - // `validate_memory_max`'s own doc for why that isn't always `v`. - let memory_max = match memory_max.map(crate::resource_limits::validate_memory_max) { - Some(Err(e)) => return (StatusCode::UNPROCESSABLE_ENTITY, e).into_response(), - Some(Ok(v)) => Some(v), - None => None, - }; - let limits = crate::resource_limits::AgentLimits { - cpu_quota: cpu_quota.map(str::to_owned), - memory_max, - }; - if let Err(e) = crate::meta::commit_resource_limits(ident.as_str(), &limits).await { - return error_response(&format!("set limits {logical}: {e:#}")); - } - let agent_dir = crate::paths::agent_runtime_dir(ident.as_str()); - let hive = state.coord.hive_env(); - let paths = match crate::coordinator::Coordinator::agent_paths(ident.as_str(), agent_dir) { - Ok(p) => p, - Err(e) => return error_response(&format!("write_dropins {logical}: {e:#}")), - }; - if let Err(e) = crate::lifecycle::write_dropins(ident.as_str(), &hive, &paths).await { - return error_response(&format!("write_dropins {logical}: {e:#}")); - } - state.coord.rescan_containers_and_emit().await; - (StatusCode::OK, "ok").into_response() -} - /// Queue a rebuild DAG for every live agent /// container. #[utoipa::path( diff --git a/hive-c0re/src/dashboard/misc_api.rs b/hive-c0re/src/dashboard/misc_api.rs index 1d0d63ec..845303f2 100644 --- a/hive-c0re/src/dashboard/misc_api.rs +++ b/hive-c0re/src/dashboard/misc_api.rs @@ -1,6 +1,5 @@ //! Remaining single-endpoint dashboard handlers: mark-all-read, operator -//! compose (`op-send`), spawn-request, hive-wide turn stats, and container -//! resources. +//! compose (`op-send`), spawn-request and hive-wide turn stats. use axum::{ extract::{Form, Path as AxumPath, State}, @@ -11,7 +10,6 @@ use serde::{Deserialize, Serialize}; use utoipa::{IntoParams, ToSchema}; use super::{AppState, Ident, error_response}; -use crate::container_stats::ContainerResource; use crate::hive_stats::HiveStats; #[derive(Deserialize, IntoParams)] @@ -43,20 +41,6 @@ pub(super) async fn api_stats_hive( .into_response() } -/// Live per-agent-container CPU + memory load from cgroup v2. -/// -/// Samples CPU over a short interval (~200 ms), so this call briefly -/// awaits. -#[utoipa::path( - get, - path = "/api/container-resources", - responses((status = 200, description = "live per-container CPU + memory load", body = Vec)), - tag = "misc_api" -)] -pub(super) async fn api_container_resources() -> Response { - axum::Json(crate::container_stats::gather().await).into_response() -} - #[derive(Serialize, ToSchema)] pub(super) struct MarkAllReadBody { marked: u64, diff --git a/hive-c0re/src/dashboard/mod.rs b/hive-c0re/src/dashboard/mod.rs index 79d808bc..6f1fec8d 100644 --- a/hive-c0re/src/dashboard/mod.rs +++ b/hive-c0re/src/dashboard/mod.rs @@ -116,7 +116,6 @@ fn app(coord: Arc) -> axum::Router { .routes(routes!(state_snapshot::api_state)) .routes(routes!(state_files::get_state_file)) .routes(routes!(misc_api::api_stats_hive)) - .routes(routes!(misc_api::api_container_resources)) .routes(routes!(misc_api::post_mark_all_read)) .routes(routes!(misc_api::post_op_send)) .routes(routes!(build_logs::get_build_logs_all)) @@ -140,7 +139,6 @@ fn app(coord: Arc) -> axum::Router { .routes(routes!(lifecycle_ops::post_rebuild)) .routes(routes!(lifecycle_ops::post_pause)) .routes(routes!(lifecycle_ops::post_resume)) - .routes(routes!(lifecycle_ops::post_resource_limits)) .routes(routes!(lifecycle_ops::post_update_all)) .routes(routes!(tombstones::post_purge_tombstone)) .routes(routes!(meta_inputs::post_meta_update)) @@ -372,7 +370,6 @@ mod router_build_probe { .routes(routes!(state_snapshot::api_state)) .routes(routes!(state_files::get_state_file)) .routes(routes!(misc_api::api_stats_hive)) - .routes(routes!(misc_api::api_container_resources)) .routes(routes!(misc_api::post_mark_all_read)) .routes(routes!(misc_api::post_op_send)) .routes(routes!(build_logs::get_build_logs_all)) @@ -396,7 +393,6 @@ mod router_build_probe { .routes(routes!(lifecycle_ops::post_rebuild)) .routes(routes!(lifecycle_ops::post_pause)) .routes(routes!(lifecycle_ops::post_resume)) - .routes(routes!(lifecycle_ops::post_resource_limits)) .routes(routes!(lifecycle_ops::post_update_all)) .routes(routes!(tombstones::post_purge_tombstone)) .routes(routes!(meta_inputs::post_meta_update)) diff --git a/hive-c0re/src/dashboard/state_snapshot.rs b/hive-c0re/src/dashboard/state_snapshot.rs index c647fd74..ac406e44 100644 --- a/hive-c0re/src/dashboard/state_snapshot.rs +++ b/hive-c0re/src/dashboard/state_snapshot.rs @@ -603,8 +603,6 @@ mod tests { status_text: None, status_set_at: None, paused: false, - cpu_quota: "200%".to_owned(), - memory_max: "4G".to_owned(), } } diff --git a/hive-c0re/src/lifecycle/host_config.rs b/hive-c0re/src/lifecycle/host_config.rs index 9e5a0b98..a0fdd10f 100644 --- a/hive-c0re/src/lifecycle/host_config.rs +++ b/hive-c0re/src/lifecycle/host_config.rs @@ -27,7 +27,6 @@ pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Re set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?; if let Some((cpu_quota, memory_max)) = stage_limits( name, - &crate::resource_limits::resource_limits_path(), &crate::resource_limits::swarm_limits_path(), &limits_dropin_path(&container), &paths.agent, @@ -97,14 +96,13 @@ impl HostMemory { /// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's. fn stage_limits( agent_name: &str, - limits_path: &Path, swarm_path: &Path, dropin_path: &Path, runtime_dir: &Path, hive: &HiveEnv, host: Option, ) -> Result> { - let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive); + let limits = limits_to_apply(agent_name, swarm_path, dropin_path, hive); let memory_max = match &limits { Some((_, memory_max)) => Some(memory_max.clone()), None => dropin_memory_max(agent_name, dropin_path), @@ -226,35 +224,27 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> { /// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits /// drop-in, or `None` to leave the drop-in at `dropin_path` as it is. /// -/// When the overrides file at `limits_path` or the swarm caps at -/// `swarm_path` can't be read, the error is logged and the agent still starts. -/// The last-applied drop-in is kept, because the hive defaults can be looser -/// than the agent's own override. +/// When the swarm caps at `swarm_path` can't be read, the error is logged and +/// the agent still starts. The last-applied drop-in is kept, because the hive +/// defaults can be looser than the swarm's cap. /// With no drop-in yet, the agent gets the hive defaults: without one it /// would run uncapped. fn limits_to_apply( agent_name: &str, - limits_path: &Path, swarm_path: &Path, dropin_path: &Path, hive: &HiveEnv, ) -> Option<(String, String)> { let hive_cpu = &hive.agent_cpu_quota; let hive_mem = &hive.agent_memory_max; - let e = match crate::resource_limits::effective_with_swarm( - limits_path, - swarm_path, - agent_name, - hive_cpu, - hive_mem, - ) { + let e = match crate::resource_limits::effective(swarm_path, agent_name, hive_cpu, hive_mem) { Ok(limits) => return Some(limits), Err(e) => e, }; if dropin_path.exists() { tracing::error!( agent = %agent_name, - path = %limits_path.display(), + path = %swarm_path.display(), error = ?e, "resource limits unreadable — keeping the last-applied limits drop-in" ); @@ -262,7 +252,7 @@ fn limits_to_apply( } else { tracing::error!( agent = %agent_name, - path = %limits_path.display(), + path = %swarm_path.display(), error = ?e, "resource limits unreadable and no limits drop-in yet — applying hive defaults" ); @@ -271,14 +261,14 @@ fn limits_to_apply( } /// Write a systemd drop-in for `container@.service` that applies -/// the agent's effective resource caps — its per-agent overrides from -/// `meta/resource-limits.json` where set, the hive-wide defaults +/// the agent's effective resource caps — the swarm's cap from +/// `meta/swarm-resource-limits.json` where set, the hive-wide defaults /// otherwise. Goes under `/run/systemd/system/...` so it's ephemeral /// (regenerated on every spawn / rebuild, except as [`limits_to_apply`] says). /// /// The weights are hive-wide (`services.hyperhive.agentCpuWeight` / -/// `agentIoWeight`) — unlike the caps they have no per-agent override in -/// `meta/resource-limits.json`, so they come straight off [`HiveEnv`]. +/// `agentIoWeight`) — unlike the caps they have no swarm cap, so they come +/// straight off [`HiveEnv`]. /// `None` (the nix option set to `null`) means the weight line is left out /// and the container keeps the kernel default. async fn set_resource_limits( @@ -848,42 +838,35 @@ mod tests { const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem"; - fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf { - dir.path().join("absent-swarm-resource-limits.json") + fn swarm_file(dir: &tempfile::TempDir) -> std::path::PathBuf { + dir.path().join("swarm-resource-limits.json") } #[test] - fn corrupt_limits_keep_an_existing_dropin() { + fn corrupt_swarm_caps_keep_an_existing_dropin() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); + let swarm = swarm_file(&dir); let dropin = dir.path().join("hyperhive-limits.conf"); - std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); + std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); assert_eq!( - limits_to_apply( - "sock", - &limits, - &absent_swarm(&dir), - &dropin, - &HiveEnv::default() - ), + limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default()), None ); assert_eq!( - std::fs::read(&limits).expect("read"), + std::fs::read(&swarm).expect("read"), TRUNCATED_LIMITS.as_bytes() ); } #[test] - fn corrupt_limits_without_a_dropin_apply_hive_defaults() { + fn corrupt_swarm_caps_without_a_dropin_apply_hive_defaults() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); - std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); + let swarm = swarm_file(&dir); + std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); let applied = limits_to_apply( "sock", - &limits, - &absent_swarm(&dir), + &swarm, &dir.path().join("absent.conf"), &HiveEnv::default(), ); @@ -891,32 +874,12 @@ mod tests { } #[test] - fn corrupt_swarm_caps_keep_an_existing_dropin() { + fn a_swarm_cap_is_applied_over_the_hive_default() { let dir = tempfile::tempdir().expect("tempdir"); - let swarm = dir.path().join("swarm-resource-limits.json"); - let dropin = dir.path().join("hyperhive-limits.conf"); - std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); - std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); - let applied = limits_to_apply( - "sock", - &dir.path().join("resource-limits.json"), - &swarm, - &dropin, - &HiveEnv::default(), - ); - assert_eq!(applied, None); - } - - #[test] - fn a_swarm_cap_is_applied_over_the_operator_override() { - let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); - let swarm = dir.path().join("swarm-resource-limits.json"); - std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits"); + let swarm = swarm_file(&dir); std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps"); let applied = limits_to_apply( "sock", - &limits, &swarm, &dir.path().join("absent.conf"), &HiveEnv::default(), @@ -925,19 +888,13 @@ mod tests { } #[test] - fn readable_limits_are_applied_over_an_existing_dropin() { + fn readable_swarm_caps_are_applied_over_an_existing_dropin() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); + let swarm = swarm_file(&dir); let dropin = dir.path().join("hyperhive-limits.conf"); - std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits"); + std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps"); std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in"); - let applied = limits_to_apply( - "sock", - &limits, - &absent_swarm(&dir), - &dropin, - &HiveEnv::default(), - ); + let applied = limits_to_apply("sock", &swarm, &dropin, &HiveEnv::default()); assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned()))); } @@ -953,8 +910,7 @@ mod tests { let runtime = dir.path().join("runtime"); let applied = stage_limits( "sock", - &dir.path().join("resource-limits.json"), - &dir.path().join("swarm-resource-limits.json"), + &swarm_file(dir), &dir.path().join("hyperhive-limits.conf"), &runtime, &HiveEnv::default(), @@ -965,40 +921,18 @@ mod tests { (applied.map(|(_, mem)| mem), copy) } - /// The agent's copy follows the swarm's cap, the one systemd enforces, over - /// the operator's override: a heap sized against the override would - /// outgrow the cgroup. - #[test] - fn the_agent_reads_the_swarm_cap_over_the_operator_override() { - let dir = tempfile::tempdir().expect("tempdir"); - std::fs::write( - dir.path().join("resource-limits.json"), - r#"{"sock": {"memory_max": "16G"}}"#, - ) - .expect("seed limits"); - std::fs::write( - dir.path().join("swarm-resource-limits.json"), - r#"{"sock": {"memory_max": "2G"}}"#, - ) - .expect("seed swarm caps"); - assert_eq!( - stage(&dir), - (Some("2G".to_owned()), Some("2147483648\n".to_owned())) - ); - } - #[test] fn the_agent_reads_the_cap_its_dropin_carries() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); + let swarm = swarm_file(&dir); - // No overrides anywhere: the hive-wide default, 4G. + // No swarm cap anywhere: the hive-wide default, 4G. assert_eq!( stage(&dir), (Some("4G".to_owned()), Some("4294967296\n".to_owned())) ); - std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits"); + std::fs::write(&swarm, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed swarm caps"); assert_eq!( stage(&dir), (Some("16G".to_owned()), Some("17179869184\n".to_owned())) @@ -1008,17 +942,17 @@ mod tests { #[test] fn every_kind_of_cap_is_written_for_the_agent() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); + let swarm = swarm_file(&dir); for (memory_max, copy) in [ ("infinity", "infinity\n"), ("50%", "8192004096\n"), ("8G", "8589934592\n"), ] { std::fs::write( - &limits, + &swarm, format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#), ) - .expect("seed limits"); + .expect("seed swarm caps"); assert_eq!( stage(&dir), (Some(memory_max.to_owned()), Some(copy.to_owned())), @@ -1072,14 +1006,14 @@ mod tests { #[test] fn a_kept_dropin_hands_the_agent_its_cap() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); + let swarm = swarm_file(&dir); let dropin = dir.path().join("hyperhive-limits.conf"); let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE); - std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits"); + std::fs::write(&swarm, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed swarm caps"); assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned())); - // Unreadable limits keep the drop-in; the copy follows the drop-in. - std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); + // Unreadable caps keep the drop-in; the copy follows the drop-in. + std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); std::fs::write( &dropin, "[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n", @@ -1103,29 +1037,21 @@ mod tests { #[test] fn an_unknown_cap_leaves_no_copy() { let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); + let swarm = swarm_file(&dir); let dropin = dir.path().join("hyperhive-limits.conf"); assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned())); - // Unreadable limits and an unreadable drop-in: nothing says what the + // Unreadable caps and an unreadable drop-in: nothing says what the // cgroup enforces. - std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); + std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); std::fs::create_dir(&dropin).expect("seed unreadable drop-in"); assert_eq!(stage(&dir), (None, None)); // A percentage with no host memory to resolve it against. - std::fs::write(&limits, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed limits"); + std::fs::write(&swarm, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed swarm caps"); let runtime = dir.path().join("runtime"); - let applied = stage_limits( - "sock", - &limits, - &dir.path().join("swarm-resource-limits.json"), - &dropin, - &runtime, - &HiveEnv::default(), - None, - ) - .expect("stage limits"); + let applied = stage_limits("sock", &swarm, &dropin, &runtime, &HiveEnv::default(), None) + .expect("stage limits"); assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned())); assert!(!runtime.join(MEMORY_CAP_FILE).exists()); } diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index 1c8f88d6..ce916fa8 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -47,8 +47,8 @@ pub struct AgentSpec { /// Stage every generated meta JSON file that exists: topology.json is /// regenerated by `reconcile` whenever the agent set changed; -/// tool-groups/capabilities/resource-limits are created lazily on -/// first write (`set_groups`/`set_caps`/`set_limits`) — absent means +/// tool-groups/capabilities are created lazily on +/// first write (`set_groups`/`set_caps`) — absent means /// every agent is on defaults, no file needed. Without staging, an /// existing-but-untracked file shows up as untracked in the meta repo, /// which can confuse nix's dirty-tree fetch. `git add` is a no-op when @@ -61,10 +61,6 @@ async fn stage_generated_meta_files(dir: &std::path::Path) -> Result<()> { crate::capabilities::capabilities_path(), "capabilities.json", ), - ( - crate::resource_limits::resource_limits_path(), - "resource-limits.json", - ), ] { if path.exists() { git(dir, &["add", name]).await?; @@ -202,7 +198,6 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> { f if f.starts_with("peer-ca-") && has_pem_ext(f) => Some("peer-ca"), "topology.json" => Some("topology"), "capabilities.json" => Some("capabilities"), - "resource-limits.json" => Some("resource-limits"), "tool-groups.json" => Some("tool-groups"), _ => None, }) @@ -352,41 +347,6 @@ pub async fn commit_capabilities(agent: &str, caps: &[String]) -> Result<()> { Ok(()) } -/// Write the resource-limits file for `agent` and commit it atomically -/// under `META_LOCK`. Same rationale as `commit_tool_groups`: the -/// working tree must never be left dirty for the next `sync_agents` to trip -/// over. -/// -/// Unlike the perm files this one is never injected into the container — -/// it's a host-side cap *on* the agent — but it lives in the same repo -/// so a limit change gets the same auditable one-commit-per-change trail. -/// -/// # Errors -/// -/// Returns an error if writing the JSON file fails or a git stage/commit -/// step fails. -pub async fn commit_resource_limits( - agent: &str, - limits: &crate::resource_limits::AgentLimits, -) -> Result<()> { - let _guard = META_LOCK.lock().await; - crate::resource_limits::set_limits(agent, limits) - .map_err(|e| anyhow::anyhow!("set resource limits for {agent}: {e}"))?; - let dir = crate::paths::meta_root(); - if crate::resource_limits::resource_limits_path().exists() { - git(&dir, &["add", "resource-limits.json"]).await?; - } - if paths_dirty(&dir, &["resource-limits.json"]).await? { - git_commit_paths( - &dir, - &format!("set resource limits for {agent}"), - &["resource-limits.json"], - ) - .await?; - } - Ok(()) -} - /// Write both perm files for `agent` (whichever are `Some`) and commit /// them in a SINGLE git commit under `META_LOCK` — the batch /// `POST /api/permissions` path. A `None` field leaves that file diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index ea283fda..db4b4aad 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -170,7 +170,7 @@ async fn dispatch(req: &HostRequest, coord: Arc) -> HostResponse { // question is only answerable on this side of the socket — // see the request's doc comment for why the client asks. HostRequest::AgentExists { name } => HostResponse::agent_exists(agent_exists(name)?), - HostRequest::AgentStatus => handle_agent_status(&coord).await, + HostRequest::AgentStatus => handle_agent_status().await, // Intercepted in `handle()` before a request ever reaches // `dispatch` — see `stream_agent_status`'s doc comment. This // arm exists only so the match stays exhaustive; reaching it @@ -184,19 +184,6 @@ async fn dispatch(req: &HostRequest, coord: Arc) -> HostResponse { // federation peer-config block) and open the web surfaces // (`hivectl open`). HostRequest::Urls => HostResponse::urls(hive_urls()), - HostRequest::SetResourceLimits { - name, - cpu_quota, - memory_max, - } => { - handle_set_resource_limits( - &coord, - name, - cpu_quota.as_deref(), - memory_max.as_deref(), - ) - .await? - } HostRequest::MatrixInvite { user, room } => { handle_matrix_invite(user, room.as_deref()).await? } @@ -293,8 +280,8 @@ async fn handle_set_paused( } /// Collect per-agent status rows for `hivectl status` and the dashboard. -async fn handle_agent_status(coord: &Arc) -> HostResponse { - match crate::container_view::build_all(&coord.hive_env()).await { +async fn handle_agent_status() -> HostResponse { + match crate::container_view::build_all().await { Ok(views) => HostResponse::agent_statuses( views .into_iter() @@ -380,66 +367,6 @@ fn agent_exists(name: &hive_types::Ident) -> Result { .with_context(|| format!("check agent state dir for {name}")) } -/// Validate + persist an agent's CPU/memory overrides, then re-apply the -/// drop-in so the change lands without waiting for a rebuild. -/// -/// Validation is here rather than only in `hivectl` because the values -/// are written verbatim into the systemd drop-in: a malformed -/// `CPUQuota=` makes systemd reject the unit, and the container stops -/// starting. Every client (CLI, dashboard, anything later) goes through -/// this path, so the guard belongs on this side of the socket. -/// -/// `None`/`None` removes the agent's entry, returning it to the -/// hive-wide defaults. -async fn handle_set_resource_limits( - coord: &Arc, - name: &hive_types::Ident, - cpu_quota: Option<&str>, - memory_max: Option<&str>, -) -> Result { - if let Some(value) = cpu_quota { - crate::resource_limits::validate_cpu_quota(value).map_err(anyhow::Error::msg)?; - } - // `validate_memory_max` returns the value to store, not just an - // ok/err verdict: a friendly spelling like "8GB" is accepted but - // normalized to the "8G" systemd's own parser actually takes. - let memory_max = memory_max - .map(crate::resource_limits::validate_memory_max) - .transpose() - .map_err(anyhow::Error::msg)?; - tracing::info!(%name, ?cpu_quota, ?memory_max, "set_resource_limits"); - let limits = crate::resource_limits::AgentLimits { - cpu_quota: cpu_quota.map(ToOwned::to_owned), - memory_max, - }; - // Goes through `meta::commit_resource_limits`, not the bare - // `resource_limits::set_limits`: the write has to be staged + - // committed under `META_LOCK` or it leaves the meta working tree - // dirty for the next `sync_agents` to trip over. - crate::meta::commit_resource_limits(name.as_str(), &limits).await?; - - // Re-apply the drop-in straight away — same three lines as the job - // queue's `WriteDropin` node. Without this the new values would sit - // in the JSON until the agent's next spawn or rebuild. - let agent_dir = crate::paths::agent_runtime_dir(name.as_str()); - let hive = coord.hive_env(); - let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?; - crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?; - - let (cpu, mem) = crate::resource_limits::effective_with_swarm( - &crate::resource_limits::resource_limits_path(), - &crate::resource_limits::swarm_limits_path(), - name.as_str(), - &hive.agent_cpu_quota, - &hive.agent_memory_max, - )?; - Ok(HostResponse::messages(vec![format!( - "{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now; the \ - Claude/JSC heap ceiling and the subagent MemoryHigh= derived from it follow at the \ - container's next start (restart it if it's running and needs them now)" - )])) -} - /// Guard: matrix provisioning needs a homeserver to provision against. /// /// Answers "is one configured", not "is one running here": the address is diff --git a/hive-c0re/src/stats/host_stats.rs b/hive-c0re/src/stats/host_stats.rs index 6a38db23..1787ddc2 100644 --- a/hive-c0re/src/stats/host_stats.rs +++ b/hive-c0re/src/stats/host_stats.rs @@ -234,8 +234,6 @@ mod tests { status_text: None, status_set_at: None, paused: false, - cpu_quota: "200%".to_owned(), - memory_max: "4G".to_owned(), } } diff --git a/hive-c0re/src/workers/swarm_limits.rs b/hive-c0re/src/workers/swarm_limits.rs index ec3b0979..a362b356 100644 --- a/hive-c0re/src/workers/swarm_limits.rs +++ b/hive-c0re/src/workers/swarm_limits.rs @@ -3,7 +3,7 @@ //! swarm-controller keeps one key per agent in the `agent-limits` bucket //! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads //! the key of each agent this hive has and stores what it finds in -//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it. +//! `swarm-resource-limits.json`, where the drop-in reads it. //! A running agent gets a changed cap at its first spawn or rebuild after //! that read. //! @@ -92,9 +92,9 @@ async fn sync(client: &async_nats::Client) -> Result<()> { /// The caps to store for `declared`, by agent. An agent with no field set /// gets no entry. /// -/// Every value goes through the validators the operator's own input path -/// uses, because it is rendered verbatim into the container's systemd -/// drop-in. One bad value refuses the whole set. +/// Every value goes through the `resource_limits` validators, because it is +/// rendered verbatim into the container's systemd drop-in. One bad value +/// refuses the whole set. fn validated(declared: &BTreeMap) -> Result> { let mut out = BTreeMap::new(); for (agent, caps) in declared { @@ -127,48 +127,38 @@ mod tests { use std::collections::BTreeMap; use super::validated; - use crate::resource_limits::AgentLimits; use swarm_queue_client::agent_limits::AgentCaps; - /// `atlas`'s answer with `atlas_key` as the value read for it, over a hive - /// chain that caps it at 800% / 16G. + /// `atlas`'s answer with `atlas_key` as the value read for it, over the + /// hive-wide defaults of 200% / 4G. fn atlas_with(atlas_key: &str) -> (String, String) { let declared = BTreeMap::from([( "atlas".to_owned(), serde_json::from_str::(atlas_key).expect("decodes"), )]); let stored = validated(&declared).expect("valid caps"); - let hive_side = BTreeMap::from([( - "atlas".to_owned(), - AgentLimits { - cpu_quota: Some("800%".to_owned()), - memory_max: Some("16G".to_owned()), - }, - )]); - crate::resource_limits::effective_from_with_swarm( - &stored, &hive_side, "atlas", "200%", "4G", - ) + crate::resource_limits::effective_from(&stored, "atlas", "200%", "4G") } /// The other half of swarm-controller's /// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`: /// the first value is the bytes that route stores. The cap replaces the - /// hive's chain; a deleted key reads as `{}` and gives the chain back. + /// hive default; a deleted key reads as `{}` and gives the default back. #[test] - fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() { + fn the_caps_the_route_stores_replace_the_hive_default_and_a_cleared_key_restores_it() { assert_eq!( atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#), ("400%".to_owned(), "8G".to_owned()) ); - assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned())); + assert_eq!(atlas_with("{}"), ("200%".to_owned(), "4G".to_owned())); } - /// Per field: a swarm memory cap leaves CPU on the hive's chain. + /// Per field: a swarm memory cap leaves CPU on the hive default. #[test] fn a_field_the_swarm_leaves_unset_stays_with_the_hive() { assert_eq!( atlas_with(r#"{"memory_max":"2G"}"#), - ("800%".to_owned(), "2G".to_owned()) + ("200%".to_owned(), "2G".to_owned()) ); } diff --git a/hive-host-sock/src/lib.rs b/hive-host-sock/src/lib.rs index e2bc0c78..4adc8233 100644 --- a/hive-host-sock/src/lib.rs +++ b/hive-host-sock/src/lib.rs @@ -211,23 +211,6 @@ pub enum HostRequest { /// this request just answers whatever it's asked. Result: /// [`HostResponse::nodes`]. QueueNodes { ids: Vec }, - /// Declare an agent's CPU/memory overrides for the per-container - /// systemd drop-in, persisted to `meta/resource-limits.json`. - /// - /// **Replace, not merge** — the pair given here becomes the agent's - /// entire entry, matching how tool-groups/capabilities are set. A - /// `None` field falls back to the hive-wide - /// `agentCpuQuota` / `agentMemoryMax`, so passing both as `None` - /// removes the entry entirely (reset to hive defaults). - /// - /// Values are passed verbatim to systemd, so the server validates - /// their shape before persisting: a malformed `CPUQuota=` makes - /// systemd reject the unit, which would stop the container starting. - SetResourceLimits { - name: Ident, - cpu_quota: Option, - memory_max: Option, - }, /// Stop managed containers hive-wide in one operator action /// (`hivectl stop`): agents plus the selected infra containers. `scope` /// selects which classes; an all-false scope means **everything** (the diff --git a/hivectl/src/agents.rs b/hivectl/src/agents.rs index f8400580..0d98eca5 100644 --- a/hivectl/src/agents.rs +++ b/hivectl/src/agents.rs @@ -1,6 +1,6 @@ //! `hivectl agent ` — everything scoped to ONE managed agent: //! container lifecycle over the host admin socket (restart/pause/resume/ -//! spawn/kill/destroy/rebuild/set-limits/choom/watch), plus the +//! spawn/kill/destroy/rebuild/choom/watch), plus the //! `quota` and `subvol` groups, whose handlers live in their own modules. //! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide //! read that lives in this module too since it shares the same daemon @@ -282,32 +282,6 @@ pub(crate) async fn run_agent(socket: &Path, name: &str, cmd: AgentCmd) -> Resul let name = crate::util::parse_ident(name)?; render(crate::client::request(socket, HostRequest::Rebuild { name }).await?) } - AgentCmd::SetLimits { - cpu_quota, - memory_max, - reset, - } => { - let name = crate::util::parse_ident(name)?; - // `--reset` is the only way to reach an all-`None` request; - // clap rejects a bare `set-limits` with neither flag, so a - // forgotten value can't silently clear the overrides. - let (cpu_quota, memory_max) = if reset { - (None, None) - } else { - (cpu_quota, memory_max) - }; - render( - crate::client::request( - socket, - HostRequest::SetResourceLimits { - name, - cpu_quota, - memory_max, - }, - ) - .await?, - ) - } AgentCmd::Choom { resume_session } => { crate::choom::choom(socket, name, resume_session.as_deref()).await } diff --git a/hivectl/src/cli.rs b/hivectl/src/cli.rs index 89348e78..b407d745 100644 --- a/hivectl/src/cli.rs +++ b/hivectl/src/cli.rs @@ -393,30 +393,6 @@ pub enum AgentCmd { }, /// Apply pending config to this managed container. Rebuild, - /// Declare this agent's CPU/memory limits, overriding the hive-wide - /// defaults. - /// - /// Replaces the agent's whole override entry rather than merging into - /// it: any limit you don't pass returns to the hive-wide default. To - /// change one and keep the other, pass both. Disk is a separate - /// resource with its own group — see `quota`. - SetLimits { - /// systemd `CPUQuota=` value, for example `400%` (100% = one full core). - #[arg(long, conflicts_with = "reset")] - cpu_quota: Option, - /// systemd `MemoryMax=` value, for example `8G`, `50%`, or `infinity`. - #[arg(long, conflicts_with = "reset")] - memory_max: Option, - /// Drop all overrides — the agent returns to the hive-wide - /// defaults. Required to clear limits, so that a `set-limits` - /// with a forgotten value can't silently reset the agent. - #[arg( - long, - conflicts_with_all = ["cpu_quota", "memory_max"], - required_unless_present_any = ["cpu_quota", "memory_max"], - )] - reset: bool, - }, /// Open an interactive Claude session inside this agent's container. /// /// A fresh session by default, or resume a prior one. Requires root