diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index 851f7e73c..3ca1043c9 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -411,13 +411,6 @@ Contents: default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same drop-in have **no** per-agent override — they're hive-wide only and come straight off `HiveEnv`, so this file has no field for them. -- `swarm-resource-limits.json` — the caps the swarm sets for this - hive's agents, same shape as `resource-limits.json`. Written only by - `workers::swarm_limits`, which copies each agent's key of the swarm's - `agent-limits` bucket here every 30 seconds. A field set here replaces - that field's answer from `resource-limits.json` and the hive-wide - default. Every value passes the same validators as the operator's - overrides first, and one bad value makes the worker change nothing here. The root agent has the meta dir RO-mounted at `/meta/`. diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index 043c34b9e..20d4dab39 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -452,24 +452,14 @@ regardless. rebuild, so changes take effect on the next lifecycle op without requiring a host rebuild. -The answer per agent and per field is, first match wins: - -1. the swarm's cap for the agent (see below); -2. the operator's per-agent override in `meta/resource-limits.json`; -3. `agentCpuQuota` / `agentMemoryMax`. - -A swarm cap replaces the hive's answer, so a hive can't raise an agent -above it. Like the rest of the drop-in, a changed swarm cap reaches a -running agent at its next spawn or rebuild. - The container can't read its own cap, so `write_dropins` also writes the effective `MemoryMax=` into the agent's runtime dir (`/run/hive/memory-max-bytes` inside) before every start: a byte count, or `infinity` for no cap. A percentage becomes bytes the way systemd resolves it, as that share of the host's physical memory rounded down to a page. A value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps -the last drop-in because it can't read the overrides or the swarm caps file, -the file carries that drop-in's `MemoryMax=`. +the last drop-in because it can't read the overrides file, the file carries +that drop-in's `MemoryMax=`. At boot the container derives two values from a byte count: `BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on @@ -479,39 +469,6 @@ changed cap reaches them at the next container start without a rebuild. `hyperhive-memory-cap-check.service` logs a warning naming the path in the container's journal. The container's `MemoryMax=` still applies. -
Implementation: swarm caps - -swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per -agent and no hive in it, so an agent that moves to another hive keeps its -caps. swarm-controller alone writes it: - -| method | path | body | -| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- | -| `GET` | `/api/agents//limits` | —; answers the stored caps, `{}` when there are none | -| `PUT` | `/api/agents//limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value | -| `DELETE` | `/api/agents//limits` | —; answers 204 | - -`PUT` refuses an agent no hive's wanted state declares, unknown fields, and -any value systemd would ignore, which would leave the agent uncapped. A -percentage has at most two decimals, no leading zero in its whole part (it -refuses `08%`), and is above zero, up to `100%` for -`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an -optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at -least one byte and below 2^64 - 1 bytes. The controller stores each value as -given. - -Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each -agent it has and stores the result in `meta/swarm-resource-limits.json`. Each -value passes the same validators as the operator's overrides first. One bad -value or a failed read leaves the file unchanged, and repeated failures raise a -dashboard banner. - -A hive's queue grant reads every key of the bucket -(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the -hive's own agents would need a roster in the auth responder. - -
- The same drop-in carries `CPUWeight=` / `IOWeight=` from `agentCpuWeight` / `agentIoWeight`. Those are a different kind of setting: the quota and the memory max are **hard caps** that throttle diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 0535bd71c..8c89734ce 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -25,9 +25,6 @@ grafana, the controller, authelia, …) → kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md) - **inference settings** — the model and effort the swarm sets for an agent, over its own. → [`inference.md`](inference.md) -- **resource caps** — the CPU and memory cap the swarm sets for an agent, - over its hive's. → - [`coordinator.md`](../scheduler/coordinator.md#container-resource-limits) - **review classifier** — planned, not implemented yet: checks a PR against its repo's own yes/no rules and posts the result as a review. → [`review-classifier.md`](review-classifier.md) diff --git a/hive-c0re/Cargo.toml b/hive-c0re/Cargo.toml index 9e8d0f0c0..ac74773ff 100644 --- a/hive-c0re/Cargo.toml +++ b/hive-c0re/Cargo.toml @@ -53,11 +53,7 @@ strum.workspace = true # Offering this hive's status to the swarm (`swarm_status`). The same crate # the swarm controller reads it with, and `kv` for the same reason: the # bucket's name and creation config belong to neither end of it alone. -swarm-queue-client = { workspace = true, features = [ - "kv", - "notices", - "agent-limits", -] } +swarm-queue-client = { workspace = true, features = ["kv", "notices"] } swarm-secret-client.workspace = true tokio.workspace = true tokio-stream.workspace = true diff --git a/hive-c0re/src/agent_config/resource_limits.rs b/hive-c0re/src/agent_config/resource_limits.rs index 880c87da0..fd68b78ae 100644 --- a/hive-c0re/src/agent_config/resource_limits.rs +++ b/hive-c0re/src/agent_config/resource_limits.rs @@ -187,86 +187,6 @@ fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Resu super::write_map(path, ¤t) } -/// Caps the swarm controller declared for this hive's agents. -/// -/// A separate file from the operator's overrides above, not a merge into it: -/// the two have different writers and different lifetimes, and a swarm -/// declaration landing in the file an operator edits would make "reset to -/// hive defaults" destroy something the operator never set. -const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json"; - -#[must_use] -pub fn swarm_limits_path() -> PathBuf { - crate::paths::meta_root().join(SWARM_LIMITS_FILE) -} - -/// Read the swarm-declared caps. An absent file is the empty map — no swarm -/// opinion, so the hive's own chain decides. A file that exists but can't be -/// read or parsed is an error, for the reason [`read`] gives: reading it as -/// empty would silently drop a cap. -pub fn read_swarm() -> std::io::Result> { - super::read_map(&swarm_limits_path()) -} - -/// Persist the swarm-declared caps atomically. -/// -/// # Errors -/// The meta dir cannot be created, or the file cannot be written. -pub fn write_swarm(map: &BTreeMap) -> std::io::Result<()> { - super::write_map(&swarm_limits_path(), map) -} - -/// [`effective_from`] with the swarm's declaration on top. -/// -/// A swarm cap **replaces** the hive's answer for that field rather than -/// joining the fallback chain as another tier: a tier a hive can out-set is -/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's -/// chain. -/// -/// The chain itself is untouched — [`effective_from`] is called exactly as -/// before and its result is what a missing swarm value falls back to. -#[must_use] -pub fn effective_from_with_swarm( - swarm: &BTreeMap, - limits: &BTreeMap, - name: &str, - hive_cpu_quota: &str, - hive_memory_max: &str, -) -> (String, String) { - let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max); - match swarm.get(name) { - Some(s) => ( - s.cpu_quota.clone().unwrap_or(cpu), - s.memory_max.clone().unwrap_or(mem), - ), - None => (cpu, mem), - } -} - -/// [`effective_from_with_swarm`] for a caller resolving a single agent, from -/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`. -/// -/// Delegates to [`effective`] rather than re-deriving the hive's answer, so -/// there stays exactly one place the fallback chain lives. Errors as [`read`] -/// does, for either file. -pub fn effective_with_swarm( - limits_path: &Path, - swarm_path: &Path, - name: &str, - hive_cpu_quota: &str, - hive_memory_max: &str, -) -> std::io::Result<(String, String)> { - let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?; - let swarm: BTreeMap = super::read_map(swarm_path)?; - Ok(match swarm.get(name) { - Some(s) => ( - s.cpu_quota.clone().unwrap_or(cpu), - s.memory_max.clone().unwrap_or(mem), - ), - None => (cpu, mem), - }) -} - /// Validate a systemd `CPUQuota=` value. Percentages only, and values /// above 100% are legitimate (one full core is 100%). /// @@ -605,52 +525,4 @@ mod tests { ); } } - - #[test] - fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() { - // The whole point of the ruling: a tier a hive can out-set is not a - // cap. The agent override below is the strongest hive-side value - // there is, and the swarm still wins. - let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]); - let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]); - assert_eq!( - effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM), - ("150%".to_owned(), "2G".to_owned()) - ); - } - - #[test] - fn a_swarm_cap_is_per_field_like_every_other_tier() { - // A swarm that caps memory and says nothing about CPU leaves CPU on - // the hive's chain — here the per-agent override, not the hive default. - let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]); - let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]); - assert_eq!( - effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM), - ("800%".to_owned(), "2G".to_owned()) - ); - } - - #[test] - fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() { - // The control that makes the two tests above mean something: the - // wrapper must be a no-op when the swarm has said nothing, for every - // shape the old chain distinguishes. If this drifts, the swarm tier - // changed behaviour it was explicitly told not to change. - let none = BTreeMap::new(); - let cases = [ - BTreeMap::new(), - BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]), - BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]), - BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]), - BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]), - ]; - for hive_side in &cases { - assert_eq!( - effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM), - effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM), - "the wrapper changed the answer with no swarm value present" - ); - } - } } diff --git a/hive-c0re/src/container_view.rs b/hive-c0re/src/container_view.rs index d55686cc8..012a9451f 100644 --- a/hive-c0re/src/container_view.rs +++ b/hive-c0re/src/container_view.rs @@ -171,14 +171,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result anyhow::Result Re if let Some((cpu_quota, memory_max)) = stage_limits( name, &crate::resource_limits::resource_limits_path(), - &crate::resource_limits::swarm_limits_path(), &limits_dropin_path(&container), &paths.agent, hive, @@ -98,13 +97,12 @@ impl HostMemory { fn stage_limits( agent_name: &str, limits_path: &Path, - swarm_path: &Path, dropin_path: &Path, runtime_dir: &Path, hive: &HiveEnv, host: Option, ) -> Result> { - let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive); + let limits = limits_to_apply(agent_name, limits_path, dropin_path, hive); let memory_max = match &limits { Some((_, memory_max)) => Some(memory_max.clone()), None => dropin_memory_max(agent_name, dropin_path), @@ -226,28 +224,20 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> { /// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits /// drop-in, or `None` to leave the drop-in at `dropin_path` as it is. /// -/// When the overrides file at `limits_path` or the swarm caps at -/// `swarm_path` can't be read, the error is logged and the agent still starts. -/// The last-applied drop-in is kept, because the hive defaults can be looser -/// than the agent's own override. +/// When the overrides file at `limits_path` can't be read, the error is +/// logged and the agent still starts. The last-applied drop-in is kept, +/// because the hive defaults can be looser than the agent's own override. /// With no drop-in yet, the agent gets the hive defaults: without one it /// would run uncapped. fn limits_to_apply( agent_name: &str, limits_path: &Path, - swarm_path: &Path, dropin_path: &Path, hive: &HiveEnv, ) -> Option<(String, String)> { let hive_cpu = &hive.agent_cpu_quota; let hive_mem = &hive.agent_memory_max; - let e = match crate::resource_limits::effective_with_swarm( - limits_path, - swarm_path, - agent_name, - hive_cpu, - hive_mem, - ) { + let e = match crate::resource_limits::effective(limits_path, agent_name, hive_cpu, hive_mem) { Ok(limits) => return Some(limits), Err(e) => e, }; @@ -848,10 +838,6 @@ mod tests { const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem"; - fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf { - dir.path().join("absent-swarm-resource-limits.json") - } - #[test] fn corrupt_limits_keep_an_existing_dropin() { let dir = tempfile::tempdir().expect("tempdir"); @@ -860,13 +846,7 @@ mod tests { std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); assert_eq!( - limits_to_apply( - "sock", - &limits, - &absent_swarm(&dir), - &dropin, - &HiveEnv::default() - ), + limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()), None ); assert_eq!( @@ -883,47 +863,12 @@ mod tests { let applied = limits_to_apply( "sock", &limits, - &absent_swarm(&dir), &dir.path().join("absent.conf"), &HiveEnv::default(), ); assert_eq!(applied, Some(("200%".to_owned(), "4G".to_owned()))); } - #[test] - fn corrupt_swarm_caps_keep_an_existing_dropin() { - let dir = tempfile::tempdir().expect("tempdir"); - let swarm = dir.path().join("swarm-resource-limits.json"); - let dropin = dir.path().join("hyperhive-limits.conf"); - std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); - std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); - let applied = limits_to_apply( - "sock", - &dir.path().join("resource-limits.json"), - &swarm, - &dropin, - &HiveEnv::default(), - ); - assert_eq!(applied, None); - } - - #[test] - fn a_swarm_cap_is_applied_over_the_operator_override() { - let dir = tempfile::tempdir().expect("tempdir"); - let limits = dir.path().join("resource-limits.json"); - let swarm = dir.path().join("swarm-resource-limits.json"); - std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits"); - std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps"); - let applied = limits_to_apply( - "sock", - &limits, - &swarm, - &dir.path().join("absent.conf"), - &HiveEnv::default(), - ); - assert_eq!(applied, Some(("200%".to_owned(), "2G".to_owned()))); - } - #[test] fn readable_limits_are_applied_over_an_existing_dropin() { let dir = tempfile::tempdir().expect("tempdir"); @@ -931,13 +876,7 @@ mod tests { let dropin = dir.path().join("hyperhive-limits.conf"); std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits"); std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in"); - let applied = limits_to_apply( - "sock", - &limits, - &absent_swarm(&dir), - &dropin, - &HiveEnv::default(), - ); + let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()); assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned()))); } @@ -954,7 +893,6 @@ mod tests { let applied = stage_limits( "sock", &dir.path().join("resource-limits.json"), - &dir.path().join("swarm-resource-limits.json"), &dir.path().join("hyperhive-limits.conf"), &runtime, &HiveEnv::default(), @@ -965,28 +903,6 @@ mod tests { (applied.map(|(_, mem)| mem), copy) } - /// The agent's copy follows the swarm's cap, the one systemd enforces, over - /// the operator's override: a heap sized against the override would - /// outgrow the cgroup. - #[test] - fn the_agent_reads_the_swarm_cap_over_the_operator_override() { - let dir = tempfile::tempdir().expect("tempdir"); - std::fs::write( - dir.path().join("resource-limits.json"), - r#"{"sock": {"memory_max": "16G"}}"#, - ) - .expect("seed limits"); - std::fs::write( - dir.path().join("swarm-resource-limits.json"), - r#"{"sock": {"memory_max": "2G"}}"#, - ) - .expect("seed swarm caps"); - assert_eq!( - stage(&dir), - (Some("2G".to_owned()), Some("2147483648\n".to_owned())) - ); - } - #[test] fn the_agent_reads_the_cap_its_dropin_carries() { let dir = tempfile::tempdir().expect("tempdir"); @@ -1119,7 +1035,6 @@ mod tests { let applied = stage_limits( "sock", &limits, - &dir.path().join("swarm-resource-limits.json"), &dropin, &runtime, &HiveEnv::default(), diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index ea283fda3..5d9839bc3 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -426,9 +426,8 @@ async fn handle_set_resource_limits( let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?; crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?; - let (cpu, mem) = crate::resource_limits::effective_with_swarm( + let (cpu, mem) = crate::resource_limits::effective( &crate::resource_limits::resource_limits_path(), - &crate::resource_limits::swarm_limits_path(), name.as_str(), &hive.agent_cpu_quota, &hive.agent_memory_max, diff --git a/hive-c0re/src/swarm_status.rs b/hive-c0re/src/swarm_status.rs index c4cfc54b6..13a1d455c 100644 --- a/hive-c0re/src/swarm_status.rs +++ b/hive-c0re/src/swarm_status.rs @@ -103,12 +103,6 @@ pub fn spawn( shutdown.clone(), )); - // The swarm's resource caps, on the same connection again. - tokio::spawn(crate::workers::swarm_limits::follow( - client.clone(), - shutdown.clone(), - )); - let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER); loop { match publish(&client, &hive).await { diff --git a/hive-c0re/src/workers/mod.rs b/hive-c0re/src/workers/mod.rs index 6f2ffa4c8..c3772e923 100644 --- a/hive-c0re/src/workers/mod.rs +++ b/hive-c0re/src/workers/mod.rs @@ -1,7 +1,6 @@ //! Background tasks and periodic sweeps: crash/login watcher, boot-time //! auto-update reconcile, the agent-sockets.json writer loop, the MCP socket listener -//! reconcile loop, knowledge-repo sync, the swarm wanted-state pull, and the -//! swarm resource-cap read. +//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull. //! Each submodule is re-exported //! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged. @@ -10,5 +9,4 @@ pub mod auto_update; pub mod crash_watch; pub mod knowledge; pub mod mcp_sockets; -pub mod swarm_limits; pub mod wanted; diff --git a/hive-c0re/src/workers/swarm_limits.rs b/hive-c0re/src/workers/swarm_limits.rs deleted file mode 100644 index ec3b09790..000000000 --- a/hive-c0re/src/workers/swarm_limits.rs +++ /dev/null @@ -1,207 +0,0 @@ -//! Reading the caps the swarm sets for this hive's agents. -//! -//! swarm-controller keeps one key per agent in the `agent-limits` bucket -//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads -//! the key of each agent this hive has and stores what it finds in -//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it. -//! A running agent gets a changed cap at its first spawn or rebuild after -//! that read. -//! -//! A failed read, or a bucket that does not answer, leaves the stored caps as -//! they are: dropping them would read exactly like the swarm clearing them. - -use std::collections::BTreeMap; -use std::time::Duration; - -use anyhow::{Context, Result}; -use swarm_queue_client::agent_limits::{self, AgentCaps, Read}; - -use crate::resource_limits::AgentLimits; -use crate::stats::sweep_health::{self, SweepHealth}; - -/// How often the caps are read. -pub const READ_INTERVAL: Duration = Duration::from_secs(30); - -/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads. -const FAILURES_BEFORE_BANNER: u32 = 3; - -/// Read the caps now and every [`READ_INTERVAL`] until shutdown. -pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver) { - let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER); - loop { - match sync(&client).await { - Ok(()) => health.record_ok(), - Err(e) => { - tracing::warn!(error = ?e, "swarm limits: reading the caps failed"); - let err = format!("{e:#}"); - health.record_err(|ctx| { - let age = ctx.since_last_ok.map_or_else( - || "no success this session".to_owned(), - |d| format!("last ok {} ago", sweep_health::fmt_age(d)), - ); - format!( - "reading the swarm's resource caps is failing ({} consecutive, {age}) \ - — agents keep the caps last read: {err}", - ctx.consecutive - ) - }); - } - } - tokio::select! { - () = tokio::time::sleep(READ_INTERVAL) => {} - _ = shutdown.changed() => { - tracing::info!("swarm limits: shutdown signal received"); - return; - } - } - } -} - -/// Read every agent's key and store the result, unless a read fails or a -/// value is refused. -async fn sync(client: &async_nats::Client) -> Result<()> { - // An unconnected client does not fail a JetStream request, it hangs on it. - swarm_queue_client::ensure_connected(client)?; - let agents = crate::lifecycle::agents_for_meta_listing() - .await - .context("enumerating this hive's agents")?; - let mut declared = BTreeMap::new(); - for spec in agents { - let read = agent_limits::read(client, &spec.name) - .await - .map_err(|e| anyhow::anyhow!(e)) - .with_context(|| format!("reading the swarm's caps for {}", spec.name))?; - match read { - Read::Current(caps) => { - declared.insert(spec.name, caps); - } - Read::NoBucket => { - tracing::debug!("swarm limits: the agent-limits bucket does not answer"); - return Ok(()); - } - } - } - let caps = validated(&declared)?; - if caps != crate::resource_limits::read_swarm().context("reading the stored swarm caps")? { - crate::resource_limits::write_swarm(&caps).context("storing the swarm caps")?; - tracing::info!(?caps, "swarm limits: stored the swarm's resource caps"); - } - Ok(()) -} - -/// The caps to store for `declared`, by agent. An agent with no field set -/// gets no entry. -/// -/// Every value goes through the validators the operator's own input path -/// uses, because it is rendered verbatim into the container's systemd -/// drop-in. One bad value refuses the whole set. -fn validated(declared: &BTreeMap) -> Result> { - let mut out = BTreeMap::new(); - for (agent, caps) in declared { - if caps.is_empty() { - continue; - } - let limits = validated_caps(caps) - .map_err(|why| anyhow::anyhow!("the swarm set an invalid cap for {agent}: {why}"))?; - out.insert(agent.clone(), limits); - } - Ok(out) -} - -fn validated_caps(caps: &AgentCaps) -> Result { - if let Some(cpu) = &caps.cpu_quota { - crate::resource_limits::validate_cpu_quota(cpu)?; - } - Ok(AgentLimits { - cpu_quota: caps.cpu_quota.clone(), - memory_max: caps - .memory_max - .as_deref() - .map(crate::resource_limits::validate_memory_max) - .transpose()?, - }) -} - -#[cfg(test)] -mod tests { - use std::collections::BTreeMap; - - use super::validated; - use crate::resource_limits::AgentLimits; - use swarm_queue_client::agent_limits::AgentCaps; - - /// `atlas`'s answer with `atlas_key` as the value read for it, over a hive - /// chain that caps it at 800% / 16G. - fn atlas_with(atlas_key: &str) -> (String, String) { - let declared = BTreeMap::from([( - "atlas".to_owned(), - serde_json::from_str::(atlas_key).expect("decodes"), - )]); - let stored = validated(&declared).expect("valid caps"); - let hive_side = BTreeMap::from([( - "atlas".to_owned(), - AgentLimits { - cpu_quota: Some("800%".to_owned()), - memory_max: Some("16G".to_owned()), - }, - )]); - crate::resource_limits::effective_from_with_swarm( - &stored, &hive_side, "atlas", "200%", "4G", - ) - } - - /// The other half of swarm-controller's - /// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`: - /// the first value is the bytes that route stores. The cap replaces the - /// hive's chain; a deleted key reads as `{}` and gives the chain back. - #[test] - fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() { - assert_eq!( - atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#), - ("400%".to_owned(), "8G".to_owned()) - ); - assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned())); - } - - /// Per field: a swarm memory cap leaves CPU on the hive's chain. - #[test] - fn a_field_the_swarm_leaves_unset_stays_with_the_hive() { - assert_eq!( - atlas_with(r#"{"memory_max":"2G"}"#), - ("800%".to_owned(), "2G".to_owned()) - ); - } - - #[test] - fn an_agent_with_no_caps_gets_no_entry() { - let declared = BTreeMap::from([("atlas".to_owned(), AgentCaps::default())]); - assert!(validated(&declared).expect("valid").is_empty()); - } - - /// The values are rendered verbatim into a systemd drop-in, so a value - /// the hive's own validators refuse, a newline smuggling in a directive - /// among them, refuses the whole set, the valid cap beside it included. - #[test] - fn an_invalid_cap_is_refused_and_nothing_is_stored() { - for bad in [ - r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#, - r#"{"memory_max":"8G\nDevicePolicy=auto"}"#, - r#"{"cpu_quota":"lots"}"#, - r#"{"memory_max":"8X"}"#, - ] { - let declared = BTreeMap::from([ - ("atlas".to_owned(), AgentCaps::default()), - ( - "iris".to_owned(), - serde_json::from_str(r#"{"memory_max":"2G"}"#).expect("decodes"), - ), - ( - "sock".to_owned(), - serde_json::from_str(bad).expect("decodes"), - ), - ]); - let err = validated(&declared).expect_err(bad); - assert!(err.to_string().contains("sock"), "{err:#}"); - } - } -} diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index 48326b2f4..f99c1a11d 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -286,22 +286,6 @@ let && !(lib.hasInfix "mint swarmctl" s) && lib.hasInfix "mint swarm-authelia-bridge" s; } - { - # swarm-controller checks no caller itself: this location's authelia - # subrequest is the operator gate for every route under `/api/`. The - # webhook location is the control, the one path to the same socket that - # skips the gate, so the first clause reads a string that can lack it. - name = "the swarm UI's API location asks authelia before reaching the controller"; - ok = - let - l = allLocal.services.nginx.virtualHosts."t.local".locations; - socket = allLocal.services.hyperhive.deploy.swarm-controller.socketPath; - in - lib.hasInfix "auth_request /__hive_authelia;" l."/api/".extraConfig - && lib.hasInfix socket l."/api/".proxyPass - && lib.hasInfix socket l."/webhook/forge/".proxyPass - && !(lib.hasInfix "auth_request" l."/webhook/forge/".extraConfig); - } { name = "a hive with matrix off serves no matrix discovery endpoint"; ok = diff --git a/swarm-controller/README.md b/swarm-controller/README.md index 4c7e12d4a..036ff2836 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -52,11 +52,6 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients. agent applies them itself. `GET` also answers the model and effort the agent last reported in its turn-state header, kept in memory → [`docs/swarm/inference.md`](../docs/swarm/inference.md). -- **Resource caps** — each agent's CPU and memory cap, one key per agent in - the `agent-limits` KV bucket, which this daemon alone writes - (`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the - agent reads it, and each cap replaces the hive's own answer for that field → - [`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits). - **Relays** — each agent's terminal and turn-state header as SSE, agent icons, the cross-repo issue report, and the UI's quick links. diff --git a/swarm-controller/src/agent_limits.rs b/swarm-controller/src/agent_limits.rs deleted file mode 100644 index 90351ab31..000000000 --- a/swarm-controller/src/agent_limits.rs +++ /dev/null @@ -1,699 +0,0 @@ -//! `/api/agents/{name}/limits` — the CPU and memory caps the swarm sets for -//! one agent. -//! -//! Each agent's caps are one value under its own key of `agent-limits` -//! (`swarm_queue_client::agent_limits`), which this daemon alone writes. The -//! hive running the agent reads the key; on the hive, a cap replaces the -//! hive's own answer for that field, and a field left unset stays with the -//! hive. -//! -//! The hive hands these values to systemd verbatim, and systemd ignores a value -//! it can't parse, leaving the agent uncapped, so they are checked here against -//! systemd's own syntax (`Caps`) as well as on the hive. - -use axum::Json; -use axum::extract::{Path, State}; -use axum::http::StatusCode; -use problem_details::ProblemDetails; -use serde::{Deserialize, Serialize}; -use swarm_queue_client::agent_limits::{self, AgentCaps}; -use utoipa::ToSchema; - -use crate::schedules::{live, queue, read, refused, write}; -use crate::{AppState, error_problem, wanted_error_status}; - -/// How many times a losing writer re-reads and re-applies, as in -/// `wanted::MAX_ATTEMPTS`. -const MAX_ATTEMPTS: usize = 5; - -/// One agent's caps. A field left out is not capped by the swarm. -/// -/// Each value is stored as given, and only in a form systemd applies: a value -/// systemd ignores ("Invalid …, ignoring") would leave the agent uncapped, -/// since a swarm cap replaces the hive's answer. -#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] -#[serde(deny_unknown_fields)] -pub struct Caps { - /// systemd `CPUQuota=`: digits, optionally `.` and one or two digits, then - /// `%`; above zero and at most `21474836.47%`. `"100%"` is one core. - #[serde(default, skip_serializing_if = "Option::is_none")] - #[schema(example = "400%")] - cpu_quota: Option, - /// systemd `MemoryMax=`: `infinity`; a percentage of physical memory - /// written as for `cpu_quota`, above zero and at most `100%`; or a size: - /// digits, optionally `.` and digits, then nothing, `B` or one of - /// `K`/`M`/`G`/`T` (1024-based, upper case), at least one byte and below - /// 2^64 - 1 bytes. - #[serde(default, skip_serializing_if = "Option::is_none")] - #[schema(example = "8G")] - memory_max: Option, -} - -impl Caps { - /// The value to store, or why it is refused. - fn into_stored(self) -> Result { - Ok(AgentCaps { - cpu_quota: self.cpu_quota.as_deref().map(cpu_quota).transpose()?, - memory_max: self.memory_max.as_deref().map(memory_max).transpose()?, - }) - } -} - -impl From for Caps { - fn from(v: AgentCaps) -> Self { - Self { - cpu_quota: v.cpu_quota, - memory_max: v.memory_max, - } - } -} - -// The accepted forms are a subset of what systemd parses (checked against -// `systemd-analyze verify`, systemd 260) and stricter than hive-c0re's own -// `validate_cpu_quota` / `validate_memory_max`, so the hive never refuses a -// stored value. - -/// systemd keeps `CPUQuota=` in hundredths of a percent in a C `int`. -const CPU_QUOTA_MAX_HUNDREDTHS: u64 = 2_147_483_647; - -fn cpu_quota(value: &str) -> Result { - match hundredths(value) { - Some(h) if (1..=CPU_QUOTA_MAX_HUNDREDTHS).contains(&h) => Ok(value.to_owned()), - _ => Err(format!( - "invalid cpu_quota {value:?}: expected a percentage above zero with at most two \ - decimals, such as \"200%\" (100% = one full core), up to \"21474836.47%\"" - )), - } -} - -fn memory_max(value: &str) -> Result { - let applies = value == "infinity" - || if value.ends_with('%') { - hundredths(value).is_some_and(|h| (1..=10_000).contains(&h)) - } else { - size_bytes(value).is_some_and(|b| b >= 1 && b < u128::from(u64::MAX)) - }; - if applies { - Ok(value.to_owned()) - } else { - Err(format!( - "invalid memory_max {value:?}: expected a size above zero such as \"8G\" \ - (suffix B, K, M, G or T, upper case), a percentage up to \"100%\" with at \ - most two decimals, or \"infinity\"" - )) - } -} - -/// `value` as systemd reads a percentage, in hundredths of a percent: digits, -/// optionally `.` and one or two digits, then `%`. `None` for anything else, -/// and for a whole part with a leading zero other than `0` itself: systemd -/// ignores `08%` and may read `010%` as octal. -fn hundredths(value: &str) -> Option { - let number = value.strip_suffix('%')?; - let (int, frac) = match number.split_once('.') { - Some((int, frac)) if (1..=2).contains(&frac.len()) && digits(frac) => (int, frac), - Some(_) => return None, - None => (number, ""), - }; - if !digits(int) || (int.len() > 1 && int.starts_with('0')) { - return None; - } - let frac = match frac.len() { - 0 => 0, - 1 => frac.parse::().ok()? * 10, - _ => frac.parse::().ok()?, - }; - int.parse::().ok()?.checked_mul(100)?.checked_add(frac) -} - -/// `value` as one term of systemd's `parse_size` reads it, in bytes: digits, -/// optionally `.` and digits, then nothing, `B`, or `K`/`M`/`G`/`T` (1024-based, -/// case-sensitive). `P` and `E`, which systemd also reads, are left out -/// because hive-c0re's validator refuses them. `None` for anything else, and for what -/// `parse_size` refuses as out of range: a whole part or fraction digits past -/// `u64`, or a whole part that, plus one when there is a fraction, is past -/// `u64` once scaled. -fn size_bytes(value: &str) -> Option { - let (number, exponent) = match value.char_indices().last()? { - (i, 'B') => (&value[..i], 0), - (i, c) => match "KMGT".find(c) { - Some(k) => (&value[..i], k + 1), - None => (value, 0), - }, - }; - let scale = 1024u128.pow(u32::try_from(exponent).ok()?); - let (int, frac) = number.split_once('.').unwrap_or((number, "")); - if !digits(int) || (number.contains('.') && !digits(frac)) { - return None; - } - let whole = u128::from(int.parse::().ok()?); - let frac_value = if frac.is_empty() { - 0 - } else { - u128::from(frac.parse::().ok()?) - }; - if whole + u128::from(frac_value > 0) > u128::from(u64::MAX) / scale { - return None; - } - let frac_bytes = frac_value * scale / 10u128.checked_pow(u32::try_from(frac.len()).ok()?)?; - Some(whole * scale + frac_bytes) -} - -fn digits(s: &str) -> bool { - !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()) -} - -fn agent_name(name: &str) -> Result { - hive_types::Ident::parse(name) - .map(hive_types::Ident::into_string) - .map_err(|reason| (StatusCode::BAD_REQUEST, reason.to_owned())) -} - -fn queue_problem(e: &anyhow::Error) -> ProblemDetails { - tracing::warn!(error = %format!("{e:#}"), "agent limits: queue failure"); - error_problem(wanted_error_status(e), &format!("{e:#}")) -} - -async fn open_or_create( - client: &async_nats::Client, -) -> Result { - agent_limits::open_or_create(client) - .await - .map_err(|e| queue_problem(&anyhow::Error::new(e))) -} - -#[utoipa::path( - get, - path = "/api/agents/{name}/limits", - params(("name" = String, Path, description = "agent name")), - responses( - (status = 200, description = "the caps the swarm sets for the agent; `{}` when it \ - sets none", body = Caps), - (status = 400, description = "not an identifier (problem+json)", body = String), - (status = 503, description = "no swarm queue is configured on this host, or it is \ - not connected (problem+json)", body = String), - (status = 500, description = "the caps could not be read (problem+json)", - body = String), - ), - tag = "agents" -)] -pub(crate) async fn get_agent_limits( - State(state): State, - Path(name): Path, -) -> Result, ProblemDetails> { - let agent = agent_name(&name).map_err(refused)?; - let client = queue(&state).map_err(refused)?; - let Some(store) = agent_limits::open(&client) - .await - .map_err(|e| queue_problem(&anyhow::Error::new(e)))? - else { - return Ok(Json(Caps::default())); - }; - let (current, _) = read::(&store, &agent) - .await - .map_err(|e| queue_problem(&e))?; - Ok(Json(current.into())) -} - -/// Set the CPU and memory caps the swarm declares for one agent. -/// -/// The value replaces the stored one whole. Whichever hive runs the agent -/// applies it; a running agent picks the caps up only at its next spawn or -/// rebuild. -#[utoipa::path( - put, - path = "/api/agents/{name}/limits", - params(("name" = String, Path, description = "agent name")), - request_body = Caps, - responses( - (status = 200, description = "the caps as stored", body = Caps), - (status = 400, description = "not an identifier, not declared on any hive, or a cap \ - is not a value systemd takes or is zero (problem+json)", body = String), - (status = 422, description = "an unknown field"), - (status = 503, description = "no swarm queue is configured on this host, or it is \ - not connected (problem+json)", body = String), - (status = 500, description = "the caps could not be written (problem+json)", - body = String), - ), - tag = "agents" -)] -pub(crate) async fn put_agent_limits( - State(state): State, - Path(name): Path, - Json(req): Json, -) -> Result, ProblemDetails> { - let agent = agent_name(&name).map_err(refused)?; - let value = req - .into_stored() - .map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?; - let client = queue(&state).map_err(refused)?; - if !live(&state).await?.contains(&agent) { - return Err(error_problem( - StatusCode::BAD_REQUEST, - &format!("no hive's wanted state declares an agent {agent:?} that is not destroyed"), - )); - } - let store = open_or_create(&client).await?; - for _ in 0..MAX_ATTEMPTS { - let (current, revision) = read::(&store, &agent) - .await - .map_err(|e| queue_problem(&e))?; - if current == value - || write(&store, &agent, &value, revision) - .await - .map_err(|e| queue_problem(&e))? - { - tracing::info!(%agent, ?value, "set agent limits"); - return Ok(Json(value.into())); - } - } - Err(queue_problem(&anyhow::anyhow!( - "gave up writing the limits of {agent} after {MAX_ATTEMPTS} conflicting writes" - ))) -} - -#[utoipa::path( - delete, - path = "/api/agents/{name}/limits", - params(("name" = String, Path, description = "agent name")), - responses( - (status = 204, description = "cleared; the hive's own answer applies again"), - (status = 400, description = "not an identifier (problem+json)", body = String), - (status = 503, description = "no swarm queue is configured on this host, or it is \ - not connected (problem+json)", body = String), - (status = 500, description = "the caps could not be cleared (problem+json)", - body = String), - ), - tag = "agents" -)] -pub(crate) async fn delete_agent_limits( - State(state): State, - Path(name): Path, -) -> Result { - let agent = agent_name(&name).map_err(refused)?; - let client = queue(&state).map_err(refused)?; - let store = open_or_create(&client).await?; - store - .delete(&agent) - .await - .map_err(|e| queue_problem(&anyhow::Error::new(e)))?; - tracing::info!(%agent, "cleared agent limits"); - Ok(StatusCode::NO_CONTENT) -} - -#[cfg(test)] -mod tests { - use super::{ - Caps, cpu_quota, delete_agent_limits, get_agent_limits, memory_max, put_agent_limits, - }; - use crate::test_nats::TestNats; - use crate::wanted::WantedWriter; - use axum::extract::{Path, State}; - use axum::http::StatusCode; - use std::sync::Arc; - use swarm_queue_client::wanted::AgentState; - - /// Of the `bad` values, systemd parses `""` (no quota) and `+5%`, and the - /// last adds a directive on a second line. It ignores every other one - /// ("Invalid CPU quota …, ignoring", checked with `systemd-analyze - /// verify`), which would leave the agent uncapped. - #[test] - fn cpu_quota_takes_a_percentage_above_zero() { - for ok in [ - "400%", - "100%", - "0.5%", - "0.01%", - "1500%", - "400.12%", - "21474836.47%", - ] { - assert_eq!(cpu_quota(ok).as_deref(), Ok(ok)); - } - for bad in [ - "0%", - "0.0%", - "0.00%", - "-50%", - "400", - "4 00%", - "%", - "lots", - "", - "1e3%", - "+5%", - "400.123%", - "0.001%", - "99999999999999999999%", - "21474836.48%", - "05%", - "08%", - "085480%", - "010%", - "00.5%", - "1.%", - ".5%", - "400%\nExecStartPre=/bin/sh -c id #%", - ] { - assert!(cpu_quota(bad).is_err(), "{bad:?} was accepted"); - } - } - - /// As above for `MemoryMax=`. Of the `bad` values, systemd also parses `3P` - /// and `2E`, which hive-c0re's validator refuses, and `8 G` and `1.G`. - #[test] - fn memory_max_takes_a_size_a_share_or_infinity_that_systemd_applies() { - for ok in [ - "8G", - "512M", - "4096K", - "1048576", - "1048576B", - "1.5B", - "0.5K", - "1.5T", - "1.23456789G", - "16383T", - "16777215T", - "18446744073709551614", - "18446744073709551614.5", - "1.00000000000000000001G", - "50%", - "12.34%", - "0.01%", - "100%", - "100.00%", - "infinity", - ] { - assert_eq!(memory_max(ok).as_deref(), Ok(ok), "{ok:?}"); - } - for bad in [ - "0", - "0G", - "0.0G", - "0.5", - "0%", - "-8G", - "150%", - "100.01%", - "12.345%", - "08%", - "085480%", - "010%", - "00.5%", - "8X", - "G", - "8 G", - "", - "eight", - "8GG", - "8gb", - "8g", - "8GB", - "4096k", - "1048576b", - "3P", - "2E", - ".5G", - "1.G", - "Infinity", - "17000000T", - "16777216T", - "16777215.5T", - "99999999999999999999T", - "18446744073709551615", - "18446744073709551616", - "1.123456789012345678901234567890G", - "8G\nDevicePolicy=auto", - ] { - assert!(memory_max(bad).is_err(), "{bad:?} was accepted"); - } - } - - /// One hive, `pr1ma`, on `client`. - fn state(client: &async_nats::Client) -> crate::AppState { - crate::AppState { - hives: Arc::new(vec![crate::HiveEntry { - name: "pr1ma".to_owned(), - domain: "pr1ma.example".to_owned(), - }]), - links: Arc::new(Vec::new()), - status: Some(Arc::new(crate::status::StatusReader::new( - client.clone(), - std::time::Duration::from_mins(1), - ))), - wanted: Some(Arc::new(WantedWriter::new(client.clone()))), - agent_status: None, - agent_icons: None, - reported_inference: None, - jobq: Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new( - hive_jobq::Graph::new(), - hive_jobq::resources::ResourceTable::new(), - ))), - webhook_secret: None, - config_prs: Arc::default(), - swarm_name: None, - auth: None, - forge: Arc::default(), - create_gate: Arc::default(), - accounts: crate::linked_accounts::Accounts::Fake(Arc::default()), - try_outs: crate::review_classifier::TryOuts::default(), - } - } - - /// A state on a real server whose `pr1ma` declares `atlas`. - async fn declared(client: &async_nats::Client) -> crate::AppState { - WantedWriter::new(client.clone()) - .set("pr1ma", "atlas", AgentState::Up) - .await - .expect("declare atlas"); - state(client) - } - - fn caps(cpu: Option<&str>, mem: Option<&str>) -> Caps { - Caps { - cpu_quota: cpu.map(str::to_owned), - memory_max: mem.map(str::to_owned), - } - } - - async fn put( - state: &crate::AppState, - agent: &str, - value: Caps, - ) -> Result, problem_details::ProblemDetails> { - put_agent_limits( - State(state.clone()), - Path(agent.to_owned()), - axum::Json(value), - ) - .await - } - - async fn get(state: &crate::AppState, agent: &str) -> Caps { - get_agent_limits(State(state.clone()), Path(agent.to_owned())) - .await - .expect("caps are read") - .0 - } - - /// The raw value a hive reads for `agent`, or `None` for no key or a - /// deleted one. - async fn published(client: &async_nats::Client, agent: &str) -> Option { - let store = swarm_queue_client::agent_limits::open_or_create(client) - .await - .expect("bucket"); - store - .get(agent) - .await - .expect("read") - .map(|v| String::from_utf8(v.to_vec()).expect("utf-8")) - } - - /// Through the real handlers and a real `nats-server`: a cap lands under - /// the agent's own key as the hive reads it, `GET` answers it, and - /// `DELETE` takes it out again. - /// - /// The published bytes are pinned literally because hive-c0re's - /// `workers::swarm_limits` tests decode this same string; a change to - /// either side's reading of it fails one of the two. - #[tokio::test] - async fn the_route_writes_the_caps_a_hive_reads_and_delete_clears_them() { - let nats = TestNats::start(); - let client = nats.client().await; - let state = declared(&client).await; - - let answer = put(&state, "atlas", caps(Some("400%"), Some("8G"))) - .await - .expect("caps are set"); - assert_eq!(answer.0, caps(Some("400%"), Some("8G"))); - assert_eq!( - published(&client, "atlas").await.as_deref(), - Some(r#"{"cpu_quota":"400%","memory_max":"8G"}"#) - ); - assert_eq!(get(&state, "atlas").await, caps(Some("400%"), Some("8G"))); - - assert_eq!( - delete_agent_limits(State(state.clone()), Path("atlas".to_owned())).await, - Ok(StatusCode::NO_CONTENT) - ); - assert_eq!(published(&client, "atlas").await, None); - assert_eq!(get(&state, "atlas").await, Caps::default()); - } - - /// A read before any cap was set answers `{}` and leaves the bucket - /// uncreated. - #[tokio::test] - async fn a_read_before_any_cap_answers_empty_and_creates_no_bucket() { - let nats = TestNats::start(); - let client = nats.client().await; - let state = state(&client); - - assert_eq!(get(&state, "atlas").await, Caps::default()); - assert!( - swarm_queue_client::agent_limits::open(&client) - .await - .expect("the queue answers") - .is_none() - ); - } - - /// A name no hive declares gets no key, so a typo cannot cap an agent - /// that does not exist yet. - #[tokio::test] - async fn an_agent_no_hive_declares_is_refused_and_nothing_is_written() { - let nats = TestNats::start(); - let client = nats.client().await; - let state = declared(&client).await; - - let problem = put(&state, "iris", caps(Some("400%"), None)) - .await - .expect_err("no hive declares iris"); - assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}"); - assert_eq!(published(&client, "iris").await, None); - } - - /// A refused cap is answered before the queue is asked: the client here - /// was never connected, and would answer 503. - #[tokio::test] - async fn a_refused_cap_is_a_400_before_the_queue() { - let client = async_nats::ConnectOptions::new() - .retry_on_initial_connect() - .connect("127.0.0.1:1") - .await - .expect("returns without a connection"); - let state = state(&client); - - let problem = put(&state, "atlas", caps(Some("0%"), None)) - .await - .expect_err("a zero cap is refused"); - assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}"); - - // The control: a valid cap on the same state does reach the queue. - let problem = put(&state, "atlas", caps(Some("400%"), None)) - .await - .expect_err("the queue is not connected"); - assert_eq!( - problem.status, - Some(StatusCode::SERVICE_UNAVAILABLE), - "{problem:?}" - ); - } - - const CAPPED: &str = r#"{"cpu_quota":"400%","memory_max":"8G"}"#; - - /// `atlas` capped, behind the real router. - async fn capped_router(client: &async_nats::Client) -> axum::Router { - let state = declared(client).await; - let _ = put(&state, "atlas", caps(Some("400%"), Some("8G"))) - .await - .expect("cap atlas"); - crate::build_app(state) - } - - async fn put_raw(app: &axum::Router, body: &'static str) -> StatusCode { - use tower::ServiceExt as _; - app.clone() - .oneshot( - axum::http::Request::put("/api/agents/atlas/limits") - .header(axum::http::header::CONTENT_TYPE, "application/json") - .body(axum::body::Body::from(body)) - .expect("request"), - ) - .await - .expect("router answers") - .status() - } - - /// A body that is not JSON is refused before the handler runs, and the - /// caps already set stay. The well-formed `{}` after it is the control: - /// the same router and request do reach the bucket. - #[tokio::test] - async fn a_malformed_body_is_a_400_and_the_caps_stay() { - let nats = TestNats::start(); - let client = nats.client().await; - let app = capped_router(&client).await; - - for body in [r#"{"cpu_quota":"#, "cpu_quota=400%", ""] { - assert_eq!( - put_raw(&app, body).await, - StatusCode::BAD_REQUEST, - "{body:?}" - ); - assert_eq!( - published(&client, "atlas").await.as_deref(), - Some(CAPPED), - "{body:?}" - ); - } - - assert_eq!(put_raw(&app, "{}").await, StatusCode::OK); - assert_eq!(published(&client, "atlas").await.as_deref(), Some("{}")); - } - - /// A value carrying a directive on a second line is a 400 through the real - /// router, and the caps already set stay. - #[tokio::test] - async fn a_cap_with_a_second_line_is_a_400_and_the_caps_stay() { - let nats = TestNats::start(); - let client = nats.client().await; - let app = capped_router(&client).await; - - for body in [ - r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#, - r#"{"memory_max":"8G\nDevicePolicy=auto"}"#, - ] { - assert_eq!(put_raw(&app, body).await, StatusCode::BAD_REQUEST, "{body}"); - assert_eq!( - published(&client, "atlas").await.as_deref(), - Some(CAPPED), - "{body}" - ); - } - } - - /// A field the body does not define is a 422 and writes nothing: a - /// misspelt `memory_max` must not read as "no memory cap". - #[tokio::test] - async fn an_unknown_field_is_a_422_and_the_caps_stay() { - let nats = TestNats::start(); - let client = nats.client().await; - let app = capped_router(&client).await; - - for body in [ - r#"{"cpu_quota":"100%","memory":"2G"}"#, - r#"{"limits":{"cpu_quota":"100%"}}"#, - ] { - assert_eq!( - put_raw(&app, body).await, - StatusCode::UNPROCESSABLE_ENTITY, - "{body}" - ); - assert_eq!( - published(&client, "atlas").await.as_deref(), - Some(CAPPED), - "{body}" - ); - } - } -} diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index cde67d31d..63c3f03e0 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -44,7 +44,6 @@ use utoipa_axum::{router::OpenApiRouter, routes}; mod agent_icon; mod agent_identity; mod agent_input; -mod agent_limits; mod agent_renewal; mod agent_state_stream; mod agent_status; @@ -3301,11 +3300,6 @@ fn build_app(state: AppState) -> axum::Router { inference::put_agent_inference, inference::delete_agent_inference )) - .routes(routes!( - agent_limits::get_agent_limits, - agent_limits::put_agent_limits, - agent_limits::delete_agent_limits - )) .routes(routes!(get_config_prs)) .routes(routes!(create_agent)) .routes(routes!(mint_agent_identity)) diff --git a/swarm-nats-auth/src/policy.rs b/swarm-nats-auth/src/policy.rs index e4a692772..b9d6db695 100644 --- a/swarm-nats-auth/src/policy.rs +++ b/swarm-nats-auth/src/policy.rs @@ -490,17 +490,6 @@ impl Policy { format!("$JS.API.STREAM.CREATE.{agent_status}"), format!("$KV.{}.{hive}.*", swarm_queue_client::agent_status::BUCKET), ]); - // Reading the caps the swarm sets for the agents this hive runs - // (`swarm_queue_client::agent_limits`): a direct get, so read-only, and - // no `STREAM.INFO` because the reader skips the lookup. - // - // Every agent's key, not just this hive's: the bucket is keyed by agent - // alone, and naming one hive's agents here needs the roster - // `hive_name`'s doc argues against. - subjects.push(format!( - "$JS.API.DIRECT.GET.{}.>", - swarm_queue_client::agent_limits::STREAM - )); subjects.extend( self.extra_hive_subjects .iter() @@ -723,51 +712,6 @@ mod tests { } } - /// A hive reads any agent's caps and can change none of them, and the - /// grant reaches no other bucket. - #[test] - fn a_hive_may_read_every_agents_caps_and_write_none() { - use swarm_queue_client::agent_limits::{STREAM, direct_get_subject, key_subject}; - - let p = policy() - .permissions("hive-alpha") - .expect("a hive is admitted"); - let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject)); - - // Its own agents and agents running elsewhere alike. - for agent in ["atlas", "iris"] { - assert!(may(&direct_get_subject(agent)), "cannot read {agent}"); - } - // A KV put or delete is a publish to the key's subject; the rest - // reshape or empty the stream. - for forbidden in [ - key_subject("atlas"), - format!("$JS.API.STREAM.CREATE.{STREAM}"), - format!("$JS.API.STREAM.UPDATE.{STREAM}"), - format!("$JS.API.STREAM.DELETE.{STREAM}"), - format!("$JS.API.STREAM.PURGE.{STREAM}"), - format!("$JS.API.STREAM.MSG.DELETE.{STREAM}"), - format!("$JS.API.CONSUMER.CREATE.{STREAM}"), - // A named consumer, bare and with a key filter, which a grant of - // `CONSUMER.CREATE..>` would cover. - format!("$JS.API.CONSUMER.CREATE.{STREAM}.c1"), - format!( - "$JS.API.CONSUMER.CREATE.{STREAM}.c1.{}", - key_subject("atlas") - ), - ] { - assert!(!may(&forbidden), "hive alpha may publish {forbidden}"); - } - // The read reaches only this bucket. - for other in [ - swarm_queue_client::agent_inference::direct_get_subject("atlas"), - swarm_queue_client::agent_schedules::direct_get_subject("atlas"), - "$JS.API.DIRECT.GET.KV_hive-wanted-beta.$KV.hive-wanted-beta.beta".to_owned(), - ] { - assert!(!may(&other), "hive alpha may publish {other}"); - } - } - #[test] fn an_extra_subject_without_the_placeholder_is_refused() { // 🩸 The option exists to put a second stream inside ONE hive's diff --git a/swarm-queue-client/Cargo.toml b/swarm-queue-client/Cargo.toml index 5496599d2..1123f9a35 100644 --- a/swarm-queue-client/Cargo.toml +++ b/swarm-queue-client/Cargo.toml @@ -47,9 +47,6 @@ agent-inference = ["async-nats/jetstream"] # `ToSchema` on the review classifier's shared types, for swarm-controller, # whose HTTP API serves the same shapes the queue carries. schema = ["dep:utoipa"] -# `agent_limits::read`, for hive-c0re, which reads the key of each agent it -# runs the same way. Explicit for the same reason as `notices`. -agent-limits = ["async-nats/jetstream"] [dependencies] # Bare (no `kv`/`jetstream`) unless a consumer opts into the `kv` feature diff --git a/swarm-queue-client/src/agent_limits.rs b/swarm-queue-client/src/agent_limits.rs deleted file mode 100644 index 77cf3b7f0..000000000 --- a/swarm-queue-client/src/agent_limits.rs +++ /dev/null @@ -1,190 +0,0 @@ -//! The per-agent resource-cap KV bucket: one key per agent, holding the CPU -//! and memory caps the swarm sets for it. -//! -//! **swarm-controller is the only writer**, and creates the bucket on its -//! first write. Every field is optional: an absent field leaves the agent on -//! its hive's own answer, so an agent with no key runs exactly as it would -//! with no swarm. -//! -//! **The hive running the agent reads the key**, not the agent: the caps go -//! into the container's systemd drop-in, which the host writes. A hive reads -//! with one `DIRECT.GET` per agent -//! ([`crate::agent_limits::direct_get_subject`]), and its grant covers every -//! key of this bucket, read-only. Scoping it to the hive's own agents would -//! need a roster in the auth responder. -//! -//! Keyed by the agent alone, like [`crate::agent_inference`]: an agent can -//! move between hives, and its caps go with it. - -/// The KV bucket caps are published into, keyed by agent name. -pub const BUCKET: &str = "agent-limits"; - -/// The stream backing [`BUCKET`]. `create_key_value` prefixes `KV_`. -pub const STREAM: &str = "KV_agent-limits"; - -/// The subject `agent`'s entry is stored under. -#[must_use] -pub fn key_subject(agent: &str) -> String { - format!("$KV.{BUCKET}.{agent}") -} - -/// The request subject that reads `agent`'s current entry. -#[must_use] -pub fn direct_get_subject(agent: &str) -> String { - format!("$JS.API.DIRECT.GET.{STREAM}.{}", key_subject(agent)) -} - -/// One agent's caps: the value under `BUCKET`/``, as systemd values -/// carried verbatim. `None` is "not capped by the swarm", never "uncapped". -#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -pub struct AgentCaps { - /// systemd `CPUQuota=`, e.g. `"400%"`. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub cpu_quota: Option, - /// systemd `MemoryMax=`, e.g. `"8G"`. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub memory_max: Option, -} - -impl AgentCaps { - /// True when no field is set, which a hive reads the same as no key. - #[must_use] - pub fn is_empty(&self) -> bool { - self.cpu_quota.is_none() && self.memory_max.is_none() - } -} - -/// Open the caps bucket for writing, creating it if nothing has yet. -/// -/// **Controller-side only.** `history: 1`: a hive wants the current value. -#[cfg(feature = "kv")] -pub async fn open_or_create( - client: &async_nats::Client, -) -> Result { - let js = async_nats::jetstream::new(client.clone()); - crate::kv::open_or_create( - &js, - BUCKET, - "agent-limits", - "Resource caps the swarm controller sets for each agent".to_owned(), - ) - .await -} - -/// Open the caps bucket for reading, or `None` when nothing has created it -/// yet, which reads the same as no caps. -/// -/// **Controller-side**, for reads that must not create the bucket. -/// -/// # Errors -/// -/// [`crate::Error::OpenBucket`] for every failure other than the bucket not -/// existing. -#[cfg(feature = "kv")] -pub async fn open( - client: &async_nats::Client, -) -> Result, crate::Error> { - let js = async_nats::jetstream::new(client.clone()); - match js.get_key_value(BUCKET).await { - Ok(store) => Ok(Some(store)), - Err(e) if crate::wanted::is_bucket_not_found(&e) => Ok(None), - Err(source) => Err(crate::Error::OpenBucket { - bucket: BUCKET.to_owned(), - source, - }), - } -} - -/// What reading one agent's key found. -#[cfg(feature = "agent-limits")] -#[derive(Debug)] -pub enum Read { - /// The bucket answered; an absent or deleted key has every field unset. - Current(AgentCaps), - /// Nothing answers for the bucket's stream: no cap has been written yet, - /// or `JetStream` is not serving it right now. Not an unset value, since - /// the second case says nothing about the agent's caps. - NoBucket, -} - -/// Read `agent`'s current caps with one direct get. -/// -/// # Errors -/// -/// A failed request or an undecodable value. A request the queue refused for -/// permissions reaches the client as a timeout, not as a refusal. -#[cfg(feature = "agent-limits")] -pub async fn read( - client: &async_nats::Client, - agent: &str, -) -> Result> { - use async_nats::jetstream::stream::DirectGetErrorKind; - - let js = async_nats::jetstream::new(client.clone()); - let stream = js.get_stream_no_info(STREAM).await?; - let message = match stream.direct_get_last_for_subject(key_subject(agent)).await { - Ok(message) => message, - Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => { - return Ok(Read::Current(AgentCaps::default())); - } - Err(e) if is_no_responders(&e) => return Ok(Read::NoBucket), - Err(e) => return Err(e.into()), - }; - // A KV delete or purge is stored as an empty marker carrying this header. - let removed = message - .headers - .get("KV-Operation") - .is_some_and(|op| matches!(op.as_str(), "DEL" | "PURGE")); - if removed { - return Ok(Read::Current(AgentCaps::default())); - } - Ok(Read::Current(serde_json::from_slice(&message.payload)?)) -} - -#[cfg(feature = "agent-limits")] -fn is_no_responders(e: &async_nats::jetstream::stream::DirectGetError) -> bool { - matches!( - e.kind(), - async_nats::jetstream::stream::DirectGetErrorKind::ErrorResponse(status, _) - if status == async_nats::StatusCode::NO_RESPONDERS - ) -} - -#[cfg(test)] -mod tests { - use super::*; - - /// The names the hive's grant in `swarm-nats-auth` is built from. - #[test] - fn names_match_the_queue_grant() { - assert_eq!(STREAM, format!("KV_{BUCKET}")); - assert_eq!(key_subject("iris"), "$KV.agent-limits.iris"); - assert_eq!( - direct_get_subject("iris"), - "$JS.API.DIRECT.GET.KV_agent-limits.$KV.agent-limits.iris" - ); - } - - /// An unset field is left out on the wire, and `{}` is every field unset. - #[test] - fn unset_fields_are_absent() { - let value = AgentCaps { - memory_max: Some("8G".to_owned()), - ..AgentCaps::default() - }; - assert_eq!( - serde_json::to_string(&value).unwrap(), - r#"{"memory_max":"8G"}"# - ); - let empty: AgentCaps = serde_json::from_str("{}").unwrap(); - assert!(empty.is_empty()); - } - - /// Fields a newer controller adds must not cost an older hive its caps. - #[test] - fn decoding_tolerates_unknown_fields() { - let decoded: AgentCaps = - serde_json::from_str(r#"{"cpu_quota":"400%","io_weight":"10"}"#).unwrap(); - assert_eq!(decoded.cpu_quota.as_deref(), Some("400%")); - } -} diff --git a/swarm-queue-client/src/lib.rs b/swarm-queue-client/src/lib.rs index 6cd0da3a4..fa5b991b6 100644 --- a/swarm-queue-client/src/lib.rs +++ b/swarm-queue-client/src/lib.rs @@ -216,11 +216,6 @@ pub mod agent_schedules; /// unconditional; writing needs `kv` and reading needs `agent-inference`. pub mod agent_inference; -/// The bucket swarm-controller sets each agent's CPU and memory caps in — one -/// key per agent, read by the hive that runs it. The names are unconditional; -/// writing needs `kv` and reading needs `agent-limits`. -pub mod agent_limits; - #[cfg(feature = "kv")] mod kv; diff --git a/swarm-queue-client/src/wanted.rs b/swarm-queue-client/src/wanted.rs index 16681441b..451039c5d 100644 --- a/swarm-queue-client/src/wanted.rs +++ b/swarm-queue-client/src/wanted.rs @@ -156,7 +156,7 @@ pub async fn open_read_only( /// `get_key_value` reports a missing bucket as `GetBucket` wrapping the /// stream lookup's `STREAM_NOT_FOUND`; every other kind is a real failure. #[cfg(feature = "kv")] -pub(crate) fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool { +fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool { use async_nats::jetstream::context::{GetStreamError, GetStreamErrorKind, KeyValueErrorKind}; e.kind() == KeyValueErrorKind::GetBucket