diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index 3ca1043c9..851f7e73c 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -411,6 +411,13 @@ Contents: default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same drop-in have **no** per-agent override — they're hive-wide only and come straight off `HiveEnv`, so this file has no field for them. +- `swarm-resource-limits.json` — the caps the swarm sets for this + hive's agents, same shape as `resource-limits.json`. Written only by + `workers::swarm_limits`, which copies each agent's key of the swarm's + `agent-limits` bucket here every 30 seconds. A field set here replaces + that field's answer from `resource-limits.json` and the hive-wide + default. Every value passes the same validators as the operator's + overrides first, and one bad value makes the worker change nothing here. The root agent has the meta dir RO-mounted at `/meta/`. diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index 20d4dab39..043c34b9e 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -452,14 +452,24 @@ regardless. rebuild, so changes take effect on the next lifecycle op without requiring a host rebuild. +The answer per agent and per field is, first match wins: + +1. the swarm's cap for the agent (see below); +2. the operator's per-agent override in `meta/resource-limits.json`; +3. `agentCpuQuota` / `agentMemoryMax`. + +A swarm cap replaces the hive's answer, so a hive can't raise an agent +above it. Like the rest of the drop-in, a changed swarm cap reaches a +running agent at its next spawn or rebuild. + The container can't read its own cap, so `write_dropins` also writes the effective `MemoryMax=` into the agent's runtime dir (`/run/hive/memory-max-bytes` inside) before every start: a byte count, or `infinity` for no cap. A percentage becomes bytes the way systemd resolves it, as that share of the host's physical memory rounded down to a page. A value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps -the last drop-in because it can't read the overrides file, the file carries -that drop-in's `MemoryMax=`. +the last drop-in because it can't read the overrides or the swarm caps file, +the file carries that drop-in's `MemoryMax=`. At boot the container derives two values from a byte count: `BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on @@ -469,6 +479,39 @@ changed cap reaches them at the next container start without a rebuild. `hyperhive-memory-cap-check.service` logs a warning naming the path in the container's journal. The container's `MemoryMax=` still applies. +
Implementation: swarm caps + +swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per +agent and no hive in it, so an agent that moves to another hive keeps its +caps. swarm-controller alone writes it: + +| method | path | body | +| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- | +| `GET` | `/api/agents//limits` | —; answers the stored caps, `{}` when there are none | +| `PUT` | `/api/agents//limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value | +| `DELETE` | `/api/agents//limits` | —; answers 204 | + +`PUT` refuses an agent no hive's wanted state declares, unknown fields, and +any value systemd would ignore, which would leave the agent uncapped. A +percentage has at most two decimals, no leading zero in its whole part (it +refuses `08%`), and is above zero, up to `100%` for +`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an +optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at +least one byte and below 2^64 - 1 bytes. The controller stores each value as +given. + +Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each +agent it has and stores the result in `meta/swarm-resource-limits.json`. Each +value passes the same validators as the operator's overrides first. One bad +value or a failed read leaves the file unchanged, and repeated failures raise a +dashboard banner. + +A hive's queue grant reads every key of the bucket +(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the +hive's own agents would need a roster in the auth responder. + +
+ The same drop-in carries `CPUWeight=` / `IOWeight=` from `agentCpuWeight` / `agentIoWeight`. Those are a different kind of setting: the quota and the memory max are **hard caps** that throttle diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 8c89734ce..0535bd71c 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -25,6 +25,9 @@ grafana, the controller, authelia, …) → kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md) - **inference settings** — the model and effort the swarm sets for an agent, over its own. → [`inference.md`](inference.md) +- **resource caps** — the CPU and memory cap the swarm sets for an agent, + over its hive's. → + [`coordinator.md`](../scheduler/coordinator.md#container-resource-limits) - **review classifier** — planned, not implemented yet: checks a PR against its repo's own yes/no rules and posts the result as a review. → [`review-classifier.md`](review-classifier.md) diff --git a/hive-c0re/Cargo.toml b/hive-c0re/Cargo.toml index ac74773ff..9e8d0f0c0 100644 --- a/hive-c0re/Cargo.toml +++ b/hive-c0re/Cargo.toml @@ -53,7 +53,11 @@ strum.workspace = true # Offering this hive's status to the swarm (`swarm_status`). The same crate # the swarm controller reads it with, and `kv` for the same reason: the # bucket's name and creation config belong to neither end of it alone. -swarm-queue-client = { workspace = true, features = ["kv", "notices"] } +swarm-queue-client = { workspace = true, features = [ + "kv", + "notices", + "agent-limits", +] } swarm-secret-client.workspace = true tokio.workspace = true tokio-stream.workspace = true diff --git a/hive-c0re/src/agent_config/resource_limits.rs b/hive-c0re/src/agent_config/resource_limits.rs index fd68b78ae..880c87da0 100644 --- a/hive-c0re/src/agent_config/resource_limits.rs +++ b/hive-c0re/src/agent_config/resource_limits.rs @@ -187,6 +187,86 @@ fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Resu super::write_map(path, ¤t) } +/// Caps the swarm controller declared for this hive's agents. +/// +/// A separate file from the operator's overrides above, not a merge into it: +/// the two have different writers and different lifetimes, and a swarm +/// declaration landing in the file an operator edits would make "reset to +/// hive defaults" destroy something the operator never set. +const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json"; + +#[must_use] +pub fn swarm_limits_path() -> PathBuf { + crate::paths::meta_root().join(SWARM_LIMITS_FILE) +} + +/// Read the swarm-declared caps. An absent file is the empty map — no swarm +/// opinion, so the hive's own chain decides. A file that exists but can't be +/// read or parsed is an error, for the reason [`read`] gives: reading it as +/// empty would silently drop a cap. +pub fn read_swarm() -> std::io::Result> { + super::read_map(&swarm_limits_path()) +} + +/// Persist the swarm-declared caps atomically. +/// +/// # Errors +/// The meta dir cannot be created, or the file cannot be written. +pub fn write_swarm(map: &BTreeMap) -> std::io::Result<()> { + super::write_map(&swarm_limits_path(), map) +} + +/// [`effective_from`] with the swarm's declaration on top. +/// +/// A swarm cap **replaces** the hive's answer for that field rather than +/// joining the fallback chain as another tier: a tier a hive can out-set is +/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's +/// chain. +/// +/// The chain itself is untouched — [`effective_from`] is called exactly as +/// before and its result is what a missing swarm value falls back to. +#[must_use] +pub fn effective_from_with_swarm( + swarm: &BTreeMap, + limits: &BTreeMap, + name: &str, + hive_cpu_quota: &str, + hive_memory_max: &str, +) -> (String, String) { + let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max); + match swarm.get(name) { + Some(s) => ( + s.cpu_quota.clone().unwrap_or(cpu), + s.memory_max.clone().unwrap_or(mem), + ), + None => (cpu, mem), + } +} + +/// [`effective_from_with_swarm`] for a caller resolving a single agent, from +/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`. +/// +/// Delegates to [`effective`] rather than re-deriving the hive's answer, so +/// there stays exactly one place the fallback chain lives. Errors as [`read`] +/// does, for either file. +pub fn effective_with_swarm( + limits_path: &Path, + swarm_path: &Path, + name: &str, + hive_cpu_quota: &str, + hive_memory_max: &str, +) -> std::io::Result<(String, String)> { + let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?; + let swarm: BTreeMap = super::read_map(swarm_path)?; + Ok(match swarm.get(name) { + Some(s) => ( + s.cpu_quota.clone().unwrap_or(cpu), + s.memory_max.clone().unwrap_or(mem), + ), + None => (cpu, mem), + }) +} + /// Validate a systemd `CPUQuota=` value. Percentages only, and values /// above 100% are legitimate (one full core is 100%). /// @@ -525,4 +605,52 @@ mod tests { ); } } + + #[test] + fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() { + // The whole point of the ruling: a tier a hive can out-set is not a + // cap. The agent override below is the strongest hive-side value + // there is, and the swarm still wins. + let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]); + let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]); + assert_eq!( + effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM), + ("150%".to_owned(), "2G".to_owned()) + ); + } + + #[test] + fn a_swarm_cap_is_per_field_like_every_other_tier() { + // A swarm that caps memory and says nothing about CPU leaves CPU on + // the hive's chain — here the per-agent override, not the hive default. + let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]); + let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]); + assert_eq!( + effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM), + ("800%".to_owned(), "2G".to_owned()) + ); + } + + #[test] + fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() { + // The control that makes the two tests above mean something: the + // wrapper must be a no-op when the swarm has said nothing, for every + // shape the old chain distinguishes. If this drifts, the swarm tier + // changed behaviour it was explicitly told not to change. + let none = BTreeMap::new(); + let cases = [ + BTreeMap::new(), + BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]), + BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]), + BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]), + BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]), + ]; + for hive_side in &cases { + assert_eq!( + effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM), + effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM), + "the wrapper changed the answer with no swarm value present" + ); + } + } } diff --git a/hive-c0re/src/container_view.rs b/hive-c0re/src/container_view.rs index 012a9451f..d55686cc8 100644 --- a/hive-c0re/src/container_view.rs +++ b/hive-c0re/src/container_view.rs @@ -171,6 +171,14 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result anyhow::Result Re if let Some((cpu_quota, memory_max)) = stage_limits( name, &crate::resource_limits::resource_limits_path(), + &crate::resource_limits::swarm_limits_path(), &limits_dropin_path(&container), &paths.agent, hive, @@ -97,12 +98,13 @@ impl HostMemory { fn stage_limits( agent_name: &str, limits_path: &Path, + swarm_path: &Path, dropin_path: &Path, runtime_dir: &Path, hive: &HiveEnv, host: Option, ) -> Result> { - let limits = limits_to_apply(agent_name, limits_path, dropin_path, hive); + let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive); let memory_max = match &limits { Some((_, memory_max)) => Some(memory_max.clone()), None => dropin_memory_max(agent_name, dropin_path), @@ -224,20 +226,28 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> { /// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits /// drop-in, or `None` to leave the drop-in at `dropin_path` as it is. /// -/// When the overrides file at `limits_path` can't be read, the error is -/// logged and the agent still starts. The last-applied drop-in is kept, -/// because the hive defaults can be looser than the agent's own override. +/// When the overrides file at `limits_path` or the swarm caps at +/// `swarm_path` can't be read, the error is logged and the agent still starts. +/// The last-applied drop-in is kept, because the hive defaults can be looser +/// than the agent's own override. /// With no drop-in yet, the agent gets the hive defaults: without one it /// would run uncapped. fn limits_to_apply( agent_name: &str, limits_path: &Path, + swarm_path: &Path, dropin_path: &Path, hive: &HiveEnv, ) -> Option<(String, String)> { let hive_cpu = &hive.agent_cpu_quota; let hive_mem = &hive.agent_memory_max; - let e = match crate::resource_limits::effective(limits_path, agent_name, hive_cpu, hive_mem) { + let e = match crate::resource_limits::effective_with_swarm( + limits_path, + swarm_path, + agent_name, + hive_cpu, + hive_mem, + ) { Ok(limits) => return Some(limits), Err(e) => e, }; @@ -838,6 +848,10 @@ mod tests { const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem"; + fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf { + dir.path().join("absent-swarm-resource-limits.json") + } + #[test] fn corrupt_limits_keep_an_existing_dropin() { let dir = tempfile::tempdir().expect("tempdir"); @@ -846,7 +860,13 @@ mod tests { std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits"); std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); assert_eq!( - limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()), + limits_to_apply( + "sock", + &limits, + &absent_swarm(&dir), + &dropin, + &HiveEnv::default() + ), None ); assert_eq!( @@ -863,12 +883,47 @@ mod tests { let applied = limits_to_apply( "sock", &limits, + &absent_swarm(&dir), &dir.path().join("absent.conf"), &HiveEnv::default(), ); assert_eq!(applied, Some(("200%".to_owned(), "4G".to_owned()))); } + #[test] + fn corrupt_swarm_caps_keep_an_existing_dropin() { + let dir = tempfile::tempdir().expect("tempdir"); + let swarm = dir.path().join("swarm-resource-limits.json"); + let dropin = dir.path().join("hyperhive-limits.conf"); + std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps"); + std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in"); + let applied = limits_to_apply( + "sock", + &dir.path().join("resource-limits.json"), + &swarm, + &dropin, + &HiveEnv::default(), + ); + assert_eq!(applied, None); + } + + #[test] + fn a_swarm_cap_is_applied_over_the_operator_override() { + let dir = tempfile::tempdir().expect("tempdir"); + let limits = dir.path().join("resource-limits.json"); + let swarm = dir.path().join("swarm-resource-limits.json"); + std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits"); + std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps"); + let applied = limits_to_apply( + "sock", + &limits, + &swarm, + &dir.path().join("absent.conf"), + &HiveEnv::default(), + ); + assert_eq!(applied, Some(("200%".to_owned(), "2G".to_owned()))); + } + #[test] fn readable_limits_are_applied_over_an_existing_dropin() { let dir = tempfile::tempdir().expect("tempdir"); @@ -876,7 +931,13 @@ mod tests { let dropin = dir.path().join("hyperhive-limits.conf"); std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits"); std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in"); - let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()); + let applied = limits_to_apply( + "sock", + &limits, + &absent_swarm(&dir), + &dropin, + &HiveEnv::default(), + ); assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned()))); } @@ -893,6 +954,7 @@ mod tests { let applied = stage_limits( "sock", &dir.path().join("resource-limits.json"), + &dir.path().join("swarm-resource-limits.json"), &dir.path().join("hyperhive-limits.conf"), &runtime, &HiveEnv::default(), @@ -903,6 +965,28 @@ mod tests { (applied.map(|(_, mem)| mem), copy) } + /// The agent's copy follows the swarm's cap, the one systemd enforces, over + /// the operator's override: a heap sized against the override would + /// outgrow the cgroup. + #[test] + fn the_agent_reads_the_swarm_cap_over_the_operator_override() { + let dir = tempfile::tempdir().expect("tempdir"); + std::fs::write( + dir.path().join("resource-limits.json"), + r#"{"sock": {"memory_max": "16G"}}"#, + ) + .expect("seed limits"); + std::fs::write( + dir.path().join("swarm-resource-limits.json"), + r#"{"sock": {"memory_max": "2G"}}"#, + ) + .expect("seed swarm caps"); + assert_eq!( + stage(&dir), + (Some("2G".to_owned()), Some("2147483648\n".to_owned())) + ); + } + #[test] fn the_agent_reads_the_cap_its_dropin_carries() { let dir = tempfile::tempdir().expect("tempdir"); @@ -1035,6 +1119,7 @@ mod tests { let applied = stage_limits( "sock", &limits, + &dir.path().join("swarm-resource-limits.json"), &dropin, &runtime, &HiveEnv::default(), diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 5d9839bc3..ea283fda3 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -426,8 +426,9 @@ async fn handle_set_resource_limits( let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?; crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?; - let (cpu, mem) = crate::resource_limits::effective( + let (cpu, mem) = crate::resource_limits::effective_with_swarm( &crate::resource_limits::resource_limits_path(), + &crate::resource_limits::swarm_limits_path(), name.as_str(), &hive.agent_cpu_quota, &hive.agent_memory_max, diff --git a/hive-c0re/src/swarm_status.rs b/hive-c0re/src/swarm_status.rs index 13a1d455c..c4cfc54b6 100644 --- a/hive-c0re/src/swarm_status.rs +++ b/hive-c0re/src/swarm_status.rs @@ -103,6 +103,12 @@ pub fn spawn( shutdown.clone(), )); + // The swarm's resource caps, on the same connection again. + tokio::spawn(crate::workers::swarm_limits::follow( + client.clone(), + shutdown.clone(), + )); + let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER); loop { match publish(&client, &hive).await { diff --git a/hive-c0re/src/workers/mod.rs b/hive-c0re/src/workers/mod.rs index c3772e923..6f2ffa4c8 100644 --- a/hive-c0re/src/workers/mod.rs +++ b/hive-c0re/src/workers/mod.rs @@ -1,6 +1,7 @@ //! Background tasks and periodic sweeps: crash/login watcher, boot-time //! auto-update reconcile, the agent-sockets.json writer loop, the MCP socket listener -//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull. +//! reconcile loop, knowledge-repo sync, the swarm wanted-state pull, and the +//! swarm resource-cap read. //! Each submodule is re-exported //! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged. @@ -9,4 +10,5 @@ pub mod auto_update; pub mod crash_watch; pub mod knowledge; pub mod mcp_sockets; +pub mod swarm_limits; pub mod wanted; diff --git a/hive-c0re/src/workers/swarm_limits.rs b/hive-c0re/src/workers/swarm_limits.rs new file mode 100644 index 000000000..ec3b09790 --- /dev/null +++ b/hive-c0re/src/workers/swarm_limits.rs @@ -0,0 +1,207 @@ +//! Reading the caps the swarm sets for this hive's agents. +//! +//! swarm-controller keeps one key per agent in the `agent-limits` bucket +//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads +//! the key of each agent this hive has and stores what it finds in +//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it. +//! A running agent gets a changed cap at its first spawn or rebuild after +//! that read. +//! +//! A failed read, or a bucket that does not answer, leaves the stored caps as +//! they are: dropping them would read exactly like the swarm clearing them. + +use std::collections::BTreeMap; +use std::time::Duration; + +use anyhow::{Context, Result}; +use swarm_queue_client::agent_limits::{self, AgentCaps, Read}; + +use crate::resource_limits::AgentLimits; +use crate::stats::sweep_health::{self, SweepHealth}; + +/// How often the caps are read. +pub const READ_INTERVAL: Duration = Duration::from_secs(30); + +/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads. +const FAILURES_BEFORE_BANNER: u32 = 3; + +/// Read the caps now and every [`READ_INTERVAL`] until shutdown. +pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver) { + let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER); + loop { + match sync(&client).await { + Ok(()) => health.record_ok(), + Err(e) => { + tracing::warn!(error = ?e, "swarm limits: reading the caps failed"); + let err = format!("{e:#}"); + health.record_err(|ctx| { + let age = ctx.since_last_ok.map_or_else( + || "no success this session".to_owned(), + |d| format!("last ok {} ago", sweep_health::fmt_age(d)), + ); + format!( + "reading the swarm's resource caps is failing ({} consecutive, {age}) \ + — agents keep the caps last read: {err}", + ctx.consecutive + ) + }); + } + } + tokio::select! { + () = tokio::time::sleep(READ_INTERVAL) => {} + _ = shutdown.changed() => { + tracing::info!("swarm limits: shutdown signal received"); + return; + } + } + } +} + +/// Read every agent's key and store the result, unless a read fails or a +/// value is refused. +async fn sync(client: &async_nats::Client) -> Result<()> { + // An unconnected client does not fail a JetStream request, it hangs on it. + swarm_queue_client::ensure_connected(client)?; + let agents = crate::lifecycle::agents_for_meta_listing() + .await + .context("enumerating this hive's agents")?; + let mut declared = BTreeMap::new(); + for spec in agents { + let read = agent_limits::read(client, &spec.name) + .await + .map_err(|e| anyhow::anyhow!(e)) + .with_context(|| format!("reading the swarm's caps for {}", spec.name))?; + match read { + Read::Current(caps) => { + declared.insert(spec.name, caps); + } + Read::NoBucket => { + tracing::debug!("swarm limits: the agent-limits bucket does not answer"); + return Ok(()); + } + } + } + let caps = validated(&declared)?; + if caps != crate::resource_limits::read_swarm().context("reading the stored swarm caps")? { + crate::resource_limits::write_swarm(&caps).context("storing the swarm caps")?; + tracing::info!(?caps, "swarm limits: stored the swarm's resource caps"); + } + Ok(()) +} + +/// The caps to store for `declared`, by agent. An agent with no field set +/// gets no entry. +/// +/// Every value goes through the validators the operator's own input path +/// uses, because it is rendered verbatim into the container's systemd +/// drop-in. One bad value refuses the whole set. +fn validated(declared: &BTreeMap) -> Result> { + let mut out = BTreeMap::new(); + for (agent, caps) in declared { + if caps.is_empty() { + continue; + } + let limits = validated_caps(caps) + .map_err(|why| anyhow::anyhow!("the swarm set an invalid cap for {agent}: {why}"))?; + out.insert(agent.clone(), limits); + } + Ok(out) +} + +fn validated_caps(caps: &AgentCaps) -> Result { + if let Some(cpu) = &caps.cpu_quota { + crate::resource_limits::validate_cpu_quota(cpu)?; + } + Ok(AgentLimits { + cpu_quota: caps.cpu_quota.clone(), + memory_max: caps + .memory_max + .as_deref() + .map(crate::resource_limits::validate_memory_max) + .transpose()?, + }) +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use super::validated; + use crate::resource_limits::AgentLimits; + use swarm_queue_client::agent_limits::AgentCaps; + + /// `atlas`'s answer with `atlas_key` as the value read for it, over a hive + /// chain that caps it at 800% / 16G. + fn atlas_with(atlas_key: &str) -> (String, String) { + let declared = BTreeMap::from([( + "atlas".to_owned(), + serde_json::from_str::(atlas_key).expect("decodes"), + )]); + let stored = validated(&declared).expect("valid caps"); + let hive_side = BTreeMap::from([( + "atlas".to_owned(), + AgentLimits { + cpu_quota: Some("800%".to_owned()), + memory_max: Some("16G".to_owned()), + }, + )]); + crate::resource_limits::effective_from_with_swarm( + &stored, &hive_side, "atlas", "200%", "4G", + ) + } + + /// The other half of swarm-controller's + /// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`: + /// the first value is the bytes that route stores. The cap replaces the + /// hive's chain; a deleted key reads as `{}` and gives the chain back. + #[test] + fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() { + assert_eq!( + atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#), + ("400%".to_owned(), "8G".to_owned()) + ); + assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned())); + } + + /// Per field: a swarm memory cap leaves CPU on the hive's chain. + #[test] + fn a_field_the_swarm_leaves_unset_stays_with_the_hive() { + assert_eq!( + atlas_with(r#"{"memory_max":"2G"}"#), + ("800%".to_owned(), "2G".to_owned()) + ); + } + + #[test] + fn an_agent_with_no_caps_gets_no_entry() { + let declared = BTreeMap::from([("atlas".to_owned(), AgentCaps::default())]); + assert!(validated(&declared).expect("valid").is_empty()); + } + + /// The values are rendered verbatim into a systemd drop-in, so a value + /// the hive's own validators refuse, a newline smuggling in a directive + /// among them, refuses the whole set, the valid cap beside it included. + #[test] + fn an_invalid_cap_is_refused_and_nothing_is_stored() { + for bad in [ + r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#, + r#"{"memory_max":"8G\nDevicePolicy=auto"}"#, + r#"{"cpu_quota":"lots"}"#, + r#"{"memory_max":"8X"}"#, + ] { + let declared = BTreeMap::from([ + ("atlas".to_owned(), AgentCaps::default()), + ( + "iris".to_owned(), + serde_json::from_str(r#"{"memory_max":"2G"}"#).expect("decodes"), + ), + ( + "sock".to_owned(), + serde_json::from_str(bad).expect("decodes"), + ), + ]); + let err = validated(&declared).expect_err(bad); + assert!(err.to_string().contains("sock"), "{err:#}"); + } + } +} diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index f99c1a11d..48326b2f4 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -286,6 +286,22 @@ let && !(lib.hasInfix "mint swarmctl" s) && lib.hasInfix "mint swarm-authelia-bridge" s; } + { + # swarm-controller checks no caller itself: this location's authelia + # subrequest is the operator gate for every route under `/api/`. The + # webhook location is the control, the one path to the same socket that + # skips the gate, so the first clause reads a string that can lack it. + name = "the swarm UI's API location asks authelia before reaching the controller"; + ok = + let + l = allLocal.services.nginx.virtualHosts."t.local".locations; + socket = allLocal.services.hyperhive.deploy.swarm-controller.socketPath; + in + lib.hasInfix "auth_request /__hive_authelia;" l."/api/".extraConfig + && lib.hasInfix socket l."/api/".proxyPass + && lib.hasInfix socket l."/webhook/forge/".proxyPass + && !(lib.hasInfix "auth_request" l."/webhook/forge/".extraConfig); + } { name = "a hive with matrix off serves no matrix discovery endpoint"; ok = diff --git a/swarm-controller/README.md b/swarm-controller/README.md index 036ff2836..4c7e12d4a 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -52,6 +52,11 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients. agent applies them itself. `GET` also answers the model and effort the agent last reported in its turn-state header, kept in memory → [`docs/swarm/inference.md`](../docs/swarm/inference.md). +- **Resource caps** — each agent's CPU and memory cap, one key per agent in + the `agent-limits` KV bucket, which this daemon alone writes + (`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the + agent reads it, and each cap replaces the hive's own answer for that field → + [`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits). - **Relays** — each agent's terminal and turn-state header as SSE, agent icons, the cross-repo issue report, and the UI's quick links. diff --git a/swarm-controller/src/agent_limits.rs b/swarm-controller/src/agent_limits.rs new file mode 100644 index 000000000..90351ab31 --- /dev/null +++ b/swarm-controller/src/agent_limits.rs @@ -0,0 +1,699 @@ +//! `/api/agents/{name}/limits` — the CPU and memory caps the swarm sets for +//! one agent. +//! +//! Each agent's caps are one value under its own key of `agent-limits` +//! (`swarm_queue_client::agent_limits`), which this daemon alone writes. The +//! hive running the agent reads the key; on the hive, a cap replaces the +//! hive's own answer for that field, and a field left unset stays with the +//! hive. +//! +//! The hive hands these values to systemd verbatim, and systemd ignores a value +//! it can't parse, leaving the agent uncapped, so they are checked here against +//! systemd's own syntax (`Caps`) as well as on the hive. + +use axum::Json; +use axum::extract::{Path, State}; +use axum::http::StatusCode; +use problem_details::ProblemDetails; +use serde::{Deserialize, Serialize}; +use swarm_queue_client::agent_limits::{self, AgentCaps}; +use utoipa::ToSchema; + +use crate::schedules::{live, queue, read, refused, write}; +use crate::{AppState, error_problem, wanted_error_status}; + +/// How many times a losing writer re-reads and re-applies, as in +/// `wanted::MAX_ATTEMPTS`. +const MAX_ATTEMPTS: usize = 5; + +/// One agent's caps. A field left out is not capped by the swarm. +/// +/// Each value is stored as given, and only in a form systemd applies: a value +/// systemd ignores ("Invalid …, ignoring") would leave the agent uncapped, +/// since a swarm cap replaces the hive's answer. +#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(deny_unknown_fields)] +pub struct Caps { + /// systemd `CPUQuota=`: digits, optionally `.` and one or two digits, then + /// `%`; above zero and at most `21474836.47%`. `"100%"` is one core. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[schema(example = "400%")] + cpu_quota: Option, + /// systemd `MemoryMax=`: `infinity`; a percentage of physical memory + /// written as for `cpu_quota`, above zero and at most `100%`; or a size: + /// digits, optionally `.` and digits, then nothing, `B` or one of + /// `K`/`M`/`G`/`T` (1024-based, upper case), at least one byte and below + /// 2^64 - 1 bytes. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[schema(example = "8G")] + memory_max: Option, +} + +impl Caps { + /// The value to store, or why it is refused. + fn into_stored(self) -> Result { + Ok(AgentCaps { + cpu_quota: self.cpu_quota.as_deref().map(cpu_quota).transpose()?, + memory_max: self.memory_max.as_deref().map(memory_max).transpose()?, + }) + } +} + +impl From for Caps { + fn from(v: AgentCaps) -> Self { + Self { + cpu_quota: v.cpu_quota, + memory_max: v.memory_max, + } + } +} + +// The accepted forms are a subset of what systemd parses (checked against +// `systemd-analyze verify`, systemd 260) and stricter than hive-c0re's own +// `validate_cpu_quota` / `validate_memory_max`, so the hive never refuses a +// stored value. + +/// systemd keeps `CPUQuota=` in hundredths of a percent in a C `int`. +const CPU_QUOTA_MAX_HUNDREDTHS: u64 = 2_147_483_647; + +fn cpu_quota(value: &str) -> Result { + match hundredths(value) { + Some(h) if (1..=CPU_QUOTA_MAX_HUNDREDTHS).contains(&h) => Ok(value.to_owned()), + _ => Err(format!( + "invalid cpu_quota {value:?}: expected a percentage above zero with at most two \ + decimals, such as \"200%\" (100% = one full core), up to \"21474836.47%\"" + )), + } +} + +fn memory_max(value: &str) -> Result { + let applies = value == "infinity" + || if value.ends_with('%') { + hundredths(value).is_some_and(|h| (1..=10_000).contains(&h)) + } else { + size_bytes(value).is_some_and(|b| b >= 1 && b < u128::from(u64::MAX)) + }; + if applies { + Ok(value.to_owned()) + } else { + Err(format!( + "invalid memory_max {value:?}: expected a size above zero such as \"8G\" \ + (suffix B, K, M, G or T, upper case), a percentage up to \"100%\" with at \ + most two decimals, or \"infinity\"" + )) + } +} + +/// `value` as systemd reads a percentage, in hundredths of a percent: digits, +/// optionally `.` and one or two digits, then `%`. `None` for anything else, +/// and for a whole part with a leading zero other than `0` itself: systemd +/// ignores `08%` and may read `010%` as octal. +fn hundredths(value: &str) -> Option { + let number = value.strip_suffix('%')?; + let (int, frac) = match number.split_once('.') { + Some((int, frac)) if (1..=2).contains(&frac.len()) && digits(frac) => (int, frac), + Some(_) => return None, + None => (number, ""), + }; + if !digits(int) || (int.len() > 1 && int.starts_with('0')) { + return None; + } + let frac = match frac.len() { + 0 => 0, + 1 => frac.parse::().ok()? * 10, + _ => frac.parse::().ok()?, + }; + int.parse::().ok()?.checked_mul(100)?.checked_add(frac) +} + +/// `value` as one term of systemd's `parse_size` reads it, in bytes: digits, +/// optionally `.` and digits, then nothing, `B`, or `K`/`M`/`G`/`T` (1024-based, +/// case-sensitive). `P` and `E`, which systemd also reads, are left out +/// because hive-c0re's validator refuses them. `None` for anything else, and for what +/// `parse_size` refuses as out of range: a whole part or fraction digits past +/// `u64`, or a whole part that, plus one when there is a fraction, is past +/// `u64` once scaled. +fn size_bytes(value: &str) -> Option { + let (number, exponent) = match value.char_indices().last()? { + (i, 'B') => (&value[..i], 0), + (i, c) => match "KMGT".find(c) { + Some(k) => (&value[..i], k + 1), + None => (value, 0), + }, + }; + let scale = 1024u128.pow(u32::try_from(exponent).ok()?); + let (int, frac) = number.split_once('.').unwrap_or((number, "")); + if !digits(int) || (number.contains('.') && !digits(frac)) { + return None; + } + let whole = u128::from(int.parse::().ok()?); + let frac_value = if frac.is_empty() { + 0 + } else { + u128::from(frac.parse::().ok()?) + }; + if whole + u128::from(frac_value > 0) > u128::from(u64::MAX) / scale { + return None; + } + let frac_bytes = frac_value * scale / 10u128.checked_pow(u32::try_from(frac.len()).ok()?)?; + Some(whole * scale + frac_bytes) +} + +fn digits(s: &str) -> bool { + !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()) +} + +fn agent_name(name: &str) -> Result { + hive_types::Ident::parse(name) + .map(hive_types::Ident::into_string) + .map_err(|reason| (StatusCode::BAD_REQUEST, reason.to_owned())) +} + +fn queue_problem(e: &anyhow::Error) -> ProblemDetails { + tracing::warn!(error = %format!("{e:#}"), "agent limits: queue failure"); + error_problem(wanted_error_status(e), &format!("{e:#}")) +} + +async fn open_or_create( + client: &async_nats::Client, +) -> Result { + agent_limits::open_or_create(client) + .await + .map_err(|e| queue_problem(&anyhow::Error::new(e))) +} + +#[utoipa::path( + get, + path = "/api/agents/{name}/limits", + params(("name" = String, Path, description = "agent name")), + responses( + (status = 200, description = "the caps the swarm sets for the agent; `{}` when it \ + sets none", body = Caps), + (status = 400, description = "not an identifier (problem+json)", body = String), + (status = 503, description = "no swarm queue is configured on this host, or it is \ + not connected (problem+json)", body = String), + (status = 500, description = "the caps could not be read (problem+json)", + body = String), + ), + tag = "agents" +)] +pub(crate) async fn get_agent_limits( + State(state): State, + Path(name): Path, +) -> Result, ProblemDetails> { + let agent = agent_name(&name).map_err(refused)?; + let client = queue(&state).map_err(refused)?; + let Some(store) = agent_limits::open(&client) + .await + .map_err(|e| queue_problem(&anyhow::Error::new(e)))? + else { + return Ok(Json(Caps::default())); + }; + let (current, _) = read::(&store, &agent) + .await + .map_err(|e| queue_problem(&e))?; + Ok(Json(current.into())) +} + +/// Set the CPU and memory caps the swarm declares for one agent. +/// +/// The value replaces the stored one whole. Whichever hive runs the agent +/// applies it; a running agent picks the caps up only at its next spawn or +/// rebuild. +#[utoipa::path( + put, + path = "/api/agents/{name}/limits", + params(("name" = String, Path, description = "agent name")), + request_body = Caps, + responses( + (status = 200, description = "the caps as stored", body = Caps), + (status = 400, description = "not an identifier, not declared on any hive, or a cap \ + is not a value systemd takes or is zero (problem+json)", body = String), + (status = 422, description = "an unknown field"), + (status = 503, description = "no swarm queue is configured on this host, or it is \ + not connected (problem+json)", body = String), + (status = 500, description = "the caps could not be written (problem+json)", + body = String), + ), + tag = "agents" +)] +pub(crate) async fn put_agent_limits( + State(state): State, + Path(name): Path, + Json(req): Json, +) -> Result, ProblemDetails> { + let agent = agent_name(&name).map_err(refused)?; + let value = req + .into_stored() + .map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?; + let client = queue(&state).map_err(refused)?; + if !live(&state).await?.contains(&agent) { + return Err(error_problem( + StatusCode::BAD_REQUEST, + &format!("no hive's wanted state declares an agent {agent:?} that is not destroyed"), + )); + } + let store = open_or_create(&client).await?; + for _ in 0..MAX_ATTEMPTS { + let (current, revision) = read::(&store, &agent) + .await + .map_err(|e| queue_problem(&e))?; + if current == value + || write(&store, &agent, &value, revision) + .await + .map_err(|e| queue_problem(&e))? + { + tracing::info!(%agent, ?value, "set agent limits"); + return Ok(Json(value.into())); + } + } + Err(queue_problem(&anyhow::anyhow!( + "gave up writing the limits of {agent} after {MAX_ATTEMPTS} conflicting writes" + ))) +} + +#[utoipa::path( + delete, + path = "/api/agents/{name}/limits", + params(("name" = String, Path, description = "agent name")), + responses( + (status = 204, description = "cleared; the hive's own answer applies again"), + (status = 400, description = "not an identifier (problem+json)", body = String), + (status = 503, description = "no swarm queue is configured on this host, or it is \ + not connected (problem+json)", body = String), + (status = 500, description = "the caps could not be cleared (problem+json)", + body = String), + ), + tag = "agents" +)] +pub(crate) async fn delete_agent_limits( + State(state): State, + Path(name): Path, +) -> Result { + let agent = agent_name(&name).map_err(refused)?; + let client = queue(&state).map_err(refused)?; + let store = open_or_create(&client).await?; + store + .delete(&agent) + .await + .map_err(|e| queue_problem(&anyhow::Error::new(e)))?; + tracing::info!(%agent, "cleared agent limits"); + Ok(StatusCode::NO_CONTENT) +} + +#[cfg(test)] +mod tests { + use super::{ + Caps, cpu_quota, delete_agent_limits, get_agent_limits, memory_max, put_agent_limits, + }; + use crate::test_nats::TestNats; + use crate::wanted::WantedWriter; + use axum::extract::{Path, State}; + use axum::http::StatusCode; + use std::sync::Arc; + use swarm_queue_client::wanted::AgentState; + + /// Of the `bad` values, systemd parses `""` (no quota) and `+5%`, and the + /// last adds a directive on a second line. It ignores every other one + /// ("Invalid CPU quota …, ignoring", checked with `systemd-analyze + /// verify`), which would leave the agent uncapped. + #[test] + fn cpu_quota_takes_a_percentage_above_zero() { + for ok in [ + "400%", + "100%", + "0.5%", + "0.01%", + "1500%", + "400.12%", + "21474836.47%", + ] { + assert_eq!(cpu_quota(ok).as_deref(), Ok(ok)); + } + for bad in [ + "0%", + "0.0%", + "0.00%", + "-50%", + "400", + "4 00%", + "%", + "lots", + "", + "1e3%", + "+5%", + "400.123%", + "0.001%", + "99999999999999999999%", + "21474836.48%", + "05%", + "08%", + "085480%", + "010%", + "00.5%", + "1.%", + ".5%", + "400%\nExecStartPre=/bin/sh -c id #%", + ] { + assert!(cpu_quota(bad).is_err(), "{bad:?} was accepted"); + } + } + + /// As above for `MemoryMax=`. Of the `bad` values, systemd also parses `3P` + /// and `2E`, which hive-c0re's validator refuses, and `8 G` and `1.G`. + #[test] + fn memory_max_takes_a_size_a_share_or_infinity_that_systemd_applies() { + for ok in [ + "8G", + "512M", + "4096K", + "1048576", + "1048576B", + "1.5B", + "0.5K", + "1.5T", + "1.23456789G", + "16383T", + "16777215T", + "18446744073709551614", + "18446744073709551614.5", + "1.00000000000000000001G", + "50%", + "12.34%", + "0.01%", + "100%", + "100.00%", + "infinity", + ] { + assert_eq!(memory_max(ok).as_deref(), Ok(ok), "{ok:?}"); + } + for bad in [ + "0", + "0G", + "0.0G", + "0.5", + "0%", + "-8G", + "150%", + "100.01%", + "12.345%", + "08%", + "085480%", + "010%", + "00.5%", + "8X", + "G", + "8 G", + "", + "eight", + "8GG", + "8gb", + "8g", + "8GB", + "4096k", + "1048576b", + "3P", + "2E", + ".5G", + "1.G", + "Infinity", + "17000000T", + "16777216T", + "16777215.5T", + "99999999999999999999T", + "18446744073709551615", + "18446744073709551616", + "1.123456789012345678901234567890G", + "8G\nDevicePolicy=auto", + ] { + assert!(memory_max(bad).is_err(), "{bad:?} was accepted"); + } + } + + /// One hive, `pr1ma`, on `client`. + fn state(client: &async_nats::Client) -> crate::AppState { + crate::AppState { + hives: Arc::new(vec![crate::HiveEntry { + name: "pr1ma".to_owned(), + domain: "pr1ma.example".to_owned(), + }]), + links: Arc::new(Vec::new()), + status: Some(Arc::new(crate::status::StatusReader::new( + client.clone(), + std::time::Duration::from_mins(1), + ))), + wanted: Some(Arc::new(WantedWriter::new(client.clone()))), + agent_status: None, + agent_icons: None, + reported_inference: None, + jobq: Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new( + hive_jobq::Graph::new(), + hive_jobq::resources::ResourceTable::new(), + ))), + webhook_secret: None, + config_prs: Arc::default(), + swarm_name: None, + auth: None, + forge: Arc::default(), + create_gate: Arc::default(), + accounts: crate::linked_accounts::Accounts::Fake(Arc::default()), + try_outs: crate::review_classifier::TryOuts::default(), + } + } + + /// A state on a real server whose `pr1ma` declares `atlas`. + async fn declared(client: &async_nats::Client) -> crate::AppState { + WantedWriter::new(client.clone()) + .set("pr1ma", "atlas", AgentState::Up) + .await + .expect("declare atlas"); + state(client) + } + + fn caps(cpu: Option<&str>, mem: Option<&str>) -> Caps { + Caps { + cpu_quota: cpu.map(str::to_owned), + memory_max: mem.map(str::to_owned), + } + } + + async fn put( + state: &crate::AppState, + agent: &str, + value: Caps, + ) -> Result, problem_details::ProblemDetails> { + put_agent_limits( + State(state.clone()), + Path(agent.to_owned()), + axum::Json(value), + ) + .await + } + + async fn get(state: &crate::AppState, agent: &str) -> Caps { + get_agent_limits(State(state.clone()), Path(agent.to_owned())) + .await + .expect("caps are read") + .0 + } + + /// The raw value a hive reads for `agent`, or `None` for no key or a + /// deleted one. + async fn published(client: &async_nats::Client, agent: &str) -> Option { + let store = swarm_queue_client::agent_limits::open_or_create(client) + .await + .expect("bucket"); + store + .get(agent) + .await + .expect("read") + .map(|v| String::from_utf8(v.to_vec()).expect("utf-8")) + } + + /// Through the real handlers and a real `nats-server`: a cap lands under + /// the agent's own key as the hive reads it, `GET` answers it, and + /// `DELETE` takes it out again. + /// + /// The published bytes are pinned literally because hive-c0re's + /// `workers::swarm_limits` tests decode this same string; a change to + /// either side's reading of it fails one of the two. + #[tokio::test] + async fn the_route_writes_the_caps_a_hive_reads_and_delete_clears_them() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + + let answer = put(&state, "atlas", caps(Some("400%"), Some("8G"))) + .await + .expect("caps are set"); + assert_eq!(answer.0, caps(Some("400%"), Some("8G"))); + assert_eq!( + published(&client, "atlas").await.as_deref(), + Some(r#"{"cpu_quota":"400%","memory_max":"8G"}"#) + ); + assert_eq!(get(&state, "atlas").await, caps(Some("400%"), Some("8G"))); + + assert_eq!( + delete_agent_limits(State(state.clone()), Path("atlas".to_owned())).await, + Ok(StatusCode::NO_CONTENT) + ); + assert_eq!(published(&client, "atlas").await, None); + assert_eq!(get(&state, "atlas").await, Caps::default()); + } + + /// A read before any cap was set answers `{}` and leaves the bucket + /// uncreated. + #[tokio::test] + async fn a_read_before_any_cap_answers_empty_and_creates_no_bucket() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = state(&client); + + assert_eq!(get(&state, "atlas").await, Caps::default()); + assert!( + swarm_queue_client::agent_limits::open(&client) + .await + .expect("the queue answers") + .is_none() + ); + } + + /// A name no hive declares gets no key, so a typo cannot cap an agent + /// that does not exist yet. + #[tokio::test] + async fn an_agent_no_hive_declares_is_refused_and_nothing_is_written() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + + let problem = put(&state, "iris", caps(Some("400%"), None)) + .await + .expect_err("no hive declares iris"); + assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}"); + assert_eq!(published(&client, "iris").await, None); + } + + /// A refused cap is answered before the queue is asked: the client here + /// was never connected, and would answer 503. + #[tokio::test] + async fn a_refused_cap_is_a_400_before_the_queue() { + let client = async_nats::ConnectOptions::new() + .retry_on_initial_connect() + .connect("127.0.0.1:1") + .await + .expect("returns without a connection"); + let state = state(&client); + + let problem = put(&state, "atlas", caps(Some("0%"), None)) + .await + .expect_err("a zero cap is refused"); + assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}"); + + // The control: a valid cap on the same state does reach the queue. + let problem = put(&state, "atlas", caps(Some("400%"), None)) + .await + .expect_err("the queue is not connected"); + assert_eq!( + problem.status, + Some(StatusCode::SERVICE_UNAVAILABLE), + "{problem:?}" + ); + } + + const CAPPED: &str = r#"{"cpu_quota":"400%","memory_max":"8G"}"#; + + /// `atlas` capped, behind the real router. + async fn capped_router(client: &async_nats::Client) -> axum::Router { + let state = declared(client).await; + let _ = put(&state, "atlas", caps(Some("400%"), Some("8G"))) + .await + .expect("cap atlas"); + crate::build_app(state) + } + + async fn put_raw(app: &axum::Router, body: &'static str) -> StatusCode { + use tower::ServiceExt as _; + app.clone() + .oneshot( + axum::http::Request::put("/api/agents/atlas/limits") + .header(axum::http::header::CONTENT_TYPE, "application/json") + .body(axum::body::Body::from(body)) + .expect("request"), + ) + .await + .expect("router answers") + .status() + } + + /// A body that is not JSON is refused before the handler runs, and the + /// caps already set stay. The well-formed `{}` after it is the control: + /// the same router and request do reach the bucket. + #[tokio::test] + async fn a_malformed_body_is_a_400_and_the_caps_stay() { + let nats = TestNats::start(); + let client = nats.client().await; + let app = capped_router(&client).await; + + for body in [r#"{"cpu_quota":"#, "cpu_quota=400%", ""] { + assert_eq!( + put_raw(&app, body).await, + StatusCode::BAD_REQUEST, + "{body:?}" + ); + assert_eq!( + published(&client, "atlas").await.as_deref(), + Some(CAPPED), + "{body:?}" + ); + } + + assert_eq!(put_raw(&app, "{}").await, StatusCode::OK); + assert_eq!(published(&client, "atlas").await.as_deref(), Some("{}")); + } + + /// A value carrying a directive on a second line is a 400 through the real + /// router, and the caps already set stay. + #[tokio::test] + async fn a_cap_with_a_second_line_is_a_400_and_the_caps_stay() { + let nats = TestNats::start(); + let client = nats.client().await; + let app = capped_router(&client).await; + + for body in [ + r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#, + r#"{"memory_max":"8G\nDevicePolicy=auto"}"#, + ] { + assert_eq!(put_raw(&app, body).await, StatusCode::BAD_REQUEST, "{body}"); + assert_eq!( + published(&client, "atlas").await.as_deref(), + Some(CAPPED), + "{body}" + ); + } + } + + /// A field the body does not define is a 422 and writes nothing: a + /// misspelt `memory_max` must not read as "no memory cap". + #[tokio::test] + async fn an_unknown_field_is_a_422_and_the_caps_stay() { + let nats = TestNats::start(); + let client = nats.client().await; + let app = capped_router(&client).await; + + for body in [ + r#"{"cpu_quota":"100%","memory":"2G"}"#, + r#"{"limits":{"cpu_quota":"100%"}}"#, + ] { + assert_eq!( + put_raw(&app, body).await, + StatusCode::UNPROCESSABLE_ENTITY, + "{body}" + ); + assert_eq!( + published(&client, "atlas").await.as_deref(), + Some(CAPPED), + "{body}" + ); + } + } +} diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index 63c3f03e0..cde67d31d 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -44,6 +44,7 @@ use utoipa_axum::{router::OpenApiRouter, routes}; mod agent_icon; mod agent_identity; mod agent_input; +mod agent_limits; mod agent_renewal; mod agent_state_stream; mod agent_status; @@ -3300,6 +3301,11 @@ fn build_app(state: AppState) -> axum::Router { inference::put_agent_inference, inference::delete_agent_inference )) + .routes(routes!( + agent_limits::get_agent_limits, + agent_limits::put_agent_limits, + agent_limits::delete_agent_limits + )) .routes(routes!(get_config_prs)) .routes(routes!(create_agent)) .routes(routes!(mint_agent_identity)) diff --git a/swarm-nats-auth/src/policy.rs b/swarm-nats-auth/src/policy.rs index b9d6db695..e4a692772 100644 --- a/swarm-nats-auth/src/policy.rs +++ b/swarm-nats-auth/src/policy.rs @@ -490,6 +490,17 @@ impl Policy { format!("$JS.API.STREAM.CREATE.{agent_status}"), format!("$KV.{}.{hive}.*", swarm_queue_client::agent_status::BUCKET), ]); + // Reading the caps the swarm sets for the agents this hive runs + // (`swarm_queue_client::agent_limits`): a direct get, so read-only, and + // no `STREAM.INFO` because the reader skips the lookup. + // + // Every agent's key, not just this hive's: the bucket is keyed by agent + // alone, and naming one hive's agents here needs the roster + // `hive_name`'s doc argues against. + subjects.push(format!( + "$JS.API.DIRECT.GET.{}.>", + swarm_queue_client::agent_limits::STREAM + )); subjects.extend( self.extra_hive_subjects .iter() @@ -712,6 +723,51 @@ mod tests { } } + /// A hive reads any agent's caps and can change none of them, and the + /// grant reaches no other bucket. + #[test] + fn a_hive_may_read_every_agents_caps_and_write_none() { + use swarm_queue_client::agent_limits::{STREAM, direct_get_subject, key_subject}; + + let p = policy() + .permissions("hive-alpha") + .expect("a hive is admitted"); + let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject)); + + // Its own agents and agents running elsewhere alike. + for agent in ["atlas", "iris"] { + assert!(may(&direct_get_subject(agent)), "cannot read {agent}"); + } + // A KV put or delete is a publish to the key's subject; the rest + // reshape or empty the stream. + for forbidden in [ + key_subject("atlas"), + format!("$JS.API.STREAM.CREATE.{STREAM}"), + format!("$JS.API.STREAM.UPDATE.{STREAM}"), + format!("$JS.API.STREAM.DELETE.{STREAM}"), + format!("$JS.API.STREAM.PURGE.{STREAM}"), + format!("$JS.API.STREAM.MSG.DELETE.{STREAM}"), + format!("$JS.API.CONSUMER.CREATE.{STREAM}"), + // A named consumer, bare and with a key filter, which a grant of + // `CONSUMER.CREATE..>` would cover. + format!("$JS.API.CONSUMER.CREATE.{STREAM}.c1"), + format!( + "$JS.API.CONSUMER.CREATE.{STREAM}.c1.{}", + key_subject("atlas") + ), + ] { + assert!(!may(&forbidden), "hive alpha may publish {forbidden}"); + } + // The read reaches only this bucket. + for other in [ + swarm_queue_client::agent_inference::direct_get_subject("atlas"), + swarm_queue_client::agent_schedules::direct_get_subject("atlas"), + "$JS.API.DIRECT.GET.KV_hive-wanted-beta.$KV.hive-wanted-beta.beta".to_owned(), + ] { + assert!(!may(&other), "hive alpha may publish {other}"); + } + } + #[test] fn an_extra_subject_without_the_placeholder_is_refused() { // 🩸 The option exists to put a second stream inside ONE hive's diff --git a/swarm-queue-client/Cargo.toml b/swarm-queue-client/Cargo.toml index 1123f9a35..5496599d2 100644 --- a/swarm-queue-client/Cargo.toml +++ b/swarm-queue-client/Cargo.toml @@ -47,6 +47,9 @@ agent-inference = ["async-nats/jetstream"] # `ToSchema` on the review classifier's shared types, for swarm-controller, # whose HTTP API serves the same shapes the queue carries. schema = ["dep:utoipa"] +# `agent_limits::read`, for hive-c0re, which reads the key of each agent it +# runs the same way. Explicit for the same reason as `notices`. +agent-limits = ["async-nats/jetstream"] [dependencies] # Bare (no `kv`/`jetstream`) unless a consumer opts into the `kv` feature diff --git a/swarm-queue-client/src/agent_limits.rs b/swarm-queue-client/src/agent_limits.rs new file mode 100644 index 000000000..77cf3b7f0 --- /dev/null +++ b/swarm-queue-client/src/agent_limits.rs @@ -0,0 +1,190 @@ +//! The per-agent resource-cap KV bucket: one key per agent, holding the CPU +//! and memory caps the swarm sets for it. +//! +//! **swarm-controller is the only writer**, and creates the bucket on its +//! first write. Every field is optional: an absent field leaves the agent on +//! its hive's own answer, so an agent with no key runs exactly as it would +//! with no swarm. +//! +//! **The hive running the agent reads the key**, not the agent: the caps go +//! into the container's systemd drop-in, which the host writes. A hive reads +//! with one `DIRECT.GET` per agent +//! ([`crate::agent_limits::direct_get_subject`]), and its grant covers every +//! key of this bucket, read-only. Scoping it to the hive's own agents would +//! need a roster in the auth responder. +//! +//! Keyed by the agent alone, like [`crate::agent_inference`]: an agent can +//! move between hives, and its caps go with it. + +/// The KV bucket caps are published into, keyed by agent name. +pub const BUCKET: &str = "agent-limits"; + +/// The stream backing [`BUCKET`]. `create_key_value` prefixes `KV_`. +pub const STREAM: &str = "KV_agent-limits"; + +/// The subject `agent`'s entry is stored under. +#[must_use] +pub fn key_subject(agent: &str) -> String { + format!("$KV.{BUCKET}.{agent}") +} + +/// The request subject that reads `agent`'s current entry. +#[must_use] +pub fn direct_get_subject(agent: &str) -> String { + format!("$JS.API.DIRECT.GET.{STREAM}.{}", key_subject(agent)) +} + +/// One agent's caps: the value under `BUCKET`/``, as systemd values +/// carried verbatim. `None` is "not capped by the swarm", never "uncapped". +#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct AgentCaps { + /// systemd `CPUQuota=`, e.g. `"400%"`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cpu_quota: Option, + /// systemd `MemoryMax=`, e.g. `"8G"`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub memory_max: Option, +} + +impl AgentCaps { + /// True when no field is set, which a hive reads the same as no key. + #[must_use] + pub fn is_empty(&self) -> bool { + self.cpu_quota.is_none() && self.memory_max.is_none() + } +} + +/// Open the caps bucket for writing, creating it if nothing has yet. +/// +/// **Controller-side only.** `history: 1`: a hive wants the current value. +#[cfg(feature = "kv")] +pub async fn open_or_create( + client: &async_nats::Client, +) -> Result { + let js = async_nats::jetstream::new(client.clone()); + crate::kv::open_or_create( + &js, + BUCKET, + "agent-limits", + "Resource caps the swarm controller sets for each agent".to_owned(), + ) + .await +} + +/// Open the caps bucket for reading, or `None` when nothing has created it +/// yet, which reads the same as no caps. +/// +/// **Controller-side**, for reads that must not create the bucket. +/// +/// # Errors +/// +/// [`crate::Error::OpenBucket`] for every failure other than the bucket not +/// existing. +#[cfg(feature = "kv")] +pub async fn open( + client: &async_nats::Client, +) -> Result, crate::Error> { + let js = async_nats::jetstream::new(client.clone()); + match js.get_key_value(BUCKET).await { + Ok(store) => Ok(Some(store)), + Err(e) if crate::wanted::is_bucket_not_found(&e) => Ok(None), + Err(source) => Err(crate::Error::OpenBucket { + bucket: BUCKET.to_owned(), + source, + }), + } +} + +/// What reading one agent's key found. +#[cfg(feature = "agent-limits")] +#[derive(Debug)] +pub enum Read { + /// The bucket answered; an absent or deleted key has every field unset. + Current(AgentCaps), + /// Nothing answers for the bucket's stream: no cap has been written yet, + /// or `JetStream` is not serving it right now. Not an unset value, since + /// the second case says nothing about the agent's caps. + NoBucket, +} + +/// Read `agent`'s current caps with one direct get. +/// +/// # Errors +/// +/// A failed request or an undecodable value. A request the queue refused for +/// permissions reaches the client as a timeout, not as a refusal. +#[cfg(feature = "agent-limits")] +pub async fn read( + client: &async_nats::Client, + agent: &str, +) -> Result> { + use async_nats::jetstream::stream::DirectGetErrorKind; + + let js = async_nats::jetstream::new(client.clone()); + let stream = js.get_stream_no_info(STREAM).await?; + let message = match stream.direct_get_last_for_subject(key_subject(agent)).await { + Ok(message) => message, + Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => { + return Ok(Read::Current(AgentCaps::default())); + } + Err(e) if is_no_responders(&e) => return Ok(Read::NoBucket), + Err(e) => return Err(e.into()), + }; + // A KV delete or purge is stored as an empty marker carrying this header. + let removed = message + .headers + .get("KV-Operation") + .is_some_and(|op| matches!(op.as_str(), "DEL" | "PURGE")); + if removed { + return Ok(Read::Current(AgentCaps::default())); + } + Ok(Read::Current(serde_json::from_slice(&message.payload)?)) +} + +#[cfg(feature = "agent-limits")] +fn is_no_responders(e: &async_nats::jetstream::stream::DirectGetError) -> bool { + matches!( + e.kind(), + async_nats::jetstream::stream::DirectGetErrorKind::ErrorResponse(status, _) + if status == async_nats::StatusCode::NO_RESPONDERS + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The names the hive's grant in `swarm-nats-auth` is built from. + #[test] + fn names_match_the_queue_grant() { + assert_eq!(STREAM, format!("KV_{BUCKET}")); + assert_eq!(key_subject("iris"), "$KV.agent-limits.iris"); + assert_eq!( + direct_get_subject("iris"), + "$JS.API.DIRECT.GET.KV_agent-limits.$KV.agent-limits.iris" + ); + } + + /// An unset field is left out on the wire, and `{}` is every field unset. + #[test] + fn unset_fields_are_absent() { + let value = AgentCaps { + memory_max: Some("8G".to_owned()), + ..AgentCaps::default() + }; + assert_eq!( + serde_json::to_string(&value).unwrap(), + r#"{"memory_max":"8G"}"# + ); + let empty: AgentCaps = serde_json::from_str("{}").unwrap(); + assert!(empty.is_empty()); + } + + /// Fields a newer controller adds must not cost an older hive its caps. + #[test] + fn decoding_tolerates_unknown_fields() { + let decoded: AgentCaps = + serde_json::from_str(r#"{"cpu_quota":"400%","io_weight":"10"}"#).unwrap(); + assert_eq!(decoded.cpu_quota.as_deref(), Some("400%")); + } +} diff --git a/swarm-queue-client/src/lib.rs b/swarm-queue-client/src/lib.rs index fa5b991b6..6cd0da3a4 100644 --- a/swarm-queue-client/src/lib.rs +++ b/swarm-queue-client/src/lib.rs @@ -216,6 +216,11 @@ pub mod agent_schedules; /// unconditional; writing needs `kv` and reading needs `agent-inference`. pub mod agent_inference; +/// The bucket swarm-controller sets each agent's CPU and memory caps in — one +/// key per agent, read by the hive that runs it. The names are unconditional; +/// writing needs `kv` and reading needs `agent-limits`. +pub mod agent_limits; + #[cfg(feature = "kv")] mod kv; diff --git a/swarm-queue-client/src/wanted.rs b/swarm-queue-client/src/wanted.rs index 451039c5d..16681441b 100644 --- a/swarm-queue-client/src/wanted.rs +++ b/swarm-queue-client/src/wanted.rs @@ -156,7 +156,7 @@ pub async fn open_read_only( /// `get_key_value` reports a missing bucket as `GetBucket` wrapping the /// stream lookup's `STREAM_NOT_FOUND`; every other kind is a real failure. #[cfg(feature = "kv")] -fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool { +pub(crate) fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool { use async_nats::jetstream::context::{GetStreamError, GetStreamErrorKind, KeyValueErrorKind}; e.kind() == KeyValueErrorKind::GetBucket