diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md
index 3ca1043c9..851f7e73c 100644
--- a/docs/agent-lifecycle/persistence.md
+++ b/docs/agent-lifecycle/persistence.md
@@ -411,6 +411,13 @@ Contents:
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
drop-in have **no** per-agent override — they're hive-wide only and
come straight off `HiveEnv`, so this file has no field for them.
+- `swarm-resource-limits.json` — the caps the swarm sets for this
+ hive's agents, same shape as `resource-limits.json`. Written only by
+ `workers::swarm_limits`, which copies each agent's key of the swarm's
+ `agent-limits` bucket here every 30 seconds. A field set here replaces
+ that field's answer from `resource-limits.json` and the hive-wide
+ default. Every value passes the same validators as the operator's
+ overrides first, and one bad value makes the worker change nothing here.
The root agent has the meta dir RO-mounted at `/meta/`.
diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md
index 20d4dab39..043c34b9e 100644
--- a/docs/scheduler/coordinator.md
+++ b/docs/scheduler/coordinator.md
@@ -452,14 +452,24 @@ regardless.
rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild.
+The answer per agent and per field is, first match wins:
+
+1. the swarm's cap for the agent (see below);
+2. the operator's per-agent override in `meta/resource-limits.json`;
+3. `agentCpuQuota` / `agentMemoryMax`.
+
+A swarm cap replaces the hive's answer, so a hive can't raise an agent
+above it. Like the rest of the drop-in, a changed swarm cap reaches a
+running agent at its next spawn or rebuild.
+
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
-the last drop-in because it can't read the overrides file, the file carries
-that drop-in's `MemoryMax=`.
+the last drop-in because it can't read the overrides or the swarm caps file,
+the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
@@ -469,6 +479,39 @@ changed cap reaches them at the next container start without a rebuild.
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
+Implementation: swarm caps
+
+swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
+agent and no hive in it, so an agent that moves to another hive keeps its
+caps. swarm-controller alone writes it:
+
+| method | path | body |
+| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
+| `GET` | `/api/agents//limits` | —; answers the stored caps, `{}` when there are none |
+| `PUT` | `/api/agents//limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
+| `DELETE` | `/api/agents//limits` | —; answers 204 |
+
+`PUT` refuses an agent no hive's wanted state declares, unknown fields, and
+any value systemd would ignore, which would leave the agent uncapped. A
+percentage has at most two decimals, no leading zero in its whole part (it
+refuses `08%`), and is above zero, up to `100%` for
+`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an
+optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at
+least one byte and below 2^64 - 1 bytes. The controller stores each value as
+given.
+
+Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
+agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
+value passes the same validators as the operator's overrides first. One bad
+value or a failed read leaves the file unchanged, and repeated failures raise a
+dashboard banner.
+
+A hive's queue grant reads every key of the bucket
+(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
+hive's own agents would need a roster in the auth responder.
+
+
+
The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle
diff --git a/docs/swarm/README.md b/docs/swarm/README.md
index 8c89734ce..0535bd71c 100644
--- a/docs/swarm/README.md
+++ b/docs/swarm/README.md
@@ -25,6 +25,9 @@ grafana, the controller, authelia, …) →
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
- **inference settings** — the model and effort the swarm sets for an agent,
over its own. → [`inference.md`](inference.md)
+- **resource caps** — the CPU and memory cap the swarm sets for an agent,
+ over its hive's. →
+ [`coordinator.md`](../scheduler/coordinator.md#container-resource-limits)
- **review classifier** — planned, not implemented yet: checks a PR against
its repo's own yes/no rules and posts the result as a review. →
[`review-classifier.md`](review-classifier.md)
diff --git a/hive-c0re/Cargo.toml b/hive-c0re/Cargo.toml
index ac74773ff..9e8d0f0c0 100644
--- a/hive-c0re/Cargo.toml
+++ b/hive-c0re/Cargo.toml
@@ -53,7 +53,11 @@ strum.workspace = true
# Offering this hive's status to the swarm (`swarm_status`). The same crate
# the swarm controller reads it with, and `kv` for the same reason: the
# bucket's name and creation config belong to neither end of it alone.
-swarm-queue-client = { workspace = true, features = ["kv", "notices"] }
+swarm-queue-client = { workspace = true, features = [
+ "kv",
+ "notices",
+ "agent-limits",
+] }
swarm-secret-client.workspace = true
tokio.workspace = true
tokio-stream.workspace = true
diff --git a/hive-c0re/src/agent_config/resource_limits.rs b/hive-c0re/src/agent_config/resource_limits.rs
index fd68b78ae..880c87da0 100644
--- a/hive-c0re/src/agent_config/resource_limits.rs
+++ b/hive-c0re/src/agent_config/resource_limits.rs
@@ -187,6 +187,86 @@ fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Resu
super::write_map(path, ¤t)
}
+/// Caps the swarm controller declared for this hive's agents.
+///
+/// A separate file from the operator's overrides above, not a merge into it:
+/// the two have different writers and different lifetimes, and a swarm
+/// declaration landing in the file an operator edits would make "reset to
+/// hive defaults" destroy something the operator never set.
+const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json";
+
+#[must_use]
+pub fn swarm_limits_path() -> PathBuf {
+ crate::paths::meta_root().join(SWARM_LIMITS_FILE)
+}
+
+/// Read the swarm-declared caps. An absent file is the empty map — no swarm
+/// opinion, so the hive's own chain decides. A file that exists but can't be
+/// read or parsed is an error, for the reason [`read`] gives: reading it as
+/// empty would silently drop a cap.
+pub fn read_swarm() -> std::io::Result> {
+ super::read_map(&swarm_limits_path())
+}
+
+/// Persist the swarm-declared caps atomically.
+///
+/// # Errors
+/// The meta dir cannot be created, or the file cannot be written.
+pub fn write_swarm(map: &BTreeMap) -> std::io::Result<()> {
+ super::write_map(&swarm_limits_path(), map)
+}
+
+/// [`effective_from`] with the swarm's declaration on top.
+///
+/// A swarm cap **replaces** the hive's answer for that field rather than
+/// joining the fallback chain as another tier: a tier a hive can out-set is
+/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's
+/// chain.
+///
+/// The chain itself is untouched — [`effective_from`] is called exactly as
+/// before and its result is what a missing swarm value falls back to.
+#[must_use]
+pub fn effective_from_with_swarm(
+ swarm: &BTreeMap,
+ limits: &BTreeMap,
+ name: &str,
+ hive_cpu_quota: &str,
+ hive_memory_max: &str,
+) -> (String, String) {
+ let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max);
+ match swarm.get(name) {
+ Some(s) => (
+ s.cpu_quota.clone().unwrap_or(cpu),
+ s.memory_max.clone().unwrap_or(mem),
+ ),
+ None => (cpu, mem),
+ }
+}
+
+/// [`effective_from_with_swarm`] for a caller resolving a single agent, from
+/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`.
+///
+/// Delegates to [`effective`] rather than re-deriving the hive's answer, so
+/// there stays exactly one place the fallback chain lives. Errors as [`read`]
+/// does, for either file.
+pub fn effective_with_swarm(
+ limits_path: &Path,
+ swarm_path: &Path,
+ name: &str,
+ hive_cpu_quota: &str,
+ hive_memory_max: &str,
+) -> std::io::Result<(String, String)> {
+ let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?;
+ let swarm: BTreeMap = super::read_map(swarm_path)?;
+ Ok(match swarm.get(name) {
+ Some(s) => (
+ s.cpu_quota.clone().unwrap_or(cpu),
+ s.memory_max.clone().unwrap_or(mem),
+ ),
+ None => (cpu, mem),
+ })
+}
+
/// Validate a systemd `CPUQuota=` value. Percentages only, and values
/// above 100% are legitimate (one full core is 100%).
///
@@ -525,4 +605,52 @@ mod tests {
);
}
}
+
+ #[test]
+ fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() {
+ // The whole point of the ruling: a tier a hive can out-set is not a
+ // cap. The agent override below is the strongest hive-side value
+ // there is, and the swarm still wins.
+ let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]);
+ let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
+ assert_eq!(
+ effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
+ ("150%".to_owned(), "2G".to_owned())
+ );
+ }
+
+ #[test]
+ fn a_swarm_cap_is_per_field_like_every_other_tier() {
+ // A swarm that caps memory and says nothing about CPU leaves CPU on
+ // the hive's chain — here the per-agent override, not the hive default.
+ let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]);
+ let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
+ assert_eq!(
+ effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
+ ("800%".to_owned(), "2G".to_owned())
+ );
+ }
+
+ #[test]
+ fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() {
+ // The control that makes the two tests above mean something: the
+ // wrapper must be a no-op when the swarm has said nothing, for every
+ // shape the old chain distinguishes. If this drifts, the swarm tier
+ // changed behaviour it was explicitly told not to change.
+ let none = BTreeMap::new();
+ let cases = [
+ BTreeMap::new(),
+ BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]),
+ BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]),
+ BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]),
+ BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]),
+ ];
+ for hive_side in &cases {
+ assert_eq!(
+ effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM),
+ effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM),
+ "the wrapper changed the answer with no swarm value present"
+ );
+ }
+ }
}
diff --git a/hive-c0re/src/container_view.rs b/hive-c0re/src/container_view.rs
index 012a9451f..d55686cc8 100644
--- a/hive-c0re/src/container_view.rs
+++ b/hive-c0re/src/container_view.rs
@@ -171,6 +171,14 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result anyhow::Result Re
if let Some((cpu_quota, memory_max)) = stage_limits(
name,
&crate::resource_limits::resource_limits_path(),
+ &crate::resource_limits::swarm_limits_path(),
&limits_dropin_path(&container),
&paths.agent,
hive,
@@ -97,12 +98,13 @@ impl HostMemory {
fn stage_limits(
agent_name: &str,
limits_path: &Path,
+ swarm_path: &Path,
dropin_path: &Path,
runtime_dir: &Path,
hive: &HiveEnv,
host: Option,
) -> Result> {
- let limits = limits_to_apply(agent_name, limits_path, dropin_path, hive);
+ let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive);
let memory_max = match &limits {
Some((_, memory_max)) => Some(memory_max.clone()),
None => dropin_memory_max(agent_name, dropin_path),
@@ -224,20 +226,28 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
///
-/// When the overrides file at `limits_path` can't be read, the error is
-/// logged and the agent still starts. The last-applied drop-in is kept,
-/// because the hive defaults can be looser than the agent's own override.
+/// When the overrides file at `limits_path` or the swarm caps at
+/// `swarm_path` can't be read, the error is logged and the agent still starts.
+/// The last-applied drop-in is kept, because the hive defaults can be looser
+/// than the agent's own override.
/// With no drop-in yet, the agent gets the hive defaults: without one it
/// would run uncapped.
fn limits_to_apply(
agent_name: &str,
limits_path: &Path,
+ swarm_path: &Path,
dropin_path: &Path,
hive: &HiveEnv,
) -> Option<(String, String)> {
let hive_cpu = &hive.agent_cpu_quota;
let hive_mem = &hive.agent_memory_max;
- let e = match crate::resource_limits::effective(limits_path, agent_name, hive_cpu, hive_mem) {
+ let e = match crate::resource_limits::effective_with_swarm(
+ limits_path,
+ swarm_path,
+ agent_name,
+ hive_cpu,
+ hive_mem,
+ ) {
Ok(limits) => return Some(limits),
Err(e) => e,
};
@@ -838,6 +848,10 @@ mod tests {
const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem";
+ fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf {
+ dir.path().join("absent-swarm-resource-limits.json")
+ }
+
#[test]
fn corrupt_limits_keep_an_existing_dropin() {
let dir = tempfile::tempdir().expect("tempdir");
@@ -846,7 +860,13 @@ mod tests {
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
assert_eq!(
- limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()),
+ limits_to_apply(
+ "sock",
+ &limits,
+ &absent_swarm(&dir),
+ &dropin,
+ &HiveEnv::default()
+ ),
None
);
assert_eq!(
@@ -863,12 +883,47 @@ mod tests {
let applied = limits_to_apply(
"sock",
&limits,
+ &absent_swarm(&dir),
&dir.path().join("absent.conf"),
&HiveEnv::default(),
);
assert_eq!(applied, Some(("200%".to_owned(), "4G".to_owned())));
}
+ #[test]
+ fn corrupt_swarm_caps_keep_an_existing_dropin() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let swarm = dir.path().join("swarm-resource-limits.json");
+ let dropin = dir.path().join("hyperhive-limits.conf");
+ std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
+ std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
+ let applied = limits_to_apply(
+ "sock",
+ &dir.path().join("resource-limits.json"),
+ &swarm,
+ &dropin,
+ &HiveEnv::default(),
+ );
+ assert_eq!(applied, None);
+ }
+
+ #[test]
+ fn a_swarm_cap_is_applied_over_the_operator_override() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let limits = dir.path().join("resource-limits.json");
+ let swarm = dir.path().join("swarm-resource-limits.json");
+ std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
+ std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps");
+ let applied = limits_to_apply(
+ "sock",
+ &limits,
+ &swarm,
+ &dir.path().join("absent.conf"),
+ &HiveEnv::default(),
+ );
+ assert_eq!(applied, Some(("200%".to_owned(), "2G".to_owned())));
+ }
+
#[test]
fn readable_limits_are_applied_over_an_existing_dropin() {
let dir = tempfile::tempdir().expect("tempdir");
@@ -876,7 +931,13 @@ mod tests {
let dropin = dir.path().join("hyperhive-limits.conf");
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in");
- let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default());
+ let applied = limits_to_apply(
+ "sock",
+ &limits,
+ &absent_swarm(&dir),
+ &dropin,
+ &HiveEnv::default(),
+ );
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
}
@@ -893,6 +954,7 @@ mod tests {
let applied = stage_limits(
"sock",
&dir.path().join("resource-limits.json"),
+ &dir.path().join("swarm-resource-limits.json"),
&dir.path().join("hyperhive-limits.conf"),
&runtime,
&HiveEnv::default(),
@@ -903,6 +965,28 @@ mod tests {
(applied.map(|(_, mem)| mem), copy)
}
+ /// The agent's copy follows the swarm's cap, the one systemd enforces, over
+ /// the operator's override: a heap sized against the override would
+ /// outgrow the cgroup.
+ #[test]
+ fn the_agent_reads_the_swarm_cap_over_the_operator_override() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ std::fs::write(
+ dir.path().join("resource-limits.json"),
+ r#"{"sock": {"memory_max": "16G"}}"#,
+ )
+ .expect("seed limits");
+ std::fs::write(
+ dir.path().join("swarm-resource-limits.json"),
+ r#"{"sock": {"memory_max": "2G"}}"#,
+ )
+ .expect("seed swarm caps");
+ assert_eq!(
+ stage(&dir),
+ (Some("2G".to_owned()), Some("2147483648\n".to_owned()))
+ );
+ }
+
#[test]
fn the_agent_reads_the_cap_its_dropin_carries() {
let dir = tempfile::tempdir().expect("tempdir");
@@ -1035,6 +1119,7 @@ mod tests {
let applied = stage_limits(
"sock",
&limits,
+ &dir.path().join("swarm-resource-limits.json"),
&dropin,
&runtime,
&HiveEnv::default(),
diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs
index 5d9839bc3..ea283fda3 100644
--- a/hive-c0re/src/server.rs
+++ b/hive-c0re/src/server.rs
@@ -426,8 +426,9 @@ async fn handle_set_resource_limits(
let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?;
crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?;
- let (cpu, mem) = crate::resource_limits::effective(
+ let (cpu, mem) = crate::resource_limits::effective_with_swarm(
&crate::resource_limits::resource_limits_path(),
+ &crate::resource_limits::swarm_limits_path(),
name.as_str(),
&hive.agent_cpu_quota,
&hive.agent_memory_max,
diff --git a/hive-c0re/src/swarm_status.rs b/hive-c0re/src/swarm_status.rs
index 13a1d455c..c4cfc54b6 100644
--- a/hive-c0re/src/swarm_status.rs
+++ b/hive-c0re/src/swarm_status.rs
@@ -103,6 +103,12 @@ pub fn spawn(
shutdown.clone(),
));
+ // The swarm's resource caps, on the same connection again.
+ tokio::spawn(crate::workers::swarm_limits::follow(
+ client.clone(),
+ shutdown.clone(),
+ ));
+
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER);
loop {
match publish(&client, &hive).await {
diff --git a/hive-c0re/src/workers/mod.rs b/hive-c0re/src/workers/mod.rs
index c3772e923..6f2ffa4c8 100644
--- a/hive-c0re/src/workers/mod.rs
+++ b/hive-c0re/src/workers/mod.rs
@@ -1,6 +1,7 @@
//! Background tasks and periodic sweeps: crash/login watcher, boot-time
//! auto-update reconcile, the agent-sockets.json writer loop, the MCP socket listener
-//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull.
+//! reconcile loop, knowledge-repo sync, the swarm wanted-state pull, and the
+//! swarm resource-cap read.
//! Each submodule is re-exported
//! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged.
@@ -9,4 +10,5 @@ pub mod auto_update;
pub mod crash_watch;
pub mod knowledge;
pub mod mcp_sockets;
+pub mod swarm_limits;
pub mod wanted;
diff --git a/hive-c0re/src/workers/swarm_limits.rs b/hive-c0re/src/workers/swarm_limits.rs
new file mode 100644
index 000000000..ec3b09790
--- /dev/null
+++ b/hive-c0re/src/workers/swarm_limits.rs
@@ -0,0 +1,207 @@
+//! Reading the caps the swarm sets for this hive's agents.
+//!
+//! swarm-controller keeps one key per agent in the `agent-limits` bucket
+//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads
+//! the key of each agent this hive has and stores what it finds in
+//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it.
+//! A running agent gets a changed cap at its first spawn or rebuild after
+//! that read.
+//!
+//! A failed read, or a bucket that does not answer, leaves the stored caps as
+//! they are: dropping them would read exactly like the swarm clearing them.
+
+use std::collections::BTreeMap;
+use std::time::Duration;
+
+use anyhow::{Context, Result};
+use swarm_queue_client::agent_limits::{self, AgentCaps, Read};
+
+use crate::resource_limits::AgentLimits;
+use crate::stats::sweep_health::{self, SweepHealth};
+
+/// How often the caps are read.
+pub const READ_INTERVAL: Duration = Duration::from_secs(30);
+
+/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
+const FAILURES_BEFORE_BANNER: u32 = 3;
+
+/// Read the caps now and every [`READ_INTERVAL`] until shutdown.
+pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver) {
+ let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER);
+ loop {
+ match sync(&client).await {
+ Ok(()) => health.record_ok(),
+ Err(e) => {
+ tracing::warn!(error = ?e, "swarm limits: reading the caps failed");
+ let err = format!("{e:#}");
+ health.record_err(|ctx| {
+ let age = ctx.since_last_ok.map_or_else(
+ || "no success this session".to_owned(),
+ |d| format!("last ok {} ago", sweep_health::fmt_age(d)),
+ );
+ format!(
+ "reading the swarm's resource caps is failing ({} consecutive, {age}) \
+ — agents keep the caps last read: {err}",
+ ctx.consecutive
+ )
+ });
+ }
+ }
+ tokio::select! {
+ () = tokio::time::sleep(READ_INTERVAL) => {}
+ _ = shutdown.changed() => {
+ tracing::info!("swarm limits: shutdown signal received");
+ return;
+ }
+ }
+ }
+}
+
+/// Read every agent's key and store the result, unless a read fails or a
+/// value is refused.
+async fn sync(client: &async_nats::Client) -> Result<()> {
+ // An unconnected client does not fail a JetStream request, it hangs on it.
+ swarm_queue_client::ensure_connected(client)?;
+ let agents = crate::lifecycle::agents_for_meta_listing()
+ .await
+ .context("enumerating this hive's agents")?;
+ let mut declared = BTreeMap::new();
+ for spec in agents {
+ let read = agent_limits::read(client, &spec.name)
+ .await
+ .map_err(|e| anyhow::anyhow!(e))
+ .with_context(|| format!("reading the swarm's caps for {}", spec.name))?;
+ match read {
+ Read::Current(caps) => {
+ declared.insert(spec.name, caps);
+ }
+ Read::NoBucket => {
+ tracing::debug!("swarm limits: the agent-limits bucket does not answer");
+ return Ok(());
+ }
+ }
+ }
+ let caps = validated(&declared)?;
+ if caps != crate::resource_limits::read_swarm().context("reading the stored swarm caps")? {
+ crate::resource_limits::write_swarm(&caps).context("storing the swarm caps")?;
+ tracing::info!(?caps, "swarm limits: stored the swarm's resource caps");
+ }
+ Ok(())
+}
+
+/// The caps to store for `declared`, by agent. An agent with no field set
+/// gets no entry.
+///
+/// Every value goes through the validators the operator's own input path
+/// uses, because it is rendered verbatim into the container's systemd
+/// drop-in. One bad value refuses the whole set.
+fn validated(declared: &BTreeMap) -> Result> {
+ let mut out = BTreeMap::new();
+ for (agent, caps) in declared {
+ if caps.is_empty() {
+ continue;
+ }
+ let limits = validated_caps(caps)
+ .map_err(|why| anyhow::anyhow!("the swarm set an invalid cap for {agent}: {why}"))?;
+ out.insert(agent.clone(), limits);
+ }
+ Ok(out)
+}
+
+fn validated_caps(caps: &AgentCaps) -> Result {
+ if let Some(cpu) = &caps.cpu_quota {
+ crate::resource_limits::validate_cpu_quota(cpu)?;
+ }
+ Ok(AgentLimits {
+ cpu_quota: caps.cpu_quota.clone(),
+ memory_max: caps
+ .memory_max
+ .as_deref()
+ .map(crate::resource_limits::validate_memory_max)
+ .transpose()?,
+ })
+}
+
+#[cfg(test)]
+mod tests {
+ use std::collections::BTreeMap;
+
+ use super::validated;
+ use crate::resource_limits::AgentLimits;
+ use swarm_queue_client::agent_limits::AgentCaps;
+
+ /// `atlas`'s answer with `atlas_key` as the value read for it, over a hive
+ /// chain that caps it at 800% / 16G.
+ fn atlas_with(atlas_key: &str) -> (String, String) {
+ let declared = BTreeMap::from([(
+ "atlas".to_owned(),
+ serde_json::from_str::(atlas_key).expect("decodes"),
+ )]);
+ let stored = validated(&declared).expect("valid caps");
+ let hive_side = BTreeMap::from([(
+ "atlas".to_owned(),
+ AgentLimits {
+ cpu_quota: Some("800%".to_owned()),
+ memory_max: Some("16G".to_owned()),
+ },
+ )]);
+ crate::resource_limits::effective_from_with_swarm(
+ &stored, &hive_side, "atlas", "200%", "4G",
+ )
+ }
+
+ /// The other half of swarm-controller's
+ /// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`:
+ /// the first value is the bytes that route stores. The cap replaces the
+ /// hive's chain; a deleted key reads as `{}` and gives the chain back.
+ #[test]
+ fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() {
+ assert_eq!(
+ atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#),
+ ("400%".to_owned(), "8G".to_owned())
+ );
+ assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned()));
+ }
+
+ /// Per field: a swarm memory cap leaves CPU on the hive's chain.
+ #[test]
+ fn a_field_the_swarm_leaves_unset_stays_with_the_hive() {
+ assert_eq!(
+ atlas_with(r#"{"memory_max":"2G"}"#),
+ ("800%".to_owned(), "2G".to_owned())
+ );
+ }
+
+ #[test]
+ fn an_agent_with_no_caps_gets_no_entry() {
+ let declared = BTreeMap::from([("atlas".to_owned(), AgentCaps::default())]);
+ assert!(validated(&declared).expect("valid").is_empty());
+ }
+
+ /// The values are rendered verbatim into a systemd drop-in, so a value
+ /// the hive's own validators refuse, a newline smuggling in a directive
+ /// among them, refuses the whole set, the valid cap beside it included.
+ #[test]
+ fn an_invalid_cap_is_refused_and_nothing_is_stored() {
+ for bad in [
+ r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#,
+ r#"{"memory_max":"8G\nDevicePolicy=auto"}"#,
+ r#"{"cpu_quota":"lots"}"#,
+ r#"{"memory_max":"8X"}"#,
+ ] {
+ let declared = BTreeMap::from([
+ ("atlas".to_owned(), AgentCaps::default()),
+ (
+ "iris".to_owned(),
+ serde_json::from_str(r#"{"memory_max":"2G"}"#).expect("decodes"),
+ ),
+ (
+ "sock".to_owned(),
+ serde_json::from_str(bad).expect("decodes"),
+ ),
+ ]);
+ let err = validated(&declared).expect_err(bad);
+ assert!(err.to_string().contains("sock"), "{err:#}");
+ }
+ }
+}
diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix
index f99c1a11d..48326b2f4 100644
--- a/nix/module-eval/core-toggle.nix
+++ b/nix/module-eval/core-toggle.nix
@@ -286,6 +286,22 @@ let
&& !(lib.hasInfix "mint swarmctl" s)
&& lib.hasInfix "mint swarm-authelia-bridge" s;
}
+ {
+ # swarm-controller checks no caller itself: this location's authelia
+ # subrequest is the operator gate for every route under `/api/`. The
+ # webhook location is the control, the one path to the same socket that
+ # skips the gate, so the first clause reads a string that can lack it.
+ name = "the swarm UI's API location asks authelia before reaching the controller";
+ ok =
+ let
+ l = allLocal.services.nginx.virtualHosts."t.local".locations;
+ socket = allLocal.services.hyperhive.deploy.swarm-controller.socketPath;
+ in
+ lib.hasInfix "auth_request /__hive_authelia;" l."/api/".extraConfig
+ && lib.hasInfix socket l."/api/".proxyPass
+ && lib.hasInfix socket l."/webhook/forge/".proxyPass
+ && !(lib.hasInfix "auth_request" l."/webhook/forge/".extraConfig);
+ }
{
name = "a hive with matrix off serves no matrix discovery endpoint";
ok =
diff --git a/swarm-controller/README.md b/swarm-controller/README.md
index 036ff2836..4c7e12d4a 100644
--- a/swarm-controller/README.md
+++ b/swarm-controller/README.md
@@ -52,6 +52,11 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
agent applies them itself. `GET` also answers the model and effort the
agent last reported in its turn-state header, kept in memory →
[`docs/swarm/inference.md`](../docs/swarm/inference.md).
+- **Resource caps** — each agent's CPU and memory cap, one key per agent in
+ the `agent-limits` KV bucket, which this daemon alone writes
+ (`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the
+ agent reads it, and each cap replaces the hive's own answer for that field →
+ [`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits).
- **Relays** — each agent's terminal and turn-state header as SSE, agent
icons, the cross-repo issue report, and the UI's quick links.
diff --git a/swarm-controller/src/agent_limits.rs b/swarm-controller/src/agent_limits.rs
new file mode 100644
index 000000000..90351ab31
--- /dev/null
+++ b/swarm-controller/src/agent_limits.rs
@@ -0,0 +1,699 @@
+//! `/api/agents/{name}/limits` — the CPU and memory caps the swarm sets for
+//! one agent.
+//!
+//! Each agent's caps are one value under its own key of `agent-limits`
+//! (`swarm_queue_client::agent_limits`), which this daemon alone writes. The
+//! hive running the agent reads the key; on the hive, a cap replaces the
+//! hive's own answer for that field, and a field left unset stays with the
+//! hive.
+//!
+//! The hive hands these values to systemd verbatim, and systemd ignores a value
+//! it can't parse, leaving the agent uncapped, so they are checked here against
+//! systemd's own syntax (`Caps`) as well as on the hive.
+
+use axum::Json;
+use axum::extract::{Path, State};
+use axum::http::StatusCode;
+use problem_details::ProblemDetails;
+use serde::{Deserialize, Serialize};
+use swarm_queue_client::agent_limits::{self, AgentCaps};
+use utoipa::ToSchema;
+
+use crate::schedules::{live, queue, read, refused, write};
+use crate::{AppState, error_problem, wanted_error_status};
+
+/// How many times a losing writer re-reads and re-applies, as in
+/// `wanted::MAX_ATTEMPTS`.
+const MAX_ATTEMPTS: usize = 5;
+
+/// One agent's caps. A field left out is not capped by the swarm.
+///
+/// Each value is stored as given, and only in a form systemd applies: a value
+/// systemd ignores ("Invalid …, ignoring") would leave the agent uncapped,
+/// since a swarm cap replaces the hive's answer.
+#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
+#[serde(deny_unknown_fields)]
+pub struct Caps {
+ /// systemd `CPUQuota=`: digits, optionally `.` and one or two digits, then
+ /// `%`; above zero and at most `21474836.47%`. `"100%"` is one core.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schema(example = "400%")]
+ cpu_quota: Option,
+ /// systemd `MemoryMax=`: `infinity`; a percentage of physical memory
+ /// written as for `cpu_quota`, above zero and at most `100%`; or a size:
+ /// digits, optionally `.` and digits, then nothing, `B` or one of
+ /// `K`/`M`/`G`/`T` (1024-based, upper case), at least one byte and below
+ /// 2^64 - 1 bytes.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schema(example = "8G")]
+ memory_max: Option,
+}
+
+impl Caps {
+ /// The value to store, or why it is refused.
+ fn into_stored(self) -> Result {
+ Ok(AgentCaps {
+ cpu_quota: self.cpu_quota.as_deref().map(cpu_quota).transpose()?,
+ memory_max: self.memory_max.as_deref().map(memory_max).transpose()?,
+ })
+ }
+}
+
+impl From for Caps {
+ fn from(v: AgentCaps) -> Self {
+ Self {
+ cpu_quota: v.cpu_quota,
+ memory_max: v.memory_max,
+ }
+ }
+}
+
+// The accepted forms are a subset of what systemd parses (checked against
+// `systemd-analyze verify`, systemd 260) and stricter than hive-c0re's own
+// `validate_cpu_quota` / `validate_memory_max`, so the hive never refuses a
+// stored value.
+
+/// systemd keeps `CPUQuota=` in hundredths of a percent in a C `int`.
+const CPU_QUOTA_MAX_HUNDREDTHS: u64 = 2_147_483_647;
+
+fn cpu_quota(value: &str) -> Result {
+ match hundredths(value) {
+ Some(h) if (1..=CPU_QUOTA_MAX_HUNDREDTHS).contains(&h) => Ok(value.to_owned()),
+ _ => Err(format!(
+ "invalid cpu_quota {value:?}: expected a percentage above zero with at most two \
+ decimals, such as \"200%\" (100% = one full core), up to \"21474836.47%\""
+ )),
+ }
+}
+
+fn memory_max(value: &str) -> Result {
+ let applies = value == "infinity"
+ || if value.ends_with('%') {
+ hundredths(value).is_some_and(|h| (1..=10_000).contains(&h))
+ } else {
+ size_bytes(value).is_some_and(|b| b >= 1 && b < u128::from(u64::MAX))
+ };
+ if applies {
+ Ok(value.to_owned())
+ } else {
+ Err(format!(
+ "invalid memory_max {value:?}: expected a size above zero such as \"8G\" \
+ (suffix B, K, M, G or T, upper case), a percentage up to \"100%\" with at \
+ most two decimals, or \"infinity\""
+ ))
+ }
+}
+
+/// `value` as systemd reads a percentage, in hundredths of a percent: digits,
+/// optionally `.` and one or two digits, then `%`. `None` for anything else,
+/// and for a whole part with a leading zero other than `0` itself: systemd
+/// ignores `08%` and may read `010%` as octal.
+fn hundredths(value: &str) -> Option {
+ let number = value.strip_suffix('%')?;
+ let (int, frac) = match number.split_once('.') {
+ Some((int, frac)) if (1..=2).contains(&frac.len()) && digits(frac) => (int, frac),
+ Some(_) => return None,
+ None => (number, ""),
+ };
+ if !digits(int) || (int.len() > 1 && int.starts_with('0')) {
+ return None;
+ }
+ let frac = match frac.len() {
+ 0 => 0,
+ 1 => frac.parse::().ok()? * 10,
+ _ => frac.parse::().ok()?,
+ };
+ int.parse::().ok()?.checked_mul(100)?.checked_add(frac)
+}
+
+/// `value` as one term of systemd's `parse_size` reads it, in bytes: digits,
+/// optionally `.` and digits, then nothing, `B`, or `K`/`M`/`G`/`T` (1024-based,
+/// case-sensitive). `P` and `E`, which systemd also reads, are left out
+/// because hive-c0re's validator refuses them. `None` for anything else, and for what
+/// `parse_size` refuses as out of range: a whole part or fraction digits past
+/// `u64`, or a whole part that, plus one when there is a fraction, is past
+/// `u64` once scaled.
+fn size_bytes(value: &str) -> Option {
+ let (number, exponent) = match value.char_indices().last()? {
+ (i, 'B') => (&value[..i], 0),
+ (i, c) => match "KMGT".find(c) {
+ Some(k) => (&value[..i], k + 1),
+ None => (value, 0),
+ },
+ };
+ let scale = 1024u128.pow(u32::try_from(exponent).ok()?);
+ let (int, frac) = number.split_once('.').unwrap_or((number, ""));
+ if !digits(int) || (number.contains('.') && !digits(frac)) {
+ return None;
+ }
+ let whole = u128::from(int.parse::().ok()?);
+ let frac_value = if frac.is_empty() {
+ 0
+ } else {
+ u128::from(frac.parse::().ok()?)
+ };
+ if whole + u128::from(frac_value > 0) > u128::from(u64::MAX) / scale {
+ return None;
+ }
+ let frac_bytes = frac_value * scale / 10u128.checked_pow(u32::try_from(frac.len()).ok()?)?;
+ Some(whole * scale + frac_bytes)
+}
+
+fn digits(s: &str) -> bool {
+ !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit())
+}
+
+fn agent_name(name: &str) -> Result {
+ hive_types::Ident::parse(name)
+ .map(hive_types::Ident::into_string)
+ .map_err(|reason| (StatusCode::BAD_REQUEST, reason.to_owned()))
+}
+
+fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
+ tracing::warn!(error = %format!("{e:#}"), "agent limits: queue failure");
+ error_problem(wanted_error_status(e), &format!("{e:#}"))
+}
+
+async fn open_or_create(
+ client: &async_nats::Client,
+) -> Result {
+ agent_limits::open_or_create(client)
+ .await
+ .map_err(|e| queue_problem(&anyhow::Error::new(e)))
+}
+
+#[utoipa::path(
+ get,
+ path = "/api/agents/{name}/limits",
+ params(("name" = String, Path, description = "agent name")),
+ responses(
+ (status = 200, description = "the caps the swarm sets for the agent; `{}` when it \
+ sets none", body = Caps),
+ (status = 400, description = "not an identifier (problem+json)", body = String),
+ (status = 503, description = "no swarm queue is configured on this host, or it is \
+ not connected (problem+json)", body = String),
+ (status = 500, description = "the caps could not be read (problem+json)",
+ body = String),
+ ),
+ tag = "agents"
+)]
+pub(crate) async fn get_agent_limits(
+ State(state): State,
+ Path(name): Path,
+) -> Result, ProblemDetails> {
+ let agent = agent_name(&name).map_err(refused)?;
+ let client = queue(&state).map_err(refused)?;
+ let Some(store) = agent_limits::open(&client)
+ .await
+ .map_err(|e| queue_problem(&anyhow::Error::new(e)))?
+ else {
+ return Ok(Json(Caps::default()));
+ };
+ let (current, _) = read::(&store, &agent)
+ .await
+ .map_err(|e| queue_problem(&e))?;
+ Ok(Json(current.into()))
+}
+
+/// Set the CPU and memory caps the swarm declares for one agent.
+///
+/// The value replaces the stored one whole. Whichever hive runs the agent
+/// applies it; a running agent picks the caps up only at its next spawn or
+/// rebuild.
+#[utoipa::path(
+ put,
+ path = "/api/agents/{name}/limits",
+ params(("name" = String, Path, description = "agent name")),
+ request_body = Caps,
+ responses(
+ (status = 200, description = "the caps as stored", body = Caps),
+ (status = 400, description = "not an identifier, not declared on any hive, or a cap \
+ is not a value systemd takes or is zero (problem+json)", body = String),
+ (status = 422, description = "an unknown field"),
+ (status = 503, description = "no swarm queue is configured on this host, or it is \
+ not connected (problem+json)", body = String),
+ (status = 500, description = "the caps could not be written (problem+json)",
+ body = String),
+ ),
+ tag = "agents"
+)]
+pub(crate) async fn put_agent_limits(
+ State(state): State,
+ Path(name): Path,
+ Json(req): Json,
+) -> Result, ProblemDetails> {
+ let agent = agent_name(&name).map_err(refused)?;
+ let value = req
+ .into_stored()
+ .map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
+ let client = queue(&state).map_err(refused)?;
+ if !live(&state).await?.contains(&agent) {
+ return Err(error_problem(
+ StatusCode::BAD_REQUEST,
+ &format!("no hive's wanted state declares an agent {agent:?} that is not destroyed"),
+ ));
+ }
+ let store = open_or_create(&client).await?;
+ for _ in 0..MAX_ATTEMPTS {
+ let (current, revision) = read::(&store, &agent)
+ .await
+ .map_err(|e| queue_problem(&e))?;
+ if current == value
+ || write(&store, &agent, &value, revision)
+ .await
+ .map_err(|e| queue_problem(&e))?
+ {
+ tracing::info!(%agent, ?value, "set agent limits");
+ return Ok(Json(value.into()));
+ }
+ }
+ Err(queue_problem(&anyhow::anyhow!(
+ "gave up writing the limits of {agent} after {MAX_ATTEMPTS} conflicting writes"
+ )))
+}
+
+#[utoipa::path(
+ delete,
+ path = "/api/agents/{name}/limits",
+ params(("name" = String, Path, description = "agent name")),
+ responses(
+ (status = 204, description = "cleared; the hive's own answer applies again"),
+ (status = 400, description = "not an identifier (problem+json)", body = String),
+ (status = 503, description = "no swarm queue is configured on this host, or it is \
+ not connected (problem+json)", body = String),
+ (status = 500, description = "the caps could not be cleared (problem+json)",
+ body = String),
+ ),
+ tag = "agents"
+)]
+pub(crate) async fn delete_agent_limits(
+ State(state): State,
+ Path(name): Path,
+) -> Result {
+ let agent = agent_name(&name).map_err(refused)?;
+ let client = queue(&state).map_err(refused)?;
+ let store = open_or_create(&client).await?;
+ store
+ .delete(&agent)
+ .await
+ .map_err(|e| queue_problem(&anyhow::Error::new(e)))?;
+ tracing::info!(%agent, "cleared agent limits");
+ Ok(StatusCode::NO_CONTENT)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::{
+ Caps, cpu_quota, delete_agent_limits, get_agent_limits, memory_max, put_agent_limits,
+ };
+ use crate::test_nats::TestNats;
+ use crate::wanted::WantedWriter;
+ use axum::extract::{Path, State};
+ use axum::http::StatusCode;
+ use std::sync::Arc;
+ use swarm_queue_client::wanted::AgentState;
+
+ /// Of the `bad` values, systemd parses `""` (no quota) and `+5%`, and the
+ /// last adds a directive on a second line. It ignores every other one
+ /// ("Invalid CPU quota …, ignoring", checked with `systemd-analyze
+ /// verify`), which would leave the agent uncapped.
+ #[test]
+ fn cpu_quota_takes_a_percentage_above_zero() {
+ for ok in [
+ "400%",
+ "100%",
+ "0.5%",
+ "0.01%",
+ "1500%",
+ "400.12%",
+ "21474836.47%",
+ ] {
+ assert_eq!(cpu_quota(ok).as_deref(), Ok(ok));
+ }
+ for bad in [
+ "0%",
+ "0.0%",
+ "0.00%",
+ "-50%",
+ "400",
+ "4 00%",
+ "%",
+ "lots",
+ "",
+ "1e3%",
+ "+5%",
+ "400.123%",
+ "0.001%",
+ "99999999999999999999%",
+ "21474836.48%",
+ "05%",
+ "08%",
+ "085480%",
+ "010%",
+ "00.5%",
+ "1.%",
+ ".5%",
+ "400%\nExecStartPre=/bin/sh -c id #%",
+ ] {
+ assert!(cpu_quota(bad).is_err(), "{bad:?} was accepted");
+ }
+ }
+
+ /// As above for `MemoryMax=`. Of the `bad` values, systemd also parses `3P`
+ /// and `2E`, which hive-c0re's validator refuses, and `8 G` and `1.G`.
+ #[test]
+ fn memory_max_takes_a_size_a_share_or_infinity_that_systemd_applies() {
+ for ok in [
+ "8G",
+ "512M",
+ "4096K",
+ "1048576",
+ "1048576B",
+ "1.5B",
+ "0.5K",
+ "1.5T",
+ "1.23456789G",
+ "16383T",
+ "16777215T",
+ "18446744073709551614",
+ "18446744073709551614.5",
+ "1.00000000000000000001G",
+ "50%",
+ "12.34%",
+ "0.01%",
+ "100%",
+ "100.00%",
+ "infinity",
+ ] {
+ assert_eq!(memory_max(ok).as_deref(), Ok(ok), "{ok:?}");
+ }
+ for bad in [
+ "0",
+ "0G",
+ "0.0G",
+ "0.5",
+ "0%",
+ "-8G",
+ "150%",
+ "100.01%",
+ "12.345%",
+ "08%",
+ "085480%",
+ "010%",
+ "00.5%",
+ "8X",
+ "G",
+ "8 G",
+ "",
+ "eight",
+ "8GG",
+ "8gb",
+ "8g",
+ "8GB",
+ "4096k",
+ "1048576b",
+ "3P",
+ "2E",
+ ".5G",
+ "1.G",
+ "Infinity",
+ "17000000T",
+ "16777216T",
+ "16777215.5T",
+ "99999999999999999999T",
+ "18446744073709551615",
+ "18446744073709551616",
+ "1.123456789012345678901234567890G",
+ "8G\nDevicePolicy=auto",
+ ] {
+ assert!(memory_max(bad).is_err(), "{bad:?} was accepted");
+ }
+ }
+
+ /// One hive, `pr1ma`, on `client`.
+ fn state(client: &async_nats::Client) -> crate::AppState {
+ crate::AppState {
+ hives: Arc::new(vec![crate::HiveEntry {
+ name: "pr1ma".to_owned(),
+ domain: "pr1ma.example".to_owned(),
+ }]),
+ links: Arc::new(Vec::new()),
+ status: Some(Arc::new(crate::status::StatusReader::new(
+ client.clone(),
+ std::time::Duration::from_mins(1),
+ ))),
+ wanted: Some(Arc::new(WantedWriter::new(client.clone()))),
+ agent_status: None,
+ agent_icons: None,
+ reported_inference: None,
+ jobq: Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
+ hive_jobq::Graph::new(),
+ hive_jobq::resources::ResourceTable::new(),
+ ))),
+ webhook_secret: None,
+ config_prs: Arc::default(),
+ swarm_name: None,
+ auth: None,
+ forge: Arc::default(),
+ create_gate: Arc::default(),
+ accounts: crate::linked_accounts::Accounts::Fake(Arc::default()),
+ try_outs: crate::review_classifier::TryOuts::default(),
+ }
+ }
+
+ /// A state on a real server whose `pr1ma` declares `atlas`.
+ async fn declared(client: &async_nats::Client) -> crate::AppState {
+ WantedWriter::new(client.clone())
+ .set("pr1ma", "atlas", AgentState::Up)
+ .await
+ .expect("declare atlas");
+ state(client)
+ }
+
+ fn caps(cpu: Option<&str>, mem: Option<&str>) -> Caps {
+ Caps {
+ cpu_quota: cpu.map(str::to_owned),
+ memory_max: mem.map(str::to_owned),
+ }
+ }
+
+ async fn put(
+ state: &crate::AppState,
+ agent: &str,
+ value: Caps,
+ ) -> Result, problem_details::ProblemDetails> {
+ put_agent_limits(
+ State(state.clone()),
+ Path(agent.to_owned()),
+ axum::Json(value),
+ )
+ .await
+ }
+
+ async fn get(state: &crate::AppState, agent: &str) -> Caps {
+ get_agent_limits(State(state.clone()), Path(agent.to_owned()))
+ .await
+ .expect("caps are read")
+ .0
+ }
+
+ /// The raw value a hive reads for `agent`, or `None` for no key or a
+ /// deleted one.
+ async fn published(client: &async_nats::Client, agent: &str) -> Option {
+ let store = swarm_queue_client::agent_limits::open_or_create(client)
+ .await
+ .expect("bucket");
+ store
+ .get(agent)
+ .await
+ .expect("read")
+ .map(|v| String::from_utf8(v.to_vec()).expect("utf-8"))
+ }
+
+ /// Through the real handlers and a real `nats-server`: a cap lands under
+ /// the agent's own key as the hive reads it, `GET` answers it, and
+ /// `DELETE` takes it out again.
+ ///
+ /// The published bytes are pinned literally because hive-c0re's
+ /// `workers::swarm_limits` tests decode this same string; a change to
+ /// either side's reading of it fails one of the two.
+ #[tokio::test]
+ async fn the_route_writes_the_caps_a_hive_reads_and_delete_clears_them() {
+ let nats = TestNats::start();
+ let client = nats.client().await;
+ let state = declared(&client).await;
+
+ let answer = put(&state, "atlas", caps(Some("400%"), Some("8G")))
+ .await
+ .expect("caps are set");
+ assert_eq!(answer.0, caps(Some("400%"), Some("8G")));
+ assert_eq!(
+ published(&client, "atlas").await.as_deref(),
+ Some(r#"{"cpu_quota":"400%","memory_max":"8G"}"#)
+ );
+ assert_eq!(get(&state, "atlas").await, caps(Some("400%"), Some("8G")));
+
+ assert_eq!(
+ delete_agent_limits(State(state.clone()), Path("atlas".to_owned())).await,
+ Ok(StatusCode::NO_CONTENT)
+ );
+ assert_eq!(published(&client, "atlas").await, None);
+ assert_eq!(get(&state, "atlas").await, Caps::default());
+ }
+
+ /// A read before any cap was set answers `{}` and leaves the bucket
+ /// uncreated.
+ #[tokio::test]
+ async fn a_read_before_any_cap_answers_empty_and_creates_no_bucket() {
+ let nats = TestNats::start();
+ let client = nats.client().await;
+ let state = state(&client);
+
+ assert_eq!(get(&state, "atlas").await, Caps::default());
+ assert!(
+ swarm_queue_client::agent_limits::open(&client)
+ .await
+ .expect("the queue answers")
+ .is_none()
+ );
+ }
+
+ /// A name no hive declares gets no key, so a typo cannot cap an agent
+ /// that does not exist yet.
+ #[tokio::test]
+ async fn an_agent_no_hive_declares_is_refused_and_nothing_is_written() {
+ let nats = TestNats::start();
+ let client = nats.client().await;
+ let state = declared(&client).await;
+
+ let problem = put(&state, "iris", caps(Some("400%"), None))
+ .await
+ .expect_err("no hive declares iris");
+ assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}");
+ assert_eq!(published(&client, "iris").await, None);
+ }
+
+ /// A refused cap is answered before the queue is asked: the client here
+ /// was never connected, and would answer 503.
+ #[tokio::test]
+ async fn a_refused_cap_is_a_400_before_the_queue() {
+ let client = async_nats::ConnectOptions::new()
+ .retry_on_initial_connect()
+ .connect("127.0.0.1:1")
+ .await
+ .expect("returns without a connection");
+ let state = state(&client);
+
+ let problem = put(&state, "atlas", caps(Some("0%"), None))
+ .await
+ .expect_err("a zero cap is refused");
+ assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}");
+
+ // The control: a valid cap on the same state does reach the queue.
+ let problem = put(&state, "atlas", caps(Some("400%"), None))
+ .await
+ .expect_err("the queue is not connected");
+ assert_eq!(
+ problem.status,
+ Some(StatusCode::SERVICE_UNAVAILABLE),
+ "{problem:?}"
+ );
+ }
+
+ const CAPPED: &str = r#"{"cpu_quota":"400%","memory_max":"8G"}"#;
+
+ /// `atlas` capped, behind the real router.
+ async fn capped_router(client: &async_nats::Client) -> axum::Router {
+ let state = declared(client).await;
+ let _ = put(&state, "atlas", caps(Some("400%"), Some("8G")))
+ .await
+ .expect("cap atlas");
+ crate::build_app(state)
+ }
+
+ async fn put_raw(app: &axum::Router, body: &'static str) -> StatusCode {
+ use tower::ServiceExt as _;
+ app.clone()
+ .oneshot(
+ axum::http::Request::put("/api/agents/atlas/limits")
+ .header(axum::http::header::CONTENT_TYPE, "application/json")
+ .body(axum::body::Body::from(body))
+ .expect("request"),
+ )
+ .await
+ .expect("router answers")
+ .status()
+ }
+
+ /// A body that is not JSON is refused before the handler runs, and the
+ /// caps already set stay. The well-formed `{}` after it is the control:
+ /// the same router and request do reach the bucket.
+ #[tokio::test]
+ async fn a_malformed_body_is_a_400_and_the_caps_stay() {
+ let nats = TestNats::start();
+ let client = nats.client().await;
+ let app = capped_router(&client).await;
+
+ for body in [r#"{"cpu_quota":"#, "cpu_quota=400%", ""] {
+ assert_eq!(
+ put_raw(&app, body).await,
+ StatusCode::BAD_REQUEST,
+ "{body:?}"
+ );
+ assert_eq!(
+ published(&client, "atlas").await.as_deref(),
+ Some(CAPPED),
+ "{body:?}"
+ );
+ }
+
+ assert_eq!(put_raw(&app, "{}").await, StatusCode::OK);
+ assert_eq!(published(&client, "atlas").await.as_deref(), Some("{}"));
+ }
+
+ /// A value carrying a directive on a second line is a 400 through the real
+ /// router, and the caps already set stay.
+ #[tokio::test]
+ async fn a_cap_with_a_second_line_is_a_400_and_the_caps_stay() {
+ let nats = TestNats::start();
+ let client = nats.client().await;
+ let app = capped_router(&client).await;
+
+ for body in [
+ r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#,
+ r#"{"memory_max":"8G\nDevicePolicy=auto"}"#,
+ ] {
+ assert_eq!(put_raw(&app, body).await, StatusCode::BAD_REQUEST, "{body}");
+ assert_eq!(
+ published(&client, "atlas").await.as_deref(),
+ Some(CAPPED),
+ "{body}"
+ );
+ }
+ }
+
+ /// A field the body does not define is a 422 and writes nothing: a
+ /// misspelt `memory_max` must not read as "no memory cap".
+ #[tokio::test]
+ async fn an_unknown_field_is_a_422_and_the_caps_stay() {
+ let nats = TestNats::start();
+ let client = nats.client().await;
+ let app = capped_router(&client).await;
+
+ for body in [
+ r#"{"cpu_quota":"100%","memory":"2G"}"#,
+ r#"{"limits":{"cpu_quota":"100%"}}"#,
+ ] {
+ assert_eq!(
+ put_raw(&app, body).await,
+ StatusCode::UNPROCESSABLE_ENTITY,
+ "{body}"
+ );
+ assert_eq!(
+ published(&client, "atlas").await.as_deref(),
+ Some(CAPPED),
+ "{body}"
+ );
+ }
+ }
+}
diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs
index 63c3f03e0..cde67d31d 100644
--- a/swarm-controller/src/main.rs
+++ b/swarm-controller/src/main.rs
@@ -44,6 +44,7 @@ use utoipa_axum::{router::OpenApiRouter, routes};
mod agent_icon;
mod agent_identity;
mod agent_input;
+mod agent_limits;
mod agent_renewal;
mod agent_state_stream;
mod agent_status;
@@ -3300,6 +3301,11 @@ fn build_app(state: AppState) -> axum::Router {
inference::put_agent_inference,
inference::delete_agent_inference
))
+ .routes(routes!(
+ agent_limits::get_agent_limits,
+ agent_limits::put_agent_limits,
+ agent_limits::delete_agent_limits
+ ))
.routes(routes!(get_config_prs))
.routes(routes!(create_agent))
.routes(routes!(mint_agent_identity))
diff --git a/swarm-nats-auth/src/policy.rs b/swarm-nats-auth/src/policy.rs
index b9d6db695..e4a692772 100644
--- a/swarm-nats-auth/src/policy.rs
+++ b/swarm-nats-auth/src/policy.rs
@@ -490,6 +490,17 @@ impl Policy {
format!("$JS.API.STREAM.CREATE.{agent_status}"),
format!("$KV.{}.{hive}.*", swarm_queue_client::agent_status::BUCKET),
]);
+ // Reading the caps the swarm sets for the agents this hive runs
+ // (`swarm_queue_client::agent_limits`): a direct get, so read-only, and
+ // no `STREAM.INFO` because the reader skips the lookup.
+ //
+ // Every agent's key, not just this hive's: the bucket is keyed by agent
+ // alone, and naming one hive's agents here needs the roster
+ // `hive_name`'s doc argues against.
+ subjects.push(format!(
+ "$JS.API.DIRECT.GET.{}.>",
+ swarm_queue_client::agent_limits::STREAM
+ ));
subjects.extend(
self.extra_hive_subjects
.iter()
@@ -712,6 +723,51 @@ mod tests {
}
}
+ /// A hive reads any agent's caps and can change none of them, and the
+ /// grant reaches no other bucket.
+ #[test]
+ fn a_hive_may_read_every_agents_caps_and_write_none() {
+ use swarm_queue_client::agent_limits::{STREAM, direct_get_subject, key_subject};
+
+ let p = policy()
+ .permissions("hive-alpha")
+ .expect("a hive is admitted");
+ let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject));
+
+ // Its own agents and agents running elsewhere alike.
+ for agent in ["atlas", "iris"] {
+ assert!(may(&direct_get_subject(agent)), "cannot read {agent}");
+ }
+ // A KV put or delete is a publish to the key's subject; the rest
+ // reshape or empty the stream.
+ for forbidden in [
+ key_subject("atlas"),
+ format!("$JS.API.STREAM.CREATE.{STREAM}"),
+ format!("$JS.API.STREAM.UPDATE.{STREAM}"),
+ format!("$JS.API.STREAM.DELETE.{STREAM}"),
+ format!("$JS.API.STREAM.PURGE.{STREAM}"),
+ format!("$JS.API.STREAM.MSG.DELETE.{STREAM}"),
+ format!("$JS.API.CONSUMER.CREATE.{STREAM}"),
+ // A named consumer, bare and with a key filter, which a grant of
+ // `CONSUMER.CREATE..>` would cover.
+ format!("$JS.API.CONSUMER.CREATE.{STREAM}.c1"),
+ format!(
+ "$JS.API.CONSUMER.CREATE.{STREAM}.c1.{}",
+ key_subject("atlas")
+ ),
+ ] {
+ assert!(!may(&forbidden), "hive alpha may publish {forbidden}");
+ }
+ // The read reaches only this bucket.
+ for other in [
+ swarm_queue_client::agent_inference::direct_get_subject("atlas"),
+ swarm_queue_client::agent_schedules::direct_get_subject("atlas"),
+ "$JS.API.DIRECT.GET.KV_hive-wanted-beta.$KV.hive-wanted-beta.beta".to_owned(),
+ ] {
+ assert!(!may(&other), "hive alpha may publish {other}");
+ }
+ }
+
#[test]
fn an_extra_subject_without_the_placeholder_is_refused() {
// 🩸 The option exists to put a second stream inside ONE hive's
diff --git a/swarm-queue-client/Cargo.toml b/swarm-queue-client/Cargo.toml
index 1123f9a35..5496599d2 100644
--- a/swarm-queue-client/Cargo.toml
+++ b/swarm-queue-client/Cargo.toml
@@ -47,6 +47,9 @@ agent-inference = ["async-nats/jetstream"]
# `ToSchema` on the review classifier's shared types, for swarm-controller,
# whose HTTP API serves the same shapes the queue carries.
schema = ["dep:utoipa"]
+# `agent_limits::read`, for hive-c0re, which reads the key of each agent it
+# runs the same way. Explicit for the same reason as `notices`.
+agent-limits = ["async-nats/jetstream"]
[dependencies]
# Bare (no `kv`/`jetstream`) unless a consumer opts into the `kv` feature
diff --git a/swarm-queue-client/src/agent_limits.rs b/swarm-queue-client/src/agent_limits.rs
new file mode 100644
index 000000000..77cf3b7f0
--- /dev/null
+++ b/swarm-queue-client/src/agent_limits.rs
@@ -0,0 +1,190 @@
+//! The per-agent resource-cap KV bucket: one key per agent, holding the CPU
+//! and memory caps the swarm sets for it.
+//!
+//! **swarm-controller is the only writer**, and creates the bucket on its
+//! first write. Every field is optional: an absent field leaves the agent on
+//! its hive's own answer, so an agent with no key runs exactly as it would
+//! with no swarm.
+//!
+//! **The hive running the agent reads the key**, not the agent: the caps go
+//! into the container's systemd drop-in, which the host writes. A hive reads
+//! with one `DIRECT.GET` per agent
+//! ([`crate::agent_limits::direct_get_subject`]), and its grant covers every
+//! key of this bucket, read-only. Scoping it to the hive's own agents would
+//! need a roster in the auth responder.
+//!
+//! Keyed by the agent alone, like [`crate::agent_inference`]: an agent can
+//! move between hives, and its caps go with it.
+
+/// The KV bucket caps are published into, keyed by agent name.
+pub const BUCKET: &str = "agent-limits";
+
+/// The stream backing [`BUCKET`]. `create_key_value` prefixes `KV_`.
+pub const STREAM: &str = "KV_agent-limits";
+
+/// The subject `agent`'s entry is stored under.
+#[must_use]
+pub fn key_subject(agent: &str) -> String {
+ format!("$KV.{BUCKET}.{agent}")
+}
+
+/// The request subject that reads `agent`'s current entry.
+#[must_use]
+pub fn direct_get_subject(agent: &str) -> String {
+ format!("$JS.API.DIRECT.GET.{STREAM}.{}", key_subject(agent))
+}
+
+/// One agent's caps: the value under `BUCKET`/``, as systemd values
+/// carried verbatim. `None` is "not capped by the swarm", never "uncapped".
+#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+pub struct AgentCaps {
+ /// systemd `CPUQuota=`, e.g. `"400%"`.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub cpu_quota: Option,
+ /// systemd `MemoryMax=`, e.g. `"8G"`.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub memory_max: Option,
+}
+
+impl AgentCaps {
+ /// True when no field is set, which a hive reads the same as no key.
+ #[must_use]
+ pub fn is_empty(&self) -> bool {
+ self.cpu_quota.is_none() && self.memory_max.is_none()
+ }
+}
+
+/// Open the caps bucket for writing, creating it if nothing has yet.
+///
+/// **Controller-side only.** `history: 1`: a hive wants the current value.
+#[cfg(feature = "kv")]
+pub async fn open_or_create(
+ client: &async_nats::Client,
+) -> Result {
+ let js = async_nats::jetstream::new(client.clone());
+ crate::kv::open_or_create(
+ &js,
+ BUCKET,
+ "agent-limits",
+ "Resource caps the swarm controller sets for each agent".to_owned(),
+ )
+ .await
+}
+
+/// Open the caps bucket for reading, or `None` when nothing has created it
+/// yet, which reads the same as no caps.
+///
+/// **Controller-side**, for reads that must not create the bucket.
+///
+/// # Errors
+///
+/// [`crate::Error::OpenBucket`] for every failure other than the bucket not
+/// existing.
+#[cfg(feature = "kv")]
+pub async fn open(
+ client: &async_nats::Client,
+) -> Result, crate::Error> {
+ let js = async_nats::jetstream::new(client.clone());
+ match js.get_key_value(BUCKET).await {
+ Ok(store) => Ok(Some(store)),
+ Err(e) if crate::wanted::is_bucket_not_found(&e) => Ok(None),
+ Err(source) => Err(crate::Error::OpenBucket {
+ bucket: BUCKET.to_owned(),
+ source,
+ }),
+ }
+}
+
+/// What reading one agent's key found.
+#[cfg(feature = "agent-limits")]
+#[derive(Debug)]
+pub enum Read {
+ /// The bucket answered; an absent or deleted key has every field unset.
+ Current(AgentCaps),
+ /// Nothing answers for the bucket's stream: no cap has been written yet,
+ /// or `JetStream` is not serving it right now. Not an unset value, since
+ /// the second case says nothing about the agent's caps.
+ NoBucket,
+}
+
+/// Read `agent`'s current caps with one direct get.
+///
+/// # Errors
+///
+/// A failed request or an undecodable value. A request the queue refused for
+/// permissions reaches the client as a timeout, not as a refusal.
+#[cfg(feature = "agent-limits")]
+pub async fn read(
+ client: &async_nats::Client,
+ agent: &str,
+) -> Result> {
+ use async_nats::jetstream::stream::DirectGetErrorKind;
+
+ let js = async_nats::jetstream::new(client.clone());
+ let stream = js.get_stream_no_info(STREAM).await?;
+ let message = match stream.direct_get_last_for_subject(key_subject(agent)).await {
+ Ok(message) => message,
+ Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => {
+ return Ok(Read::Current(AgentCaps::default()));
+ }
+ Err(e) if is_no_responders(&e) => return Ok(Read::NoBucket),
+ Err(e) => return Err(e.into()),
+ };
+ // A KV delete or purge is stored as an empty marker carrying this header.
+ let removed = message
+ .headers
+ .get("KV-Operation")
+ .is_some_and(|op| matches!(op.as_str(), "DEL" | "PURGE"));
+ if removed {
+ return Ok(Read::Current(AgentCaps::default()));
+ }
+ Ok(Read::Current(serde_json::from_slice(&message.payload)?))
+}
+
+#[cfg(feature = "agent-limits")]
+fn is_no_responders(e: &async_nats::jetstream::stream::DirectGetError) -> bool {
+ matches!(
+ e.kind(),
+ async_nats::jetstream::stream::DirectGetErrorKind::ErrorResponse(status, _)
+ if status == async_nats::StatusCode::NO_RESPONDERS
+ )
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// The names the hive's grant in `swarm-nats-auth` is built from.
+ #[test]
+ fn names_match_the_queue_grant() {
+ assert_eq!(STREAM, format!("KV_{BUCKET}"));
+ assert_eq!(key_subject("iris"), "$KV.agent-limits.iris");
+ assert_eq!(
+ direct_get_subject("iris"),
+ "$JS.API.DIRECT.GET.KV_agent-limits.$KV.agent-limits.iris"
+ );
+ }
+
+ /// An unset field is left out on the wire, and `{}` is every field unset.
+ #[test]
+ fn unset_fields_are_absent() {
+ let value = AgentCaps {
+ memory_max: Some("8G".to_owned()),
+ ..AgentCaps::default()
+ };
+ assert_eq!(
+ serde_json::to_string(&value).unwrap(),
+ r#"{"memory_max":"8G"}"#
+ );
+ let empty: AgentCaps = serde_json::from_str("{}").unwrap();
+ assert!(empty.is_empty());
+ }
+
+ /// Fields a newer controller adds must not cost an older hive its caps.
+ #[test]
+ fn decoding_tolerates_unknown_fields() {
+ let decoded: AgentCaps =
+ serde_json::from_str(r#"{"cpu_quota":"400%","io_weight":"10"}"#).unwrap();
+ assert_eq!(decoded.cpu_quota.as_deref(), Some("400%"));
+ }
+}
diff --git a/swarm-queue-client/src/lib.rs b/swarm-queue-client/src/lib.rs
index fa5b991b6..6cd0da3a4 100644
--- a/swarm-queue-client/src/lib.rs
+++ b/swarm-queue-client/src/lib.rs
@@ -216,6 +216,11 @@ pub mod agent_schedules;
/// unconditional; writing needs `kv` and reading needs `agent-inference`.
pub mod agent_inference;
+/// The bucket swarm-controller sets each agent's CPU and memory caps in — one
+/// key per agent, read by the hive that runs it. The names are unconditional;
+/// writing needs `kv` and reading needs `agent-limits`.
+pub mod agent_limits;
+
#[cfg(feature = "kv")]
mod kv;
diff --git a/swarm-queue-client/src/wanted.rs b/swarm-queue-client/src/wanted.rs
index 451039c5d..16681441b 100644
--- a/swarm-queue-client/src/wanted.rs
+++ b/swarm-queue-client/src/wanted.rs
@@ -156,7 +156,7 @@ pub async fn open_read_only(
/// `get_key_value` reports a missing bucket as `GetBucket` wrapping the
/// stream lookup's `STREAM_NOT_FOUND`; every other kind is a real failure.
#[cfg(feature = "kv")]
-fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool {
+pub(crate) fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool {
use async_nats::jetstream::context::{GetStreamError, GetStreamErrorKind, KeyValueErrorKind};
e.kind() == KeyValueErrorKind::GetBucket