Compare commits
20 changed files with 14 additions and 1489 deletions
|
|
@ -411,13 +411,6 @@ Contents:
|
|||
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
|
||||
drop-in have **no** per-agent override — they're hive-wide only and
|
||||
come straight off `HiveEnv`, so this file has no field for them.
|
||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||
hive's agents, same shape as `resource-limits.json`. Written only by
|
||||
`workers::swarm_limits`, which copies each agent's key of the swarm's
|
||||
`agent-limits` bucket here every 30 seconds. A field set here replaces
|
||||
that field's answer from `resource-limits.json` and the hive-wide
|
||||
default. Every value passes the same validators as the operator's
|
||||
overrides first, and one bad value makes the worker change nothing here.
|
||||
|
||||
The root agent has the meta dir RO-mounted at `/meta/`.
|
||||
|
||||
|
|
|
|||
|
|
@ -452,24 +452,14 @@ regardless.
|
|||
rebuild, so changes take effect on the next lifecycle op without
|
||||
requiring a host rebuild.
|
||||
|
||||
The answer per agent and per field is, first match wins:
|
||||
|
||||
1. the swarm's cap for the agent (see below);
|
||||
2. the operator's per-agent override in `meta/resource-limits.json`;
|
||||
3. `agentCpuQuota` / `agentMemoryMax`.
|
||||
|
||||
A swarm cap replaces the hive's answer, so a hive can't raise an agent
|
||||
above it. Like the rest of the drop-in, a changed swarm cap reaches a
|
||||
running agent at its next spawn or rebuild.
|
||||
|
||||
The container can't read its own cap, so `write_dropins` also writes the
|
||||
effective `MemoryMax=` into the agent's runtime dir
|
||||
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
|
||||
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
|
||||
it, as that share of the host's physical memory rounded down to a page. A
|
||||
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
|
||||
the last drop-in because it can't read the overrides or the swarm caps file,
|
||||
the file carries that drop-in's `MemoryMax=`.
|
||||
the last drop-in because it can't read the overrides file, the file carries
|
||||
that drop-in's `MemoryMax=`.
|
||||
|
||||
At boot the container derives two values from a byte count:
|
||||
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
|
||||
|
|
@ -479,39 +469,6 @@ changed cap reaches them at the next container start without a rebuild.
|
|||
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
|
||||
container's journal. The container's `MemoryMax=` still applies.
|
||||
|
||||
<details><summary>Implementation: swarm caps</summary>
|
||||
|
||||
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
|
||||
agent and no hive in it, so an agent that moves to another hive keeps its
|
||||
caps. swarm-controller alone writes it:
|
||||
|
||||
| method | path | body |
|
||||
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
|
||||
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
|
||||
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
|
||||
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
|
||||
|
||||
`PUT` refuses an agent no hive's wanted state declares, unknown fields, and
|
||||
any value systemd would ignore, which would leave the agent uncapped. A
|
||||
percentage has at most two decimals, no leading zero in its whole part (it
|
||||
refuses `08%`), and is above zero, up to `100%` for
|
||||
`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an
|
||||
optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at
|
||||
least one byte and below 2^64 - 1 bytes. The controller stores each value as
|
||||
given.
|
||||
|
||||
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
||||
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
||||
value passes the same validators as the operator's overrides first. One bad
|
||||
value or a failed read leaves the file unchanged, and repeated failures raise a
|
||||
dashboard banner.
|
||||
|
||||
A hive's queue grant reads every key of the bucket
|
||||
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
|
||||
hive's own agents would need a roster in the auth responder.
|
||||
|
||||
</details>
|
||||
|
||||
The same drop-in carries `CPUWeight=` / `IOWeight=` from
|
||||
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
|
||||
setting: the quota and the memory max are **hard caps** that throttle
|
||||
|
|
|
|||
|
|
@ -25,9 +25,6 @@ grafana, the controller, authelia, …) →
|
|||
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
|
||||
- **inference settings** — the model and effort the swarm sets for an agent,
|
||||
over its own. → [`inference.md`](inference.md)
|
||||
- **resource caps** — the CPU and memory cap the swarm sets for an agent,
|
||||
over its hive's. →
|
||||
[`coordinator.md`](../scheduler/coordinator.md#container-resource-limits)
|
||||
- **review classifier** — planned, not implemented yet: checks a PR against
|
||||
its repo's own yes/no rules and posts the result as a review. →
|
||||
[`review-classifier.md`](review-classifier.md)
|
||||
|
|
|
|||
|
|
@ -53,11 +53,7 @@ strum.workspace = true
|
|||
# Offering this hive's status to the swarm (`swarm_status`). The same crate
|
||||
# the swarm controller reads it with, and `kv` for the same reason: the
|
||||
# bucket's name and creation config belong to neither end of it alone.
|
||||
swarm-queue-client = { workspace = true, features = [
|
||||
"kv",
|
||||
"notices",
|
||||
"agent-limits",
|
||||
] }
|
||||
swarm-queue-client = { workspace = true, features = ["kv", "notices"] }
|
||||
swarm-secret-client.workspace = true
|
||||
tokio.workspace = true
|
||||
tokio-stream.workspace = true
|
||||
|
|
|
|||
|
|
@ -187,86 +187,6 @@ fn set_limits_at(path: &Path, name: &str, limits: &AgentLimits) -> std::io::Resu
|
|||
super::write_map(path, ¤t)
|
||||
}
|
||||
|
||||
/// Caps the swarm controller declared for this hive's agents.
|
||||
///
|
||||
/// A separate file from the operator's overrides above, not a merge into it:
|
||||
/// the two have different writers and different lifetimes, and a swarm
|
||||
/// declaration landing in the file an operator edits would make "reset to
|
||||
/// hive defaults" destroy something the operator never set.
|
||||
const SWARM_LIMITS_FILE: &str = "swarm-resource-limits.json";
|
||||
|
||||
#[must_use]
|
||||
pub fn swarm_limits_path() -> PathBuf {
|
||||
crate::paths::meta_root().join(SWARM_LIMITS_FILE)
|
||||
}
|
||||
|
||||
/// Read the swarm-declared caps. An absent file is the empty map — no swarm
|
||||
/// opinion, so the hive's own chain decides. A file that exists but can't be
|
||||
/// read or parsed is an error, for the reason [`read`] gives: reading it as
|
||||
/// empty would silently drop a cap.
|
||||
pub fn read_swarm() -> std::io::Result<BTreeMap<String, AgentLimits>> {
|
||||
super::read_map(&swarm_limits_path())
|
||||
}
|
||||
|
||||
/// Persist the swarm-declared caps atomically.
|
||||
///
|
||||
/// # Errors
|
||||
/// The meta dir cannot be created, or the file cannot be written.
|
||||
pub fn write_swarm(map: &BTreeMap<String, AgentLimits>) -> std::io::Result<()> {
|
||||
super::write_map(&swarm_limits_path(), map)
|
||||
}
|
||||
|
||||
/// [`effective_from`] with the swarm's declaration on top.
|
||||
///
|
||||
/// A swarm cap **replaces** the hive's answer for that field rather than
|
||||
/// joining the fallback chain as another tier: a tier a hive can out-set is
|
||||
/// not a cap. Per field, so a swarm memory ceiling leaves CPU on the hive's
|
||||
/// chain.
|
||||
///
|
||||
/// The chain itself is untouched — [`effective_from`] is called exactly as
|
||||
/// before and its result is what a missing swarm value falls back to.
|
||||
#[must_use]
|
||||
pub fn effective_from_with_swarm(
|
||||
swarm: &BTreeMap<String, AgentLimits>,
|
||||
limits: &BTreeMap<String, AgentLimits>,
|
||||
name: &str,
|
||||
hive_cpu_quota: &str,
|
||||
hive_memory_max: &str,
|
||||
) -> (String, String) {
|
||||
let (cpu, mem) = effective_from(limits, name, hive_cpu_quota, hive_memory_max);
|
||||
match swarm.get(name) {
|
||||
Some(s) => (
|
||||
s.cpu_quota.clone().unwrap_or(cpu),
|
||||
s.memory_max.clone().unwrap_or(mem),
|
||||
),
|
||||
None => (cpu, mem),
|
||||
}
|
||||
}
|
||||
|
||||
/// [`effective_from_with_swarm`] for a caller resolving a single agent, from
|
||||
/// the operator overrides at `limits_path` and the swarm caps at `swarm_path`.
|
||||
///
|
||||
/// Delegates to [`effective`] rather than re-deriving the hive's answer, so
|
||||
/// there stays exactly one place the fallback chain lives. Errors as [`read`]
|
||||
/// does, for either file.
|
||||
pub fn effective_with_swarm(
|
||||
limits_path: &Path,
|
||||
swarm_path: &Path,
|
||||
name: &str,
|
||||
hive_cpu_quota: &str,
|
||||
hive_memory_max: &str,
|
||||
) -> std::io::Result<(String, String)> {
|
||||
let (cpu, mem) = effective(limits_path, name, hive_cpu_quota, hive_memory_max)?;
|
||||
let swarm: BTreeMap<String, AgentLimits> = super::read_map(swarm_path)?;
|
||||
Ok(match swarm.get(name) {
|
||||
Some(s) => (
|
||||
s.cpu_quota.clone().unwrap_or(cpu),
|
||||
s.memory_max.clone().unwrap_or(mem),
|
||||
),
|
||||
None => (cpu, mem),
|
||||
})
|
||||
}
|
||||
|
||||
/// Validate a systemd `CPUQuota=` value. Percentages only, and values
|
||||
/// above 100% are legitimate (one full core is 100%).
|
||||
///
|
||||
|
|
@ -605,52 +525,4 @@ mod tests {
|
|||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swarm_cap_replaces_the_hive_answer_rather_than_joining_the_chain() {
|
||||
// The whole point of the ruling: a tier a hive can out-set is not a
|
||||
// cap. The agent override below is the strongest hive-side value
|
||||
// there is, and the swarm still wins.
|
||||
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), Some("16G")))]);
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(Some("150%"), Some("2G")))]);
|
||||
assert_eq!(
|
||||
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
("150%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swarm_cap_is_per_field_like_every_other_tier() {
|
||||
// A swarm that caps memory and says nothing about CPU leaves CPU on
|
||||
// the hive's chain — here the per-agent override, not the hive default.
|
||||
let hive_side = BTreeMap::from([("sock".to_owned(), limits(Some("800%"), None))]);
|
||||
let swarm = BTreeMap::from([("sock".to_owned(), limits(None, Some("2G")))]);
|
||||
assert_eq!(
|
||||
effective_from_with_swarm(&swarm, &hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
("800%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_empty_swarm_map_leaves_the_existing_chain_bit_for_bit() {
|
||||
// The control that makes the two tests above mean something: the
|
||||
// wrapper must be a no-op when the swarm has said nothing, for every
|
||||
// shape the old chain distinguishes. If this drifts, the swarm tier
|
||||
// changed behaviour it was explicitly told not to change.
|
||||
let none = BTreeMap::new();
|
||||
let cases = [
|
||||
BTreeMap::new(),
|
||||
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), Some("8G")))]),
|
||||
BTreeMap::from([("sock".to_owned(), limits(Some("400%"), None))]),
|
||||
BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]),
|
||||
BTreeMap::from([("other".to_owned(), limits(Some("100%"), Some("1G")))]),
|
||||
];
|
||||
for hive_side in &cases {
|
||||
assert_eq!(
|
||||
effective_from_with_swarm(&none, hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
effective_from(hive_side, "sock", HIVE_CPU, HIVE_MEM),
|
||||
"the wrapper changed the answer with no swarm value present"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -171,14 +171,6 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
|
|||
);
|
||||
std::collections::BTreeMap::new()
|
||||
});
|
||||
let swarm_limits = crate::resource_limits::read_swarm().unwrap_or_else(|e| {
|
||||
tracing::error!(
|
||||
path = %crate::resource_limits::swarm_limits_path().display(),
|
||||
error = ?e,
|
||||
"swarm resource caps unreadable — container rows ignore them"
|
||||
);
|
||||
std::collections::BTreeMap::new()
|
||||
});
|
||||
let mut out = Vec::new();
|
||||
for c in &raw {
|
||||
let Some(logical) = c.strip_prefix(AGENT_PREFIX) else {
|
||||
|
|
@ -243,8 +235,7 @@ pub async fn build_all(hive: &crate::coordinator::HiveEnv) -> anyhow::Result<Vec
|
|||
(None, None)
|
||||
};
|
||||
let paused = Coordinator::is_paused(&logical);
|
||||
let (cpu_quota, memory_max) = crate::resource_limits::effective_from_with_swarm(
|
||||
&swarm_limits,
|
||||
let (cpu_quota, memory_max) = crate::resource_limits::effective_from(
|
||||
&limits,
|
||||
logical.as_str(),
|
||||
&hive.agent_cpu_quota,
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@ pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Re
|
|||
if let Some((cpu_quota, memory_max)) = stage_limits(
|
||||
name,
|
||||
&crate::resource_limits::resource_limits_path(),
|
||||
&crate::resource_limits::swarm_limits_path(),
|
||||
&limits_dropin_path(&container),
|
||||
&paths.agent,
|
||||
hive,
|
||||
|
|
@ -98,13 +97,12 @@ impl HostMemory {
|
|||
fn stage_limits(
|
||||
agent_name: &str,
|
||||
limits_path: &Path,
|
||||
swarm_path: &Path,
|
||||
dropin_path: &Path,
|
||||
runtime_dir: &Path,
|
||||
hive: &HiveEnv,
|
||||
host: Option<HostMemory>,
|
||||
) -> Result<Option<(String, String)>> {
|
||||
let limits = limits_to_apply(agent_name, limits_path, swarm_path, dropin_path, hive);
|
||||
let limits = limits_to_apply(agent_name, limits_path, dropin_path, hive);
|
||||
let memory_max = match &limits {
|
||||
Some((_, memory_max)) => Some(memory_max.clone()),
|
||||
None => dropin_memory_max(agent_name, dropin_path),
|
||||
|
|
@ -226,28 +224,20 @@ fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
|
|||
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
|
||||
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
|
||||
///
|
||||
/// When the overrides file at `limits_path` or the swarm caps at
|
||||
/// `swarm_path` can't be read, the error is logged and the agent still starts.
|
||||
/// The last-applied drop-in is kept, because the hive defaults can be looser
|
||||
/// than the agent's own override.
|
||||
/// When the overrides file at `limits_path` can't be read, the error is
|
||||
/// logged and the agent still starts. The last-applied drop-in is kept,
|
||||
/// because the hive defaults can be looser than the agent's own override.
|
||||
/// With no drop-in yet, the agent gets the hive defaults: without one it
|
||||
/// would run uncapped.
|
||||
fn limits_to_apply(
|
||||
agent_name: &str,
|
||||
limits_path: &Path,
|
||||
swarm_path: &Path,
|
||||
dropin_path: &Path,
|
||||
hive: &HiveEnv,
|
||||
) -> Option<(String, String)> {
|
||||
let hive_cpu = &hive.agent_cpu_quota;
|
||||
let hive_mem = &hive.agent_memory_max;
|
||||
let e = match crate::resource_limits::effective_with_swarm(
|
||||
limits_path,
|
||||
swarm_path,
|
||||
agent_name,
|
||||
hive_cpu,
|
||||
hive_mem,
|
||||
) {
|
||||
let e = match crate::resource_limits::effective(limits_path, agent_name, hive_cpu, hive_mem) {
|
||||
Ok(limits) => return Some(limits),
|
||||
Err(e) => e,
|
||||
};
|
||||
|
|
@ -848,10 +838,6 @@ mod tests {
|
|||
|
||||
const TRUNCATED_LIMITS: &str = "{\n \"sock\": { \"cpu_quota\": \"50%\", \"mem";
|
||||
|
||||
fn absent_swarm(dir: &tempfile::TempDir) -> std::path::PathBuf {
|
||||
dir.path().join("absent-swarm-resource-limits.json")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_limits_keep_an_existing_dropin() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
|
|
@ -860,13 +846,7 @@ mod tests {
|
|||
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
|
||||
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
||||
assert_eq!(
|
||||
limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&absent_swarm(&dir),
|
||||
&dropin,
|
||||
&HiveEnv::default()
|
||||
),
|
||||
limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
|
|
@ -883,47 +863,12 @@ mod tests {
|
|||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&absent_swarm(&dir),
|
||||
&dir.path().join("absent.conf"),
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
assert_eq!(applied, Some(("200%".to_owned(), "4G".to_owned())));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_swarm_caps_keep_an_existing_dropin() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let swarm = dir.path().join("swarm-resource-limits.json");
|
||||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
std::fs::write(&swarm, TRUNCATED_LIMITS).expect("seed swarm caps");
|
||||
std::fs::write(&dropin, "MemoryMax=1G\n").expect("seed drop-in");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&dir.path().join("resource-limits.json"),
|
||||
&swarm,
|
||||
&dropin,
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
assert_eq!(applied, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swarm_cap_is_applied_over_the_operator_override() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let limits = dir.path().join("resource-limits.json");
|
||||
let swarm = dir.path().join("swarm-resource-limits.json");
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
|
||||
std::fs::write(&swarm, r#"{"sock": {"memory_max": "2G"}}"#).expect("seed swarm caps");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&swarm,
|
||||
&dir.path().join("absent.conf"),
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
assert_eq!(applied, Some(("200%".to_owned(), "2G".to_owned())));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn readable_limits_are_applied_over_an_existing_dropin() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
|
|
@ -931,13 +876,7 @@ mod tests {
|
|||
let dropin = dir.path().join("hyperhive-limits.conf");
|
||||
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
|
||||
std::fs::write(&dropin, "MemoryMax=8G\n").expect("seed drop-in");
|
||||
let applied = limits_to_apply(
|
||||
"sock",
|
||||
&limits,
|
||||
&absent_swarm(&dir),
|
||||
&dropin,
|
||||
&HiveEnv::default(),
|
||||
);
|
||||
let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default());
|
||||
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
|
||||
}
|
||||
|
||||
|
|
@ -954,7 +893,6 @@ mod tests {
|
|||
let applied = stage_limits(
|
||||
"sock",
|
||||
&dir.path().join("resource-limits.json"),
|
||||
&dir.path().join("swarm-resource-limits.json"),
|
||||
&dir.path().join("hyperhive-limits.conf"),
|
||||
&runtime,
|
||||
&HiveEnv::default(),
|
||||
|
|
@ -965,28 +903,6 @@ mod tests {
|
|||
(applied.map(|(_, mem)| mem), copy)
|
||||
}
|
||||
|
||||
/// The agent's copy follows the swarm's cap, the one systemd enforces, over
|
||||
/// the operator's override: a heap sized against the override would
|
||||
/// outgrow the cgroup.
|
||||
#[test]
|
||||
fn the_agent_reads_the_swarm_cap_over_the_operator_override() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
std::fs::write(
|
||||
dir.path().join("resource-limits.json"),
|
||||
r#"{"sock": {"memory_max": "16G"}}"#,
|
||||
)
|
||||
.expect("seed limits");
|
||||
std::fs::write(
|
||||
dir.path().join("swarm-resource-limits.json"),
|
||||
r#"{"sock": {"memory_max": "2G"}}"#,
|
||||
)
|
||||
.expect("seed swarm caps");
|
||||
assert_eq!(
|
||||
stage(&dir),
|
||||
(Some("2G".to_owned()), Some("2147483648\n".to_owned()))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_agent_reads_the_cap_its_dropin_carries() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
|
|
@ -1119,7 +1035,6 @@ mod tests {
|
|||
let applied = stage_limits(
|
||||
"sock",
|
||||
&limits,
|
||||
&dir.path().join("swarm-resource-limits.json"),
|
||||
&dropin,
|
||||
&runtime,
|
||||
&HiveEnv::default(),
|
||||
|
|
|
|||
|
|
@ -426,9 +426,8 @@ async fn handle_set_resource_limits(
|
|||
let paths = Coordinator::agent_paths(name.as_str(), agent_dir)?;
|
||||
crate::lifecycle::write_dropins(name.as_str(), &hive, &paths).await?;
|
||||
|
||||
let (cpu, mem) = crate::resource_limits::effective_with_swarm(
|
||||
let (cpu, mem) = crate::resource_limits::effective(
|
||||
&crate::resource_limits::resource_limits_path(),
|
||||
&crate::resource_limits::swarm_limits_path(),
|
||||
name.as_str(),
|
||||
&hive.agent_cpu_quota,
|
||||
&hive.agent_memory_max,
|
||||
|
|
|
|||
|
|
@ -103,12 +103,6 @@ pub fn spawn(
|
|||
shutdown.clone(),
|
||||
));
|
||||
|
||||
// The swarm's resource caps, on the same connection again.
|
||||
tokio::spawn(crate::workers::swarm_limits::follow(
|
||||
client.clone(),
|
||||
shutdown.clone(),
|
||||
));
|
||||
|
||||
let mut health = SweepHealth::new("swarm_status_publish", "warn", FAILURES_BEFORE_BANNER);
|
||||
loop {
|
||||
match publish(&client, &hive).await {
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
//! Background tasks and periodic sweeps: crash/login watcher, boot-time
|
||||
//! auto-update reconcile, the agent-sockets.json writer loop, the MCP socket listener
|
||||
//! reconcile loop, knowledge-repo sync, the swarm wanted-state pull, and the
|
||||
//! swarm resource-cap read.
|
||||
//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull.
|
||||
//! Each submodule is re-exported
|
||||
//! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged.
|
||||
|
||||
|
|
@ -10,5 +9,4 @@ pub mod auto_update;
|
|||
pub mod crash_watch;
|
||||
pub mod knowledge;
|
||||
pub mod mcp_sockets;
|
||||
pub mod swarm_limits;
|
||||
pub mod wanted;
|
||||
|
|
|
|||
|
|
@ -1,207 +0,0 @@
|
|||
//! Reading the caps the swarm sets for this hive's agents.
|
||||
//!
|
||||
//! swarm-controller keeps one key per agent in the `agent-limits` bucket
|
||||
//! ([`swarm_queue_client::agent_limits`]). Every [`READ_INTERVAL`] this reads
|
||||
//! the key of each agent this hive has and stores what it finds in
|
||||
//! `swarm-resource-limits.json`, where the drop-in and the dashboard read it.
|
||||
//! A running agent gets a changed cap at its first spawn or rebuild after
|
||||
//! that read.
|
||||
//!
|
||||
//! A failed read, or a bucket that does not answer, leaves the stored caps as
|
||||
//! they are: dropping them would read exactly like the swarm clearing them.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use swarm_queue_client::agent_limits::{self, AgentCaps, Read};
|
||||
|
||||
use crate::resource_limits::AgentLimits;
|
||||
use crate::stats::sweep_health::{self, SweepHealth};
|
||||
|
||||
/// How often the caps are read.
|
||||
pub const READ_INTERVAL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// At [`READ_INTERVAL`] this is ~90 seconds of failing reads.
|
||||
const FAILURES_BEFORE_BANNER: u32 = 3;
|
||||
|
||||
/// Read the caps now and every [`READ_INTERVAL`] until shutdown.
|
||||
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
||||
let mut health = SweepHealth::new("swarm_limits_read", "warn", FAILURES_BEFORE_BANNER);
|
||||
loop {
|
||||
match sync(&client).await {
|
||||
Ok(()) => health.record_ok(),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "swarm limits: reading the caps failed");
|
||||
let err = format!("{e:#}");
|
||||
health.record_err(|ctx| {
|
||||
let age = ctx.since_last_ok.map_or_else(
|
||||
|| "no success this session".to_owned(),
|
||||
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
|
||||
);
|
||||
format!(
|
||||
"reading the swarm's resource caps is failing ({} consecutive, {age}) \
|
||||
— agents keep the caps last read: {err}",
|
||||
ctx.consecutive
|
||||
)
|
||||
});
|
||||
}
|
||||
}
|
||||
tokio::select! {
|
||||
() = tokio::time::sleep(READ_INTERVAL) => {}
|
||||
_ = shutdown.changed() => {
|
||||
tracing::info!("swarm limits: shutdown signal received");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Read every agent's key and store the result, unless a read fails or a
|
||||
/// value is refused.
|
||||
async fn sync(client: &async_nats::Client) -> Result<()> {
|
||||
// An unconnected client does not fail a JetStream request, it hangs on it.
|
||||
swarm_queue_client::ensure_connected(client)?;
|
||||
let agents = crate::lifecycle::agents_for_meta_listing()
|
||||
.await
|
||||
.context("enumerating this hive's agents")?;
|
||||
let mut declared = BTreeMap::new();
|
||||
for spec in agents {
|
||||
let read = agent_limits::read(client, &spec.name)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
.with_context(|| format!("reading the swarm's caps for {}", spec.name))?;
|
||||
match read {
|
||||
Read::Current(caps) => {
|
||||
declared.insert(spec.name, caps);
|
||||
}
|
||||
Read::NoBucket => {
|
||||
tracing::debug!("swarm limits: the agent-limits bucket does not answer");
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
}
|
||||
let caps = validated(&declared)?;
|
||||
if caps != crate::resource_limits::read_swarm().context("reading the stored swarm caps")? {
|
||||
crate::resource_limits::write_swarm(&caps).context("storing the swarm caps")?;
|
||||
tracing::info!(?caps, "swarm limits: stored the swarm's resource caps");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The caps to store for `declared`, by agent. An agent with no field set
|
||||
/// gets no entry.
|
||||
///
|
||||
/// Every value goes through the validators the operator's own input path
|
||||
/// uses, because it is rendered verbatim into the container's systemd
|
||||
/// drop-in. One bad value refuses the whole set.
|
||||
fn validated(declared: &BTreeMap<String, AgentCaps>) -> Result<BTreeMap<String, AgentLimits>> {
|
||||
let mut out = BTreeMap::new();
|
||||
for (agent, caps) in declared {
|
||||
if caps.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let limits = validated_caps(caps)
|
||||
.map_err(|why| anyhow::anyhow!("the swarm set an invalid cap for {agent}: {why}"))?;
|
||||
out.insert(agent.clone(), limits);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn validated_caps(caps: &AgentCaps) -> Result<AgentLimits, String> {
|
||||
if let Some(cpu) = &caps.cpu_quota {
|
||||
crate::resource_limits::validate_cpu_quota(cpu)?;
|
||||
}
|
||||
Ok(AgentLimits {
|
||||
cpu_quota: caps.cpu_quota.clone(),
|
||||
memory_max: caps
|
||||
.memory_max
|
||||
.as_deref()
|
||||
.map(crate::resource_limits::validate_memory_max)
|
||||
.transpose()?,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use super::validated;
|
||||
use crate::resource_limits::AgentLimits;
|
||||
use swarm_queue_client::agent_limits::AgentCaps;
|
||||
|
||||
/// `atlas`'s answer with `atlas_key` as the value read for it, over a hive
|
||||
/// chain that caps it at 800% / 16G.
|
||||
fn atlas_with(atlas_key: &str) -> (String, String) {
|
||||
let declared = BTreeMap::from([(
|
||||
"atlas".to_owned(),
|
||||
serde_json::from_str::<AgentCaps>(atlas_key).expect("decodes"),
|
||||
)]);
|
||||
let stored = validated(&declared).expect("valid caps");
|
||||
let hive_side = BTreeMap::from([(
|
||||
"atlas".to_owned(),
|
||||
AgentLimits {
|
||||
cpu_quota: Some("800%".to_owned()),
|
||||
memory_max: Some("16G".to_owned()),
|
||||
},
|
||||
)]);
|
||||
crate::resource_limits::effective_from_with_swarm(
|
||||
&stored, &hive_side, "atlas", "200%", "4G",
|
||||
)
|
||||
}
|
||||
|
||||
/// The other half of swarm-controller's
|
||||
/// `agent_limits::the_route_writes_the_caps_a_hive_reads_and_delete_clears_them`:
|
||||
/// the first value is the bytes that route stores. The cap replaces the
|
||||
/// hive's chain; a deleted key reads as `{}` and gives the chain back.
|
||||
#[test]
|
||||
fn the_caps_the_route_stores_replace_the_hive_chain_and_a_cleared_key_restores_it() {
|
||||
assert_eq!(
|
||||
atlas_with(r#"{"cpu_quota":"400%","memory_max":"8G"}"#),
|
||||
("400%".to_owned(), "8G".to_owned())
|
||||
);
|
||||
assert_eq!(atlas_with("{}"), ("800%".to_owned(), "16G".to_owned()));
|
||||
}
|
||||
|
||||
/// Per field: a swarm memory cap leaves CPU on the hive's chain.
|
||||
#[test]
|
||||
fn a_field_the_swarm_leaves_unset_stays_with_the_hive() {
|
||||
assert_eq!(
|
||||
atlas_with(r#"{"memory_max":"2G"}"#),
|
||||
("800%".to_owned(), "2G".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_agent_with_no_caps_gets_no_entry() {
|
||||
let declared = BTreeMap::from([("atlas".to_owned(), AgentCaps::default())]);
|
||||
assert!(validated(&declared).expect("valid").is_empty());
|
||||
}
|
||||
|
||||
/// The values are rendered verbatim into a systemd drop-in, so a value
|
||||
/// the hive's own validators refuse, a newline smuggling in a directive
|
||||
/// among them, refuses the whole set, the valid cap beside it included.
|
||||
#[test]
|
||||
fn an_invalid_cap_is_refused_and_nothing_is_stored() {
|
||||
for bad in [
|
||||
r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#,
|
||||
r#"{"memory_max":"8G\nDevicePolicy=auto"}"#,
|
||||
r#"{"cpu_quota":"lots"}"#,
|
||||
r#"{"memory_max":"8X"}"#,
|
||||
] {
|
||||
let declared = BTreeMap::from([
|
||||
("atlas".to_owned(), AgentCaps::default()),
|
||||
(
|
||||
"iris".to_owned(),
|
||||
serde_json::from_str(r#"{"memory_max":"2G"}"#).expect("decodes"),
|
||||
),
|
||||
(
|
||||
"sock".to_owned(),
|
||||
serde_json::from_str(bad).expect("decodes"),
|
||||
),
|
||||
]);
|
||||
let err = validated(&declared).expect_err(bad);
|
||||
assert!(err.to_string().contains("sock"), "{err:#}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -286,22 +286,6 @@ let
|
|||
&& !(lib.hasInfix "mint swarmctl" s)
|
||||
&& lib.hasInfix "mint swarm-authelia-bridge" s;
|
||||
}
|
||||
{
|
||||
# swarm-controller checks no caller itself: this location's authelia
|
||||
# subrequest is the operator gate for every route under `/api/`. The
|
||||
# webhook location is the control, the one path to the same socket that
|
||||
# skips the gate, so the first clause reads a string that can lack it.
|
||||
name = "the swarm UI's API location asks authelia before reaching the controller";
|
||||
ok =
|
||||
let
|
||||
l = allLocal.services.nginx.virtualHosts."t.local".locations;
|
||||
socket = allLocal.services.hyperhive.deploy.swarm-controller.socketPath;
|
||||
in
|
||||
lib.hasInfix "auth_request /__hive_authelia;" l."/api/".extraConfig
|
||||
&& lib.hasInfix socket l."/api/".proxyPass
|
||||
&& lib.hasInfix socket l."/webhook/forge/".proxyPass
|
||||
&& !(lib.hasInfix "auth_request" l."/webhook/forge/".extraConfig);
|
||||
}
|
||||
{
|
||||
name = "a hive with matrix off serves no matrix discovery endpoint";
|
||||
ok =
|
||||
|
|
|
|||
|
|
@ -52,11 +52,6 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|||
agent applies them itself. `GET` also answers the model and effort the
|
||||
agent last reported in its turn-state header, kept in memory →
|
||||
[`docs/swarm/inference.md`](../docs/swarm/inference.md).
|
||||
- **Resource caps** — each agent's CPU and memory cap, one key per agent in
|
||||
the `agent-limits` KV bucket, which this daemon alone writes
|
||||
(`GET`/`PUT`/`DELETE /api/agents/{name}/limits`); the hive running the
|
||||
agent reads it, and each cap replaces the hive's own answer for that field →
|
||||
[`docs/scheduler/coordinator.md`](../docs/scheduler/coordinator.md#container-resource-limits).
|
||||
- **Relays** — each agent's terminal and turn-state header as SSE, agent
|
||||
icons, the cross-repo issue report, and the UI's quick links.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,699 +0,0 @@
|
|||
//! `/api/agents/{name}/limits` — the CPU and memory caps the swarm sets for
|
||||
//! one agent.
|
||||
//!
|
||||
//! Each agent's caps are one value under its own key of `agent-limits`
|
||||
//! (`swarm_queue_client::agent_limits`), which this daemon alone writes. The
|
||||
//! hive running the agent reads the key; on the hive, a cap replaces the
|
||||
//! hive's own answer for that field, and a field left unset stays with the
|
||||
//! hive.
|
||||
//!
|
||||
//! The hive hands these values to systemd verbatim, and systemd ignores a value
|
||||
//! it can't parse, leaving the agent uncapped, so they are checked here against
|
||||
//! systemd's own syntax (`Caps`) as well as on the hive.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::StatusCode;
|
||||
use problem_details::ProblemDetails;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use swarm_queue_client::agent_limits::{self, AgentCaps};
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use crate::schedules::{live, queue, read, refused, write};
|
||||
use crate::{AppState, error_problem, wanted_error_status};
|
||||
|
||||
/// How many times a losing writer re-reads and re-applies, as in
|
||||
/// `wanted::MAX_ATTEMPTS`.
|
||||
const MAX_ATTEMPTS: usize = 5;
|
||||
|
||||
/// One agent's caps. A field left out is not capped by the swarm.
|
||||
///
|
||||
/// Each value is stored as given, and only in a form systemd applies: a value
|
||||
/// systemd ignores ("Invalid …, ignoring") would leave the agent uncapped,
|
||||
/// since a swarm cap replaces the hive's answer.
|
||||
#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Caps {
|
||||
/// systemd `CPUQuota=`: digits, optionally `.` and one or two digits, then
|
||||
/// `%`; above zero and at most `21474836.47%`. `"100%"` is one core.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
#[schema(example = "400%")]
|
||||
cpu_quota: Option<String>,
|
||||
/// systemd `MemoryMax=`: `infinity`; a percentage of physical memory
|
||||
/// written as for `cpu_quota`, above zero and at most `100%`; or a size:
|
||||
/// digits, optionally `.` and digits, then nothing, `B` or one of
|
||||
/// `K`/`M`/`G`/`T` (1024-based, upper case), at least one byte and below
|
||||
/// 2^64 - 1 bytes.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
#[schema(example = "8G")]
|
||||
memory_max: Option<String>,
|
||||
}
|
||||
|
||||
impl Caps {
|
||||
/// The value to store, or why it is refused.
|
||||
fn into_stored(self) -> Result<AgentCaps, String> {
|
||||
Ok(AgentCaps {
|
||||
cpu_quota: self.cpu_quota.as_deref().map(cpu_quota).transpose()?,
|
||||
memory_max: self.memory_max.as_deref().map(memory_max).transpose()?,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl From<AgentCaps> for Caps {
|
||||
fn from(v: AgentCaps) -> Self {
|
||||
Self {
|
||||
cpu_quota: v.cpu_quota,
|
||||
memory_max: v.memory_max,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The accepted forms are a subset of what systemd parses (checked against
|
||||
// `systemd-analyze verify`, systemd 260) and stricter than hive-c0re's own
|
||||
// `validate_cpu_quota` / `validate_memory_max`, so the hive never refuses a
|
||||
// stored value.
|
||||
|
||||
/// systemd keeps `CPUQuota=` in hundredths of a percent in a C `int`.
|
||||
const CPU_QUOTA_MAX_HUNDREDTHS: u64 = 2_147_483_647;
|
||||
|
||||
fn cpu_quota(value: &str) -> Result<String, String> {
|
||||
match hundredths(value) {
|
||||
Some(h) if (1..=CPU_QUOTA_MAX_HUNDREDTHS).contains(&h) => Ok(value.to_owned()),
|
||||
_ => Err(format!(
|
||||
"invalid cpu_quota {value:?}: expected a percentage above zero with at most two \
|
||||
decimals, such as \"200%\" (100% = one full core), up to \"21474836.47%\""
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn memory_max(value: &str) -> Result<String, String> {
|
||||
let applies = value == "infinity"
|
||||
|| if value.ends_with('%') {
|
||||
hundredths(value).is_some_and(|h| (1..=10_000).contains(&h))
|
||||
} else {
|
||||
size_bytes(value).is_some_and(|b| b >= 1 && b < u128::from(u64::MAX))
|
||||
};
|
||||
if applies {
|
||||
Ok(value.to_owned())
|
||||
} else {
|
||||
Err(format!(
|
||||
"invalid memory_max {value:?}: expected a size above zero such as \"8G\" \
|
||||
(suffix B, K, M, G or T, upper case), a percentage up to \"100%\" with at \
|
||||
most two decimals, or \"infinity\""
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
/// `value` as systemd reads a percentage, in hundredths of a percent: digits,
|
||||
/// optionally `.` and one or two digits, then `%`. `None` for anything else,
|
||||
/// and for a whole part with a leading zero other than `0` itself: systemd
|
||||
/// ignores `08%` and may read `010%` as octal.
|
||||
fn hundredths(value: &str) -> Option<u64> {
|
||||
let number = value.strip_suffix('%')?;
|
||||
let (int, frac) = match number.split_once('.') {
|
||||
Some((int, frac)) if (1..=2).contains(&frac.len()) && digits(frac) => (int, frac),
|
||||
Some(_) => return None,
|
||||
None => (number, ""),
|
||||
};
|
||||
if !digits(int) || (int.len() > 1 && int.starts_with('0')) {
|
||||
return None;
|
||||
}
|
||||
let frac = match frac.len() {
|
||||
0 => 0,
|
||||
1 => frac.parse::<u64>().ok()? * 10,
|
||||
_ => frac.parse::<u64>().ok()?,
|
||||
};
|
||||
int.parse::<u64>().ok()?.checked_mul(100)?.checked_add(frac)
|
||||
}
|
||||
|
||||
/// `value` as one term of systemd's `parse_size` reads it, in bytes: digits,
|
||||
/// optionally `.` and digits, then nothing, `B`, or `K`/`M`/`G`/`T` (1024-based,
|
||||
/// case-sensitive). `P` and `E`, which systemd also reads, are left out
|
||||
/// because hive-c0re's validator refuses them. `None` for anything else, and for what
|
||||
/// `parse_size` refuses as out of range: a whole part or fraction digits past
|
||||
/// `u64`, or a whole part that, plus one when there is a fraction, is past
|
||||
/// `u64` once scaled.
|
||||
fn size_bytes(value: &str) -> Option<u128> {
|
||||
let (number, exponent) = match value.char_indices().last()? {
|
||||
(i, 'B') => (&value[..i], 0),
|
||||
(i, c) => match "KMGT".find(c) {
|
||||
Some(k) => (&value[..i], k + 1),
|
||||
None => (value, 0),
|
||||
},
|
||||
};
|
||||
let scale = 1024u128.pow(u32::try_from(exponent).ok()?);
|
||||
let (int, frac) = number.split_once('.').unwrap_or((number, ""));
|
||||
if !digits(int) || (number.contains('.') && !digits(frac)) {
|
||||
return None;
|
||||
}
|
||||
let whole = u128::from(int.parse::<u64>().ok()?);
|
||||
let frac_value = if frac.is_empty() {
|
||||
0
|
||||
} else {
|
||||
u128::from(frac.parse::<u64>().ok()?)
|
||||
};
|
||||
if whole + u128::from(frac_value > 0) > u128::from(u64::MAX) / scale {
|
||||
return None;
|
||||
}
|
||||
let frac_bytes = frac_value * scale / 10u128.checked_pow(u32::try_from(frac.len()).ok()?)?;
|
||||
Some(whole * scale + frac_bytes)
|
||||
}
|
||||
|
||||
fn digits(s: &str) -> bool {
|
||||
!s.is_empty() && s.bytes().all(|b| b.is_ascii_digit())
|
||||
}
|
||||
|
||||
fn agent_name(name: &str) -> Result<String, (StatusCode, String)> {
|
||||
hive_types::Ident::parse(name)
|
||||
.map(hive_types::Ident::into_string)
|
||||
.map_err(|reason| (StatusCode::BAD_REQUEST, reason.to_owned()))
|
||||
}
|
||||
|
||||
fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
|
||||
tracing::warn!(error = %format!("{e:#}"), "agent limits: queue failure");
|
||||
error_problem(wanted_error_status(e), &format!("{e:#}"))
|
||||
}
|
||||
|
||||
async fn open_or_create(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<async_nats::jetstream::kv::Store, ProblemDetails> {
|
||||
agent_limits::open_or_create(client)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&anyhow::Error::new(e)))
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/agents/{name}/limits",
|
||||
params(("name" = String, Path, description = "agent name")),
|
||||
responses(
|
||||
(status = 200, description = "the caps the swarm sets for the agent; `{}` when it \
|
||||
sets none", body = Caps),
|
||||
(status = 400, description = "not an identifier (problem+json)", body = String),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
(status = 500, description = "the caps could not be read (problem+json)",
|
||||
body = String),
|
||||
),
|
||||
tag = "agents"
|
||||
)]
|
||||
pub(crate) async fn get_agent_limits(
|
||||
State(state): State<AppState>,
|
||||
Path(name): Path<String>,
|
||||
) -> Result<Json<Caps>, ProblemDetails> {
|
||||
let agent = agent_name(&name).map_err(refused)?;
|
||||
let client = queue(&state).map_err(refused)?;
|
||||
let Some(store) = agent_limits::open(&client)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&anyhow::Error::new(e)))?
|
||||
else {
|
||||
return Ok(Json(Caps::default()));
|
||||
};
|
||||
let (current, _) = read::<AgentCaps>(&store, &agent)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&e))?;
|
||||
Ok(Json(current.into()))
|
||||
}
|
||||
|
||||
/// Set the CPU and memory caps the swarm declares for one agent.
|
||||
///
|
||||
/// The value replaces the stored one whole. Whichever hive runs the agent
|
||||
/// applies it; a running agent picks the caps up only at its next spawn or
|
||||
/// rebuild.
|
||||
#[utoipa::path(
|
||||
put,
|
||||
path = "/api/agents/{name}/limits",
|
||||
params(("name" = String, Path, description = "agent name")),
|
||||
request_body = Caps,
|
||||
responses(
|
||||
(status = 200, description = "the caps as stored", body = Caps),
|
||||
(status = 400, description = "not an identifier, not declared on any hive, or a cap \
|
||||
is not a value systemd takes or is zero (problem+json)", body = String),
|
||||
(status = 422, description = "an unknown field"),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
(status = 500, description = "the caps could not be written (problem+json)",
|
||||
body = String),
|
||||
),
|
||||
tag = "agents"
|
||||
)]
|
||||
pub(crate) async fn put_agent_limits(
|
||||
State(state): State<AppState>,
|
||||
Path(name): Path<String>,
|
||||
Json(req): Json<Caps>,
|
||||
) -> Result<Json<Caps>, ProblemDetails> {
|
||||
let agent = agent_name(&name).map_err(refused)?;
|
||||
let value = req
|
||||
.into_stored()
|
||||
.map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
|
||||
let client = queue(&state).map_err(refused)?;
|
||||
if !live(&state).await?.contains(&agent) {
|
||||
return Err(error_problem(
|
||||
StatusCode::BAD_REQUEST,
|
||||
&format!("no hive's wanted state declares an agent {agent:?} that is not destroyed"),
|
||||
));
|
||||
}
|
||||
let store = open_or_create(&client).await?;
|
||||
for _ in 0..MAX_ATTEMPTS {
|
||||
let (current, revision) = read::<AgentCaps>(&store, &agent)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&e))?;
|
||||
if current == value
|
||||
|| write(&store, &agent, &value, revision)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&e))?
|
||||
{
|
||||
tracing::info!(%agent, ?value, "set agent limits");
|
||||
return Ok(Json(value.into()));
|
||||
}
|
||||
}
|
||||
Err(queue_problem(&anyhow::anyhow!(
|
||||
"gave up writing the limits of {agent} after {MAX_ATTEMPTS} conflicting writes"
|
||||
)))
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
delete,
|
||||
path = "/api/agents/{name}/limits",
|
||||
params(("name" = String, Path, description = "agent name")),
|
||||
responses(
|
||||
(status = 204, description = "cleared; the hive's own answer applies again"),
|
||||
(status = 400, description = "not an identifier (problem+json)", body = String),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
(status = 500, description = "the caps could not be cleared (problem+json)",
|
||||
body = String),
|
||||
),
|
||||
tag = "agents"
|
||||
)]
|
||||
pub(crate) async fn delete_agent_limits(
|
||||
State(state): State<AppState>,
|
||||
Path(name): Path<String>,
|
||||
) -> Result<StatusCode, ProblemDetails> {
|
||||
let agent = agent_name(&name).map_err(refused)?;
|
||||
let client = queue(&state).map_err(refused)?;
|
||||
let store = open_or_create(&client).await?;
|
||||
store
|
||||
.delete(&agent)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&anyhow::Error::new(e)))?;
|
||||
tracing::info!(%agent, "cleared agent limits");
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
Caps, cpu_quota, delete_agent_limits, get_agent_limits, memory_max, put_agent_limits,
|
||||
};
|
||||
use crate::test_nats::TestNats;
|
||||
use crate::wanted::WantedWriter;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::StatusCode;
|
||||
use std::sync::Arc;
|
||||
use swarm_queue_client::wanted::AgentState;
|
||||
|
||||
/// Of the `bad` values, systemd parses `""` (no quota) and `+5%`, and the
|
||||
/// last adds a directive on a second line. It ignores every other one
|
||||
/// ("Invalid CPU quota …, ignoring", checked with `systemd-analyze
|
||||
/// verify`), which would leave the agent uncapped.
|
||||
#[test]
|
||||
fn cpu_quota_takes_a_percentage_above_zero() {
|
||||
for ok in [
|
||||
"400%",
|
||||
"100%",
|
||||
"0.5%",
|
||||
"0.01%",
|
||||
"1500%",
|
||||
"400.12%",
|
||||
"21474836.47%",
|
||||
] {
|
||||
assert_eq!(cpu_quota(ok).as_deref(), Ok(ok));
|
||||
}
|
||||
for bad in [
|
||||
"0%",
|
||||
"0.0%",
|
||||
"0.00%",
|
||||
"-50%",
|
||||
"400",
|
||||
"4 00%",
|
||||
"%",
|
||||
"lots",
|
||||
"",
|
||||
"1e3%",
|
||||
"+5%",
|
||||
"400.123%",
|
||||
"0.001%",
|
||||
"99999999999999999999%",
|
||||
"21474836.48%",
|
||||
"05%",
|
||||
"08%",
|
||||
"085480%",
|
||||
"010%",
|
||||
"00.5%",
|
||||
"1.%",
|
||||
".5%",
|
||||
"400%\nExecStartPre=/bin/sh -c id #%",
|
||||
] {
|
||||
assert!(cpu_quota(bad).is_err(), "{bad:?} was accepted");
|
||||
}
|
||||
}
|
||||
|
||||
/// As above for `MemoryMax=`. Of the `bad` values, systemd also parses `3P`
|
||||
/// and `2E`, which hive-c0re's validator refuses, and `8 G` and `1.G`.
|
||||
#[test]
|
||||
fn memory_max_takes_a_size_a_share_or_infinity_that_systemd_applies() {
|
||||
for ok in [
|
||||
"8G",
|
||||
"512M",
|
||||
"4096K",
|
||||
"1048576",
|
||||
"1048576B",
|
||||
"1.5B",
|
||||
"0.5K",
|
||||
"1.5T",
|
||||
"1.23456789G",
|
||||
"16383T",
|
||||
"16777215T",
|
||||
"18446744073709551614",
|
||||
"18446744073709551614.5",
|
||||
"1.00000000000000000001G",
|
||||
"50%",
|
||||
"12.34%",
|
||||
"0.01%",
|
||||
"100%",
|
||||
"100.00%",
|
||||
"infinity",
|
||||
] {
|
||||
assert_eq!(memory_max(ok).as_deref(), Ok(ok), "{ok:?}");
|
||||
}
|
||||
for bad in [
|
||||
"0",
|
||||
"0G",
|
||||
"0.0G",
|
||||
"0.5",
|
||||
"0%",
|
||||
"-8G",
|
||||
"150%",
|
||||
"100.01%",
|
||||
"12.345%",
|
||||
"08%",
|
||||
"085480%",
|
||||
"010%",
|
||||
"00.5%",
|
||||
"8X",
|
||||
"G",
|
||||
"8 G",
|
||||
"",
|
||||
"eight",
|
||||
"8GG",
|
||||
"8gb",
|
||||
"8g",
|
||||
"8GB",
|
||||
"4096k",
|
||||
"1048576b",
|
||||
"3P",
|
||||
"2E",
|
||||
".5G",
|
||||
"1.G",
|
||||
"Infinity",
|
||||
"17000000T",
|
||||
"16777216T",
|
||||
"16777215.5T",
|
||||
"99999999999999999999T",
|
||||
"18446744073709551615",
|
||||
"18446744073709551616",
|
||||
"1.123456789012345678901234567890G",
|
||||
"8G\nDevicePolicy=auto",
|
||||
] {
|
||||
assert!(memory_max(bad).is_err(), "{bad:?} was accepted");
|
||||
}
|
||||
}
|
||||
|
||||
/// One hive, `pr1ma`, on `client`.
|
||||
fn state(client: &async_nats::Client) -> crate::AppState {
|
||||
crate::AppState {
|
||||
hives: Arc::new(vec![crate::HiveEntry {
|
||||
name: "pr1ma".to_owned(),
|
||||
domain: "pr1ma.example".to_owned(),
|
||||
}]),
|
||||
links: Arc::new(Vec::new()),
|
||||
status: Some(Arc::new(crate::status::StatusReader::new(
|
||||
client.clone(),
|
||||
std::time::Duration::from_mins(1),
|
||||
))),
|
||||
wanted: Some(Arc::new(WantedWriter::new(client.clone()))),
|
||||
agent_status: None,
|
||||
agent_icons: None,
|
||||
reported_inference: None,
|
||||
jobq: Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
|
||||
hive_jobq::Graph::new(),
|
||||
hive_jobq::resources::ResourceTable::new(),
|
||||
))),
|
||||
webhook_secret: None,
|
||||
config_prs: Arc::default(),
|
||||
swarm_name: None,
|
||||
auth: None,
|
||||
forge: Arc::default(),
|
||||
create_gate: Arc::default(),
|
||||
accounts: crate::linked_accounts::Accounts::Fake(Arc::default()),
|
||||
try_outs: crate::review_classifier::TryOuts::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// A state on a real server whose `pr1ma` declares `atlas`.
|
||||
async fn declared(client: &async_nats::Client) -> crate::AppState {
|
||||
WantedWriter::new(client.clone())
|
||||
.set("pr1ma", "atlas", AgentState::Up)
|
||||
.await
|
||||
.expect("declare atlas");
|
||||
state(client)
|
||||
}
|
||||
|
||||
fn caps(cpu: Option<&str>, mem: Option<&str>) -> Caps {
|
||||
Caps {
|
||||
cpu_quota: cpu.map(str::to_owned),
|
||||
memory_max: mem.map(str::to_owned),
|
||||
}
|
||||
}
|
||||
|
||||
async fn put(
|
||||
state: &crate::AppState,
|
||||
agent: &str,
|
||||
value: Caps,
|
||||
) -> Result<axum::Json<Caps>, problem_details::ProblemDetails> {
|
||||
put_agent_limits(
|
||||
State(state.clone()),
|
||||
Path(agent.to_owned()),
|
||||
axum::Json(value),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn get(state: &crate::AppState, agent: &str) -> Caps {
|
||||
get_agent_limits(State(state.clone()), Path(agent.to_owned()))
|
||||
.await
|
||||
.expect("caps are read")
|
||||
.0
|
||||
}
|
||||
|
||||
/// The raw value a hive reads for `agent`, or `None` for no key or a
|
||||
/// deleted one.
|
||||
async fn published(client: &async_nats::Client, agent: &str) -> Option<String> {
|
||||
let store = swarm_queue_client::agent_limits::open_or_create(client)
|
||||
.await
|
||||
.expect("bucket");
|
||||
store
|
||||
.get(agent)
|
||||
.await
|
||||
.expect("read")
|
||||
.map(|v| String::from_utf8(v.to_vec()).expect("utf-8"))
|
||||
}
|
||||
|
||||
/// Through the real handlers and a real `nats-server`: a cap lands under
|
||||
/// the agent's own key as the hive reads it, `GET` answers it, and
|
||||
/// `DELETE` takes it out again.
|
||||
///
|
||||
/// The published bytes are pinned literally because hive-c0re's
|
||||
/// `workers::swarm_limits` tests decode this same string; a change to
|
||||
/// either side's reading of it fails one of the two.
|
||||
#[tokio::test]
|
||||
async fn the_route_writes_the_caps_a_hive_reads_and_delete_clears_them() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
|
||||
let answer = put(&state, "atlas", caps(Some("400%"), Some("8G")))
|
||||
.await
|
||||
.expect("caps are set");
|
||||
assert_eq!(answer.0, caps(Some("400%"), Some("8G")));
|
||||
assert_eq!(
|
||||
published(&client, "atlas").await.as_deref(),
|
||||
Some(r#"{"cpu_quota":"400%","memory_max":"8G"}"#)
|
||||
);
|
||||
assert_eq!(get(&state, "atlas").await, caps(Some("400%"), Some("8G")));
|
||||
|
||||
assert_eq!(
|
||||
delete_agent_limits(State(state.clone()), Path("atlas".to_owned())).await,
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
);
|
||||
assert_eq!(published(&client, "atlas").await, None);
|
||||
assert_eq!(get(&state, "atlas").await, Caps::default());
|
||||
}
|
||||
|
||||
/// A read before any cap was set answers `{}` and leaves the bucket
|
||||
/// uncreated.
|
||||
#[tokio::test]
|
||||
async fn a_read_before_any_cap_answers_empty_and_creates_no_bucket() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = state(&client);
|
||||
|
||||
assert_eq!(get(&state, "atlas").await, Caps::default());
|
||||
assert!(
|
||||
swarm_queue_client::agent_limits::open(&client)
|
||||
.await
|
||||
.expect("the queue answers")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
/// A name no hive declares gets no key, so a typo cannot cap an agent
|
||||
/// that does not exist yet.
|
||||
#[tokio::test]
|
||||
async fn an_agent_no_hive_declares_is_refused_and_nothing_is_written() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
|
||||
let problem = put(&state, "iris", caps(Some("400%"), None))
|
||||
.await
|
||||
.expect_err("no hive declares iris");
|
||||
assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}");
|
||||
assert_eq!(published(&client, "iris").await, None);
|
||||
}
|
||||
|
||||
/// A refused cap is answered before the queue is asked: the client here
|
||||
/// was never connected, and would answer 503.
|
||||
#[tokio::test]
|
||||
async fn a_refused_cap_is_a_400_before_the_queue() {
|
||||
let client = async_nats::ConnectOptions::new()
|
||||
.retry_on_initial_connect()
|
||||
.connect("127.0.0.1:1")
|
||||
.await
|
||||
.expect("returns without a connection");
|
||||
let state = state(&client);
|
||||
|
||||
let problem = put(&state, "atlas", caps(Some("0%"), None))
|
||||
.await
|
||||
.expect_err("a zero cap is refused");
|
||||
assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}");
|
||||
|
||||
// The control: a valid cap on the same state does reach the queue.
|
||||
let problem = put(&state, "atlas", caps(Some("400%"), None))
|
||||
.await
|
||||
.expect_err("the queue is not connected");
|
||||
assert_eq!(
|
||||
problem.status,
|
||||
Some(StatusCode::SERVICE_UNAVAILABLE),
|
||||
"{problem:?}"
|
||||
);
|
||||
}
|
||||
|
||||
const CAPPED: &str = r#"{"cpu_quota":"400%","memory_max":"8G"}"#;
|
||||
|
||||
/// `atlas` capped, behind the real router.
|
||||
async fn capped_router(client: &async_nats::Client) -> axum::Router {
|
||||
let state = declared(client).await;
|
||||
let _ = put(&state, "atlas", caps(Some("400%"), Some("8G")))
|
||||
.await
|
||||
.expect("cap atlas");
|
||||
crate::build_app(state)
|
||||
}
|
||||
|
||||
async fn put_raw(app: &axum::Router, body: &'static str) -> StatusCode {
|
||||
use tower::ServiceExt as _;
|
||||
app.clone()
|
||||
.oneshot(
|
||||
axum::http::Request::put("/api/agents/atlas/limits")
|
||||
.header(axum::http::header::CONTENT_TYPE, "application/json")
|
||||
.body(axum::body::Body::from(body))
|
||||
.expect("request"),
|
||||
)
|
||||
.await
|
||||
.expect("router answers")
|
||||
.status()
|
||||
}
|
||||
|
||||
/// A body that is not JSON is refused before the handler runs, and the
|
||||
/// caps already set stay. The well-formed `{}` after it is the control:
|
||||
/// the same router and request do reach the bucket.
|
||||
#[tokio::test]
|
||||
async fn a_malformed_body_is_a_400_and_the_caps_stay() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let app = capped_router(&client).await;
|
||||
|
||||
for body in [r#"{"cpu_quota":"#, "cpu_quota=400%", ""] {
|
||||
assert_eq!(
|
||||
put_raw(&app, body).await,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"{body:?}"
|
||||
);
|
||||
assert_eq!(
|
||||
published(&client, "atlas").await.as_deref(),
|
||||
Some(CAPPED),
|
||||
"{body:?}"
|
||||
);
|
||||
}
|
||||
|
||||
assert_eq!(put_raw(&app, "{}").await, StatusCode::OK);
|
||||
assert_eq!(published(&client, "atlas").await.as_deref(), Some("{}"));
|
||||
}
|
||||
|
||||
/// A value carrying a directive on a second line is a 400 through the real
|
||||
/// router, and the caps already set stay.
|
||||
#[tokio::test]
|
||||
async fn a_cap_with_a_second_line_is_a_400_and_the_caps_stay() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let app = capped_router(&client).await;
|
||||
|
||||
for body in [
|
||||
r#"{"cpu_quota":"400%\nExecStartPre=/bin/sh -c id #%"}"#,
|
||||
r#"{"memory_max":"8G\nDevicePolicy=auto"}"#,
|
||||
] {
|
||||
assert_eq!(put_raw(&app, body).await, StatusCode::BAD_REQUEST, "{body}");
|
||||
assert_eq!(
|
||||
published(&client, "atlas").await.as_deref(),
|
||||
Some(CAPPED),
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A field the body does not define is a 422 and writes nothing: a
|
||||
/// misspelt `memory_max` must not read as "no memory cap".
|
||||
#[tokio::test]
|
||||
async fn an_unknown_field_is_a_422_and_the_caps_stay() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let app = capped_router(&client).await;
|
||||
|
||||
for body in [
|
||||
r#"{"cpu_quota":"100%","memory":"2G"}"#,
|
||||
r#"{"limits":{"cpu_quota":"100%"}}"#,
|
||||
] {
|
||||
assert_eq!(
|
||||
put_raw(&app, body).await,
|
||||
StatusCode::UNPROCESSABLE_ENTITY,
|
||||
"{body}"
|
||||
);
|
||||
assert_eq!(
|
||||
published(&client, "atlas").await.as_deref(),
|
||||
Some(CAPPED),
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -44,7 +44,6 @@ use utoipa_axum::{router::OpenApiRouter, routes};
|
|||
mod agent_icon;
|
||||
mod agent_identity;
|
||||
mod agent_input;
|
||||
mod agent_limits;
|
||||
mod agent_renewal;
|
||||
mod agent_state_stream;
|
||||
mod agent_status;
|
||||
|
|
@ -3301,11 +3300,6 @@ fn build_app(state: AppState) -> axum::Router {
|
|||
inference::put_agent_inference,
|
||||
inference::delete_agent_inference
|
||||
))
|
||||
.routes(routes!(
|
||||
agent_limits::get_agent_limits,
|
||||
agent_limits::put_agent_limits,
|
||||
agent_limits::delete_agent_limits
|
||||
))
|
||||
.routes(routes!(get_config_prs))
|
||||
.routes(routes!(create_agent))
|
||||
.routes(routes!(mint_agent_identity))
|
||||
|
|
|
|||
|
|
@ -490,17 +490,6 @@ impl Policy {
|
|||
format!("$JS.API.STREAM.CREATE.{agent_status}"),
|
||||
format!("$KV.{}.{hive}.*", swarm_queue_client::agent_status::BUCKET),
|
||||
]);
|
||||
// Reading the caps the swarm sets for the agents this hive runs
|
||||
// (`swarm_queue_client::agent_limits`): a direct get, so read-only, and
|
||||
// no `STREAM.INFO` because the reader skips the lookup.
|
||||
//
|
||||
// Every agent's key, not just this hive's: the bucket is keyed by agent
|
||||
// alone, and naming one hive's agents here needs the roster
|
||||
// `hive_name`'s doc argues against.
|
||||
subjects.push(format!(
|
||||
"$JS.API.DIRECT.GET.{}.>",
|
||||
swarm_queue_client::agent_limits::STREAM
|
||||
));
|
||||
subjects.extend(
|
||||
self.extra_hive_subjects
|
||||
.iter()
|
||||
|
|
@ -723,51 +712,6 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
/// A hive reads any agent's caps and can change none of them, and the
|
||||
/// grant reaches no other bucket.
|
||||
#[test]
|
||||
fn a_hive_may_read_every_agents_caps_and_write_none() {
|
||||
use swarm_queue_client::agent_limits::{STREAM, direct_get_subject, key_subject};
|
||||
|
||||
let p = policy()
|
||||
.permissions("hive-alpha")
|
||||
.expect("a hive is admitted");
|
||||
let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject));
|
||||
|
||||
// Its own agents and agents running elsewhere alike.
|
||||
for agent in ["atlas", "iris"] {
|
||||
assert!(may(&direct_get_subject(agent)), "cannot read {agent}");
|
||||
}
|
||||
// A KV put or delete is a publish to the key's subject; the rest
|
||||
// reshape or empty the stream.
|
||||
for forbidden in [
|
||||
key_subject("atlas"),
|
||||
format!("$JS.API.STREAM.CREATE.{STREAM}"),
|
||||
format!("$JS.API.STREAM.UPDATE.{STREAM}"),
|
||||
format!("$JS.API.STREAM.DELETE.{STREAM}"),
|
||||
format!("$JS.API.STREAM.PURGE.{STREAM}"),
|
||||
format!("$JS.API.STREAM.MSG.DELETE.{STREAM}"),
|
||||
format!("$JS.API.CONSUMER.CREATE.{STREAM}"),
|
||||
// A named consumer, bare and with a key filter, which a grant of
|
||||
// `CONSUMER.CREATE.<stream>.>` would cover.
|
||||
format!("$JS.API.CONSUMER.CREATE.{STREAM}.c1"),
|
||||
format!(
|
||||
"$JS.API.CONSUMER.CREATE.{STREAM}.c1.{}",
|
||||
key_subject("atlas")
|
||||
),
|
||||
] {
|
||||
assert!(!may(&forbidden), "hive alpha may publish {forbidden}");
|
||||
}
|
||||
// The read reaches only this bucket.
|
||||
for other in [
|
||||
swarm_queue_client::agent_inference::direct_get_subject("atlas"),
|
||||
swarm_queue_client::agent_schedules::direct_get_subject("atlas"),
|
||||
"$JS.API.DIRECT.GET.KV_hive-wanted-beta.$KV.hive-wanted-beta.beta".to_owned(),
|
||||
] {
|
||||
assert!(!may(&other), "hive alpha may publish {other}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_extra_subject_without_the_placeholder_is_refused() {
|
||||
// 🩸 The option exists to put a second stream inside ONE hive's
|
||||
|
|
|
|||
|
|
@ -47,9 +47,6 @@ agent-inference = ["async-nats/jetstream"]
|
|||
# `ToSchema` on the review classifier's shared types, for swarm-controller,
|
||||
# whose HTTP API serves the same shapes the queue carries.
|
||||
schema = ["dep:utoipa"]
|
||||
# `agent_limits::read`, for hive-c0re, which reads the key of each agent it
|
||||
# runs the same way. Explicit for the same reason as `notices`.
|
||||
agent-limits = ["async-nats/jetstream"]
|
||||
|
||||
[dependencies]
|
||||
# Bare (no `kv`/`jetstream`) unless a consumer opts into the `kv` feature
|
||||
|
|
|
|||
|
|
@ -1,190 +0,0 @@
|
|||
//! The per-agent resource-cap KV bucket: one key per agent, holding the CPU
|
||||
//! and memory caps the swarm sets for it.
|
||||
//!
|
||||
//! **swarm-controller is the only writer**, and creates the bucket on its
|
||||
//! first write. Every field is optional: an absent field leaves the agent on
|
||||
//! its hive's own answer, so an agent with no key runs exactly as it would
|
||||
//! with no swarm.
|
||||
//!
|
||||
//! **The hive running the agent reads the key**, not the agent: the caps go
|
||||
//! into the container's systemd drop-in, which the host writes. A hive reads
|
||||
//! with one `DIRECT.GET` per agent
|
||||
//! ([`crate::agent_limits::direct_get_subject`]), and its grant covers every
|
||||
//! key of this bucket, read-only. Scoping it to the hive's own agents would
|
||||
//! need a roster in the auth responder.
|
||||
//!
|
||||
//! Keyed by the agent alone, like [`crate::agent_inference`]: an agent can
|
||||
//! move between hives, and its caps go with it.
|
||||
|
||||
/// The KV bucket caps are published into, keyed by agent name.
|
||||
pub const BUCKET: &str = "agent-limits";
|
||||
|
||||
/// The stream backing [`BUCKET`]. `create_key_value` prefixes `KV_`.
|
||||
pub const STREAM: &str = "KV_agent-limits";
|
||||
|
||||
/// The subject `agent`'s entry is stored under.
|
||||
#[must_use]
|
||||
pub fn key_subject(agent: &str) -> String {
|
||||
format!("$KV.{BUCKET}.{agent}")
|
||||
}
|
||||
|
||||
/// The request subject that reads `agent`'s current entry.
|
||||
#[must_use]
|
||||
pub fn direct_get_subject(agent: &str) -> String {
|
||||
format!("$JS.API.DIRECT.GET.{STREAM}.{}", key_subject(agent))
|
||||
}
|
||||
|
||||
/// One agent's caps: the value under `BUCKET`/`<agent>`, as systemd values
|
||||
/// carried verbatim. `None` is "not capped by the swarm", never "uncapped".
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct AgentCaps {
|
||||
/// systemd `CPUQuota=`, e.g. `"400%"`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub cpu_quota: Option<String>,
|
||||
/// systemd `MemoryMax=`, e.g. `"8G"`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub memory_max: Option<String>,
|
||||
}
|
||||
|
||||
impl AgentCaps {
|
||||
/// True when no field is set, which a hive reads the same as no key.
|
||||
#[must_use]
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.cpu_quota.is_none() && self.memory_max.is_none()
|
||||
}
|
||||
}
|
||||
|
||||
/// Open the caps bucket for writing, creating it if nothing has yet.
|
||||
///
|
||||
/// **Controller-side only.** `history: 1`: a hive wants the current value.
|
||||
#[cfg(feature = "kv")]
|
||||
pub async fn open_or_create(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<async_nats::jetstream::kv::Store, crate::Error> {
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
crate::kv::open_or_create(
|
||||
&js,
|
||||
BUCKET,
|
||||
"agent-limits",
|
||||
"Resource caps the swarm controller sets for each agent".to_owned(),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Open the caps bucket for reading, or `None` when nothing has created it
|
||||
/// yet, which reads the same as no caps.
|
||||
///
|
||||
/// **Controller-side**, for reads that must not create the bucket.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// [`crate::Error::OpenBucket`] for every failure other than the bucket not
|
||||
/// existing.
|
||||
#[cfg(feature = "kv")]
|
||||
pub async fn open(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<Option<async_nats::jetstream::kv::Store>, crate::Error> {
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
match js.get_key_value(BUCKET).await {
|
||||
Ok(store) => Ok(Some(store)),
|
||||
Err(e) if crate::wanted::is_bucket_not_found(&e) => Ok(None),
|
||||
Err(source) => Err(crate::Error::OpenBucket {
|
||||
bucket: BUCKET.to_owned(),
|
||||
source,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// What reading one agent's key found.
|
||||
#[cfg(feature = "agent-limits")]
|
||||
#[derive(Debug)]
|
||||
pub enum Read {
|
||||
/// The bucket answered; an absent or deleted key has every field unset.
|
||||
Current(AgentCaps),
|
||||
/// Nothing answers for the bucket's stream: no cap has been written yet,
|
||||
/// or `JetStream` is not serving it right now. Not an unset value, since
|
||||
/// the second case says nothing about the agent's caps.
|
||||
NoBucket,
|
||||
}
|
||||
|
||||
/// Read `agent`'s current caps with one direct get.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// A failed request or an undecodable value. A request the queue refused for
|
||||
/// permissions reaches the client as a timeout, not as a refusal.
|
||||
#[cfg(feature = "agent-limits")]
|
||||
pub async fn read(
|
||||
client: &async_nats::Client,
|
||||
agent: &str,
|
||||
) -> Result<Read, Box<dyn std::error::Error + Send + Sync>> {
|
||||
use async_nats::jetstream::stream::DirectGetErrorKind;
|
||||
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
let stream = js.get_stream_no_info(STREAM).await?;
|
||||
let message = match stream.direct_get_last_for_subject(key_subject(agent)).await {
|
||||
Ok(message) => message,
|
||||
Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => {
|
||||
return Ok(Read::Current(AgentCaps::default()));
|
||||
}
|
||||
Err(e) if is_no_responders(&e) => return Ok(Read::NoBucket),
|
||||
Err(e) => return Err(e.into()),
|
||||
};
|
||||
// A KV delete or purge is stored as an empty marker carrying this header.
|
||||
let removed = message
|
||||
.headers
|
||||
.get("KV-Operation")
|
||||
.is_some_and(|op| matches!(op.as_str(), "DEL" | "PURGE"));
|
||||
if removed {
|
||||
return Ok(Read::Current(AgentCaps::default()));
|
||||
}
|
||||
Ok(Read::Current(serde_json::from_slice(&message.payload)?))
|
||||
}
|
||||
|
||||
#[cfg(feature = "agent-limits")]
|
||||
fn is_no_responders(e: &async_nats::jetstream::stream::DirectGetError) -> bool {
|
||||
matches!(
|
||||
e.kind(),
|
||||
async_nats::jetstream::stream::DirectGetErrorKind::ErrorResponse(status, _)
|
||||
if status == async_nats::StatusCode::NO_RESPONDERS
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The names the hive's grant in `swarm-nats-auth` is built from.
|
||||
#[test]
|
||||
fn names_match_the_queue_grant() {
|
||||
assert_eq!(STREAM, format!("KV_{BUCKET}"));
|
||||
assert_eq!(key_subject("iris"), "$KV.agent-limits.iris");
|
||||
assert_eq!(
|
||||
direct_get_subject("iris"),
|
||||
"$JS.API.DIRECT.GET.KV_agent-limits.$KV.agent-limits.iris"
|
||||
);
|
||||
}
|
||||
|
||||
/// An unset field is left out on the wire, and `{}` is every field unset.
|
||||
#[test]
|
||||
fn unset_fields_are_absent() {
|
||||
let value = AgentCaps {
|
||||
memory_max: Some("8G".to_owned()),
|
||||
..AgentCaps::default()
|
||||
};
|
||||
assert_eq!(
|
||||
serde_json::to_string(&value).unwrap(),
|
||||
r#"{"memory_max":"8G"}"#
|
||||
);
|
||||
let empty: AgentCaps = serde_json::from_str("{}").unwrap();
|
||||
assert!(empty.is_empty());
|
||||
}
|
||||
|
||||
/// Fields a newer controller adds must not cost an older hive its caps.
|
||||
#[test]
|
||||
fn decoding_tolerates_unknown_fields() {
|
||||
let decoded: AgentCaps =
|
||||
serde_json::from_str(r#"{"cpu_quota":"400%","io_weight":"10"}"#).unwrap();
|
||||
assert_eq!(decoded.cpu_quota.as_deref(), Some("400%"));
|
||||
}
|
||||
}
|
||||
|
|
@ -216,11 +216,6 @@ pub mod agent_schedules;
|
|||
/// unconditional; writing needs `kv` and reading needs `agent-inference`.
|
||||
pub mod agent_inference;
|
||||
|
||||
/// The bucket swarm-controller sets each agent's CPU and memory caps in — one
|
||||
/// key per agent, read by the hive that runs it. The names are unconditional;
|
||||
/// writing needs `kv` and reading needs `agent-limits`.
|
||||
pub mod agent_limits;
|
||||
|
||||
#[cfg(feature = "kv")]
|
||||
mod kv;
|
||||
|
||||
|
|
|
|||
|
|
@ -156,7 +156,7 @@ pub async fn open_read_only(
|
|||
/// `get_key_value` reports a missing bucket as `GetBucket` wrapping the
|
||||
/// stream lookup's `STREAM_NOT_FOUND`; every other kind is a real failure.
|
||||
#[cfg(feature = "kv")]
|
||||
pub(crate) fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool {
|
||||
fn is_bucket_not_found(e: &async_nats::jetstream::context::KeyValueError) -> bool {
|
||||
use async_nats::jetstream::context::{GetStreamError, GetStreamErrorKind, KeyValueErrorKind};
|
||||
|
||||
e.kind() == KeyValueErrorKind::GetBucket
|
||||
|
|
|
|||
Loading…
Reference in a new issue