Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
15 changed files with 295 additions and 769 deletions

View file

@ -403,8 +403,7 @@ Contents:
Written by `resource_limits::set_limits`; read where Written by `resource_limits::set_limits`; read where
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected `lifecycle::write_dropins` generates the systemd drop-in, **not** injected
into the container — these are host-side caps on the container, so into the container — these are host-side caps on the container, so
the capped party never sets them. It sees only the resulting memory cap, the capped party never sees or sets them. Fallback is per *field*:
as a byte count or `infinity` in its runtime dir (below). Fallback is per *field*:
an absent file, absent agent, or absent field falls back to the an absent file, absent agent, or absent field falls back to the
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`, hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
so an agent can override only its memory and still track the hive so an agent can override only its memory and still track the hive
@ -509,12 +508,6 @@ containers can't start.
- `/run/hyperhive/host.sock` — admin socket (host-side CLI). - `/run/hyperhive/host.sock` — admin socket (host-side CLI).
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket - `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
(bind-mounted into the container as `/run/hive/mcp.sock`). (bind-mounted into the container as `/run/hive/mcp.sock`).
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
`MemoryMax=` as a byte count (a percentage resolved against the host's
physical memory) or `infinity`, written by `lifecycle::write_dropins`
before every start. Absent only when hive-c0re can't tell the cap. The
container reads it at boot as `/run/hive/memory-max-bytes`
(`nix/agent-modules/memory-cap.nix`).
On startup, `Coordinator::register_agent` drops any prior socket On startup, `Coordinator::register_agent` drops any prior socket
task before rebinding — idempotent so a hive-c0re restart followed task before rebinding — idempotent so a hive-c0re restart followed

View file

@ -452,23 +452,6 @@ regardless.
rebuild, so changes take effect on the next lifecycle op without rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild. requiring a host rebuild.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the overrides file, the file carries
that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
`hive-subagent-daemon` (two thirds). Neither is in the agent's build, so a
changed cap reaches them at the next container start without a rebuild.
`infinity` sets neither. A missing or malformed file sets neither as well, and
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
The same drop-in carries `CPUWeight=` / `IOWeight=` from The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of `agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle setting: the quota and the memory max are **hard caps** that throttle

View file

@ -122,36 +122,79 @@ fn resolve(limits: &AgentLimits, hive_cpu_quota: &str, hive_memory_max: &str) ->
(cpu, mem) (cpu, mem)
} }
/// Effective `MemoryMax=` string for `name` against an already-loaded
/// map — the memory-only half of [`effective_from`], split out because
/// [`effective_memory_bytes_from`] doesn't have a CPU quota to pass.
/// No single-agent convenience wrapper (unlike [`effective`] /
/// [`effective_from`]): every current caller already has a loaded map
/// in hand, so one would just be dead code.
#[must_use]
fn effective_memory_max_from(
limits: &BTreeMap<String, AgentLimits>,
name: &str,
hive_memory_max: &str,
) -> String {
limits
.get(name)
.and_then(|l| l.memory_max.clone())
.unwrap_or_else(|| hive_memory_max.to_owned())
}
/// Effective `MemoryMax=` for `name`, as a raw byte count, against an
/// already-loaded map — the form `render_flake_with_lookup` uses so it
/// doesn't re-read `resource-limits.json` once per agent (same
/// reasoning as [`effective_from`] / `container_view::build_all`). No
/// single-agent convenience wrapper, for the same reason as
/// [`effective_memory_max_from`] — every current caller already has a
/// loaded map in hand.
///
/// Logs a `warn!` naming `name` when the effective value is a RAM
/// percentage: unlike `"infinity"`, a percentage IS a real, resolvable
/// cap — resolving one just needs host `MemTotal`, which this module
/// doesn't track — so today it degrades to `None` (no derived heap
/// ceiling) rather than silently guessing wrong against `MemTotal`.
#[must_use]
pub fn effective_memory_bytes_from(
limits: &BTreeMap<String, AgentLimits>,
name: &str,
hive_memory_max: &str,
) -> Option<u64> {
let mem = effective_memory_max_from(limits, name, hive_memory_max);
if mem.ends_with('%') {
tracing::warn!(
agent = name,
memory_max = %mem,
"effective MemoryMax= is a RAM percentage; can't derive a JSC heap ceiling from it \
without host MemTotal — leaving BUN_JSC_forceRAMSize unset for this agent"
);
}
parse_bytes(&mem)
}
/// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`, /// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`,
/// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`) /// a bare byte count, optionally with a decimal like `"1.5G"`) into a
/// into a raw byte count, accepting exactly the single-term forms systemd's /// raw byte count. systemd's `K`/`M`/`G`/`T` suffixes are IEC binary
/// `parse_size` does: a `K`/`M`/`G`/`T`/`P`/`E` suffix, IEC binary /// (1024-based), not decimal — this matches. Pure integer arithmetic
/// (1024-based), a `B` suffix, or none. Suffixes are case-sensitive, and a
/// redundant `B` after a multiplier (`4GB`) is rejected, as systemd rejects
/// both and then leaves the unit uncapped. Pure integer arithmetic
/// throughout (via `u128` headroom) rather than `f64`: a `MemoryMax=` /// throughout (via `u128` headroom) rather than `f64`: a `MemoryMax=`
/// value is always a small non-negative decimal (enforced by /// value is always a small non-negative decimal (enforced by
/// [`is_plain_number`] upstream in [`validate_memory_max`]), so floats /// [`is_plain_number`] upstream in [`validate_memory_max`]), so floats
/// would only add rounding / sign-loss risk for no benefit. Returns /// would only add rounding / sign-loss risk for no benefit. Returns
/// `None` for `"infinity"` and for percentages, which need the host's /// `None` for `"infinity"` and percentages; see
/// `MemTotal` to resolve. /// [`effective_memory_bytes_from`] for why those can't be turned into a
/// byte count here.
#[must_use] #[must_use]
pub fn parse_bytes(value: &str) -> Option<u64> { pub fn parse_bytes(value: &str) -> Option<u64> {
if value == "infinity" || value.ends_with('%') { if value == "infinity" || value.ends_with('%') {
return None; return None;
} }
let (mantissa, exponent) = [ let (mantissa, exponent) = [('K', 1u32), ('M', 2), ('G', 3), ('T', 4)]
('E', 6u32), .into_iter()
('P', 5), .find_map(|(suffix, exp)| {
('T', 4), value
('G', 3), .strip_suffix([suffix, suffix.to_ascii_lowercase()])
('M', 2), .map(|m| (m, exp))
('K', 1), })
('B', 0), .unwrap_or((value, 0));
]
.into_iter()
.find_map(|(suffix, exp)| value.strip_suffix(suffix).map(|m| (m, exp)))
.unwrap_or((value, 0));
let scale = u128::from(1024u64.checked_pow(exponent)?); let scale = u128::from(1024u64.checked_pow(exponent)?);
let (int_part, frac_part) = mantissa.split_once('.').unwrap_or((mantissa, "")); let (int_part, frac_part) = mantissa.split_once('.').unwrap_or((mantissa, ""));
let whole: u128 = int_part.parse().ok()?; let whole: u128 = int_part.parse().ok()?;
@ -473,14 +516,8 @@ mod tests {
parse_bytes("1.5G"), parse_bytes("1.5G"),
Some(1024 * 1024 * 1024 + 512 * 1024 * 1024) Some(1024 * 1024 * 1024 + 512 * 1024 * 1024)
); );
assert_eq!(parse_bytes("1048576B"), Some(1_048_576)); // Lowercase suffixes accepted, matching `normalized_byte_size`.
assert_eq!(parse_bytes("3P"), Some(3 << 50)); assert_eq!(parse_bytes("4g"), Some(4 * 1024 * 1024 * 1024));
assert_eq!(parse_bytes("2E"), Some(2 << 60));
assert_eq!(
parse_bytes("16E"),
None,
"past u64, which systemd refuses too"
);
} }
#[test] #[test]
@ -491,38 +528,50 @@ mod tests {
#[test] #[test]
fn parse_bytes_rejects_garbage() { fn parse_bytes_rejects_garbage() {
// systemd's suffixes are case-sensitive and take no `B` after a for v in ["", "abc", "-8G", "8Gi"] {
// multiplier, so it leaves a unit with any of these uncapped.
for v in [
"", "abc", "-8G", "8Gi", "4g", "4k", "1024b", "4GB", "4MB", "B", ".5G",
] {
assert_eq!(parse_bytes(v), None, "{v} should not parse"); assert_eq!(parse_bytes(v), None, "{v} should not parse");
} }
} }
#[test] #[test]
fn every_size_the_validator_stores_has_a_byte_count() { fn effective_memory_bytes_falls_back_to_hive_default() {
for (input, bytes) in [ // Empty map, i.e. no per-agent override — exercises the
("1048576", 1_048_576), // "use the hive-wide default" arm end to end.
("1048576B", 1_048_576), let empty = BTreeMap::new();
("1048576b", 1_048_576), assert_eq!(
("4096K", 4096 << 10), effective_memory_bytes_from(&empty, "nobody-configured-this-agent", "4G"),
("4096kb", 4096 << 10), Some(4 * 1024 * 1024 * 1024)
("512M", 512 << 20), );
("512mB", 512 << 20), assert_eq!(
("4G", 4 << 30), effective_memory_bytes_from(&empty, "nobody-configured-this-agent", "infinity"),
("4g", 4 << 30), None
("4GB", 4 << 30), );
("1.5G", 3 << 29), }
("2T", 2 << 40),
("2tb", 2 << 40), #[test]
] { fn effective_memory_bytes_from_prefers_per_agent_override() {
let stored = validate_memory_max(input).expect(input); let map = BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]);
assert_eq!( assert_eq!(
parse_bytes(&stored), effective_memory_bytes_from(&map, "sock", HIVE_MEM),
Some(bytes), Some(8 * 1024 * 1024 * 1024)
"{input} stored as {stored}" );
); // A different agent not in the map still falls back to the
} // hive-wide default from the same loaded map (no re-read).
assert_eq!(
effective_memory_bytes_from(&map, "iris", HIVE_MEM),
Some(4 * 1024 * 1024 * 1024)
);
}
/// A percentage cap is real and resolvable in principle, but this
/// module has no host `MemTotal` to resolve it against — must
/// degrade to `None` (no derived heap ceiling) rather than silently
/// treating it as unbounded or guessing a number. `warn!` firing is
/// exercised for coverage but not asserted on (no tracing test
/// subscriber wired up here) — the `None` return is the contract.
#[test]
fn effective_memory_bytes_from_returns_none_for_percentage() {
let map = BTreeMap::from([("sock".to_owned(), limits(None, Some("50%")))]);
assert_eq!(effective_memory_bytes_from(&map, "sock", HIVE_MEM), None);
} }
} }

View file

@ -1,7 +1,6 @@
//! Per-container host-side config: the nspawn conf rewrite (bind mounts, //! Per-container host-side config: the nspawn conf rewrite (bind mounts,
//! network isolation, forwarded credentials), the systemd resource-limits //! network isolation, forwarded credentials), the systemd resource-limits
//! drop-in, the agent's copy of its memory cap, and the `write_dropins` verb //! drop-in, and the `write_dropins` verb that re-applies both.
//! that re-applies them.
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@ -17,22 +16,20 @@ use super::{
}; };
/// Re-apply the per-container host-side config: nspawn flags (bind /// Re-apply the per-container host-side config: nspawn flags (bind
/// mounts etc.), the systemd resource-limits drop-in, the agent's copy of /// mounts etc.), the systemd resource-limits drop-in, and a daemon
/// its memory cap, and a daemon reload so all of it takes effect on the next /// reload so both take effect on the next unit (re)start. Idempotent —
/// unit (re)start. Idempotent — the job queue's `WriteDropin` node, also /// the job queue's `WriteDropin` node, also folded into every `Swap`
/// folded into every `Swap` (rebuild is the reconcile verb). /// (rebuild is the reconcile verb).
pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> { pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
validate(name)?; validate(name)?;
let container = container_name(name); let container = container_name(name);
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?; set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
if let Some((cpu_quota, memory_max)) = stage_limits( if let Some((cpu_quota, memory_max)) = limits_to_apply(
name, name,
&crate::resource_limits::resource_limits_path(), &crate::resource_limits::resource_limits_path(),
&limits_dropin_path(&container), &limits_dropin_path(&container),
&paths.agent,
hive, hive,
HostMemory::read(), ) {
)? {
set_resource_limits( set_resource_limits(
&container, &container,
&cpu_quota, &cpu_quota,
@ -54,173 +51,6 @@ fn limits_dropin_path(container: &str) -> PathBuf {
)) ))
} }
/// File in the agent's runtime dir, `/run/hive/` in the container, holding the
/// `MemoryMax=` its drop-in carries: a byte count, or `infinity` for no cap.
/// `nix/agent-modules/memory-cap.nix` reads it at boot under this name, and
/// treats a missing file as a fault.
const MEMORY_CAP_FILE: &str = "memory-max-bytes";
/// The host's physical memory and page size, the two numbers systemd resolves
/// a `MemoryMax=` percentage against.
#[derive(Clone, Copy, Debug)]
struct HostMemory {
bytes: u64,
page: u64,
}
impl HostMemory {
/// `sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE)`, as systemd's
/// `physical_memory()` computes it. systemd also caps that by the root
/// cgroup's `memory.max`, which the host's root cgroup does not have.
fn read() -> Option<Self> {
// SAFETY: `sysconf(3)` only reads system configuration.
let (pages, page) = unsafe {
(
libc::sysconf(libc::_SC_PHYS_PAGES),
libc::sysconf(libc::_SC_PAGESIZE),
)
};
let pages = u64::try_from(pages).ok().filter(|&p| p > 0)?;
let page = u64::try_from(page).ok().filter(|&p| p > 0)?;
Some(Self {
bytes: pages.checked_mul(page)?,
page,
})
}
}
/// [`limits_to_apply`], plus handing the memory cap to the agent through
/// [`MEMORY_CAP_FILE`] in `runtime_dir`. The container cannot read the cap
/// systemd enforces on it, so this file is how it learns the value it sizes
/// its heap and its subagents against. It is written on every call, from the
/// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's.
fn stage_limits(
agent_name: &str,
limits_path: &Path,
dropin_path: &Path,
runtime_dir: &Path,
hive: &HiveEnv,
host: Option<HostMemory>,
) -> Result<Option<(String, String)>> {
let limits = limits_to_apply(agent_name, limits_path, dropin_path, hive);
let memory_max = match &limits {
Some((_, memory_max)) => Some(memory_max.clone()),
None => dropin_memory_max(agent_name, dropin_path),
};
let cap = memory_max.and_then(|m| memory_cap_value(agent_name, &m, host));
write_memory_cap(runtime_dir, cap.as_deref())?;
Ok(limits)
}
/// The `MemoryMax=` the drop-in at `path` applies: its last assignment, as
/// systemd reads it, or `infinity`, systemd's default, when it has none or its
/// last one is empty. `None` when the file can't be read.
fn dropin_memory_max(agent_name: &str, path: &Path) -> Option<String> {
let body = match std::fs::read_to_string(path) {
Ok(body) => body,
Err(e) => {
tracing::warn!(
agent = %agent_name,
path = %path.display(),
error = ?e,
"limits drop-in unreadable; the agent gets no memory cap"
);
return None;
}
};
let value = body
.lines()
.rev()
.find_map(|line| line.trim().strip_prefix("MemoryMax="))
.map(str::trim)
.filter(|v| !v.is_empty())
.unwrap_or("infinity");
Some(value.to_owned())
}
/// What [`MEMORY_CAP_FILE`] says for `memory_max`, read the way systemd reads
/// `MemoryMax=`: a size is its byte count, and a percentage is that share of
/// `host`'s physical memory, rounded down to a page. A value systemd ignores
/// (zero, over 100%, more than two decimals, unparseable) leaves the unit
/// uncapped, so it is `infinity` here. `None` for a percentage when `host` is
/// unknown.
fn memory_cap_value(
agent_name: &str,
memory_max: &str,
host: Option<HostMemory>,
) -> Option<String> {
let bytes = if let Some(percent) = memory_max.strip_suffix('%') {
let Some(host) = host else {
tracing::warn!(
agent = agent_name,
memory_max,
"host physical memory unknown; the agent gets no memory cap"
);
return None;
};
permyriad(percent).and_then(|p| scale_physical_memory(host, p))
} else {
crate::resource_limits::parse_bytes(memory_max)
};
Some(
bytes
.filter(|&b| b > 0)
.map_or_else(|| "infinity".to_owned(), |b| b.to_string()),
)
}
/// `percent` (the part before `%`) in hundredths of a percent, with systemd's
/// `parse_permyriad` rules: digits, at most two decimals, at most 100%.
fn permyriad(percent: &str) -> Option<u64> {
let (int, frac) = percent.split_once('.').unwrap_or((percent, ""));
if int.is_empty() || !int.bytes().all(|b| b.is_ascii_digit()) {
return None;
}
let hundredths = match frac.as_bytes() {
[] if !percent.contains('.') => 0,
[d] if d.is_ascii_digit() => u64::from(d - b'0') * 10,
[d1, d2] if d1.is_ascii_digit() && d2.is_ascii_digit() => {
u64::from(d1 - b'0') * 10 + u64::from(d2 - b'0')
}
_ => return None,
};
let value = int
.parse::<u64>()
.ok()?
.checked_mul(100)?
.checked_add(hundredths)?;
(value <= 10_000).then_some(value)
}
/// systemd's `physical_memory_scale(permyriad, 10000)`: whole pages, rounded
/// down, and the exact total at 100%.
fn scale_physical_memory(host: HostMemory, permyriad: u64) -> Option<u64> {
if permyriad == 10_000 {
return Some(host.bytes);
}
let pages = u128::from(host.bytes / host.page) * u128::from(permyriad) / 10_000;
u64::try_from(pages * u128::from(host.page)).ok()
}
/// Write `cap` into [`MEMORY_CAP_FILE`], or remove the file when the cap is
/// unknown, which the agent reports at boot.
fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
let path = runtime_dir.join(MEMORY_CAP_FILE);
let Some(cap) = cap else {
return match std::fs::remove_file(&path) {
Err(e) if e.kind() != std::io::ErrorKind::NotFound => {
Err(e).with_context(|| format!("remove {}", path.display()))
}
_ => Ok(()),
};
};
std::fs::create_dir_all(runtime_dir)
.with_context(|| format!("create agent runtime dir {}", runtime_dir.display()))?;
let tmp = runtime_dir.join(format!(".{MEMORY_CAP_FILE}.tmp"));
std::fs::write(&tmp, format!("{cap}\n")).with_context(|| format!("write {}", tmp.display()))?;
std::fs::rename(&tmp, &path).with_context(|| format!("rename onto {}", path.display()))
}
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits /// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is. /// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
/// ///
@ -659,10 +489,8 @@ async fn set_nspawn_flags(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::{
BindMount, HostMemory, MEMORY_CAP_FILE, QUEUE_CLIENT_ID_CREDENTIAL, BindMount, QUEUE_CLIENT_ID_CREDENTIAL, QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs,
QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs, bind_own_agent_dirs, bind_own_agent_dirs, holds_manage_root_agent, limits_to_apply, queue_agent_credentials,
holds_manage_root_agent, limits_to_apply, memory_cap_value, queue_agent_credentials,
stage_limits,
}; };
use crate::coordinator::HiveEnv; use crate::coordinator::HiveEnv;
@ -879,169 +707,4 @@ mod tests {
let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()); let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default());
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned()))); assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
} }
/// 4 000 003 pages of 4 KiB: odd, so rounding to a page shows.
const HOST: HostMemory = HostMemory {
bytes: 4_000_003 * 4096,
page: 4096,
};
/// Runs [`stage_limits`] against the files in `dir` and returns the
/// drop-in's `MemoryMax=` and the agent's copy of it, `None` when absent.
fn stage(dir: &tempfile::TempDir) -> (Option<String>, Option<String>) {
let runtime = dir.path().join("runtime");
let applied = stage_limits(
"sock",
&dir.path().join("resource-limits.json"),
&dir.path().join("hyperhive-limits.conf"),
&runtime,
&HiveEnv::default(),
Some(HOST),
)
.expect("stage limits");
let copy = std::fs::read_to_string(runtime.join(MEMORY_CAP_FILE)).ok();
(applied.map(|(_, mem)| mem), copy)
}
#[test]
fn the_agent_reads_the_cap_its_dropin_carries() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
// No overrides anywhere: the hive-wide default, 4G.
assert_eq!(
stage(&dir),
(Some("4G".to_owned()), Some("4294967296\n".to_owned()))
);
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
assert_eq!(
stage(&dir),
(Some("16G".to_owned()), Some("17179869184\n".to_owned()))
);
}
#[test]
fn every_kind_of_cap_is_written_for_the_agent() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
for (memory_max, copy) in [
("infinity", "infinity\n"),
("50%", "8192004096\n"),
("8G", "8589934592\n"),
] {
std::fs::write(
&limits,
format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#),
)
.expect("seed limits");
assert_eq!(
stage(&dir),
(Some(memory_max.to_owned()), Some(copy.to_owned())),
"{memory_max}"
);
}
}
#[test]
fn a_cap_is_read_the_way_systemd_reads_memory_max() {
for (memory_max, cap) in [
("4G", "4294967296"),
("1048576B", "1048576"),
("3P", "3377699720527872"),
("infinity", "infinity"),
("50%", "8192004096"),
("33.33%", "5460787200"),
("12.5%", "2048000000"),
("100%", "16384012288"),
// systemd ignores these and the unit runs uncapped.
("0", "infinity"),
("4g", "infinity"),
("4GB", "infinity"),
("0%", "infinity"),
("150%", "infinity"),
("12.345%", "infinity"),
("lots", "infinity"),
] {
assert_eq!(
memory_cap_value("sock", memory_max, Some(HOST)).as_deref(),
Some(cap),
"{memory_max}"
);
}
assert_eq!(memory_cap_value("sock", "50%", None), None);
assert_eq!(
memory_cap_value("sock", "8G", None).as_deref(),
Some("8589934592")
);
}
#[test]
fn the_host_memory_systemd_would_read_is_found() {
let host = HostMemory::read().expect("sysconf");
assert!(
host.page >= 4096 && host.bytes.is_multiple_of(host.page),
"{host:?}"
);
}
#[test]
fn a_kept_dropin_hands_the_agent_its_cap() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
let dropin = dir.path().join("hyperhive-limits.conf");
let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE);
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned()));
// Unreadable limits keep the drop-in; the copy follows the drop-in.
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
std::fs::write(
&dropin,
"[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n",
)
.expect("seed drop-in");
assert_eq!(stage(&dir), (None, Some("2147483648\n".to_owned())));
// A kept drop-in with no copy beside it, as after an upgrade.
std::fs::remove_file(&copy).expect("remove copy");
assert_eq!(stage(&dir), (None, Some("2147483648\n".to_owned())));
// The last assignment is the one systemd applies.
std::fs::write(&dropin, "[Service]\nMemoryMax=2G\nMemoryMax=3G\n").expect("seed drop-in");
assert_eq!(stage(&dir), (None, Some("3221225472\n".to_owned())));
// No `MemoryMax=` is systemd's default.
std::fs::write(&dropin, "[Service]\nCPUQuota=200%\n").expect("seed drop-in");
assert_eq!(stage(&dir), (None, Some("infinity\n".to_owned())));
}
#[test]
fn an_unknown_cap_leaves_no_copy() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
let dropin = dir.path().join("hyperhive-limits.conf");
assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned()));
// Unreadable limits and an unreadable drop-in: nothing says what the
// cgroup enforces.
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
std::fs::create_dir(&dropin).expect("seed unreadable drop-in");
assert_eq!(stage(&dir), (None, None));
// A percentage with no host memory to resolve it against.
std::fs::write(&limits, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed limits");
let runtime = dir.path().join("runtime");
let applied = stage_limits(
"sock",
&limits,
&dropin,
&runtime,
&HiveEnv::default(),
None,
)
.expect("stage limits");
assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned()));
assert!(!runtime.join(MEMORY_CAP_FILE).exists());
}
} }

View file

@ -88,7 +88,17 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> {
let dir = crate::paths::meta_root(); let dir = crate::paths::meta_root();
std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?; std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?;
let new_flake = render_hive_flake(hive, agents)?; let new_flake = render_flake(
&hive.hyperhive_flake,
&hive.hyperhive_docs_flake,
&hive.nixpkgs_flake,
hive.claude_code_path.as_deref(),
hive.dashboard_port,
&hive.operator_pronouns,
&hive.context_window_tokens,
&hive.agent_memory_max,
agents,
)?;
let flake_path = dir.join("flake.nix"); let flake_path = dir.join("flake.nix");
let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default(); let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default();
let initial = !dir.join(".git").exists(); let initial = !dir.join(".git").exists();
@ -434,21 +444,6 @@ pub async fn commit_perms(
Ok(()) Ok(())
} }
/// [`render_flake`] with its inputs taken from `hive`, as [`sync_agents`]
/// renders it.
fn render_hive_flake(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<String> {
render_flake(
&hive.hyperhive_flake,
&hive.hyperhive_docs_flake,
&hive.nixpkgs_flake,
hive.claude_code_path.as_deref(),
hive.dashboard_port,
&hive.operator_pronouns,
&hive.context_window_tokens,
agents,
)
}
#[allow( #[allow(
clippy::too_many_arguments, clippy::too_many_arguments,
reason = "many genuine flake inputs (source flakes, port, pronouns, tokens, \ reason = "many genuine flake inputs (source flakes, port, pronouns, tokens, \
@ -462,6 +457,7 @@ fn render_flake(
dashboard_port: u16, dashboard_port: u16,
operator_pronouns: &str, operator_pronouns: &str,
context_window_tokens: &std::collections::HashMap<String, u64>, context_window_tokens: &std::collections::HashMap<String, u64>,
hive_memory_max: &str,
agents: &[AgentSpec], agents: &[AgentSpec],
) -> Result<String> { ) -> Result<String> {
render_flake_with_lookup( render_flake_with_lookup(
@ -472,6 +468,7 @@ fn render_flake(
dashboard_port, dashboard_port,
operator_pronouns, operator_pronouns,
context_window_tokens, context_window_tokens,
hive_memory_max,
agents, agents,
agent_canonical_inputs, agent_canonical_inputs,
) )
@ -864,6 +861,7 @@ fn render_flake_with_lookup<F>(
dashboard_port: u16, dashboard_port: u16,
operator_pronouns: &str, operator_pronouns: &str,
context_window_tokens: &std::collections::HashMap<String, u64>, context_window_tokens: &std::collections::HashMap<String, u64>,
hive_memory_max: &str,
agents: &[AgentSpec], agents: &[AgentSpec],
lookup: F, lookup: F,
) -> Result<String> ) -> Result<String>
@ -945,7 +943,7 @@ where
let pronouns_escaped = operator_pronouns.replace('\\', "\\\\").replace('"', "\\\""); let pronouns_escaped = operator_pronouns.replace('\\', "\\\\").replace('"', "\\\"");
let _ = writeln!( let _ = writeln!(
out, out,
" dashboardPort = {dashboard_port};\n operatorPronouns = \"{pronouns_escaped}\";\n mkAgent = {{ name, isManager, port, toolGroups ? null, capabilities ? null }}:" " dashboardPort = {dashboard_port};\n operatorPronouns = \"{pronouns_escaped}\";\n mkAgent = {{ name, isManager, port, toolGroups ? null, capabilities ? null, memoryMaxBytes ? null }}:"
); );
out.push_str( out.push_str(
r#" let r#" let
@ -1096,6 +1094,7 @@ where
# nspawn itself already call this container, so logs join to # nspawn itself already call this container, so logs join to
# both without a prefix transform. # both without a prefix transform.
networking.hostName = "h-${name}"; networking.hostName = "h-${name}";
services.hyperhive.agent.claudeMemoryMaxBytes = memoryMaxBytes;
programs.git.config.user = { programs.git.config.user = {
name = name; name = name;
email = "${name}@hyperhive.local"; email = "${name}@hyperhive.local";
@ -1178,9 +1177,12 @@ where
); );
// Propagated, never read as empty: an agent with no tool-groups entry // Propagated, never read as empty: an agent with no tool-groups entry
// renders `toolGroups = null` and gets `AGENT_DEFAULT`, which fails // renders `toolGroups = null` and gets `AGENT_DEFAULT`, which fails
// open for any agent whose explicit entry is narrower. // open for any agent whose explicit entry is narrower. Likewise an
// agent with no resource-limits entry gets the hive `memoryMaxBytes`,
// a larger heap ceiling than a tighter override.
let tool_groups_map = crate::tool_groups::read()?; let tool_groups_map = crate::tool_groups::read()?;
let capabilities_map = crate::capabilities::read()?; let capabilities_map = crate::capabilities::read()?;
let resource_limits_map = crate::resource_limits::read()?;
for spec in agents { for spec in agents {
// Emit `toolGroups = "group1,group2"` when the operator has // Emit `toolGroups = "group1,group2"` when the operator has
// explicitly configured groups for this agent. Absent entry = null // explicitly configured groups for this agent. Absent entry = null
@ -1206,15 +1208,29 @@ where
let joined = caps.join(","); let joined = caps.join(",");
format!("\"{joined}\"") format!("\"{joined}\"")
}; };
// Effective `MemoryMax=` for this agent (per-agent override, else
// the hive-wide default), turned into a raw byte count so
// the agent's claude-settings module can derive a JSC heap ceiling
// (see `services.hyperhive.agent.claudeMemoryMaxBytes`). `null` when the
// effective value is `"infinity"` or a RAM percentage — no
// byte count to derive, dependent env var stays unset, same as
// today's no-cap behavior.
let memory_max_attr = crate::resource_limits::effective_memory_bytes_from(
&resource_limits_map,
&spec.name,
hive_memory_max,
)
.map_or_else(|| "null".to_owned(), |b| b.to_string());
let _ = writeln!( let _ = writeln!(
out, out,
" {} = mkAgent {{ name = \"{}\"; isManager = {}; port = {}; toolGroups = {}; capabilities = {}; }};", " {} = mkAgent {{ name = \"{}\"; isManager = {}; port = {}; toolGroups = {}; capabilities = {}; memoryMaxBytes = {}; }};",
spec.name, spec.name,
spec.name, spec.name,
if spec.is_manager { "true" } else { "false" }, if spec.is_manager { "true" } else { "false" },
spec.port, spec.port,
tool_groups_attr, tool_groups_attr,
capabilities_attr, capabilities_attr,
memory_max_attr,
); );
} }
out.push_str(" };\n };\n}\n"); out.push_str(" };\n };\n}\n");
@ -1452,6 +1468,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1499,6 +1516,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1518,6 +1536,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1559,6 +1578,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1588,6 +1608,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1609,6 +1630,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1640,6 +1662,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[ &[
sample_spec("argus", false, 9001), sample_spec("argus", false, 9001),
sample_spec("bitburner", false, 9002), sample_spec("bitburner", false, 9002),
@ -1674,6 +1697,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
|_| Vec::new(), |_| Vec::new(),
) )
@ -1713,6 +1737,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1759,6 +1784,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -1940,6 +1966,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1977,6 +2004,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -2022,6 +2050,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -2115,6 +2144,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -2190,6 +2220,7 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -2215,20 +2246,32 @@ mod tests {
); );
} }
/// The agent reads its memory cap at boot from the copy /// The JSC-heap-ceiling fix needs the effective per-agent memory cap
/// `lifecycle::write_dropins` hands it, so the cap is no build input. /// threaded into the flake as raw bytes, since nspawn hides the real
/// cgroup cap from inside the container. An
/// agent with no `resource-limits.json` override falls back to the
/// hive-wide default passed to `render_flake` (`"4G"` in every test
/// in this module) — this locks in the byte-count conversion end to
/// end through the actual render path (not just `parse_bytes`
/// in isolation).
#[test] #[test]
fn render_hive_flake_is_identical_across_memory_caps() { fn render_flake_derives_memory_max_bytes_from_hive_default() {
let hive = |cap: &str| HiveEnv { let out = render_flake(
agent_memory_max: cap.to_owned(), "github:example/hyperhive",
..HiveEnv::default() "path:/nix/store/bbbb-hyperhive-docs-source",
}; "path:/nix/store/aaaa-nixpkgs-source",
let agents = [sample_spec("alice", false, 9001)]; None,
let small = render_hive_flake(&hive("4G"), &agents).expect("render 4G"); 8000,
let large = render_hive_flake(&hive("16G"), &agents).expect("render 16G"); "she/her",
assert_eq!( &std::collections::HashMap::new(),
small, large, "4G",
"the rendered flake must not depend on the memory cap" &[sample_spec("alice", false, 9001)],
)
.expect("render flake");
let want_bytes = 4u64 * 1024 * 1024 * 1024;
assert!(
out.contains(&format!("memoryMaxBytes = {want_bytes};")),
"memoryMaxBytes must reflect the hive-wide default in bytes:\n{out}"
); );
} }
} }

View file

@ -433,9 +433,10 @@ async fn handle_set_resource_limits(
&hive.agent_memory_max, &hive.agent_memory_max,
)?; )?;
Ok(HostResponse::messages(vec![format!( Ok(HostResponse::messages(vec![format!(
"{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now; the \ "{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now (restart \
Claude/JSC heap ceiling and the subagent MemoryHigh= derived from it follow at the \ the container if it's running and needs the new cap immediately), but the derived \
container's next start (restart it if it's running and needs them now)" Claude/JSC heap ceiling is baked in at build time, so it needs a REBUILD \
(`hivectl agent {name} rebuild`) to actually track this change"
)])) )]))
} }

View file

@ -19,6 +19,11 @@ let
# meta.rs `otel_config`). Options declared in `otel.nix`, which also # meta.rs `otel_config`). Options declared in `otel.nix`, which also
# exports the generic OTLP environment this container's producers read. # exports the generic OTLP environment this container's producers read.
otelCfg = config.services.hyperhive.agent.otel; otelCfg = config.services.hyperhive.agent.otel;
# Effective per-agent MemoryMax=, in bytes, injected by meta.rs's
# per-agent flake render (`services.hyperhive.agent.claudeMemoryMaxBytes`). `null`
# when the effective cap is unbounded ("infinity") or a RAM
# percentage — see `resource_limits::effective_memory_bytes`.
memoryMaxBytes = config.services.hyperhive.agent.claudeMemoryMaxBytes;
# Base claude-code environment applied to every agent regardless of OTEL. # Base claude-code environment applied to every agent regardless of OTEL.
# Shipped via the managed settings `env` block so claude and `hivectl # Shipped via the managed settings `env` block so claude and `hivectl
# choom` both inherit them without a launch wrapper. # choom` both inherit them without a launch wrapper.
@ -42,6 +47,19 @@ let
CLAUDE_CODE_RESUME_INTERRUPTED_TURN = "1"; CLAUDE_CODE_RESUME_INTERRUPTED_TURN = "1";
# Use the simpler system prompt variant suited to headless operation. # Use the simpler system prompt variant suited to headless operation.
CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT = "1"; CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT = "1";
}
# Bun/JavaScriptCore (the runtime under claude-code's `.claude-wrapped`
# binary) sizes its default heap ceiling off the container's visible
# `/proc/meminfo`, which under nspawn is the *host's* physical RAM, not
# the systemd `MemoryMax=` cgroup cap actually enforced on this
# container — the cap lives on the host's outer cgroup and is invisible
# from inside (confirmed: `/sys/fs/cgroup/memory.max` reads `max` at
# every level from in here). That mismatch lets JSC's heap grow well
# past the real wall before GC kicks in hard, causing choom sessions to
# hang. Pin JSC's ceiling to 75% of the *actual* effective cap instead,
# once it's known at build time.
// lib.optionalAttrs (memoryMaxBytes != null) {
BUN_JSC_forceRAMSize = toString (memoryMaxBytes * 75 / 100);
}; };
# Claude Code's own telemetry switches — what it emits, and whether it # Claude Code's own telemetry switches — what it emits, and whether it
# emits at all. Everything an OTEL SDK reads generically (endpoint, # emits at all. Everything an OTEL SDK reads generically (endpoint,
@ -81,6 +99,31 @@ in
''; '';
}; };
# Build-time implementation surface for the JSC-heap-ceiling fix:
# meta.rs's per-agent flake render injects this from the effective
# `MemoryMax=` (per-agent `resource-limits.json` override, else the
# hive-wide `services.hyperhive.agentMemoryMax`) — see
# `resource_limits::effective_memory_bytes_from`. Not meant to be set
# directly in an agent.nix, same convention as `services.hyperhive.agent.otel.*`
# above; the host option (or `hivectl agent <name> set-limits`) is
# the real operator knob, and this only reflects the value baked in at
# the agent's *last rebuild* — `set-limits` still applies the
# cgroup cap live via a drop-in reload, but this derived heap ceiling
# needs a rebuild to pick up a new value.
options.services.hyperhive.agent.claudeMemoryMaxBytes = lib.mkOption {
type = lib.types.nullOr lib.types.ints.positive;
default = null;
internal = true;
description = ''
Effective per-agent memory cap in bytes, when it's a plain
byte-size value (null for an unbounded or percentage-based cap).
This is the whole container's cap, not claude's share of it —
the name records its first consumer, `BUN_JSC_forceRAMSize` in
`baseClaudeEnv`. `mcp.nix` reads it too, to size the subagent
daemon's `MemoryHigh=` against the container it runs in.
'';
};
options.services.hyperhive.agent.claudeMarketplaces = lib.mkOption { options.services.hyperhive.agent.claudeMarketplaces = lib.mkOption {
type = lib.types.listOf lib.types.str; type = lib.types.listOf lib.types.str;
default = [ default = [
@ -214,7 +257,9 @@ in
environment.etc."claude-code/managed-settings.json".source = environment.etc."claude-code/managed-settings.json".source =
let let
baseSettings = "${config.services.hyperhive.agent.packages.assets}/share/hyperhive/prompts/claude-settings.json"; baseSettings = "${config.services.hyperhive.agent.packages.assets}/share/hyperhive/prompts/claude-settings.json";
# Merge base env (always) with OTEL env (when enabled). # Merge base env (always) with OTEL env (when enabled). jq is always
# run — `baseClaudeEnv` can carry per-agent values (e.g.
# BUN_JSC_forceRAMSize) that can't live in the static store asset.
allEnv = allEnv =
baseClaudeEnv baseClaudeEnv
// lib.optionalAttrs otelCfg.enable otelSettingsEnv // lib.optionalAttrs otelCfg.enable otelSettingsEnv

View file

@ -41,11 +41,6 @@ let
which the hive writes into every container, so no agent build sets it. which the hive writes into every container, so no agent build sets it.
Remove the setting. Remove the setting.
''; '';
memoryMaxBytesRemoved = ''
An agent reads its memory cap at boot from /run/hive/memory-max-bytes,
which the hive writes before every start, so no agent build sets it.
Remove the setting.
'';
in in
{ {
imports = [ imports = [
@ -65,7 +60,6 @@ in
./logs.nix ./logs.nix
./matrix.nix ./matrix.nix
./mcp.nix ./mcp.nix
./memory-cap.nix
./network.nix ./network.nix
./otel.nix ./otel.nix
./packages.nix ./packages.nix
@ -100,13 +94,6 @@ in
# Both spellings, for the reason given above. # Both spellings, for the reason given above.
(lib.mkRemovedOptionModule [ "hyperhive" "hiveName" ] hiveNameRemoved) (lib.mkRemovedOptionModule [ "hyperhive" "hiveName" ] hiveNameRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "hiveName" ] hiveNameRemoved) (lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "hiveName" ] hiveNameRemoved)
(lib.mkRemovedOptionModule [ "hyperhive" "claudeMemoryMaxBytes" ] memoryMaxBytesRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"agent"
"claudeMemoryMaxBytes"
] memoryMaxBytesRemoved)
]; ];
options.services.hyperhive.agent.icon = lib.mkOption { options.services.hyperhive.agent.icon = lib.mkOption {

View file

@ -11,6 +11,25 @@
}: }:
let let
userName = config.services.hyperhive.agent.user.name; userName = config.services.hyperhive.agent.user.name;
# This container's own effective `MemoryMax=` in bytes, baked in per
# agent by meta.rs's flake render — see
# `services.hyperhive.agent.claudeMemoryMaxBytes` in ./claude-settings.nix. `null`
# when the cap is `infinity` or a RAM percentage, i.e. when the module
# has no byte count to size anything against.
containerMemoryMaxBytes = config.services.hyperhive.agent.claudeMemoryMaxBytes;
# Two thirds of the container's cap, as the soft ceiling on everything
# the subagent daemon runs. The daemon spawns nested `claude` sessions
# as plain children, so its cgroup already *is* the "all subagents"
# cgroup and a unit-level ceiling bounds the set without a slice.
#
# `MemoryHigh=` and not `MemoryMax=`: this throttles rather than walls.
# Past it the kernel reclaims aggressively and the cgroup stalls, so a
# subagent that overshoots gets visibly slow and the remaining third
# stays available for the agent's own turn — but a subagent that
# genuinely needs more than two thirds still gets it when the container
# has the memory to spare, which is what keeps overprovisioning
# (several agents that rarely compile at the same time) working.
subagentMemoryHigh = containerMemoryMaxBytes * 2 / 3;
# Set on the harness only for an ACP agent on the opencode preset # Set on the harness only for an ACP agent on the opencode preset
# (./agent-service.nix). # (./agent-service.nix).
acpApiKeyEnv = config.systemd.services.hive-agent.environment.HIVE_ACP_API_KEY_ENV or null; acpApiKeyEnv = config.systemd.services.hive-agent.environment.HIVE_ACP_API_KEY_ENV or null;
@ -456,11 +475,12 @@ in
# actually chose and leaves the daemon serving, which is also what # actually chose and leaves the daemon serving, which is also what
# lets it report the kill rather than vanish and be restarted. # lets it report the kill rather than vanish and be restarted.
OOMPolicy = "continue"; OOMPolicy = "continue";
# `MemoryHigh=` is a drop-in from ./memory-cap.nix, sized at boot from
# the container's cap.
User = userName; User = userName;
Group = userName; Group = userName;
} }
// lib.optionalAttrs (containerMemoryMaxBytes != null) {
MemoryHigh = toString subagentMemoryHigh;
}
// lib.optionalAttrs subagentReadsStore { // lib.optionalAttrs subagentReadsStore {
# Bare id: inherits the credential the container manager passed in, # Bare id: inherits the credential the container manager passed in,
# the same form ./queue-identity.nix uses. # the same form ./queue-identity.nix uses.

View file

@ -1,98 +0,0 @@
# The container's memory cap, read at boot rather than built in, so a changed
# cap needs no rebuild.
#
# The cap is `MemoryMax=` on the host's `container@` unit, which nothing in
# here can read: `/sys/fs/cgroup/memory.max` reads `max` at every level from
# inside. Before every start hive-c0re (`lifecycle::host_config`) writes it to
# `/run/hive/memory-max-bytes` as a byte count, a percentage resolved against
# the host's RAM as systemd resolves it, or `infinity` for no cap. Two values
# are derived from a byte count, and neither is set for `infinity`. A missing
# or malformed file sets neither too, and `hyperhive-memory-cap-check` logs a
# warning at boot:
#
# - `BUN_JSC_forceRAMSize`, 75% of the cap. Bun/JavaScriptCore, the runtime
# under claude-code, sizes its heap ceiling off `/proc/meminfo`, which under
# nspawn is the host's RAM, so without it the heap grows well past the real
# wall before GC kicks in hard and choom sessions hang. An environment
# generator hands it to every unit of the container's systemd, `machinectl
# shell` sessions (`hivectl choom`) included; `/etc/set-environment` exports
# it in login shells that no unit starts (`nixos-container root-login`).
# `hive-agent` and `hive-subagent-daemon` spawn `claude` as a plain child,
# which inherits it, so neither unit may set or unset it.
# - `MemoryHigh=` on `hive-subagent-daemon`, two thirds of the cap, as a
# generator drop-in. The daemon spawns nested `claude` sessions as plain
# children, so its cgroup already is the "all subagents" cgroup. `High`, not
# `Max`: past it the kernel reclaims hard and the subagents slow down, so the
# last third stays free for the agent's own turn, but a subagent that needs
# more still gets it when the container has memory to spare.
#
# Tested by `checks.script-test-agent-memory-cap` and
# `checks.module-eval-agent-memory`.
{ pkgs, ... }:
let
# Sets `bytes`. Exits for `infinity`, and runs `noCap` when there is no file
# or its first line is neither `infinity` nor a plain byte count. A leading
# zero would read as octal, and 16 digits or more could overflow
# `bytes * 75`. Shell builtins only.
readCap = noCap: ''
bytes=
[ -r /run/hive/memory-max-bytes ] && read -r bytes < /run/hive/memory-max-bytes
case "$bytes" in
infinity) exit 0 ;;
"" | 0* | *[!0-9]* | ????????????????*) ${noCap} ;;
esac
'';
envGenerator = pkgs.writeShellScript "hyperhive-memory-cap-env" ''
${readCap "exit 0"}
printf 'BUN_JSC_forceRAMSize=%s\n' "$((bytes * 75 / 100))"
'';
# `$1` is the generator's normal-priority output dir. A drop-in there merges
# with the unit's own file in /etc.
unitGenerator = pkgs.writeShellScript "hyperhive-memory-cap" ''
${readCap "exit 0"}
dir="$1/hive-subagent-daemon.service.d"
${pkgs.coreutils}/bin/mkdir -p "$dir"
printf '[Service]\nMemoryHigh=%s\n' "$((bytes * 2 / 3))" > "$dir/50-memory-high.conf"
'';
# The generators run before the container's journal starts and an nspawn
# container has no /dev/kmsg, so the boot warning for a missing or malformed
# cap file comes from a unit. `<4>` is the journal's warning priority.
check = pkgs.writeShellScript "hyperhive-memory-cap-check" ''
${readCap ''
if [ -e /run/hive/memory-max-bytes ]; then
why="holds neither a byte count nor infinity"
else
why="is missing"
fi
echo "<4>/run/hive/memory-max-bytes $why: no BUN_JSC_forceRAMSize heap ceiling and no MemoryHigh= on hive-subagent-daemon; the host's MemoryMax= on the container still applies." >&2
exit 0
''}
'';
# Sourced by every shell's `/etc/set-environment`, so POSIX sh only.
profile = pkgs.writeText "hyperhive-memory-cap.sh" ''
__hyperhive_cap=$(${envGenerator})
case "$__hyperhive_cap" in
BUN_JSC_forceRAMSize=?*) export "$__hyperhive_cap" ;;
esac
unset __hyperhive_cap
'';
in
{
environment.etc."systemd/system-environment-generators/50-hyperhive-memory-cap".source =
envGenerator;
systemd.generators.hyperhive-memory-cap = unitGenerator;
systemd.services.hyperhive-memory-cap-check = {
description = "Warn when the container's memory cap is unusable";
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = check;
};
};
environment.etc."hyperhive/memory-cap.sh".source = profile;
environment.extraInit = ". /etc/hyperhive/memory-cap.sh";
}

View file

@ -45,6 +45,10 @@
[ "hyperhive" "claudeMarketplaces" ] [ "hyperhive" "claudeMarketplaces" ]
[ "services" "hyperhive" "agent" "claudeMarketplaces" ] [ "services" "hyperhive" "agent" "claudeMarketplaces" ]
) )
(lib.mkRenamedOptionModule
[ "hyperhive" "claudeMemoryMaxBytes" ]
[ "services" "hyperhive" "agent" "claudeMemoryMaxBytes" ]
)
(lib.mkRenamedOptionModule (lib.mkRenamedOptionModule
[ "hyperhive" "claudePlugins" ] [ "hyperhive" "claudePlugins" ]
[ "services" "hyperhive" "agent" "claudePlugins" ] [ "services" "hyperhive" "agent" "claudePlugins" ]

View file

@ -138,12 +138,6 @@ in
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;
}; };
# Runs ./agent-modules/memory-cap.nix's generators and login-shell snippet
# against sample cap files.
script-test-agent-memory-cap = import ./script-tests/agent-memory-cap.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-agent-github-bao = import ./module-eval/agent-github-bao.nix { module-eval-agent-github-bao = import ./module-eval/agent-github-bao.nix {
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;

View file

@ -22,37 +22,39 @@ let
agentHarness agentHarness
; ;
agentDefault = agent { }; # The memory-pressure pair. `claudeMemoryMaxBytes` is the container's own
# cap, rendered per agent by meta.rs — the capped arm is the one every
# real deploy gets, the uncapped arm is a hive that set `infinity` or a
# RAM percentage and so hands the module no byte count to size against.
agentCapped = agent { claudeMemoryMaxBytes = 8589934592; };
agentUncapped = agent { };
agentBashDaemon = machine: machine.systemd.services.hive-bash-daemon; agentBashDaemon = machine: machine.systemd.services.hive-bash-daemon;
agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon; agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon;
cases = [ cases = [
{ {
# The ceiling is sized at boot from the container's cap # Two thirds of the container's cap, and a SOFT ceiling: the daemon's
# (../agent-modules/memory-cap.nix). One in the unit would be the cap at # cgroup holds every nested claude, so `MemoryHigh=` throttles the
# the last rebuild, and every agent would carry its own hive's number. # subagent set as a whole before the kernel picks a victim, while the
name = "the subagent daemon's memory ceiling is not built into its unit"; # absent `MemoryMax=` is what still lets one subagent use more than
ok = !((agentSubagentDaemon agentDefault).serviceConfig ? MemoryHigh); # its share on a container that has the memory free. A hard cap here
} # would trade the silent kill for a guaranteed wall, which is the
{ # shape this deliberately does not have.
# claude's heap ceiling reaches it only as inherited environment: the name = "the subagent daemon throttles at two thirds of the container's memory";
# environment generator fills the manager's, systemd hands that to every
# unit, and both units spawn claude as a plain child. A unit that sets
# or unsets the variable itself cuts that. Without the two generators the
# cap reaches nothing, and without the check unit nothing says so.
name = "claude's units take the heap ceiling from the memory cap read at boot";
ok = ok =
let let
leavesIt = c = (agentSubagentDaemon agentCapped).serviceConfig;
unit:
!(unit.environment ? BUN_JSC_forceRAMSize)
&& !(lib.elem "BUN_JSC_forceRAMSize" (lib.toList (unit.serviceConfig.UnsetEnvironment or [ ])));
in in
leavesIt (agentHarness agentDefault) c.MemoryHigh == "5726623061" && !(c ? MemoryMax);
&& leavesIt (agentSubagentDaemon agentDefault) }
&& agentDefault.environment.etc ? "systemd/system-environment-generators/50-hyperhive-memory-cap" {
&& agentDefault.systemd.generators ? hyperhive-memory-cap # What makes the case above able to fail. With no byte count for the
&& lib.elem "multi-user.target" agentDefault.systemd.services.hyperhive-memory-cap-check.wantedBy; # container there is no fraction to take, and a hardcoded fallback
# would be a number about some other hive's machine — so the unit
# renders no ceiling at all rather than a fabricated one.
name = "an agent with no byte-valued memory cap renders no subagent ceiling";
ok = !((agentSubagentDaemon agentUncapped).serviceConfig ? MemoryHigh);
} }
{ {
# The sign is the whole property, and it is easy to write backwards: # The sign is the whole property, and it is easy to write backwards:
@ -66,8 +68,8 @@ let
name = "the OOM killer prefers a subagent over the agent's own session"; name = "the OOM killer prefers a subagent over the agent's own session";
ok = ok =
let let
sub = (agentSubagentDaemon agentDefault).serviceConfig.OOMScoreAdjust; sub = (agentSubagentDaemon agentUncapped).serviceConfig.OOMScoreAdjust;
own = (agentHarness agentDefault).serviceConfig.OOMScoreAdjust; own = (agentHarness agentUncapped).serviceConfig.OOMScoreAdjust;
in in
sub > 0 && own < 0 && sub > own; sub > 0 && own < 0 && sub > own;
} }
@ -79,13 +81,13 @@ let
# chose, which is the blast radius that made the preference a bad # chose, which is the blast radius that made the preference a bad
# trade in the first place. # trade in the first place.
name = "one subagent losing the OOM draw does not stop the daemon"; name = "one subagent losing the OOM draw does not stop the daemon";
ok = (agentSubagentDaemon agentDefault).serviceConfig.OOMPolicy == "continue"; ok = (agentSubagentDaemon agentUncapped).serviceConfig.OOMPolicy == "continue";
} }
{ {
# Each bash task is a process in the daemon's cgroup; under `stop` one # Each bash task is a process in the daemon's cgroup; under `stop` one
# OOM-killed task would end every running sibling task. # OOM-killed task would end every running sibling task.
name = "one bash task losing the OOM draw does not stop the daemon"; name = "one bash task losing the OOM draw does not stop the daemon";
ok = (agentBashDaemon agentDefault).serviceConfig.OOMPolicy or null == "continue"; ok = (agentBashDaemon agentUncapped).serviceConfig.OOMPolicy or null == "continue";
} }
]; ];
in in

View file

@ -1,44 +0,0 @@
# `checks.script-test-agent-memory-cap` — runs the four readers of
# `/run/hive/memory-max-bytes` in ../agent-modules/memory-cap.nix: the
# environment generator, the unit generator, the boot check that warns about
# an unusable cap, and the login-shell snippet `/etc/set-environment` sources.
# The cases are in ./agent-memory-cap.sh.
#
# What runs is the rendered scripts. The one edit is the cap file's path, moved
# under the build directory, and the snippet's reference to the environment
# generator, pointed at that edited copy.
{
pkgs,
lib,
self,
nixosSystem,
}:
let
inherit
(import ../module-eval/lib.nix {
inherit
pkgs
lib
self
nixosSystem
;
})
agentWith
;
machine = agentWith { };
in
pkgs.runCommand "hyperhive-script-test-agent-memory-cap"
{
ENV_GENERATOR =
machine.environment.etc."systemd/system-environment-generators/50-hyperhive-memory-cap".source;
UNIT_GENERATOR = machine.systemd.generators.hyperhive-memory-cap;
CHECK = machine.systemd.services.hyperhive-memory-cap-check.serviceConfig.ExecStart;
PROFILE = machine.environment.etc."hyperhive/memory-cap.sh".source;
EXTRA_INIT = machine.environment.extraInit;
SHELLS = "${pkgs.dash}/bin/dash ${pkgs.bash}/bin/sh";
}
''
${pkgs.bash}/bin/bash ${./agent-memory-cap.sh}
touch "$out"
''

View file

@ -1,116 +0,0 @@
# Cases for ./agent-memory-cap.nix. Each case writes a cap file (or none), runs
# both generators and the boot check with a clean environment and compares what
# they produced, then sources the login-shell snippet under each POSIX shell
# and compares what a child process sees.
set -uo pipefail
failures=0
count=0
fail() {
echo "FAILED: $case: $*" >&2
failures=$((failures + 1))
}
dir=$(mktemp -d)
cap=$dir/memory-max-bytes
sed "s|/run/hive/memory-max-bytes|$cap|g" "$ENV_GENERATOR" >"$dir/env-generator"
sed "s|/run/hive/memory-max-bytes|$cap|g" "$UNIT_GENERATOR" >"$dir/unit-generator"
sed "s|/run/hive/memory-max-bytes|$cap|g" "$CHECK" >"$dir/check"
chmod +x "$dir/env-generator" "$dir/unit-generator" "$dir/check"
sed "s|$ENV_GENERATOR|$dir/env-generator|g" "$PROFILE" >"$dir/profile"
envbin=$(command -v env)
case="setup"
for g in env-generator unit-generator check; do
if grep -q '/run/hive/memory-max-bytes' "$dir/$g"; then fail "$g still names /run/hive after the rewrite"; fi
if ! grep -qF "$cap" "$dir/$g"; then fail "the rewrite did not reach $g's read"; fi
done
if ! grep -qF "$dir/env-generator" "$dir/profile"; then fail "the snippet does not call the rewritten generator"; fi
if ! printf '%s\n' "$EXTRA_INIT" | grep -qxF '. /etc/hyperhive/memory-cap.sh'; then
fail "/etc/set-environment does not source the snippet"
fi
# check <description> <expected BUN_JSC_forceRAMSize or ""> <expected MemoryHigh or "">
# <why the boot check warns, or "">
check() {
case=$1
count=$((count + 1))
local want_env=${2:+BUN_JSC_forceRAMSize=$2} want_high=$3 want_why=${4:-} got
got=$(env -i "$dir/env-generator")
if [ "$got" != "$want_env" ]; then
fail "environment generator printed:"$'\n'"$got"$'\n'"wanted:"$'\n'"$want_env"
fi
local out=$dir/generator-out
rm -rf "$out"
mkdir -p "$out/normal" "$out/early" "$out/late"
env -i "$dir/unit-generator" "$out/normal" "$out/early" "$out/late"
local dropin=$out/normal/hive-subagent-daemon.service.d/50-memory-high.conf
if [ -z "$want_high" ]; then
got=$(find "$out" -type f)
if [ -n "$got" ]; then fail "unit generator wrote:"$'\n'"$got"; fi
else
got=$(cat "$dropin" 2>&1)
if [ "$got" != "[Service]"$'\n'"MemoryHigh=$want_high" ]; then
fail "unit generator's drop-in:"$'\n'"$got"
fi
fi
# The warning goes to stderr with the journal's `<4>` priority prefix, and
# nothing goes anywhere for a byte count or `infinity`.
got=$(env -i "$dir/check" 2>&1)
local want_warn=
if [ -n "$want_why" ]; then
want_warn="<4>$cap $want_why: no BUN_JSC_forceRAMSize heap ceiling and no MemoryHigh= on hive-subagent-daemon; the host's MemoryMax= on the container still applies."
fi
if [ "$got" != "$want_warn" ]; then
fail "boot check printed:"$'\n'"$got"$'\n'"wanted:"$'\n'"$want_warn"
fi
local sh
for sh in $SHELLS; do
# Only what the snippet exported reaches the child's environment. No PATH:
# the snippet must not need one.
got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(BUN_|LD_PRELOAD=|__hyperhive)')
if [ "$got" != "$want_env" ]; then
fail "$sh: a child of the sourcing shell saw:"$'\n'"$got"
fi
done
}
printf '8589934592\n' >"$cap"
check "an 8G cap" 6442450944 5726623061
printf '4294967296' >"$cap"
check "a line without a newline is kept" 3221225472 2863311530
printf '4294967296\nLD_PRELOAD=/x\n' >"$cap"
check "only the first line is read" 3221225472 2863311530
printf 'infinity\n' >"$cap"
check "infinity sets nothing and warns of nothing" "" "" ""
printf 'infinity' >"$cap"
check "infinity without a newline" "" "" ""
printf 'infinityx\n' >"$cap"
check "a word that only starts with infinity" "" "" "holds neither a byte count nor infinity"
printf '8G\n' >"$cap"
check "a size with a suffix is ignored" "" "" "holds neither a byte count nor infinity"
printf '0123\n' >"$cap"
check "a leading zero is ignored" "" "" "holds neither a byte count nor infinity"
printf '1234567890123456\n' >"$cap"
check "sixteen digits are ignored" "" "" "holds neither a byte count nor infinity"
: >"$cap"
check "an empty file" "" "" "holds neither a byte count nor infinity"
rm -f "$cap"
check "no file" "" "" "is missing"
echo "$count cases, $failures failed"
[ "$failures" -eq 0 ]