Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
Author SHA1 Message Date
atlas
fb5eb9edc1 agents: read every MemoryMax= size systemd enforces
`parse_bytes` knew only K/M/G/T, in either case. A size with the bare
`B` unit (`1048576B`), which `validate_memory_max` stores and systemd
enforces, became `infinity` in the agent's cap file: the cgroup was
capped while the agent was told there was no cap. It now takes exactly
the single-term forms systemd's `parse_size` does: no suffix, `B`, or
K/M/G/T/P/E, case-sensitive. Lowercase units and `4GB`, which systemd
refuses and so leaves the unit uncapped, read as `infinity` too.

A test runs every suffix form the validator accepts through it and
checks the stored value parses to the right byte count.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
eaca3c4eb3 agents: always write the memory cap, infinity for none
The cap file was removed for a cap of `infinity` or a RAM percentage,
so the boot check warned on every start of such an agent, for a
legitimate config. The file is now written on every start:

- a size: its byte count, as before;
- `infinity`: the literal `infinity`. The generators set no JSC heap
  ceiling and no subagent MemoryHigh=, and the boot check is quiet;
- a percentage: resolved on the host the way systemd resolves
  MemoryMax=N% for the container@ unit. That is permyriad (at most two
  decimals, at most 100%) of sysconf(_SC_PHYS_PAGES) *
  sysconf(_SC_PAGESIZE), rounded down to a page;
- a value systemd ignores (zero, over 100%, more than two decimals,
  unparseable) leaves the unit uncapped, so it is `infinity`.

The file is missing only when hive-c0re can't tell the cap: the
limits file and the kept drop-in are both unreadable, or a percentage
meets an unknown host memory size. The boot check now warns only for
a missing file or a first line that is neither a byte count nor
`infinity`.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
57955a6c53 agents: hand over the kept drop-in's cap, warn at boot without one
When hive-c0re can't read resource-limits.json and a limits drop-in
already exists, it keeps the drop-in, but it wrote no memory cap file. A
container whose runtime dir had no file yet (hive-c0re upgraded, /run
not cleared) then started with no heap ceiling and no subagent
MemoryHigh=, silently. `stage_limits` now writes the file on every
call: from the new limits, or from the last `MemoryMax=` in the kept
drop-in (`infinity` when it has none or can't be read).

A missing or malformed cap file at boot is no longer silent. The
generators run before the container's journal and an nspawn container
has no /dev/kmsg, so a oneshot `hyperhive-memory-cap-check.service`
logs a warning naming the path. The agent then runs with no JSC heap
ceiling and no subagent MemoryHigh=; the container's MemoryMax= still
applies.

module-eval-agent-memory gains a case pinning how the ceiling reaches
claude: `hive-agent` and `hive-subagent-daemon` neither set nor unset
BUN_JSC_forceRAMSize, both generators and the check unit are installed.

Part of #5081 (slice S0-2).
2026-10-12 10:59:12 +02:00
atlas
0d619e215e agents: read the memory cap at boot, not from the build
An agent's memory cap was rendered into its flake (`memoryMaxBytes` →
`services.hyperhive.agent.claudeMemoryMaxBytes`), so the closure depended
on it and a changed cap reached the JSC heap ceiling and the subagent
`MemoryHigh=` only after a rebuild.

hive-c0re now writes the effective `MemoryMax=` in bytes to the agent's
runtime dir (`/run/hive/memory-max-bytes` inside) in `write_dropins`,
from the same value the drop-in gets. The file is removed for
`infinity` or a RAM percentage, and left alone when the drop-in is kept.

Inside the container, `nix/agent-modules/memory-cap.nix` reads it at
boot: an environment generator sets `BUN_JSC_forceRAMSize` (75%) for
every unit and `machinectl shell`, `/etc/set-environment` exports it in
login shells, and a unit generator writes `MemoryHigh=` (2/3) as a
drop-in for `hive-subagent-daemon`.

The render drops `memoryMaxBytes`, the agent option is removed (both
spellings fail eval with a removal message), and
`effective_memory_bytes_from` goes with its only caller.

Part of #5081 (slice S0-2).
2026-10-12 10:59:11 +02:00
15 changed files with 769 additions and 295 deletions

View file

@ -403,7 +403,8 @@ Contents:
Written by `resource_limits::set_limits`; read where Written by `resource_limits::set_limits`; read where
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected `lifecycle::write_dropins` generates the systemd drop-in, **not** injected
into the container — these are host-side caps on the container, so into the container — these are host-side caps on the container, so
the capped party never sees or sets them. Fallback is per *field*: the capped party never sets them. It sees only the resulting memory cap,
as a byte count or `infinity` in its runtime dir (below). Fallback is per *field*:
an absent file, absent agent, or absent field falls back to the an absent file, absent agent, or absent field falls back to the
hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`, hive-wide `services.hyperhive.c0re.agentCpuQuota` / `agentMemoryMax`,
so an agent can override only its memory and still track the hive so an agent can override only its memory and still track the hive
@ -508,6 +509,12 @@ containers can't start.
- `/run/hyperhive/host.sock` — admin socket (host-side CLI). - `/run/hyperhive/host.sock` — admin socket (host-side CLI).
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket - `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
(bind-mounted into the container as `/run/hive/mcp.sock`). (bind-mounted into the container as `/run/hive/mcp.sock`).
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
`MemoryMax=` as a byte count (a percentage resolved against the host's
physical memory) or `infinity`, written by `lifecycle::write_dropins`
before every start. Absent only when hive-c0re can't tell the cap. The
container reads it at boot as `/run/hive/memory-max-bytes`
(`nix/agent-modules/memory-cap.nix`).
On startup, `Coordinator::register_agent` drops any prior socket On startup, `Coordinator::register_agent` drops any prior socket
task before rebinding — idempotent so a hive-c0re restart followed task before rebinding — idempotent so a hive-c0re restart followed

View file

@ -452,6 +452,23 @@ regardless.
rebuild, so changes take effect on the next lifecycle op without rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild. requiring a host rebuild.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the overrides file, the file carries
that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
`hive-subagent-daemon` (two thirds). Neither is in the agent's build, so a
changed cap reaches them at the next container start without a rebuild.
`infinity` sets neither. A missing or malformed file sets neither as well, and
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
The same drop-in carries `CPUWeight=` / `IOWeight=` from The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of `agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle setting: the quota and the memory max are **hard caps** that throttle

View file

@ -122,78 +122,35 @@ fn resolve(limits: &AgentLimits, hive_cpu_quota: &str, hive_memory_max: &str) ->
(cpu, mem) (cpu, mem)
} }
/// Effective `MemoryMax=` string for `name` against an already-loaded
/// map — the memory-only half of [`effective_from`], split out because
/// [`effective_memory_bytes_from`] doesn't have a CPU quota to pass.
/// No single-agent convenience wrapper (unlike [`effective`] /
/// [`effective_from`]): every current caller already has a loaded map
/// in hand, so one would just be dead code.
#[must_use]
fn effective_memory_max_from(
limits: &BTreeMap<String, AgentLimits>,
name: &str,
hive_memory_max: &str,
) -> String {
limits
.get(name)
.and_then(|l| l.memory_max.clone())
.unwrap_or_else(|| hive_memory_max.to_owned())
}
/// Effective `MemoryMax=` for `name`, as a raw byte count, against an
/// already-loaded map — the form `render_flake_with_lookup` uses so it
/// doesn't re-read `resource-limits.json` once per agent (same
/// reasoning as [`effective_from`] / `container_view::build_all`). No
/// single-agent convenience wrapper, for the same reason as
/// [`effective_memory_max_from`] — every current caller already has a
/// loaded map in hand.
///
/// Logs a `warn!` naming `name` when the effective value is a RAM
/// percentage: unlike `"infinity"`, a percentage IS a real, resolvable
/// cap — resolving one just needs host `MemTotal`, which this module
/// doesn't track — so today it degrades to `None` (no derived heap
/// ceiling) rather than silently guessing wrong against `MemTotal`.
#[must_use]
pub fn effective_memory_bytes_from(
limits: &BTreeMap<String, AgentLimits>,
name: &str,
hive_memory_max: &str,
) -> Option<u64> {
let mem = effective_memory_max_from(limits, name, hive_memory_max);
if mem.ends_with('%') {
tracing::warn!(
agent = name,
memory_max = %mem,
"effective MemoryMax= is a RAM percentage; can't derive a JSC heap ceiling from it \
without host MemTotal — leaving BUN_JSC_forceRAMSize unset for this agent"
);
}
parse_bytes(&mem)
}
/// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`, /// Parse a systemd `MemoryMax=`-style byte-size value (`"4G"`, `"512M"`,
/// a bare byte count, optionally with a decimal like `"1.5G"`) into a /// `"1048576B"`, a bare byte count, optionally with a decimal like `"1.5G"`)
/// raw byte count. systemd's `K`/`M`/`G`/`T` suffixes are IEC binary /// into a raw byte count, accepting exactly the single-term forms systemd's
/// (1024-based), not decimal — this matches. Pure integer arithmetic /// `parse_size` does: a `K`/`M`/`G`/`T`/`P`/`E` suffix, IEC binary
/// (1024-based), a `B` suffix, or none. Suffixes are case-sensitive, and a
/// redundant `B` after a multiplier (`4GB`) is rejected, as systemd rejects
/// both and then leaves the unit uncapped. Pure integer arithmetic
/// throughout (via `u128` headroom) rather than `f64`: a `MemoryMax=` /// throughout (via `u128` headroom) rather than `f64`: a `MemoryMax=`
/// value is always a small non-negative decimal (enforced by /// value is always a small non-negative decimal (enforced by
/// [`is_plain_number`] upstream in [`validate_memory_max`]), so floats /// [`is_plain_number`] upstream in [`validate_memory_max`]), so floats
/// would only add rounding / sign-loss risk for no benefit. Returns /// would only add rounding / sign-loss risk for no benefit. Returns
/// `None` for `"infinity"` and percentages; see /// `None` for `"infinity"` and for percentages, which need the host's
/// [`effective_memory_bytes_from`] for why those can't be turned into a /// `MemTotal` to resolve.
/// byte count here.
#[must_use] #[must_use]
pub fn parse_bytes(value: &str) -> Option<u64> { pub fn parse_bytes(value: &str) -> Option<u64> {
if value == "infinity" || value.ends_with('%') { if value == "infinity" || value.ends_with('%') {
return None; return None;
} }
let (mantissa, exponent) = [('K', 1u32), ('M', 2), ('G', 3), ('T', 4)] let (mantissa, exponent) = [
('E', 6u32),
('P', 5),
('T', 4),
('G', 3),
('M', 2),
('K', 1),
('B', 0),
]
.into_iter() .into_iter()
.find_map(|(suffix, exp)| { .find_map(|(suffix, exp)| value.strip_suffix(suffix).map(|m| (m, exp)))
value
.strip_suffix([suffix, suffix.to_ascii_lowercase()])
.map(|m| (m, exp))
})
.unwrap_or((value, 0)); .unwrap_or((value, 0));
let scale = u128::from(1024u64.checked_pow(exponent)?); let scale = u128::from(1024u64.checked_pow(exponent)?);
let (int_part, frac_part) = mantissa.split_once('.').unwrap_or((mantissa, "")); let (int_part, frac_part) = mantissa.split_once('.').unwrap_or((mantissa, ""));
@ -516,8 +473,14 @@ mod tests {
parse_bytes("1.5G"), parse_bytes("1.5G"),
Some(1024 * 1024 * 1024 + 512 * 1024 * 1024) Some(1024 * 1024 * 1024 + 512 * 1024 * 1024)
); );
// Lowercase suffixes accepted, matching `normalized_byte_size`. assert_eq!(parse_bytes("1048576B"), Some(1_048_576));
assert_eq!(parse_bytes("4g"), Some(4 * 1024 * 1024 * 1024)); assert_eq!(parse_bytes("3P"), Some(3 << 50));
assert_eq!(parse_bytes("2E"), Some(2 << 60));
assert_eq!(
parse_bytes("16E"),
None,
"past u64, which systemd refuses too"
);
} }
#[test] #[test]
@ -528,50 +491,38 @@ mod tests {
#[test] #[test]
fn parse_bytes_rejects_garbage() { fn parse_bytes_rejects_garbage() {
for v in ["", "abc", "-8G", "8Gi"] { // systemd's suffixes are case-sensitive and take no `B` after a
// multiplier, so it leaves a unit with any of these uncapped.
for v in [
"", "abc", "-8G", "8Gi", "4g", "4k", "1024b", "4GB", "4MB", "B", ".5G",
] {
assert_eq!(parse_bytes(v), None, "{v} should not parse"); assert_eq!(parse_bytes(v), None, "{v} should not parse");
} }
} }
#[test] #[test]
fn effective_memory_bytes_falls_back_to_hive_default() { fn every_size_the_validator_stores_has_a_byte_count() {
// Empty map, i.e. no per-agent override — exercises the for (input, bytes) in [
// "use the hive-wide default" arm end to end. ("1048576", 1_048_576),
let empty = BTreeMap::new(); ("1048576B", 1_048_576),
("1048576b", 1_048_576),
("4096K", 4096 << 10),
("4096kb", 4096 << 10),
("512M", 512 << 20),
("512mB", 512 << 20),
("4G", 4 << 30),
("4g", 4 << 30),
("4GB", 4 << 30),
("1.5G", 3 << 29),
("2T", 2 << 40),
("2tb", 2 << 40),
] {
let stored = validate_memory_max(input).expect(input);
assert_eq!( assert_eq!(
effective_memory_bytes_from(&empty, "nobody-configured-this-agent", "4G"), parse_bytes(&stored),
Some(4 * 1024 * 1024 * 1024) Some(bytes),
); "{input} stored as {stored}"
assert_eq!(
effective_memory_bytes_from(&empty, "nobody-configured-this-agent", "infinity"),
None
); );
} }
#[test]
fn effective_memory_bytes_from_prefers_per_agent_override() {
let map = BTreeMap::from([("sock".to_owned(), limits(None, Some("8G")))]);
assert_eq!(
effective_memory_bytes_from(&map, "sock", HIVE_MEM),
Some(8 * 1024 * 1024 * 1024)
);
// A different agent not in the map still falls back to the
// hive-wide default from the same loaded map (no re-read).
assert_eq!(
effective_memory_bytes_from(&map, "iris", HIVE_MEM),
Some(4 * 1024 * 1024 * 1024)
);
}
/// A percentage cap is real and resolvable in principle, but this
/// module has no host `MemTotal` to resolve it against — must
/// degrade to `None` (no derived heap ceiling) rather than silently
/// treating it as unbounded or guessing a number. `warn!` firing is
/// exercised for coverage but not asserted on (no tracing test
/// subscriber wired up here) — the `None` return is the contract.
#[test]
fn effective_memory_bytes_from_returns_none_for_percentage() {
let map = BTreeMap::from([("sock".to_owned(), limits(None, Some("50%")))]);
assert_eq!(effective_memory_bytes_from(&map, "sock", HIVE_MEM), None);
} }
} }

View file

@ -1,6 +1,7 @@
//! Per-container host-side config: the nspawn conf rewrite (bind mounts, //! Per-container host-side config: the nspawn conf rewrite (bind mounts,
//! network isolation, forwarded credentials), the systemd resource-limits //! network isolation, forwarded credentials), the systemd resource-limits
//! drop-in, and the `write_dropins` verb that re-applies both. //! drop-in, the agent's copy of its memory cap, and the `write_dropins` verb
//! that re-applies them.
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@ -16,20 +17,22 @@ use super::{
}; };
/// Re-apply the per-container host-side config: nspawn flags (bind /// Re-apply the per-container host-side config: nspawn flags (bind
/// mounts etc.), the systemd resource-limits drop-in, and a daemon /// mounts etc.), the systemd resource-limits drop-in, the agent's copy of
/// reload so both take effect on the next unit (re)start. Idempotent — /// its memory cap, and a daemon reload so all of it takes effect on the next
/// the job queue's `WriteDropin` node, also folded into every `Swap` /// unit (re)start. Idempotent — the job queue's `WriteDropin` node, also
/// (rebuild is the reconcile verb). /// folded into every `Swap` (rebuild is the reconcile verb).
pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> { pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
validate(name)?; validate(name)?;
let container = container_name(name); let container = container_name(name);
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?; set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?;
if let Some((cpu_quota, memory_max)) = limits_to_apply( if let Some((cpu_quota, memory_max)) = stage_limits(
name, name,
&crate::resource_limits::resource_limits_path(), &crate::resource_limits::resource_limits_path(),
&limits_dropin_path(&container), &limits_dropin_path(&container),
&paths.agent,
hive, hive,
) { HostMemory::read(),
)? {
set_resource_limits( set_resource_limits(
&container, &container,
&cpu_quota, &cpu_quota,
@ -51,6 +54,173 @@ fn limits_dropin_path(container: &str) -> PathBuf {
)) ))
} }
/// File in the agent's runtime dir, `/run/hive/` in the container, holding the
/// `MemoryMax=` its drop-in carries: a byte count, or `infinity` for no cap.
/// `nix/agent-modules/memory-cap.nix` reads it at boot under this name, and
/// treats a missing file as a fault.
const MEMORY_CAP_FILE: &str = "memory-max-bytes";
/// The host's physical memory and page size, the two numbers systemd resolves
/// a `MemoryMax=` percentage against.
#[derive(Clone, Copy, Debug)]
struct HostMemory {
bytes: u64,
page: u64,
}
impl HostMemory {
/// `sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE)`, as systemd's
/// `physical_memory()` computes it. systemd also caps that by the root
/// cgroup's `memory.max`, which the host's root cgroup does not have.
fn read() -> Option<Self> {
// SAFETY: `sysconf(3)` only reads system configuration.
let (pages, page) = unsafe {
(
libc::sysconf(libc::_SC_PHYS_PAGES),
libc::sysconf(libc::_SC_PAGESIZE),
)
};
let pages = u64::try_from(pages).ok().filter(|&p| p > 0)?;
let page = u64::try_from(page).ok().filter(|&p| p > 0)?;
Some(Self {
bytes: pages.checked_mul(page)?,
page,
})
}
}
/// [`limits_to_apply`], plus handing the memory cap to the agent through
/// [`MEMORY_CAP_FILE`] in `runtime_dir`. The container cannot read the cap
/// systemd enforces on it, so this file is how it learns the value it sizes
/// its heap and its subagents against. It is written on every call, from the
/// `MemoryMax=` the agent will start under: the new one, or the kept drop-in's.
fn stage_limits(
agent_name: &str,
limits_path: &Path,
dropin_path: &Path,
runtime_dir: &Path,
hive: &HiveEnv,
host: Option<HostMemory>,
) -> Result<Option<(String, String)>> {
let limits = limits_to_apply(agent_name, limits_path, dropin_path, hive);
let memory_max = match &limits {
Some((_, memory_max)) => Some(memory_max.clone()),
None => dropin_memory_max(agent_name, dropin_path),
};
let cap = memory_max.and_then(|m| memory_cap_value(agent_name, &m, host));
write_memory_cap(runtime_dir, cap.as_deref())?;
Ok(limits)
}
/// The `MemoryMax=` the drop-in at `path` applies: its last assignment, as
/// systemd reads it, or `infinity`, systemd's default, when it has none or its
/// last one is empty. `None` when the file can't be read.
fn dropin_memory_max(agent_name: &str, path: &Path) -> Option<String> {
let body = match std::fs::read_to_string(path) {
Ok(body) => body,
Err(e) => {
tracing::warn!(
agent = %agent_name,
path = %path.display(),
error = ?e,
"limits drop-in unreadable; the agent gets no memory cap"
);
return None;
}
};
let value = body
.lines()
.rev()
.find_map(|line| line.trim().strip_prefix("MemoryMax="))
.map(str::trim)
.filter(|v| !v.is_empty())
.unwrap_or("infinity");
Some(value.to_owned())
}
/// What [`MEMORY_CAP_FILE`] says for `memory_max`, read the way systemd reads
/// `MemoryMax=`: a size is its byte count, and a percentage is that share of
/// `host`'s physical memory, rounded down to a page. A value systemd ignores
/// (zero, over 100%, more than two decimals, unparseable) leaves the unit
/// uncapped, so it is `infinity` here. `None` for a percentage when `host` is
/// unknown.
fn memory_cap_value(
agent_name: &str,
memory_max: &str,
host: Option<HostMemory>,
) -> Option<String> {
let bytes = if let Some(percent) = memory_max.strip_suffix('%') {
let Some(host) = host else {
tracing::warn!(
agent = agent_name,
memory_max,
"host physical memory unknown; the agent gets no memory cap"
);
return None;
};
permyriad(percent).and_then(|p| scale_physical_memory(host, p))
} else {
crate::resource_limits::parse_bytes(memory_max)
};
Some(
bytes
.filter(|&b| b > 0)
.map_or_else(|| "infinity".to_owned(), |b| b.to_string()),
)
}
/// `percent` (the part before `%`) in hundredths of a percent, with systemd's
/// `parse_permyriad` rules: digits, at most two decimals, at most 100%.
fn permyriad(percent: &str) -> Option<u64> {
let (int, frac) = percent.split_once('.').unwrap_or((percent, ""));
if int.is_empty() || !int.bytes().all(|b| b.is_ascii_digit()) {
return None;
}
let hundredths = match frac.as_bytes() {
[] if !percent.contains('.') => 0,
[d] if d.is_ascii_digit() => u64::from(d - b'0') * 10,
[d1, d2] if d1.is_ascii_digit() && d2.is_ascii_digit() => {
u64::from(d1 - b'0') * 10 + u64::from(d2 - b'0')
}
_ => return None,
};
let value = int
.parse::<u64>()
.ok()?
.checked_mul(100)?
.checked_add(hundredths)?;
(value <= 10_000).then_some(value)
}
/// systemd's `physical_memory_scale(permyriad, 10000)`: whole pages, rounded
/// down, and the exact total at 100%.
fn scale_physical_memory(host: HostMemory, permyriad: u64) -> Option<u64> {
if permyriad == 10_000 {
return Some(host.bytes);
}
let pages = u128::from(host.bytes / host.page) * u128::from(permyriad) / 10_000;
u64::try_from(pages * u128::from(host.page)).ok()
}
/// Write `cap` into [`MEMORY_CAP_FILE`], or remove the file when the cap is
/// unknown, which the agent reports at boot.
fn write_memory_cap(runtime_dir: &Path, cap: Option<&str>) -> Result<()> {
let path = runtime_dir.join(MEMORY_CAP_FILE);
let Some(cap) = cap else {
return match std::fs::remove_file(&path) {
Err(e) if e.kind() != std::io::ErrorKind::NotFound => {
Err(e).with_context(|| format!("remove {}", path.display()))
}
_ => Ok(()),
};
};
std::fs::create_dir_all(runtime_dir)
.with_context(|| format!("create agent runtime dir {}", runtime_dir.display()))?;
let tmp = runtime_dir.join(format!(".{MEMORY_CAP_FILE}.tmp"));
std::fs::write(&tmp, format!("{cap}\n")).with_context(|| format!("write {}", tmp.display()))?;
std::fs::rename(&tmp, &path).with_context(|| format!("rename onto {}", path.display()))
}
/// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits /// The `(CPUQuota, MemoryMax)` to write into `agent_name`'s limits
/// drop-in, or `None` to leave the drop-in at `dropin_path` as it is. /// drop-in, or `None` to leave the drop-in at `dropin_path` as it is.
/// ///
@ -489,8 +659,10 @@ async fn set_nspawn_flags(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::{
BindMount, QUEUE_CLIENT_ID_CREDENTIAL, QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs, BindMount, HostMemory, MEMORY_CAP_FILE, QUEUE_CLIENT_ID_CREDENTIAL,
bind_own_agent_dirs, holds_manage_root_agent, limits_to_apply, queue_agent_credentials, QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs, bind_own_agent_dirs,
holds_manage_root_agent, limits_to_apply, memory_cap_value, queue_agent_credentials,
stage_limits,
}; };
use crate::coordinator::HiveEnv; use crate::coordinator::HiveEnv;
@ -707,4 +879,169 @@ mod tests {
let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default()); let applied = limits_to_apply("sock", &limits, &dropin, &HiveEnv::default());
assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned()))); assert_eq!(applied, Some(("200%".to_owned(), "1G".to_owned())));
} }
/// 4 000 003 pages of 4 KiB: odd, so rounding to a page shows.
const HOST: HostMemory = HostMemory {
bytes: 4_000_003 * 4096,
page: 4096,
};
/// Runs [`stage_limits`] against the files in `dir` and returns the
/// drop-in's `MemoryMax=` and the agent's copy of it, `None` when absent.
fn stage(dir: &tempfile::TempDir) -> (Option<String>, Option<String>) {
let runtime = dir.path().join("runtime");
let applied = stage_limits(
"sock",
&dir.path().join("resource-limits.json"),
&dir.path().join("hyperhive-limits.conf"),
&runtime,
&HiveEnv::default(),
Some(HOST),
)
.expect("stage limits");
let copy = std::fs::read_to_string(runtime.join(MEMORY_CAP_FILE)).ok();
(applied.map(|(_, mem)| mem), copy)
}
#[test]
fn the_agent_reads_the_cap_its_dropin_carries() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
// No overrides anywhere: the hive-wide default, 4G.
assert_eq!(
stage(&dir),
(Some("4G".to_owned()), Some("4294967296\n".to_owned()))
);
std::fs::write(&limits, r#"{"sock": {"memory_max": "16G"}}"#).expect("seed limits");
assert_eq!(
stage(&dir),
(Some("16G".to_owned()), Some("17179869184\n".to_owned()))
);
}
#[test]
fn every_kind_of_cap_is_written_for_the_agent() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
for (memory_max, copy) in [
("infinity", "infinity\n"),
("50%", "8192004096\n"),
("8G", "8589934592\n"),
] {
std::fs::write(
&limits,
format!(r#"{{"sock": {{"memory_max": "{memory_max}"}}}}"#),
)
.expect("seed limits");
assert_eq!(
stage(&dir),
(Some(memory_max.to_owned()), Some(copy.to_owned())),
"{memory_max}"
);
}
}
#[test]
fn a_cap_is_read_the_way_systemd_reads_memory_max() {
for (memory_max, cap) in [
("4G", "4294967296"),
("1048576B", "1048576"),
("3P", "3377699720527872"),
("infinity", "infinity"),
("50%", "8192004096"),
("33.33%", "5460787200"),
("12.5%", "2048000000"),
("100%", "16384012288"),
// systemd ignores these and the unit runs uncapped.
("0", "infinity"),
("4g", "infinity"),
("4GB", "infinity"),
("0%", "infinity"),
("150%", "infinity"),
("12.345%", "infinity"),
("lots", "infinity"),
] {
assert_eq!(
memory_cap_value("sock", memory_max, Some(HOST)).as_deref(),
Some(cap),
"{memory_max}"
);
}
assert_eq!(memory_cap_value("sock", "50%", None), None);
assert_eq!(
memory_cap_value("sock", "8G", None).as_deref(),
Some("8589934592")
);
}
#[test]
fn the_host_memory_systemd_would_read_is_found() {
let host = HostMemory::read().expect("sysconf");
assert!(
host.page >= 4096 && host.bytes.is_multiple_of(host.page),
"{host:?}"
);
}
#[test]
fn a_kept_dropin_hands_the_agent_its_cap() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
let dropin = dir.path().join("hyperhive-limits.conf");
let copy = dir.path().join("runtime").join(MEMORY_CAP_FILE);
std::fs::write(&limits, r#"{"sock": {"memory_max": "1G"}}"#).expect("seed limits");
assert_eq!(stage(&dir).1, Some("1073741824\n".to_owned()));
// Unreadable limits keep the drop-in; the copy follows the drop-in.
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
std::fs::write(
&dropin,
"[Unit]\nConditionPathIsDirectory=/rt/sock\n\n[Service]\nRestartSec=5\nMemoryMax=2G\nCPUQuota=200%\n",
)
.expect("seed drop-in");
assert_eq!(stage(&dir), (None, Some("2147483648\n".to_owned())));
// A kept drop-in with no copy beside it, as after an upgrade.
std::fs::remove_file(&copy).expect("remove copy");
assert_eq!(stage(&dir), (None, Some("2147483648\n".to_owned())));
// The last assignment is the one systemd applies.
std::fs::write(&dropin, "[Service]\nMemoryMax=2G\nMemoryMax=3G\n").expect("seed drop-in");
assert_eq!(stage(&dir), (None, Some("3221225472\n".to_owned())));
// No `MemoryMax=` is systemd's default.
std::fs::write(&dropin, "[Service]\nCPUQuota=200%\n").expect("seed drop-in");
assert_eq!(stage(&dir), (None, Some("infinity\n".to_owned())));
}
#[test]
fn an_unknown_cap_leaves_no_copy() {
let dir = tempfile::tempdir().expect("tempdir");
let limits = dir.path().join("resource-limits.json");
let dropin = dir.path().join("hyperhive-limits.conf");
assert_eq!(stage(&dir).1, Some("4294967296\n".to_owned()));
// Unreadable limits and an unreadable drop-in: nothing says what the
// cgroup enforces.
std::fs::write(&limits, TRUNCATED_LIMITS).expect("seed limits");
std::fs::create_dir(&dropin).expect("seed unreadable drop-in");
assert_eq!(stage(&dir), (None, None));
// A percentage with no host memory to resolve it against.
std::fs::write(&limits, r#"{"sock": {"memory_max": "50%"}}"#).expect("seed limits");
let runtime = dir.path().join("runtime");
let applied = stage_limits(
"sock",
&limits,
&dropin,
&runtime,
&HiveEnv::default(),
None,
)
.expect("stage limits");
assert_eq!(applied.map(|(_, mem)| mem), Some("50%".to_owned()));
assert!(!runtime.join(MEMORY_CAP_FILE).exists());
}
} }

View file

@ -88,17 +88,7 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> {
let dir = crate::paths::meta_root(); let dir = crate::paths::meta_root();
std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?; std::fs::create_dir_all(&dir).with_context(|| format!("create {}", dir.display()))?;
let new_flake = render_flake( let new_flake = render_hive_flake(hive, agents)?;
&hive.hyperhive_flake,
&hive.hyperhive_docs_flake,
&hive.nixpkgs_flake,
hive.claude_code_path.as_deref(),
hive.dashboard_port,
&hive.operator_pronouns,
&hive.context_window_tokens,
&hive.agent_memory_max,
agents,
)?;
let flake_path = dir.join("flake.nix"); let flake_path = dir.join("flake.nix");
let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default(); let on_disk = std::fs::read_to_string(&flake_path).unwrap_or_default();
let initial = !dir.join(".git").exists(); let initial = !dir.join(".git").exists();
@ -444,6 +434,21 @@ pub async fn commit_perms(
Ok(()) Ok(())
} }
/// [`render_flake`] with its inputs taken from `hive`, as [`sync_agents`]
/// renders it.
fn render_hive_flake(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<String> {
render_flake(
&hive.hyperhive_flake,
&hive.hyperhive_docs_flake,
&hive.nixpkgs_flake,
hive.claude_code_path.as_deref(),
hive.dashboard_port,
&hive.operator_pronouns,
&hive.context_window_tokens,
agents,
)
}
#[allow( #[allow(
clippy::too_many_arguments, clippy::too_many_arguments,
reason = "many genuine flake inputs (source flakes, port, pronouns, tokens, \ reason = "many genuine flake inputs (source flakes, port, pronouns, tokens, \
@ -457,7 +462,6 @@ fn render_flake(
dashboard_port: u16, dashboard_port: u16,
operator_pronouns: &str, operator_pronouns: &str,
context_window_tokens: &std::collections::HashMap<String, u64>, context_window_tokens: &std::collections::HashMap<String, u64>,
hive_memory_max: &str,
agents: &[AgentSpec], agents: &[AgentSpec],
) -> Result<String> { ) -> Result<String> {
render_flake_with_lookup( render_flake_with_lookup(
@ -468,7 +472,6 @@ fn render_flake(
dashboard_port, dashboard_port,
operator_pronouns, operator_pronouns,
context_window_tokens, context_window_tokens,
hive_memory_max,
agents, agents,
agent_canonical_inputs, agent_canonical_inputs,
) )
@ -861,7 +864,6 @@ fn render_flake_with_lookup<F>(
dashboard_port: u16, dashboard_port: u16,
operator_pronouns: &str, operator_pronouns: &str,
context_window_tokens: &std::collections::HashMap<String, u64>, context_window_tokens: &std::collections::HashMap<String, u64>,
hive_memory_max: &str,
agents: &[AgentSpec], agents: &[AgentSpec],
lookup: F, lookup: F,
) -> Result<String> ) -> Result<String>
@ -943,7 +945,7 @@ where
let pronouns_escaped = operator_pronouns.replace('\\', "\\\\").replace('"', "\\\""); let pronouns_escaped = operator_pronouns.replace('\\', "\\\\").replace('"', "\\\"");
let _ = writeln!( let _ = writeln!(
out, out,
" dashboardPort = {dashboard_port};\n operatorPronouns = \"{pronouns_escaped}\";\n mkAgent = {{ name, isManager, port, toolGroups ? null, capabilities ? null, memoryMaxBytes ? null }}:" " dashboardPort = {dashboard_port};\n operatorPronouns = \"{pronouns_escaped}\";\n mkAgent = {{ name, isManager, port, toolGroups ? null, capabilities ? null }}:"
); );
out.push_str( out.push_str(
r#" let r#" let
@ -1094,7 +1096,6 @@ where
# nspawn itself already call this container, so logs join to # nspawn itself already call this container, so logs join to
# both without a prefix transform. # both without a prefix transform.
networking.hostName = "h-${name}"; networking.hostName = "h-${name}";
services.hyperhive.agent.claudeMemoryMaxBytes = memoryMaxBytes;
programs.git.config.user = { programs.git.config.user = {
name = name; name = name;
email = "${name}@hyperhive.local"; email = "${name}@hyperhive.local";
@ -1177,12 +1178,9 @@ where
); );
// Propagated, never read as empty: an agent with no tool-groups entry // Propagated, never read as empty: an agent with no tool-groups entry
// renders `toolGroups = null` and gets `AGENT_DEFAULT`, which fails // renders `toolGroups = null` and gets `AGENT_DEFAULT`, which fails
// open for any agent whose explicit entry is narrower. Likewise an // open for any agent whose explicit entry is narrower.
// agent with no resource-limits entry gets the hive `memoryMaxBytes`,
// a larger heap ceiling than a tighter override.
let tool_groups_map = crate::tool_groups::read()?; let tool_groups_map = crate::tool_groups::read()?;
let capabilities_map = crate::capabilities::read()?; let capabilities_map = crate::capabilities::read()?;
let resource_limits_map = crate::resource_limits::read()?;
for spec in agents { for spec in agents {
// Emit `toolGroups = "group1,group2"` when the operator has // Emit `toolGroups = "group1,group2"` when the operator has
// explicitly configured groups for this agent. Absent entry = null // explicitly configured groups for this agent. Absent entry = null
@ -1208,29 +1206,15 @@ where
let joined = caps.join(","); let joined = caps.join(",");
format!("\"{joined}\"") format!("\"{joined}\"")
}; };
// Effective `MemoryMax=` for this agent (per-agent override, else
// the hive-wide default), turned into a raw byte count so
// the agent's claude-settings module can derive a JSC heap ceiling
// (see `services.hyperhive.agent.claudeMemoryMaxBytes`). `null` when the
// effective value is `"infinity"` or a RAM percentage — no
// byte count to derive, dependent env var stays unset, same as
// today's no-cap behavior.
let memory_max_attr = crate::resource_limits::effective_memory_bytes_from(
&resource_limits_map,
&spec.name,
hive_memory_max,
)
.map_or_else(|| "null".to_owned(), |b| b.to_string());
let _ = writeln!( let _ = writeln!(
out, out,
" {} = mkAgent {{ name = \"{}\"; isManager = {}; port = {}; toolGroups = {}; capabilities = {}; memoryMaxBytes = {}; }};", " {} = mkAgent {{ name = \"{}\"; isManager = {}; port = {}; toolGroups = {}; capabilities = {}; }};",
spec.name, spec.name,
spec.name, spec.name,
if spec.is_manager { "true" } else { "false" }, if spec.is_manager { "true" } else { "false" },
spec.port, spec.port,
tool_groups_attr, tool_groups_attr,
capabilities_attr, capabilities_attr,
memory_max_attr,
); );
} }
out.push_str(" };\n };\n}\n"); out.push_str(" };\n };\n}\n");
@ -1468,7 +1452,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1516,7 +1499,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1536,7 +1518,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1578,7 +1559,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1608,7 +1588,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1630,7 +1609,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1662,7 +1640,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[ &[
sample_spec("argus", false, 9001), sample_spec("argus", false, 9001),
sample_spec("bitburner", false, 9002), sample_spec("bitburner", false, 9002),
@ -1697,7 +1674,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
|_| Vec::new(), |_| Vec::new(),
) )
@ -1737,7 +1713,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -1784,7 +1759,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -1966,7 +1940,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -2004,7 +1977,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake"); .expect("render flake");
@ -2050,7 +2022,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -2144,7 +2115,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -2220,7 +2190,6 @@ mod tests {
8000, 8000,
"she/her", "she/her",
&std::collections::HashMap::new(), &std::collections::HashMap::new(),
"4G",
&[sample_spec("alice", false, 9001)], &[sample_spec("alice", false, 9001)],
) )
.expect("render flake") .expect("render flake")
@ -2246,32 +2215,20 @@ mod tests {
); );
} }
/// The JSC-heap-ceiling fix needs the effective per-agent memory cap /// The agent reads its memory cap at boot from the copy
/// threaded into the flake as raw bytes, since nspawn hides the real /// `lifecycle::write_dropins` hands it, so the cap is no build input.
/// cgroup cap from inside the container. An
/// agent with no `resource-limits.json` override falls back to the
/// hive-wide default passed to `render_flake` (`"4G"` in every test
/// in this module) — this locks in the byte-count conversion end to
/// end through the actual render path (not just `parse_bytes`
/// in isolation).
#[test] #[test]
fn render_flake_derives_memory_max_bytes_from_hive_default() { fn render_hive_flake_is_identical_across_memory_caps() {
let out = render_flake( let hive = |cap: &str| HiveEnv {
"github:example/hyperhive", agent_memory_max: cap.to_owned(),
"path:/nix/store/bbbb-hyperhive-docs-source", ..HiveEnv::default()
"path:/nix/store/aaaa-nixpkgs-source", };
None, let agents = [sample_spec("alice", false, 9001)];
8000, let small = render_hive_flake(&hive("4G"), &agents).expect("render 4G");
"she/her", let large = render_hive_flake(&hive("16G"), &agents).expect("render 16G");
&std::collections::HashMap::new(), assert_eq!(
"4G", small, large,
&[sample_spec("alice", false, 9001)], "the rendered flake must not depend on the memory cap"
)
.expect("render flake");
let want_bytes = 4u64 * 1024 * 1024 * 1024;
assert!(
out.contains(&format!("memoryMaxBytes = {want_bytes};")),
"memoryMaxBytes must reflect the hive-wide default in bytes:\n{out}"
); );
} }
} }

View file

@ -433,10 +433,9 @@ async fn handle_set_resource_limits(
&hive.agent_memory_max, &hive.agent_memory_max,
)?; )?;
Ok(HostResponse::messages(vec![format!( Ok(HostResponse::messages(vec![format!(
"{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now (restart \ "{name}: CPUQuota={cpu} MemoryMax={mem} — the cgroup cap itself is live now; the \
the container if it's running and needs the new cap immediately), but the derived \ Claude/JSC heap ceiling and the subagent MemoryHigh= derived from it follow at the \
Claude/JSC heap ceiling is baked in at build time, so it needs a REBUILD \ container's next start (restart it if it's running and needs them now)"
(`hivectl agent {name} rebuild`) to actually track this change"
)])) )]))
} }

View file

@ -19,11 +19,6 @@ let
# meta.rs `otel_config`). Options declared in `otel.nix`, which also # meta.rs `otel_config`). Options declared in `otel.nix`, which also
# exports the generic OTLP environment this container's producers read. # exports the generic OTLP environment this container's producers read.
otelCfg = config.services.hyperhive.agent.otel; otelCfg = config.services.hyperhive.agent.otel;
# Effective per-agent MemoryMax=, in bytes, injected by meta.rs's
# per-agent flake render (`services.hyperhive.agent.claudeMemoryMaxBytes`). `null`
# when the effective cap is unbounded ("infinity") or a RAM
# percentage — see `resource_limits::effective_memory_bytes`.
memoryMaxBytes = config.services.hyperhive.agent.claudeMemoryMaxBytes;
# Base claude-code environment applied to every agent regardless of OTEL. # Base claude-code environment applied to every agent regardless of OTEL.
# Shipped via the managed settings `env` block so claude and `hivectl # Shipped via the managed settings `env` block so claude and `hivectl
# choom` both inherit them without a launch wrapper. # choom` both inherit them without a launch wrapper.
@ -47,19 +42,6 @@ let
CLAUDE_CODE_RESUME_INTERRUPTED_TURN = "1"; CLAUDE_CODE_RESUME_INTERRUPTED_TURN = "1";
# Use the simpler system prompt variant suited to headless operation. # Use the simpler system prompt variant suited to headless operation.
CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT = "1"; CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT = "1";
}
# Bun/JavaScriptCore (the runtime under claude-code's `.claude-wrapped`
# binary) sizes its default heap ceiling off the container's visible
# `/proc/meminfo`, which under nspawn is the *host's* physical RAM, not
# the systemd `MemoryMax=` cgroup cap actually enforced on this
# container — the cap lives on the host's outer cgroup and is invisible
# from inside (confirmed: `/sys/fs/cgroup/memory.max` reads `max` at
# every level from in here). That mismatch lets JSC's heap grow well
# past the real wall before GC kicks in hard, causing choom sessions to
# hang. Pin JSC's ceiling to 75% of the *actual* effective cap instead,
# once it's known at build time.
// lib.optionalAttrs (memoryMaxBytes != null) {
BUN_JSC_forceRAMSize = toString (memoryMaxBytes * 75 / 100);
}; };
# Claude Code's own telemetry switches — what it emits, and whether it # Claude Code's own telemetry switches — what it emits, and whether it
# emits at all. Everything an OTEL SDK reads generically (endpoint, # emits at all. Everything an OTEL SDK reads generically (endpoint,
@ -99,31 +81,6 @@ in
''; '';
}; };
# Build-time implementation surface for the JSC-heap-ceiling fix:
# meta.rs's per-agent flake render injects this from the effective
# `MemoryMax=` (per-agent `resource-limits.json` override, else the
# hive-wide `services.hyperhive.agentMemoryMax`) — see
# `resource_limits::effective_memory_bytes_from`. Not meant to be set
# directly in an agent.nix, same convention as `services.hyperhive.agent.otel.*`
# above; the host option (or `hivectl agent <name> set-limits`) is
# the real operator knob, and this only reflects the value baked in at
# the agent's *last rebuild* — `set-limits` still applies the
# cgroup cap live via a drop-in reload, but this derived heap ceiling
# needs a rebuild to pick up a new value.
options.services.hyperhive.agent.claudeMemoryMaxBytes = lib.mkOption {
type = lib.types.nullOr lib.types.ints.positive;
default = null;
internal = true;
description = ''
Effective per-agent memory cap in bytes, when it's a plain
byte-size value (null for an unbounded or percentage-based cap).
This is the whole container's cap, not claude's share of it —
the name records its first consumer, `BUN_JSC_forceRAMSize` in
`baseClaudeEnv`. `mcp.nix` reads it too, to size the subagent
daemon's `MemoryHigh=` against the container it runs in.
'';
};
options.services.hyperhive.agent.claudeMarketplaces = lib.mkOption { options.services.hyperhive.agent.claudeMarketplaces = lib.mkOption {
type = lib.types.listOf lib.types.str; type = lib.types.listOf lib.types.str;
default = [ default = [
@ -257,9 +214,7 @@ in
environment.etc."claude-code/managed-settings.json".source = environment.etc."claude-code/managed-settings.json".source =
let let
baseSettings = "${config.services.hyperhive.agent.packages.assets}/share/hyperhive/prompts/claude-settings.json"; baseSettings = "${config.services.hyperhive.agent.packages.assets}/share/hyperhive/prompts/claude-settings.json";
# Merge base env (always) with OTEL env (when enabled). jq is always # Merge base env (always) with OTEL env (when enabled).
# run — `baseClaudeEnv` can carry per-agent values (e.g.
# BUN_JSC_forceRAMSize) that can't live in the static store asset.
allEnv = allEnv =
baseClaudeEnv baseClaudeEnv
// lib.optionalAttrs otelCfg.enable otelSettingsEnv // lib.optionalAttrs otelCfg.enable otelSettingsEnv

View file

@ -41,6 +41,11 @@ let
which the hive writes into every container, so no agent build sets it. which the hive writes into every container, so no agent build sets it.
Remove the setting. Remove the setting.
''; '';
memoryMaxBytesRemoved = ''
An agent reads its memory cap at boot from /run/hive/memory-max-bytes,
which the hive writes before every start, so no agent build sets it.
Remove the setting.
'';
in in
{ {
imports = [ imports = [
@ -60,6 +65,7 @@ in
./logs.nix ./logs.nix
./matrix.nix ./matrix.nix
./mcp.nix ./mcp.nix
./memory-cap.nix
./network.nix ./network.nix
./otel.nix ./otel.nix
./packages.nix ./packages.nix
@ -94,6 +100,13 @@ in
# Both spellings, for the reason given above. # Both spellings, for the reason given above.
(lib.mkRemovedOptionModule [ "hyperhive" "hiveName" ] hiveNameRemoved) (lib.mkRemovedOptionModule [ "hyperhive" "hiveName" ] hiveNameRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "hiveName" ] hiveNameRemoved) (lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "hiveName" ] hiveNameRemoved)
(lib.mkRemovedOptionModule [ "hyperhive" "claudeMemoryMaxBytes" ] memoryMaxBytesRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"agent"
"claudeMemoryMaxBytes"
] memoryMaxBytesRemoved)
]; ];
options.services.hyperhive.agent.icon = lib.mkOption { options.services.hyperhive.agent.icon = lib.mkOption {

View file

@ -11,25 +11,6 @@
}: }:
let let
userName = config.services.hyperhive.agent.user.name; userName = config.services.hyperhive.agent.user.name;
# This container's own effective `MemoryMax=` in bytes, baked in per
# agent by meta.rs's flake render — see
# `services.hyperhive.agent.claudeMemoryMaxBytes` in ./claude-settings.nix. `null`
# when the cap is `infinity` or a RAM percentage, i.e. when the module
# has no byte count to size anything against.
containerMemoryMaxBytes = config.services.hyperhive.agent.claudeMemoryMaxBytes;
# Two thirds of the container's cap, as the soft ceiling on everything
# the subagent daemon runs. The daemon spawns nested `claude` sessions
# as plain children, so its cgroup already *is* the "all subagents"
# cgroup and a unit-level ceiling bounds the set without a slice.
#
# `MemoryHigh=` and not `MemoryMax=`: this throttles rather than walls.
# Past it the kernel reclaims aggressively and the cgroup stalls, so a
# subagent that overshoots gets visibly slow and the remaining third
# stays available for the agent's own turn — but a subagent that
# genuinely needs more than two thirds still gets it when the container
# has the memory to spare, which is what keeps overprovisioning
# (several agents that rarely compile at the same time) working.
subagentMemoryHigh = containerMemoryMaxBytes * 2 / 3;
# Set on the harness only for an ACP agent on the opencode preset # Set on the harness only for an ACP agent on the opencode preset
# (./agent-service.nix). # (./agent-service.nix).
acpApiKeyEnv = config.systemd.services.hive-agent.environment.HIVE_ACP_API_KEY_ENV or null; acpApiKeyEnv = config.systemd.services.hive-agent.environment.HIVE_ACP_API_KEY_ENV or null;
@ -475,12 +456,11 @@ in
# actually chose and leaves the daemon serving, which is also what # actually chose and leaves the daemon serving, which is also what
# lets it report the kill rather than vanish and be restarted. # lets it report the kill rather than vanish and be restarted.
OOMPolicy = "continue"; OOMPolicy = "continue";
# `MemoryHigh=` is a drop-in from ./memory-cap.nix, sized at boot from
# the container's cap.
User = userName; User = userName;
Group = userName; Group = userName;
} }
// lib.optionalAttrs (containerMemoryMaxBytes != null) {
MemoryHigh = toString subagentMemoryHigh;
}
// lib.optionalAttrs subagentReadsStore { // lib.optionalAttrs subagentReadsStore {
# Bare id: inherits the credential the container manager passed in, # Bare id: inherits the credential the container manager passed in,
# the same form ./queue-identity.nix uses. # the same form ./queue-identity.nix uses.

View file

@ -0,0 +1,98 @@
# The container's memory cap, read at boot rather than built in, so a changed
# cap needs no rebuild.
#
# The cap is `MemoryMax=` on the host's `container@` unit, which nothing in
# here can read: `/sys/fs/cgroup/memory.max` reads `max` at every level from
# inside. Before every start hive-c0re (`lifecycle::host_config`) writes it to
# `/run/hive/memory-max-bytes` as a byte count, a percentage resolved against
# the host's RAM as systemd resolves it, or `infinity` for no cap. Two values
# are derived from a byte count, and neither is set for `infinity`. A missing
# or malformed file sets neither too, and `hyperhive-memory-cap-check` logs a
# warning at boot:
#
# - `BUN_JSC_forceRAMSize`, 75% of the cap. Bun/JavaScriptCore, the runtime
# under claude-code, sizes its heap ceiling off `/proc/meminfo`, which under
# nspawn is the host's RAM, so without it the heap grows well past the real
# wall before GC kicks in hard and choom sessions hang. An environment
# generator hands it to every unit of the container's systemd, `machinectl
# shell` sessions (`hivectl choom`) included; `/etc/set-environment` exports
# it in login shells that no unit starts (`nixos-container root-login`).
# `hive-agent` and `hive-subagent-daemon` spawn `claude` as a plain child,
# which inherits it, so neither unit may set or unset it.
# - `MemoryHigh=` on `hive-subagent-daemon`, two thirds of the cap, as a
# generator drop-in. The daemon spawns nested `claude` sessions as plain
# children, so its cgroup already is the "all subagents" cgroup. `High`, not
# `Max`: past it the kernel reclaims hard and the subagents slow down, so the
# last third stays free for the agent's own turn, but a subagent that needs
# more still gets it when the container has memory to spare.
#
# Tested by `checks.script-test-agent-memory-cap` and
# `checks.module-eval-agent-memory`.
{ pkgs, ... }:
let
# Sets `bytes`. Exits for `infinity`, and runs `noCap` when there is no file
# or its first line is neither `infinity` nor a plain byte count. A leading
# zero would read as octal, and 16 digits or more could overflow
# `bytes * 75`. Shell builtins only.
readCap = noCap: ''
bytes=
[ -r /run/hive/memory-max-bytes ] && read -r bytes < /run/hive/memory-max-bytes
case "$bytes" in
infinity) exit 0 ;;
"" | 0* | *[!0-9]* | ????????????????*) ${noCap} ;;
esac
'';
envGenerator = pkgs.writeShellScript "hyperhive-memory-cap-env" ''
${readCap "exit 0"}
printf 'BUN_JSC_forceRAMSize=%s\n' "$((bytes * 75 / 100))"
'';
# `$1` is the generator's normal-priority output dir. A drop-in there merges
# with the unit's own file in /etc.
unitGenerator = pkgs.writeShellScript "hyperhive-memory-cap" ''
${readCap "exit 0"}
dir="$1/hive-subagent-daemon.service.d"
${pkgs.coreutils}/bin/mkdir -p "$dir"
printf '[Service]\nMemoryHigh=%s\n' "$((bytes * 2 / 3))" > "$dir/50-memory-high.conf"
'';
# The generators run before the container's journal starts and an nspawn
# container has no /dev/kmsg, so the boot warning for a missing or malformed
# cap file comes from a unit. `<4>` is the journal's warning priority.
check = pkgs.writeShellScript "hyperhive-memory-cap-check" ''
${readCap ''
if [ -e /run/hive/memory-max-bytes ]; then
why="holds neither a byte count nor infinity"
else
why="is missing"
fi
echo "<4>/run/hive/memory-max-bytes $why: no BUN_JSC_forceRAMSize heap ceiling and no MemoryHigh= on hive-subagent-daemon; the host's MemoryMax= on the container still applies." >&2
exit 0
''}
'';
# Sourced by every shell's `/etc/set-environment`, so POSIX sh only.
profile = pkgs.writeText "hyperhive-memory-cap.sh" ''
__hyperhive_cap=$(${envGenerator})
case "$__hyperhive_cap" in
BUN_JSC_forceRAMSize=?*) export "$__hyperhive_cap" ;;
esac
unset __hyperhive_cap
'';
in
{
environment.etc."systemd/system-environment-generators/50-hyperhive-memory-cap".source =
envGenerator;
systemd.generators.hyperhive-memory-cap = unitGenerator;
systemd.services.hyperhive-memory-cap-check = {
description = "Warn when the container's memory cap is unusable";
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = check;
};
};
environment.etc."hyperhive/memory-cap.sh".source = profile;
environment.extraInit = ". /etc/hyperhive/memory-cap.sh";
}

View file

@ -45,10 +45,6 @@
[ "hyperhive" "claudeMarketplaces" ] [ "hyperhive" "claudeMarketplaces" ]
[ "services" "hyperhive" "agent" "claudeMarketplaces" ] [ "services" "hyperhive" "agent" "claudeMarketplaces" ]
) )
(lib.mkRenamedOptionModule
[ "hyperhive" "claudeMemoryMaxBytes" ]
[ "services" "hyperhive" "agent" "claudeMemoryMaxBytes" ]
)
(lib.mkRenamedOptionModule (lib.mkRenamedOptionModule
[ "hyperhive" "claudePlugins" ] [ "hyperhive" "claudePlugins" ]
[ "services" "hyperhive" "agent" "claudePlugins" ] [ "services" "hyperhive" "agent" "claudePlugins" ]

View file

@ -138,6 +138,12 @@ in
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;
}; };
# Runs ./agent-modules/memory-cap.nix's generators and login-shell snippet
# against sample cap files.
script-test-agent-memory-cap = import ./script-tests/agent-memory-cap.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-agent-github-bao = import ./module-eval/agent-github-bao.nix { module-eval-agent-github-bao = import ./module-eval/agent-github-bao.nix {
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;

View file

@ -22,39 +22,37 @@ let
agentHarness agentHarness
; ;
# The memory-pressure pair. `claudeMemoryMaxBytes` is the container's own agentDefault = agent { };
# cap, rendered per agent by meta.rs — the capped arm is the one every
# real deploy gets, the uncapped arm is a hive that set `infinity` or a
# RAM percentage and so hands the module no byte count to size against.
agentCapped = agent { claudeMemoryMaxBytes = 8589934592; };
agentUncapped = agent { };
agentBashDaemon = machine: machine.systemd.services.hive-bash-daemon; agentBashDaemon = machine: machine.systemd.services.hive-bash-daemon;
agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon; agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon;
cases = [ cases = [
{ {
# Two thirds of the container's cap, and a SOFT ceiling: the daemon's # The ceiling is sized at boot from the container's cap
# cgroup holds every nested claude, so `MemoryHigh=` throttles the # (../agent-modules/memory-cap.nix). One in the unit would be the cap at
# subagent set as a whole before the kernel picks a victim, while the # the last rebuild, and every agent would carry its own hive's number.
# absent `MemoryMax=` is what still lets one subagent use more than name = "the subagent daemon's memory ceiling is not built into its unit";
# its share on a container that has the memory free. A hard cap here ok = !((agentSubagentDaemon agentDefault).serviceConfig ? MemoryHigh);
# would trade the silent kill for a guaranteed wall, which is the
# shape this deliberately does not have.
name = "the subagent daemon throttles at two thirds of the container's memory";
ok =
let
c = (agentSubagentDaemon agentCapped).serviceConfig;
in
c.MemoryHigh == "5726623061" && !(c ? MemoryMax);
} }
{ {
# What makes the case above able to fail. With no byte count for the # claude's heap ceiling reaches it only as inherited environment: the
# container there is no fraction to take, and a hardcoded fallback # environment generator fills the manager's, systemd hands that to every
# would be a number about some other hive's machine — so the unit # unit, and both units spawn claude as a plain child. A unit that sets
# renders no ceiling at all rather than a fabricated one. # or unsets the variable itself cuts that. Without the two generators the
name = "an agent with no byte-valued memory cap renders no subagent ceiling"; # cap reaches nothing, and without the check unit nothing says so.
ok = !((agentSubagentDaemon agentUncapped).serviceConfig ? MemoryHigh); name = "claude's units take the heap ceiling from the memory cap read at boot";
ok =
let
leavesIt =
unit:
!(unit.environment ? BUN_JSC_forceRAMSize)
&& !(lib.elem "BUN_JSC_forceRAMSize" (lib.toList (unit.serviceConfig.UnsetEnvironment or [ ])));
in
leavesIt (agentHarness agentDefault)
&& leavesIt (agentSubagentDaemon agentDefault)
&& agentDefault.environment.etc ? "systemd/system-environment-generators/50-hyperhive-memory-cap"
&& agentDefault.systemd.generators ? hyperhive-memory-cap
&& lib.elem "multi-user.target" agentDefault.systemd.services.hyperhive-memory-cap-check.wantedBy;
} }
{ {
# The sign is the whole property, and it is easy to write backwards: # The sign is the whole property, and it is easy to write backwards:
@ -68,8 +66,8 @@ let
name = "the OOM killer prefers a subagent over the agent's own session"; name = "the OOM killer prefers a subagent over the agent's own session";
ok = ok =
let let
sub = (agentSubagentDaemon agentUncapped).serviceConfig.OOMScoreAdjust; sub = (agentSubagentDaemon agentDefault).serviceConfig.OOMScoreAdjust;
own = (agentHarness agentUncapped).serviceConfig.OOMScoreAdjust; own = (agentHarness agentDefault).serviceConfig.OOMScoreAdjust;
in in
sub > 0 && own < 0 && sub > own; sub > 0 && own < 0 && sub > own;
} }
@ -81,13 +79,13 @@ let
# chose, which is the blast radius that made the preference a bad # chose, which is the blast radius that made the preference a bad
# trade in the first place. # trade in the first place.
name = "one subagent losing the OOM draw does not stop the daemon"; name = "one subagent losing the OOM draw does not stop the daemon";
ok = (agentSubagentDaemon agentUncapped).serviceConfig.OOMPolicy == "continue"; ok = (agentSubagentDaemon agentDefault).serviceConfig.OOMPolicy == "continue";
} }
{ {
# Each bash task is a process in the daemon's cgroup; under `stop` one # Each bash task is a process in the daemon's cgroup; under `stop` one
# OOM-killed task would end every running sibling task. # OOM-killed task would end every running sibling task.
name = "one bash task losing the OOM draw does not stop the daemon"; name = "one bash task losing the OOM draw does not stop the daemon";
ok = (agentBashDaemon agentUncapped).serviceConfig.OOMPolicy or null == "continue"; ok = (agentBashDaemon agentDefault).serviceConfig.OOMPolicy or null == "continue";
} }
]; ];
in in

View file

@ -0,0 +1,44 @@
# `checks.script-test-agent-memory-cap` — runs the four readers of
# `/run/hive/memory-max-bytes` in ../agent-modules/memory-cap.nix: the
# environment generator, the unit generator, the boot check that warns about
# an unusable cap, and the login-shell snippet `/etc/set-environment` sources.
# The cases are in ./agent-memory-cap.sh.
#
# What runs is the rendered scripts. The one edit is the cap file's path, moved
# under the build directory, and the snippet's reference to the environment
# generator, pointed at that edited copy.
{
pkgs,
lib,
self,
nixosSystem,
}:
let
inherit
(import ../module-eval/lib.nix {
inherit
pkgs
lib
self
nixosSystem
;
})
agentWith
;
machine = agentWith { };
in
pkgs.runCommand "hyperhive-script-test-agent-memory-cap"
{
ENV_GENERATOR =
machine.environment.etc."systemd/system-environment-generators/50-hyperhive-memory-cap".source;
UNIT_GENERATOR = machine.systemd.generators.hyperhive-memory-cap;
CHECK = machine.systemd.services.hyperhive-memory-cap-check.serviceConfig.ExecStart;
PROFILE = machine.environment.etc."hyperhive/memory-cap.sh".source;
EXTRA_INIT = machine.environment.extraInit;
SHELLS = "${pkgs.dash}/bin/dash ${pkgs.bash}/bin/sh";
}
''
${pkgs.bash}/bin/bash ${./agent-memory-cap.sh}
touch "$out"
''

View file

@ -0,0 +1,116 @@
# Cases for ./agent-memory-cap.nix. Each case writes a cap file (or none), runs
# both generators and the boot check with a clean environment and compares what
# they produced, then sources the login-shell snippet under each POSIX shell
# and compares what a child process sees.
set -uo pipefail
failures=0
count=0
fail() {
echo "FAILED: $case: $*" >&2
failures=$((failures + 1))
}
dir=$(mktemp -d)
cap=$dir/memory-max-bytes
sed "s|/run/hive/memory-max-bytes|$cap|g" "$ENV_GENERATOR" >"$dir/env-generator"
sed "s|/run/hive/memory-max-bytes|$cap|g" "$UNIT_GENERATOR" >"$dir/unit-generator"
sed "s|/run/hive/memory-max-bytes|$cap|g" "$CHECK" >"$dir/check"
chmod +x "$dir/env-generator" "$dir/unit-generator" "$dir/check"
sed "s|$ENV_GENERATOR|$dir/env-generator|g" "$PROFILE" >"$dir/profile"
envbin=$(command -v env)
case="setup"
for g in env-generator unit-generator check; do
if grep -q '/run/hive/memory-max-bytes' "$dir/$g"; then fail "$g still names /run/hive after the rewrite"; fi
if ! grep -qF "$cap" "$dir/$g"; then fail "the rewrite did not reach $g's read"; fi
done
if ! grep -qF "$dir/env-generator" "$dir/profile"; then fail "the snippet does not call the rewritten generator"; fi
if ! printf '%s\n' "$EXTRA_INIT" | grep -qxF '. /etc/hyperhive/memory-cap.sh'; then
fail "/etc/set-environment does not source the snippet"
fi
# check <description> <expected BUN_JSC_forceRAMSize or ""> <expected MemoryHigh or "">
# <why the boot check warns, or "">
check() {
case=$1
count=$((count + 1))
local want_env=${2:+BUN_JSC_forceRAMSize=$2} want_high=$3 want_why=${4:-} got
got=$(env -i "$dir/env-generator")
if [ "$got" != "$want_env" ]; then
fail "environment generator printed:"$'\n'"$got"$'\n'"wanted:"$'\n'"$want_env"
fi
local out=$dir/generator-out
rm -rf "$out"
mkdir -p "$out/normal" "$out/early" "$out/late"
env -i "$dir/unit-generator" "$out/normal" "$out/early" "$out/late"
local dropin=$out/normal/hive-subagent-daemon.service.d/50-memory-high.conf
if [ -z "$want_high" ]; then
got=$(find "$out" -type f)
if [ -n "$got" ]; then fail "unit generator wrote:"$'\n'"$got"; fi
else
got=$(cat "$dropin" 2>&1)
if [ "$got" != "[Service]"$'\n'"MemoryHigh=$want_high" ]; then
fail "unit generator's drop-in:"$'\n'"$got"
fi
fi
# The warning goes to stderr with the journal's `<4>` priority prefix, and
# nothing goes anywhere for a byte count or `infinity`.
got=$(env -i "$dir/check" 2>&1)
local want_warn=
if [ -n "$want_why" ]; then
want_warn="<4>$cap $want_why: no BUN_JSC_forceRAMSize heap ceiling and no MemoryHigh= on hive-subagent-daemon; the host's MemoryMax= on the container still applies."
fi
if [ "$got" != "$want_warn" ]; then
fail "boot check printed:"$'\n'"$got"$'\n'"wanted:"$'\n'"$want_warn"
fi
local sh
for sh in $SHELLS; do
# Only what the snippet exported reaches the child's environment. No PATH:
# the snippet must not need one.
got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(BUN_|LD_PRELOAD=|__hyperhive)')
if [ "$got" != "$want_env" ]; then
fail "$sh: a child of the sourcing shell saw:"$'\n'"$got"
fi
done
}
printf '8589934592\n' >"$cap"
check "an 8G cap" 6442450944 5726623061
printf '4294967296' >"$cap"
check "a line without a newline is kept" 3221225472 2863311530
printf '4294967296\nLD_PRELOAD=/x\n' >"$cap"
check "only the first line is read" 3221225472 2863311530
printf 'infinity\n' >"$cap"
check "infinity sets nothing and warns of nothing" "" "" ""
printf 'infinity' >"$cap"
check "infinity without a newline" "" "" ""
printf 'infinityx\n' >"$cap"
check "a word that only starts with infinity" "" "" "holds neither a byte count nor infinity"
printf '8G\n' >"$cap"
check "a size with a suffix is ignored" "" "" "holds neither a byte count nor infinity"
printf '0123\n' >"$cap"
check "a leading zero is ignored" "" "" "holds neither a byte count nor infinity"
printf '1234567890123456\n' >"$cap"
check "sixteen digits are ignored" "" "" "holds neither a byte count nor infinity"
: >"$cap"
check "an empty file" "" "" "holds neither a byte count nor infinity"
rm -f "$cap"
check "no file" "" "" "is missing"
echo "$count cases, $failures failed"
[ "$failures" -eq 0 ]