Watch
0
0
Fork
You've already forked hyperhive
0

hand agents the dashboard port, pinned claude and otel labels at container start

The three values that differ per host (deploy.hive-controller.dashboardPort,
claudeCodePackage and otel.extraResourceAttributes) no longer go into the
meta flake. hive-priv writes them into the container's
/etc/hyperhive-hive-facts with the hive name and domain, and the agent
reads them at boot:

- HIVE_DASHBOARD_PORT reaches every unit through the hive-facts generator;
  the web UI still defaults to 7000 without it.
- `claude` on PATH is a wrapper that runs HYPERHIVE_CLAUDE_CODE_PATH's
  claude when it is there, and the container's own claude-code otherwise.
  The host's serve.json keeps gc-rooting the pinned build.
- OTEL_RESOURCE_ATTRIBUTES moves out of the units' Environment= into a
  boot-time generator, which appends HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES.

The agent options services.hyperhive.agent.claudeCodePath (now a removed
option) and otel.extraResourceAttributes go; only the meta render set them.
render_flake takes no HiveEnv any more, so the memory-cap render test and
render_hive_flake_at, which only existed to vary HiveEnv into the render,
go with it.

Refs #4857
This commit is contained in:
atlas 2026-10-21 23:47:35 +02:00 • committed by mara
commit 9113651c10
23 changed files with 408 additions and 335 deletions

View file

@ -118,7 +118,10 @@ declares one flake input per agent (`agent-<n>.url =
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`) and one "git+<forge>/agent-configs/<n>.git?rev=<sha>"`) and one
`nixosConfigurations.<n>` output per agent. Each output wraps `nixosConfigurations.<n>` output per agent. Each output wraps
`inputs.agent-<n>.nixosModules.default` with the identity + `inputs.agent-<n>.nixosModules.default` with the identity +
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module. `HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
(the dashboard port, the pinned claude-code, extra OTEL labels) aren't in it:
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
start (see `docs/process/conventions.md`).
Containers run against `--flake /var/lib/hyperhive/meta#<n>`. Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
The declared input url is the head of `main` on the agent's forge config repo The declared input url is the head of `main` on the agent's forge config repo
@ -195,8 +198,7 @@ The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix` (see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
is a fixed boilerplate that exports `nixosModules.default = is a fixed boilerplate that exports `nixosModules.default =
import ./agent.nix`; the meta flake imports that module and import ./agent.nix`; the meta flake imports that module and
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` / wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
`HIVE_DASHBOARD_PORT`.
### Tag state machine ### Tag state machine

View file

@ -39,11 +39,15 @@ Four env vars cover the identity surface, read by
name. name.
Inside an agent container, the hive's name and domain don't come from the Inside an agent container, the hive's name and domain don't come from the
agent's build. `hive-priv` writes them to `/etc/hyperhive-hive-facts` before agent's build, and neither do the hive's other per-host values: the dashboard
every container start. The agent's environment generator port (`HIVE_DASHBOARD_PORT`), the pinned claude-code
(`nix/agent-modules/hive-facts.nix`) passes them to every unit, including (`HYPERHIVE_CLAUDE_CODE_PATH`) and extra OTEL labels
`hivectl choom` sessions, and `/etc/set-environment` exports them in login (`HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`). `hive-priv` writes them to
shells that no unit starts (`nixos-container root-login`). Neither is part of `/etc/hyperhive-hive-facts` before every container start, leaving out any the
hive doesn't set; the agent then uses its own default. The agent's environment
generator (`nix/agent-modules/hive-facts.nix`) passes them to every unit,
including `hivectl choom` sessions, and `/etc/set-environment` exports them in login
shells that no unit starts (`nixos-container root-login`). None is part of
an agent's closure, and an agent started on another hive reports that hive. an agent's closure, and an agent started on another hive reports that hive.
`hive_name` + `swarm_name` are **distinct** from `hive_name` + `swarm_name` are **distinct** from

View file

@ -145,9 +145,9 @@ and every agent runs that build.
What crosses is the **store path**, not the derivation. Containers What crosses is the **store path**, not the derivation. Containers
share the host's `/nix/store`, so the binary is already reachable share the host's `/nix/store`, so the binary is already reachable
inside them with its whole closure; hive-c0re writes the path into each inside them with its whole closure. `hive-priv` writes the path into each
agent's flake as a string literal and the agent module symlinks container's `/etc/hyperhive-hive-facts` at start, and the agent's `claude`
`bin/claude` onto PATH. Two things rule out the natural-seeming alternatives: a wrapper on PATH runs its `bin/claude`. Two things rule out the natural-seeming alternatives: a
`path:/nix/store/<pkg>` flake input is re-copied into the store as a `path:/nix/store/<pkg>` flake input is re-copied into the store as a
reference-less `-source` (so the runtime closure never arrives), and reference-less `-source` (so the runtime closure never arrives), and
`lib.types.package` fed a bare path runs `builtins.storePath`, which `lib.types.package` fed a bare path runs `builtins.storePath`, which
@ -171,7 +171,7 @@ expected.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
The catch is that a path written into a generated flake is text, not a The catch is that a path handed to the container at start is text, not a
reference — the container's closure doesn't keep the binary alive. reference — the container's closure doesn't keep the binary alive.
The **host** does: it interpolates the package into The **host** does: it interpolates the package into
`/etc/hyperhive/serve.json`, so it lands in the host's system closure `/etc/hyperhive/serve.json`, so it lands in the host's system closure
@ -180,7 +180,7 @@ string context, which is the load-bearing detail; discard the context
anywhere on that path and `nix-collect-garbage` will eventually take anywhere on that path and `nix-collect-garbage` will eventually take
the hive's `claude` out from under it. The price of the root is that the hive's `claude` out from under it. The price of the root is that
`nix-collect-garbage` can't reclaim an old `claude-code` until every `nix-collect-garbage` can't reclaim an old `claude-code` until every
agent has rebuilt past it and the old generations are gone. agent has restarted past it and the old generations are gone.
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

View file

@ -190,7 +190,7 @@ two-tier architecture, the security model, and how the options interact.
The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`, The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container `OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container
wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every wide**, for every unit and in `/etc/profile`, so every
process in an agent container exports to the hive's collector without any process in an agent container exports to the hive's collector without any
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
yourself from a tool call or a shell you open in the container. yourself from a tool call or a shell you open in the container.
@ -209,8 +209,9 @@ the push (see [Authenticated ingest](#authenticated-ingest)). The container
doesn't set it. doesn't set it.
Append additional labels for one hive via `extraResourceAttributes` (see Append additional labels for one hive via `extraResourceAttributes` (see
option reference above); pass custom per-data-point labels with option reference above). Agents get them at container start, so a change
`hive-metric --labels` (see below). reaches an agent at its next start, not through a rebuild. Pass custom
per-data-point labels with `hive-metric --labels` (see below).
## Log severity ## Log severity

View file

@ -33,11 +33,13 @@ harness unit's PATH. By default that's the `claude-code` in the agent's
own nixpkgs (the meta flake's `nixpkgs` input) via own nixpkgs (the meta flake's `nixpkgs` input) via
`environment.systemPackages`. Since that's typically a release channel and `environment.systemPackages`. Since that's typically a release channel and
this package moves fast, the operator can pin one per hive with this package moves fast, the operator can pin one per hive with
`services.hyperhive.deploy.hive-controller.claudeCodePackage`: hive-c0re writes its `services.hyperhive.deploy.hive-controller.claudeCodePackage`. `hive-priv`
store path into each agent's flake, and `claude` on PATH becomes a symlink to it writes its store path into the container's `/etc/hyperhive-hive-facts` as
instead of the container's own `claude-code` — so there's only ever one `HYPERHIVE_CLAUDE_CODE_PATH` before every start. `claude` on PATH is a wrapper
`claude` in the container. Agents pick up a new build on their that runs that build when the path exists, and the container's own
next rebuild, not live. See docs/process/gotchas.md::`claude-code` is unfree. `claude-code` when the variable is unset or the path no longer exists, saying
so on stderr in the second case. A bump reaches an agent at its next container
start, not at a rebuild. See docs/process/gotchas.md::`claude-code` is unfree.
Hive-enforced settings ship at `/etc/claude-code/managed-settings.json` Hive-enforced settings ship at `/etc/claude-code/managed-settings.json`
(claude-code's canonical managed-settings path — precedence #1, (claude-code's canonical managed-settings path — precedence #1,

View file

@ -43,8 +43,8 @@ pub struct Coordinator {
/// Git rev of the hyperhive checkout this host runs. Pins the meta /// Git rev of the hyperhive checkout this host runs. Pins the meta
/// flake's `hyperhive` input; `None` makes the meta render fail. /// flake's `hyperhive` input; `None` makes the meta render fail.
pub hyperhive_rev: Option<String>, pub hyperhive_rev: Option<String>,
/// Store path of the `claude-code` build every agent runs, written /// Store path of the `claude-code` build every agent runs, handed to
/// into each per-agent flake as `services.hyperhive.agent.claudeCodePath`. Set by /// each container at start as `HYPERHIVE_CLAUDE_CODE_PATH`. Set by
/// the NixOS module option /// the NixOS module option
/// `services.hyperhive.deploy.hive-controller.claudeCodePackage` /// `services.hyperhive.deploy.hive-controller.claudeCodePackage`
/// (which resolves the package and hands us its path). `None` = every /// (which resolves the package and hands us its path). `None` = every
@ -55,9 +55,9 @@ pub struct Coordinator {
/// closure and all. The host module is what keeps it from being /// closure and all. The host module is what keeps it from being
/// garbage-collected — see that option. /// garbage-collected — see that option.
pub claude_code_path: Option<String>, pub claude_code_path: Option<String>,
/// TCP port the host's hive-c0re dashboard listens on. Inlined into /// TCP port the host's hive-c0re dashboard listens on. Handed to each
/// each per-agent flake so the agent's web UI can build the right /// container at start as `HIVE_DASHBOARD_PORT`, so the agent's web UI
/// rebuild-button URL pointing back at the dashboard. /// can build the rebuild-button URL pointing back at the dashboard.
pub dashboard_port: u16, pub dashboard_port: u16,
/// Per-agent systemd `CPUQuota=` value (e.g. `"200%"`). Written into /// Per-agent systemd `CPUQuota=` value (e.g. `"200%"`). Written into
/// the `container@h-<name>.service.d/` drop-in on every spawn/rebuild. /// the `container@h-<name>.service.d/` drop-in on every spawn/rebuild.
@ -193,7 +193,7 @@ pub struct HiveEnv {
pub hyperhive_rev: Option<String>, pub hyperhive_rev: Option<String>,
/// Store path of the `claude-code` agents run, or `None` for "each /// Store path of the `claude-code` agents run, or `None` for "each
/// agent keeps the one out of its own nixpkgs". Travels into the /// agent keeps the one out of its own nixpkgs". Travels into the
/// container as `services.hyperhive.agent.claudeCodePath` — a plain string, kept /// container as `HYPERHIVE_CLAUDE_CODE_PATH` — a plain string, kept
/// alive host-side by the module that resolved it. /// alive host-side by the module that resolved it.
pub claude_code_path: Option<String>, pub claude_code_path: Option<String>,
pub dashboard_port: u16, pub dashboard_port: u16,

View file

@ -25,7 +25,7 @@ use super::{
pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> { pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> {
validate(name)?; validate(name)?;
let container = container_name(name); let container = container_name(name);
set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?; set_nspawn_flags(&container, hive, &paths.agent, &paths.claude, &paths.notes).await?;
hand_permissions(name, &paths.agent); hand_permissions(name, &paths.agent);
if let Some((cpu_quota, memory_max)) = stage_limits( if let Some((cpu_quota, memory_max)) = stage_limits(
name, name,
@ -496,6 +496,7 @@ fn queue_agent_credentials(agent_name: &str, dir: Option<&Path>) -> Vec<Credenti
)] )]
async fn set_nspawn_flags( async fn set_nspawn_flags(
container: &str, container: &str,
hive: &HiveEnv,
runtime_dir: &Path, runtime_dir: &Path,
claude_dir: &Path, claude_dir: &Path,
notes_dir: &Path, notes_dir: &Path,
@ -669,7 +670,7 @@ async fn set_nspawn_flags(
&binds, &binds,
isolation, isolation,
&load_creds, &load_creds,
super::hive_facts_from_env(), super::hive_facts(hive),
) )
.await .await
} }

View file

@ -169,14 +169,27 @@ pub fn network_isolation_from_env() -> Result<hive_priv_sock::NetworkIsolation>
network_isolation_from_vars(bridge.as_deref(), subnet.as_deref()) network_isolation_from_vars(bridge.as_deref(), subnet.as_deref())
} }
/// The hive's facts every container is handed at runtime, from hive-c0re's /// The hive's facts every container is handed at runtime, so that no agent
/// own unit environment (`nix/host-modules/hive-c0re/environment.nix`). An /// build carries a value that differs between hosts. The name, domain and
/// unset or empty variable is `None`. /// OTEL resource labels come from hive-c0re's own unit environment
fn hive_facts_from_env() -> hive_priv_sock::HiveFacts { /// (`nix/host-modules/hive-c0re/environment.nix`), where an unset or empty
let read = |name: &str| std::env::var(name).ok().filter(|v| !v.is_empty()); /// variable is `None`; the dashboard port and pinned claude from `hive`.
fn hive_facts(hive: &HiveEnv) -> hive_priv_sock::HiveFacts {
hive_facts_from(hive, |name| std::env::var(name).ok())
}
/// [`hive_facts`] over a given environment lookup, for tests.
fn hive_facts_from(
hive: &HiveEnv,
var: impl Fn(&str) -> Option<String>,
) -> hive_priv_sock::HiveFacts {
let read = |name: &str| var(name).filter(|v| !v.is_empty());
hive_priv_sock::HiveFacts { hive_priv_sock::HiveFacts {
name: read("HYPERHIVE_HIVE_NAME"), name: read("HYPERHIVE_HIVE_NAME"),
domain: read("HYPERHIVE_HIVE_DOMAIN"), domain: read("HYPERHIVE_HIVE_DOMAIN"),
dashboard_port: Some(hive.dashboard_port),
claude_code_path: hive.claude_code_path.clone(),
otel_extra_resource_attributes: read("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES"),
} }
} }

View file

@ -527,3 +527,39 @@ async fn run_with_deadline_returns_output_of_a_fast_child() {
.expect("echo finishes"); .expect("echo finishes");
assert_eq!(String::from_utf8_lossy(&out.stdout).trim(), "hi"); assert_eq!(String::from_utf8_lossy(&out.stdout).trim(), "hi");
} }
/// The three per-host values reach the container as facts: the dashboard
/// port and pinned claude from the hive's config, the OTEL labels from the
/// daemon's environment.
#[test]
fn hive_facts_carry_the_per_host_values() {
let hive = HiveEnv {
dashboard_port: 7123,
claude_code_path: Some("/nix/store/0abc-claude-code".to_owned()),
..HiveEnv::default()
};
let facts = hive_facts_from(&hive, |name| {
(name == "HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES").then(|| "team=a".to_owned())
});
assert_eq!(facts.dashboard_port, Some(7123));
assert_eq!(
facts.claude_code_path.as_deref(),
Some("/nix/store/0abc-claude-code")
);
assert_eq!(
facts.otel_extra_resource_attributes.as_deref(),
Some("team=a")
);
}
/// Unset and empty both leave a fact out, so the agent falls back to its
/// own default rather than reading an empty value.
#[test]
fn hive_facts_leave_unset_values_out() {
let facts = hive_facts_from(&HiveEnv::default(), |name| {
(name == "HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES").then(String::new)
});
assert_eq!(facts.claude_code_path, None);
assert_eq!(facts.otel_extra_resource_attributes, None);
assert_eq!(facts.name, None);
}

View file

@ -431,23 +431,12 @@ fn pinned_git_url(forge_git_url: &str, rev: Option<&str>) -> Result<String> {
Ok(format!("git+{forge_git_url}?rev={rev}")) Ok(format!("git+{forge_git_url}?rev={rev}"))
} }
/// [`render_flake`] with its inputs taken from `hive`, as [`sync_agents`] /// [`render_flake`] with the hyperhive pin taken from `hive`, as
/// renders it. /// [`sync_agents`] renders it. Nothing else in `hive` reaches the flake: the
/// values that differ per host go to the container at start
/// (`lifecycle::hive_facts`).
fn render_hive_flake(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<String> { fn render_hive_flake(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<String> {
render_hive_flake_at(hive, &hyperhive_input_url(hive)?, agents) render_flake(&hyperhive_input_url(hive)?, agents)
}
fn render_hive_flake_at(
hive: &HiveEnv,
hyperhive_url: &str,
agents: &[AgentSpec],
) -> Result<String> {
render_flake(
hyperhive_url,
hive.claude_code_path.as_deref(),
hive.dashboard_port,
agents,
)
} }
/// Canonical inputs meta knows how to dedup. An agent that declares one /// Canonical inputs meta knows how to dedup. An agent that declares one
@ -629,19 +618,6 @@ fn forwarded_env_vars() -> Vec<(&'static str, String)> {
.collect() .collect()
} }
/// Extra `OTEL_RESOURCE_ATTRIBUTES` entries for every agent, read off
/// hive-c0re's own unit env
/// (`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`).
/// `None` when unset or empty. This is the only OTEL value the renderer
/// writes into the flake: agents export to the hive collector by default and
/// find it by name, so there is no enable flag, endpoint or protocol to
/// carry.
fn otel_extra_resource_attributes() -> Option<String> {
std::env::var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES")
.ok()
.filter(|v| !v.is_empty())
}
/// Remove the CA files earlier renders embedded next to `flake.nix` /// Remove the CA files earlier renders embedded next to `flake.nix`
/// (`hive-ca.pem`, and the per-peer `peer-ca-<N>.pem` before it), returning /// (`hive-ca.pem`, and the per-peer `peer-ca-<N>.pem` before it), returning
/// every filename removed so the caller stages the deletion. Nothing writes /// every filename removed so the caller stages the deletion. Nothing writes
@ -764,12 +740,7 @@ fn canonical_inputs_in_lock(raw: &str) -> Vec<&'static str> {
reason = "templated string-builder for the meta flake — the length is one \ reason = "templated string-builder for the meta flake — the length is one \
contiguous fmt block, splitting it would just hide the shape" contiguous fmt block, splitting it would just hide the shape"
)] )]
fn render_flake( fn render_flake(hyperhive_url: &str, agents: &[AgentSpec]) -> Result<String> {
hyperhive_url: &str,
claude_code_path: Option<&str>,
dashboard_port: u16,
agents: &[AgentSpec],
) -> Result<String> {
use std::fmt::Write as _; use std::fmt::Write as _;
let mut out = String::new(); let mut out = String::new();
out.push_str("{\n description = \"hyperhive deployed agents\";\n inputs = {\n"); out.push_str("{\n description = \"hyperhive deployed agents\";\n inputs = {\n");
@ -816,10 +787,8 @@ fn render_flake(
); );
} }
} }
out.push_str(" };\n outputs =\n { self, hyperhive, ... }@inputs:\n let\n"); out.push_str(
let _ = writeln!( " };\n outputs =\n { self, hyperhive, ... }@inputs:\n let\n mkAgent = { name, port, archetype ? null }:\n",
out,
" dashboardPort = {dashboard_port};\n mkAgent = {{ name, port, archetype ? null }}:"
); );
out.push_str( out.push_str(
r#" let r#" let
@ -842,32 +811,6 @@ fn render_flake(
{ {
"#, "#,
); );
// The `claude-code` agents run, as a bare store path rather than a
// flake input: containers share the host's `/nix/store`, so the
// binary is already there with its closure and has nothing to
// travel. A string literal is also the only shape that evaluates —
// `lib.types.package` on a bare path runs `builtins.storePath`,
// which pure eval rejects. The agent module puts its `bin/` on the
// harness PATH; the host module holds the gc root, since a path
// spelled out here is text and references nothing.
// `None` = no override; agents keep their own nixpkgs' `claude-code`.
if let Some(path) = claude_code_path {
let _ = writeln!(
out,
" services.hyperhive.agent.claudeCodePath = \"{path}\";"
);
}
// Extra OTEL resource labels (`deploy.hive-controller.otel.*`) are the one
// OTEL value rendered here; enablement and the collector endpoint are
// agent-module defaults.
if let Some(attrs) = otel_extra_resource_attributes() {
let esc = |s: &str| s.replace('\\', "\\\\").replace('"', "\\\"");
let _ = writeln!(
out,
" services.hyperhive.agent.otel.extraResourceAttributes = \"{}\";",
esc(&attrs)
);
}
// Agent-facing service URLs (`services.hyperhive.agent.forge.url`, `services.hyperhive.agent.matrix.url`): // Agent-facing service URLs (`services.hyperhive.agent.forge.url`, `services.hyperhive.agent.matrix.url`):
// emit the host's real values as build-time agent config, the same // emit the host's real values as build-time agent config, the same
// host-state -> agent-module shape as the CA block above. // host-state -> agent-module shape as the CA block above.
@ -959,7 +902,6 @@ fn render_flake(
systemd.services.${service}.environment = { systemd.services.${service}.environment = {
HIVE_PORT = toString port; HIVE_PORT = toString port;
HIVE_LABEL = name; HIVE_LABEL = name;
HIVE_DASHBOARD_PORT = toString dashboardPort;
", ",
); );
// Forwarded env vars — picked up from hive-c0re's own systemd unit // Forwarded env vars — picked up from hive-c0re's own systemd unit
@ -1282,13 +1224,8 @@ mod tests {
} }
} }
fn render( fn render(hyperhive_url: &str, agents: &[AgentSpec]) -> String {
hyperhive_url: &str, render_flake(hyperhive_url, agents).expect("render")
claude_code_path: Option<&str>,
dashboard_port: u16,
agents: &[AgentSpec],
) -> String {
render_flake(hyperhive_url, claude_code_path, dashboard_port, agents).expect("render")
} }
#[test] #[test]
@ -1309,8 +1246,6 @@ mod tests {
fn the_input_a_rebuild_relocks_is_a_pinned_forge_url() { fn the_input_a_rebuild_relocks_is_a_pinned_forge_url() {
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("iris", 9001)], &[sample_spec("iris", 9001)],
); );
let url = out let url = out
@ -1344,8 +1279,6 @@ mod tests {
fn render_flake_pins_hyperhive_by_url_and_follows_its_nixpkgs() { fn render_flake_pins_hyperhive_by_url_and_follows_its_nixpkgs() {
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
); );
assert!( assert!(
@ -1374,8 +1307,6 @@ mod tests {
fn render_flake_names_each_agent_container_after_the_agent() { fn render_flake_names_each_agent_container_after_the_agent() {
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
); );
assert!( assert!(
@ -1401,49 +1332,6 @@ mod tests {
); );
} }
#[test]
fn render_flake_pins_claude_path_without_adding_an_input() {
// The host-pinned claude travels as a bare store path assigned to
// an option — deliberately NOT as a flake input. Containers share
// the host store, so the build is already reachable; making it an
// input would re-copy it as a reference-less `-source` and strip
// the closure the binary actually needs.
let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
Some("/nix/store/cccc-claude-code-2.1.220"),
8000,
&[sample_spec("alice", 9001)],
);
assert!(
out.contains(
"services.hyperhive.agent.claudeCodePath = \"/nix/store/cccc-claude-code-2.1.220\";"
),
"claude path assigned as a plain string literal:\n{out}"
);
assert!(
!out.contains("claude-code-2.1.220\".url"),
"the pinned claude must not become a flake input:\n{out}"
);
}
#[test]
fn render_flake_omits_claude_path_when_unset() {
// `None` = no host-level pin: the option is left undefined so the
// agent module keeps its own nixpkgs' `claude-code` (and keeps it
// in `environment.systemPackages`, which is what makes the
// unpinned case self-contained).
let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)],
);
assert!(
!out.contains("claudeCodePath"),
"no claude assignment when unpinned:\n{out}"
);
}
#[test] #[test]
fn render_flake_emits_follows_for_agents_declaring_nixpkgs() { fn render_flake_emits_follows_for_agents_declaring_nixpkgs() {
// `bitburner` + `dmatrix` declare `nixpkgs` at their root, while // `bitburner` + `dmatrix` declare `nixpkgs` at their root, while
@ -1454,8 +1342,6 @@ mod tests {
}; };
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[ &[
sample_spec("argus", 9001), sample_spec("argus", 9001),
with_nixpkgs("bitburner", 9002), with_nixpkgs("bitburner", 9002),
@ -1482,8 +1368,6 @@ mod tests {
fn render_flake_builds_every_agent_from_agent_base() { fn render_flake_builds_every_agent_from_agent_base() {
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001), sample_spec("ruth", 9002)], &[sample_spec("alice", 9001), sample_spec("ruth", 9002)],
); );
assert_eq!( assert_eq!(
@ -1507,8 +1391,6 @@ mod tests {
fn render_flake_skips_canonical_follows_when_the_lock_declares_none() { fn render_flake_skips_canonical_follows_when_the_lock_declares_none() {
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
); );
// No agent-side follows when the lock declares nothing // No agent-side follows when the lock declares nothing
@ -1538,8 +1420,6 @@ mod tests {
}; };
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[alice, android], &[alice, android],
); );
assert!( assert!(
@ -1588,8 +1468,6 @@ mod tests {
} }
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
); );
unsafe { unsafe {
@ -1629,8 +1507,6 @@ mod tests {
} }
render( render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
) )
}; };
@ -1824,8 +1700,6 @@ mod tests {
} }
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
); );
unsafe { unsafe {
@ -1855,8 +1729,6 @@ mod tests {
}; };
let out = render( let out = render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[at("alice", 9001, "aaaa"), at("bob", 9002, "bbbb")], &[at("alice", 9001, "aaaa"), at("bob", 9002, "bbbb")],
); );
assert!( assert!(
@ -1882,8 +1754,6 @@ mod tests {
let args = |agents: &[AgentSpec]| { let args = |agents: &[AgentSpec]| {
render_flake( render_flake(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
agents, agents,
) )
}; };
@ -2038,8 +1908,6 @@ mod tests {
let render = || { let render = || {
render( render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
) )
}; };
@ -2100,11 +1968,12 @@ mod tests {
} }
#[test] #[test]
fn render_flake_carries_no_otel_enablement() { fn render_flake_carries_no_otel_config_and_no_per_host_value() {
// The agent module defaults otel on and finds the collector by name, so // The agent module defaults otel on and finds the collector by name, so
// the flake carries no enable, endpoint or protocol line however the // the flake carries no enable, endpoint or protocol line however the
// host's OTEL env is set. Only the per-hive resource labels are // host's OTEL env is set. The resource labels, dashboard port and
// rendered, and only when set. // pinned claude differ per host and reach the agent at container start
// (`lifecycle::hive_facts`), so none of them is rendered either.
// //
// Serialised against every other env-mutating test in the crate. // Serialised against every other env-mutating test in the crate.
// `stats::otel_metrics::tests` perturbs OTEL env too, and lands in this // `stats::otel_metrics::tests` perturbs OTEL env too, and lands in this
@ -2115,8 +1984,6 @@ mod tests {
let render = || { let render = || {
render( render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
) )
}; };
@ -2139,10 +2006,11 @@ mod tests {
std::env::remove_var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES"); std::env::remove_var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES");
} }
// Presence control: the render is the real flake, carrying the pin. // Presence controls: the render is the real flake, carrying the pin and
// the harness env block the dashboard port used to sit in.
assert!( assert!(
plain.contains("rev=0123abcd"), plain.contains("rev=0123abcd") && plain.contains("HIVE_PORT = toString port;"),
"the hyperhive pin line must be rendered:\n{plain}" "the hyperhive pin and the harness env must be rendered:\n{plain}"
); );
for (name, flake) in [("plain", &plain), ("labelled", &labelled)] { for (name, flake) in [("plain", &plain), ("labelled", &labelled)] {
for needle in [ for needle in [
@ -2150,6 +2018,11 @@ mod tests {
"agent.otel.endpoint", "agent.otel.endpoint",
"agent.otel.protocol", "agent.otel.protocol",
"c.example", "c.example",
"extraResourceAttributes",
"deployment.environment",
"claudeCodePath",
"dashboardPort",
"HIVE_DASHBOARD_PORT",
] { ] {
assert!( assert!(
!flake.contains(needle), !flake.contains(needle),
@ -2157,16 +2030,6 @@ mod tests {
); );
} }
} }
assert!(
!plain.contains("agent.otel.extraResourceAttributes"),
"extraResourceAttributes must not appear when unset:\n{plain}"
);
assert!(
labelled.contains(
"services.hyperhive.agent.otel.extraResourceAttributes = \"deployment.environment=prod\";"
),
"extraResourceAttributes must be rendered when set:\n{labelled}"
);
} }
#[test] #[test]
fn render_flake_injects_github_disable_only_when_signalled() { fn render_flake_injects_github_disable_only_when_signalled() {
@ -2181,8 +2044,6 @@ mod tests {
let render = || { let render = || {
render( render(
"git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd",
None,
8000,
&[sample_spec("alice", 9001)], &[sample_spec("alice", 9001)],
) )
}; };
@ -2206,33 +2067,6 @@ mod tests {
"github.enable = false must be injected when the host disables it:\n{disabled}" "github.enable = false must be injected when the host disables it:\n{disabled}"
); );
} }
/// The agent reads its memory cap at boot from the copy
/// `lifecycle::write_dropins` hands it, so the cap is no build input.
#[test]
fn render_hive_flake_is_identical_across_memory_caps() {
let hive = |cap: &str| HiveEnv {
agent_memory_max: cap.to_owned(),
..HiveEnv::default()
};
let agents = [sample_spec("alice", 9001)];
let small = render_hive_flake_at(
&hive("4G"),
"git+https://forge.example/h.git?rev=abc",
&agents,
)
.expect("render");
let large = render_hive_flake_at(
&hive("16G"),
"git+https://forge.example/h.git?rev=abc",
&agents,
)
.expect("render");
assert_eq!(
small, large,
"the rendered flake must not depend on the memory cap"
);
}
} }
#[cfg(test)] #[cfg(test)]

View file

@ -11,8 +11,8 @@
//! ⚠️ **The lock has to be crate-wide, not per-module.** `hive-bash-mcp`'s //! ⚠️ **The lock has to be crate-wide, not per-module.** `hive-bash-mcp`'s
//! sibling helper records why: two separate per-module mutexes serialise //! sibling helper records why: two separate per-module mutexes serialise
//! nothing against each other, and that produced a CI-only flake. The pair that //! nothing against each other, and that produced a CI-only flake. The pair that
//! motivated this one is `meta::tests` (which renders a flake from //! motivated this one is `meta::tests` (which renders a flake with
//! `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`) and `stats::otel_metrics::tests` (which asserts which //! `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES` set) and `stats::otel_metrics::tests` (which asserts which
//! variable enables the exporter) — different modules, same process env, same //! variable enables the exporter) — different modules, same process env, same
//! binary. //! binary.
//! //!

View file

@ -187,6 +187,15 @@ pub struct HiveFacts {
pub name: Option<String>, pub name: Option<String>,
/// `HYPERHIVE_HIVE_DOMAIN`: the hive's DNS domain. /// `HYPERHIVE_HIVE_DOMAIN`: the hive's DNS domain.
pub domain: Option<String>, pub domain: Option<String>,
/// `HIVE_DASHBOARD_PORT`: the port this hive's dashboard listens on.
pub dashboard_port: Option<u16>,
/// `HYPERHIVE_CLAUDE_CODE_PATH`: store path of the `claude-code` the
/// hive pins for its agents. The host's system closure keeps it alive;
/// the container only gets the path.
pub claude_code_path: Option<String>,
/// `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`: comma-separated `k=v`
/// entries the agent appends to its `OTEL_RESOURCE_ATTRIBUTES`.
pub otel_extra_resource_attributes: Option<String>,
} }
/// Wall-clock bound hive-priv puts on one [`PrivRequest::MeasureAgentDisk`] /// Wall-clock bound hive-priv puts on one [`PrivRequest::MeasureAgentDisk`]

View file

@ -367,24 +367,54 @@ fn container_rootfs(container: &str) -> PathBuf {
/// Render [`HiveFacts`] as the systemd environment file the agent loads, /// Render [`HiveFacts`] as the systemd environment file the agent loads,
/// one `KEY=value` line per present fact. /// one `KEY=value` line per present fact.
/// ///
/// A value is refused unless it is DNS-name characters only, so it can carry /// Each value is refused, never escaped, unless it is made only of the
/// neither a newline (a second assignment) nor anything an environment file /// characters its fact can hold. None of those sets admits a newline (a
/// or a shell would interpret. /// second assignment), whitespace, a quote, `\` or `$`: the agent hands each
/// line to systemd and its shells, which would read those as syntax.
fn render_hive_facts(facts: &HiveFacts) -> Result<String> { fn render_hive_facts(facts: &HiveFacts) -> Result<String> {
use std::fmt::Write as _; use std::fmt::Write as _;
let dns = |b: u8| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_');
let port = facts.dashboard_port.map(|p| p.to_string());
let mut out = String::new(); let mut out = String::new();
for (key, value) in [ for (key, value, max_len, allowed) in [
("HYPERHIVE_HIVE_NAME", &facts.name), (
("HYPERHIVE_HIVE_DOMAIN", &facts.domain), "HYPERHIVE_HIVE_NAME",
facts.name.as_ref(),
253,
&dns as &dyn Fn(u8) -> bool,
),
("HYPERHIVE_HIVE_DOMAIN", facts.domain.as_ref(), 253, &dns),
("HIVE_DASHBOARD_PORT", port.as_ref(), 5, &|b: u8| {
b.is_ascii_digit()
}),
// A store path's name characters, plus the `/` of its prefix; the
// prefix itself is checked below.
(
"HYPERHIVE_CLAUDE_CODE_PATH",
facts.claude_code_path.as_ref(),
256,
&|b: u8| dns(b) || matches!(b, b'+' | b'?' | b'=' | b'/'),
),
// `k=v,k=v`, values percent-encoded as the OTEL spec has them.
(
"HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES",
facts.otel_extra_resource_attributes.as_ref(),
1024,
&|b: u8| dns(b) || matches!(b, b'=' | b',' | b'/' | b':' | b'%' | b'+' | b'@'),
),
] { ] {
let Some(value) = value else { continue }; let Some(value) = value else { continue };
if value.is_empty() if value.is_empty() || value.len() > max_len || !value.bytes().all(allowed) {
|| value.len() > 253 bail!(
|| !value "invalid {key} value {value:?}: must be 1-{max_len} chars of its fact's character set"
.bytes() );
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_')) }
if key == "HYPERHIVE_CLAUDE_CODE_PATH"
&& !value
.strip_prefix("/nix/store/")
.is_some_and(|name| !name.is_empty() && !name.contains('/'))
{ {
bail!("invalid {key} value {value:?}: must be 1-253 chars of [A-Za-z0-9._-]"); bail!("invalid {key} value {value:?}: must be a top-level /nix/store path");
} }
let _ = writeln!(out, "{key}={value}"); let _ = writeln!(out, "{key}={value}");
} }
@ -711,14 +741,15 @@ mod tests {
let both = HiveFacts { let both = HiveFacts {
name: Some("pr1ma".to_owned()), name: Some("pr1ma".to_owned()),
domain: Some("pr1ma.darkest.space".to_owned()), domain: Some("pr1ma.darkest.space".to_owned()),
..HiveFacts::default()
}; };
assert_eq!( assert_eq!(
render_hive_facts(&both).unwrap(), render_hive_facts(&both).unwrap(),
"HYPERHIVE_HIVE_NAME=pr1ma\nHYPERHIVE_HIVE_DOMAIN=pr1ma.darkest.space\n" "HYPERHIVE_HIVE_NAME=pr1ma\nHYPERHIVE_HIVE_DOMAIN=pr1ma.darkest.space\n"
); );
let domain_only = HiveFacts { let domain_only = HiveFacts {
name: None,
domain: Some("h1.example.test".to_owned()), domain: Some("h1.example.test".to_owned()),
..HiveFacts::default()
}; };
assert_eq!( assert_eq!(
render_hive_facts(&domain_only).unwrap(), render_hive_facts(&domain_only).unwrap(),
@ -734,7 +765,66 @@ mod tests {
for bad in ["", "pr1ma\nLD_PRELOAD=/x", "a b", "a\"b", "$(x)"] { for bad in ["", "pr1ma\nLD_PRELOAD=/x", "a b", "a\"b", "$(x)"] {
let facts = HiveFacts { let facts = HiveFacts {
name: Some(bad.to_owned()), name: Some(bad.to_owned()),
domain: None, ..HiveFacts::default()
};
assert!(
render_hive_facts(&facts).is_err(),
"{bad:?} must be refused"
);
}
}
/// The three per-host values agents used to get at build time arrive
/// here, one line each.
#[test]
fn hive_facts_render_the_per_host_values() {
let facts = HiveFacts {
dashboard_port: Some(7123),
claude_code_path: Some("/nix/store/0abc-claude-code-2.1.220".to_owned()),
otel_extra_resource_attributes: Some(
"deployment.environment=prod,team=a%2Fb".to_owned(),
),
..HiveFacts::default()
};
assert_eq!(
render_hive_facts(&facts).unwrap(),
"HIVE_DASHBOARD_PORT=7123\n\
HYPERHIVE_CLAUDE_CODE_PATH=/nix/store/0abc-claude-code-2.1.220\n\
HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=deployment.environment=prod,team=a%2Fb\n"
);
}
/// The claude path is a top-level store path or nothing, and neither
/// value may carry generator syntax.
#[test]
fn hive_facts_refuse_bad_claude_paths_and_resource_attributes() {
for bad in [
"/nix/store/",
"/nix/store/abc-claude/bin/claude",
"/tmp/claude",
"nix/store/abc-claude",
"/nix/store/abc-claude\nLD_PRELOAD=/x",
] {
let facts = HiveFacts {
claude_code_path: Some(bad.to_owned()),
..HiveFacts::default()
};
assert!(
render_hive_facts(&facts).is_err(),
"{bad:?} must be refused"
);
}
for bad in [
"a=b\nPATH=/x",
"a=${HOME}",
"a=b c",
"a=\"b\"",
"a=b\\c",
"",
] {
let facts = HiveFacts {
otel_extra_resource_attributes: Some(bad.to_owned()),
..HiveFacts::default()
}; };
assert!( assert!(
render_hive_facts(&facts).is_err(), render_hive_facts(&facts).is_err(),
@ -752,10 +842,7 @@ mod tests {
bridge: "hive0".to_owned(), bridge: "hive0".to_owned(),
gateway_ip: "10.42.0.1".to_owned(), gateway_ip: "10.42.0.1".to_owned(),
}; };
let facts = HiveFacts { let facts = HiveFacts::default();
name: None,
domain: None,
};
for name in ["hive-forge", "hive-matrix"] { for name in ["hive-forge", "hive-matrix"] {
let errs = [ let errs = [
handle_write_nspawn_flags(name, &[], &isolation, &[], &facts).unwrap_err(), handle_write_nspawn_flags(name, &[], &isolation, &[], &facts).unwrap_err(),

View file

@ -28,6 +28,28 @@ let
which the hive writes before every start, so no agent build sets it. which the hive writes before every start, so no agent build sets it.
Remove the setting. Remove the setting.
''; '';
claudeCodePathRemoved = ''
An agent reads the claude-code its hive pins at runtime, from
HYPERHIVE_CLAUDE_CODE_PATH in /etc/hyperhive-hive-facts, so no agent build
sets it. Pin it host-side with
services.hyperhive.deploy.hive-controller.claudeCodePackage.
'';
# `claude` on PATH. A hive that pins a claude-code
# (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its
# store path over at container start as `HYPERHIVE_CLAUDE_CODE_PATH`, and
# the host's system closure keeps it alive: nothing in this container
# references it. With no pin, or a path that is not there, this container's
# own claude-code runs. `exec -a claude` keeps the bare name in argv[0],
# which is how the harness finds its claude child (`find_claude_child`).
ownClaude = import ../claude-code-no-alsa.nix lib pkgs.claude-code;
claude = pkgs.writeShellScriptBin "claude" ''
pinned=''${HYPERHIVE_CLAUDE_CODE_PATH:-}
if [ -n "$pinned" ] && [ -x "$pinned/bin/claude" ]; then
exec -a claude "$pinned/bin/claude" "$@"
fi
exec -a claude ${ownClaude}/bin/claude "$@"
'';
in in
{ {
imports = [ imports = [
@ -68,6 +90,12 @@ in
"agent" "agent"
"claudeMemoryMaxBytes" "claudeMemoryMaxBytes"
] memoryMaxBytesRemoved) ] memoryMaxBytesRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"agent"
"claudeCodePath"
] claudeCodePathRemoved)
]; ];
options.services.hyperhive.agent.icon = lib.mkOption { options.services.hyperhive.agent.icon = lib.mkOption {
@ -87,33 +115,6 @@ in
''; '';
}; };
options.services.hyperhive.agent.claudeCodePath = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/nix/store/…-claude-code-2.1.220";
description = ''
Store path of the `claude-code` this agent runs, or `null` (the
default) to use the `claude-code` from the container's own
nixpkgs.
Set by the generated meta flake when the operator sets
`services.hyperhive.deploy.hive-controller.claudeCodePackage` host-side, so a hive
can run a claude built from a *different* nixpkgs than the one
its agents evaluate against — a release channel can trail
unstable by weeks on this one package. It arrives as a path and
not a package because agents share the host's `/nix/store`, so
the build is already reachable here with its whole closure and
has nothing to travel.
When set, `claude` on PATH is a symlink to this path's
`bin/claude` and the container's own `claude-code` is dropped, so
there is only ever one claude in the container. Note that neither
the symlink nor anything else in the container's closure *refers*
to the target — keeping it alive is the host's job, see
`services.hyperhive.deploy.hive-controller.claudeCodePackage`.
'';
};
config = { config = {
assertions = [ assertions = [
# Guard the inputs-routed-as-output pattern: the agent flake.nix is # Guard the inputs-routed-as-output pattern: the agent flake.nix is
@ -240,28 +241,7 @@ in
config.services.hyperhive.agent.packages.hive-metric config.services.hyperhive.agent.packages.hive-metric
] ]
++ [ ++ [
( claude
if config.services.hyperhive.agent.claudeCodePath == null then
import ../claude-code-no-alsa.nix lib pkgs.claude-code
else
# Host-pinned claude: a symlink farm around a path the
# container was handed as text. It has to be a derivation —
# `environment.systemPackages` coerces a store-path *string*
# with `toDerivation`, i.e. `builtins.storePath`, which pure
# evaluation rejects (`systemd.services.*.path` does the same,
# which is why the harness gets this via PATH like everything
# else rather than a unit-level entry). Interpolating the path
# into the builder is just text, so it evaluates anywhere.
#
# The symlink registers no store reference — the target isn't
# among this derivation's inputs, so nothing here keeps the
# binary alive. That is deliberate and it is the host's job:
# see `services.hyperhive.deploy.hive-controller.claudeCodePackage`.
pkgs.runCommandLocal "claude-code-pinned" { } ''
mkdir -p "$out/bin"
ln -s ${config.services.hyperhive.agent.claudeCodePath}/bin/claude "$out/bin/claude"
''
)
] ]
++ (with pkgs; [ ++ (with pkgs; [
bashInteractive bashInteractive

View file

@ -2,8 +2,17 @@
# agent's closure does not carry it. # agent's closure does not carry it.
# #
# hive-priv writes `/etc/hyperhive-hive-facts` into the container before every # hive-priv writes `/etc/hyperhive-hive-facts` into the container before every
# start (`HYPERHIVE_HIVE_NAME`, `HYPERHIVE_HIVE_DOMAIN`, one `KEY=value` line # start, one `KEY=value` line each:
# each). Two readers hand those lines on: #
# - `HYPERHIVE_HIVE_NAME`, `HYPERHIVE_HIVE_DOMAIN`;
# - `HIVE_DASHBOARD_PORT`, read by the harness's web UI (7000 when absent);
# - `HYPERHIVE_CLAUDE_CODE_PATH`, the claude-code the hive pins (`claude` in
# ./default.nix runs this container's own when absent);
# - `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`, appended to
# `OTEL_RESOURCE_ATTRIBUTES` by ./otel.nix (no extra labels when absent).
#
# Any of them may be absent: a hive-c0re older than the agent's build writes
# fewer. Two readers hand the lines on:
# #
# - an environment generator, for the container's systemd: it passes them to # - an environment generator, for the container's systemd: it passes them to
# every unit it starts, `machinectl shell` sessions (`hivectl choom`) # every unit it starts, `machinectl shell` sessions (`hivectl choom`)
@ -23,8 +32,8 @@
let let
userName = config.services.hyperhive.agent.user.name; userName = config.services.hyperhive.agent.user.name;
# Shell builtins only: nothing is assumed on PATH this early. Only the two # Shell builtins only: nothing is assumed on PATH this early. Only the known
# known keys are passed on. `|| [ -n "$key" ]` keeps a last line that has # keys are passed on. `|| [ -n "$key" ]` keeps a last line that has
# no trailing newline. # no trailing newline.
generator = pkgs.writeShellScript "hyperhive-hive-facts" '' generator = pkgs.writeShellScript "hyperhive-hive-facts" ''
hive=unknown hive=unknown
@ -32,7 +41,8 @@ let
while IFS='=' read -r key value || [ -n "$key" ]; do while IFS='=' read -r key value || [ -n "$key" ]; do
case "$key" in case "$key" in
HYPERHIVE_HIVE_NAME) hive=$value ;; HYPERHIVE_HIVE_NAME) hive=$value ;;
HYPERHIVE_HIVE_DOMAIN) ;; HYPERHIVE_HIVE_DOMAIN | HIVE_DASHBOARD_PORT) ;;
HYPERHIVE_CLAUDE_CODE_PATH | HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES) ;;
*) continue ;; *) continue ;;
esac esac
printf '%s=%s\n' "$key" "$value" printf '%s=%s\n' "$key" "$value"

View file

@ -42,9 +42,17 @@ let
# agent, and their metric names already tell them apart. No `hive`: the # agent, and their metric names already tell them apart. No `hive`: the
# swarm collector stamps it from the receiver that accepted the hive's # swarm collector stamps it from the receiver that accepted the hive's
# push (nix/host-modules/swarm-otel.nix), so the build names no hive. # push (nix/host-modules/swarm-otel.nix), so the build names no hive.
resourceAttributes = resourceAttributes = "service.name=hyperhive-agent,agent=${userName},swarm=${swarmDisplayName}";
"service.name=hyperhive-agent,agent=${userName},swarm=${swarmDisplayName}"
+ lib.optionalString (cfg.extraResourceAttributes != "") ",${cfg.extraResourceAttributes}"; # Prints `OTEL_RESOURCE_ATTRIBUTES`: the labels above, then the hive's own
# (`HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`), which differ per host and so
# arrive at container start. That variable comes from ./hive-facts.nix's
# generator, which runs before this one and is absent when the hive sets no
# labels.
resourceGenerator = pkgs.writeShellScript "hyperhive-otel-resource" ''
printf 'OTEL_RESOURCE_ATTRIBUTES=%s%s\n' ${lib.escapeShellArg resourceAttributes} \
"''${HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES:+,$HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES}"
'';
# The variables an OTEL SDK reads on its own, in any language, in any # The variables an OTEL SDK reads on its own, in any language, in any
# process here. # process here.
@ -62,7 +70,6 @@ let
# which Prometheus/Mimir-family backends (incl. grafana-lgtm) # which Prometheus/Mimir-family backends (incl. grafana-lgtm)
# silently drop without a deltatocumulative processor. # silently drop without a deltatocumulative processor.
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE = "cumulative"; OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE = "cumulative";
OTEL_RESOURCE_ATTRIBUTES = resourceAttributes;
}; };
# The exporter's NAME picks the wire protocol, so `protocol` has to be # The exporter's NAME picks the wire protocol, so `protocol` has to be
@ -118,19 +125,6 @@ in
''; '';
}; };
extraResourceAttributes = lib.mkOption {
type = lib.types.str;
default = "";
internal = true;
description = ''
Extra comma-separated entries appended to
`OTEL_RESOURCE_ATTRIBUTES` after the built-in
`service.name` / `agent` / `swarm` labels.
Host-driven via
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
'';
};
debug = lib.mkOption { debug = lib.mkOption {
type = lib.types.bool; type = lib.types.bool;
default = false; default = false;
@ -167,6 +161,14 @@ in
systemd.globalEnvironment = producerEnv; systemd.globalEnvironment = producerEnv;
environment.variables = producerEnv; environment.variables = producerEnv;
# `OTEL_RESOURCE_ATTRIBUTES` is in neither: a unit's own `Environment=`
# would override what the generator computes at boot. Units get it from
# the manager's environment; login shells run the same generator after
# ./hive-facts.nix's snippet has exported the hive's labels.
environment.etc."systemd/system-environment-generators/60-hyperhive-otel-resource".source =
resourceGenerator;
environment.extraInit = lib.mkAfter ''export "$(${resourceGenerator})"'';
# The journal this container writes is reachable from inside it and # The journal this container writes is reachable from inside it and
# from nowhere else, which is why the forwarder runs here rather than # from nowhere else, which is why the forwarder runs here rather than
# the hive reading the container's journal from outside: the host-side # the hive reading the container's journal from outside: the host-side

View file

@ -110,9 +110,9 @@ let
# in whole rather than interpolated — that is also the ONLY thing # in whole rather than interpolated — that is also the ONLY thing
# gc-rooting it: the resulting string carries store context, so this # gc-rooting it: the resulting string carries store context, so this
# /etc entry genuinely references the package and the host's system # /etc entry genuinely references the package and the host's system
# closure holds it alive. Nothing container-side can — meta.rs writes # closure holds it alive. Nothing container-side can: hive-c0re hands
# the path into each agent's generated flake as a plain string # the path to each container as text (`HYPERHIVE_CLAUDE_CODE_PATH`),
# literal, and text is not a reference. Hence the assertion below: do # and text is not a reference. Hence the assertion below: do
# NOT discard this context, and do not route the path through # NOT discard this context, and do not route the path through
# anything that drops it. The failure mode is a garbage-collected # anything that drops it. The failure mode is a garbage-collected
# `claude` and a hive that can't take a turn, weeks after the commit # `claude` and a hive that can't take a turn, weeks after the commit

View file

@ -326,8 +326,9 @@
}; };
}; };
# What this hive hands its own agents through the meta-flake render. Per # What this hive hands its own agents at container start, through
# host, so two hives may set them differently. # `/etc/hyperhive-hive-facts`. Per host, so two hives may set them
# differently.
options.services.hyperhive.deploy.hive-controller = { options.services.hyperhive.deploy.hive-controller = {
claudeCodePackage = lib.mkOption { claudeCodePackage = lib.mkOption {
type = lib.types.nullOr lib.types.package; type = lib.types.nullOr lib.types.package;
@ -350,22 +351,22 @@
What travels into the container is the **store path**, not the What travels into the container is the **store path**, not the
derivation: agents share the host's `/nix/store`, so the binary derivation: agents share the host's `/nix/store`, so the binary
and its full closure are already reachable there — nothing and its full closure are already reachable there — nothing
needs rebuilding or copying. hive-c0re writes the path into needs rebuilding or copying. hive-c0re hands the path to each
each agent's generated flake as a plain string literal (a bare container at start as `HYPERHIVE_CLAUDE_CODE_PATH`, and the agent's
path fed to `lib.types.package` would run `builtins.storePath`, `claude` on PATH runs it, or the container's own `claude-code` when
which is illegal under pure evaluation) and the agent module the variable is absent or names nothing. A change reaches an agent
puts its `bin/` on the harness's PATH. at its next container start, not through a rebuild.
The flip side of a plain string is that nothing in the agent's The flip side of a plain string is that nothing in the agent's
own closure refers to it, so the container cannot keep it own closure refers to it, so the container cannot keep it
alive. The **host** does that instead: this package is alive. The **host** does that instead: this package is
interpolated into `/etc/hyperhive/serve.json`, which puts it in interpolated into `/etc/hyperhive/serve.json`, which puts it in
the host's system closure — so it is gc-rooted by the running the host's system closure, so it is gc-rooted by the running
generation for exactly as long as that generation is the one generation. ⚠️ An agent started under an older generation keeps
the agents were rendered from. The cost is that that generation's path until its next start, and only the old
`nix-collect-garbage` cannot reclaim an old `claude-code` until generation roots it: collect old generations after restarting
every agent has been rebuilt past it and the old generations the agents, or an agent still running the old path falls back to
are gone. its own `claude-code` at its next `claude` spawn.
The swarm's `swarm-claude-login` and `swarm-review-classifier` The swarm's `swarm-claude-login` and `swarm-review-classifier`
containers, where this host runs them, follow the same pin and containers, where this host runs them, follow the same pin and
@ -377,7 +378,11 @@
dashboardPort = lib.mkOption { dashboardPort = lib.mkOption {
type = lib.types.port; type = lib.types.port;
default = 7000; default = 7000;
description = "TCP port this hive's hive-c0re dashboard listens on."; description = ''
TCP port this hive's hive-c0re dashboard listens on. Agents get it
at container start as `HIVE_DASHBOARD_PORT`, for their web UI's
links back to the dashboard.
'';
}; };
}; };
} }

View file

@ -236,7 +236,7 @@ in
}; };
# Read by this hive's hive-c0re, which stamps it on its own export and # Read by this hive's hive-c0re, which stamps it on its own export and
# renders it into its agents', so it is a per-host choice. # hands it to its agents at container start, so it is a per-host choice.
options.services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes = lib.mkOption { options.services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = ""; default = "";

View file

@ -123,6 +123,25 @@ let
&& lib.all (e: s.exporters.${e}.endpoint == endpoint) s.service.pipelines.logs.exporters && lib.all (e: s.exporters.${e}.endpoint == endpoint) s.service.pipelines.logs.exporters
&& !(agentNoOtel.systemd.globalEnvironment ? OTEL_EXPORTER_OTLP_ENDPOINT); && !(agentNoOtel.systemd.globalEnvironment ? OTEL_EXPORTER_OTLP_ENDPOINT);
} }
{
# The hive's extra labels arrive at container start, so the build
# cannot hold the whole `OTEL_RESOURCE_ATTRIBUTES`: a boot-time
# generator computes it, and a unit-level `Environment=` copy would
# override that. The endpoint beside it is the presence control for the
# two absence lookups; `agentNoOtel` is the control for the generator.
name = "an agent's OTEL resource labels come from a boot-time generator, not the build";
ok =
let
generator =
m: m.environment.etc ? "systemd/system-environment-generators/60-hyperhive-otel-resource";
in
agentDefault.systemd.globalEnvironment ? OTEL_EXPORTER_OTLP_ENDPOINT
&& !(agentDefault.systemd.globalEnvironment ? OTEL_RESOURCE_ATTRIBUTES)
&& agentDefault.environment.variables ? OTEL_EXPORTER_OTLP_ENDPOINT
&& !(agentDefault.environment.variables ? OTEL_RESOURCE_ATTRIBUTES)
&& generator agentDefault
&& !(generator agentNoOtel);
}
{ {
# Presence control for the two cases above: with the switch off there # Presence control for the two cases above: with the switch off there
# is no collector in the container at all, so their passing is about # is no collector in the container at all, so their passing is about

View file

@ -282,7 +282,7 @@ let
{ {
# The CLI is the one the agents run, not the container's own nixpkgs'. # The CLI is the one the agents run, not the container's own nixpkgs'.
# Unpinned is the control: it falls back to the container's own `claude-code`. # Unpinned is the control: it falls back to the container's own `claude-code`.
# serve.json is what hive-c0re renders every agent's `claudeCodePath` from. # serve.json is what hive-c0re hands every agent's `HYPERHIVE_CLAUDE_CODE_PATH` from.
name = "the agents, the claude login minter and the review classifier run the claude-code this host pins"; name = "the agents, the claude login minter and the review classifier run the claude-code this host pins";
ok = ok =
let let

View file

@ -1,7 +1,9 @@
# `checks.script-test-agent-hive-facts` — runs the two readers of # `checks.script-test-agent-hive-facts` — runs the two readers of
# `/etc/hyperhive-hive-facts` in ../agent-modules/hive-facts.nix: the systemd # `/etc/hyperhive-hive-facts` in ../agent-modules/hive-facts.nix: the systemd
# environment generator and the login-shell snippet `/etc/set-environment` # environment generator and the login-shell snippet `/etc/set-environment`
# sources. The cases are in ./agent-hive-facts.sh. # sources. Also the two consumers of what they pass on: ../agent-modules/otel.nix's
# `OTEL_RESOURCE_ATTRIBUTES` generator and ../agent-modules/default.nix's `claude`.
# The cases are in ./agent-hive-facts.sh.
# #
# What runs is the rendered generator and snippet. The one edit is the facts # What runs is the rendered generator and snippet. The one edit is the facts
# file's path, moved under the build directory, and the snippet's reference to # file's path, moved under the build directory, and the snippet's reference to
@ -26,6 +28,11 @@ let
; ;
machine = agentWith { services.hyperhive.agent.user.name = "a1"; }; machine = agentWith { services.hyperhive.agent.user.name = "a1"; };
# As text: building it would build the claude-code it falls back to.
claude = lib.findFirst (
p: (p.name or "") == "claude"
) (throw "no `claude` in systemPackages") machine.environment.systemPackages;
in in
pkgs.runCommand "hyperhive-script-test-agent-hive-facts" pkgs.runCommand "hyperhive-script-test-agent-hive-facts"
{ {
@ -33,6 +40,9 @@ pkgs.runCommand "hyperhive-script-test-agent-hive-facts"
machine.environment.etc."systemd/system-environment-generators/50-hyperhive-hive-facts".source; machine.environment.etc."systemd/system-environment-generators/50-hyperhive-hive-facts".source;
PROFILE = machine.environment.etc."hyperhive/hive-facts.sh".source; PROFILE = machine.environment.etc."hyperhive/hive-facts.sh".source;
EXTRA_INIT = machine.environment.extraInit; EXTRA_INIT = machine.environment.extraInit;
OTEL_GENERATOR =
machine.environment.etc."systemd/system-environment-generators/60-hyperhive-otel-resource".source;
CLAUDE_WRAPPER = builtins.unsafeDiscardStringContext claude.text;
SHELLS = "${pkgs.dash}/bin/dash ${pkgs.bash}/bin/sh"; SHELLS = "${pkgs.dash}/bin/dash ${pkgs.bash}/bin/sh";
} }
'' ''

View file

@ -40,7 +40,7 @@ check() {
for sh in $SHELLS; do for sh in $SHELLS; do
# Only what the snippet exported reaches the child's environment. No PATH: # Only what the snippet exported reaches the child's environment. No PATH:
# the snippet must not need one. # the snippet must not need one.
got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(HYPERHIVE_|CLAUDE_|LD_PRELOAD=|__hyperhive)' | sort) got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(HYPERHIVE_|HIVE_DASHBOARD_PORT=|CLAUDE_|LD_PRELOAD=|__hyperhive)' | sort)
if [ "$got" != "$(printf '%s\n' "$want" | sort)" ]; then if [ "$got" != "$(printf '%s\n' "$want" | sort)" ]; then
fail "$sh: a child of the sourcing shell saw:"$'\n'"$got" fail "$sh: a child of the sourcing shell saw:"$'\n'"$got"
fi fi
@ -61,11 +61,69 @@ check "a last line without a newline is kept" "HYPERHIVE_HIVE_DOMAIN=pr1ma.examp
HYPERHIVE_HIVE_NAME=pr1ma HYPERHIVE_HIVE_NAME=pr1ma
CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=pr1ma-a1" CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=pr1ma-a1"
printf 'HIVE_DASHBOARD_PORT=7123\nHYPERHIVE_CLAUDE_CODE_PATH=/nix/store/0abc-claude-code\nHYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod\n' >"$facts"
check "the per-host values, a value holding = kept whole" "HIVE_DASHBOARD_PORT=7123
HYPERHIVE_CLAUDE_CODE_PATH=/nix/store/0abc-claude-code
HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod
CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1"
: >"$facts" : >"$facts"
check "an empty file (no facts)" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" check "an empty file (no facts)" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1"
rm -f "$facts" rm -f "$facts"
check "no file" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" check "no file" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1"
# OTEL_RESOURCE_ATTRIBUTES, for units (the generator, run with what the
# hive-facts generator printed) and for login shells (the extraInit line, after
# the snippet). The base labels come from the build; the hive's are appended.
base="service.name=hyperhive-agent,agent=a1,swarm=unknown"
otel_check() {
case=$1
count=$((count + 1))
local want="OTEL_RESOURCE_ATTRIBUTES=$2" got sh passed
mapfile -t passed < <(env -i "$dir/generator")
got=$(env -i "${passed[@]}" "$OTEL_GENERATOR")
if [ "$got" != "$want" ]; then fail "generator printed: $got, wanted: $want"; fi
for sh in $SHELLS; do
got=$(env -i "$sh" -c ". '$dir/profile'; $otel_init; exec '$envbin'" | grep '^OTEL_RESOURCE_ATTRIBUTES=')
if [ "$got" != "$want" ]; then fail "$sh: a login shell saw: $got"; fi
done
}
case="otel setup"
otel_init=$(printf '%s\n' "$EXTRA_INIT" | grep -F "$OTEL_GENERATOR")
if [ -z "$otel_init" ]; then fail "/etc/set-environment does not run the OTEL resource generator"; fi
if [ "$(printf '%s\n' "$EXTRA_INIT" | grep -n . | grep -F "$OTEL_GENERATOR" | cut -d: -f1)" -lt \
"$(printf '%s\n' "$EXTRA_INIT" | grep -nxF '. /etc/hyperhive/hive-facts.sh' | cut -d: -f1)" ]; then
fail "/etc/set-environment runs the OTEL resource generator before the hive-facts snippet"
fi
printf 'HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod\n' >"$facts"
otel_check "the hive's labels are appended" "$base,team=a,env=prod"
rm -f "$facts"
otel_check "no hive labels: the base alone, no trailing comma" "$base"
# `claude`: the hive's pinned build when it is there, this container's own
# otherwise. Both builds are stand-ins that print which one ran.
mkdir -p "$dir/own/bin" "$dir/pinned/bin"
printf '#!%s\necho own "$@"\n' "$BASH" >"$dir/own/bin/claude"
printf '#!%s\necho pinned "$@"\n' "$BASH" >"$dir/pinned/bin/claude"
chmod +x "$dir/own/bin/claude" "$dir/pinned/bin/claude"
printf '%s' "$CLAUDE_WRAPPER" | sed -E "s|/nix/store/[^ ]+/bin/claude|$dir/own/bin/claude|" >"$dir/claude"
case="claude setup"
if ! grep -qF "$dir/own/bin/claude" "$dir/claude"; then fail "the rewrite did not reach the fallback claude"; fi
claude_check() {
case=$1
count=$((count + 1))
local got pin=()
if [ -n "$2" ]; then pin=("HYPERHIVE_CLAUDE_CODE_PATH=$2"); fi
# Under this shell: the shebang names the build's bash, not in this sandbox.
got=$(env -i "${pin[@]}" "$BASH" "$dir/claude" -p hi)
if [ "$got" != "$3 -p hi" ]; then fail "ran: $got, wanted: $3 -p hi"; fi
}
claude_check "the pinned claude when the hive hands one over" "$dir/pinned" pinned
claude_check "the container's own claude with no pin" "" own
claude_check "the container's own claude when the pinned path is gone" "$dir/gone" own
echo "$count cases, $failures failed" echo "$count cases, $failures failed"
[ "$failures" -eq 0 ] [ "$failures" -eq 0 ]