diff --git a/docs/agent-lifecycle/config-changes.md b/docs/agent-lifecycle/config-changes.md index cd5862d4..ac36494a 100644 --- a/docs/agent-lifecycle/config-changes.md +++ b/docs/agent-lifecycle/config-changes.md @@ -118,7 +118,10 @@ declares one flake input per agent (`agent-.url = "git+/agent-configs/.git?rev="`) and one `nixosConfigurations.` output per agent. Each output wraps `inputs.agent-.nixosModules.default` with the identity + -`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module. +`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host +(the dashboard port, the pinned claude-code, extra OTEL labels) aren't in it: +`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at +start (see `docs/process/conventions.md`). Containers run against `--flake /var/lib/hyperhive/meta#`. The declared input url is the head of `main` on the agent's forge config repo @@ -195,8 +198,7 @@ The container's `--flake` ref is `/var/lib/hyperhive/meta#` (see "Meta flake" above). The agent's own `applied//flake.nix` is a fixed boilerplate that exports `nixosModules.default = import ./agent.nix`; the meta flake imports that module and -wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` / -`HIVE_DASHBOARD_PORT`. +wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`. ### Tag state machine diff --git a/docs/process/conventions.md b/docs/process/conventions.md index adc9ec66..2ab12c09 100644 --- a/docs/process/conventions.md +++ b/docs/process/conventions.md @@ -39,11 +39,15 @@ Four env vars cover the identity surface, read by name. Inside an agent container, the hive's name and domain don't come from the -agent's build. `hive-priv` writes them to `/etc/hyperhive-hive-facts` before -every container start. The agent's environment generator -(`nix/agent-modules/hive-facts.nix`) passes them to every unit, including -`hivectl choom` sessions, and `/etc/set-environment` exports them in login -shells that no unit starts (`nixos-container root-login`). Neither is part of +agent's build, and neither do the hive's other per-host values: the dashboard +port (`HIVE_DASHBOARD_PORT`), the pinned claude-code +(`HYPERHIVE_CLAUDE_CODE_PATH`) and extra OTEL labels +(`HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`). `hive-priv` writes them to +`/etc/hyperhive-hive-facts` before every container start, leaving out any the +hive doesn't set; the agent then uses its own default. The agent's environment +generator (`nix/agent-modules/hive-facts.nix`) passes them to every unit, +including `hivectl choom` sessions, and `/etc/set-environment` exports them in login +shells that no unit starts (`nixos-container root-login`). None is part of an agent's closure, and an agent started on another hive reports that hive. `hive_name` + `swarm_name` are **distinct** from diff --git a/docs/process/gotchas.md b/docs/process/gotchas.md index 343b7a61..4f16ca6a 100644 --- a/docs/process/gotchas.md +++ b/docs/process/gotchas.md @@ -145,9 +145,9 @@ and every agent runs that build. What crosses is the **store path**, not the derivation. Containers share the host's `/nix/store`, so the binary is already reachable -inside them with its whole closure; hive-c0re writes the path into each -agent's flake as a string literal and the agent module symlinks -`bin/claude` onto PATH. Two things rule out the natural-seeming alternatives: a +inside them with its whole closure. `hive-priv` writes the path into each +container's `/etc/hyperhive-hive-facts` at start, and the agent's `claude` +wrapper on PATH runs its `bin/claude`. Two things rule out the natural-seeming alternatives: a `path:/nix/store/` flake input is re-copied into the store as a reference-less `-source` (so the runtime closure never arrives), and `lib.types.package` fed a bare path runs `builtins.storePath`, which @@ -171,7 +171,7 @@ expected. -The catch is that a path written into a generated flake is text, not a +The catch is that a path handed to the container at start is text, not a reference — the container's closure doesn't keep the binary alive. The **host** does: it interpolates the package into `/etc/hyperhive/serve.json`, so it lands in the host's system closure @@ -180,7 +180,7 @@ string context, which is the load-bearing detail; discard the context anywhere on that path and `nix-collect-garbage` will eventually take the hive's `claude` out from under it. The price of the root is that `nix-collect-garbage` can't reclaim an old `claude-code` until every -agent has rebuilt past it and the old generations are gone. +agent has restarted past it and the old generations are gone. diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index 9151eb51..01678185 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -190,7 +190,7 @@ two-tier architecture, the security model, and how the options interact. The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`, `OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container -wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every +wide**, for every unit and in `/etc/profile`, so every process in an agent container exports to the hive's collector without any per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run yourself from a tool call or a shell you open in the container. @@ -209,8 +209,9 @@ the push (see [Authenticated ingest](#authenticated-ingest)). The container doesn't set it. Append additional labels for one hive via `extraResourceAttributes` (see -option reference above); pass custom per-data-point labels with -`hive-metric --labels` (see below). +option reference above). Agents get them at container start, so a change +reaches an agent at its next start, not through a rebuild. Pass custom +per-data-point labels with `hive-metric --labels` (see below). ## Log severity diff --git a/docs/turn-loop/claude-invocation.md b/docs/turn-loop/claude-invocation.md index 21539961..00cb3a15 100644 --- a/docs/turn-loop/claude-invocation.md +++ b/docs/turn-loop/claude-invocation.md @@ -33,11 +33,13 @@ harness unit's PATH. By default that's the `claude-code` in the agent's own nixpkgs (the meta flake's `nixpkgs` input) via `environment.systemPackages`. Since that's typically a release channel and this package moves fast, the operator can pin one per hive with -`services.hyperhive.deploy.hive-controller.claudeCodePackage`: hive-c0re writes its -store path into each agent's flake, and `claude` on PATH becomes a symlink to it -instead of the container's own `claude-code` — so there's only ever one -`claude` in the container. Agents pick up a new build on their -next rebuild, not live. See docs/process/gotchas.md::`claude-code` is unfree. +`services.hyperhive.deploy.hive-controller.claudeCodePackage`. `hive-priv` +writes its store path into the container's `/etc/hyperhive-hive-facts` as +`HYPERHIVE_CLAUDE_CODE_PATH` before every start. `claude` on PATH is a wrapper +that runs that build when the path exists, and the container's own +`claude-code` when the variable is unset or the path no longer exists, saying +so on stderr in the second case. A bump reaches an agent at its next container +start, not at a rebuild. See docs/process/gotchas.md::`claude-code` is unfree. Hive-enforced settings ship at `/etc/claude-code/managed-settings.json` (claude-code's canonical managed-settings path — precedence #1, diff --git a/hive-c0re/src/coordinator.rs b/hive-c0re/src/coordinator.rs index d752f947..bcb72ed4 100644 --- a/hive-c0re/src/coordinator.rs +++ b/hive-c0re/src/coordinator.rs @@ -43,8 +43,8 @@ pub struct Coordinator { /// Git rev of the hyperhive checkout this host runs. Pins the meta /// flake's `hyperhive` input; `None` makes the meta render fail. pub hyperhive_rev: Option, - /// Store path of the `claude-code` build every agent runs, written - /// into each per-agent flake as `services.hyperhive.agent.claudeCodePath`. Set by + /// Store path of the `claude-code` build every agent runs, handed to + /// each container at start as `HYPERHIVE_CLAUDE_CODE_PATH`. Set by /// the NixOS module option /// `services.hyperhive.deploy.hive-controller.claudeCodePackage` /// (which resolves the package and hands us its path). `None` = every @@ -55,9 +55,9 @@ pub struct Coordinator { /// closure and all. The host module is what keeps it from being /// garbage-collected — see that option. pub claude_code_path: Option, - /// TCP port the host's hive-c0re dashboard listens on. Inlined into - /// each per-agent flake so the agent's web UI can build the right - /// rebuild-button URL pointing back at the dashboard. + /// TCP port the host's hive-c0re dashboard listens on. Handed to each + /// container at start as `HIVE_DASHBOARD_PORT`, so the agent's web UI + /// can build the rebuild-button URL pointing back at the dashboard. pub dashboard_port: u16, /// Per-agent systemd `CPUQuota=` value (e.g. `"200%"`). Written into /// the `container@h-.service.d/` drop-in on every spawn/rebuild. @@ -193,7 +193,7 @@ pub struct HiveEnv { pub hyperhive_rev: Option, /// Store path of the `claude-code` agents run, or `None` for "each /// agent keeps the one out of its own nixpkgs". Travels into the - /// container as `services.hyperhive.agent.claudeCodePath` — a plain string, kept + /// container as `HYPERHIVE_CLAUDE_CODE_PATH` — a plain string, kept /// alive host-side by the module that resolved it. pub claude_code_path: Option, pub dashboard_port: u16, diff --git a/hive-c0re/src/lifecycle/host_config.rs b/hive-c0re/src/lifecycle/host_config.rs index 4204ede5..bb20df7e 100644 --- a/hive-c0re/src/lifecycle/host_config.rs +++ b/hive-c0re/src/lifecycle/host_config.rs @@ -25,7 +25,7 @@ use super::{ pub async fn write_dropins(name: &str, hive: &HiveEnv, paths: &AgentPaths) -> Result<()> { validate(name)?; let container = container_name(name); - set_nspawn_flags(&container, &paths.agent, &paths.claude, &paths.notes).await?; + set_nspawn_flags(&container, hive, &paths.agent, &paths.claude, &paths.notes).await?; hand_permissions(name, &paths.agent); if let Some((cpu_quota, memory_max)) = stage_limits( name, @@ -496,6 +496,7 @@ fn queue_agent_credentials(agent_name: &str, dir: Option<&Path>) -> Vec Result network_isolation_from_vars(bridge.as_deref(), subnet.as_deref()) } -/// The hive's facts every container is handed at runtime, from hive-c0re's -/// own unit environment (`nix/host-modules/hive-c0re/environment.nix`). An -/// unset or empty variable is `None`. -fn hive_facts_from_env() -> hive_priv_sock::HiveFacts { - let read = |name: &str| std::env::var(name).ok().filter(|v| !v.is_empty()); +/// The hive's facts every container is handed at runtime, so that no agent +/// build carries a value that differs between hosts. The name, domain and +/// OTEL resource labels come from hive-c0re's own unit environment +/// (`nix/host-modules/hive-c0re/environment.nix`), where an unset or empty +/// variable is `None`; the dashboard port and pinned claude from `hive`. +fn hive_facts(hive: &HiveEnv) -> hive_priv_sock::HiveFacts { + hive_facts_from(hive, |name| std::env::var(name).ok()) +} + +/// [`hive_facts`] over a given environment lookup, for tests. +fn hive_facts_from( + hive: &HiveEnv, + var: impl Fn(&str) -> Option, +) -> hive_priv_sock::HiveFacts { + let read = |name: &str| var(name).filter(|v| !v.is_empty()); hive_priv_sock::HiveFacts { name: read("HYPERHIVE_HIVE_NAME"), domain: read("HYPERHIVE_HIVE_DOMAIN"), + dashboard_port: Some(hive.dashboard_port), + claude_code_path: hive.claude_code_path.clone(), + otel_extra_resource_attributes: read("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES"), } } diff --git a/hive-c0re/src/lifecycle/tests.rs b/hive-c0re/src/lifecycle/tests.rs index 0ea0d190..b68ef27e 100644 --- a/hive-c0re/src/lifecycle/tests.rs +++ b/hive-c0re/src/lifecycle/tests.rs @@ -527,3 +527,39 @@ async fn run_with_deadline_returns_output_of_a_fast_child() { .expect("echo finishes"); assert_eq!(String::from_utf8_lossy(&out.stdout).trim(), "hi"); } + +/// The three per-host values reach the container as facts: the dashboard +/// port and pinned claude from the hive's config, the OTEL labels from the +/// daemon's environment. +#[test] +fn hive_facts_carry_the_per_host_values() { + let hive = HiveEnv { + dashboard_port: 7123, + claude_code_path: Some("/nix/store/0abc-claude-code".to_owned()), + ..HiveEnv::default() + }; + let facts = hive_facts_from(&hive, |name| { + (name == "HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES").then(|| "team=a".to_owned()) + }); + assert_eq!(facts.dashboard_port, Some(7123)); + assert_eq!( + facts.claude_code_path.as_deref(), + Some("/nix/store/0abc-claude-code") + ); + assert_eq!( + facts.otel_extra_resource_attributes.as_deref(), + Some("team=a") + ); +} + +/// Unset and empty both leave a fact out, so the agent falls back to its +/// own default rather than reading an empty value. +#[test] +fn hive_facts_leave_unset_values_out() { + let facts = hive_facts_from(&HiveEnv::default(), |name| { + (name == "HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES").then(String::new) + }); + assert_eq!(facts.claude_code_path, None); + assert_eq!(facts.otel_extra_resource_attributes, None); + assert_eq!(facts.name, None); +} diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index 715886b6..430c65ae 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -431,23 +431,12 @@ fn pinned_git_url(forge_git_url: &str, rev: Option<&str>) -> Result { Ok(format!("git+{forge_git_url}?rev={rev}")) } -/// [`render_flake`] with its inputs taken from `hive`, as [`sync_agents`] -/// renders it. +/// [`render_flake`] with the hyperhive pin taken from `hive`, as +/// [`sync_agents`] renders it. Nothing else in `hive` reaches the flake: the +/// values that differ per host go to the container at start +/// (`lifecycle::hive_facts`). fn render_hive_flake(hive: &HiveEnv, agents: &[AgentSpec]) -> Result { - render_hive_flake_at(hive, &hyperhive_input_url(hive)?, agents) -} - -fn render_hive_flake_at( - hive: &HiveEnv, - hyperhive_url: &str, - agents: &[AgentSpec], -) -> Result { - render_flake( - hyperhive_url, - hive.claude_code_path.as_deref(), - hive.dashboard_port, - agents, - ) + render_flake(&hyperhive_input_url(hive)?, agents) } /// Canonical inputs meta knows how to dedup. An agent that declares one @@ -629,19 +618,6 @@ fn forwarded_env_vars() -> Vec<(&'static str, String)> { .collect() } -/// Extra `OTEL_RESOURCE_ATTRIBUTES` entries for every agent, read off -/// hive-c0re's own unit env -/// (`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`). -/// `None` when unset or empty. This is the only OTEL value the renderer -/// writes into the flake: agents export to the hive collector by default and -/// find it by name, so there is no enable flag, endpoint or protocol to -/// carry. -fn otel_extra_resource_attributes() -> Option { - std::env::var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES") - .ok() - .filter(|v| !v.is_empty()) -} - /// Remove the CA files earlier renders embedded next to `flake.nix` /// (`hive-ca.pem`, and the per-peer `peer-ca-.pem` before it), returning /// every filename removed so the caller stages the deletion. Nothing writes @@ -764,12 +740,7 @@ fn canonical_inputs_in_lock(raw: &str) -> Vec<&'static str> { reason = "templated string-builder for the meta flake — the length is one \ contiguous fmt block, splitting it would just hide the shape" )] -fn render_flake( - hyperhive_url: &str, - claude_code_path: Option<&str>, - dashboard_port: u16, - agents: &[AgentSpec], -) -> Result { +fn render_flake(hyperhive_url: &str, agents: &[AgentSpec]) -> Result { use std::fmt::Write as _; let mut out = String::new(); out.push_str("{\n description = \"hyperhive deployed agents\";\n inputs = {\n"); @@ -816,10 +787,8 @@ fn render_flake( ); } } - out.push_str(" };\n outputs =\n { self, hyperhive, ... }@inputs:\n let\n"); - let _ = writeln!( - out, - " dashboardPort = {dashboard_port};\n mkAgent = {{ name, port, archetype ? null }}:" + out.push_str( + " };\n outputs =\n { self, hyperhive, ... }@inputs:\n let\n mkAgent = { name, port, archetype ? null }:\n", ); out.push_str( r#" let @@ -842,32 +811,6 @@ fn render_flake( { "#, ); - // The `claude-code` agents run, as a bare store path rather than a - // flake input: containers share the host's `/nix/store`, so the - // binary is already there with its closure and has nothing to - // travel. A string literal is also the only shape that evaluates — - // `lib.types.package` on a bare path runs `builtins.storePath`, - // which pure eval rejects. The agent module puts its `bin/` on the - // harness PATH; the host module holds the gc root, since a path - // spelled out here is text and references nothing. - // `None` = no override; agents keep their own nixpkgs' `claude-code`. - if let Some(path) = claude_code_path { - let _ = writeln!( - out, - " services.hyperhive.agent.claudeCodePath = \"{path}\";" - ); - } - // Extra OTEL resource labels (`deploy.hive-controller.otel.*`) are the one - // OTEL value rendered here; enablement and the collector endpoint are - // agent-module defaults. - if let Some(attrs) = otel_extra_resource_attributes() { - let esc = |s: &str| s.replace('\\', "\\\\").replace('"', "\\\""); - let _ = writeln!( - out, - " services.hyperhive.agent.otel.extraResourceAttributes = \"{}\";", - esc(&attrs) - ); - } // Agent-facing service URLs (`services.hyperhive.agent.forge.url`, `services.hyperhive.agent.matrix.url`): // emit the host's real values as build-time agent config, the same // host-state -> agent-module shape as the CA block above. @@ -959,7 +902,6 @@ fn render_flake( systemd.services.${service}.environment = { HIVE_PORT = toString port; HIVE_LABEL = name; - HIVE_DASHBOARD_PORT = toString dashboardPort; ", ); // Forwarded env vars — picked up from hive-c0re's own systemd unit @@ -1282,13 +1224,8 @@ mod tests { } } - fn render( - hyperhive_url: &str, - claude_code_path: Option<&str>, - dashboard_port: u16, - agents: &[AgentSpec], - ) -> String { - render_flake(hyperhive_url, claude_code_path, dashboard_port, agents).expect("render") + fn render(hyperhive_url: &str, agents: &[AgentSpec]) -> String { + render_flake(hyperhive_url, agents).expect("render") } #[test] @@ -1309,8 +1246,6 @@ mod tests { fn the_input_a_rebuild_relocks_is_a_pinned_forge_url() { let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("iris", 9001)], ); let url = out @@ -1344,8 +1279,6 @@ mod tests { fn render_flake_pins_hyperhive_by_url_and_follows_its_nixpkgs() { let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ); assert!( @@ -1374,8 +1307,6 @@ mod tests { fn render_flake_names_each_agent_container_after_the_agent() { let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ); assert!( @@ -1401,49 +1332,6 @@ mod tests { ); } - #[test] - fn render_flake_pins_claude_path_without_adding_an_input() { - // The host-pinned claude travels as a bare store path assigned to - // an option — deliberately NOT as a flake input. Containers share - // the host store, so the build is already reachable; making it an - // input would re-copy it as a reference-less `-source` and strip - // the closure the binary actually needs. - let out = render( - "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - Some("/nix/store/cccc-claude-code-2.1.220"), - 8000, - &[sample_spec("alice", 9001)], - ); - assert!( - out.contains( - "services.hyperhive.agent.claudeCodePath = \"/nix/store/cccc-claude-code-2.1.220\";" - ), - "claude path assigned as a plain string literal:\n{out}" - ); - assert!( - !out.contains("claude-code-2.1.220\".url"), - "the pinned claude must not become a flake input:\n{out}" - ); - } - - #[test] - fn render_flake_omits_claude_path_when_unset() { - // `None` = no host-level pin: the option is left undefined so the - // agent module keeps its own nixpkgs' `claude-code` (and keeps it - // in `environment.systemPackages`, which is what makes the - // unpinned case self-contained). - let out = render( - "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, - &[sample_spec("alice", 9001)], - ); - assert!( - !out.contains("claudeCodePath"), - "no claude assignment when unpinned:\n{out}" - ); - } - #[test] fn render_flake_emits_follows_for_agents_declaring_nixpkgs() { // `bitburner` + `dmatrix` declare `nixpkgs` at their root, while @@ -1454,8 +1342,6 @@ mod tests { }; let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[ sample_spec("argus", 9001), with_nixpkgs("bitburner", 9002), @@ -1482,8 +1368,6 @@ mod tests { fn render_flake_builds_every_agent_from_agent_base() { let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001), sample_spec("ruth", 9002)], ); assert_eq!( @@ -1507,8 +1391,6 @@ mod tests { fn render_flake_skips_canonical_follows_when_the_lock_declares_none() { let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ); // No agent-side follows when the lock declares nothing @@ -1538,8 +1420,6 @@ mod tests { }; let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[alice, android], ); assert!( @@ -1588,8 +1468,6 @@ mod tests { } let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ); unsafe { @@ -1629,8 +1507,6 @@ mod tests { } render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ) }; @@ -1824,8 +1700,6 @@ mod tests { } let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ); unsafe { @@ -1855,8 +1729,6 @@ mod tests { }; let out = render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[at("alice", 9001, "aaaa"), at("bob", 9002, "bbbb")], ); assert!( @@ -1882,8 +1754,6 @@ mod tests { let args = |agents: &[AgentSpec]| { render_flake( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, agents, ) }; @@ -2038,8 +1908,6 @@ mod tests { let render = || { render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ) }; @@ -2100,11 +1968,12 @@ mod tests { } #[test] - fn render_flake_carries_no_otel_enablement() { + fn render_flake_carries_no_otel_config_and_no_per_host_value() { // The agent module defaults otel on and finds the collector by name, so // the flake carries no enable, endpoint or protocol line however the - // host's OTEL env is set. Only the per-hive resource labels are - // rendered, and only when set. + // host's OTEL env is set. The resource labels, dashboard port and + // pinned claude differ per host and reach the agent at container start + // (`lifecycle::hive_facts`), so none of them is rendered either. // // Serialised against every other env-mutating test in the crate. // `stats::otel_metrics::tests` perturbs OTEL env too, and lands in this @@ -2115,8 +1984,6 @@ mod tests { let render = || { render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ) }; @@ -2139,10 +2006,11 @@ mod tests { std::env::remove_var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES"); } - // Presence control: the render is the real flake, carrying the pin. + // Presence controls: the render is the real flake, carrying the pin and + // the harness env block the dashboard port used to sit in. assert!( - plain.contains("rev=0123abcd"), - "the hyperhive pin line must be rendered:\n{plain}" + plain.contains("rev=0123abcd") && plain.contains("HIVE_PORT = toString port;"), + "the hyperhive pin and the harness env must be rendered:\n{plain}" ); for (name, flake) in [("plain", &plain), ("labelled", &labelled)] { for needle in [ @@ -2150,6 +2018,11 @@ mod tests { "agent.otel.endpoint", "agent.otel.protocol", "c.example", + "extraResourceAttributes", + "deployment.environment", + "claudeCodePath", + "dashboardPort", + "HIVE_DASHBOARD_PORT", ] { assert!( !flake.contains(needle), @@ -2157,16 +2030,6 @@ mod tests { ); } } - assert!( - !plain.contains("agent.otel.extraResourceAttributes"), - "extraResourceAttributes must not appear when unset:\n{plain}" - ); - assert!( - labelled.contains( - "services.hyperhive.agent.otel.extraResourceAttributes = \"deployment.environment=prod\";" - ), - "extraResourceAttributes must be rendered when set:\n{labelled}" - ); } #[test] fn render_flake_injects_github_disable_only_when_signalled() { @@ -2181,8 +2044,6 @@ mod tests { let render = || { render( "git+https://forge.example/hyperhive/hyperhive.git?rev=0123abcd", - None, - 8000, &[sample_spec("alice", 9001)], ) }; @@ -2206,33 +2067,6 @@ mod tests { "github.enable = false must be injected when the host disables it:\n{disabled}" ); } - - /// The agent reads its memory cap at boot from the copy - /// `lifecycle::write_dropins` hands it, so the cap is no build input. - #[test] - fn render_hive_flake_is_identical_across_memory_caps() { - let hive = |cap: &str| HiveEnv { - agent_memory_max: cap.to_owned(), - ..HiveEnv::default() - }; - let agents = [sample_spec("alice", 9001)]; - let small = render_hive_flake_at( - &hive("4G"), - "git+https://forge.example/h.git?rev=abc", - &agents, - ) - .expect("render"); - let large = render_hive_flake_at( - &hive("16G"), - "git+https://forge.example/h.git?rev=abc", - &agents, - ) - .expect("render"); - assert_eq!( - small, large, - "the rendered flake must not depend on the memory cap" - ); - } } #[cfg(test)] diff --git a/hive-c0re/src/test_env.rs b/hive-c0re/src/test_env.rs index f280eae9..1be54a5a 100644 --- a/hive-c0re/src/test_env.rs +++ b/hive-c0re/src/test_env.rs @@ -11,8 +11,8 @@ //! ⚠️ **The lock has to be crate-wide, not per-module.** `hive-bash-mcp`'s //! sibling helper records why: two separate per-module mutexes serialise //! nothing against each other, and that produced a CI-only flake. The pair that -//! motivated this one is `meta::tests` (which renders a flake from -//! `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`) and `stats::otel_metrics::tests` (which asserts which +//! motivated this one is `meta::tests` (which renders a flake with +//! `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES` set) and `stats::otel_metrics::tests` (which asserts which //! variable enables the exporter) — different modules, same process env, same //! binary. //! diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index 86e3679f..331c50d2 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -187,6 +187,15 @@ pub struct HiveFacts { pub name: Option, /// `HYPERHIVE_HIVE_DOMAIN`: the hive's DNS domain. pub domain: Option, + /// `HIVE_DASHBOARD_PORT`: the port this hive's dashboard listens on. + pub dashboard_port: Option, + /// `HYPERHIVE_CLAUDE_CODE_PATH`: store path of the `claude-code` the + /// hive pins for its agents. The host's system closure keeps it alive; + /// the container only gets the path. + pub claude_code_path: Option, + /// `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`: comma-separated `k=v` + /// entries the agent appends to its `OTEL_RESOURCE_ATTRIBUTES`. + pub otel_extra_resource_attributes: Option, } /// Wall-clock bound hive-priv puts on one [`PrivRequest::MeasureAgentDisk`] diff --git a/hive-priv/src/container_config.rs b/hive-priv/src/container_config.rs index 64352042..0fc3e297 100644 --- a/hive-priv/src/container_config.rs +++ b/hive-priv/src/container_config.rs @@ -367,24 +367,54 @@ fn container_rootfs(container: &str) -> PathBuf { /// Render [`HiveFacts`] as the systemd environment file the agent loads, /// one `KEY=value` line per present fact. /// -/// A value is refused unless it is DNS-name characters only, so it can carry -/// neither a newline (a second assignment) nor anything an environment file -/// or a shell would interpret. +/// Each value is refused, never escaped, unless it is made only of the +/// characters its fact can hold. None of those sets admits a newline (a +/// second assignment), whitespace, a quote, `\` or `$`: the agent hands each +/// line to systemd and its shells, which would read those as syntax. fn render_hive_facts(facts: &HiveFacts) -> Result { use std::fmt::Write as _; + let dns = |b: u8| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_'); + let port = facts.dashboard_port.map(|p| p.to_string()); let mut out = String::new(); - for (key, value) in [ - ("HYPERHIVE_HIVE_NAME", &facts.name), - ("HYPERHIVE_HIVE_DOMAIN", &facts.domain), + for (key, value, max_len, allowed) in [ + ( + "HYPERHIVE_HIVE_NAME", + facts.name.as_ref(), + 253, + &dns as &dyn Fn(u8) -> bool, + ), + ("HYPERHIVE_HIVE_DOMAIN", facts.domain.as_ref(), 253, &dns), + ("HIVE_DASHBOARD_PORT", port.as_ref(), 5, &|b: u8| { + b.is_ascii_digit() + }), + // A store path's name characters, plus the `/` of its prefix; the + // prefix itself is checked below. + ( + "HYPERHIVE_CLAUDE_CODE_PATH", + facts.claude_code_path.as_ref(), + 256, + &|b: u8| dns(b) || matches!(b, b'+' | b'?' | b'=' | b'/'), + ), + // `k=v,k=v`, values percent-encoded as the OTEL spec has them. + ( + "HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES", + facts.otel_extra_resource_attributes.as_ref(), + 1024, + &|b: u8| dns(b) || matches!(b, b'=' | b',' | b'/' | b':' | b'%' | b'+' | b'@'), + ), ] { let Some(value) = value else { continue }; - if value.is_empty() - || value.len() > 253 - || !value - .bytes() - .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_')) + if value.is_empty() || value.len() > max_len || !value.bytes().all(allowed) { + bail!( + "invalid {key} value {value:?}: must be 1-{max_len} chars of its fact's character set" + ); + } + if key == "HYPERHIVE_CLAUDE_CODE_PATH" + && !value + .strip_prefix("/nix/store/") + .is_some_and(|name| !name.is_empty() && !name.contains('/')) { - bail!("invalid {key} value {value:?}: must be 1-253 chars of [A-Za-z0-9._-]"); + bail!("invalid {key} value {value:?}: must be a top-level /nix/store path"); } let _ = writeln!(out, "{key}={value}"); } @@ -711,14 +741,15 @@ mod tests { let both = HiveFacts { name: Some("pr1ma".to_owned()), domain: Some("pr1ma.darkest.space".to_owned()), + ..HiveFacts::default() }; assert_eq!( render_hive_facts(&both).unwrap(), "HYPERHIVE_HIVE_NAME=pr1ma\nHYPERHIVE_HIVE_DOMAIN=pr1ma.darkest.space\n" ); let domain_only = HiveFacts { - name: None, domain: Some("h1.example.test".to_owned()), + ..HiveFacts::default() }; assert_eq!( render_hive_facts(&domain_only).unwrap(), @@ -734,7 +765,66 @@ mod tests { for bad in ["", "pr1ma\nLD_PRELOAD=/x", "a b", "a\"b", "$(x)"] { let facts = HiveFacts { name: Some(bad.to_owned()), - domain: None, + ..HiveFacts::default() + }; + assert!( + render_hive_facts(&facts).is_err(), + "{bad:?} must be refused" + ); + } + } + + /// The three per-host values agents used to get at build time arrive + /// here, one line each. + #[test] + fn hive_facts_render_the_per_host_values() { + let facts = HiveFacts { + dashboard_port: Some(7123), + claude_code_path: Some("/nix/store/0abc-claude-code-2.1.220".to_owned()), + otel_extra_resource_attributes: Some( + "deployment.environment=prod,team=a%2Fb".to_owned(), + ), + ..HiveFacts::default() + }; + assert_eq!( + render_hive_facts(&facts).unwrap(), + "HIVE_DASHBOARD_PORT=7123\n\ + HYPERHIVE_CLAUDE_CODE_PATH=/nix/store/0abc-claude-code-2.1.220\n\ + HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=deployment.environment=prod,team=a%2Fb\n" + ); + } + + /// The claude path is a top-level store path or nothing, and neither + /// value may carry generator syntax. + #[test] + fn hive_facts_refuse_bad_claude_paths_and_resource_attributes() { + for bad in [ + "/nix/store/", + "/nix/store/abc-claude/bin/claude", + "/tmp/claude", + "nix/store/abc-claude", + "/nix/store/abc-claude\nLD_PRELOAD=/x", + ] { + let facts = HiveFacts { + claude_code_path: Some(bad.to_owned()), + ..HiveFacts::default() + }; + assert!( + render_hive_facts(&facts).is_err(), + "{bad:?} must be refused" + ); + } + for bad in [ + "a=b\nPATH=/x", + "a=${HOME}", + "a=b c", + "a=\"b\"", + "a=b\\c", + "", + ] { + let facts = HiveFacts { + otel_extra_resource_attributes: Some(bad.to_owned()), + ..HiveFacts::default() }; assert!( render_hive_facts(&facts).is_err(), @@ -752,10 +842,7 @@ mod tests { bridge: "hive0".to_owned(), gateway_ip: "10.42.0.1".to_owned(), }; - let facts = HiveFacts { - name: None, - domain: None, - }; + let facts = HiveFacts::default(); for name in ["hive-forge", "hive-matrix"] { let errs = [ handle_write_nspawn_flags(name, &[], &isolation, &[], &facts).unwrap_err(), diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index 6047ad5c..6f4f5705 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -28,6 +28,33 @@ let which the hive writes before every start, so no agent build sets it. Remove the setting. ''; + claudeCodePathRemoved = '' + An agent reads the claude-code its hive pins at runtime, from + HYPERHIVE_CLAUDE_CODE_PATH in /etc/hyperhive-hive-facts, so no agent build + sets it. Pin it host-side with + services.hyperhive.deploy.hive-controller.claudeCodePackage. + ''; + + # `claude` on PATH. A hive that pins a claude-code + # (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its + # store path over at container start as `HYPERHIVE_CLAUDE_CODE_PATH`, and + # the host's system closure keeps it alive: nothing in this container + # references it. With no pin, or a path that is not there, this container's + # own claude-code runs; a pin that is set but gone (collected after a bump + # while this container still holds the old facts) says so on stderr, since + # the two builds can differ. `exec -a claude` keeps the bare name in argv[0], + # which is how the harness finds its claude child (`find_claude_child`). + ownClaude = import ../claude-code-no-alsa.nix lib pkgs.claude-code; + claude = pkgs.writeShellScriptBin "claude" '' + pinned=''${HYPERHIVE_CLAUDE_CODE_PATH:-} + if [ -n "$pinned" ] && [ -x "$pinned/bin/claude" ]; then + exec -a claude "$pinned/bin/claude" "$@" + fi + if [ -n "$pinned" ]; then + echo "claude: $pinned/bin/claude, the hive's pinned claude-code, is missing; running this container's own" >&2 + fi + exec -a claude ${ownClaude}/bin/claude "$@" + ''; in { imports = [ @@ -68,6 +95,12 @@ in "agent" "claudeMemoryMaxBytes" ] memoryMaxBytesRemoved) + (lib.mkRemovedOptionModule [ + "services" + "hyperhive" + "agent" + "claudeCodePath" + ] claudeCodePathRemoved) ]; options.services.hyperhive.agent.icon = lib.mkOption { @@ -87,33 +120,6 @@ in ''; }; - options.services.hyperhive.agent.claudeCodePath = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/nix/store/…-claude-code-2.1.220"; - description = '' - Store path of the `claude-code` this agent runs, or `null` (the - default) to use the `claude-code` from the container's own - nixpkgs. - - Set by the generated meta flake when the operator sets - `services.hyperhive.deploy.hive-controller.claudeCodePackage` host-side, so a hive - can run a claude built from a *different* nixpkgs than the one - its agents evaluate against — a release channel can trail - unstable by weeks on this one package. It arrives as a path and - not a package because agents share the host's `/nix/store`, so - the build is already reachable here with its whole closure and - has nothing to travel. - - When set, `claude` on PATH is a symlink to this path's - `bin/claude` and the container's own `claude-code` is dropped, so - there is only ever one claude in the container. Note that neither - the symlink nor anything else in the container's closure *refers* - to the target — keeping it alive is the host's job, see - `services.hyperhive.deploy.hive-controller.claudeCodePackage`. - ''; - }; - config = { assertions = [ # Guard the inputs-routed-as-output pattern: the agent flake.nix is @@ -240,28 +246,7 @@ in config.services.hyperhive.agent.packages.hive-metric ] ++ [ - ( - if config.services.hyperhive.agent.claudeCodePath == null then - import ../claude-code-no-alsa.nix lib pkgs.claude-code - else - # Host-pinned claude: a symlink farm around a path the - # container was handed as text. It has to be a derivation — - # `environment.systemPackages` coerces a store-path *string* - # with `toDerivation`, i.e. `builtins.storePath`, which pure - # evaluation rejects (`systemd.services.*.path` does the same, - # which is why the harness gets this via PATH like everything - # else rather than a unit-level entry). Interpolating the path - # into the builder is just text, so it evaluates anywhere. - # - # The symlink registers no store reference — the target isn't - # among this derivation's inputs, so nothing here keeps the - # binary alive. That is deliberate and it is the host's job: - # see `services.hyperhive.deploy.hive-controller.claudeCodePackage`. - pkgs.runCommandLocal "claude-code-pinned" { } '' - mkdir -p "$out/bin" - ln -s ${config.services.hyperhive.agent.claudeCodePath}/bin/claude "$out/bin/claude" - '' - ) + claude ] ++ (with pkgs; [ bashInteractive diff --git a/nix/agent-modules/hive-facts.nix b/nix/agent-modules/hive-facts.nix index c5e33e06..864965bb 100644 --- a/nix/agent-modules/hive-facts.nix +++ b/nix/agent-modules/hive-facts.nix @@ -2,8 +2,17 @@ # agent's closure does not carry it. # # hive-priv writes `/etc/hyperhive-hive-facts` into the container before every -# start (`HYPERHIVE_HIVE_NAME`, `HYPERHIVE_HIVE_DOMAIN`, one `KEY=value` line -# each). Two readers hand those lines on: +# start, one `KEY=value` line each: +# +# - `HYPERHIVE_HIVE_NAME`, `HYPERHIVE_HIVE_DOMAIN`; +# - `HIVE_DASHBOARD_PORT`, read by the harness's web UI (7000 when absent); +# - `HYPERHIVE_CLAUDE_CODE_PATH`, the claude-code the hive pins (`claude` in +# ./default.nix runs this container's own when absent); +# - `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`, appended to +# `OTEL_RESOURCE_ATTRIBUTES` by ./otel.nix (no extra labels when absent). +# +# Any of them may be absent: a hive-c0re older than the agent's build writes +# fewer. Two readers hand the lines on: # # - an environment generator, for the container's systemd: it passes them to # every unit it starts, `machinectl shell` sessions (`hivectl choom`) @@ -23,8 +32,8 @@ let userName = config.services.hyperhive.agent.user.name; - # Shell builtins only: nothing is assumed on PATH this early. Only the two - # known keys are passed on. `|| [ -n "$key" ]` keeps a last line that has + # Shell builtins only: nothing is assumed on PATH this early. Only the known + # keys are passed on. `|| [ -n "$key" ]` keeps a last line that has # no trailing newline. generator = pkgs.writeShellScript "hyperhive-hive-facts" '' hive=unknown @@ -32,7 +41,8 @@ let while IFS='=' read -r key value || [ -n "$key" ]; do case "$key" in HYPERHIVE_HIVE_NAME) hive=$value ;; - HYPERHIVE_HIVE_DOMAIN) ;; + HYPERHIVE_HIVE_DOMAIN | HIVE_DASHBOARD_PORT) ;; + HYPERHIVE_CLAUDE_CODE_PATH | HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES) ;; *) continue ;; esac printf '%s=%s\n' "$key" "$value" diff --git a/nix/agent-modules/otel.nix b/nix/agent-modules/otel.nix index d0f9046b..9ddbdbf8 100644 --- a/nix/agent-modules/otel.nix +++ b/nix/agent-modules/otel.nix @@ -42,9 +42,17 @@ let # agent, and their metric names already tell them apart. No `hive`: the # swarm collector stamps it from the receiver that accepted the hive's # push (nix/host-modules/swarm-otel.nix), so the build names no hive. - resourceAttributes = - "service.name=hyperhive-agent,agent=${userName},swarm=${swarmDisplayName}" - + lib.optionalString (cfg.extraResourceAttributes != "") ",${cfg.extraResourceAttributes}"; + resourceAttributes = "service.name=hyperhive-agent,agent=${userName},swarm=${swarmDisplayName}"; + + # Prints `OTEL_RESOURCE_ATTRIBUTES`: the labels above, then the hive's own + # (`HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`), which differ per host and so + # arrive at container start. That variable comes from ./hive-facts.nix's + # generator, which runs before this one and is absent when the hive sets no + # labels. + resourceGenerator = pkgs.writeShellScript "hyperhive-otel-resource" '' + printf 'OTEL_RESOURCE_ATTRIBUTES=%s%s\n' ${lib.escapeShellArg resourceAttributes} \ + "''${HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES:+,$HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES}" + ''; # The variables an OTEL SDK reads on its own, in any language, in any # process here. @@ -62,7 +70,6 @@ let # which Prometheus/Mimir-family backends (incl. grafana-lgtm) # silently drop without a deltatocumulative processor. OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE = "cumulative"; - OTEL_RESOURCE_ATTRIBUTES = resourceAttributes; }; # The exporter's NAME picks the wire protocol, so `protocol` has to be @@ -118,19 +125,6 @@ in ''; }; - extraResourceAttributes = lib.mkOption { - type = lib.types.str; - default = ""; - internal = true; - description = '' - Extra comma-separated entries appended to - `OTEL_RESOURCE_ATTRIBUTES` after the built-in - `service.name` / `agent` / `swarm` labels. - Host-driven via - `services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`. - ''; - }; - debug = lib.mkOption { type = lib.types.bool; default = false; @@ -167,6 +161,14 @@ in systemd.globalEnvironment = producerEnv; environment.variables = producerEnv; + # `OTEL_RESOURCE_ATTRIBUTES` is in neither: a unit's own `Environment=` + # would override what the generator computes at boot. Units get it from + # the manager's environment; login shells run the same generator after + # ./hive-facts.nix's snippet has exported the hive's labels. + environment.etc."systemd/system-environment-generators/60-hyperhive-otel-resource".source = + resourceGenerator; + environment.extraInit = lib.mkAfter ''export "$(${resourceGenerator})"''; + # The journal this container writes is reachable from inside it and # from nowhere else, which is why the forwarder runs here rather than # the hive reading the container's journal from outside: the host-side diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index 9b38328f..0f2d39a0 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -110,9 +110,9 @@ let # in whole rather than interpolated — that is also the ONLY thing # gc-rooting it: the resulting string carries store context, so this # /etc entry genuinely references the package and the host's system - # closure holds it alive. Nothing container-side can — meta.rs writes - # the path into each agent's generated flake as a plain string - # literal, and text is not a reference. Hence the assertion below: do + # closure holds it alive. Nothing container-side can: hive-c0re hands + # the path to each container as text (`HYPERHIVE_CLAUDE_CODE_PATH`), + # and text is not a reference. Hence the assertion below: do # NOT discard this context, and do not route the path through # anything that drops it. The failure mode is a garbage-collected # `claude` and a hive that can't take a turn, weeks after the commit diff --git a/nix/host-modules/hive-c0re/options.nix b/nix/host-modules/hive-c0re/options.nix index ebabd39e..17267da8 100644 --- a/nix/host-modules/hive-c0re/options.nix +++ b/nix/host-modules/hive-c0re/options.nix @@ -326,8 +326,9 @@ }; }; - # What this hive hands its own agents through the meta-flake render. Per - # host, so two hives may set them differently. + # What this hive hands its own agents at container start, through + # `/etc/hyperhive-hive-facts`. Per host, so two hives may set them + # differently. options.services.hyperhive.deploy.hive-controller = { claudeCodePackage = lib.mkOption { type = lib.types.nullOr lib.types.package; @@ -350,22 +351,23 @@ What travels into the container is the **store path**, not the derivation: agents share the host's `/nix/store`, so the binary and its full closure are already reachable there — nothing - needs rebuilding or copying. hive-c0re writes the path into - each agent's generated flake as a plain string literal (a bare - path fed to `lib.types.package` would run `builtins.storePath`, - which is illegal under pure evaluation) and the agent module - puts its `bin/` on the harness's PATH. + needs rebuilding or copying. hive-c0re hands the path to each + container at start as `HYPERHIVE_CLAUDE_CODE_PATH`, and the agent's + `claude` on PATH runs it, or the container's own `claude-code` when + the variable is absent or names nothing. A change reaches an agent + at its next container start, not through a rebuild. The flip side of a plain string is that nothing in the agent's own closure refers to it, so the container cannot keep it alive. The **host** does that instead: this package is interpolated into `/etc/hyperhive/serve.json`, which puts it in - the host's system closure — so it is gc-rooted by the running - generation for exactly as long as that generation is the one - the agents were rendered from. The cost is that - `nix-collect-garbage` cannot reclaim an old `claude-code` until - every agent has been rebuilt past it and the old generations - are gone. + the host's system closure, so it is gc-rooted by the running + generation. ⚠️ An agent started under an older generation keeps + that generation's path until its next start, and only the old + generation roots it: collect old generations after restarting + the agents, or an agent still running the old path falls back to + its own `claude-code` at its next `claude` spawn, saying so on + stderr. The swarm's `swarm-claude-login` and `swarm-review-classifier` containers, where this host runs them, follow the same pin and @@ -377,7 +379,11 @@ dashboardPort = lib.mkOption { type = lib.types.port; default = 7000; - description = "TCP port this hive's hive-c0re dashboard listens on."; + description = '' + TCP port this hive's hive-c0re dashboard listens on. Agents get it + at container start as `HIVE_DASHBOARD_PORT`, for their web UI's + links back to the dashboard. + ''; }; }; } diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index b162f8b8..e2379eb9 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -236,7 +236,7 @@ in }; # Read by this hive's hive-c0re, which stamps it on its own export and - # renders it into its agents', so it is a per-host choice. + # hands it to its agents at container start, so it is a per-host choice. options.services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes = lib.mkOption { type = lib.types.str; default = ""; diff --git a/nix/module-eval/agent-otel.nix b/nix/module-eval/agent-otel.nix index a9df766f..d92614a1 100644 --- a/nix/module-eval/agent-otel.nix +++ b/nix/module-eval/agent-otel.nix @@ -123,6 +123,25 @@ let && lib.all (e: s.exporters.${e}.endpoint == endpoint) s.service.pipelines.logs.exporters && !(agentNoOtel.systemd.globalEnvironment ? OTEL_EXPORTER_OTLP_ENDPOINT); } + { + # The hive's extra labels arrive at container start, so the build + # cannot hold the whole `OTEL_RESOURCE_ATTRIBUTES`: a boot-time + # generator computes it, and a unit-level `Environment=` copy would + # override that. The endpoint beside it is the presence control for the + # two absence lookups; `agentNoOtel` is the control for the generator. + name = "an agent's OTEL resource labels come from a boot-time generator, not the build"; + ok = + let + generator = + m: m.environment.etc ? "systemd/system-environment-generators/60-hyperhive-otel-resource"; + in + agentDefault.systemd.globalEnvironment ? OTEL_EXPORTER_OTLP_ENDPOINT + && !(agentDefault.systemd.globalEnvironment ? OTEL_RESOURCE_ATTRIBUTES) + && agentDefault.environment.variables ? OTEL_EXPORTER_OTLP_ENDPOINT + && !(agentDefault.environment.variables ? OTEL_RESOURCE_ATTRIBUTES) + && generator agentDefault + && !(generator agentNoOtel); + } { # Presence control for the two cases above: with the switch off there # is no collector in the container at all, so their passing is about diff --git a/nix/module-eval/swarm-services-switch.nix b/nix/module-eval/swarm-services-switch.nix index 5b319322..40d38dcb 100644 --- a/nix/module-eval/swarm-services-switch.nix +++ b/nix/module-eval/swarm-services-switch.nix @@ -282,7 +282,7 @@ let { # The CLI is the one the agents run, not the container's own nixpkgs'. # Unpinned is the control: it falls back to the container's own `claude-code`. - # serve.json is what hive-c0re renders every agent's `claudeCodePath` from. + # serve.json is what hive-c0re hands every agent's `HYPERHIVE_CLAUDE_CODE_PATH` from. name = "the agents, the claude login minter and the review classifier run the claude-code this host pins"; ok = let diff --git a/nix/script-tests/agent-hive-facts.nix b/nix/script-tests/agent-hive-facts.nix index 82c1a2c1..560c705e 100644 --- a/nix/script-tests/agent-hive-facts.nix +++ b/nix/script-tests/agent-hive-facts.nix @@ -1,7 +1,9 @@ # `checks.script-test-agent-hive-facts` — runs the two readers of # `/etc/hyperhive-hive-facts` in ../agent-modules/hive-facts.nix: the systemd # environment generator and the login-shell snippet `/etc/set-environment` -# sources. The cases are in ./agent-hive-facts.sh. +# sources. Also the two consumers of what they pass on: ../agent-modules/otel.nix's +# `OTEL_RESOURCE_ATTRIBUTES` generator and ../agent-modules/default.nix's `claude`. +# The cases are in ./agent-hive-facts.sh. # # What runs is the rendered generator and snippet. The one edit is the facts # file's path, moved under the build directory, and the snippet's reference to @@ -26,6 +28,11 @@ let ; machine = agentWith { services.hyperhive.agent.user.name = "a1"; }; + + # As text: building it would build the claude-code it falls back to. + claude = lib.findFirst ( + p: (p.name or "") == "claude" + ) (throw "no `claude` in systemPackages") machine.environment.systemPackages; in pkgs.runCommand "hyperhive-script-test-agent-hive-facts" { @@ -33,6 +40,9 @@ pkgs.runCommand "hyperhive-script-test-agent-hive-facts" machine.environment.etc."systemd/system-environment-generators/50-hyperhive-hive-facts".source; PROFILE = machine.environment.etc."hyperhive/hive-facts.sh".source; EXTRA_INIT = machine.environment.extraInit; + OTEL_GENERATOR = + machine.environment.etc."systemd/system-environment-generators/60-hyperhive-otel-resource".source; + CLAUDE_WRAPPER = builtins.unsafeDiscardStringContext claude.text; SHELLS = "${pkgs.dash}/bin/dash ${pkgs.bash}/bin/sh"; } '' diff --git a/nix/script-tests/agent-hive-facts.sh b/nix/script-tests/agent-hive-facts.sh index cddb1bf7..58f57ac7 100644 --- a/nix/script-tests/agent-hive-facts.sh +++ b/nix/script-tests/agent-hive-facts.sh @@ -40,7 +40,7 @@ check() { for sh in $SHELLS; do # Only what the snippet exported reaches the child's environment. No PATH: # the snippet must not need one. - got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(HYPERHIVE_|CLAUDE_|LD_PRELOAD=|__hyperhive)' | sort) + got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(HYPERHIVE_|HIVE_DASHBOARD_PORT=|CLAUDE_|LD_PRELOAD=|__hyperhive)' | sort) if [ "$got" != "$(printf '%s\n' "$want" | sort)" ]; then fail "$sh: a child of the sourcing shell saw:"$'\n'"$got" fi @@ -61,11 +61,73 @@ check "a last line without a newline is kept" "HYPERHIVE_HIVE_DOMAIN=pr1ma.examp HYPERHIVE_HIVE_NAME=pr1ma CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=pr1ma-a1" +printf 'HIVE_DASHBOARD_PORT=7123\nHYPERHIVE_CLAUDE_CODE_PATH=/nix/store/0abc-claude-code\nHYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod\n' >"$facts" +check "the per-host values, a value holding = kept whole" "HIVE_DASHBOARD_PORT=7123 +HYPERHIVE_CLAUDE_CODE_PATH=/nix/store/0abc-claude-code +HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod +CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" + : >"$facts" check "an empty file (no facts)" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" rm -f "$facts" check "no file" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" +# OTEL_RESOURCE_ATTRIBUTES, for units (the generator, run with what the +# hive-facts generator printed) and for login shells (the extraInit line, after +# the snippet). The base labels come from the build; the hive's are appended. +base="service.name=hyperhive-agent,agent=a1,swarm=unknown" +otel_check() { + case=$1 + count=$((count + 1)) + local want="OTEL_RESOURCE_ATTRIBUTES=$2" got sh passed + mapfile -t passed < <(env -i "$dir/generator") + got=$(env -i "${passed[@]}" "$OTEL_GENERATOR") + if [ "$got" != "$want" ]; then fail "generator printed: $got, wanted: $want"; fi + for sh in $SHELLS; do + got=$(env -i "$sh" -c ". '$dir/profile'; $otel_init; exec '$envbin'" | grep '^OTEL_RESOURCE_ATTRIBUTES=') + if [ "$got" != "$want" ]; then fail "$sh: a login shell saw: $got"; fi + done +} + +case="otel setup" +otel_init=$(printf '%s\n' "$EXTRA_INIT" | grep -F "$OTEL_GENERATOR") +if [ -z "$otel_init" ]; then fail "/etc/set-environment does not run the OTEL resource generator"; fi +if [ "$(printf '%s\n' "$EXTRA_INIT" | grep -n . | grep -F "$OTEL_GENERATOR" | cut -d: -f1)" -lt \ + "$(printf '%s\n' "$EXTRA_INIT" | grep -nxF '. /etc/hyperhive/hive-facts.sh' | cut -d: -f1)" ]; then + fail "/etc/set-environment runs the OTEL resource generator before the hive-facts snippet" +fi + +printf 'HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod\n' >"$facts" +otel_check "the hive's labels are appended" "$base,team=a,env=prod" +rm -f "$facts" +otel_check "no hive labels: the base alone, no trailing comma" "$base" + +# `claude`: the hive's pinned build when it is there, this container's own +# otherwise. Both builds are stand-ins that print which one ran. +mkdir -p "$dir/own/bin" "$dir/pinned/bin" +printf '#!%s\necho own "$@"\n' "$BASH" >"$dir/own/bin/claude" +printf '#!%s\necho pinned "$@"\n' "$BASH" >"$dir/pinned/bin/claude" +chmod +x "$dir/own/bin/claude" "$dir/pinned/bin/claude" +printf '%s' "$CLAUDE_WRAPPER" | sed -E "s|/nix/store/[^ ]+/bin/claude|$dir/own/bin/claude|" >"$dir/claude" +case="claude setup" +if ! grep -qF "$dir/own/bin/claude" "$dir/claude"; then fail "the rewrite did not reach the fallback claude"; fi +# claude_check +claude_check() { + case=$1 + count=$((count + 1)) + local got said pin=() + if [ -n "$2" ]; then pin=("HYPERHIVE_CLAUDE_CODE_PATH=$2"); fi + # Under this shell: the shebang names the build's bash, not in this sandbox. + got=$(env -i "${pin[@]}" "$BASH" "$dir/claude" -p hi 2>"$dir/stderr") + said=$(cat "$dir/stderr") + if [ "$got" != "$3 -p hi" ]; then fail "ran: $got, wanted: $3 -p hi"; fi + if [ "$said" != "$4" ]; then fail "said on stderr: '$said', wanted: '$4'"; fi +} +claude_check "the pinned claude when the hive hands one over, silently" "$dir/pinned" pinned "" +claude_check "the container's own claude with no pin, silently" "" own "" +claude_check "the container's own claude when the pinned path is gone, saying so" "$dir/gone" own \ + "claude: $dir/gone/bin/claude, the hive's pinned claude-code, is missing; running this container's own" + echo "$count cases, $failures failed" [ "$failures" -eq 0 ]