From b08176f089ad1521f552e35fd1c0915254e25489 Mon Sep 17 00:00:00 2001 From: atlas Date: Mon, 27 Jul 2026 13:06:22 +0200 Subject: [PATCH] feat(#2693): let the operator pin the claude-code every agent runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Agents run whatever `claude-code` the meta flake's `nixpkgs` resolves to, and that is normally a release channel. This one package moves fast enough that stable trails unstable by weeks — 26.05 is on 2.1.187 while unstable carries 2.1.220 — and an agent cannot fix it for itself: it only ever sees the single nixpkgs hive-c0re injects, so an `agent.nix` has no other tree to reach for. New host option `services.hyperhive.c0re.claudeCodePackage` takes the package directly and rides the existing `hyperhiveDocs` threading path — serveConfigJson -> HiveEnv -> render_flake — to reach each agent as `hyperhive.claudeCodePath`. Null (the default) is today's behaviour. What travels is the store *path*, as a plain string literal, not a flake input: containers share the host's `/nix/store`, so the build is already reachable inside them with its whole closure and has nothing to travel. An input would be worse than useless — a `path:/nix/store/` input is re-copied as a reference-less `-source`, which strips exactly the closure the binary needs. The catch is that a path written into a generated flake is text, so nothing in the container's closure keeps the binary alive. The host does that instead, and gets it for free: the package is interpolated into `/etc/hyperhive/serve.json`, `builtins.toJSON` preserves string context, so the /etc entry references it and the system closure gc-roots it for as long as that generation is the one the agents were rendered from. An assertion pins that property, because losing the context is invisible at eval and at deploy — it would surface only as every agent failing to spawn `claude` whenever the next gc ran. Container side wraps the path in a symlink farm rather than putting it on PATH directly: `systemd.services..path` and `environment.systemPackages` both coerce a store-path *string* through `lib.toDerivation`, i.e. `builtins.storePath`, which pure evaluation rejects. Interpolating the path into a builder is just text and evaluates anywhere. `claude-code` drops out of systemPackages when a pin is set, so there is exactly one claude in the container. Refs #2693 --- docs/gotchas.md | 41 +++++++++++++ docs/turn-loop/claude-invocation.md | 11 ++++ hive-c0re/src/coordinator.rs | 20 +++++++ hive-c0re/src/meta.rs | 81 ++++++++++++++++++++++++++ nix/agent-modules/default.nix | 52 ++++++++++++++++- nix/host-modules/hive-c0re/default.nix | 38 ++++++++++++ nix/host-modules/hive-c0re/options.nix | 39 +++++++++++++ 7 files changed, 281 insertions(+), 1 deletion(-) diff --git a/docs/gotchas.md b/docs/gotchas.md index 119bc38e..6cbc7f9c 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -84,6 +84,47 @@ scoped, only this one package. This is needed because each per-agent operator's host-level `allowUnfree` does **not** propagate in. Operators don't need to set anything on their side. +That same isolation is why an agent can't pick a claude out of a +*different* nixpkgs by itself: a container only ever sees the one +nixpkgs the meta flake injects, so an `agent.nix` naming the host's +`nixpkgs-unstable` has nothing to name. A release channel can trail +unstable by weeks on this package, which is what +`services.hyperhive.c0re.claudeCodePackage` is for — set it host-side +and every agent runs that build. + +What crosses is the **store path**, not the derivation. Containers +share the host's `/nix/store`, so the binary is already reachable +inside them with its whole closure; hive-c0re writes the path into each +agent's flake as a string literal and the agent module symlinks +`bin/claude` onto PATH. Two things rule out the obvious alternatives: a +`path:/nix/store/` flake input is re-copied into the store as a +reference-less `-source` (so the runtime closure never arrives), and +`lib.types.package` fed a bare path runs `builtins.storePath`, which +pure evaluation rejects. `hyperhive.docs.source` gets away with being +an input only because a docs tree has no runtime dependencies. + +The `storePath` trap is worth spelling out, because it is not confined +to options the operator writes: **any** option of type `package` fed a +store-path *string* coerces through `lib.toDerivation`, i.e. +`builtins.storePath`. `environment.systemPackages` and +`systemd.services..path` both do it (the latter takes plain +strings like `/run/wrappers` happily, but anything under +`builtins.storeDir` is treated as a package). So a path handed to the +container as text has to be wrapped in a real derivation — a symlink +farm built from the interpolated string — before it can go anywhere a +package is expected. + +The catch is that a path written into a generated flake is text, not a +reference — the container's closure does not keep the binary alive. +The **host** does: the package is interpolated into +`/etc/hyperhive/serve.json`, so it lands in the host's system closure +and is gc-rooted by the running generation. `builtins.toJSON` preserves +string context, which is the load-bearing detail; discard the context +anywhere on that path and `nix-collect-garbage` will eventually take +the hive's `claude` out from under it. The price of the root is that an +old `claude-code` can't be reclaimed until every agent has rebuilt past +it and the old generations are gone. + ## Claude credentials are per-agent `/var/lib/hyperhive/agents//claude/` bind-mounts to diff --git a/docs/turn-loop/claude-invocation.md b/docs/turn-loop/claude-invocation.md index 9d93619e..b9d2f26c 100644 --- a/docs/turn-loop/claude-invocation.md +++ b/docs/turn-loop/claude-invocation.md @@ -21,6 +21,17 @@ auto-reset / retry decisions in `drive_turn`. The lib returns everything it parsed from a turn (usage, cost, context window, resolved model) as `Telemetry`, which the policy layer applies to the bus. +**Which `claude` binary.** The bare name `claude`, resolved off the +harness unit's PATH. By default that's the `claude-code` in the agent's +own nixpkgs (the meta flake's `nixpkgs` input) via +`environment.systemPackages`. Since that's usually a release channel and +this package moves fast, the operator can pin one hive-wide with +`services.hyperhive.c0re.claudeCodePackage`: its store path is written +into each agent's flake, and `claude` on PATH becomes a symlink to it +instead of the container's own `claude-code` — so there's only ever one +`claude` in the container. Agents pick up a new build on their +next rebuild, not live. See docs/gotchas.md::`claude-code` is unfree. + Hive-enforced settings ship at `/etc/claude-code/managed-settings.json` (claude-code's canonical managed-settings path — precedence #1, read-only, un-overridable), wired in `nix/agent-modules/claude-settings.nix` diff --git a/hive-c0re/src/coordinator.rs b/hive-c0re/src/coordinator.rs index c08927df..ee561c89 100644 --- a/hive-c0re/src/coordinator.rs +++ b/hive-c0re/src/coordinator.rs @@ -62,6 +62,17 @@ pub struct Coordinator { /// "nixpkgs"` is set in the host flake). Empty string = legacy /// `follows = "hyperhive/nixpkgs"` behaviour. pub nixpkgs_flake: String, + /// Store path of the `claude-code` build every agent runs, written + /// into each per-agent flake as `hyperhive.claudeCodePath`. Set by + /// the NixOS module option `services.hyperhive.c0re.claudeCodePackage` + /// (which resolves the package and hands us its path). Empty string = + /// every agent keeps the `claude-code` from its own nixpkgs. + /// + /// A path rather than a flake input because containers share the + /// host's `/nix/store`: the binary is already reachable inside them, + /// closure and all. The host module is what keeps it from being + /// garbage-collected — see that option. + pub claude_code_path: String, /// TCP port the host's hive-c0re dashboard listens on. Inlined into /// each per-agent flake so the agent's web UI can build the right /// rebuild-button URL pointing back at the dashboard. @@ -213,6 +224,11 @@ pub struct HiveEnv { /// so a doc edit only re-locks this input, not the whole source. pub hyperhive_docs_flake: String, pub nixpkgs_flake: String, + /// Store path of the `claude-code` agents run, or empty for "each + /// agent keeps the one out of its own nixpkgs". Travels into the + /// container as `hyperhive.claudeCodePath` — a plain string, kept + /// alive host-side by the module that resolved it. + pub claude_code_path: String, pub dashboard_port: u16, pub operator_pronouns: String, pub context_window_tokens: std::collections::HashMap, @@ -235,6 +251,7 @@ impl Default for HiveEnv { hyperhive_flake: "/etc/hyperhive".to_string(), hyperhive_docs_flake: String::new(), nixpkgs_flake: String::new(), + claude_code_path: String::new(), dashboard_port: 7000, operator_pronouns: "she/her".to_string(), context_window_tokens: std::collections::HashMap::from([ @@ -474,6 +491,7 @@ impl Coordinator { hyperhive_flake, hyperhive_docs_flake, nixpkgs_flake, + claude_code_path, dashboard_port, operator_pronouns, context_window_tokens, @@ -520,6 +538,7 @@ impl Coordinator { hyperhive_flake, hyperhive_docs_flake, nixpkgs_flake, + claude_code_path, dashboard_port, operator_pronouns, context_window_tokens, @@ -553,6 +572,7 @@ impl Coordinator { hyperhive_flake: self.hyperhive_flake.clone(), hyperhive_docs_flake: self.hyperhive_docs_flake.clone(), nixpkgs_flake: self.nixpkgs_flake.clone(), + claude_code_path: self.claude_code_path.clone(), dashboard_port: self.dashboard_port, operator_pronouns: self.operator_pronouns.clone(), context_window_tokens: self.context_window_tokens.clone(), diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index bc1fb6f4..b36183fc 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -92,6 +92,7 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> { &hive.hyperhive_flake, &hive.hyperhive_docs_flake, &hive.nixpkgs_flake, + &hive.claude_code_path, hive.dashboard_port, &hive.operator_pronouns, &hive.context_window_tokens, @@ -650,6 +651,7 @@ fn render_flake( hyperhive_flake: &str, docs_flake: &str, nixpkgs_flake: &str, + claude_code_path: &str, dashboard_port: u16, operator_pronouns: &str, context_window_tokens: &std::collections::HashMap, @@ -660,6 +662,7 @@ fn render_flake( hyperhive_flake, docs_flake, nixpkgs_flake, + claude_code_path, dashboard_port, operator_pronouns, context_window_tokens, @@ -937,6 +940,7 @@ fn render_flake_with_lookup( hyperhive_flake: &str, docs_flake: &str, nixpkgs_flake: &str, + claude_code_path: &str, dashboard_port: u16, operator_pronouns: &str, context_window_tokens: &std::collections::HashMap, @@ -1051,6 +1055,21 @@ where if !docs_flake.is_empty() { out.push_str(" hyperhive.docs.source = inputs.\"hyperhive-docs\".outPath;\n"); } + // The `claude-code` agents run, as a bare store path rather than a + // flake input: containers share the host's `/nix/store`, so the + // binary is already there with its closure and has nothing to + // travel. A string literal is also the only shape that evaluates — + // `lib.types.package` on a bare path runs `builtins.storePath`, + // which pure eval rejects. The agent module puts its `bin/` on the + // harness PATH; the host module holds the gc root, since a path + // spelled out here is text and references nothing. + // Empty = no override; agents keep their own nixpkgs' `claude-code`. + if !claude_code_path.is_empty() { + let _ = writeln!( + out, + " hyperhive.claudeCodePath = \"{claude_code_path}\";" + ); + } // CA trust: embed every hive-trusted CA so each agent validates them at // build time. The list is the hive's own self-signed CA (when active) // plus every peer-hive root CA (`swarm.peers..caCert`) — a peer CA is @@ -1561,6 +1580,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1605,6 +1625,7 @@ mod tests { "github:example/hyperhive", "", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1617,6 +1638,57 @@ mod tests { ); } + #[test] + fn render_flake_pins_claude_path_without_adding_an_input() { + // The host-pinned claude travels as a bare store path assigned to + // an option — deliberately NOT as a flake input. Containers share + // the host store, so the build is already reachable; making it an + // input would re-copy it as a reference-less `-source` and strip + // the closure the binary actually needs. + let out = render_flake( + "github:example/hyperhive", + "path:/nix/store/bbbb-hyperhive-docs-source", + "path:/nix/store/aaaa-nixpkgs-source", + "/nix/store/cccc-claude-code-2.1.220", + 8000, + "she/her", + &std::collections::HashMap::new(), + "4G", + &[sample_spec("alice", false, 9001)], + ); + assert!( + out.contains("hyperhive.claudeCodePath = \"/nix/store/cccc-claude-code-2.1.220\";"), + "claude path assigned as a plain string literal:\n{out}" + ); + assert!( + !out.contains("claude-code-2.1.220\".url"), + "the pinned claude must not become a flake input:\n{out}" + ); + } + + #[test] + fn render_flake_omits_claude_path_when_unset() { + // Empty = no host-level pin: the option is left undefined so the + // agent module keeps its own nixpkgs' `claude-code` (and keeps it + // in `environment.systemPackages`, which is what makes the + // unpinned case self-contained). + let out = render_flake( + "github:example/hyperhive", + "path:/nix/store/bbbb-hyperhive-docs-source", + "path:/nix/store/aaaa-nixpkgs-source", + "", + 8000, + "she/her", + &std::collections::HashMap::new(), + "4G", + &[sample_spec("alice", false, 9001)], + ); + assert!( + !out.contains("claudeCodePath"), + "no claude assignment when unpinned:\n{out}" + ); + } + #[test] fn render_flake_falls_back_to_follows_when_nixpkgs_flake_empty() { // Empty nixpkgs_flake → legacy follows behaviour (backward compat @@ -1625,6 +1697,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1655,6 +1728,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1688,6 +1762,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1723,6 +1798,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1761,6 +1837,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1803,6 +1880,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1886,6 +1964,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1971,6 +2050,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -2013,6 +2093,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index 810ab3b5..ffec918e 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -61,6 +61,33 @@ ''; }; + options.hyperhive.claudeCodePath = lib.mkOption { + type = lib.types.str; + default = ""; + example = "/nix/store/…-claude-code-2.1.220"; + description = '' + Store path of the `claude-code` this agent runs, or `""` (the + default) to use the `claude-code` from the container's own + nixpkgs. + + Set by the generated meta flake when the operator sets + `services.hyperhive.c0re.claudeCodePackage` host-side, so a hive + can run a claude built from a *different* nixpkgs than the one + its agents evaluate against — a release channel can trail + unstable by weeks on this one package. It arrives as a path and + not a package because agents share the host's `/nix/store`, so + the build is already reachable here with its whole closure and + has nothing to travel. + + When set, `claude` on PATH is a symlink to this path's + `bin/claude` and the container's own `claude-code` is dropped, so + there is only ever one claude in the container. Note that neither + the symlink nor anything else in the container's closure *refers* + to the target — keeping it alive is the host's job, see + `services.hyperhive.c0re.claudeCodePackage`. + ''; + }; + config = { assertions = [ # Guard the inputs-routed-as-output pattern: the agent flake.nix is @@ -183,8 +210,31 @@ environment.systemPackages = [ config.hyperhive.packages.hive-metric ] + ++ [ + ( + if config.hyperhive.claudeCodePath == "" then + pkgs.claude-code + else + # Host-pinned claude: a symlink farm around a path the + # container was handed as text. It has to be a derivation — + # `environment.systemPackages` coerces a store-path *string* + # with `toDerivation`, i.e. `builtins.storePath`, which pure + # evaluation rejects (`systemd.services.*.path` does the same, + # which is why the harness gets this via PATH like everything + # else rather than a unit-level entry). Interpolating the path + # into the builder is just text, so it evaluates anywhere. + # + # The symlink registers no store reference — the target isn't + # among this derivation's inputs, so nothing here keeps the + # binary alive. That is deliberate and it is the host's job: + # see `services.hyperhive.c0re.claudeCodePackage`. + pkgs.runCommandLocal "claude-code-pinned" { } '' + mkdir -p "$out/bin" + ln -s ${config.hyperhive.claudeCodePath}/bin/claude "$out/bin/claude" + '' + ) + ] ++ (with pkgs; [ - claude-code bashInteractive coreutils-full # procps for pkill — used by the web UI's /api/cancel to SIGINT the diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index 5372ed7a..06aab90e 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -59,6 +59,22 @@ let fi ''; + # Store path of the `claude-code` every agent runs, or "" for "each + # agent keeps the one out of its own nixpkgs". meta.rs writes it into + # each agent's generated flake as a plain string literal, and the agent + # module puts its `bin/` on the harness PATH. + # + # This interpolation is also the package's gc root, and the only one: + # it carries store context, `builtins.toJSON` preserves that, so the + # /etc entry below genuinely references the package and the host's + # system closure holds it alive. Nothing on the container side can — + # a path spelled out in a generated flake is text, not a reference. + # Hence the assertion further down: do NOT discard this context, and + # do not hand meta.rs the path by a route that drops it. The failure + # mode is a garbage-collected `claude` and a hive that can't take a + # turn, weeks after the commit that caused it. + claudeCodePath = if cfg.claudeCodePackage == null then "" else "${cfg.claudeCodePackage}"; + # The `hive-c0re serve` config JSON. Keys are snake_case to match the # `ServeConfig` serde shape the daemon deserialises (the # container-injected HiveEnv fields, flattened, plus the hive-c0re-local @@ -76,6 +92,7 @@ let hyperhive_flake = cfg.hyperhiveFlake; hyperhive_docs_flake = cfg.hyperhiveDocs; nixpkgs_flake = cfg.nixpkgsFlake; + claude_code_path = claudeCodePath; dashboard_port = cfg.dashboardPort; operator_pronouns = cfg.operatorPronouns; context_window_tokens = cfg.contextWindowTokens; @@ -94,6 +111,27 @@ in ]; config = lib.mkIf cfg.enable { + assertions = [ + { + # The pinned claude reaches agents as a bare path, so this + # string's store context is the whole reason the binary survives + # a `nix-collect-garbage`. Losing it is invisible at eval and at + # deploy — it only shows up as every agent failing to spawn + # `claude`, at whatever unrelated moment the gc runs. Cheap + # enough to just check. + assertion = cfg.claudeCodePackage == null || builtins.hasContext claudeCodePath; + message = '' + services.hyperhive.c0re.claudeCodePackage lost its store + context on the way into /etc/hyperhive/serve.json, so the + package is no longer gc-rooted by the system closure and + `nix-collect-garbage` may delete the claude every agent runs. + Something on that path discarded the context (e.g. + builtins.unsafeDiscardStringContext, toString, or reading the + path back out of a plain file) — undo it. + ''; + } + ]; + environment.systemPackages = [ cfg.package pkgs.git diff --git a/nix/host-modules/hive-c0re/options.nix b/nix/host-modules/hive-c0re/options.nix index eee0237c..bd10cecd 100644 --- a/nix/host-modules/hive-c0re/options.nix +++ b/nix/host-modules/hive-c0re/options.nix @@ -128,6 +128,45 @@ of the host's channel. ''; }; + claudeCodePackage = lib.mkOption { + type = lib.types.nullOr lib.types.package; + default = null; + example = lib.literalExpression "inputs.nixpkgs-unstable.legacyPackages.x86_64-linux.claude-code"; + description = '' + The `claude-code` build every agent runs, or `null` (the + default) to leave each agent on the `claude-code` from its own + nixpkgs — i.e. whatever `nixpkgsFlake` resolves to. + + This is the one binary the whole hive is built around, and it + moves fast enough that a release channel routinely trails + unstable by weeks on it. An agent cannot fix that for itself: + agents evaluate against the single nixpkgs hive-c0re injects, + so an `agent.nix` has no other tree to reach for. Set this from + a second nixpkgs in the host flake and every agent follows, + without moving the nixpkgs the rest of the container is built + from. + + What travels into the container is the **store path**, not the + derivation: agents share the host's `/nix/store`, so the binary + and its full closure are already reachable there — nothing + needs rebuilding or copying. hive-c0re writes the path into + each agent's generated flake as a plain string literal (a bare + path fed to `lib.types.package` would run `builtins.storePath`, + which is illegal under pure evaluation) and the agent module + puts its `bin/` on the harness's PATH. + + The flip side of a plain string is that nothing in the agent's + own closure refers to it, so the container cannot keep it + alive. The **host** does that instead: this package is + interpolated into `/etc/hyperhive/serve.json`, which puts it in + the host's system closure — so it is gc-rooted by the running + generation for exactly as long as that generation is the one + the agents were rendered from. The cost is that + `nix-collect-garbage` cannot reclaim an old `claude-code` until + every agent has been rebuilt past it and the old generations + are gone. + ''; + }; dashboardPort = lib.mkOption { type = lib.types.port; default = 7000;