diff --git a/docs/gotchas.md b/docs/gotchas.md index 119bc38e..6cbc7f9c 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -84,6 +84,47 @@ scoped, only this one package. This is needed because each per-agent operator's host-level `allowUnfree` does **not** propagate in. Operators don't need to set anything on their side. +That same isolation is why an agent can't pick a claude out of a +*different* nixpkgs by itself: a container only ever sees the one +nixpkgs the meta flake injects, so an `agent.nix` naming the host's +`nixpkgs-unstable` has nothing to name. A release channel can trail +unstable by weeks on this package, which is what +`services.hyperhive.c0re.claudeCodePackage` is for — set it host-side +and every agent runs that build. + +What crosses is the **store path**, not the derivation. Containers +share the host's `/nix/store`, so the binary is already reachable +inside them with its whole closure; hive-c0re writes the path into each +agent's flake as a string literal and the agent module symlinks +`bin/claude` onto PATH. Two things rule out the obvious alternatives: a +`path:/nix/store/` flake input is re-copied into the store as a +reference-less `-source` (so the runtime closure never arrives), and +`lib.types.package` fed a bare path runs `builtins.storePath`, which +pure evaluation rejects. `hyperhive.docs.source` gets away with being +an input only because a docs tree has no runtime dependencies. + +The `storePath` trap is worth spelling out, because it is not confined +to options the operator writes: **any** option of type `package` fed a +store-path *string* coerces through `lib.toDerivation`, i.e. +`builtins.storePath`. `environment.systemPackages` and +`systemd.services..path` both do it (the latter takes plain +strings like `/run/wrappers` happily, but anything under +`builtins.storeDir` is treated as a package). So a path handed to the +container as text has to be wrapped in a real derivation — a symlink +farm built from the interpolated string — before it can go anywhere a +package is expected. + +The catch is that a path written into a generated flake is text, not a +reference — the container's closure does not keep the binary alive. +The **host** does: the package is interpolated into +`/etc/hyperhive/serve.json`, so it lands in the host's system closure +and is gc-rooted by the running generation. `builtins.toJSON` preserves +string context, which is the load-bearing detail; discard the context +anywhere on that path and `nix-collect-garbage` will eventually take +the hive's `claude` out from under it. The price of the root is that an +old `claude-code` can't be reclaimed until every agent has rebuilt past +it and the old generations are gone. + ## Claude credentials are per-agent `/var/lib/hyperhive/agents//claude/` bind-mounts to diff --git a/docs/turn-loop/claude-invocation.md b/docs/turn-loop/claude-invocation.md index 9d93619e..b9d2f26c 100644 --- a/docs/turn-loop/claude-invocation.md +++ b/docs/turn-loop/claude-invocation.md @@ -21,6 +21,17 @@ auto-reset / retry decisions in `drive_turn`. The lib returns everything it parsed from a turn (usage, cost, context window, resolved model) as `Telemetry`, which the policy layer applies to the bus. +**Which `claude` binary.** The bare name `claude`, resolved off the +harness unit's PATH. By default that's the `claude-code` in the agent's +own nixpkgs (the meta flake's `nixpkgs` input) via +`environment.systemPackages`. Since that's usually a release channel and +this package moves fast, the operator can pin one hive-wide with +`services.hyperhive.c0re.claudeCodePackage`: its store path is written +into each agent's flake, and `claude` on PATH becomes a symlink to it +instead of the container's own `claude-code` — so there's only ever one +`claude` in the container. Agents pick up a new build on their +next rebuild, not live. See docs/gotchas.md::`claude-code` is unfree. + Hive-enforced settings ship at `/etc/claude-code/managed-settings.json` (claude-code's canonical managed-settings path — precedence #1, read-only, un-overridable), wired in `nix/agent-modules/claude-settings.nix` diff --git a/hive-c0re/src/coordinator.rs b/hive-c0re/src/coordinator.rs index c08927df..ee561c89 100644 --- a/hive-c0re/src/coordinator.rs +++ b/hive-c0re/src/coordinator.rs @@ -62,6 +62,17 @@ pub struct Coordinator { /// "nixpkgs"` is set in the host flake). Empty string = legacy /// `follows = "hyperhive/nixpkgs"` behaviour. pub nixpkgs_flake: String, + /// Store path of the `claude-code` build every agent runs, written + /// into each per-agent flake as `hyperhive.claudeCodePath`. Set by + /// the NixOS module option `services.hyperhive.c0re.claudeCodePackage` + /// (which resolves the package and hands us its path). Empty string = + /// every agent keeps the `claude-code` from its own nixpkgs. + /// + /// A path rather than a flake input because containers share the + /// host's `/nix/store`: the binary is already reachable inside them, + /// closure and all. The host module is what keeps it from being + /// garbage-collected — see that option. + pub claude_code_path: String, /// TCP port the host's hive-c0re dashboard listens on. Inlined into /// each per-agent flake so the agent's web UI can build the right /// rebuild-button URL pointing back at the dashboard. @@ -213,6 +224,11 @@ pub struct HiveEnv { /// so a doc edit only re-locks this input, not the whole source. pub hyperhive_docs_flake: String, pub nixpkgs_flake: String, + /// Store path of the `claude-code` agents run, or empty for "each + /// agent keeps the one out of its own nixpkgs". Travels into the + /// container as `hyperhive.claudeCodePath` — a plain string, kept + /// alive host-side by the module that resolved it. + pub claude_code_path: String, pub dashboard_port: u16, pub operator_pronouns: String, pub context_window_tokens: std::collections::HashMap, @@ -235,6 +251,7 @@ impl Default for HiveEnv { hyperhive_flake: "/etc/hyperhive".to_string(), hyperhive_docs_flake: String::new(), nixpkgs_flake: String::new(), + claude_code_path: String::new(), dashboard_port: 7000, operator_pronouns: "she/her".to_string(), context_window_tokens: std::collections::HashMap::from([ @@ -474,6 +491,7 @@ impl Coordinator { hyperhive_flake, hyperhive_docs_flake, nixpkgs_flake, + claude_code_path, dashboard_port, operator_pronouns, context_window_tokens, @@ -520,6 +538,7 @@ impl Coordinator { hyperhive_flake, hyperhive_docs_flake, nixpkgs_flake, + claude_code_path, dashboard_port, operator_pronouns, context_window_tokens, @@ -553,6 +572,7 @@ impl Coordinator { hyperhive_flake: self.hyperhive_flake.clone(), hyperhive_docs_flake: self.hyperhive_docs_flake.clone(), nixpkgs_flake: self.nixpkgs_flake.clone(), + claude_code_path: self.claude_code_path.clone(), dashboard_port: self.dashboard_port, operator_pronouns: self.operator_pronouns.clone(), context_window_tokens: self.context_window_tokens.clone(), diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index bc1fb6f4..b36183fc 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -92,6 +92,7 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> { &hive.hyperhive_flake, &hive.hyperhive_docs_flake, &hive.nixpkgs_flake, + &hive.claude_code_path, hive.dashboard_port, &hive.operator_pronouns, &hive.context_window_tokens, @@ -650,6 +651,7 @@ fn render_flake( hyperhive_flake: &str, docs_flake: &str, nixpkgs_flake: &str, + claude_code_path: &str, dashboard_port: u16, operator_pronouns: &str, context_window_tokens: &std::collections::HashMap, @@ -660,6 +662,7 @@ fn render_flake( hyperhive_flake, docs_flake, nixpkgs_flake, + claude_code_path, dashboard_port, operator_pronouns, context_window_tokens, @@ -937,6 +940,7 @@ fn render_flake_with_lookup( hyperhive_flake: &str, docs_flake: &str, nixpkgs_flake: &str, + claude_code_path: &str, dashboard_port: u16, operator_pronouns: &str, context_window_tokens: &std::collections::HashMap, @@ -1051,6 +1055,21 @@ where if !docs_flake.is_empty() { out.push_str(" hyperhive.docs.source = inputs.\"hyperhive-docs\".outPath;\n"); } + // The `claude-code` agents run, as a bare store path rather than a + // flake input: containers share the host's `/nix/store`, so the + // binary is already there with its closure and has nothing to + // travel. A string literal is also the only shape that evaluates — + // `lib.types.package` on a bare path runs `builtins.storePath`, + // which pure eval rejects. The agent module puts its `bin/` on the + // harness PATH; the host module holds the gc root, since a path + // spelled out here is text and references nothing. + // Empty = no override; agents keep their own nixpkgs' `claude-code`. + if !claude_code_path.is_empty() { + let _ = writeln!( + out, + " hyperhive.claudeCodePath = \"{claude_code_path}\";" + ); + } // CA trust: embed every hive-trusted CA so each agent validates them at // build time. The list is the hive's own self-signed CA (when active) // plus every peer-hive root CA (`swarm.peers..caCert`) — a peer CA is @@ -1561,6 +1580,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1605,6 +1625,7 @@ mod tests { "github:example/hyperhive", "", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1617,6 +1638,57 @@ mod tests { ); } + #[test] + fn render_flake_pins_claude_path_without_adding_an_input() { + // The host-pinned claude travels as a bare store path assigned to + // an option — deliberately NOT as a flake input. Containers share + // the host store, so the build is already reachable; making it an + // input would re-copy it as a reference-less `-source` and strip + // the closure the binary actually needs. + let out = render_flake( + "github:example/hyperhive", + "path:/nix/store/bbbb-hyperhive-docs-source", + "path:/nix/store/aaaa-nixpkgs-source", + "/nix/store/cccc-claude-code-2.1.220", + 8000, + "she/her", + &std::collections::HashMap::new(), + "4G", + &[sample_spec("alice", false, 9001)], + ); + assert!( + out.contains("hyperhive.claudeCodePath = \"/nix/store/cccc-claude-code-2.1.220\";"), + "claude path assigned as a plain string literal:\n{out}" + ); + assert!( + !out.contains("claude-code-2.1.220\".url"), + "the pinned claude must not become a flake input:\n{out}" + ); + } + + #[test] + fn render_flake_omits_claude_path_when_unset() { + // Empty = no host-level pin: the option is left undefined so the + // agent module keeps its own nixpkgs' `claude-code` (and keeps it + // in `environment.systemPackages`, which is what makes the + // unpinned case self-contained). + let out = render_flake( + "github:example/hyperhive", + "path:/nix/store/bbbb-hyperhive-docs-source", + "path:/nix/store/aaaa-nixpkgs-source", + "", + 8000, + "she/her", + &std::collections::HashMap::new(), + "4G", + &[sample_spec("alice", false, 9001)], + ); + assert!( + !out.contains("claudeCodePath"), + "no claude assignment when unpinned:\n{out}" + ); + } + #[test] fn render_flake_falls_back_to_follows_when_nixpkgs_flake_empty() { // Empty nixpkgs_flake → legacy follows behaviour (backward compat @@ -1625,6 +1697,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1655,6 +1728,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1688,6 +1762,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1723,6 +1798,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1761,6 +1837,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1803,6 +1880,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1886,6 +1964,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -1971,6 +2050,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), @@ -2013,6 +2093,7 @@ mod tests { "github:example/hyperhive", "path:/nix/store/bbbb-hyperhive-docs-source", "path:/nix/store/aaaa-nixpkgs-source", + "", 8000, "she/her", &std::collections::HashMap::new(), diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index 810ab3b5..ffec918e 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -61,6 +61,33 @@ ''; }; + options.hyperhive.claudeCodePath = lib.mkOption { + type = lib.types.str; + default = ""; + example = "/nix/store/…-claude-code-2.1.220"; + description = '' + Store path of the `claude-code` this agent runs, or `""` (the + default) to use the `claude-code` from the container's own + nixpkgs. + + Set by the generated meta flake when the operator sets + `services.hyperhive.c0re.claudeCodePackage` host-side, so a hive + can run a claude built from a *different* nixpkgs than the one + its agents evaluate against — a release channel can trail + unstable by weeks on this one package. It arrives as a path and + not a package because agents share the host's `/nix/store`, so + the build is already reachable here with its whole closure and + has nothing to travel. + + When set, `claude` on PATH is a symlink to this path's + `bin/claude` and the container's own `claude-code` is dropped, so + there is only ever one claude in the container. Note that neither + the symlink nor anything else in the container's closure *refers* + to the target — keeping it alive is the host's job, see + `services.hyperhive.c0re.claudeCodePackage`. + ''; + }; + config = { assertions = [ # Guard the inputs-routed-as-output pattern: the agent flake.nix is @@ -183,8 +210,31 @@ environment.systemPackages = [ config.hyperhive.packages.hive-metric ] + ++ [ + ( + if config.hyperhive.claudeCodePath == "" then + pkgs.claude-code + else + # Host-pinned claude: a symlink farm around a path the + # container was handed as text. It has to be a derivation — + # `environment.systemPackages` coerces a store-path *string* + # with `toDerivation`, i.e. `builtins.storePath`, which pure + # evaluation rejects (`systemd.services.*.path` does the same, + # which is why the harness gets this via PATH like everything + # else rather than a unit-level entry). Interpolating the path + # into the builder is just text, so it evaluates anywhere. + # + # The symlink registers no store reference — the target isn't + # among this derivation's inputs, so nothing here keeps the + # binary alive. That is deliberate and it is the host's job: + # see `services.hyperhive.c0re.claudeCodePackage`. + pkgs.runCommandLocal "claude-code-pinned" { } '' + mkdir -p "$out/bin" + ln -s ${config.hyperhive.claudeCodePath}/bin/claude "$out/bin/claude" + '' + ) + ] ++ (with pkgs; [ - claude-code bashInteractive coreutils-full # procps for pkill — used by the web UI's /api/cancel to SIGINT the diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index 5372ed7a..06aab90e 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -59,6 +59,22 @@ let fi ''; + # Store path of the `claude-code` every agent runs, or "" for "each + # agent keeps the one out of its own nixpkgs". meta.rs writes it into + # each agent's generated flake as a plain string literal, and the agent + # module puts its `bin/` on the harness PATH. + # + # This interpolation is also the package's gc root, and the only one: + # it carries store context, `builtins.toJSON` preserves that, so the + # /etc entry below genuinely references the package and the host's + # system closure holds it alive. Nothing on the container side can — + # a path spelled out in a generated flake is text, not a reference. + # Hence the assertion further down: do NOT discard this context, and + # do not hand meta.rs the path by a route that drops it. The failure + # mode is a garbage-collected `claude` and a hive that can't take a + # turn, weeks after the commit that caused it. + claudeCodePath = if cfg.claudeCodePackage == null then "" else "${cfg.claudeCodePackage}"; + # The `hive-c0re serve` config JSON. Keys are snake_case to match the # `ServeConfig` serde shape the daemon deserialises (the # container-injected HiveEnv fields, flattened, plus the hive-c0re-local @@ -76,6 +92,7 @@ let hyperhive_flake = cfg.hyperhiveFlake; hyperhive_docs_flake = cfg.hyperhiveDocs; nixpkgs_flake = cfg.nixpkgsFlake; + claude_code_path = claudeCodePath; dashboard_port = cfg.dashboardPort; operator_pronouns = cfg.operatorPronouns; context_window_tokens = cfg.contextWindowTokens; @@ -94,6 +111,27 @@ in ]; config = lib.mkIf cfg.enable { + assertions = [ + { + # The pinned claude reaches agents as a bare path, so this + # string's store context is the whole reason the binary survives + # a `nix-collect-garbage`. Losing it is invisible at eval and at + # deploy — it only shows up as every agent failing to spawn + # `claude`, at whatever unrelated moment the gc runs. Cheap + # enough to just check. + assertion = cfg.claudeCodePackage == null || builtins.hasContext claudeCodePath; + message = '' + services.hyperhive.c0re.claudeCodePackage lost its store + context on the way into /etc/hyperhive/serve.json, so the + package is no longer gc-rooted by the system closure and + `nix-collect-garbage` may delete the claude every agent runs. + Something on that path discarded the context (e.g. + builtins.unsafeDiscardStringContext, toString, or reading the + path back out of a plain file) — undo it. + ''; + } + ]; + environment.systemPackages = [ cfg.package pkgs.git diff --git a/nix/host-modules/hive-c0re/options.nix b/nix/host-modules/hive-c0re/options.nix index eee0237c..bd10cecd 100644 --- a/nix/host-modules/hive-c0re/options.nix +++ b/nix/host-modules/hive-c0re/options.nix @@ -128,6 +128,45 @@ of the host's channel. ''; }; + claudeCodePackage = lib.mkOption { + type = lib.types.nullOr lib.types.package; + default = null; + example = lib.literalExpression "inputs.nixpkgs-unstable.legacyPackages.x86_64-linux.claude-code"; + description = '' + The `claude-code` build every agent runs, or `null` (the + default) to leave each agent on the `claude-code` from its own + nixpkgs — i.e. whatever `nixpkgsFlake` resolves to. + + This is the one binary the whole hive is built around, and it + moves fast enough that a release channel routinely trails + unstable by weeks on it. An agent cannot fix that for itself: + agents evaluate against the single nixpkgs hive-c0re injects, + so an `agent.nix` has no other tree to reach for. Set this from + a second nixpkgs in the host flake and every agent follows, + without moving the nixpkgs the rest of the container is built + from. + + What travels into the container is the **store path**, not the + derivation: agents share the host's `/nix/store`, so the binary + and its full closure are already reachable there — nothing + needs rebuilding or copying. hive-c0re writes the path into + each agent's generated flake as a plain string literal (a bare + path fed to `lib.types.package` would run `builtins.storePath`, + which is illegal under pure evaluation) and the agent module + puts its `bin/` on the harness's PATH. + + The flip side of a plain string is that nothing in the agent's + own closure refers to it, so the container cannot keep it + alive. The **host** does that instead: this package is + interpolated into `/etc/hyperhive/serve.json`, which puts it in + the host's system closure — so it is gc-rooted by the running + generation for exactly as long as that generation is the one + the agents were rendered from. The cost is that + `nix-collect-garbage` cannot reclaim an old `claude-code` until + every agent has been rebuilt past it and the old generations + are gone. + ''; + }; dashboardPort = lib.mkOption { type = lib.types.port; default = 7000;