From 3a4b8d987323b2dcad8b9bf87a37f41115874218 Mon Sep 17 00:00:00 2001 From: atlas Date: Sat, 4 Jul 2026 12:37:43 +0200 Subject: [PATCH] fix(#1748): route all nix invocations through the host daemon (NIX_REMOTE=daemon) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root contexts (systemd services running as root, PID 1) default to store=auto which resolves to the LOCAL nix store — bypassing the host daemon, its remote builders (muede-pc2), and any prebuilt derivation outputs already in the shared store. This causes spurious full rebuilds of agent toplevels that the host already built and cached. Two changes: harness-base.nix: - Add systemd.globalEnvironment.NIX_REMOTE = "daemon" — sets DefaultEnvironment in systemd.conf so every unit in the container inherits NIX_REMOTE=daemon. Non-root contexts already default to the daemon socket; this only matters for root services that would otherwise use the local store. - Add NIX_REMOTE = "daemon" to environment.variables so interactive shells also have it set (redundant with /etc/profile.d/nix-daemon.sh but explicit and profile-agnostic). hive-c0re.nix (hive-priv service): - Add NIX_REMOTE = "daemon" to the service environment. hive-priv runs as root and invokes nixos-container update + nix prebuild; these must route through the host daemon so they see the shared store and remote builders, not a private local store. The sandbox-fallback = true in harness-base.nix is kept as a belt- and-suspenders fallback but becomes a no-op for the common case once nix routes through the daemon (the daemon builds on the host where sandboxing works). --- nix/modules/hive-c0re.nix | 6 ++++++ nix/templates/harness-base.nix | 23 +++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/nix/modules/hive-c0re.nix b/nix/modules/hive-c0re.nix index 1a3f5360..c0119b40 100644 --- a/nix/modules/hive-c0re.nix +++ b/nix/modules/hive-c0re.nix @@ -1215,6 +1215,12 @@ in # errors out. Point HOME at the StateDirectory below (persistent, # so the cache survives across rebuilds). HOME = "/var/lib/hive-priv"; + # hive-priv runs as root. Root nix defaults to store=auto which + # resolves to the LOCAL store — bypassing the host daemon, its + # remote builders, and prebuilt derivation outputs. Force daemon + # routing so nixos-container update and the nix prebuild see the + # same store and substituters as every other build context. + NIX_REMOTE = "daemon"; }; serviceConfig = { ExecStart = "${cfg.package}/bin/hive-priv"; diff --git a/nix/templates/harness-base.nix b/nix/templates/harness-base.nix index 9733d9b7..0629116b 100644 --- a/nix/templates/harness-base.nix +++ b/nix/templates/harness-base.nix @@ -1394,6 +1394,10 @@ in HIVE_DEFAULT_EFFORT = config.hyperhive.effortLevel; HIVE_ASSETS_DIR = "${pkgs.hyperhive-assets}/share/hyperhive"; SHELL = "${pkgs.bashInteractive}/bin/bash"; + # Route interactive-shell nix invocations through the host daemon. + # Redundant with /etc/profile.d/nix-daemon.sh but ensures it's set + # regardless of which profile files are sourced. + NIX_REMOTE = "daemon"; } // lib.optionalAttrs (!config.hyperhive.autoCompact) { # Zero watermark disables proactive compaction; the reactive path @@ -1449,8 +1453,27 @@ in # local builds rather than failing on the missing user-namespace. # See `docs/gotchas.md::Containerized nix-daemon needs # sandbox-fallback = true` + `docs/security.md` for the rationale. + # + # Note: with NIX_REMOTE=daemon below this becomes a no-op for the + # common case — daemon-routed builds run on the host where sandboxing + # works. It stays as a belt-and-suspenders fallback for any context + # that bypasses the daemon (e.g. direct nix-store invocations). nix.settings.sandbox-fallback = lib.mkForce true; + # Route ALL nix invocations in this container through the host + # nix-daemon socket, regardless of whether the caller is root or + # non-root. Without this, root contexts (PID 1, systemd services + # running as root) default to store=auto which resolves to the LOCAL + # store — bypassing the shared daemon, its remote builders, and the + # host's prebuilt derivation cache, causing spurious full rebuilds. + # + # systemd.globalEnvironment sets DefaultEnvironment in systemd.conf, + # so every unit started by PID 1 inherits NIX_REMOTE=daemon. + # Non-root nix clients already default to the daemon socket, so this + # is a no-op for them; it only matters for root services that would + # otherwise silently use the local store. + systemd.globalEnvironment.NIX_REMOTE = "daemon"; + # `claude-code` is unfree. Each per-agent container's nixosConfiguration # evaluates its own `nixpkgs` instance, so the operator's host-level # `nixpkgs.config.allowUnfreePredicate` does not propagate into here —