fix(#2673): set nix fallback in agent + CI containers

Container nix invocations hard-failed whenever the remote builder
muede-pc2 was unreachable, while the identical build on the host
succeeded. Both go through the same host nix-daemon, so the difference
looked impossible.

The cause is that `fallback` is a client-side option: the nix client
transmits it to the daemon per connection (`tryFallback` in setOptions),
so the caller decides whether a failed remote dispatch may degrade to a
local build — even when the build itself runs on the host daemon under
NIX_REMOTE=daemon. Only genuinely daemon-side settings such as
`builders` are inherited from the host. The containers never set
`fallback`, so they took nix's default of false.

Set it in the agent-container base module and in the CI container, and
correct the hive-ci comment that claimed fallback was inherited from
the host daemon along with buildMachines and max-jobs.

Verified in an agent container: `nix fmt` fails outright on the remote
store's connection reset, while the same command with fallback enabled
reports the same connection error and then builds locally and succeeds.
This commit is contained in:
atlas 2026-07-25 19:33:11 +02:00 committed by mara
commit 65a0686297
2 changed files with 26 additions and 2 deletions

View file

@ -131,6 +131,22 @@
# that bypasses the daemon (e.g. direct nix-store invocations).
nix.settings.sandbox-fallback = lib.mkForce true;
# Fall back to a local build when a remote builder can't be reached,
# instead of hard-failing the whole invocation.
#
# This is NOT redundant with the host daemon's own setting, and that
# is the subtle part: `fallback` (protocol `tryFallback`) is a *client*
# option. Every nix client transmits it to the daemon on connect, so
# the client — i.e. this container — decides whether a failed remote
# dispatch may degrade to a local build, even though the build itself
# executes on the host daemon under NIX_REMOTE=daemon below. Contrast
# `builders`, which is genuinely daemon-side and which an untrusted
# client cannot override. Without this, a container inherits nix's
# default `false`, so a momentarily unreachable remote builder kills
# the invocation while the exact same build started on the host
# succeeds.
nix.settings.fallback = true;
# Route ALL nix invocations in this container through the host
# nix-daemon socket, regardless of whether the caller is root or
# non-root. Without this, root contexts (PID 1, systemd services

View file

@ -205,8 +205,9 @@ in
# Route the container's nix through the HOST nix-daemon (as the agent
# containers do) instead of an in-container daemon: with
# NIX_REMOTE=daemon below, builds run on the host daemon and inherit
# its buildMachines + max-jobs + fallback, so a remote builder that's
# down degrades to a local build instead of hard-failing the check.
# its buildMachines + max-jobs. Note that `fallback` is NOT inherited
# that way — it's a client-side option transmitted per connection, so
# the container sets it itself (see nix.settings.fallback below).
# Bind the *directory* (not the socket file) so the mount stays live
# across a host nix-daemon restart, which recreates the socket inode.
"/nix/var/nix/daemon-socket" = {
@ -252,6 +253,13 @@ in
# through the host daemon (the daemon governs sandboxing).
# See docs/gotchas.md and harness-base.nix.
nix.settings.sandbox-fallback = lib.mkForce true;
# Degrade to a local build when a remote builder is unreachable
# rather than failing the check. `fallback` is a client-side
# option (transmitted per daemon connection), so routing through
# the host daemon does NOT inherit the host's value — the
# container has to set it. Same reasoning as the agent
# containers; see nix/agent-modules/default.nix.
nix.settings.fallback = true;
nix.settings.experimental-features = [
"nix-command"
"flakes"