Watch
0
0
Fork
You've already forked hyperhive
0

add a per-agent nix builds mode, host or none, carried like privateUsers

This commit is contained in:
atlas 2026-11-03 00:30:55 +01:00 • committed by mara
commit 5485151ba6
2 changed files with 618 additions and 58 deletions

View file

@ -306,7 +306,10 @@ nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.con
### Containerized nix-daemon needs `sandbox-fallback = true`
Agent containers bind-mount the host's nix-daemon socket. nspawn
nixos-containers' start script bind-mounts the host's nix-daemon socket
directory into every container (hyperhive adds no bind for it; an agent
with `services.hyperhive.agent.nixBuilds = "none"` gets it shadowed, see
`docs/trust-boundary/security.md`). nspawn
containers don't get user-namespaces by default, so `nix build`
invocations _inside_ the container can't set up the build sandbox
and fail outright if the host daemon's

View file

@ -174,15 +174,53 @@ before building `agent_notes_dir(name)`, so a `..` component can't traverse.
### Background
Agent containers bind-mount the host's `nix-daemon` socket. The host daemon may
nixos-containers' start script bind-mounts the host's `nix-daemon` socket
directory (`/nix/var/nix/daemon-socket`) into every container, hyperhive's
agents included; hyperhive adds no bind for it. The host daemon may
have `sandbox-fallback = false` (strict NixOS defaults), which causes `nix build`
inside nspawn containers to fail — containers lack kernel user namespaces, so nix
can't set up its build sandbox. the agent modules set `sandbox-fallback = true`
so that builds fall back to unsandboxed execution rather than failing outright.
### Per-agent nix builds mode
`services.hyperhive.agent.nixBuilds` picks where an agent's nix builds run. It
has two values, and defaults to `host`:
- **`host`**: builds go to the host daemon through the socket above. The daemon
forks them in its own cgroup, so they run **outside the agent container's
CPU and memory caps**, which sit on `container-<name>.service`.
- **`none`**: the container has no route to the host daemon. hive-priv adds
`--tmpfs=/nix/var/nix/daemon-socket` to the container's nspawn flags. nspawn
mounts its custom mounts sorted by destination, then by kind (a bind before
a tmpfs), so the empty tmpfs lands on top of the start script's bind
whatever order the flags are in, and the container finds no `socket` in the
directory. `NIX_REMOTE=daemon` stays, so a build fails with `cannot connect
to socket at '/nix/var/nix/daemon-socket/socket'`, which names the cause. The
agent's `/nix/store` and its read-only nix database stay bound, so store
paths already there stay readable; only a realisation of a new path, or any
`nix` command that needs the daemon, fails.
The mode reaches the container as `privateUsers` does: the swarm builder reads
the option from the evaluated config into the agent's closure record
(`nix_builds`), hive-c0re reads the record when it writes the container's
flags and sends the mode to hive-priv, and the flags apply at the container's
next start.
- A record or builder from before the field reads `host`.
- A mode this build doesn't know (a record written by a newer swarm) reads as
`none`, never `host`: an agent that asked for a mode its hive can't give gets
no builds, not builds outside its limits.
- A record hive-c0re can't read, or none, sends no mode, and hive-priv keeps
what the container's last write chose.
- A hive-c0re or hive-priv from before the option ignores it, so the agent keeps
host builds until you upgrade both.
`none` cuts the daemon only. It leaves the agent's egress alone.
### Threat model
Unsandboxed nix builds run as `nixbld` users (non-root, typically UIDs 30001-30010).
Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up).
Without sandbox isolation, a build derivation's builder script has read access to
any file in the container that the nixbld user can read.