add a per-agent nix builds mode, host or none, carried like privateUsers
This commit is contained in:
parent
b9484e62a0
commit
5485151ba6
2 changed files with 618 additions and 58 deletions
|
|
@ -306,7 +306,10 @@ nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.con
|
|||
|
||||
### Containerized nix-daemon needs `sandbox-fallback = true`
|
||||
|
||||
Agent containers bind-mount the host's nix-daemon socket. nspawn
|
||||
nixos-containers' start script bind-mounts the host's nix-daemon socket
|
||||
directory into every container (hyperhive adds no bind for it; an agent
|
||||
with `services.hyperhive.agent.nixBuilds = "none"` gets it shadowed, see
|
||||
`docs/trust-boundary/security.md`). nspawn
|
||||
containers don't get user-namespaces by default, so `nix build`
|
||||
invocations _inside_ the container can't set up the build sandbox
|
||||
and fail outright if the host daemon's
|
||||
|
|
|
|||
|
|
@ -174,15 +174,53 @@ before building `agent_notes_dir(name)`, so a `..` component can't traverse.
|
|||
|
||||
### Background
|
||||
|
||||
Agent containers bind-mount the host's `nix-daemon` socket. The host daemon may
|
||||
nixos-containers' start script bind-mounts the host's `nix-daemon` socket
|
||||
directory (`/nix/var/nix/daemon-socket`) into every container, hyperhive's
|
||||
agents included; hyperhive adds no bind for it. The host daemon may
|
||||
have `sandbox-fallback = false` (strict NixOS defaults), which causes `nix build`
|
||||
inside nspawn containers to fail — containers lack kernel user namespaces, so nix
|
||||
can't set up its build sandbox. the agent modules set `sandbox-fallback = true`
|
||||
so that builds fall back to unsandboxed execution rather than failing outright.
|
||||
|
||||
### Per-agent nix builds mode
|
||||
|
||||
`services.hyperhive.agent.nixBuilds` picks where an agent's nix builds run. It
|
||||
has two values, and defaults to `host`:
|
||||
|
||||
- **`host`**: builds go to the host daemon through the socket above. The daemon
|
||||
forks them in its own cgroup, so they run **outside the agent container's
|
||||
CPU and memory caps**, which sit on `container-<name>.service`.
|
||||
- **`none`**: the container has no route to the host daemon. hive-priv adds
|
||||
`--tmpfs=/nix/var/nix/daemon-socket` to the container's nspawn flags. nspawn
|
||||
mounts its custom mounts sorted by destination, then by kind (a bind before
|
||||
a tmpfs), so the empty tmpfs lands on top of the start script's bind
|
||||
whatever order the flags are in, and the container finds no `socket` in the
|
||||
directory. `NIX_REMOTE=daemon` stays, so a build fails with `cannot connect
|
||||
to socket at '/nix/var/nix/daemon-socket/socket'`, which names the cause. The
|
||||
agent's `/nix/store` and its read-only nix database stay bound, so store
|
||||
paths already there stay readable; only a realisation of a new path, or any
|
||||
`nix` command that needs the daemon, fails.
|
||||
|
||||
The mode reaches the container as `privateUsers` does: the swarm builder reads
|
||||
the option from the evaluated config into the agent's closure record
|
||||
(`nix_builds`), hive-c0re reads the record when it writes the container's
|
||||
flags and sends the mode to hive-priv, and the flags apply at the container's
|
||||
next start.
|
||||
|
||||
- A record or builder from before the field reads `host`.
|
||||
- A mode this build doesn't know (a record written by a newer swarm) reads as
|
||||
`none`, never `host`: an agent that asked for a mode its hive can't give gets
|
||||
no builds, not builds outside its limits.
|
||||
- A record hive-c0re can't read, or none, sends no mode, and hive-priv keeps
|
||||
what the container's last write chose.
|
||||
- A hive-c0re or hive-priv from before the option ignores it, so the agent keeps
|
||||
host builds until you upgrade both.
|
||||
|
||||
`none` cuts the daemon only. It leaves the agent's egress alone.
|
||||
|
||||
### Threat model
|
||||
|
||||
Unsandboxed nix builds run as `nixbld` users (non-root, typically UIDs 30001-30010).
|
||||
Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up).
|
||||
Without sandbox isolation, a build derivation's builder script has read access to
|
||||
any file in the container that the nixbld user can read.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue