112 lines
6.4 KiB
Markdown
112 lines
6.4 KiB
Markdown
# Nix builds and credential isolation
|
|
|
|
The mechanism and threat model for `nix build` inside an agent container, from
|
|
the [security model](../../2-handbook/agents/security.md).
|
|
|
|
## Background
|
|
|
|
nixos-containers' start script bind-mounts the host's `nix-daemon` socket
|
|
directory (`/nix/var/nix/daemon-socket`) into every container, hyperhive's
|
|
agents included; hyperhive adds no bind for it. The host daemon may
|
|
have `sandbox-fallback = false` (strict NixOS defaults), which causes `nix build`
|
|
inside nspawn containers to fail — containers lack kernel user namespaces, so nix
|
|
can't set up its build sandbox. the agent modules set `sandbox-fallback = true`
|
|
so that builds fall back to unsandboxed execution rather than failing outright.
|
|
|
|
## Per-agent nix builds mode
|
|
|
|
`services.hyperhive.agent.nixBuilds` picks where an agent's nix builds run. It
|
|
has two values, and defaults to `host`:
|
|
|
|
- **`host`**: builds go to the host daemon through the socket above. The daemon
|
|
forks them in its own cgroup, so they run **outside the agent container's
|
|
CPU and memory caps**, which sit on `container-<name>.service`.
|
|
- **`none`**: the container has no route to the host daemon. hive-priv adds
|
|
`--tmpfs=/nix/var/nix/daemon-socket` to the container's nspawn flags. nspawn
|
|
mounts its custom mounts sorted by destination, then by kind (a bind before
|
|
a tmpfs), so the empty tmpfs lands on top of the start script's bind
|
|
whatever order the flags are in, and the container finds no `socket` in the
|
|
directory. `NIX_REMOTE=daemon` stays, so a build fails with `cannot connect
|
|
to socket at '/nix/var/nix/daemon-socket/socket'`, which names the cause. The
|
|
agent's `/nix/store` and its read-only nix database stay bound, so store
|
|
paths already there stay readable; only a realisation of a new path, or any
|
|
`nix` command that needs the daemon, fails.
|
|
The agent's own NixOS config turns its `nix-daemon` socket and service off in
|
|
this mode: the tmpfs is writable, so the unit would otherwise listen on it.
|
|
Under `host` the directory is the host's read-only bind, which the socket
|
|
unit's `ConditionPathIsReadWrite` rejects, so that unit never starts.
|
|
|
|
The mode reaches the container as `privateUsers` does: the swarm builder reads
|
|
the option from the evaluated config into the agent's closure record
|
|
(`nix_builds`), hive-c0re reads the record when it writes the container's
|
|
flags and sends the mode to hive-priv, and the flags apply at the container's
|
|
next start.
|
|
|
|
- An agent whose module doesn't declare the option reads `host`. When the builder
|
|
can't read the option (its eval fails or times out), the record carries no
|
|
mode, and so does a record or builder from before the field. hive-c0re then
|
|
sends no mode, and hive-priv keeps what the container's last write chose.
|
|
- A mode this build doesn't know (a record written by a newer swarm) reads as
|
|
`none`, never `host`: an agent that asked for a mode its hive can't give gets
|
|
no builds, not builds outside its limits.
|
|
- A record hive-c0re can't read, or none, sends no mode either, and hive-priv
|
|
keeps what the container's last write chose.
|
|
- A hive-c0re or hive-priv from before the option ignores it, so the agent keeps
|
|
host builds until you upgrade both.
|
|
|
|
`none` cuts the daemon only. It leaves the agent's egress alone.
|
|
|
|
Under `host` an agent shares the daemon with the hive's own closure pull, and
|
|
gains nothing from it. The daemon trusts the swarm cache's public key, but it
|
|
has no cache address and no pull token (hive-c0re fetches with a netrc readable
|
|
only by `hive-core`), so an agent can't fetch from the private cache. It can
|
|
still ask the daemon to substitute from the public caches the host configures, as
|
|
before. See
|
|
[hive-c0re-privilege.md](hive-c0re-privilege.md#closures-from-the-swarm-cache).
|
|
|
|
## Threat model
|
|
|
|
Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up).
|
|
Without sandbox isolation, a build derivation's builder script has read access to
|
|
any file in the container that the nixbld user can read.
|
|
|
|
The blast radius also has a **network** dimension. `swarm-ci-runner` runs its
|
|
unsandboxed builds of untrusted PR code in its own private netns behind the
|
|
bridge: a build reaches the forge by name and can't reach host-loopback
|
|
services — including the core dashboard at `127.0.0.1:<dashboard_port>`, which
|
|
has no application-layer auth of its own (see [`docs/4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md)). The `0600`
|
|
token policy bounds file reads; network isolation bounds network reach.
|
|
|
|
**What's NOT exposed**:
|
|
|
|
- `/home/<name>/.claude/` — `.credentials.json`, `history.jsonl` and
|
|
`settings.json` are `0600`; `projects/` and `sessions/` are `0700`. All owned
|
|
by the per-agent user `<name>`, so nixbld users can't read any of them.
|
|
⚠️ The **directory itself is `0755`**, on purpose: `hive-core` is a different
|
|
user and needs read+execute to list it so `claude_has_session` can detect a
|
|
valid session (`ensure_claude_dir`, `hive-c0re/src/lifecycle/setup.rs`).
|
|
The per-file mode is therefore the whole protection here — anything added to
|
|
this directory at a default mode is world-readable, which isn't
|
|
hypothetical: `plugins/` and `.last-cleanup` already are.
|
|
- `$HYPERHIVE_STATE_DIR/forge-token` (= `/agents/<name>/state/forge-token`)
|
|
— the token hive-c0re wrote at mode `0600`, chowned to the per-agent
|
|
uid:gid, before swarm-controller took over minting. Nothing writes it any
|
|
more; it stays as the fallback for an agent without a store identity.
|
|
nixbld users can't read it. The swarm-minted token lives under
|
|
`/run/hive-agent-forge-token/` (`0700` directory, `0400` file), outside
|
|
the state dir.
|
|
- `/run/hive-agent-proxy-credential/proxy.env` — the agent's egress-proxy
|
|
credential inside its proxy URL (`0700` directory, `0600` file, tmpfs). Units
|
|
load it as an environment file, so the credential is also in the environment of
|
|
every process they start. It authenticates this agent to the hive's squid and
|
|
nothing else.
|
|
|
|
**Policy**: all credential files written to agent state directories MUST be mode
|
|
`0600` or stricter. Don't create world-readable secret files in agent state dirs.
|
|
|
|
## Long-term fix
|
|
|
|
The proper fix is to enable user namespaces inside nspawn containers
|
|
(`--private-users=inherit` in `EXTRA_NSPAWN_FLAGS`) so nix can set up its real
|
|
sandbox and `sandbox-fallback` becomes a true last resort. This requires verifying
|
|
bind-mount compatibility with user namespace UID mapping first.
|