mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity
hyperhive/docs/4-implementation/trust-boundary/nix-builds.md
iris 30a8f71bb5 docs: the trust-boundary pages keep the operator guidance, the mechanism moves to implementation 
boundary.md keeps the model, the design rule, why network isolation
matters and host admin socket access; the hive-priv socket activation and
the per-agent socket directory move to 4-implementation/trust-boundary.
security.md keeps the agent trust model and the path-name rule; the nix
builds mode with its threat model, and the hive-c0re privilege separation,
move to their own pages there. Headings and text move as they were, with
one level shaved off the heading depth. Links, code and nix pointers, and
the docs index follow the moves.
2026-11-06 19:45:11 +01:00

5.9 KiB
Raw Blame History

Nix builds and credential isolation

The mechanism and threat model for nix build inside an agent container, from the security model.

Background

nixos-containers' start script bind-mounts the host's nix-daemon socket directory (/nix/var/nix/daemon-socket) into every container, hyperhive's agents included; hyperhive adds no bind for it. The host daemon may have sandbox-fallback = false (strict NixOS defaults), which causes nix build inside nspawn containers to fail — containers lack kernel user namespaces, so nix can't set up its build sandbox. the agent modules set sandbox-fallback = true so that builds fall back to unsandboxed execution rather than failing outright.

Per-agent nix builds mode

services.hyperhive.agent.nixBuilds picks where an agent's nix builds run. It has two values, and defaults to host:

  • host: builds go to the host daemon through the socket above. The daemon forks them in its own cgroup, so they run outside the agent container's CPU and memory caps, which sit on container-<name>.service.
  • none: the container has no route to the host daemon. hive-priv adds --tmpfs=/nix/var/nix/daemon-socket to the container's nspawn flags. nspawn mounts its custom mounts sorted by destination, then by kind (a bind before a tmpfs), so the empty tmpfs lands on top of the start script's bind whatever order the flags are in, and the container finds no socket in the directory. NIX_REMOTE=daemon stays, so a build fails with cannot connect to socket at '/nix/var/nix/daemon-socket/socket', which names the cause. The agent's /nix/store and its read-only nix database stay bound, so store paths already there stay readable; only a realisation of a new path, or any nix command that needs the daemon, fails. The agent's own NixOS config turns its nix-daemon socket and service off in this mode: the tmpfs is writable, so the unit would otherwise listen on it. Under host the directory is the host's read-only bind, which the socket unit's ConditionPathIsReadWrite rejects, so that unit never starts.

The mode reaches the container as privateUsers does: the swarm builder reads the option from the evaluated config into the agent's closure record (nix_builds), hive-c0re reads the record when it writes the container's flags and sends the mode to hive-priv, and the flags apply at the container's next start.

  • An agent whose module doesn't declare the option reads host. When the builder can't read the option (its eval fails or times out), the record carries no mode, and so does a record or builder from before the field. hive-c0re then sends no mode, and hive-priv keeps what the container's last write chose.
  • A mode this build doesn't know (a record written by a newer swarm) reads as none, never host: an agent that asked for a mode its hive can't give gets no builds, not builds outside its limits.
  • A record hive-c0re can't read, or none, sends no mode either, and hive-priv keeps what the container's last write chose.
  • A hive-c0re or hive-priv from before the option ignores it, so the agent keeps host builds until you upgrade both.

none cuts the daemon only. It leaves the agent's egress alone.

Threat model

Unsandboxed nix builds run as nixbld users (non-root; nrBuildUsers is at least 32, so UIDs 30001 and up). Without sandbox isolation, a build derivation's builder script has read access to any file in the container that the nixbld user can read.

The blast radius also has a network dimension. swarm-ci-runner runs its unsandboxed builds of untrusted PR code in its own private netns behind the bridge: a build reaches the forge by name and can't reach host-loopback services — including the core dashboard at 127.0.0.1:<dashboard_port>, which has no application-layer auth of its own (see docs/4-implementation/scheduler/ci.md). The 0600 token policy bounds file reads; network isolation bounds network reach.

What's NOT exposed:

  • /home/<name>/.claude/ — .credentials.json, history.jsonl and settings.json are 0600; projects/ and sessions/ are 0700. All owned by the per-agent user <name>, so nixbld users can't read any of them. ⚠️ The directory itself is 0755, on purpose: hive-core is a different user and needs read+execute to list it so claude_has_session can detect a valid session (ensure_claude_dir, hive-c0re/src/lifecycle/setup.rs). The per-file mode is therefore the whole protection here — anything added to this directory at a default mode is world-readable, which isn't hypothetical: plugins/ and .last-cleanup already are.
  • $HYPERHIVE_STATE_DIR/forge-token (= /agents/<name>/state/forge-token) — the token hive-c0re wrote at mode 0600, chowned to the per-agent uid:gid, before swarm-controller took over minting. Nothing writes it any more; it stays as the fallback for an agent without a store identity. nixbld users can't read it. The swarm-minted token lives under /run/hive-agent-forge-token/ (0700 directory, 0400 file), outside the state dir.
  • /run/hive-agent-proxy-credential/proxy.env — the agent's egress-proxy credential inside its proxy URL (0700 directory, 0600 file, tmpfs). Units load it as an environment file, so the credential is also in the environment of every process they start. It authenticates this agent to the hive's squid and nothing else.

Policy: all credential files written to agent state directories MUST be mode 0600 or stricter. Don't create world-readable secret files in agent state dirs.

Long-term fix

The proper fix is to enable user namespaces inside nspawn containers (--private-users=inherit in EXTRA_NSPAWN_FLAGS) so nix can set up its real sandbox and sandbox-fallback becomes a true last resort. This requires verifying bind-mount compatibility with user namespace UID mapping first.