diff --git a/docs/2-handbook/agent-lifecycle/persistence.md b/docs/2-handbook/agent-lifecycle/persistence.md index 4db31cde..d9b864d0 100644 --- a/docs/2-handbook/agent-lifecycle/persistence.md +++ b/docs/2-handbook/agent-lifecycle/persistence.md @@ -767,7 +767,7 @@ container lifetime: user**, `0751`, not recursive. hive-priv creates it `0751 root` on the host before every start; the harness binds its sockets there as the agent user. Why the mode matters: - [`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir). + [`boundary-mechanism.md`](../../4-implementation/trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir). A namespaced agent has the dir bound at `/agents//sockets` and linked from `/run/hive-agent/`; the activation chowns nothing, because a mount point under `/run` would belong to root and the id mapping would diff --git a/docs/2-handbook/trust-boundary/boundary.md b/docs/2-handbook/trust-boundary/boundary.md index d6fc058d..99b81d83 100644 --- a/docs/2-handbook/trust-boundary/boundary.md +++ b/docs/2-handbook/trust-boundary/boundary.md @@ -81,74 +81,7 @@ The boundary rests on three layers: 3. **Privsep** — defence in depth on the core process; `hive-c0re` runs as the unprivileged `hive-core` user and delegates root operations to `hive-priv`, a narrow socket-activated helper. See - [`docs/2-handbook/trust-boundary/security.md`](security.md) for the privilege boundary table. - -### hive-priv socket activation - -`hive-priv` is **always** socket-activated by the `hive-priv.socket` -systemd unit. The unit binds `/run/hive/priv.sock` with -`SocketGroup=hive-core` and mode `0660` and passes the ready listener -to the helper as fd 3 (`LISTEN_FDS`). The helper requires this and -bails if it isn't socket-activated. - -⚠️ Intentionally, no self-bind fallback exists: if `hive-priv` bound -the socket itself, it would create the file owned by root's primary -group rather than `hive-core`, and a `hive-core` client couldn't -connect the way the socket unit's `SocketGroup` grant intends. -Requiring socket activation everywhere keeps dev and prod on the -exact same path, so the group grant always holds. - -### the per-agent socket dir - -**Three principals that share no group** reach `/run/hive-agent//`, -which is why its mode is what it's: - -| principal | reaches | needs | -| ------------------- | ---------------------------------------- | ------------ | -| the agent's harness | binds + unlinks `agent.sock`, `web.sock` | owner, `rwx` | -| `hive-c0re` | dials `agent.sock` (todo wakes) | traverse | -| the gateway's nginx | dials `web.sock` | traverse | - -The last two land in "other," so the dir is **`0751`, owned by the -agent's container uid/gid** — `o=--x` is traverse without listing, and -both sockets are `0666`, which is all a dialer needs. - - - -**One mechanism creates it, one sets its owner.** Before every start, -hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751 -root:root`, no symlink followed) and leaves an existing one alone. The -container's own activation (`hive-agent-user-migrate`) then chowns it to -the agent user and sets `0751`. Without a user namespace -that uid is the host inode's owner. (A namespaced agent, see "User namespace" in -`docs/2-handbook/agent-lifecycle/persistence.md`, gets the dir from `EnsureAgentSocketDir` -owned by its data owner instead, because the container can't chown through -the id map.) Don't add another host-side chown or chmod: -two owners of one path revert each other. Until the container activates, -the dir is `0751 root`: nothing but root can plant a socket in it. - - - -The mode is load-bearing, not cosmetic. Write permission on a -_directory_ is what confers the right to unlink its entries, whoever owns -them, and the sticky bit is the only thing that would restrain that (it -isn't set here). A world-writable socket dir therefore lets anything -able to reach the path delete an agent's socket and bind its own. On the -host, any non-root process whose `/run` is writable can do that, such as -a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view -of `/run` read-only. Dropping `o=w` removes that permission rather than -qualifying it. - -⚠️ **The gateway's nginx and dnsmasq are host services, next to -`hive-c0re`** (see `docs/4-implementation/networking/gateway.md`) — there is no namespace between -them and the rest of the host. That costs no network isolation: nginx -binds the host's `:80`/`:443` and reaches `localhost` upstreams, which -would require opening a netns anyway. -🔑 It does mean nothing _implicitly_ scopes the privileged reload verb — -see [`docs/2-handbook/trust-boundary/security.md`](security.md#hive-c0re-privilege-separation) for -how `PrivRequest::ReloadGatewayNginx`'s containment works. - -⚠️ This dir has exactly one writer, so it needs no world write at all. + [`docs/4-implementation/trust-boundary/hive-c0re-privilege.md`](../../4-implementation/trust-boundary/hive-c0re-privilege.md) for the privilege boundary table. ### host admin socket access (`hivectl`) @@ -168,3 +101,9 @@ reach the socket path; the socket's own `0660 hive-admin` mode gates the connection, and the per-agent subdirs under it keep their own restrictive perms. Keep `adminUsers` to trusted operators — membership is equivalent to root over the hive. + +## Implementation detail + +How `hive-priv` is socket-activated and how the per-agent socket directory +gets its owner and mode: +[socket mechanics](../../4-implementation/trust-boundary/boundary-mechanism.md). diff --git a/docs/2-handbook/trust-boundary/security.md b/docs/2-handbook/trust-boundary/security.md index a6a25fcb..68efd23b 100644 --- a/docs/2-handbook/trust-boundary/security.md +++ b/docs/2-handbook/trust-boundary/security.md @@ -170,233 +170,9 @@ it first. The agent socket's `GetAgentMeta` takes `name` as a serde-validated `hive_types::Ident` (or falls back to `Ident::parse` for the "self" case) before building `agent_notes_dir(name)`, so a `..` component can't traverse. -## Nix builds and credential isolation +## Implementation detail -### Background - -nixos-containers' start script bind-mounts the host's `nix-daemon` socket -directory (`/nix/var/nix/daemon-socket`) into every container, hyperhive's -agents included; hyperhive adds no bind for it. The host daemon may -have `sandbox-fallback = false` (strict NixOS defaults), which causes `nix build` -inside nspawn containers to fail — containers lack kernel user namespaces, so nix -can't set up its build sandbox. the agent modules set `sandbox-fallback = true` -so that builds fall back to unsandboxed execution rather than failing outright. - -### Per-agent nix builds mode - -`services.hyperhive.agent.nixBuilds` picks where an agent's nix builds run. It -has two values, and defaults to `host`: - -- **`host`**: builds go to the host daemon through the socket above. The daemon - forks them in its own cgroup, so they run **outside the agent container's - CPU and memory caps**, which sit on `container-.service`. -- **`none`**: the container has no route to the host daemon. hive-priv adds - `--tmpfs=/nix/var/nix/daemon-socket` to the container's nspawn flags. nspawn - mounts its custom mounts sorted by destination, then by kind (a bind before - a tmpfs), so the empty tmpfs lands on top of the start script's bind - whatever order the flags are in, and the container finds no `socket` in the - directory. `NIX_REMOTE=daemon` stays, so a build fails with `cannot connect -to socket at '/nix/var/nix/daemon-socket/socket'`, which names the cause. The - agent's `/nix/store` and its read-only nix database stay bound, so store - paths already there stay readable; only a realisation of a new path, or any - `nix` command that needs the daemon, fails. - The agent's own NixOS config turns its `nix-daemon` socket and service off in - this mode: the tmpfs is writable, so the unit would otherwise listen on it. - Under `host` the directory is the host's read-only bind, which the socket - unit's `ConditionPathIsReadWrite` rejects, so that unit never starts. - -The mode reaches the container as `privateUsers` does: the swarm builder reads -the option from the evaluated config into the agent's closure record -(`nix_builds`), hive-c0re reads the record when it writes the container's -flags and sends the mode to hive-priv, and the flags apply at the container's -next start. - -- An agent whose module doesn't declare the option reads `host`. When the builder - can't read the option (its eval fails or times out), the record carries no - mode, and so does a record or builder from before the field. hive-c0re then - sends no mode, and hive-priv keeps what the container's last write chose. -- A mode this build doesn't know (a record written by a newer swarm) reads as - `none`, never `host`: an agent that asked for a mode its hive can't give gets - no builds, not builds outside its limits. -- A record hive-c0re can't read, or none, sends no mode either, and hive-priv - keeps what the container's last write chose. -- A hive-c0re or hive-priv from before the option ignores it, so the agent keeps - host builds until you upgrade both. - -`none` cuts the daemon only. It leaves the agent's egress alone. - -### Threat model - -Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up). -Without sandbox isolation, a build derivation's builder script has read access to -any file in the container that the nixbld user can read. - -The blast radius also has a **network** dimension. `swarm-ci-runner` runs its -unsandboxed builds of untrusted PR code in its own private netns behind the -bridge: a build reaches the forge by name and can't reach host-loopback -services — including the core dashboard at `127.0.0.1:`, which -has no application-layer auth of its own (see [`docs/4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md)). The `0600` -token policy bounds file reads; network isolation bounds network reach. - -**What's NOT exposed**: - -- `/home//.claude/` — `.credentials.json`, `history.jsonl` and - `settings.json` are `0600`; `projects/` and `sessions/` are `0700`. All owned - by the per-agent user ``, so nixbld users can't read any of them. - ⚠️ The **directory itself is `0755`**, on purpose: `hive-core` is a different - user and needs read+execute to list it so `claude_has_session` can detect a - valid session (`ensure_claude_dir`, `hive-c0re/src/lifecycle/setup.rs`). - The per-file mode is therefore the whole protection here — anything added to - this directory at a default mode is world-readable, which isn't - hypothetical: `plugins/` and `.last-cleanup` already are. -- `$HYPERHIVE_STATE_DIR/forge-token` (= `/agents//state/forge-token`) - — the token hive-c0re wrote at mode `0600`, chowned to the per-agent - uid:gid, before swarm-controller took over minting. Nothing writes it any - more; it stays as the fallback for an agent without a store identity. - nixbld users can't read it. The swarm-minted token lives under - `/run/hive-agent-forge-token/` (`0700` directory, `0400` file), outside - the state dir. -- `/run/hive-agent-proxy-credential/proxy.env` — the agent's egress-proxy - credential inside its proxy URL (`0700` directory, `0600` file, tmpfs). Units - load it as an environment file, so the credential is also in the environment of - every process they start. It authenticates this agent to the hive's squid and - nothing else. - -**Policy**: all credential files written to agent state directories MUST be mode -`0600` or stricter. Don't create world-readable secret files in agent state dirs. - -### Long-term fix - -The proper fix is to enable user namespaces inside nspawn containers -(`--private-users=inherit` in `EXTRA_NSPAWN_FLAGS`) so nix can set up its real -sandbox and `sandbox-fallback` becomes a true last resort. This requires verifying -bind-mount compatibility with user namespace UID mapping first. - -## hive-c0re privilege separation - -### Background - -`hive-c0re` runs as the unprivileged system user `hive-core` -(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't -directly invoke `nixos-container` or act on a system -unit (`systemctl reload nginx`) — those require root. `hive-priv` fills -this gap. - -⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly -scopes it.** Its containment is the unit name hard-coded in `hive-priv`: -a caller can't name the unit, so it can't steer the verb at another -service. **A privileged verb needs something bounding what it can act -on; when that isn't a namespace, it has to be a constant the caller -can't supply.** - -### hive-priv - -`hive-priv` is a minimal privileged helper that runs as root, socket-activated -at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the -`hive-core` user can connect). `hive-c0re` calls it via `priv_client` -for every operation that genuinely requires root. - -**Narrow interface** — `PrivRequest` variants map 1:1 to specific -known operations; there is no arbitrary command pass-through: - -| Operation | What it runs | -| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `StartContainer` / `StopContainer` | `nixos-container start/stop ` | -| `KillContainer` | `machinectl kill --signal=SIGKILL` (`nixos-container` has no kill verb) | -| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-.service --since @ -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. | -| `CreateContainer` / `UpdateContainer` | `nixos-container create --system-path `; for an update, `nix-env -p --set ` and a reload of the unit. `` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one | -| `RealiseClosure` | `nix-store --realise --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-` and the agent name valid. It downloads and never builds | -| `DestroyContainer` | `nixos-container destroy ` | -| `ListContainers` | `nixos-container list` | -| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | -| `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner | -| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner | -| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs ` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names | -| `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named | -| `DaemonReload` | `systemctl daemon-reload` | -| `ControlInfraContainer` | `systemctl nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) | -| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` | -| `SetAgentPaused` | create / remove the `//harness/paused` marker that parks an agent's turn loop | -| `EnsureAgentSubvolume` | `btrfs subvolume create /` for a new agent — no-op when the path exists or the filesystem isn't btrfs | -| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first | -| `DeleteAgentSubvolume` | `btrfs subvolume delete /` — purge path only, never a plain destroy | -| `EnsureBtrfsQuota` | `btrfs quota enable `. Operator opt-in — enabling forces a full rescan | -| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw /` | -| `SetSubvolumeQuota` | `btrfs qgroup limit /` | -| `MeasureAgentDisk` | `du -s -x -b -c -P` over `//state`, `//work` and `/var/lib/nixos-containers/h-`, killed after 10 min. hive-priv derives the paths from the name; the request carries none | -| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r ` — the frozen point-in-time copy a migration streams from | -| `DeleteAgentSnapshot` | `btrfs subvolume delete ` | -| `SendAgentSnapshotToFile` | `btrfs send [-p ] ` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export | -| `SendAgentSnapshotToFd` | `btrfs send [-p ] ` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation | - -**Strings that reach root-loaded config** — `WriteResourceLimits` and -`WriteNspawnFlags` copy request strings into a systemd drop-in and an env -file that systemd and the nixos-containers start script read, so hive-priv -refuses, never escapes, any value outside its shape. `memory_max` is -`infinity`, a decimal percentage or a decimal size with an optional -`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is -1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A -bind or credential path is absolute with no whitespace, control byte, `\`, -`$`, backtick, `"` or `:`. The size limits of a value stay with systemd. - -**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild -takes the closure the swarm built instead of building it. Root-owned sources -alone decide what root accepts: - -- The cache signs. The swarm's cache (attic) holds the signing key and signs - each narinfo it serves; the private key never leaves it. The builder pushes - with a push token and holds no signing key. -- The address is the host's configuration: - `services.hyperhive.swarm.attic.domain` and `cache` become - `HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses - it, and `hive-priv` refuses an address that isn't `https`. -- The key and the pull token come from the secret store. The root unit - `hive-priv-cache-facts` reads the hive's own copies under the hive's own store - identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`, - files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`, - which `hive-c0re` writes for the probe, and it refuses a file in the root - directory that isn't a plain file owned by root or that others can read, and - the directory under the same rule. -- Nix verifies. For that one call `hive-priv` sets `substituters` and - `trusted-public-keys` to the swarm cache and its key, and `require-sigs true`. - A path without a signature by that key isn't stored; `hive-priv` answers - `not available` and logs the refusal at error level. `--max-jobs 0` means no - answer, whatever the cache sends, starts a build as root. -- A hive realises only paths whose signature checks. The request carries an - agent name and a store path, both validated, and no address, key or file; any - other field never reaches the verb. - -What a compromised `hive-c0re` can do with this verb: ask for any store path -that has the shape of one and that the swarm cache signed, which fills the -store with paths the swarm vouches for. It can't make root accept a path from -a cache or under a key of its choosing, because it writes none of the places -those come from. The trust anchor is the cache's signing key and the store that -hands out its public key, as for every other credential. - -**Container allowlist** — `hive-priv` validates every request against -an allowlist before any operation: the allowlist accepts only names -matching the agent-name convention (`h-` prefix, char-validated) -and rejects every other container name, the swarm's `hive-forge` and -`hive-matrix` included: those belong to the host that runs them, not to -a hive. `hive-gateway` is a host unit, -not a container, so it's not in this list — see `ReloadGatewayNginx` -above for how the hard-coded unit name scopes its access instead. - -**Socket-activated** — systemd starts `hive-priv` on the first -incoming connection (`LISTEN_FDS=1`); it's not running between calls. -The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem -writes to only the paths `hive-priv` legitimately needs. The unit also sets -`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and -`ProtectKernelModules`: it changes no execution domain, schedules nothing in -real time, reads no kernel log, and loads no module. It sets none of -`RestrictNamespaces` (`nsenter` into a container needs `setns`), -`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or -`SystemCallFilter`; those need a watched deploy to judge. - -### Privilege boundary summary - -| Component | Runs as | Privilege needed for | -| ---------------------------- | -------------- | --------------------------------------------- | -| `hive-c0re` | `hive-core` | broker, HTTP dashboard | -| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes | -| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving | +- How in-container `nix build` stays apart from the credentials, and its + threat model: [nix builds](../../4-implementation/trust-boundary/nix-builds.md). +- How `hive-c0re` runs unprivileged and delegates root operations: + [privilege separation](../../4-implementation/trust-boundary/hive-c0re-privilege.md). diff --git a/docs/4-implementation/networking/gateway.md b/docs/4-implementation/networking/gateway.md index ff5590ea..68fe9f49 100644 --- a/docs/4-implementation/networking/gateway.md +++ b/docs/4-implementation/networking/gateway.md @@ -281,7 +281,7 @@ Every agent binds its web UI on a unix-domain socket at nginx reaches `web.sock` through `o=--x` (traverse) and the socket's own `0666`. The gateway is one of three principals sharing that dir and doesn't own its ownership rules — see - [`docs/2-handbook/trust-boundary/boundary.md`](../../2-handbook/trust-boundary/boundary.md#the-per-agent-socket-dir). + [`docs/4-implementation/trust-boundary/boundary-mechanism.md`](../trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir). 3. **Marker gate**. After a successful `bind_unix`, the harness drops `/hyperhive-socket-bound` next to the socket. c0re's `agent_sockets::write` filters its JSON map by marker presence — diff --git a/docs/4-implementation/trust-boundary/boundary-mechanism.md b/docs/4-implementation/trust-boundary/boundary-mechanism.md new file mode 100644 index 00000000..fa59b242 --- /dev/null +++ b/docs/4-implementation/trust-boundary/boundary-mechanism.md @@ -0,0 +1,72 @@ +# Socket mechanics behind the operator/agent boundary + +How systemd activates `hive-priv` and how the per-agent socket directory +gets its owner and mode, the two pieces of the boundary described in +[the operator/agent boundary](../../2-handbook/trust-boundary/boundary.md). + +## hive-priv socket activation + +`hive-priv` is **always** socket-activated by the `hive-priv.socket` +systemd unit. The unit binds `/run/hive/priv.sock` with +`SocketGroup=hive-core` and mode `0660` and passes the ready listener +to the helper as fd 3 (`LISTEN_FDS`). The helper requires this and +bails if it isn't socket-activated. + +⚠️ Intentionally, no self-bind fallback exists: if `hive-priv` bound +the socket itself, it would create the file owned by root's primary +group rather than `hive-core`, and a `hive-core` client couldn't +connect the way the socket unit's `SocketGroup` grant intends. +Requiring socket activation everywhere keeps dev and prod on the +exact same path, so the group grant always holds. + +## the per-agent socket dir + +**Three principals that share no group** reach `/run/hive-agent//`, +which is why its mode is what it's: + +| principal | reaches | needs | +| ------------------- | ---------------------------------------- | ------------ | +| the agent's harness | binds + unlinks `agent.sock`, `web.sock` | owner, `rwx` | +| `hive-c0re` | dials `agent.sock` (todo wakes) | traverse | +| the gateway's nginx | dials `web.sock` | traverse | + +The last two land in "other," so the dir is **`0751`, owned by the +agent's container uid/gid** — `o=--x` is traverse without listing, and +both sockets are `0666`, which is all a dialer needs. + + + +**One mechanism creates it, one sets its owner.** Before every start, +hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751 +root:root`, no symlink followed) and leaves an existing one alone. The +container's own activation (`hive-agent-user-migrate`) then chowns it to +the agent user and sets `0751`. Without a user namespace +that uid is the host inode's owner. (A namespaced agent, see "User namespace" in +`docs/2-handbook/agent-lifecycle/persistence.md`, gets the dir from `EnsureAgentSocketDir` +owned by its data owner instead, because the container can't chown through +the id map.) Don't add another host-side chown or chmod: +two owners of one path revert each other. Until the container activates, +the dir is `0751 root`: nothing but root can plant a socket in it. + + + +The mode is load-bearing, not cosmetic. Write permission on a +_directory_ is what confers the right to unlink its entries, whoever owns +them, and the sticky bit is the only thing that would restrain that (it +isn't set here). A world-writable socket dir therefore lets anything +able to reach the path delete an agent's socket and bind its own. On the +host, any non-root process whose `/run` is writable can do that, such as +a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view +of `/run` read-only. Dropping `o=w` removes that permission rather than +qualifying it. + +⚠️ **The gateway's nginx and dnsmasq are host services, next to +`hive-c0re`** (see `docs/4-implementation/networking/gateway.md`) — there is no namespace between +them and the rest of the host. That costs no network isolation: nginx +binds the host's `:80`/`:443` and reaches `localhost` upstreams, which +would require opening a netns anyway. +🔑 It does mean nothing _implicitly_ scopes the privileged reload verb — +see [`docs/4-implementation/trust-boundary/hive-c0re-privilege.md`](hive-c0re-privilege.md) for +how `PrivRequest::ReloadGatewayNginx`'s containment works. + +⚠️ This dir has exactly one writer, so it needs no world write at all. diff --git a/docs/4-implementation/trust-boundary/hive-c0re-privilege.md b/docs/4-implementation/trust-boundary/hive-c0re-privilege.md new file mode 100644 index 00000000..62d3fd65 --- /dev/null +++ b/docs/4-implementation/trust-boundary/hive-c0re-privilege.md @@ -0,0 +1,131 @@ +# hive-c0re privilege separation + +How `hive-c0re` runs unprivileged and delegates its root operations to +`hive-priv`, from the [security model](../../2-handbook/trust-boundary/security.md). + +## Background + +`hive-c0re` runs as the unprivileged system user `hive-core` +(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't +directly invoke `nixos-container` or act on a system +unit (`systemctl reload nginx`) — those require root. `hive-priv` fills +this gap. + +⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly +scopes it.** Its containment is the unit name hard-coded in `hive-priv`: +a caller can't name the unit, so it can't steer the verb at another +service. **A privileged verb needs something bounding what it can act +on; when that isn't a namespace, it has to be a constant the caller +can't supply.** + +## hive-priv + +`hive-priv` is a minimal privileged helper that runs as root, socket-activated +at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the +`hive-core` user can connect). `hive-c0re` calls it via `priv_client` +for every operation that genuinely requires root. + +**Narrow interface** — `PrivRequest` variants map 1:1 to specific +known operations; there is no arbitrary command pass-through: + +| Operation | What it runs | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `StartContainer` / `StopContainer` | `nixos-container start/stop ` | +| `KillContainer` | `machinectl kill --signal=SIGKILL` (`nixos-container` has no kill verb) | +| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-.service --since @ -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. | +| `CreateContainer` / `UpdateContainer` | `nixos-container create --system-path `; for an update, `nix-env -p --set ` and a reload of the unit. `` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one | +| `RealiseClosure` | `nix-store --realise --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-` and the agent name valid. It downloads and never builds | +| `DestroyContainer` | `nixos-container destroy ` | +| `ListContainers` | `nixos-container list` | +| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | +| `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner | +| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner | +| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs ` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names | +| `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named | +| `DaemonReload` | `systemctl daemon-reload` | +| `ControlInfraContainer` | `systemctl nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) | +| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` | +| `SetAgentPaused` | create / remove the `//harness/paused` marker that parks an agent's turn loop | +| `EnsureAgentSubvolume` | `btrfs subvolume create /` for a new agent — no-op when the path exists or the filesystem isn't btrfs | +| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first | +| `DeleteAgentSubvolume` | `btrfs subvolume delete /` — purge path only, never a plain destroy | +| `EnsureBtrfsQuota` | `btrfs quota enable `. Operator opt-in — enabling forces a full rescan | +| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw /` | +| `SetSubvolumeQuota` | `btrfs qgroup limit /` | +| `MeasureAgentDisk` | `du -s -x -b -c -P` over `//state`, `//work` and `/var/lib/nixos-containers/h-`, killed after 10 min. hive-priv derives the paths from the name; the request carries none | +| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r ` — the frozen point-in-time copy a migration streams from | +| `DeleteAgentSnapshot` | `btrfs subvolume delete ` | +| `SendAgentSnapshotToFile` | `btrfs send [-p ] ` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export | +| `SendAgentSnapshotToFd` | `btrfs send [-p ] ` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation | + +**Strings that reach root-loaded config** — `WriteResourceLimits` and +`WriteNspawnFlags` copy request strings into a systemd drop-in and an env +file that systemd and the nixos-containers start script read, so hive-priv +refuses, never escapes, any value outside its shape. `memory_max` is +`infinity`, a decimal percentage or a decimal size with an optional +`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is +1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A +bind or credential path is absolute with no whitespace, control byte, `\`, +`$`, backtick, `"` or `:`. The size limits of a value stay with systemd. + +**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild +takes the closure the swarm built instead of building it. Root-owned sources +alone decide what root accepts: + +- The cache signs. The swarm's cache (attic) holds the signing key and signs + each narinfo it serves; the private key never leaves it. The builder pushes + with a push token and holds no signing key. +- The address is the host's configuration: + `services.hyperhive.swarm.attic.domain` and `cache` become + `HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses + it, and `hive-priv` refuses an address that isn't `https`. +- The key and the pull token come from the secret store. The root unit + `hive-priv-cache-facts` reads the hive's own copies under the hive's own store + identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`, + files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`, + which `hive-c0re` writes for the probe, and it refuses a file in the root + directory that isn't a plain file owned by root or that others can read, and + the directory under the same rule. +- Nix verifies. For that one call `hive-priv` sets `substituters` and + `trusted-public-keys` to the swarm cache and its key, and `require-sigs true`. + A path without a signature by that key isn't stored; `hive-priv` answers + `not available` and logs the refusal at error level. `--max-jobs 0` means no + answer, whatever the cache sends, starts a build as root. +- A hive realises only paths whose signature checks. The request carries an + agent name and a store path, both validated, and no address, key or file; any + other field never reaches the verb. + +What a compromised `hive-c0re` can do with this verb: ask for any store path +that has the shape of one and that the swarm cache signed, which fills the +store with paths the swarm vouches for. It can't make root accept a path from +a cache or under a key of its choosing, because it writes none of the places +those come from. The trust anchor is the cache's signing key and the store that +hands out its public key, as for every other credential. + +**Container allowlist** — `hive-priv` validates every request against +an allowlist before any operation: the allowlist accepts only names +matching the agent-name convention (`h-` prefix, char-validated) +and rejects every other container name, the swarm's `hive-forge` and +`hive-matrix` included: those belong to the host that runs them, not to +a hive. `hive-gateway` is a host unit, +not a container, so it's not in this list — see `ReloadGatewayNginx` +above for how the hard-coded unit name scopes its access instead. + +**Socket-activated** — systemd starts `hive-priv` on the first +incoming connection (`LISTEN_FDS=1`); it's not running between calls. +The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem +writes to only the paths `hive-priv` legitimately needs. The unit also sets +`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and +`ProtectKernelModules`: it changes no execution domain, schedules nothing in +real time, reads no kernel log, and loads no module. It sets none of +`RestrictNamespaces` (`nsenter` into a container needs `setns`), +`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or +`SystemCallFilter`; those need a watched deploy to judge. + +## Privilege boundary summary + +| Component | Runs as | Privilege needed for | +| ---------------------------- | -------------- | --------------------------------------------- | +| `hive-c0re` | `hive-core` | broker, HTTP dashboard | +| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes | +| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving | diff --git a/docs/4-implementation/trust-boundary/nix-builds.md b/docs/4-implementation/trust-boundary/nix-builds.md new file mode 100644 index 00000000..5e9047ed --- /dev/null +++ b/docs/4-implementation/trust-boundary/nix-builds.md @@ -0,0 +1,104 @@ +# Nix builds and credential isolation + +The mechanism and threat model for `nix build` inside an agent container, from +the [security model](../../2-handbook/trust-boundary/security.md). + +## Background + +nixos-containers' start script bind-mounts the host's `nix-daemon` socket +directory (`/nix/var/nix/daemon-socket`) into every container, hyperhive's +agents included; hyperhive adds no bind for it. The host daemon may +have `sandbox-fallback = false` (strict NixOS defaults), which causes `nix build` +inside nspawn containers to fail — containers lack kernel user namespaces, so nix +can't set up its build sandbox. the agent modules set `sandbox-fallback = true` +so that builds fall back to unsandboxed execution rather than failing outright. + +## Per-agent nix builds mode + +`services.hyperhive.agent.nixBuilds` picks where an agent's nix builds run. It +has two values, and defaults to `host`: + +- **`host`**: builds go to the host daemon through the socket above. The daemon + forks them in its own cgroup, so they run **outside the agent container's + CPU and memory caps**, which sit on `container-.service`. +- **`none`**: the container has no route to the host daemon. hive-priv adds + `--tmpfs=/nix/var/nix/daemon-socket` to the container's nspawn flags. nspawn + mounts its custom mounts sorted by destination, then by kind (a bind before + a tmpfs), so the empty tmpfs lands on top of the start script's bind + whatever order the flags are in, and the container finds no `socket` in the + directory. `NIX_REMOTE=daemon` stays, so a build fails with `cannot connect +to socket at '/nix/var/nix/daemon-socket/socket'`, which names the cause. The + agent's `/nix/store` and its read-only nix database stay bound, so store + paths already there stay readable; only a realisation of a new path, or any + `nix` command that needs the daemon, fails. + The agent's own NixOS config turns its `nix-daemon` socket and service off in + this mode: the tmpfs is writable, so the unit would otherwise listen on it. + Under `host` the directory is the host's read-only bind, which the socket + unit's `ConditionPathIsReadWrite` rejects, so that unit never starts. + +The mode reaches the container as `privateUsers` does: the swarm builder reads +the option from the evaluated config into the agent's closure record +(`nix_builds`), hive-c0re reads the record when it writes the container's +flags and sends the mode to hive-priv, and the flags apply at the container's +next start. + +- An agent whose module doesn't declare the option reads `host`. When the builder + can't read the option (its eval fails or times out), the record carries no + mode, and so does a record or builder from before the field. hive-c0re then + sends no mode, and hive-priv keeps what the container's last write chose. +- A mode this build doesn't know (a record written by a newer swarm) reads as + `none`, never `host`: an agent that asked for a mode its hive can't give gets + no builds, not builds outside its limits. +- A record hive-c0re can't read, or none, sends no mode either, and hive-priv + keeps what the container's last write chose. +- A hive-c0re or hive-priv from before the option ignores it, so the agent keeps + host builds until you upgrade both. + +`none` cuts the daemon only. It leaves the agent's egress alone. + +## Threat model + +Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up). +Without sandbox isolation, a build derivation's builder script has read access to +any file in the container that the nixbld user can read. + +The blast radius also has a **network** dimension. `swarm-ci-runner` runs its +unsandboxed builds of untrusted PR code in its own private netns behind the +bridge: a build reaches the forge by name and can't reach host-loopback +services — including the core dashboard at `127.0.0.1:`, which +has no application-layer auth of its own (see [`docs/4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md)). The `0600` +token policy bounds file reads; network isolation bounds network reach. + +**What's NOT exposed**: + +- `/home//.claude/` — `.credentials.json`, `history.jsonl` and + `settings.json` are `0600`; `projects/` and `sessions/` are `0700`. All owned + by the per-agent user ``, so nixbld users can't read any of them. + ⚠️ The **directory itself is `0755`**, on purpose: `hive-core` is a different + user and needs read+execute to list it so `claude_has_session` can detect a + valid session (`ensure_claude_dir`, `hive-c0re/src/lifecycle/setup.rs`). + The per-file mode is therefore the whole protection here — anything added to + this directory at a default mode is world-readable, which isn't + hypothetical: `plugins/` and `.last-cleanup` already are. +- `$HYPERHIVE_STATE_DIR/forge-token` (= `/agents//state/forge-token`) + — the token hive-c0re wrote at mode `0600`, chowned to the per-agent + uid:gid, before swarm-controller took over minting. Nothing writes it any + more; it stays as the fallback for an agent without a store identity. + nixbld users can't read it. The swarm-minted token lives under + `/run/hive-agent-forge-token/` (`0700` directory, `0400` file), outside + the state dir. +- `/run/hive-agent-proxy-credential/proxy.env` — the agent's egress-proxy + credential inside its proxy URL (`0700` directory, `0600` file, tmpfs). Units + load it as an environment file, so the credential is also in the environment of + every process they start. It authenticates this agent to the hive's squid and + nothing else. + +**Policy**: all credential files written to agent state directories MUST be mode +`0600` or stricter. Don't create world-readable secret files in agent state dirs. + +## Long-term fix + +The proper fix is to enable user namespaces inside nspawn containers +(`--private-users=inherit` in `EXTRA_NSPAWN_FLAGS`) so nix can set up its real +sandbox and `sandbox-fallback` becomes a true last resort. This requires verifying +bind-mount compatibility with user namespace UID mapping first. diff --git a/docs/5-project/process/gotchas.md b/docs/5-project/process/gotchas.md index 43c73a5c..1fe5d943 100644 --- a/docs/5-project/process/gotchas.md +++ b/docs/5-project/process/gotchas.md @@ -309,14 +309,14 @@ nix build "git+/internal/meta.git?rev=#nixosConfigurations.argus.con nixos-containers' start script bind-mounts the host's nix-daemon socket directory into every container (hyperhive adds no bind for it; an agent with `services.hyperhive.agent.nixBuilds = "none"` gets it shadowed, see -`docs/2-handbook/trust-boundary/security.md`). nspawn +`docs/4-implementation/trust-boundary/nix-builds.md`). nspawn containers don't get user-namespaces by default, so `nix build` invocations _inside_ the container can't set up the build sandbox and fail outright if the host daemon's `nix.settings.sandbox-fallback` is `false` (nixpkgs default). `nix/agent-modules/default.nix` does `lib.mkForce true` so builds fall back to unsandboxed local builds rather than failing. Security -implications: `docs/2-handbook/trust-boundary/security.md`. +implications: `docs/4-implementation/trust-boundary/nix-builds.md`. ### Linking workspace binaries locally needs `nix develop` diff --git a/docs/README.md b/docs/README.md index 0d5ff323..8adc4c11 100644 --- a/docs/README.md +++ b/docs/README.md @@ -141,6 +141,15 @@ Nothing of it sits in this tree. - **How does an agent get a todo when someone assigns it an issue, and lose it when the issue closes?** → [`4-implementation/swarm/notify-router.md`](4-implementation/swarm/notify-router.md). +### Trust boundary mechanics + +- **How does systemd activate `hive-priv`, and how does the per-agent socket + directory get its owner and mode?** → [`4-implementation/trust-boundary/boundary-mechanism.md`](4-implementation/trust-boundary/boundary-mechanism.md). +- **How does `hive-c0re` run unprivileged and delegate root operations?** → + [`4-implementation/trust-boundary/hive-c0re-privilege.md`](4-implementation/trust-boundary/hive-c0re-privilege.md). +- **How do in-container nix builds stay apart from the credentials?** → + [`4-implementation/trust-boundary/nix-builds.md`](4-implementation/trust-boundary/nix-builds.md). + ### Web UI internals - **How's the dashboard built, and how does the terminal render?** → diff --git a/hive-priv-sock/README.md b/hive-priv-sock/README.md index 8b607a7c..f4163713 100644 --- a/hive-priv-sock/README.md +++ b/hive-priv-sock/README.md @@ -17,5 +17,5 @@ admin socket. Serde-derived request/response types only — no server or client logic. Both sides import them so the shapes stay in sync. See `docs/2-handbook/trust-boundary/boundary.md` + -`docs/2-handbook/trust-boundary/security.md` for the privilege boundary these types sit on, and +`docs/4-implementation/trust-boundary/hive-c0re-privilege.md` for the privilege boundary these types sit on, and `hive-priv/README` for the helper itself. diff --git a/hive-priv/README.md b/hive-priv/README.md index cf87319e..0ce3a7db 100644 --- a/hive-priv/README.md +++ b/hive-priv/README.md @@ -12,7 +12,7 @@ coordination logic (broker, HTTP, scheduling) stays in the _unprivileged_ Privsep. `hive-c0re` runs as the unprivileged `hive-core` user so a bug or a prompt-injection in the large daemon can't directly wield root. The few root operations it needs are funnelled through this small, auditable helper instead. -See `docs/2-handbook/trust-boundary/boundary.md` and `docs/2-handbook/trust-boundary/security.md` for the privilege boundary. +See `docs/2-handbook/trust-boundary/boundary.md` and `docs/4-implementation/trust-boundary/hive-c0re-privilege.md` for the privilege boundary. ## Security model diff --git a/hive-priv/src/main.rs b/hive-priv/src/main.rs index 904a0841..b80c8e29 100644 --- a/hive-priv/src/main.rs +++ b/hive-priv/src/main.rs @@ -109,7 +109,7 @@ fn check_socket_activation( fn socket_listener() -> Result { // hive-priv is ALWAYS socket-activated by the `hive-priv.socket` unit // (fd 3 via LISTEN_FDS). There is intentionally no self-bind fallback, - // so dev and prod take the same path; see docs/2-handbook/trust-boundary/boundary.md. + // so dev and prod take the same path; see docs/4-implementation/trust-boundary/boundary-mechanism.md. check_socket_activation( std::env::var("LISTEN_FDS").ok().as_deref(), std::env::var("LISTEN_PID").ok().as_deref(), diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index 778cac88..2ee8ce80 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -200,7 +200,7 @@ in # in-container `nix build` invocations fall back to unsandboxed # local builds rather than failing on the missing user-namespace. # See `docs/5-project/process/gotchas.md::Containerized nix-daemon needs - # sandbox-fallback = true` + `docs/2-handbook/trust-boundary/security.md` for the rationale. + # sandbox-fallback = true` + `docs/4-implementation/trust-boundary/nix-builds.md` for the rationale. # # Note: with NIX_REMOTE=daemon below this becomes a no-op for the # common case — daemon-routed builds run on the host where sandboxing diff --git a/nix/agent-modules/user.nix b/nix/agent-modules/user.nix index 76c1540a..0459ddfc 100644 --- a/nix/agent-modules/user.nix +++ b/nix/agent-modules/user.nix @@ -290,7 +290,7 @@ in # The socket dir is bind-mounted from the host, where hive-priv creates # it `0751 root`. The harness binds its sockets here as this user; the # 0751 is what lets hive-c0re and nginx dial them without listing - # (docs/2-handbook/trust-boundary/boundary.md, "the per-agent socket dir"). Not + # (docs/4-implementation/trust-boundary/boundary-mechanism.md, "the per-agent socket dir"). Not # recursive: only the harness writes inside it. `test -d` and `chmod` # follow symlinks, so a link here is refused rather than handed over. socketDir="/run/hive-agent/$userName"