Watch
0
0
Fork
You've already forked hyperhive
0

docs: the trust-boundary pages keep the operator guidance, the mechanism moves to implementation

boundary.md keeps the model, the design rule, why network isolation
matters and host admin socket access; the hive-priv socket activation and
the per-agent socket directory move to 4-implementation/trust-boundary.
security.md keeps the agent trust model and the path-name rule; the nix
builds mode with its threat model, and the hive-c0re privilege separation,
move to their own pages there. Headings and text move as they were, with
one level shaved off the heading depth. Links, code and nix pointers, and
the docs index follow the moves.
This commit is contained in:
iris 2026-11-06 19:07:29 +01:00 • committed by mara
commit 30a8f71bb5
3 changed files with 338 additions and 307 deletions

View file

@ -767,7 +767,7 @@ container lifetime:
user**, `0751`, not recursive. hive-priv creates it `0751 root`
on the host before every start; the harness binds its sockets
there as the agent user. Why the mode matters:
[`boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
[`boundary-mechanism.md`](../../4-implementation/trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir).
A namespaced agent has the dir bound at `/agents/<name>/sockets` and
linked from `/run/hive-agent/<name>`; the activation chowns nothing, because
a mount point under `/run` would belong to root and the id mapping would

View file

@ -81,74 +81,7 @@ The boundary rests on three layers:
3. **Privsep** — defence in depth on the core process; `hive-c0re`
runs as the unprivileged `hive-core` user and delegates root
operations to `hive-priv`, a narrow socket-activated helper. See
[`docs/2-handbook/trust-boundary/security.md`](security.md) for the privilege boundary table.
### hive-priv socket activation
`hive-priv` is **always** socket-activated by the `hive-priv.socket`
systemd unit. The unit binds `/run/hive/priv.sock` with
`SocketGroup=hive-core` and mode `0660` and passes the ready listener
to the helper as fd 3 (`LISTEN_FDS`). The helper requires this and
bails if it isn't socket-activated.
⚠️ Intentionally, no self-bind fallback exists: if `hive-priv` bound
the socket itself, it would create the file owned by root's primary
group rather than `hive-core`, and a `hive-core` client couldn't
connect the way the socket unit's `SocketGroup` grant intends.
Requiring socket activation everywhere keeps dev and prod on the
exact same path, so the group grant always holds.
### the per-agent socket dir
**Three principals that share no group** reach `/run/hive-agent/<name>/`,
which is why its mode is what it's:
| principal | reaches | needs |
| ------------------- | ---------------------------------------- | ------------ |
| the agent's harness | binds + unlinks `agent.sock`, `web.sock` | owner, `rwx` |
| `hive-c0re` | dials `agent.sock` (todo wakes) | traverse |
| the gateway's nginx | dials `web.sock` | traverse |
The last two land in "other," so the dir is **`0751`, owned by the
agent's container uid/gid** — `o=--x` is traverse without listing, and
both sockets are `0666`, which is all a dialer needs.
<!-- vale write-good.Passive = NO -->
**One mechanism creates it, one sets its owner.** Before every start,
hive-priv creates the dir when missing (`EnsureAgentSocketDir`, `0751
root:root`, no symlink followed) and leaves an existing one alone. The
container's own activation (`hive-agent-user-migrate`) then chowns it to
the agent user and sets `0751`. Without a user namespace
that uid is the host inode's owner. (A namespaced agent, see "User namespace" in
`docs/2-handbook/agent-lifecycle/persistence.md`, gets the dir from `EnsureAgentSocketDir`
owned by its data owner instead, because the container can't chown through
the id map.) Don't add another host-side chown or chmod:
two owners of one path revert each other. Until the container activates,
the dir is `0751 root`: nothing but root can plant a socket in it.
<!-- vale write-good.Passive = YES -->
The mode is load-bearing, not cosmetic. Write permission on a
_directory_ is what confers the right to unlink its entries, whoever owns
them, and the sticky bit is the only thing that would restrain that (it
isn't set here). A world-writable socket dir therefore lets anything
able to reach the path delete an agent's socket and bind its own. On the
host, any non-root process whose `/run` is writable can do that, such as
a login session or dnsmasq. nginx only dials: `ProtectSystem=strict` makes its view
of `/run` read-only. Dropping `o=w` removes that permission rather than
qualifying it.
⚠️ **The gateway's nginx and dnsmasq are host services, next to
`hive-c0re`** (see `docs/4-implementation/networking/gateway.md`) — there is no namespace between
them and the rest of the host. That costs no network isolation: nginx
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which
would require opening a netns anyway.
🔑 It does mean nothing _implicitly_ scopes the privileged reload verb —
see [`docs/2-handbook/trust-boundary/security.md`](security.md#hive-c0re-privilege-separation) for
how `PrivRequest::ReloadGatewayNginx`'s containment works.
⚠️ This dir has exactly one writer, so it needs no world write at all.
[`docs/4-implementation/trust-boundary/hive-c0re-privilege.md`](../../4-implementation/trust-boundary/hive-c0re-privilege.md) for the privilege boundary table.
### host admin socket access (`hivectl`)
@ -168,3 +101,9 @@ reach the socket path; the socket's own `0660 hive-admin` mode gates
the connection, and the per-agent subdirs under it keep their own
restrictive perms. Keep `adminUsers` to trusted operators — membership
is equivalent to root over the hive.
## Implementation detail
How `hive-priv` is socket-activated and how the per-agent socket directory
gets its owner and mode:
[socket mechanics](../../4-implementation/trust-boundary/boundary-mechanism.md).

View file

@ -170,233 +170,9 @@ it first. The agent socket's `GetAgentMeta` takes `name` as a serde-validated
`hive_types::Ident` (or falls back to `Ident::parse` for the "self" case)
before building `agent_notes_dir(name)`, so a `..` component can't traverse.
## Nix builds and credential isolation
## Implementation detail
### Background
nixos-containers' start script bind-mounts the host's `nix-daemon` socket
directory (`/nix/var/nix/daemon-socket`) into every container, hyperhive's
agents included; hyperhive adds no bind for it. The host daemon may
have `sandbox-fallback = false` (strict NixOS defaults), which causes `nix build`
inside nspawn containers to fail — containers lack kernel user namespaces, so nix
can't set up its build sandbox. the agent modules set `sandbox-fallback = true`
so that builds fall back to unsandboxed execution rather than failing outright.
### Per-agent nix builds mode
`services.hyperhive.agent.nixBuilds` picks where an agent's nix builds run. It
has two values, and defaults to `host`:
- **`host`**: builds go to the host daemon through the socket above. The daemon
forks them in its own cgroup, so they run **outside the agent container's
CPU and memory caps**, which sit on `container-<name>.service`.
- **`none`**: the container has no route to the host daemon. hive-priv adds
`--tmpfs=/nix/var/nix/daemon-socket` to the container's nspawn flags. nspawn
mounts its custom mounts sorted by destination, then by kind (a bind before
a tmpfs), so the empty tmpfs lands on top of the start script's bind
whatever order the flags are in, and the container finds no `socket` in the
directory. `NIX_REMOTE=daemon` stays, so a build fails with `cannot connect
to socket at '/nix/var/nix/daemon-socket/socket'`, which names the cause. The
agent's `/nix/store` and its read-only nix database stay bound, so store
paths already there stay readable; only a realisation of a new path, or any
`nix` command that needs the daemon, fails.
The agent's own NixOS config turns its `nix-daemon` socket and service off in
this mode: the tmpfs is writable, so the unit would otherwise listen on it.
Under `host` the directory is the host's read-only bind, which the socket
unit's `ConditionPathIsReadWrite` rejects, so that unit never starts.
The mode reaches the container as `privateUsers` does: the swarm builder reads
the option from the evaluated config into the agent's closure record
(`nix_builds`), hive-c0re reads the record when it writes the container's
flags and sends the mode to hive-priv, and the flags apply at the container's
next start.
- An agent whose module doesn't declare the option reads `host`. When the builder
can't read the option (its eval fails or times out), the record carries no
mode, and so does a record or builder from before the field. hive-c0re then
sends no mode, and hive-priv keeps what the container's last write chose.
- A mode this build doesn't know (a record written by a newer swarm) reads as
`none`, never `host`: an agent that asked for a mode its hive can't give gets
no builds, not builds outside its limits.
- A record hive-c0re can't read, or none, sends no mode either, and hive-priv
keeps what the container's last write chose.
- A hive-c0re or hive-priv from before the option ignores it, so the agent keeps
host builds until you upgrade both.
`none` cuts the daemon only. It leaves the agent's egress alone.
### Threat model
Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up).
Without sandbox isolation, a build derivation's builder script has read access to
any file in the container that the nixbld user can read.
The blast radius also has a **network** dimension. `swarm-ci-runner` runs its
unsandboxed builds of untrusted PR code in its own private netns behind the
bridge: a build reaches the forge by name and can't reach host-loopback
services — including the core dashboard at `127.0.0.1:<dashboard_port>`, which
has no application-layer auth of its own (see [`docs/4-implementation/scheduler/ci.md`](../../4-implementation/scheduler/ci.md)). The `0600`
token policy bounds file reads; network isolation bounds network reach.
**What's NOT exposed**:
- `/home/<name>/.claude/` — `.credentials.json`, `history.jsonl` and
`settings.json` are `0600`; `projects/` and `sessions/` are `0700`. All owned
by the per-agent user `<name>`, so nixbld users can't read any of them.
⚠️ The **directory itself is `0755`**, on purpose: `hive-core` is a different
user and needs read+execute to list it so `claude_has_session` can detect a
valid session (`ensure_claude_dir`, `hive-c0re/src/lifecycle/setup.rs`).
The per-file mode is therefore the whole protection here — anything added to
this directory at a default mode is world-readable, which isn't
hypothetical: `plugins/` and `.last-cleanup` already are.
- `$HYPERHIVE_STATE_DIR/forge-token` (= `/agents/<name>/state/forge-token`)
— the token hive-c0re wrote at mode `0600`, chowned to the per-agent
uid:gid, before swarm-controller took over minting. Nothing writes it any
more; it stays as the fallback for an agent without a store identity.
nixbld users can't read it. The swarm-minted token lives under
`/run/hive-agent-forge-token/` (`0700` directory, `0400` file), outside
the state dir.
- `/run/hive-agent-proxy-credential/proxy.env` — the agent's egress-proxy
credential inside its proxy URL (`0700` directory, `0600` file, tmpfs). Units
load it as an environment file, so the credential is also in the environment of
every process they start. It authenticates this agent to the hive's squid and
nothing else.
**Policy**: all credential files written to agent state directories MUST be mode
`0600` or stricter. Don't create world-readable secret files in agent state dirs.
### Long-term fix
The proper fix is to enable user namespaces inside nspawn containers
(`--private-users=inherit` in `EXTRA_NSPAWN_FLAGS`) so nix can set up its real
sandbox and `sandbox-fallback` becomes a true last resort. This requires verifying
bind-mount compatibility with user namespace UID mapping first.
## hive-c0re privilege separation
### Background
`hive-c0re` runs as the unprivileged system user `hive-core`
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't
directly invoke `nixos-container` or act on a system
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
this gap.
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
a caller can't name the unit, so it can't steer the verb at another
service. **A privileged verb needs something bounding what it can act
on; when that isn't a namespace, it has to be a constant the caller
can't supply.**
### hive-priv
`hive-priv` is a minimal privileged helper that runs as root, socket-activated
at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the
`hive-core` user can connect). `hive-c0re` calls it via `priv_client`
for every operation that genuinely requires root.
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
known operations; there is no arbitrary command pass-through:
| Operation | What it runs |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
| `DaemonReload` | `systemctl daemon-reload` |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
**Strings that reach root-loaded config** — `WriteResourceLimits` and
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
file that systemd and the nixos-containers start script read, so hive-priv
refuses, never escapes, any value outside its shape. `memory_max` is
`infinity`, a decimal percentage or a decimal size with an optional
`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is
1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A
bind or credential path is absolute with no whitespace, control byte, `\`,
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
takes the closure the swarm built instead of building it. Root-owned sources
alone decide what root accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key.
- The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache` become
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
it, and `hive-priv` refuses an address that isn't `https`.
- The key and the pull token come from the secret store. The root unit
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
which `hive-c0re` writes for the probe, and it refuses a file in the root
directory that isn't a plain file owned by root or that others can read, and
the directory under the same rule.
- Nix verifies. For that one call `hive-priv` sets `substituters` and
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
A path without a signature by that key isn't stored; `hive-priv` answers
`not available` and logs the refusal at error level. `--max-jobs 0` means no
answer, whatever the cache sends, starts a build as root.
- A hive realises only paths whose signature checks. The request carries an
agent name and a store path, both validated, and no address, key or file; any
other field never reaches the verb.
What a compromised `hive-c0re` can do with this verb: ask for any store path
that has the shape of one and that the swarm cache signed, which fills the
store with paths the swarm vouches for. It can't make root accept a path from
a cache or under a key of its choosing, because it writes none of the places
those come from. The trust anchor is the cache's signing key and the store that
hands out its public key, as for every other credential.
**Container allowlist** — `hive-priv` validates every request against
an allowlist before any operation: the allowlist accepts only names
matching the agent-name convention (`h-` prefix, char-validated)
and rejects every other container name, the swarm's `hive-forge` and
`hive-matrix` included: those belong to the host that runs them, not to
a hive. `hive-gateway` is a host unit,
not a container, so it's not in this list — see `ReloadGatewayNginx`
above for how the hard-coded unit name scopes its access instead.
**Socket-activated** — systemd starts `hive-priv` on the first
incoming connection (`LISTEN_FDS=1`); it's not running between calls.
The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem
writes to only the paths `hive-priv` legitimately needs. The unit also sets
`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and
`ProtectKernelModules`: it changes no execution domain, schedules nothing in
real time, reads no kernel log, and loads no module. It sets none of
`RestrictNamespaces` (`nsenter` into a container needs `setns`),
`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or
`SystemCallFilter`; those need a watched deploy to judge.
### Privilege boundary summary
| Component | Runs as | Privilege needed for |
| ---------------------------- | -------------- | --------------------------------------------- |
| `hive-c0re` | `hive-core` | broker, HTTP dashboard |
| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes |
| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving |
- How in-container `nix build` stays apart from the credentials, and its
threat model: [nix builds](../../4-implementation/trust-boundary/nix-builds.md).
- How `hive-c0re` runs unprivileged and delegates root operations:
[privilege separation](../../4-implementation/trust-boundary/hive-c0re-privilege.md).