30a8f71bb5
boundary.md keeps the model, the design rule, why network isolation matters and host admin socket access; the hive-priv socket activation and the per-agent socket directory move to 4-implementation/trust-boundary. security.md keeps the agent trust model and the path-name rule; the nix builds mode with its threat model, and the hive-c0re privilege separation, move to their own pages there. Headings and text move as they were, with one level shaved off the heading depth. Links, code and nix pointers, and the docs index follow the moves.
131 lines
25 KiB
Markdown
131 lines
25 KiB
Markdown
# hive-c0re privilege separation
|
|
|
|
How `hive-c0re` runs unprivileged and delegates its root operations to
|
|
`hive-priv`, from the [security model](../../2-handbook/trust-boundary/security.md).
|
|
|
|
## Background
|
|
|
|
`hive-c0re` runs as the unprivileged system user `hive-core`
|
|
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't
|
|
directly invoke `nixos-container` or act on a system
|
|
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
|
|
this gap.
|
|
|
|
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
|
|
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
|
|
a caller can't name the unit, so it can't steer the verb at another
|
|
service. **A privileged verb needs something bounding what it can act
|
|
on; when that isn't a namespace, it has to be a constant the caller
|
|
can't supply.**
|
|
|
|
## hive-priv
|
|
|
|
`hive-priv` is a minimal privileged helper that runs as root, socket-activated
|
|
at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the
|
|
`hive-core` user can connect). `hive-c0re` calls it via `priv_client`
|
|
for every operation that genuinely requires root.
|
|
|
|
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
|
|
known operations; there is no arbitrary command pass-through:
|
|
|
|
| Operation | What it runs |
|
|
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
|
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
|
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
|
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one |
|
|
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
|
|
| `DestroyContainer` | `nixos-container destroy <name>` |
|
|
| `ListContainers` | `nixos-container list` |
|
|
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
|
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
|
|
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
|
|
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
|
|
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
|
| `DaemonReload` | `systemctl daemon-reload` |
|
|
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
|
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
|
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
|
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
|
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
|
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
|
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
|
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
|
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
|
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
|
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
|
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
|
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
|
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
|
|
|
**Strings that reach root-loaded config** — `WriteResourceLimits` and
|
|
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
|
|
file that systemd and the nixos-containers start script read, so hive-priv
|
|
refuses, never escapes, any value outside its shape. `memory_max` is
|
|
`infinity`, a decimal percentage or a decimal size with an optional
|
|
`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is
|
|
1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A
|
|
bind or credential path is absolute with no whitespace, control byte, `\`,
|
|
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
|
|
|
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
|
|
takes the closure the swarm built instead of building it. Root-owned sources
|
|
alone decide what root accepts:
|
|
|
|
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
|
each narinfo it serves; the private key never leaves it. The builder pushes
|
|
with a push token and holds no signing key.
|
|
- The address is the host's configuration:
|
|
`services.hyperhive.swarm.attic.domain` and `cache` become
|
|
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
|
|
it, and `hive-priv` refuses an address that isn't `https`.
|
|
- The key and the pull token come from the secret store. The root unit
|
|
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
|
|
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
|
|
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
|
|
which `hive-c0re` writes for the probe, and it refuses a file in the root
|
|
directory that isn't a plain file owned by root or that others can read, and
|
|
the directory under the same rule.
|
|
- Nix verifies. For that one call `hive-priv` sets `substituters` and
|
|
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
|
|
A path without a signature by that key isn't stored; `hive-priv` answers
|
|
`not available` and logs the refusal at error level. `--max-jobs 0` means no
|
|
answer, whatever the cache sends, starts a build as root.
|
|
- A hive realises only paths whose signature checks. The request carries an
|
|
agent name and a store path, both validated, and no address, key or file; any
|
|
other field never reaches the verb.
|
|
|
|
What a compromised `hive-c0re` can do with this verb: ask for any store path
|
|
that has the shape of one and that the swarm cache signed, which fills the
|
|
store with paths the swarm vouches for. It can't make root accept a path from
|
|
a cache or under a key of its choosing, because it writes none of the places
|
|
those come from. The trust anchor is the cache's signing key and the store that
|
|
hands out its public key, as for every other credential.
|
|
|
|
**Container allowlist** — `hive-priv` validates every request against
|
|
an allowlist before any operation: the allowlist accepts only names
|
|
matching the agent-name convention (`h-` prefix, char-validated)
|
|
and rejects every other container name, the swarm's `hive-forge` and
|
|
`hive-matrix` included: those belong to the host that runs them, not to
|
|
a hive. `hive-gateway` is a host unit,
|
|
not a container, so it's not in this list — see `ReloadGatewayNginx`
|
|
above for how the hard-coded unit name scopes its access instead.
|
|
|
|
**Socket-activated** — systemd starts `hive-priv` on the first
|
|
incoming connection (`LISTEN_FDS=1`); it's not running between calls.
|
|
The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem
|
|
writes to only the paths `hive-priv` legitimately needs. The unit also sets
|
|
`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and
|
|
`ProtectKernelModules`: it changes no execution domain, schedules nothing in
|
|
real time, reads no kernel log, and loads no module. It sets none of
|
|
`RestrictNamespaces` (`nsenter` into a container needs `setns`),
|
|
`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or
|
|
`SystemCallFilter`; those need a watched deploy to judge.
|
|
|
|
## Privilege boundary summary
|
|
|
|
| Component | Runs as | Privilege needed for |
|
|
| ---------------------------- | -------------- | --------------------------------------------- |
|
|
| `hive-c0re` | `hive-core` | broker, HTTP dashboard |
|
|
| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes |
|
|
| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving |
|