177 lines
39 KiB
Markdown
177 lines
39 KiB
Markdown
# hive-c0re privilege separation
|
|
|
|
How `hive-c0re` runs unprivileged and delegates its root operations to
|
|
`hive-priv`, from the [security model](../../2-handbook/agents/security.md).
|
|
|
|
## Background
|
|
|
|
`hive-c0re` runs as the unprivileged system user `hive-core`
|
|
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't
|
|
directly invoke `nixos-container` or act on a system
|
|
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
|
|
this gap.
|
|
|
|
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
|
|
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
|
|
a caller can't name the unit, so it can't steer the verb at another
|
|
service. **A privileged verb needs something bounding what it can act
|
|
on; when that isn't a namespace, it has to be a constant the caller
|
|
can't supply.**
|
|
|
|
## hive-priv
|
|
|
|
`hive-priv` is a minimal privileged helper that runs as root, socket-activated
|
|
at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the
|
|
`hive-core` user can connect). `hive-c0re` calls it via `priv_client`
|
|
for every operation that genuinely requires root.
|
|
|
|
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
|
|
known operations; there is no arbitrary command pass-through:
|
|
|
|
| Operation | What it runs |
|
|
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
|
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
|
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
|
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv first runs `nix store copy-sigs --store daemon` for that path from the cache named in `/run/hyperhive-nix/cache.conf`, with the netrc beside it, which imports the cache's signature for a path that was already in the store (on a failure it logs a warning and the check goes on); it then runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
|
|
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
|
|
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when the fragment is missing or lacks a setting |
|
|
| `DestroyContainer` | `nixos-container destroy <name>` |
|
|
| `ListContainers` | `nixos-container list` |
|
|
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
|
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
|
|
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
|
|
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
|
|
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
|
| `DaemonReload` | `systemctl daemon-reload` |
|
|
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
|
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
|
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
|
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
|
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
|
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
|
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
|
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
|
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
|
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
|
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
|
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
|
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
|
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
|
|
|
**Strings that reach root-loaded config** — `WriteResourceLimits` and
|
|
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
|
|
file that systemd and the nixos-containers start script read, so hive-priv
|
|
refuses, never escapes, any value outside its shape. `memory_max` is
|
|
`infinity`, a decimal percentage or a decimal size with an optional
|
|
`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is
|
|
1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A
|
|
bind or credential path is absolute with no whitespace, control byte, `\`,
|
|
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
|
|
|
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
|
|
built with `nix-store --realise <path> --max-jobs 0`, and `hive-priv` applies it
|
|
only when the swarm's key signed it. Pulling needs no root. The command is a
|
|
request to the host's nix daemon, which substitutes each path of the closure
|
|
from the swarm cache or from its other substituters (`cache.nixos.org`),
|
|
whichever has it, and stores a path from an unprivileged client only if a key
|
|
in its `trusted-public-keys` signed it. Root-owned sources alone decide what
|
|
the daemon accepts:
|
|
|
|
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
|
each narinfo it serves; the private key never leaves it. The builder pushes
|
|
with a push token and holds no signing key.
|
|
- The address is the host's configuration:
|
|
`services.hyperhive.swarm.attic.domain` and `cache`. The daemon holds it as a
|
|
substituter; `hive-c0re` has no address and names no source.
|
|
- The key and the pull token come from the secret store. The root unit
|
|
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
|
|
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
|
|
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
|
|
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
|
|
address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the
|
|
host's nix.conf includes (`!include`, so a host whose unit hasn't run yet
|
|
still starts). The fragment holds no secret. `hive-priv` reads nothing from
|
|
`/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
|
|
- Only root reads the token. The netrc the fragment names,
|
|
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
|
|
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
|
|
default `/etc/nix/netrc`), because `netrc-file` takes one path and would
|
|
replace the host's. The unit merges again on every run, every ten minutes.
|
|
The daemon is root and reads the file itself, so `hive-core` has no copy of
|
|
the token for the pull. The probe's copy under `/run/hyperhive/attic/` is
|
|
separate.
|
|
- The daemon enforces. It refuses a path from an unprivileged client unless a
|
|
trusted key signed it; a client can't lower `require-sigs` or add keys or
|
|
substituters. `--max-jobs 0` forbids a build, so a path no substituter has
|
|
fails.
|
|
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
|
|
toplevel, it first imports the cache's signature for that path with
|
|
`nix store copy-sigs` (a path the host built carries none), then runs
|
|
`nix store verify --no-contents --sigs-needed 1` with
|
|
`trusted-public-keys` set to the key in the root file, and refuses a path that
|
|
is absent or that the key didn't sign. The daemon's trusted keys also include
|
|
the host's other caches, so this check is the one that counts only the swarm's.
|
|
- When the fragment is missing or lacks a setting, `hive-c0re` asks `hive-priv` for
|
|
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
|
|
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
|
|
`this hive has no cache facts yet`.
|
|
|
|
**What an agent with daemon access can fetch.** Agent containers on
|
|
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). The
|
|
daemon holds the swarm cache as a substituter with the pull token, so any client
|
|
of it can have a swarm closure substituted, as it can have a path from
|
|
`cache.nixos.org`. That gives an agent nothing it can't read already: a closure
|
|
is world-readable in `/nix/store` once it's on the host, it holds no secret
|
|
(every secret comes from the store at run time), and the agent never sees the
|
|
token, which stays in root-only `/run/hyperhive-nix/netrc`. What the agent gains
|
|
is the chance to pull another hive's closure that isn't on this host yet. The
|
|
daemon takes a path the swarm key signed from any source, and only the swarm can
|
|
produce one.
|
|
|
|
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
|
|
signed, which fills the store with paths the swarm vouches for, and then ask
|
|
`hive-priv` to apply any such path to any agent. Before this change it could
|
|
already do both through `RealiseClosure` and an apply; the former check bound an
|
|
apply to the path `hive-priv` itself downloaded for that agent, and not to the
|
|
agent the swarm built it for. It can't make root accept a path from a cache or
|
|
under a key of its choosing, because it writes none of the places those come
|
|
from. The trust anchor is the cache's signing key and the store that hands out
|
|
its public key, as for every other credential.
|
|
|
|
**Mixed versions on one host.** `hive-priv` and `hive-c0re` restart separately.
|
|
A new `hive-c0re` with an older `hive-priv`: the older one refuses an apply of a
|
|
path it didn't download, so `hive-c0re` answers that refusal by sending the
|
|
older `RealiseClosure`, which downloads and pins the path (already in the
|
|
store), and applies again. A new `hive-priv` with an older `hive-c0re`: it still
|
|
accepts `RealiseClosure`, answers `Ok` for a path already in the store with the
|
|
swarm's signature and `not available` for one that isn't, and applies. Both
|
|
shims go in the release after this one.
|
|
|
|
**Container allowlist** — `hive-priv` validates every request against
|
|
an allowlist before any operation: the allowlist accepts only names
|
|
matching the agent-name convention (`h-` prefix, char-validated)
|
|
and rejects every other container name, the swarm's `hive-forge` and
|
|
`hive-matrix` included: those belong to the host that runs them, not to
|
|
a hive. `hive-gateway` is a host unit,
|
|
not a container, so it's not in this list — see `ReloadGatewayNginx`
|
|
above for how the hard-coded unit name scopes its access instead.
|
|
|
|
**Socket-activated** — systemd starts `hive-priv` on the first
|
|
incoming connection (`LISTEN_FDS=1`); it's not running between calls.
|
|
The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem
|
|
writes to only the paths `hive-priv` legitimately needs. The unit also sets
|
|
`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and
|
|
`ProtectKernelModules`: it changes no execution domain, schedules nothing in
|
|
real time, reads no kernel log, and loads no module. It sets none of
|
|
`RestrictNamespaces` (`nsenter` into a container needs `setns`),
|
|
`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or
|
|
`SystemCallFilter`; those need a watched deploy to judge.
|
|
|
|
## Privilege boundary summary
|
|
|
|
| Component | Runs as | Privilege needed for |
|
|
| ---------------------------- | -------------- | --------------------------------------------- |
|
|
| `hive-c0re` | `hive-core` | broker, HTTP dashboard |
|
|
| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes |
|
|
| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving |
|