ca51bb9d54
HIVE_PRIV_EXE, set by the module to the /run/hive-rootfs-check link, is checked at start and used for the drop-in's ExecStartPre. Without it the running binary's path is used as before, and an unlinked one gets an error that names the cause.
183 lines
39 KiB
Markdown
183 lines
39 KiB
Markdown
# hive-c0re privilege separation
|
|
|
|
How `hive-c0re` runs unprivileged and delegates its root operations to
|
|
`hive-priv`, from the [security model](../../2-handbook/agents/security.md).
|
|
|
|
## Background
|
|
|
|
`hive-c0re` runs as the unprivileged system user `hive-core`
|
|
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't
|
|
directly invoke `nixos-container` or act on a system
|
|
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
|
|
this gap.
|
|
|
|
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
|
|
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
|
|
a caller can't name the unit, so it can't steer the verb at another
|
|
service. **A privileged verb needs something bounding what it can act
|
|
on; when that isn't a namespace, it has to be a constant the caller
|
|
can't supply.**
|
|
|
|
## hive-priv
|
|
|
|
`hive-priv` is a minimal privileged helper that runs as root, socket-activated
|
|
at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the
|
|
`hive-core` user can connect). `hive-c0re` calls it via `priv_client`
|
|
for every operation that genuinely requires root.
|
|
|
|
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
|
|
known operations; there is no arbitrary command pass-through:
|
|
|
|
| Operation | What it runs |
|
|
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
|
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
|
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re logs the lines under the failed start's `nixos-container` tag. |
|
|
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv first runs `nix store copy-sigs --store daemon` for that path from the cache named in `/run/hyperhive-nix/cache.conf`, with the netrc beside it, which imports the cache's signature for a path that was already in the store (on a failure it logs a warning and the check goes on); it then runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
|
|
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
|
|
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when the fragment is missing or lacks a setting |
|
|
| `DestroyContainer` | `nixos-container destroy <name>` |
|
|
| `ListContainers` | `nixos-container list` |
|
|
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
|
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
|
|
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
|
|
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`MemoryHigh=`/`MemorySwapMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and, when the binary it runs is a store path, a GC root on that build |
|
|
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
|
| `DaemonReload` | `systemctl daemon-reload` |
|
|
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
|
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
|
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
|
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
|
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
|
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
|
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
|
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
|
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
|
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
|
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
|
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
|
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
|
|
|
**Strings that reach root-loaded config** — `WriteResourceLimits` and
|
|
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
|
|
file that systemd and the nixos-containers start script read, so hive-priv
|
|
refuses, never escapes, any value outside its shape. `memory_max` is
|
|
`infinity`, a decimal percentage or a decimal size with an optional
|
|
`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is
|
|
1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A
|
|
bind or credential path is absolute with no whitespace, control byte, `\`,
|
|
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
|
|
|
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
|
|
built with `nix-store --realise <path> --max-jobs 0`, and `hive-priv` applies it
|
|
only when the swarm's key signed it. Pulling needs no root. The command is a
|
|
request to the host's nix daemon, which substitutes each path of the closure
|
|
from the swarm cache or from its other substituters (`cache.nixos.org`),
|
|
whichever has it, and stores a path from an unprivileged client only if a key
|
|
in its `trusted-public-keys` signed it. Root-owned sources alone decide what
|
|
the daemon accepts:
|
|
|
|
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
|
each narinfo it serves; the private key never leaves it. The builder pushes
|
|
with a push token and holds no signing key.
|
|
- The address is the host's configuration:
|
|
`services.hyperhive.swarm.attic.domain` and `cache`. The daemon holds it as a
|
|
substituter; `hive-c0re` has no address and names no source.
|
|
- The key and the pull token come from the secret store. The root unit
|
|
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
|
|
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
|
|
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
|
|
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
|
|
address), `extra-trusted-public-keys` (the key), `netrc-file` and
|
|
`ssl-cert-file`, which the host's nix.conf includes (`!include`, so a host
|
|
whose unit hasn't run yet still starts). The fragment holds no secret.
|
|
`ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`): the host's
|
|
own CA bundle plus the swarm's root, so nix clients on the host trust the
|
|
cache's certificate while the host's system trust stays without the root.
|
|
`hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
|
|
writes for the probe.
|
|
- Only root reads the token. The netrc the fragment names,
|
|
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
|
|
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
|
|
default `/etc/nix/netrc`), because `netrc-file` takes one path and would
|
|
replace the host's. The unit merges again on every run, every ten minutes.
|
|
The daemon is root and reads the file itself, so `hive-core` has no copy of
|
|
the token for the pull. The probe's copy under `/run/hyperhive/attic/` is
|
|
separate.
|
|
- The daemon enforces. It refuses a path from an unprivileged client unless a
|
|
trusted key signed it; a client can't lower `require-sigs` or add keys or
|
|
substituters. `--max-jobs 0` forbids a build, so a path no substituter has
|
|
fails.
|
|
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
|
|
toplevel, it first imports the cache's signature for that path with
|
|
`nix store copy-sigs` (a path the host built carries none), then runs
|
|
`nix store verify --no-contents --sigs-needed 1` with
|
|
`trusted-public-keys` set to the key in the root file, and refuses a path that
|
|
is absent or that the key didn't sign. The daemon's trusted keys also include
|
|
the host's other caches, so this check is the one that counts only the swarm's.
|
|
- When the fragment is missing or lacks a setting, `hive-c0re` asks `hive-priv` for
|
|
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
|
|
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
|
|
`this hive has no cache facts yet`.
|
|
|
|
**What an agent with daemon access can fetch.** Agent containers on
|
|
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). The
|
|
daemon holds the swarm cache as a substituter with the pull token, so any client
|
|
of it can have a swarm closure substituted, as it can have a path from
|
|
`cache.nixos.org`. That gives an agent nothing it can't read already: a closure
|
|
is world-readable in `/nix/store` once it's on the host, it holds no secret
|
|
(every secret comes from the store at run time), and the agent never sees the
|
|
token, which stays in root-only `/run/hyperhive-nix/netrc`. What the agent gains
|
|
is the chance to pull another hive's closure that isn't on this host yet. The
|
|
daemon takes a path the swarm key signed from any source, and only the swarm can
|
|
produce one.
|
|
|
|
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
|
|
signed, which fills the store with paths the swarm vouches for, and then ask
|
|
`hive-priv` to apply any such path to any agent. Before this change it could
|
|
already do both through `RealiseClosure` and an apply; the former check bound an
|
|
apply to the path `hive-priv` itself downloaded for that agent, and not to the
|
|
agent the swarm built it for. It can't make root accept a path from a cache or
|
|
under a key of its choosing, because it writes none of the places those come
|
|
from. The trust anchor is the cache's signing key and the store that hands out
|
|
its public key, as for every other credential.
|
|
|
|
**Mixed versions on one host.** `hive-priv` and `hive-c0re` restart separately.
|
|
A new `hive-c0re` with an older `hive-priv`: the older one refuses an apply of a
|
|
path it didn't download, so `hive-c0re` answers that refusal by sending the
|
|
older `RealiseClosure`, which downloads and pins the path (already in the
|
|
store), and applies again. A new `hive-priv` with an older `hive-c0re`: it still
|
|
accepts `RealiseClosure`, answers `Ok` for a path already in the store with the
|
|
swarm's signature and `not available` for one that isn't, and applies. Both
|
|
shims go in the release after this one.
|
|
|
|
**Container allowlist** — `hive-priv` validates every request against
|
|
an allowlist before any operation: the allowlist accepts only names
|
|
matching the agent-name convention (`h-` prefix, char-validated)
|
|
and rejects every other container name, the swarm's `hive-forge` and
|
|
`hive-matrix` included: those belong to the host that runs them, not to
|
|
a hive. `hive-gateway` is a host unit,
|
|
not a container, so it's not in this list — see `ReloadGatewayNginx`
|
|
above for how the hard-coded unit name scopes its access instead.
|
|
|
|
**Socket-activated** — systemd starts `hive-priv` on the first
|
|
incoming connection (`LISTEN_FDS=1`); it's not running between calls.
|
|
The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem
|
|
writes to only the paths `hive-priv` legitimately needs. A `ReadWritePaths`
|
|
entry that's missing fails the unit with `226/NAMESPACE`, so tmpfiles rules
|
|
create `/etc/nixos-containers`, `/var/lib/nixos-containers` and
|
|
`/run/hive-agent` on a hive that has no declared container yet. The unit also sets
|
|
`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and
|
|
`ProtectKernelModules`: it changes no execution domain, schedules nothing in
|
|
real time, reads no kernel log, and loads no module. It sets none of
|
|
`RestrictNamespaces` (`nsenter` into a container needs `setns`),
|
|
`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or
|
|
`SystemCallFilter`; those need a watched deploy to judge.
|
|
|
|
## Privilege boundary summary
|
|
|
|
| Component | Runs as | Privilege needed for |
|
|
| ---------------------------- | -------------- | --------------------------------------------- |
|
|
| `hive-c0re` | `hive-core` | broker, HTTP dashboard |
|
|
| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes |
|
|
| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving |
|