hyperhive/docs/4-implementation/trust-boundary/hive-c0re-privilege.md

hive-c0re privilege separation

How hive-c0re runs unprivileged and delegates its root operations to hive-priv, from the security model.

Background

hive-c0re runs as the unprivileged system user hive-core (/var/lib/hyperhive owned by hive-core:hive-core). It can't directly invoke nixos-container or act on a system unit (systemctl reload nginx) — those require root. hive-priv fills this gap.

⚠️ ReloadGatewayNginx acts on a host unit, so nothing implicitly scopes it. Its containment is the unit name hard-coded in hive-priv: a caller can't name the unit, so it can't steer the verb at another service. A privileged verb needs something bounding what it can act on; when that isn't a namespace, it has to be a constant the caller can't supply.

hive-priv

hive-priv is a minimal privileged helper that runs as root, socket-activated at /run/hive/priv.sock (mode 0660, group hive-core — only the hive-core user can connect). hive-c0re calls it via priv_client for every operation that genuinely requires root.

Narrow interface — PrivRequest variants map 1:1 to specific known operations; there is no arbitrary command pass-through:

Operation	What it runs
StartContainer / StopContainer	nixos-container start/stop <name>
KillContainer	machinectl kill <machine> --signal=SIGKILL (nixos-container has no kill verb)
StartFailureLog	journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps <t> to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves.
CreateContainer / UpdateContainer	nixos-container create <name> --system-path <toplevel>; for an update, nix-env -p <profile> --set <toplevel> and a reload of the unit. <toplevel> must already be in the store with a signature by the swarm cache's key: hive-priv first runs nix store copy-sigs --store daemon for that path from the cache named in /run/hyperhive-nix/cache.conf, with the netrc beside it, which imports the cache's signature for a path that was already in the store (on a failure it logs a warning and the check goes on); it then runs nix store verify --no-contents --sigs-needed 1 with trusted-public-keys set to the key in root-owned /run/hyperhive-priv/attic/public-key, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one
RealiseClosure	Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers Ok when the path is in the store and signed by the swarm key (what CreateContainer needs), else not available: <reason>. The path must be /nix/store/<32 chars>-<name> and the agent name valid
RefreshCacheFacts	systemctl start hive-priv-cache-facts.service, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when the fragment is missing or lacks a setting
DestroyContainer	nixos-container destroy <name>
ListContainers	nixos-container list
ReloadGatewayNginx	systemctl reload/start/reset-failed nginx (host unit; the unit name is hard-coded, not a parameter)
WriteNspawnFlags	write /etc/nixos-containers/<container>.conf (bind-mount list, network isolation vars, and --ephemeral after the root's salvage). hive-priv first copies a stopped container's root to the agent's work/, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets private_users it also adds --private-users=pick, mounts every bind owneridmap, and once chowns the agent's state, harness, work and claude dirs (never config, following no link) to the data owner
EnsureAgentSocketDir	mkdirat /run/hive-agent/<name> 0751 root:root under an O_NOFOLLOW parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets private_users it then chowns the dir to the data owner
WriteResourceLimits	write CPUQuota=/MemoryMax=/CPUWeight=/IOWeight= systemd drop-in for agent container, with an ExecStartPre= that runs hive-priv check-rootfs <container> (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names
RemoveServiceDropin	remove container@<name>.service.d/ drop-in on destroy, and the GC root of the hive-priv build it named
DaemonReload	systemctl daemon-reload
ControlInfraContainer	systemctl <action> nginx.service for the gateway — the InfraContainer enum is the allowlist, and serde rejects unknown names at the wire boundary (hive-c0re, the forge and the matrix have no variant, so no request can name them)
SyncAgentTmpfiles	unlink /etc/tmpfiles.d/hyperhive-agents.conf and return Ok
SetAgentPaused	create / remove the <state>/<name>/harness/paused marker that parks an agent's turn loop
EnsureAgentSubvolume	btrfs subvolume create <state>/<name> for a new agent — no-op when the path exists or the filesystem isn't btrfs
UpgradeAgentSubvolume	migrate an existing plain state dir into a subvolume: create, cp -a --reflink=auto, atomic swap. Operator opt-in, and the caller stops the agent first
DeleteAgentSubvolume	btrfs subvolume delete <state>/<name> — purge path only, never a plain destroy
EnsureBtrfsQuota	btrfs quota enable <AGENT_STATE_ROOT>. Operator opt-in — enabling forces a full rescan
ReadSubvolumeUsage	btrfs qgroup show -f --raw <state>/<name>
SetSubvolumeQuota	btrfs qgroup limit <bytes|none> <state>/<name>
MeasureAgentDisk	du -s -x -b -c -P over <state>/<name>/state, <state>/<name>/work and /var/lib/nixos-containers/h-<name>, killed after 10 min. hive-priv derives the paths from the name; the request carries none
SnapshotAgentSubvolume	btrfs subvolume snapshot -r <agent_root> <snapshot_path> — the frozen point-in-time copy a migration streams from
DeleteAgentSnapshot	btrfs subvolume delete <snapshot_path>
SendAgentSnapshotToFile	btrfs send [-p <parent>] <snapshot> into a bare filename under MIGRATE_STAGING_ROOT; refuses to overwrite an existing export
SendAgentSnapshotToFd	btrfs send [-p <parent>] <snapshot> into a file descriptor passed with the request (SCM_RIGHTS). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation

Strings that reach root-loaded config — WriteResourceLimits and WriteNspawnFlags copy request strings into a systemd drop-in and an env file that systemd and the nixos-containers start script read, so hive-priv refuses, never escapes, any value outside its shape. memory_max is infinity, a decimal percentage or a decimal size with an optional B/K/M/G/T/P/E; cpu_quota is a decimal percentage; bridge is 1-15 ASCII letters, digits, -, _ or .; gateway_ip is an IPv4 literal. A bind or credential path is absolute with no whitespace, control byte, \, $, backtick, " or :. The size limits of a value stay with systemd.

Closures from the swarm cache — hive-c0re pulls the closure the swarm built with nix-store --realise <path> --max-jobs 0, and hive-priv applies it only when the swarm's key signed it. Pulling needs no root. The command is a request to the host's nix daemon, which substitutes each path of the closure from the swarm cache or from its other substituters (cache.nixos.org), whichever has it, and stores a path from an unprivileged client only if a key in its trusted-public-keys signed it. Root-owned sources alone decide what the daemon accepts:

• The cache signs. The swarm's cache (attic) holds the signing key and signs each narinfo it serves; the private key never leaves it. The builder pushes with a push token and holds no signing key.
• The address is the host's configuration: services.hyperhive.swarm.attic.domain and cache. The daemon holds it as a substituter; hive-c0re has no address and names no source.
• The key and the pull token come from the secret store. The root unit hive-priv-cache-facts reads the hive's own copies under the hive's own store identity and writes the key to /run/hyperhive-priv/attic/public-key (directory 0700, file 0600, root). It also writes a nix.conf fragment, /run/hyperhive-nix/cache.conf (0644), with extra-substituters (the address), extra-trusted-public-keys (the key) and netrc-file, which the host's nix.conf includes (!include, so a host whose unit hasn't run yet still starts). The fragment holds no secret. hive-priv reads nothing from /run/hyperhive/attic/, which hive-c0re writes for the probe.
• Only root reads the token. The netrc the fragment names, /run/hyperhive-nix/netrc (root:root 0600), holds the token's entry plus the entries of the host's own netrc (nix.settings.netrc-file, else nix's default /etc/nix/netrc), because netrc-file takes one path and would replace the host's. The unit merges again on every run, every ten minutes. The daemon is root and reads the file itself, so hive-core has no copy of the token for the pull. The probe's copy under /run/hyperhive/attic/ is separate.
• The daemon enforces. It refuses a path from an unprivileged client unless a trusted key signed it; a client can't lower require-sigs or add keys or substituters. --max-jobs 0 forbids a build, so a path no substituter has fails.
• hive-priv verifies. Before CreateContainer or UpdateContainer applies a toplevel, it first imports the cache's signature for that path with nix store copy-sigs (a path the host built carries none), then runs nix store verify --no-contents --sigs-needed 1 with trusted-public-keys set to the key in the root file, and refuses a path that is absent or that the key didn't sign. The daemon's trusted keys also include the host's other caches, so this check is the one that counts only the swarm's.
• When the fragment is missing or lacks a setting, hive-c0re asks hive-priv for RefreshCacheFacts, which starts the facts unit and waits up to 60 s, instead of waiting for the unit's timer. A host the swarm hasn't given facts to answers this hive has no cache facts yet.

What an agent with daemon access can fetch. Agent containers on nixBuilds = host reach the same daemon (nix-builds.md). The daemon holds the swarm cache as a substituter with the pull token, so any client of it can have a swarm closure substituted, as it can have a path from cache.nixos.org. That gives an agent nothing it can't read already: a closure is world-readable in /nix/store once it's on the host, it holds no secret (every secret comes from the store at run time), and the agent never sees the token, which stays in root-only /run/hyperhive-nix/netrc. What the agent gains is the chance to pull another hive's closure that isn't on this host yet. The daemon takes a path the swarm key signed from any source, and only the swarm can produce one.

What a compromised hive-c0re can do: pull any store path that the swarm cache signed, which fills the store with paths the swarm vouches for, and then ask hive-priv to apply any such path to any agent. Before this change it could already do both through RealiseClosure and an apply; the former check bound an apply to the path hive-priv itself downloaded for that agent, and not to the agent the swarm built it for. It can't make root accept a path from a cache or under a key of its choosing, because it writes none of the places those come from. The trust anchor is the cache's signing key and the store that hands out its public key, as for every other credential.

Mixed versions on one host. hive-priv and hive-c0re restart separately. A new hive-c0re with an older hive-priv: the older one refuses an apply of a path it didn't download, so hive-c0re answers that refusal by sending the older RealiseClosure, which downloads and pins the path (already in the store), and applies again. A new hive-priv with an older hive-c0re: it still accepts RealiseClosure, answers Ok for a path already in the store with the swarm's signature and not available for one that isn't, and applies. Both shims go in the release after this one.

Container allowlist — hive-priv validates every request against an allowlist before any operation: the allowlist accepts only names matching the agent-name convention (h- prefix, char-validated) and rejects every other container name, the swarm's hive-forge and hive-matrix included: those belong to the host that runs them, not to a hive. hive-gateway is a host unit, not a container, so it's not in this list — see ReloadGatewayNginx above for how the hard-coded unit name scopes its access instead.

Socket-activated — systemd starts hive-priv on the first incoming connection (LISTEN_FDS=1); it's not running between calls. The ProtectSystem=strict + ReadWritePaths sandbox limits filesystem writes to only the paths hive-priv legitimately needs. The unit also sets LockPersonality, RestrictRealtime, ProtectKernelLogs and ProtectKernelModules: it changes no execution domain, schedules nothing in real time, reads no kernel log, and loads no module. It sets none of RestrictNamespaces (nsenter into a container needs setns), RestrictAddressFamilies, PrivateDevices, CapabilityBoundingSet or SystemCallFilter; those need a watched deploy to judge.

Privilege boundary summary

Component	Runs as	Privilege needed for
hive-c0re	hive-core	broker, HTTP dashboard
hive-priv	root	container lifecycle, bind mounts, cred writes
hive-ag3nt (per-container)	per-agent user	turn execution, MCP serving