0e84fe8bc4
The handbook's sections were agent-lifecycle, integrations, swarm, tools, trust-boundary and web-ui, which do not follow how the system is built, so a reader could not tell which layer a page is about. Its pages now sit in swarm/, hives/ and agents/, with the accounts an agent can hold outside the system in external-accounts/. - swarm/: forge, matrix and the knowledge repo join the swarm pages, and so does the swarmctl reference. - hives/: the dashboard, hivectl and its reference, and the operator/agent boundary, with a README that takes over the old web-ui and tools pages. - agents/: the roster, persistence, config changes, the agent's own web page (was web-ui/agent) and the security model, with a README. - external-accounts/: GitHub and GitLab. Pages are moved whole; none is split yet. Every reference follows: relative links in the docs, `docs/2-handbook/...` paths in code comments, the nix checks that regenerate the CLI references and the ignore list, and docs/README.md, whose handbook section is now by layer.
131 lines
25 KiB
Markdown
131 lines
25 KiB
Markdown
# hive-c0re privilege separation
|
|
|
|
How `hive-c0re` runs unprivileged and delegates its root operations to
|
|
`hive-priv`, from the [security model](../../2-handbook/agents/security.md).
|
|
|
|
## Background
|
|
|
|
`hive-c0re` runs as the unprivileged system user `hive-core`
|
|
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It can't
|
|
directly invoke `nixos-container` or act on a system
|
|
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
|
|
this gap.
|
|
|
|
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
|
|
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
|
|
a caller can't name the unit, so it can't steer the verb at another
|
|
service. **A privileged verb needs something bounding what it can act
|
|
on; when that isn't a namespace, it has to be a constant the caller
|
|
can't supply.**
|
|
|
|
## hive-priv
|
|
|
|
`hive-priv` is a minimal privileged helper that runs as root, socket-activated
|
|
at `/run/hive/priv.sock` (mode `0660`, group `hive-core` — only the
|
|
`hive-core` user can connect). `hive-c0re` calls it via `priv_client`
|
|
for every operation that genuinely requires root.
|
|
|
|
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
|
|
known operations; there is no arbitrary command pass-through:
|
|
|
|
| Operation | What it runs |
|
|
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
|
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
|
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
|
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one |
|
|
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
|
|
| `DestroyContainer` | `nixos-container destroy <name>` |
|
|
| `ListContainers` | `nixos-container list` |
|
|
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
|
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
|
|
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
|
|
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
|
|
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
|
| `DaemonReload` | `systemctl daemon-reload` |
|
|
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
|
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
|
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
|
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
|
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
|
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
|
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
|
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
|
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
|
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
|
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
|
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
|
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
|
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
|
|
|
**Strings that reach root-loaded config** — `WriteResourceLimits` and
|
|
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
|
|
file that systemd and the nixos-containers start script read, so hive-priv
|
|
refuses, never escapes, any value outside its shape. `memory_max` is
|
|
`infinity`, a decimal percentage or a decimal size with an optional
|
|
`B`/`K`/`M`/`G`/`T`/`P`/`E`; `cpu_quota` is a decimal percentage; `bridge` is
|
|
1-15 ASCII letters, digits, `-`, `_` or `.`; `gateway_ip` is an IPv4 literal. A
|
|
bind or credential path is absolute with no whitespace, control byte, `\`,
|
|
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
|
|
|
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
|
|
takes the closure the swarm built instead of building it. Root-owned sources
|
|
alone decide what root accepts:
|
|
|
|
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
|
each narinfo it serves; the private key never leaves it. The builder pushes
|
|
with a push token and holds no signing key.
|
|
- The address is the host's configuration:
|
|
`services.hyperhive.swarm.attic.domain` and `cache` become
|
|
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
|
|
it, and `hive-priv` refuses an address that isn't `https`.
|
|
- The key and the pull token come from the secret store. The root unit
|
|
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
|
|
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
|
|
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
|
|
which `hive-c0re` writes for the probe, and it refuses a file in the root
|
|
directory that isn't a plain file owned by root or that others can read, and
|
|
the directory under the same rule.
|
|
- Nix verifies. For that one call `hive-priv` sets `substituters` and
|
|
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
|
|
A path without a signature by that key isn't stored; `hive-priv` answers
|
|
`not available` and logs the refusal at error level. `--max-jobs 0` means no
|
|
answer, whatever the cache sends, starts a build as root.
|
|
- A hive realises only paths whose signature checks. The request carries an
|
|
agent name and a store path, both validated, and no address, key or file; any
|
|
other field never reaches the verb.
|
|
|
|
What a compromised `hive-c0re` can do with this verb: ask for any store path
|
|
that has the shape of one and that the swarm cache signed, which fills the
|
|
store with paths the swarm vouches for. It can't make root accept a path from
|
|
a cache or under a key of its choosing, because it writes none of the places
|
|
those come from. The trust anchor is the cache's signing key and the store that
|
|
hands out its public key, as for every other credential.
|
|
|
|
**Container allowlist** — `hive-priv` validates every request against
|
|
an allowlist before any operation: the allowlist accepts only names
|
|
matching the agent-name convention (`h-` prefix, char-validated)
|
|
and rejects every other container name, the swarm's `hive-forge` and
|
|
`hive-matrix` included: those belong to the host that runs them, not to
|
|
a hive. `hive-gateway` is a host unit,
|
|
not a container, so it's not in this list — see `ReloadGatewayNginx`
|
|
above for how the hard-coded unit name scopes its access instead.
|
|
|
|
**Socket-activated** — systemd starts `hive-priv` on the first
|
|
incoming connection (`LISTEN_FDS=1`); it's not running between calls.
|
|
The `ProtectSystem=strict` + `ReadWritePaths` sandbox limits filesystem
|
|
writes to only the paths `hive-priv` legitimately needs. The unit also sets
|
|
`LockPersonality`, `RestrictRealtime`, `ProtectKernelLogs` and
|
|
`ProtectKernelModules`: it changes no execution domain, schedules nothing in
|
|
real time, reads no kernel log, and loads no module. It sets none of
|
|
`RestrictNamespaces` (`nsenter` into a container needs `setns`),
|
|
`RestrictAddressFamilies`, `PrivateDevices`, `CapabilityBoundingSet` or
|
|
`SystemCallFilter`; those need a watched deploy to judge.
|
|
|
|
## Privilege boundary summary
|
|
|
|
| Component | Runs as | Privilege needed for |
|
|
| ---------------------------- | -------------- | --------------------------------------------- |
|
|
| `hive-c0re` | `hive-core` | broker, HTTP dashboard |
|
|
| `hive-priv` | `root` | container lifecycle, bind mounts, cred writes |
|
|
| `hive-ag3nt` (per-container) | per-agent user | turn execution, MCP serving |
|