hive-c0re pulls closures through the nix daemon; hive-priv verifies the swarm signature before applying
This commit is contained in:
atlas
mara
parent
448b703995
commit
ecd4445680
1 changed files with 1257 additions and 584 deletions
|
|
@ -28,35 +28,36 @@ for every operation that genuinely requires root.
|
|||
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
|
||||
known operations; there is no arbitrary command pass-through:
|
||||
|
||||
| Operation | What it runs |
|
||||
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
||||
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
||||
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one |
|
||||
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
||||
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
||||
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
||||
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
||||
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
||||
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
||||
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
||||
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
||||
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
||||
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
||||
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
||||
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
||||
| Operation | What it runs |
|
||||
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
||||
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
||||
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
|
||||
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
|
||||
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its cache fragment is missing |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
|
||||
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
|
||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
|
||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
|
||||
| `DaemonReload` | `systemctl daemon-reload` |
|
||||
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
|
||||
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
||||
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
||||
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
||||
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
|
||||
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
|
||||
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
|
||||
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
|
||||
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
|
||||
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
|
||||
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
|
||||
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
|
||||
|
||||
**Strings that reach root-loaded config** — `WriteResourceLimits` and
|
||||
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
|
||||
|
|
@ -68,39 +69,61 @@ refuses, never escapes, any value outside its shape. `memory_max` is
|
|||
bind or credential path is absolute with no whitespace, control byte, `\`,
|
||||
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
||||
|
||||
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
|
||||
takes the closure the swarm built instead of building it. Root-owned sources
|
||||
alone decide what root accepts:
|
||||
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
|
||||
built through the host's nix daemon, and `hive-priv` applies it only when the
|
||||
swarm's key signed it. Pulling needs no root: `nix-store --realise` of a store
|
||||
path is a request to the daemon. Root-owned sources alone decide what the
|
||||
daemon accepts:
|
||||
|
||||
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
||||
each narinfo it serves; the private key never leaves it. The builder pushes
|
||||
with a push token and holds no signing key.
|
||||
- The address is the host's configuration:
|
||||
`services.hyperhive.swarm.attic.domain` and `cache` become
|
||||
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
|
||||
it, and `hive-priv` refuses an address that isn't `https`.
|
||||
- The key and the pull token come from the secret store. The root unit
|
||||
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
|
||||
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
|
||||
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
|
||||
which `hive-c0re` writes for the probe, and it refuses a file in the root
|
||||
directory that isn't a plain file owned by root or that others can read, and
|
||||
the directory under the same rule.
|
||||
- Nix verifies. For that one call `hive-priv` sets `substituters` and
|
||||
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
|
||||
A path without a signature by that key isn't stored; `hive-priv` answers
|
||||
`not available` and logs the refusal at error level. `--max-jobs 0` means no
|
||||
answer, whatever the cache sends, starts a build as root.
|
||||
- A hive realises only paths whose signature checks. The request carries an
|
||||
agent name and a store path, both validated, and no address, key or file; any
|
||||
other field never reaches the verb.
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`. The root unit
|
||||
`hive-priv-cache-facts` lists it as `extra-trusted-substituters` in a nix.conf
|
||||
fragment, `/run/hyperhive-nix/cache.conf`, that the host's nix.conf includes
|
||||
(`!include`, so a host whose unit hasn't run yet still starts).
|
||||
- The key and the pull token come from the secret store. The unit reads the
|
||||
hive's own copies under the hive's own store identity and writes the key to
|
||||
`/run/hyperhive-priv/attic/public-key` and the token to `netrc` there
|
||||
(directory `0700`, files `0600`, root), and the key into the fragment as
|
||||
`extra-trusted-public-keys` and the netrc's path as `netrc-file`. The fragment
|
||||
holds no secret. `hive-c0re` reads its cache address, and nothing else, from
|
||||
it; `hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
|
||||
writes for the probe.
|
||||
- The daemon enforces. `hive-core` is an unprivileged client: the daemon takes
|
||||
its `--option substituters` only for an address in `trusted-substituters`,
|
||||
ignores its `trusted-public-keys`, `require-sigs` and `netrc-file`, and
|
||||
refuses a path no trusted key signed. `--max-jobs 0` means no answer, whatever
|
||||
the cache sends, starts a build.
|
||||
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
|
||||
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
|
||||
`trusted-public-keys` set to the key in the root file, and refuses a path that
|
||||
is absent or that the key didn't sign. The daemon's trusted keys also include
|
||||
the host's other caches, so this check is the one that counts only the swarm's.
|
||||
- When the fragment is missing, `hive-c0re` asks `hive-priv` for `RefreshCacheFacts`,
|
||||
which starts the facts unit and waits up to 60 s, instead of waiting for the
|
||||
unit's timer. A host the swarm hasn't given facts to answers
|
||||
`this hive has no cache facts yet`.
|
||||
|
||||
What a compromised `hive-c0re` can do with this verb: ask for any store path
|
||||
that has the shape of one and that the swarm cache signed, which fills the
|
||||
store with paths the swarm vouches for. It can't make root accept a path from
|
||||
a cache or under a key of its choosing, because it writes none of the places
|
||||
those come from. The trust anchor is the cache's signing key and the store that
|
||||
hands out its public key, as for every other credential.
|
||||
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
|
||||
signed, which fills the store with paths the swarm vouches for, and then ask
|
||||
`hive-priv` to apply any such path to any agent. Before this change it could
|
||||
already do both through `RealiseClosure` and an apply; the former check bound an
|
||||
apply to the path `hive-priv` itself downloaded for that agent, and not to the
|
||||
agent the swarm built it for. It can't make root accept a path from a cache or
|
||||
under a key of its choosing, because it writes none of the places those come
|
||||
from. The trust anchor is the cache's signing key and the store that hands out
|
||||
its public key, as for every other credential.
|
||||
|
||||
**Mixed versions on one host.** `hive-priv` and `hive-c0re` restart separately.
|
||||
A new `hive-c0re` with an older `hive-priv`: the older one refuses an apply of a
|
||||
path it didn't download, so `hive-c0re` answers that refusal by sending the
|
||||
older `RealiseClosure`, which downloads and pins the path (already in the
|
||||
store), and applies again. A new `hive-priv` with an older `hive-c0re`: it still
|
||||
accepts `RealiseClosure`, answers `Ok` for a path already in the store with the
|
||||
swarm's signature and `not available` for one that isn't, and applies. Both
|
||||
shims go in the release after this one.
|
||||
|
||||
**Container allowlist** — `hive-priv` validates every request against
|
||||
an allowlist before any operation: the allowlist accepts only names
|
||||
|
|
|
|||
Loading…
Reference in a new issue