mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re pulls closures through the nix daemon; hive-priv verifies the swarm signature before applying

Browse source
This commit is contained in:
atlas 2026-11-07 13:20:55 +01:00 • committed by mara
commit ecd4445680
5 changed files with 1257 additions and 584 deletions
Download patch file Download diff file Expand all files Collapse all files

10
docs/2-handbook/agents/config-changes.md
View file

@ -154,11 +154,11 @@ the swarm builder built the agent from the updated repo (the
`MetaSync`; its `Create` reads the record once.
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
the agent's record in the swarm's `agent-closures` bucket (the controller
writes it from the builder's reply) and have `hive-priv` download the closure
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
`CreateContainer`), which accepts only the path its last `RealiseClosure` for
that agent returned. On the wire the
writes it from the builder's reply) and pull the closure it names from the
swarm cache through the host's nix daemon. `Swap` and `Create` then give
`hive-priv` the store path (`toplevel` on `UpdateContainer` and
`CreateContainer`), which accepts only a path in the store that carries the
swarm cache's signature. On the wire the
`hive_priv_sock::PrivRequest` variants carry a required field:
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
A rebuild with no record, or none the builder pushed, fails with an error

20
docs/2-handbook/swarm/credentials-reference.md
View file

@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
| secret | generated by | lives at |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address, the key and the netrc's path as nix.conf settings that the host's nix.conf includes. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. `hive-core` reads only the fragment, for the address. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal

10
docs/4-implementation/scheduler/coordinator.md
View file

@ -42,11 +42,11 @@ because there is no malformed spec to reject.
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied |
| Node | Wraps |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Prebuild` | pull the swarm's closure for the agent into the store through the host's nix daemon while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; pulls the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied |
Cheap — no build slot:

59
docs/4-implementation/swarm/builder.md
View file

@ -452,7 +452,8 @@ you change `swarm.attic.domain`, delete the stale `url` leaf and restart
replaces the stored one with the key `attic cache info` prints, so a cache
that loses its database and gets a new key needs no step: the next run of the
unit stores the new key, the controller copies it to the hives within five
minutes, and `hive-priv-cache-facts` reads it within ten.
minutes, and `hive-priv-cache-facts` reads it within ten (or at once, when a
rebuild finds the host without facts).
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any
@ -532,14 +533,17 @@ identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
token goes out in a header and in the netrc; a hive that finds another scheme
writes nothing.
`hive-priv` doesn't read those files. The key that must have signed a download and
the pull token come from `hive-priv-cache-facts`, a root unit that reads the
hive's own `attic/public-key` and `attic/pull-token` from the store under the
hive's own store identity every ten minutes and writes them to
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`,
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes
decides nothing `hive-priv` trusts.
`hive-priv` and the nix daemon don't read those files. The key that must have
signed a download and the pull token come from `hive-priv-cache-facts`, a root
unit that reads the hive's own `attic/public-key` and `attic/pull-token` from
the store under the hive's own store identity every ten minutes and writes them
to `/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). It also writes
the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
includes: the cache's address (the host's `services.hyperhive.swarm.attic.domain`
and `cache`) as a trusted substituter, the key as a trusted key and the netrc's
path. It restarts `nix-daemon` only when the fragment changed after the daemon
started, which matters for CppNix; Lix re-reads nix.conf on every connection. A
file `hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from
@ -581,17 +585,20 @@ means check attic's fingerprint format; it says nothing about an attacker. A
### Rebuilds download the closure
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
realise the record's store path (`RealiseClosure`). `hive-priv` runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
the host's configured cache address, and the public key and netrc from the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
source, and nix stores the path only when the cache's key signed it. The host's
nix configuration stays as it was.
still up, reads the agent's record in `agent-closures` and has `hive-c0re` pull
the record's store path. `hive-c0re` isn't root and needs no root: it runs
`nix-store --realise <path> --max-jobs 0 --option substituters <cache address>`
through the host's nix daemon. The address is the one in the fragment above, the
only substituter the daemon lets an unprivileged client name. The daemon
ignores a client's keys, `require-sigs` and netrc, so it stores the path only
when a key it trusts signed it. When the fragment is missing, `hive-c0re` first
asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s) instead of waiting for
its timer. A host that still has no facts, a signature refusal and a path the
cache lacks each fail with their own message.
`Swap` then does the same lookup and download, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. A stopped agent skips
`Swap` then does the same lookup and pull, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. `hive-priv` applies it only
when `nix store verify --sigs-needed 1` finds the swarm key's signature on it. A stopped agent skips
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
same and creates the container from that path.
@ -608,13 +615,13 @@ an unreachable cache costs is one function.
Mixed versions:
| hive | controller or record | what happens |
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
| old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
| hive | controller or record | what happens |
| ---- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the apply of a path it didn't download; `hive-c0re` has it download the path (already in the store) and applies again |
## Credentials

135
docs/4-implementation/trust-boundary/hive-c0re-privilege.md
View file

@ -28,35 +28,36 @@ for every operation that genuinely requires root.
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
known operations; there is no arbitrary command pass-through:
| Operation | What it runs |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
| `DaemonReload` | `systemctl daemon-reload` |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
| Operation | What it runs |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its cache fragment is missing |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
| `DaemonReload` | `systemctl daemon-reload` |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
**Strings that reach root-loaded config** — `WriteResourceLimits` and
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env
@ -68,39 +69,61 @@ refuses, never escapes, any value outside its shape. `memory_max` is
bind or credential path is absolute with no whitespace, control byte, `\`,
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild
takes the closure the swarm built instead of building it. Root-owned sources
alone decide what root accepts:
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
built through the host's nix daemon, and `hive-priv` applies it only when the
swarm's key signed it. Pulling needs no root: `nix-store --realise` of a store
path is a request to the daemon. Root-owned sources alone decide what the
daemon accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key.
- The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache` become
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses
it, and `hive-priv` refuses an address that isn't `https`.
- The key and the pull token come from the secret store. The root unit
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`,
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`,
which `hive-c0re` writes for the probe, and it refuses a file in the root
directory that isn't a plain file owned by root or that others can read, and
the directory under the same rule.
- Nix verifies. For that one call `hive-priv` sets `substituters` and
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`.
A path without a signature by that key isn't stored; `hive-priv` answers
`not available` and logs the refusal at error level. `--max-jobs 0` means no
answer, whatever the cache sends, starts a build as root.
- A hive realises only paths whose signature checks. The request carries an
agent name and a store path, both validated, and no address, key or file; any
other field never reaches the verb.
`services.hyperhive.swarm.attic.domain` and `cache`. The root unit
`hive-priv-cache-facts` lists it as `extra-trusted-substituters` in a nix.conf
fragment, `/run/hyperhive-nix/cache.conf`, that the host's nix.conf includes
(`!include`, so a host whose unit hasn't run yet still starts).
- The key and the pull token come from the secret store. The unit reads the
hive's own copies under the hive's own store identity and writes the key to
`/run/hyperhive-priv/attic/public-key` and the token to `netrc` there
(directory `0700`, files `0600`, root), and the key into the fragment as
`extra-trusted-public-keys` and the netrc's path as `netrc-file`. The fragment
holds no secret. `hive-c0re` reads its cache address, and nothing else, from
it; `hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
writes for the probe.
- The daemon enforces. `hive-core` is an unprivileged client: the daemon takes
its `--option substituters` only for an address in `trusted-substituters`,
ignores its `trusted-public-keys`, `require-sigs` and `netrc-file`, and
refuses a path no trusted key signed. `--max-jobs 0` means no answer, whatever
the cache sends, starts a build.
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
`trusted-public-keys` set to the key in the root file, and refuses a path that
is absent or that the key didn't sign. The daemon's trusted keys also include
the host's other caches, so this check is the one that counts only the swarm's.
- When the fragment is missing, `hive-c0re` asks `hive-priv` for `RefreshCacheFacts`,
which starts the facts unit and waits up to 60 s, instead of waiting for the
unit's timer. A host the swarm hasn't given facts to answers
`this hive has no cache facts yet`.
What a compromised `hive-c0re` can do with this verb: ask for any store path
that has the shape of one and that the swarm cache signed, which fills the
store with paths the swarm vouches for. It can't make root accept a path from
a cache or under a key of its choosing, because it writes none of the places
those come from. The trust anchor is the cache's signing key and the store that
hands out its public key, as for every other credential.
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
signed, which fills the store with paths the swarm vouches for, and then ask
`hive-priv` to apply any such path to any agent. Before this change it could
already do both through `RealiseClosure` and an apply; the former check bound an
apply to the path `hive-priv` itself downloaded for that agent, and not to the
agent the swarm built it for. It can't make root accept a path from a cache or
under a key of its choosing, because it writes none of the places those come
from. The trust anchor is the cache's signing key and the store that hands out
its public key, as for every other credential.
**Mixed versions on one host.** `hive-priv` and `hive-c0re` restart separately.
A new `hive-c0re` with an older `hive-priv`: the older one refuses an apply of a
path it didn't download, so `hive-c0re` answers that refusal by sending the
older `RealiseClosure`, which downloads and pins the path (already in the
store), and applies again. A new `hive-priv` with an older `hive-c0re`: it still
accepts `RealiseClosure`, answers `Ok` for a path already in the store with the
swarm's signature and `not available` for one that isn't, and applies. Both
shims go in the release after this one.
**Container allowlist** — `hive-priv` validates every request against
an allowlist before any operation: the allowlist accepts only names