mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re pulls closures through the nix daemon; hive-priv verifies the swarm signature before applying

Browse source
This commit is contained in:
atlas 2026-11-07 13:20:55 +01:00 • committed by mara
commit ecd4445680
21 changed files with 1257 additions and 584 deletions
Download patch file Download diff file Expand all files Collapse all files

10
docs/2-handbook/agents/config-changes.md
View file

@ -154,11 +154,11 @@ the swarm builder built the agent from the updated repo (the
`MetaSync`; its `Create` reads the record once. `MetaSync`; its `Create` reads the record once.
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read - **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
the agent's record in the swarm's `agent-closures` bucket (the controller the agent's record in the swarm's `agent-closures` bucket (the controller
writes it from the builder's reply) and have `hive-priv` download the closure writes it from the builder's reply) and pull the closure it names from the
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then swarm cache through the host's nix daemon. `Swap` and `Create` then give
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and `hive-priv` the store path (`toplevel` on `UpdateContainer` and
`CreateContainer`), which accepts only the path its last `RealiseClosure` for `CreateContainer`), which accepts only a path in the store that carries the
that agent returned. On the wire the swarm cache's signature. On the wire the
`hive_priv_sock::PrivRequest` variants carry a required field: `hive_priv_sock::PrivRequest` variants carry a required field:
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`. `{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
A rebuild with no record, or none the builder pushed, fails with an error A rebuild with no record, or none the builder pushed, fails with an error

20
docs/2-handbook/swarm/credentials-reference.md
View file

@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
### Hive-level — one of each per hive ### Hive-level — one of each per hive
| secret | generated by | lives at | | secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. | | the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address, the key and the netrc's path as nix.conf settings that the host's nix.conf includes. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. `hive-core` reads only the fragment, for the address. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | | the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk | | this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct | | squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | | the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal ### Store identities, one per principal

10
docs/4-implementation/scheduler/coordinator.md
View file

@ -42,11 +42,11 @@ because there is no malformed spec to reject.
Nix-heavy — hold one of the `buildSlots` permits for the node's duration: Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps | | Node | Wraps |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead | | `Prebuild` | pull the swarm's closure for the agent into the store through the host's nix daemon while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node | | `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied | | `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; pulls the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied |
Cheap — no build slot: Cheap — no build slot:

59
docs/4-implementation/swarm/builder.md
View file

@ -452,7 +452,8 @@ you change `swarm.attic.domain`, delete the stale `url` leaf and restart
replaces the stored one with the key `attic cache info` prints, so a cache replaces the stored one with the key `attic cache info` prints, so a cache
that loses its database and gets a new key needs no step: the next run of the that loses its database and gets a new key needs no step: the next run of the
unit stores the new key, the controller copies it to the hives within five unit stores the new key, the controller copies it to the hives within five
minutes, and `hive-priv-cache-facts` reads it within ten. minutes, and `hive-priv-cache-facts` reads it within ten (or at once, when a
rebuild finds the host without facts).
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>` `swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any and `swarm` on every host, whether or not it runs the cache, so a builder on any
@ -532,14 +533,17 @@ identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
token goes out in a header and in the netrc; a hive that finds another scheme token goes out in a header and in the netrc; a hive that finds another scheme
writes nothing. writes nothing.
`hive-priv` doesn't read those files. The key that must have signed a download and `hive-priv` and the nix daemon don't read those files. The key that must have
the pull token come from `hive-priv-cache-facts`, a root unit that reads the signed a download and the pull token come from `hive-priv-cache-facts`, a root
hive's own `attic/public-key` and `attic/pull-token` from the store under the unit that reads the hive's own `attic/public-key` and `attic/pull-token` from
hive's own store identity every ten minutes and writes them to the store under the hive's own store identity every ten minutes and writes them
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's to `/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). It also writes
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`, the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes includes: the cache's address (the host's `services.hyperhive.swarm.attic.domain`
decides nothing `hive-priv` trusts. and `cache`) as a trusted substituter, the key as a trusted key and the netrc's
path. It restarts `nix-daemon` only when the fragment changed after the daemon
started, which matters for CppNix; Lix re-reads nix.conf on every connection. A
file `hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
The same pass checks each of the hive's agents. It reads the agent's record in The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from `agent-closures`, then fetches the closure's narinfo from
@ -581,17 +585,20 @@ means check attic's fingerprint format; it says nothing about an attacker. A
### Rebuilds download the closure ### Rebuilds download the closure
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to still up, reads the agent's record in `agent-closures` and has `hive-c0re` pull
realise the record's store path (`RealiseClosure`). `hive-priv` runs the record's store path. `hive-c0re` isn't root and needs no root: it runs
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with `nix-store --realise <path> --max-jobs 0 --option substituters <cache address>`
the host's configured cache address, and the public key and netrc from the through the host's nix daemon. The address is the one in the fragment above, the
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and only substituter the daemon lets an unprivileged client name. The daemon
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only ignores a client's keys, `require-sigs` and netrc, so it stores the path only
source, and nix stores the path only when the cache's key signed it. The host's when a key it trusts signed it. When the fragment is missing, `hive-c0re` first
nix configuration stays as it was. asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s) instead of waiting for
its timer. A host that still has no facts, a signature refusal and a path the
cache lacks each fail with their own message.
`Swap` then does the same lookup and download, a no-op once the path is in the `Swap` then does the same lookup and pull, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. A stopped agent skips store, and gives `hive-priv` the store path to apply. `hive-priv` applies it only
when `nix store verify --sigs-needed 1` finds the swarm key's signature on it. A stopped agent skips
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the `Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
same and creates the container from that path. same and creates the container from that path.
@ -608,13 +615,13 @@ an unreachable cache costs is one function.
Mixed versions: Mixed versions:
| hive | controller or record | what happens | | hive | controller or record | what happens |
| ---- | -------------------- | ------------------------------------------------------------------------------------------ | | ---- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| old | any | builds locally, as before | | old | any | builds locally, as before |
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it | | new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
| new | cache down | the download fails; same as no record, with the download's error in the log | | new | cache down | the download fails; same as no record, with the download's error in the log |
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well | | new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails | | new | old `hive-priv` | it refuses the apply of a path it didn't download; `hive-c0re` has it download the path (already in the store) and applies again |
## Credentials ## Credentials

135
docs/4-implementation/trust-boundary/hive-c0re-privilege.md
View file

@ -28,35 +28,36 @@ for every operation that genuinely requires root.
**Narrow interface** — `PrivRequest` variants map 1:1 to specific **Narrow interface** — `PrivRequest` variants map 1:1 to specific
known operations; there is no arbitrary command pass-through: known operations; there is no arbitrary command pass-through:
| Operation | What it runs | | Operation | What it runs |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` | | `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) | | `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. | | `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must be the store path the last successful `RealiseClosure` returned for that agent (in memory, per agent), so hive-c0re can't name a path never realised for the agent; hive-priv builds nothing. The binding doesn't mean hive-c0re can't choose the closure: it can still ask hive-priv to download any swarm-signed path for any agent and then apply it. The swarm key and the cache bound that to a cross-assignment risk (one agent gets another's closure), not an arbitrary-path one | | `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | `nix-store --realise <store_path> --max-jobs 0` against the swarm cache, with the host's configured cache address, and the public key and netrc from root-owned `/run/hyperhive-priv/attic/`, as per-call settings that replace the host's substituters and trusted keys and require a signature. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid. It downloads and never builds | | `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
| `DestroyContainer` | `nixos-container destroy <name>` | | `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its cache fragment is missing |
| `ListContainers` | `nixos-container list` | | `DestroyContainer` | `nixos-container destroy <name>` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `ListContainers` | `nixos-container list` |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner | | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner | | `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted; for an agent whose closure record sets `private_users` it also adds `--private-users=pick`, mounts every bind `owneridmap`, and once chowns the agent's `state`, `harness`, `work` and `claude` dirs (never `config`, following no link) to the data owner |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names | | `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else; for an agent whose closure record sets `private_users` it then chowns the dir to the data owner |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
| `DaemonReload` | `systemctl daemon-reload` | | `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) | | `DaemonReload` | `systemctl daemon-reload` |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` | | `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop | | `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs | | `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first | | `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy | | `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
| `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan | | `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
| `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` | | `EnsureBtrfsQuota` | `btrfs quota enable <AGENT_STATE_ROOT>`. Operator opt-in — enabling forces a full rescan |
| `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` | | `ReadSubvolumeUsage` | `btrfs qgroup show -f --raw <state>/<name>` |
| `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none | | `SetSubvolumeQuota` | `btrfs qgroup limit <bytes\|none> <state>/<name>` |
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from | | `MeasureAgentDisk` | `du -s -x -b -c -P` over `<state>/<name>/state`, `<state>/<name>/work` and `/var/lib/nixos-containers/h-<name>`, killed after 10 min. hive-priv derives the paths from the name; the request carries none |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` | | `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export | | `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation | | `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
**Strings that reach root-loaded config** — `WriteResourceLimits` and **Strings that reach root-loaded config** — `WriteResourceLimits` and
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env `WriteNspawnFlags` copy request strings into a systemd drop-in and an env
@ -68,39 +69,61 @@ refuses, never escapes, any value outside its shape. `memory_max` is
bind or credential path is absolute with no whitespace, control byte, `\`, bind or credential path is absolute with no whitespace, control byte, `\`,
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd. `$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
**Closures from the swarm cache** — `RealiseClosure` is how a hive's rebuild **Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
takes the closure the swarm built instead of building it. Root-owned sources built through the host's nix daemon, and `hive-priv` applies it only when the
alone decide what root accepts: swarm's key signed it. Pulling needs no root: `nix-store --realise` of a store
path is a request to the daemon. Root-owned sources alone decide what the
daemon accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs - The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key. with a push token and holds no signing key.
- The address is the host's configuration: - The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache` become `services.hyperhive.swarm.attic.domain` and `cache`. The root unit
`HIVE_PRIV_CACHE_URL` in the `hive-priv` unit. No request and no file chooses `hive-priv-cache-facts` lists it as `extra-trusted-substituters` in a nix.conf
it, and `hive-priv` refuses an address that isn't `https`. fragment, `/run/hyperhive-nix/cache.conf`, that the host's nix.conf includes
- The key and the pull token come from the secret store. The root unit (`!include`, so a host whose unit hasn't run yet still starts).
`hive-priv-cache-facts` reads the hive's own copies under the hive's own store - The key and the pull token come from the secret store. The unit reads the
identity and writes them to `/run/hyperhive-priv/attic/` (directory `0700`, hive's own copies under the hive's own store identity and writes the key to
files `0600`, root). `hive-priv` reads nothing from `/run/hyperhive/attic/`, `/run/hyperhive-priv/attic/public-key` and the token to `netrc` there
which `hive-c0re` writes for the probe, and it refuses a file in the root (directory `0700`, files `0600`, root), and the key into the fragment as
directory that isn't a plain file owned by root or that others can read, and `extra-trusted-public-keys` and the netrc's path as `netrc-file`. The fragment
the directory under the same rule. holds no secret. `hive-c0re` reads its cache address, and nothing else, from
- Nix verifies. For that one call `hive-priv` sets `substituters` and it; `hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
`trusted-public-keys` to the swarm cache and its key, and `require-sigs true`. writes for the probe.
A path without a signature by that key isn't stored; `hive-priv` answers - The daemon enforces. `hive-core` is an unprivileged client: the daemon takes
`not available` and logs the refusal at error level. `--max-jobs 0` means no its `--option substituters` only for an address in `trusted-substituters`,
answer, whatever the cache sends, starts a build as root. ignores its `trusted-public-keys`, `require-sigs` and `netrc-file`, and
- A hive realises only paths whose signature checks. The request carries an refuses a path no trusted key signed. `--max-jobs 0` means no answer, whatever
agent name and a store path, both validated, and no address, key or file; any the cache sends, starts a build.
other field never reaches the verb. - `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
`trusted-public-keys` set to the key in the root file, and refuses a path that
is absent or that the key didn't sign. The daemon's trusted keys also include
the host's other caches, so this check is the one that counts only the swarm's.
- When the fragment is missing, `hive-c0re` asks `hive-priv` for `RefreshCacheFacts`,
which starts the facts unit and waits up to 60 s, instead of waiting for the
unit's timer. A host the swarm hasn't given facts to answers
`this hive has no cache facts yet`.
What a compromised `hive-c0re` can do with this verb: ask for any store path What a compromised `hive-c0re` can do: pull any store path that the swarm cache
that has the shape of one and that the swarm cache signed, which fills the signed, which fills the store with paths the swarm vouches for, and then ask
store with paths the swarm vouches for. It can't make root accept a path from `hive-priv` to apply any such path to any agent. Before this change it could
a cache or under a key of its choosing, because it writes none of the places already do both through `RealiseClosure` and an apply; the former check bound an
those come from. The trust anchor is the cache's signing key and the store that apply to the path `hive-priv` itself downloaded for that agent, and not to the
hands out its public key, as for every other credential. agent the swarm built it for. It can't make root accept a path from a cache or
under a key of its choosing, because it writes none of the places those come
from. The trust anchor is the cache's signing key and the store that hands out
its public key, as for every other credential.
**Mixed versions on one host.** `hive-priv` and `hive-c0re` restart separately.
A new `hive-c0re` with an older `hive-priv`: the older one refuses an apply of a
path it didn't download, so `hive-c0re` answers that refusal by sending the
older `RealiseClosure`, which downloads and pins the path (already in the
store), and applies again. A new `hive-priv` with an older `hive-c0re`: it still
accepts `RealiseClosure`, answers `Ok` for a path already in the store with the
swarm's signature and `not available` for one that isn't, and applies. Both
shims go in the release after this one.
**Container allowlist** — `hive-priv` validates every request against **Container allowlist** — `hive-priv` validates every request against
an allowlist before any operation: the allowlist accepts only names an allowlist before any operation: the allowlist accepts only names

484
hive-c0re/src/cache_pull.rs Normal file
View file

@ -0,0 +1,484 @@
//! Pull the swarm's closure into the store through the host's nix daemon.
//!
//! hive-c0re is not root and does not need to be: `nix-store --realise` of a
//! store path is a request to the daemon, which substitutes it. What the daemon
//! accepts is root's: the root unit `hive-priv-cache-facts` writes a nix.conf
//! fragment ([`CONF_FRAGMENT`]) that lists the swarm cache as a
//! `trusted-substituter`, adds its key to `trusted-public-keys` and names the
//! netrc. An unprivileged client may name a substituter only from that list,
//! cannot lower `require-sigs` or replace the keys, and a path without a
//! signature by a trusted key is refused. hive-priv then applies the toplevel
//! only if the swarm's key signed it.
//!
//! The cache's address is read from the fragment, so the address named is
//! byte for byte the one the daemon trusts. A missing fragment means this host
//! has not been given its cache facts; [`pull`] first asks hive-priv to run the
//! facts unit, which is how the swarm's copy of the facts reaches the host
//! without waiting for the unit's timer.
use std::time::Duration;
use anyhow::Result;
use hive_priv_sock::valid_closure_path;
/// The nix.conf fragment `hive-priv-cache-facts` writes (root-owned, readable).
pub const CONF_FRAGMENT: &str = "/run/hyperhive-nix/cache.conf";
/// The fragment's line that lists the cache the daemon lets a client name.
const URL_SETTING: &str = "extra-trusted-substituters";
/// Wall-clock bound on one pull. A stalled transfer is bounded by nix's own
/// `stalled-download-timeout`; this covers the rest.
const PULL_TIMEOUT: Duration = Duration::from_mins(30);
/// Why a closure was not pulled.
#[derive(Debug, PartialEq, Eq)]
pub enum PullError {
/// This host has no cache facts: no fragment, no address in it, or a
/// daemon that has not loaded it. Waiting for the facts unit fixes it.
FactsMissing(String),
/// Nix refused the path: no signature by a key this host trusts.
SignatureRefused(String),
/// The cache does not have the path.
NotFound(String),
/// Anything else nix said, or could not do.
Failed(String),
/// Nix was still running at [`PULL_TIMEOUT`].
TimedOut,
}
impl std::fmt::Display for PullError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::FactsMissing(why) => write!(
f,
"this hive has no cache facts yet ({why}); a later deploy retries"
),
Self::SignatureRefused(last) => write!(
f,
"nix refused the closure, which carries no signature by a key this host trusts: {last}"
),
Self::NotFound(last) => write!(f, "the swarm cache does not have the closure: {last}"),
Self::Failed(last) => f.write_str(last),
Self::TimedOut => write!(f, "nix-store timed out after {}s", PULL_TIMEOUT.as_secs()),
}
}
}
impl std::error::Error for PullError {}
/// What a `nix-store` run came to.
pub enum Run {
Exited { success: bool, stderr: String },
TimedOut,
}
/// What [`pull`] does on the host, so a test can stand in for it.
pub trait Effects: Sync {
/// The fragment's text; `None` when it cannot be read.
async fn fragment(&self) -> Option<String>;
/// Run the facts unit through hive-priv and wait for it.
async fn refresh(&self) -> Result<()>;
/// Run `nix-store` with `args`, bounded by [`PULL_TIMEOUT`].
async fn nix_store(&self, args: Vec<String>) -> std::io::Result<Run>;
}
/// The cache's address in `fragment`: the value of its
/// `extra-trusted-substituters` line, when that is one https address.
#[must_use]
pub fn cache_url(fragment: &str) -> Option<&str> {
fragment.lines().find_map(|line| {
let (name, value) = line.split_once('=')?;
let value = value.trim();
(name.trim() == URL_SETTING
&& value.starts_with("https://")
&& !value.contains(char::is_whitespace))
.then_some(value)
})
}
/// The `nix-store` arguments that fetch `store_path` and nothing else.
///
/// - `--option substituters <url>` names the one cache the daemon lists as
/// trusted. An unprivileged client's `substituters` that is not on that list
/// empties the list and fails the pull, so the address must be the fragment's.
/// - `--max-jobs 0` forbids a local build, so a path the cache lacks fails.
/// - No `trusted-public-keys`, `require-sigs` or `netrc-file`: the daemon
/// ignores them from an unprivileged client and applies its own.
#[must_use]
pub fn pull_args(store_path: &str, url: &str) -> Vec<String> {
[
"--realise",
store_path,
"--max-jobs",
"0",
"--option",
"substituters",
url,
]
.map(str::to_owned)
.into()
}
/// Name nix's refusal from its stderr. A refusal for want of a signature
/// or a daemon that does not trust the address is told apart from a path the
/// cache lacks and from a cache that is down.
fn classify(stderr: &str) -> PullError {
let last = stderr
.lines()
.rev()
.find(|l| !l.trim().is_empty())
.unwrap_or("nix-store failed with no message")
.trim()
.to_owned();
let lower = stderr.to_lowercase();
if lower.contains("untrusted substituter") {
PullError::FactsMissing(
"the nix daemon does not list the swarm cache as a trusted substituter".to_owned(),
)
} else if lower.contains("not signed by any of the keys") || lower.contains("signature") {
PullError::SignatureRefused(last)
} else if lower.contains("don't know how to build")
|| lower.contains("no substituter that can build")
{
PullError::NotFound(last)
} else {
PullError::Failed(last)
}
}
/// Pull `store_path` into the store.
///
/// # Errors
/// [`PullError`]: no cache facts on this host (after one refresh), nix
/// refusing the signature, the cache lacking the path, nix failing otherwise,
/// or a timeout.
pub async fn pull(store_path: &str) -> Result<(), PullError> {
pull_with(&Real, store_path).await
}
async fn pull_with(fx: &impl Effects, store_path: &str) -> Result<(), PullError> {
if !valid_closure_path(store_path) {
return Err(PullError::Failed(format!(
"{store_path:?} is not a store path"
)));
}
let url = address_after_refresh(fx).await?;
match fx
.nix_store(pull_args(store_path, &url))
.await
.map_err(|e| PullError::Failed(format!("running nix-store: {e}")))?
{
Run::Exited { success: true, .. } => Ok(()),
Run::Exited { stderr, .. } => Err(classify(&stderr)),
Run::TimedOut => Err(PullError::TimedOut),
}
}
/// The cache's address, after one facts refresh when the host has none.
async fn address_after_refresh(fx: &impl Effects) -> Result<String, PullError> {
if let Some(url) = address(fx).await {
return Ok(url);
}
let refreshed = fx.refresh().await;
address(fx).await.ok_or_else(|| {
PullError::FactsMissing(match refreshed {
Ok(()) => "the facts unit ran and the swarm has given this hive none".to_owned(),
Err(e) => format!("refreshing them failed: {e:#}"),
})
})
}
async fn address(fx: &impl Effects) -> Option<String> {
fx.fragment()
.await
.as_deref()
.and_then(cache_url)
.map(str::to_owned)
}
/// The host: the fragment on disk, hive-priv for the refresh, `nix-store` from
/// the system profile through the daemon.
struct Real;
impl Effects for Real {
async fn fragment(&self) -> Option<String> {
tokio::fs::read_to_string(CONF_FRAGMENT).await.ok()
}
async fn refresh(&self) -> Result<()> {
crate::priv_client::refresh_cache_facts().await
}
async fn nix_store(&self, args: Vec<String>) -> std::io::Result<Run> {
let mut cmd = tokio::process::Command::new("nix-store");
// Explicit, so the store is the daemon's whatever the environment.
cmd.args(args)
.env("NIX_REMOTE", "daemon")
.stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::null())
.kill_on_drop(true);
match tokio::time::timeout(PULL_TIMEOUT, cmd.output()).await {
Ok(out) => {
let out = out?;
Ok(Run::Exited {
success: out.status.success(),
stderr: String::from_utf8_lossy(&out.stderr).into_owned(),
})
}
Err(_) => Ok(Run::TimedOut),
}
}
}
#[cfg(test)]
mod tests {
use std::sync::Mutex;
use super::*;
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const URL: &str = "https://cache.example/swarm";
fn fragment(url: &str) -> String {
format!(
"extra-trusted-substituters = {url}\nextra-trusted-public-keys = swarm:AAAA\nnetrc-file = /run/hyperhive-priv/attic/netrc"
)
}
/// A host that answers from a script: the fragment as it is now (and the
/// one a refresh leaves), what `nix-store` comes to, and a log of calls.
struct Fake {
fragment: Mutex<Option<String>>,
after_refresh: Option<String>,
refresh_fails: bool,
nix: Mutex<Option<Run>>,
calls: Mutex<Vec<String>>,
}
impl Fake {
fn new(fragment: Option<String>, nix: Run) -> Self {
Self {
fragment: Mutex::new(fragment),
after_refresh: None,
refresh_fails: false,
nix: Mutex::new(Some(nix)),
calls: Mutex::new(Vec::new()),
}
}
fn calls(&self) -> Vec<String> {
self.calls.lock().unwrap().clone()
}
}
impl Effects for Fake {
async fn fragment(&self) -> Option<String> {
self.calls.lock().unwrap().push("fragment".to_owned());
self.fragment.lock().unwrap().clone()
}
async fn refresh(&self) -> Result<()> {
self.calls.lock().unwrap().push("refresh".to_owned());
if self.refresh_fails {
anyhow::bail!("hive-priv unreachable");
}
*self.fragment.lock().unwrap() = self.after_refresh.clone();
Ok(())
}
async fn nix_store(&self, args: Vec<String>) -> std::io::Result<Run> {
self.calls
.lock()
.unwrap()
.push(format!("nix-store {}", args.join(" ")));
Ok(self.nix.lock().unwrap().take().expect("one run"))
}
}
fn exited(success: bool, stderr: &str) -> Run {
Run::Exited {
success,
stderr: stderr.to_owned(),
}
}
fn option<'a>(args: &'a [String], name: &str) -> Option<&'a str> {
args.windows(3)
.find(|w| w[0] == "--option" && w[1] == name)
.map(|w| w[2].as_str())
}
/// An unprivileged client can name only the trusted address and cannot set
/// keys, signatures or the netrc, so the command carries exactly that, and
/// never lets nix build.
#[test]
fn the_command_names_only_the_swarm_cache_and_never_builds() {
let args = pull_args(PATH, URL);
assert_eq!(args[..2], ["--realise", PATH]);
let jobs = args.iter().position(|a| a == "--max-jobs").expect("flag");
assert_eq!(args[jobs + 1], "0");
assert_eq!(option(&args, "substituters"), Some(URL));
let options: Vec<_> = args.iter().filter(|a| *a == "--option").collect();
assert_eq!(options.len(), 1, "{args:?}");
for forbidden in ["trusted-public-keys", "require-sigs", "netrc-file"] {
assert_eq!(option(&args, forbidden), None, "{forbidden}");
}
assert!(!args.iter().any(|a| a.starts_with("extra-")), "{args:?}");
}
#[test]
fn the_address_is_the_fragments_https_line() {
assert_eq!(cache_url(&fragment(URL)), Some(URL));
for bad in [
String::new(),
"netrc-file = /x".to_owned(),
fragment("http://cache.example/swarm"),
fragment("https://cache.example/a b"),
fragment(""),
// The key's line is not the address's.
"extra-trusted-public-keys = https://cache.example/swarm".to_owned(),
] {
assert_eq!(cache_url(&bad), None, "{bad:?}");
}
}
#[tokio::test]
async fn with_the_fragment_in_place_it_pulls_without_a_refresh() {
let fx = Fake::new(Some(fragment(URL)), exited(true, ""));
pull_with(&fx, PATH).await.expect("pulled");
assert_eq!(
fx.calls(),
[
"fragment".to_owned(),
format!("nix-store {}", pull_args(PATH, URL).join(" "))
]
);
}
/// Missing facts are refreshed once, and the pull then goes ahead with the
/// address the refresh wrote.
#[tokio::test]
async fn a_missing_fragment_is_refreshed_before_the_pull() {
let mut fx = Fake::new(None, exited(true, ""));
fx.after_refresh = Some(fragment(URL));
pull_with(&fx, PATH).await.expect("pulled");
assert_eq!(
fx.calls(),
[
"fragment".to_owned(),
"refresh".to_owned(),
"fragment".to_owned(),
format!("nix-store {}", pull_args(PATH, URL).join(" "))
]
);
}
/// Facts that are still absent after the refresh, whether the unit ran or
/// hive-priv could not start it, stop the pull before nix is asked.
#[tokio::test]
async fn facts_still_missing_after_the_refresh_are_reported_and_nix_is_not_run() {
for (case, refresh_fails, want) in [
("unit ran", false, "none"),
("refresh failed", true, "hive-priv unreachable"),
] {
let mut fx = Fake::new(None, exited(true, ""));
fx.refresh_fails = refresh_fails;
let err = pull_with(&fx, PATH).await.expect_err(case);
let PullError::FactsMissing(why) = &err else {
panic!("{case}: {err:?}");
};
assert!(why.contains(want), "{case}: {why}");
assert!(
format!("{err}").contains("a later deploy retries"),
"{case}"
);
assert!(
!fx.calls().iter().any(|c| c.starts_with("nix-store")),
"{case}"
);
}
}
#[tokio::test]
async fn a_fragment_without_an_address_is_refreshed_like_a_missing_one() {
let mut fx = Fake::new(Some("netrc-file = /x".to_owned()), exited(true, ""));
fx.after_refresh = Some(fragment(URL));
pull_with(&fx, PATH).await.expect("pulled");
assert!(fx.calls().contains(&"refresh".to_owned()));
}
#[test]
fn nixs_refusals_are_told_apart() {
for (stderr, want) in [
(
"warning: ignoring substitute for '/nix/store/x' from 'https://c', as it's not signed by any of the keys in 'trusted-public-keys'\nerror: build of '/nix/store/x' failed",
"signature",
),
(
"error: cannot add path 'x' because it lacks a signature by a trusted key",
"signature",
),
(
"warning: ignoring untrusted substituter 'https://c', you are not a trusted user.\ndon't know how to build these paths:\nerror: build of 'x' failed",
"facts",
),
(
"don't know how to build these paths:\n /nix/store/x\nerror: build of '/nix/store/x' failed",
"not found",
),
(
"error: path '/nix/store/x' is required, but there is no substituter that can build it",
"not found",
),
(
"error: unable to download 'https://c/x.narinfo': Couldn't connect to server",
"other",
),
] {
let kind = match classify(stderr) {
PullError::SignatureRefused(_) => "signature",
PullError::FactsMissing(_) => "facts",
PullError::NotFound(_) => "not found",
PullError::Failed(_) => "other",
PullError::TimedOut => "timeout",
};
assert_eq!(kind, want, "{stderr}");
}
// The last line is what the operator reads.
let PullError::Failed(last) = classify("warning: a\n\nerror: Couldn't connect\n\n") else {
panic!("not a plain failure");
};
assert_eq!(last, "error: Couldn't connect");
}
#[tokio::test]
async fn a_failed_run_is_mapped_and_a_timeout_is_a_timeout() {
let fx = Fake::new(
Some(fragment(URL)),
exited(
false,
"warning: ignoring substitute from 'https://c', as it's not signed by any of the keys",
),
);
assert!(matches!(
pull_with(&fx, PATH).await,
Err(PullError::SignatureRefused(_))
));
let fx = Fake::new(Some(fragment(URL)), Run::TimedOut);
assert_eq!(pull_with(&fx, PATH).await, Err(PullError::TimedOut));
}
/// Nothing but a store path is ever handed to nix.
#[tokio::test]
async fn only_a_store_path_is_pulled() {
for bad in ["", "main", "/etc/passwd", "--help", "/nix/store/x"] {
let fx = Fake::new(Some(fragment(URL)), exited(true, ""));
assert!(
matches!(pull_with(&fx, bad).await, Err(PullError::Failed(_))),
"{bad}"
);
assert!(fx.calls().is_empty(), "{bad}");
}
}
}

8
hive-c0re/src/job_queue/exec.rs
View file

@ -51,8 +51,8 @@ pub(super) trait Host: Sync {
fn pinned_closure(&self, name: &str) -> Option<String>; fn pinned_closure(&self, name: &str) -> Option<String>;
/// Drop the agent's pin. /// Drop the agent's pin.
fn unpin_closure(&self, name: &str); fn unpin_closure(&self, name: &str);
/// Download `store_path` from the swarm's cache into the store, never /// Download `store_path` from the swarm's cache into the store through the
/// building it. /// host's nix daemon, never building it.
async fn realise_closure(&self, name: &str, store_path: &str) -> Result<()>; async fn realise_closure(&self, name: &str, store_path: &str) -> Result<()>;
/// Re-apply drop-ins, then point the container at the closure `toplevel`. /// Re-apply drop-ins, then point the container at the closure `toplevel`.
async fn swap_update( async fn swap_update(
@ -128,7 +128,9 @@ impl Host for RealHost {
} }
async fn realise_closure(&self, name: &str, store_path: &str) -> Result<()> { async fn realise_closure(&self, name: &str, store_path: &str) -> Result<()> {
crate::priv_client::realise_closure(name, store_path).await crate::cache_pull::pull(store_path).await?;
tracing::debug!(%name, %store_path, "pulled the swarm's closure through the nix daemon");
Ok(())
} }
async fn swap_update( async fn swap_update(

1
hive-c0re/src/main.rs
View file

@ -17,6 +17,7 @@ mod actions;
mod agent_config; mod agent_config;
mod agent_set; mod agent_set;
mod attic; mod attic;
mod cache_pull;
mod container_view; mod container_view;
mod coordinator; mod coordinator;
mod dashboard; mod dashboard;

106
hive-c0re/src/priv_client.rs
View file

@ -267,9 +267,15 @@ pub async fn kill_container(name: &str) -> Result<()> {
pub async fn update_container_streaming( pub async fn update_container_streaming(
name: &str, name: &str,
toplevel: &str, toplevel: &str,
on_line: impl FnMut(PrivStream, &str), mut on_line: impl FnMut(PrivStream, &str),
) -> Result<()> { ) -> Result<()> {
ok(call_streaming(&update_container_request(name, toplevel), on_line).await?) let req = update_container_request(name, toplevel);
let first = ok(call_streaming(&req, &mut on_line).await?);
if !older_hive_priv_refused(&first) {
return first;
}
ask_older_hive_priv_to_download(name, toplevel).await?;
ok(call_streaming(&req, &mut on_line).await?)
} }
fn update_container_request(name: &str, toplevel: &str) -> PrivRequest { fn update_container_request(name: &str, toplevel: &str) -> PrivRequest {
@ -285,9 +291,15 @@ fn update_container_request(name: &str, toplevel: &str) -> PrivRequest {
pub async fn create_container_streaming( pub async fn create_container_streaming(
name: &str, name: &str,
toplevel: &str, toplevel: &str,
on_line: impl FnMut(PrivStream, &str), mut on_line: impl FnMut(PrivStream, &str),
) -> Result<()> { ) -> Result<()> {
ok(call_streaming(&create_container_request(name, toplevel), on_line).await?) let req = create_container_request(name, toplevel);
let first = ok(call_streaming(&req, &mut on_line).await?);
if !older_hive_priv_refused(&first) {
return first;
}
ask_older_hive_priv_to_download(name, toplevel).await?;
ok(call_streaming(&req, &mut on_line).await?)
} }
fn create_container_request(name: &str, toplevel: &str) -> PrivRequest { fn create_container_request(name: &str, toplevel: &str) -> PrivRequest {
@ -298,17 +310,49 @@ fn create_container_request(name: &str, toplevel: &str) -> PrivRequest {
} }
} }
/// See [`PrivRequest::RealiseClosure`]. A path hive-priv could not download /// What an older hive-priv, which applies only a path it downloaded itself,
/// reads as an error whose message starts `not available: `. /// says to an apply of any other.
/// const OLDER_PIN_REFUSAL: &str = "is not the closure realised for this agent";
/// # Errors
/// The call failed, or hive-priv answered `not available`. /// Whether `applied` is an older hive-priv refusing an apply because it did not
pub async fn realise_closure(agent: &str, store_path: &str) -> Result<()> { /// download the path itself (hive-c0re pulled it through the nix daemon). A
/// hive-priv of the same release never refuses that way, so a retry through
/// [`ask_older_hive_priv_to_download`] happens only while the two restart out of
/// step.
fn older_hive_priv_refused(applied: &Result<()>) -> bool {
applied
.as_ref()
.is_err_and(|e| format!("{e:#}").contains(OLDER_PIN_REFUSAL))
}
/// Ask an older hive-priv to download `toplevel`, as it did before, which also
/// pins the path for the apply that follows. The path is already in the store,
/// so this is a lookup.
async fn ask_older_hive_priv_to_download(name: &str, toplevel: &str) -> Result<()> {
tracing::warn!(%name, %toplevel, "an older hive-priv refused a closure it did not download; asking it to");
ok(call(&PrivRequest::RealiseClosure { ok(call(&PrivRequest::RealiseClosure {
agent: agent.to_owned(), agent: name.to_owned(),
store_path: store_path.to_owned(), store_path: toplevel.to_owned(),
}) })
.await?) .await?)
.context("asking the older hive-priv to download the closure")
}
/// How long hive-c0re waits for hive-priv's facts refresh: hive-priv bounds
/// the unit at 60 s.
const REFRESH_WAIT: std::time::Duration = std::time::Duration::from_secs(90);
/// Run the root unit that reads this hive's cache facts from the store and
/// writes the nix.conf fragment, and wait for it. An older hive-priv does not
/// know the op and answers an error.
///
/// # Errors
/// The call failed or did not answer in time, or the unit failed.
pub async fn refresh_cache_facts() -> Result<()> {
within(REFRESH_WAIT, async {
ok(call(&PrivRequest::RefreshCacheFacts).await?)
})
.await
} }
pub async fn destroy_container(name: &str) -> Result<()> { pub async fn destroy_container(name: &str) -> Result<()> {
@ -684,9 +728,11 @@ mod tests {
use super::{ use super::{
BindMount, CredentialMount, HiveFacts, NetworkIsolation, NixBuilds, PrivRequest, BindMount, CredentialMount, HiveFacts, NetworkIsolation, NixBuilds, PrivRequest,
PrivResponse, PrivStream, call_at, call_streaming_at, call_with_fd_at, check, PrivResponse, PrivStream, call_at, call_streaming_at, call_with_fd_at, check,
create_container_request, parse_qgroup_usage, resource_limits_request, create_container_request, older_hive_priv_refused, parse_qgroup_usage,
send_snapshot_request, update_container_request, within, write_nspawn_flags_request, resource_limits_request, send_snapshot_request, update_container_request, within,
write_nspawn_flags_request,
}; };
use anyhow::Context as _;
use serde_json::{Value, json}; use serde_json::{Value, json};
use std::io::{Read as _, Write as _}; use std::io::{Read as _, Write as _};
use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd, RawFd}; use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd, RawFd};
@ -697,6 +743,38 @@ mod tests {
serde_json::to_value(req).expect("a request serialises") serde_json::to_value(req).expect("a request serialises")
} }
const REFUSAL: &str = "atlas: /nix/store/x is not the closure realised for this agent (none)";
/// An older hive-priv's refusal of a path it did not download is the
/// signal to ask it to download; a success and any other failure are not.
#[test]
fn only_an_older_hive_privs_pin_refusal_asks_for_a_download() {
assert!(older_hive_priv_refused(&Err(anyhow::anyhow!(REFUSAL))));
// As the streaming call words it: the error wraps hive-priv's text.
assert!(older_hive_priv_refused(
&Err(anyhow::anyhow!(REFUSAL)).context("nixos-container create")
));
assert!(!older_hive_priv_refused(&Ok(())));
for other in [
"nixos-container create failed",
"atlas: refusing /nix/store/x: has no signature by the swarm cache's key",
"connect to hive-priv socket",
] {
assert!(
!older_hive_priv_refused(&Err(anyhow::anyhow!(other))),
"{other}"
);
}
}
#[test]
fn the_cache_facts_refresh_names_no_unit() {
assert_eq!(
wire(&PrivRequest::RefreshCacheFacts),
json!({"op": "refresh_cache_facts"})
);
}
#[test] #[test]
fn create_and_update_keep_the_name_and_toplevel_apart_and_stream() { fn create_and_update_keep_the_name_and_toplevel_apart_and_stream() {
assert_eq!( assert_eq!(

55
hive-priv-sock/src/lib.rs
View file

@ -110,9 +110,10 @@ pub fn valid_meta_rev(rev: &str) -> bool {
/// Whether `path` is a store path in the shape nix prints one: `/nix/store/`, /// Whether `path` is a store path in the shape nix prints one: `/nix/store/`,
/// 32 lowercase letters or digits, `-`, then a name of the characters store /// 32 lowercase letters or digits, `-`, then a name of the characters store
/// names allow. The path crosses the wire in [`PrivRequest::RealiseClosure`], /// names allow. The path crosses the wire in [`PrivRequest::UpdateContainer`]
/// [`PrivRequest::UpdateContainer`] and [`PrivRequest::CreateContainer`], so /// and [`PrivRequest::CreateContainer`] (and the older
/// `hive-priv` checks it before it hands it to nix. /// [`PrivRequest::RealiseClosure`]), so `hive-priv` checks it before it hands
/// it to nix, and hive-c0re before it names it to nix.
#[must_use] #[must_use]
pub fn valid_closure_path(path: &str) -> bool { pub fn valid_closure_path(path: &str) -> bool {
let Some(rest) = path.strip_prefix("/nix/store/") else { let Some(rest) = path.strip_prefix("/nix/store/") else {
@ -131,11 +132,6 @@ pub fn valid_closure_path(path: &str) -> bool {
}) })
} }
/// What a failed [`PrivRequest::RealiseClosure`] says when nix refused a path
/// because no trusted key signed it, so a caller can tell that apart from a
/// cache that is down.
pub const SIGNATURE_REFUSED: &str = "no signature by the swarm cache's key";
/// Root of per-agent state directories on the host. /// Root of per-agent state directories on the host.
/// Subdirectory layout: `<AGENT_STATE_ROOT>/<name>/state/<file>`. /// Subdirectory layout: `<AGENT_STATE_ROOT>/<name>/state/<file>`.
/// Used by `WriteAgentStateFile` to derive the write path so the /// Used by `WriteAgentStateFile` to derive the write path so the
@ -269,8 +265,8 @@ pub enum PrivRequest {
KillContainer { name: String }, KillContainer { name: String },
/// Point `name`'s container profile at `toplevel`, then reload the unit if /// Point `name`'s container profile at `toplevel`, then reload the unit if
/// it runs. hive-priv never builds: `toplevel` must be the path the last /// it runs. hive-priv never builds: `toplevel` must already be in the store
/// successful `RealiseClosure` for `name` returned, and it refuses any other. /// and signed by the swarm cache's key, and it refuses any other.
/// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages /// When `stream` is `true`, hive-priv sends `PrivEvent::Line` messages
/// as the process runs, then a terminal `PrivEvent::Done`. /// as the process runs, then a terminal `PrivEvent::Done`.
UpdateContainer { UpdateContainer {
@ -292,18 +288,24 @@ pub enum PrivRequest {
toplevel: String, toplevel: String,
}, },
/// `nix-store --realise <store_path>` from the swarm's binary cache, never /// Kept for an older hive-c0re that still asks hive-priv to download:
/// building: the closure the swarm built for `agent`, downloaded into the /// hive-c0re now pulls through the nix daemon and hive-priv downloads
/// store ahead of the update that applies it. /// nothing. The answer is `Ok` when `store_path` is already in the store
/// and signed by the swarm cache's key (what `CreateContainer` and
/// `UpdateContainer` require), else an error that starts `not available: `.
/// `store_path` is checked by [`valid_closure_path`].
/// ///
/// The request names no cache and no key: hive-priv takes the cache's /// An older hive-priv answers the same request by downloading, and pins the
/// address from its own unit's environment and the key and the pull token /// path for the apply that follows; hive-c0re sends it to one only when
/// from files only root can write, and nix accepts the path only with a /// that older hive-priv refused an apply.
/// signature by that key. `store_path` is checked by
/// [`valid_closure_path`]. A path nix could not substitute answers with an
/// error that starts `not available: `.
RealiseClosure { agent: String, store_path: String }, RealiseClosure { agent: String, store_path: String },
/// `systemctl start hive-priv-cache-facts.service`, bounded at 60 s: the
/// root unit that reads this hive's cache facts from the store and writes
/// the nix.conf fragment the daemon trusts. hive-c0re asks for it when the
/// fragment is missing. The request names no unit.
RefreshCacheFacts,
/// `nixos-container destroy <name>` /// `nixos-container destroy <name>`
DestroyContainer { name: String }, DestroyContainer { name: String },
@ -880,6 +882,21 @@ mod tests {
assert!(serde_json::from_str::<PrivRequest>(without).is_err()); assert!(serde_json::from_str::<PrivRequest>(without).is_err());
} }
/// The facts refresh names no unit and carries no field: a stray field is
/// dropped when it is read, so a request cannot choose what to start.
#[test]
fn the_cache_facts_refresh_carries_nothing() {
let line = r#"{"op":"refresh_cache_facts","unit":"sshd.service"}"#;
assert!(matches!(
serde_json::from_str::<PrivRequest>(line),
Ok(PrivRequest::RefreshCacheFacts)
));
assert_eq!(
serde_json::to_string(&PrivRequest::RefreshCacheFacts).unwrap(),
r#"{"op":"refresh_cache_facts"}"#
);
}
/// `toplevel` is required: hive-priv has no flake to build one from. /// `toplevel` is required: hive-priv has no flake to build one from.
#[test] #[test]
fn a_request_without_a_toplevel_is_refused() { fn a_request_without_a_toplevel_is_refused() {

17
hive-priv/README.md
View file

@ -46,15 +46,14 @@ Each module's `//!` header lists its ops.
`container_toplevel_action` (in `src/container.rs`) applies a closure that is `container_toplevel_action` (in `src/container.rs`) applies a closure that is
already in the store; hive-priv builds nothing. hive-c0re names the closure already in the store; hive-priv builds nothing. hive-c0re names the closure
(`toplevel` on `CreateContainer` / `UpdateContainer`), downloaded beforehand (`toplevel` on `CreateContainer` / `UpdateContainer`), pulled beforehand by
with `RealiseClosure`. hive-priv remembers, per agent, the path its last hive-c0re through the nix daemon. hive-priv applies a path only when
successful `RealiseClosure` returned (in memory), and applies only that path to `nix store verify --no-contents --sigs-needed 1`, with the key in root-owned
that agent. It refuses a path never realised for the agent, another agent's `/run/hyperhive-priv/attic/public-key`, finds the swarm cache's signature on it.
and the host's own, and logs an error naming the agent and both paths. A It refuses a path that is absent, unsigned, signed by another key or not a store
restart between the download and the apply empties the record, so the apply path, and logs an error naming the agent and the path. hive-priv keeps no record
fails and the job retries. hive-priv does not re-check the signature at apply of what was pulled, so a restart between the pull and the apply loses nothing.
time: `RealiseClosure` enforces it, and the binding limits an apply to what that `RefreshCacheFacts` starts the root unit that refreshes the cache facts.
call stored.
`create` passes the path to `nixos-container create --system-path`, so `create` passes the path to `nixos-container create --system-path`, so
`nixos-container`'s own `buildFlake()` never runs. Without the flag, it builds `nixos-container`'s own `buildFlake()` never runs. Without the flag, it builds

719
hive-priv/src/closure.rs
View file

@ -1,42 +1,34 @@
//! `RealiseClosure`: download the closure the swarm built for an agent from //! The check on a closure before it is applied, and the two small verbs around
//! the swarm's binary cache into the store, without ever building. //! the swarm cache.
//! //!
//! What this accepts is decided by root-owned sources alone: the cache's //! hive-c0re (unprivileged) downloads the swarm's closure through the host's
//! address is the host's configuration, handed to the unit in //! nix daemon. `CreateContainer` and `UpdateContainer` then apply a toplevel
//! [`CACHE_URL_ENV`], and the cache's public key and the pull token's netrc are //! only when it is in the store and carries a signature by the swarm cache's
//! files a root unit keeps in [`ROOT_FACTS_DIR`]. Nothing in the request, and //! key, which `nix store verify` checks against the key in [`ROOT_FACTS_DIR`].
//! nothing hive-c0re (unprivileged) writes, chooses a cache or a key. They go to //! The key is a file a root unit keeps; nothing in the request, and nothing
//! nix's daemon as per-call settings, which the daemon applies for a trusted //! hive-c0re writes, chooses it.
//! client such as this one (root), so the host's nix configuration is untouched.
use std::os::unix::fs::MetadataExt as _; use std::os::unix::fs::MetadataExt as _;
use std::path::{Path, PathBuf}; use std::path::Path;
use anyhow::{Context as _, Result, bail}; use anyhow::{Context as _, Result, bail};
use hive_priv_sock::{SIGNATURE_REFUSED, valid_closure_path}; use hive_priv_sock::valid_closure_path;
use crate::container::{BoundedRun, run_bounded}; use crate::container::{BoundedRun, run_bounded};
use crate::shared::validate_agent_name; use crate::shared::{RealSpawn, Spawn, validate_agent_name};
/// The unit's environment variable naming the cache: `https://<domain>/<cache>`
/// from the host's `services.hyperhive.swarm.attic` options.
pub(crate) const CACHE_URL_ENV: &str = "HIVE_PRIV_CACHE_URL";
/// Where the root unit `hive-priv-cache-facts` keeps `public-key` and `netrc`. /// Where the root unit `hive-priv-cache-facts` keeps `public-key` and `netrc`.
/// Not `/run/hyperhive/attic`, which hive-c0re writes. /// Not `/run/hyperhive/attic`, which hive-c0re writes.
const ROOT_FACTS_DIR: &str = "/run/hyperhive-priv/attic"; const ROOT_FACTS_DIR: &str = "/run/hyperhive-priv/attic";
/// Wall-clock bound on one download. A stalled transfer is bounded by nix's /// The root unit that reads this hive's cache facts from the store.
/// own `stalled-download-timeout`; this covers the rest. const FACTS_UNIT: &str = "hive-priv-cache-facts.service";
const REALISE_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(30);
/// The cache this host trusts. /// Wall-clock bound on one `nix store verify`, which reads a path's metadata.
#[derive(Debug, PartialEq, Eq)] const VERIFY_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(1);
struct Cache {
substituter: String, /// Wall-clock bound on the facts unit: a store login and three small writes.
public_key: String, const REFRESH_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(1);
netrc: PathBuf,
}
/// Whether `path` is a regular file or directory owned by `owner` that nobody /// Whether `path` is a regular file or directory owned by `owner` that nobody
/// else can read or write. /// else can read or write.
@ -64,305 +56,180 @@ fn private_to(path: &Path, owner: u32, directory: bool) -> Result<()> {
Ok(()) Ok(())
} }
impl Cache { /// The swarm cache's public key in `dir`, as `<name>:<key>`.
/// The cache at `url`, trusting the key in `dir`.
///
/// # Errors
/// No `url`, one that is not a plain `https` address (the token would go
/// out in clear), a `dir` or file that is missing, not owned by `owner` or
/// open to others, or a key that is not `<name>:<key>`.
fn read(url: Option<&str>, dir: &Path, owner: u32) -> Result<Self> {
let url = url
.map(str::trim)
.filter(|u| !u.is_empty())
.context("this host names no cache")?;
if !url.starts_with("https://") || url.contains(char::is_whitespace) {
bail!("the cache address is not an https URL");
}
private_to(dir, owner, true)?;
let key_file = dir.join("public-key");
private_to(&key_file, owner, false)?;
let netrc = dir.join("netrc");
private_to(&netrc, owner, false)?;
let key = std::fs::read_to_string(&key_file)
.with_context(|| format!("reading {}", key_file.display()))?;
let key = key.trim();
if key.contains(char::is_whitespace)
|| key
.split_once(':')
.is_none_or(|(name, key)| name.is_empty() || key.is_empty())
{
bail!("the cache's public key is not <name>:<key>");
}
Ok(Self {
substituter: url.to_owned(),
public_key: key.to_owned(),
netrc,
})
}
}
/// The `nix-store` arguments that fetch `store_path` and nothing else.
///
/// - `--max-jobs 0` forbids a local build, so a path the cache lacks fails.
/// - `substituters` and `trusted-public-keys` are replaced, not extended: the
/// swarm cache is the only source and its key the only one that counts.
/// - `require-sigs true` makes nix refuse a path no trusted key signed.
fn realise_args(store_path: &str, cache: &Cache) -> Vec<String> {
let mut args: Vec<String> = ["--realise", store_path, "--max-jobs", "0"]
.map(str::to_owned)
.into();
let netrc = cache.netrc.display().to_string();
for (name, value) in [
("substituters", cache.substituter.as_str()),
("trusted-public-keys", cache.public_key.as_str()),
("require-sigs", "true"),
("netrc-file", netrc.as_str()),
] {
args.extend(["--option".to_owned(), name.to_owned(), value.to_owned()]);
}
args
}
/// Why nix did not realise the path, from its last stderr line. A refusal for
/// want of a signature is named, so the caller can tell it from a cache that
/// is down.
fn reason(stderr: &str) -> String {
let last = stderr.lines().last().unwrap_or("").trim();
if last.to_lowercase().contains("signature") {
format!("{SIGNATURE_REFUSED}: {last}")
} else {
last.to_owned()
}
}
/// `RealiseClosure`.
/// ///
/// # Errors /// # Errors
/// Bad arguments, or `not available: <reason>` for anything that left the path /// A `dir` or key file that is missing, not owned by `owner` or open to
/// out of the store: no cache or key on this host, nix failing or timing out. /// others, or a key that is not `<name>:<key>`.
pub(crate) async fn realise_closure(agent: &str, store_path: &str) -> Result<(String, String)> { fn swarm_key(dir: &Path, owner: u32) -> Result<String> {
let url = std::env::var(CACHE_URL_ENV).ok(); private_to(dir, owner, true)?;
let done = realise_with( let key_file = dir.join("public-key");
url.as_deref(), private_to(&key_file, owner, false)?;
Path::new(ROOT_FACTS_DIR), let key = std::fs::read_to_string(&key_file)
0, .with_context(|| format!("reading {}", key_file.display()))?;
agent, let key = key.trim();
store_path, if key.contains(char::is_whitespace)
) || key
.await?; .split_once(':')
if let Ok(mut realised) = REALISED.lock() { .is_none_or(|(name, key)| name.is_empty() || key.is_empty())
remember(&mut realised, agent, store_path); {
bail!("the cache's public key is not <name>:<key>");
} }
Ok(done) Ok(key.to_owned())
} }
/// The store path each agent's last successful `RealiseClosure` put in the /// The `nix` arguments that check `toplevel` is in the store and signed by
/// store. `CreateContainer` / `UpdateContainer` apply only that path for that /// `key`, and by no other key.
/// agent. hive-priv is a long-running process, so the entry outlives the
/// calls between the download and the apply; a restart in between empties it
/// and the apply is refused.
static REALISED: std::sync::Mutex<Vec<(String, String)>> = std::sync::Mutex::new(Vec::new());
fn remember(realised: &mut Vec<(String, String)>, agent: &str, store_path: &str) {
realised.retain(|(a, _)| a != agent);
realised.push((agent.to_owned(), store_path.to_owned()));
}
/// The path recorded for `agent`.
fn lookup<'a>(realised: &'a [(String, String)], agent: &str) -> Option<&'a str> {
realised
.iter()
.find(|(a, _)| a == agent)
.map(|(_, p)| p.as_str())
}
/// `toplevel` when it is a store path and the one `agent`'s last
/// `RealiseClosure` returned (`realised`); else the refusal, which is also
/// logged at error level with the agent and both paths.
/// ///
/// The request comes from hive-c0re, which is not root. A signed path is not /// - `--no-contents` checks signatures and validity, not the NAR hash.
/// enough: it could be another agent's closure or the host's own. /// - `--sigs-needed 1` with `trusted-public-keys` replaced by `key` means one
/// /// signature by the swarm cache's key; the host's other keys do not count.
/// # Errors fn verify_args(toplevel: &str, key: &str) -> Vec<String> {
/// `toplevel` is not a store path, or is not `realised`. [
pub(crate) fn applicable<'a>( "--extra-experimental-features",
agent: &str, "nix-command",
toplevel: &'a str, "store",
realised: Option<&str>, "verify",
) -> Result<&'a str> { "--no-contents",
"--sigs-needed",
"1",
"--option",
"trusted-public-keys",
key,
toplevel,
]
.map(str::to_owned)
.into()
}
/// What `nix store verify` reports through its exit status, as a bit set: 2 is
/// a path no trusted key signed, 4 a path that is not in the store.
fn verdict(code: Option<i32>, toplevel: &str, stderr: &str) -> Result<()> {
match code {
Some(0) => Ok(()),
Some(c) if c & 4 != 0 => bail!("{toplevel} is not in the store"),
Some(c) if c & 2 != 0 => bail!("{toplevel} has no signature by the swarm cache's key"),
_ => bail!(
"nix store verify failed: {}",
stderr.lines().last().unwrap_or("").trim()
),
}
}
/// Whether `toplevel` is in the store with a signature by the key in `dir`.
async fn check_signed(spawn: &impl Spawn, dir: &Path, owner: u32, toplevel: &str) -> Result<()> {
if !valid_closure_path(toplevel) { if !valid_closure_path(toplevel) {
bail!("{toplevel:?} is not a store path"); bail!("{toplevel:?} is not a store path");
} }
if realised == Some(toplevel) { let key = swarm_key(dir, owner)?;
return Ok(toplevel); let mut cmd = spawn.command("nix");
cmd.args(verify_args(toplevel, &key));
match run_bounded(cmd, VERIFY_TIMEOUT)
.await
.context("running nix store verify")?
{
BoundedRun::Exited { status, stderr } => verdict(status.code(), toplevel, &stderr),
BoundedRun::TimedOut => bail!(
"nix store verify timed out after {}s; killed it",
VERIFY_TIMEOUT.as_secs()
),
} }
let realised = realised.unwrap_or("none");
tracing::error!(
%agent, requested = %toplevel, %realised,
"refusing to apply a closure that is not the one realised for the agent"
);
bail!("{agent}: {toplevel} is not the closure realised for this agent ({realised})")
} }
/// [`applicable`] against what `RealiseClosure` recorded for `agent`. /// `toplevel` when it is a store path that is in the store with a signature by
/// the swarm cache's key; else the refusal, which is also logged at error
/// level with the agent and the path.
///
/// The request comes from hive-c0re, which is not root and can ask for any
/// path of that shape. A path in the store is not enough: a build of the host's
/// own, or a path from another cache, carries no signature by this key.
/// ///
/// # Errors /// # Errors
/// As [`applicable`]. /// `toplevel` is not a store path, is not in the store, or carries no
pub(crate) fn applicable_for<'a>(agent: &str, toplevel: &'a str) -> Result<&'a str> { /// signature by the swarm cache's key, or the key cannot be read.
let realised = REALISED pub(crate) async fn applicable_for<'a>(agent: &str, toplevel: &'a str) -> Result<&'a str> {
.lock() applicable_with(&RealSpawn, Path::new(ROOT_FACTS_DIR), 0, agent, toplevel).await
.ok()
.and_then(|r| lookup(&r, agent).map(str::to_owned));
applicable(agent, toplevel, realised.as_deref())
} }
/// [`realise_closure`] with the cache's address and the key's directory and async fn applicable_with<'a>(
/// owner named, so a test can stand in for the host. spawn: &impl Spawn,
async fn realise_with(
url: Option<&str>,
dir: &Path, dir: &Path,
owner: u32, owner: u32,
agent: &str, agent: &str,
store_path: &str, toplevel: &'a str,
) -> Result<(String, String)> { ) -> Result<&'a str> {
match check_signed(spawn, dir, owner, toplevel).await {
Ok(()) => Ok(toplevel),
Err(e) => {
tracing::error!(
%agent, requested = %toplevel, error = %format!("{e:#}"),
"refusing to apply a closure the swarm cache did not sign"
);
bail!("{agent}: refusing {toplevel}: {e:#}")
}
}
}
/// `RealiseClosure`, from an older hive-c0re: nothing is downloaded. `Ok` when
/// the path is already what an apply requires, else `not available: <reason>`.
///
/// # Errors
/// Bad arguments, or `not available: <reason>`.
pub(crate) async fn realise_closure(agent: &str, store_path: &str) -> Result<(String, String)> {
validate_agent_name(agent)?; validate_agent_name(agent)?;
if !valid_closure_path(store_path) { if !valid_closure_path(store_path) {
bail!("{store_path:?} is not a store path"); bail!("{store_path:?} is not a store path");
} }
let outcome = async { match check_signed(&RealSpawn, Path::new(ROOT_FACTS_DIR), 0, store_path).await {
let cache = Cache::read(url, dir, owner)?; Ok(()) => Ok(("present".to_owned(), String::new())),
let mut cmd = tokio::process::Command::new("nix-store"); Err(e) => bail!(
cmd.args(realise_args(store_path, &cache)); "not available: {e:#}; hive-priv no longer downloads, hive-c0re pulls through the nix daemon"
match run_bounded(cmd, REALISE_TIMEOUT) ),
.await
.context("running nix-store")?
{
BoundedRun::Exited { status, .. } if status.success() => Ok(()),
BoundedRun::Exited { stderr, .. } => bail!("{}", reason(&stderr)),
BoundedRun::TimedOut => bail!(
"nix-store timed out after {}s; killed it",
REALISE_TIMEOUT.as_secs()
),
}
} }
.await; }
match outcome {
Ok(()) => { /// `RefreshCacheFacts`: start the root unit that reads this hive's cache facts
tracing::info!(%agent, %store_path, "substituted the agent's closure"); /// from the store and writes the nix.conf fragment, and wait for it. The unit
Ok(("substituted".to_owned(), String::new())) /// is fixed here, not named by the request.
} ///
Err(e) => { /// # Errors
let reason = format!("{e:#}"); /// The unit failed to start or did not finish within [`REFRESH_TIMEOUT`].
if reason.contains(SIGNATURE_REFUSED) { pub(crate) async fn refresh_cache_facts() -> Result<(String, String)> {
tracing::error!(%agent, %store_path, %reason, "the cache's closure was refused"); refresh_with(&RealSpawn, REFRESH_TIMEOUT).await
} }
bail!("not available: {reason}")
async fn refresh_with(spawn: &impl Spawn, limit: std::time::Duration) -> Result<(String, String)> {
let mut cmd = spawn.command("systemctl");
cmd.args(["start", FACTS_UNIT]);
match run_bounded(cmd, limit)
.await
.with_context(|| format!("running systemctl start {FACTS_UNIT}"))?
{
BoundedRun::Exited { status, .. } if status.success() => {
tracing::info!("refreshed the cache facts");
Ok((String::new(), String::new()))
} }
BoundedRun::Exited { status, stderr } => bail!(
"systemctl start {FACTS_UNIT} exited {status}: {}",
stderr.lines().last().unwrap_or("").trim()
),
BoundedRun::TimedOut => bail!(
"{FACTS_UNIT} did not finish within {}s; killed the wait",
limit.as_secs()
),
} }
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::os::unix::fs::PermissionsExt as _; use std::os::unix::fs::PermissionsExt as _;
use std::path::PathBuf;
use super::*; use super::*;
use crate::shared::test_support::scratch; use crate::shared::test_support::{ScriptSpawn, scratch};
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas"; const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const URL: &str = "https://cache.example/swarm"; const KEY: &str = "swarm:AAAA";
fn cache() -> Cache {
Cache {
substituter: URL.to_owned(),
public_key: "swarm:AAAA".to_owned(),
netrc: PathBuf::from("/run/hyperhive-priv/attic/netrc"),
}
}
const OTHER: &str = "/nix/store/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-nixos-system-iris";
#[test]
fn the_path_realised_for_the_agent_is_applied() {
assert_eq!(applicable("atlas", PATH, Some(PATH)).unwrap(), PATH);
}
/// Another agent's realised path, a valid-shaped path never realised, and
/// a restart that emptied the record are all refused, naming the agent
/// and both paths.
#[test]
fn any_other_path_is_refused() {
for (case, realised) in [
("another path was realised for the agent", Some(OTHER)),
("nothing was realised for the agent", None),
] {
let err = applicable("atlas", PATH, realised).unwrap_err();
let msg = format!("{err:#}");
assert!(
msg.contains("atlas")
&& msg.contains(PATH)
&& msg.contains(realised.unwrap_or("none")),
"{case}: {msg}"
);
}
}
#[test]
fn only_a_store_path_is_applied_even_when_it_was_recorded() {
for bad in [
"",
"main",
"/etc/passwd",
"/nix/store/x",
&format!("{PATH}/bin"),
] {
let err = applicable("atlas", bad, Some(bad)).unwrap_err();
assert!(format!("{err:#}").contains("not a store path"), "{bad}");
}
}
/// Each agent keeps its own entry, and a newer download replaces the
/// older one.
#[test]
fn a_record_is_per_agent_and_the_latest_wins() {
let mut realised = Vec::new();
remember(&mut realised, "atlas", PATH);
remember(&mut realised, "iris", OTHER);
assert_eq!(lookup(&realised, "atlas"), Some(PATH));
assert_eq!(lookup(&realised, "iris"), Some(OTHER));
remember(&mut realised, "atlas", OTHER);
assert_eq!(lookup(&realised, "atlas"), Some(OTHER));
assert_eq!(lookup(&realised, "nobody"), None);
assert_eq!(realised.len(), 2);
}
fn option<'a>(args: &'a [String], name: &str) -> Option<&'a str> {
args.windows(3)
.find(|w| w[0] == "--option" && w[1] == name)
.map(|w| w[2].as_str())
}
#[test]
fn the_command_never_builds() {
let args = realise_args(PATH, &cache());
assert_eq!(args[..2], ["--realise", PATH]);
let jobs = args.iter().position(|a| a == "--max-jobs").expect("flag");
assert_eq!(args[jobs + 1], "0");
}
#[test]
fn only_the_swarm_cache_and_its_key_count_and_signatures_are_required() {
let args = realise_args(PATH, &cache());
assert_eq!(option(&args, "substituters"), Some(URL));
assert_eq!(option(&args, "trusted-public-keys"), Some("swarm:AAAA"));
assert_eq!(option(&args, "require-sigs"), Some("true"));
assert_eq!(
option(&args, "netrc-file"),
Some("/run/hyperhive-priv/attic/netrc")
);
// The settings that would add to the defaults are not used.
assert!(!args.iter().any(|a| a.starts_with("extra-")), "{args:?}");
}
/// The uid the test runs as, which stands in for root. /// The uid the test runs as, which stands in for root.
fn me() -> u32 { fn me() -> u32 {
@ -387,126 +254,169 @@ mod tests {
dir dir
} }
#[test] /// A `nix` that behaves like `nix store verify --sigs-needed 1` over a store
fn the_cache_is_the_hosts_address_and_the_root_files_key() { /// holding `PATH`, signed by `KEY` only: exit 0 for that path under that key,
let dir = facts("swarm:AAAA"); /// 2 for that path under any other key, 4 for any other path. It exits 99
let cache = Cache::read(Some(URL), &dir, me()).expect("read"); /// for anything but the verify invocation, and logs its arguments.
assert_eq!(cache.substituter, URL); fn nix(log: &Path) -> ScriptSpawn {
assert_eq!(cache.public_key, "swarm:AAAA"); ScriptSpawn(format!(
assert_eq!(cache.netrc, dir.join("netrc")); r#"echo "$@" >>{log}
[ "$1 $2 $3 $4 $5" = "--extra-experimental-features nix-command store verify --no-contents" ] || exit 99
[ "$6 $7 $8 $9" = "--sigs-needed 1 --option trusted-public-keys" ] || exit 99
[ "${{11}}" = {PATH} ] || exit 4
[ "${{10}}" = {KEY} ] || exit 2
exit 0"#,
log = log.display()
))
} }
/// A `substituter` or `public-key` file beside the facts, as hive-c0re #[tokio::test]
/// writes them, decides nothing: the address is the argument and the key is async fn a_toplevel_signed_by_the_swarm_key_is_applied() {
/// the root file's. let log = scratch().join("calls");
#[test] let got = applicable_with(&nix(&log), &facts(KEY), me(), "atlas", PATH)
fn the_address_is_never_read_from_a_file_and_the_key_only_from_the_root_files() { .await
let dir = facts("swarm:ROOT"); .expect("signed");
put( assert_eq!(got, PATH);
&dir.join("substituter"), // The key checked against is the one in the root file.
"https://evil.example/cache\n", let calls = std::fs::read_to_string(&log).expect("log");
0o600, assert!(
calls.contains(&format!("trusted-public-keys {KEY} {PATH}")),
"{calls}"
); );
let cache = Cache::read(Some(URL), &dir, me()).expect("read");
assert_eq!(cache.substituter, URL);
assert_eq!(cache.public_key, "swarm:ROOT");
// The directory hive-c0re writes is not the one this reads.
assert!(!ROOT_FACTS_DIR.starts_with("/run/hyperhive/"));
assert_eq!(ROOT_FACTS_DIR, "/run/hyperhive-priv/attic");
// And without an address from the host there is no cache, whatever the
// directory holds.
for none in [None, Some(""), Some(" ")] {
assert!(Cache::read(none, &dir, me()).is_err(), "{none:?}");
}
} }
#[test] /// A path no swarm key signed, one the store lacks, and a key file that
fn a_file_that_is_not_the_roots_or_is_open_to_others_is_refused() { /// names another key are each refused, naming the agent and the path.
let me = me(); #[tokio::test]
// Control: the facts as the unit leaves them are accepted. async fn an_unsigned_or_absent_toplevel_is_refused() {
assert!(Cache::read(Some(URL), &facts("swarm:AAAA"), me).is_ok()); let log = scratch().join("calls");
// Another owner than the one trusted. let other = "/nix/store/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-nixos-system-iris";
assert!(Cache::read(Some(URL), &facts("swarm:AAAA"), me + 1).is_err()); for (case, dir, path, want) in [
// The key or the netrc readable by others, or the directory open. ("absent", facts(KEY), other, "is not in the store"),
for (name, mode) in [("public-key", 0o644), ("netrc", 0o644), ("netrc", 0o660)] { (
let dir = facts("swarm:AAAA"); "unsigned by the key on file",
std::fs::set_permissions(dir.join(name), std::fs::Permissions::from_mode(mode)) facts("other:BBBB"),
.expect("chmod"); PATH,
assert!(Cache::read(Some(URL), &dir, me).is_err(), "{name} {mode:o}"); "has no signature by the swarm cache's key",
} ),
let dir = facts("swarm:AAAA"); ] {
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)).expect("chmod"); let err = applicable_with(&nix(&log), &dir, me(), "atlas", path)
assert!(Cache::read(Some(URL), &dir, me).is_err()); .await
// A symlink in place of the key. .expect_err(case);
let dir = facts("swarm:AAAA"); let msg = format!("{err:#}");
std::fs::remove_file(dir.join("public-key")).expect("rm");
std::os::unix::fs::symlink(dir.join("netrc"), dir.join("public-key")).expect("link");
assert!(Cache::read(Some(URL), &dir, me).is_err());
}
#[test]
fn missing_files_and_unusable_values_are_refused() {
let me = me();
for url in ["http://cache.example/swarm", "https://cache.example/a b"] {
assert!( assert!(
Cache::read(Some(url), &facts("swarm:AAAA"), me).is_err(), msg.contains("atlas") && msg.contains(path) && msg.contains(want),
"{url}" "{case}: {msg}"
); );
} }
for key in ["nokeyname", "swarm: AAAA", ":AAAA", "swarm:"] {
assert!(Cache::read(Some(URL), &facts(key), me).is_err(), "{key:?}");
}
let dir = facts("swarm:AAAA");
std::fs::remove_file(dir.join("netrc")).expect("rm");
assert!(Cache::read(Some(URL), &dir, me).is_err());
assert!(Cache::read(Some(URL), &dir.join("nowhere"), me).is_err());
} }
/// Missing facts reach the caller as `not available`, which a rebuild /// A tool failure other than the two verdicts is a refusal too, with the
/// reports as a closure that cannot be pulled. /// tool's last line.
#[tokio::test] #[tokio::test]
async fn without_the_root_files_the_answer_is_not_available() { async fn a_failing_verify_refuses() {
let nowhere = scratch().join("nowhere"); let spawn = ScriptSpawn("echo 'error: cannot connect' >&2; exit 1".to_owned());
let err = realise_with(Some(URL), &nowhere, 0, "atlas", PATH) let err = applicable_with(&spawn, &facts(KEY), me(), "atlas", PATH)
.await .await
.expect_err("no facts"); .expect_err("refused");
assert!(format!("{err:#}").starts_with("not available: "), "{err:#}"); assert!(format!("{err:#}").contains("cannot connect"), "{err:#}");
let err = realise_with(None, &facts("swarm:AAAA"), me(), "atlas", PATH)
.await
.expect_err("no address");
assert!(format!("{err:#}").starts_with("not available: "), "{err:#}");
} }
/// The request carries an agent and a path and nothing that chooses a #[tokio::test]
/// cache: fields beyond those are dropped when it is read. async fn only_a_store_path_is_verified() {
#[test] // A stand-in that would accept anything, so only the shape check refuses.
fn the_request_cannot_carry_an_address_or_a_key() { let yes = ScriptSpawn("exit 0".to_owned());
let line = r#"{"op":"realise_closure","agent":"atlas","store_path":"/nix/store/x", for bad in [
"substituter":"https://evil.example/c","public_key":"evil:AAAA","netrc":"/tmp/x"}"#; "",
let hive_priv_sock::PrivRequest::RealiseClosure { agent, store_path } = "main",
serde_json::from_str(line).expect("reads") "/etc/passwd",
else { "/nix/store/x",
panic!("not a RealiseClosure"); &format!("{PATH}/bin"),
}; ] {
assert_eq!( let err = applicable_with(&yes, &facts(KEY), me(), "atlas", bad)
(agent.as_str(), store_path.as_str()), .await
("atlas", "/nix/store/x") .expect_err(bad);
assert!(format!("{err:#}").contains("not a store path"), "{bad}");
}
// Control: the same stand-in accepts a store path.
assert!(
applicable_with(&yes, &facts(KEY), me(), "atlas", PATH)
.await
.is_ok()
); );
} }
#[test] /// Without the root key there is nothing to verify against: refused, and
fn a_signature_refusal_is_named_and_another_failure_is_not() { /// nix is not asked.
let refused = #[tokio::test]
reason("error: cannot add path 'x' because it lacks a signature by a trusted key"); async fn without_the_root_key_nothing_is_applied() {
assert!(refused.starts_with(SIGNATURE_REFUSED), "{refused}"); let log = scratch().join("calls");
let down = let nowhere = scratch().join("nowhere");
reason("warning: x\nerror: unable to download 'https://c/x.narinfo': Couldn't connect"); let err = applicable_with(&nix(&log), &nowhere, 0, "atlas", PATH)
assert!(!down.contains(SIGNATURE_REFUSED), "{down}"); .await
assert!(down.contains("Couldn't connect"), "{down}"); .expect_err("no key");
assert!(format!("{err:#}").contains("missing"), "{err:#}");
assert!(!log.exists(), "nix ran without a key");
} }
#[test]
fn the_key_is_the_root_files_and_only_a_private_one() {
let me = me();
assert_eq!(swarm_key(&facts(KEY), me).unwrap(), KEY);
// Another owner than the one trusted.
assert!(swarm_key(&facts(KEY), me + 1).is_err());
// The key open to others, or the directory open.
let dir = facts(KEY);
std::fs::set_permissions(
dir.join("public-key"),
std::fs::Permissions::from_mode(0o644),
)
.expect("chmod");
assert!(swarm_key(&dir, me).is_err());
let dir = facts(KEY);
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)).expect("chmod");
assert!(swarm_key(&dir, me).is_err());
// A symlink in place of the key.
let dir = facts(KEY);
std::fs::remove_file(dir.join("public-key")).expect("rm");
std::os::unix::fs::symlink(dir.join("netrc"), dir.join("public-key")).expect("link");
assert!(swarm_key(&dir, me).is_err());
for key in ["nokeyname", "swarm: AAAA", ":AAAA", "swarm:"] {
assert!(swarm_key(&facts(key), me).is_err(), "{key:?}");
}
// The directory hive-c0re writes is not the one this reads.
assert_eq!(ROOT_FACTS_DIR, "/run/hyperhive-priv/attic");
}
#[test]
fn the_command_counts_only_the_swarm_key() {
let args = verify_args(PATH, KEY);
assert_eq!(args.last().map(String::as_str), Some(PATH));
let at = args
.iter()
.position(|a| a == "--sigs-needed")
.expect("flag");
assert_eq!(args[at + 1], "1");
let at = args
.iter()
.position(|a| a == "trusted-public-keys")
.expect("option");
assert_eq!(args[at - 1], "--option");
assert_eq!(args[at + 1], KEY);
// The setting that would add to the host's keys is not used.
assert!(
!args.iter().any(|a| a.starts_with("extra-trusted")),
"{args:?}"
);
}
/// An older hive-c0re's download request downloads nothing: it succeeds
/// only for a path an apply would accept.
#[tokio::test] #[tokio::test]
async fn bad_arguments_are_refused_before_anything_runs() { async fn the_older_download_request_answers_not_available_for_a_path_not_in_the_store() {
// The real `nix` is not on the path of the test, so this is refused for
// want of the root key before anything runs; the shape and name checks
// come first.
for (agent, path) in [ for (agent, path) in [
("Bad Agent", PATH), ("Bad Agent", PATH),
("", PATH), ("", PATH),
@ -517,5 +427,34 @@ mod tests {
let err = realise_closure(agent, path).await.expect_err("refused"); let err = realise_closure(agent, path).await.expect_err("refused");
assert!(!format!("{err:#}").contains("not available"), "{err:#}"); assert!(!format!("{err:#}").contains("not available"), "{err:#}");
} }
let err = realise_closure("atlas", PATH).await.expect_err("no key");
let msg = format!("{err:#}");
assert!(
msg.starts_with("not available: ") && msg.contains("no longer downloads"),
"{msg}"
);
}
/// The refresh starts the fixed unit and nothing else, and says why it
/// failed or timed out.
#[tokio::test]
async fn the_refresh_starts_the_facts_unit_and_reports_a_failure() {
let log = scratch().join("calls");
let ok = ScriptSpawn(format!("echo \"$0 $@\" >>{}", log.display()));
refresh_with(&ok, REFRESH_TIMEOUT).await.expect("started");
assert_eq!(
std::fs::read_to_string(&log).expect("log").trim(),
"systemctl start hive-priv-cache-facts.service"
);
let failing = ScriptSpawn("echo 'Unit not found.' >&2; exit 5".to_owned());
let err = refresh_with(&failing, REFRESH_TIMEOUT)
.await
.expect_err("failed");
assert!(format!("{err:#}").contains("Unit not found."), "{err:#}");
let slow = ScriptSpawn("sleep 30".to_owned());
let err = refresh_with(&slow, std::time::Duration::from_millis(100))
.await
.expect_err("timed out");
assert!(format!("{err:#}").contains("did not finish"), "{err:#}");
} }
} }

24
hive-priv/src/container.rs
View file

@ -207,9 +207,10 @@ async fn destroy_container_with(host: &Host<impl Spawn>, name: &str) -> Result<(
} }
/// Shared body for `CreateContainer` / `UpdateContainer`: validate the name /// Shared body for `CreateContainer` / `UpdateContainer`: validate the name
/// and that `toplevel` is the closure realised for the agent, and apply it (streaming line events to `writer` when /// and that `toplevel` is in the store with a signature by the swarm cache's
/// `stream` is set). Nothing is built here: `toplevel` is a closure already in /// key, and apply it (streaming line events to `writer` when `stream` is set).
/// the store, which `RealiseClosure` put there. /// Nothing is built here: `toplevel` is a closure already in the store, which
/// hive-c0re pulled through the nix daemon.
/// ///
/// `create` passes `--system-path` so `nixos-container`'s own `buildFlake()` /// `create` passes `--system-path` so `nixos-container`'s own `buildFlake()`
/// never runs; it builds to a relative `.tmp` out-link that concurrent calls /// never runs; it builds to a relative `.tmp` out-link that concurrent calls
@ -228,12 +229,12 @@ pub(crate) async fn container_toplevel_action(
writer: &mut OwnedWriteHalf, writer: &mut OwnedWriteHalf,
) -> Result<(String, String)> { ) -> Result<(String, String)> {
validate_agent_name(name)?; validate_agent_name(name)?;
let toplevel = crate::closure::applicable_for(name, toplevel)?; let toplevel = crate::closure::applicable_for(name, toplevel).await?;
apply_toplevel(&Host::REAL, verb, name, toplevel, stream, writer).await apply_toplevel(&Host::REAL, verb, name, toplevel, stream, writer).await
} }
/// [`container_toplevel_action`] past the check that `toplevel` is the /// [`container_toplevel_action`] past the check that `toplevel` is signed by
/// closure realised for `name`. /// the swarm cache's key.
async fn apply_toplevel( async fn apply_toplevel(
host: &Host<impl Spawn>, host: &Host<impl Spawn>,
verb: &str, verb: &str,
@ -987,11 +988,12 @@ mod host_tests {
// ---- the apply gate ---- // ---- the apply gate ----
/// Only the closure realised for the agent is applied. Nothing is /// Only a closure the swarm key signed is applied. A test has no swarm key
/// realised for these names, so each request is refused at the gate: the /// at the root path, so each request is refused at the gate: the message
/// message is the gate's, not a `nixos-container` failure. /// is the gate's, not a `nixos-container` failure. The signed and unsigned
/// verdicts are tested in `closure`.
#[tokio::test] #[tokio::test]
async fn an_unrealised_or_foreign_closure_is_refused_at_the_gate() { async fn a_closure_without_the_swarm_key_to_check_it_against_is_refused_at_the_gate() {
for verb in ["create", "update"] { for verb in ["create", "update"] {
let (writer, _peer) = &mut pipe(); let (writer, _peer) = &mut pipe();
let err = container_toplevel_action(verb, "never-realised", TOPLEVEL, false, writer) let err = container_toplevel_action(verb, "never-realised", TOPLEVEL, false, writer)
@ -999,7 +1001,7 @@ mod host_tests {
.unwrap_err(); .unwrap_err();
let msg = format!("{err:#}"); let msg = format!("{err:#}");
assert!( assert!(
msg.contains("is not the closure realised for this agent (none)"), msg.contains(&format!("never-realised: refusing {TOPLEVEL}")),
"{verb}: {msg}" "{verb}: {msg}"
); );
} }

6
hive-priv/src/main.rs
View file

@ -24,8 +24,8 @@
//! [`prestart`] (the check of a container's root that runs before each start, //! [`prestart`] (the check of a container's root that runs before each start,
//! as a sub-command), [`storage`] (btrfs subvolumes, quotas, snapshots, disk //! as a sub-command), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles), //! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles),
//! [`infra`] (gateway nginx, infra containers) and [`closure`] (downloads from //! [`infra`] (gateway nginx, infra containers) and [`closure`] (the swarm-signature
//! the swarm cache). Helpers more than one family uses are in [`shared`]. //! check before an apply, and the cache-facts refresh). Helpers more than one family uses are in [`shared`].
mod closure; mod closure;
mod container; mod container;
@ -403,6 +403,8 @@ async fn exec(
ref store_path, ref store_path,
} => closure::realise_closure(agent, store_path).await, } => closure::realise_closure(agent, store_path).await,
PrivRequest::RefreshCacheFacts => closure::refresh_cache_facts().await,
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await, PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,
PrivRequest::ListContainers => container::container_run(&["list"]).await, PrivRequest::ListContainers => container::container_run(&["list"]).await,

8
nix/host-modules/hive-c0re/default.nix
View file

@ -315,9 +315,11 @@ in
# StateDirectory (/var/lib/hyperhive), RuntimeDirectory # StateDirectory (/var/lib/hyperhive), RuntimeDirectory
# (/run/hyperhive), and ReadWritePaths below, which systemd keeps # (/run/hyperhive), and ReadWritePaths below, which systemd keeps
# writable. Beyond the managed directories: # writable. Beyond the managed directories:
# - nix is invoked directly (lifecycle, meta, flake_check), but # - the one nix call is the closure pull (src/cache_pull.rs):
# NIX_REMOTE=daemon routes all store writes through the host # `nix-store --realise` with NIX_REMOTE=daemon, so the host daemon
# daemon — hive-c0re never writes to /nix itself. # writes the store — hive-c0re never writes to /nix itself. It
# connects to the daemon's socket, which RestrictAddressFamilies
# (deferred above) must keep allowing.
# - nix build worktrees live in PrivateTmp, not /tmp. # - nix build worktrees live in PrivateTmp, not /tmp.
# - /etc writes (bind-mount edits) go through hive-priv via the # - /etc writes (bind-mount edits) go through hive-priv via the

43
nix/host-modules/hive-priv-cache.nix
View file

@ -1,18 +1,21 @@
# The binary-cache facts hive-priv (root) trusts, kept where only root can # The binary-cache facts the host's nix daemon and hive-priv (root) trust, kept
# change them. # where only root can change them.
# #
# hive-priv downloads an agent's closure from the swarm cache when a rebuild # hive-c0re downloads an agent's closure from the swarm cache through the
# asks it to. What decides whether a download is accepted — which cache, and # host's nix daemon, and hive-priv applies it only if the swarm's key signed
# which key must have signed it — must not come from a file the unprivileged # it. What decides whether a download is accepted — which cache the daemon
# hive-c0re writes, or a compromised hive-c0re could have root accept a # lets an unprivileged client name, and which key must have signed it — must
# closure from a cache of its own. So: # not come from a file the unprivileged hive-c0re writes, or a compromised
# hive-c0re could have root accept a closure from a cache of its own. So:
# #
# - the cache's address is this host's configuration # - the cache's address is this host's configuration
# ({option}`services.hyperhive.swarm.attic.domain` and `cache`), handed to # ({option}`services.hyperhive.swarm.attic.domain` and `cache`);
# hive-priv in its unit's environment;
# - the cache's public key and the pull token are read from the store by a root # - the cache's public key and the pull token are read from the store by a root
# oneshot, `hive-priv-cache-facts`, under this hive's own store identity, and # oneshot, `hive-priv-cache-facts`, under this hive's own store identity. The
# land in `/run/hyperhive-priv/attic/` (root, `0700`; files `0600`). # key and the token land in `/run/hyperhive-priv/attic/` (root, `0700`; files
# `0600`), and the address, the key and the netrc's path in a nix.conf
# fragment, `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
# includes (`!include`, so a host whose unit has not run yet still starts).
# #
# hive-c0re keeps its own copies under `/run/hyperhive/attic/` for the probe, # hive-c0re keeps its own copies under `/run/hyperhive/attic/` for the probe,
# which only reports. # which only reports.
@ -42,6 +45,9 @@ let
&& tls.baoHiveIdentityKeyFile != null; && tls.baoHiveIdentityKeyFile != null;
dir = "/run/hyperhive-priv/attic"; dir = "/run/hyperhive-priv/attic";
confDir = "/run/hyperhive-nix";
confFile = "${confDir}/cache.conf";
url = "https://${atticCfg.domain}/${atticCfg.cache}";
# Where the store runs on this host, the first login waits for its container. # Where the store runs on this host, the first login waits for its container.
# No `wants`: a pull would start the container mid-switch on the old closure. # No `wants`: a pull would start the container mid-switch on the old closure.
@ -52,15 +58,16 @@ let
in in
{ {
config = lib.mkIf enabled { config = lib.mkIf enabled {
systemd.tmpfiles.rules = [ "d /run/hyperhive-priv 0700 root root - -" ]; systemd.tmpfiles.rules = [
"d /run/hyperhive-priv 0700 root root - -"
"d ${confDir} 0755 root root - -"
];
# The URL nix is told is the only substituter for the download; no request # `!include` tolerates a fragment that does not exist yet.
# chooses it. nix.extraOptions = "!include ${confFile}";
systemd.services.hive-priv.environment.HIVE_PRIV_CACHE_URL =
"https://${atticCfg.domain}/${atticCfg.cache}";
systemd.services.hive-priv-cache-facts = { systemd.services.hive-priv-cache-facts = {
description = "Read the swarm cache's public key and pull token for hive-priv"; description = "Read the swarm cache's public key and pull token for the host's nix daemon and hive-priv";
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ]; wants = [ "network-online.target" ];
after = [ after = [
@ -94,7 +101,7 @@ in
script = import ./lib/attic-root-facts.nix { inherit lib; } { script = import ./lib/attic-root-facts.nix { inherit lib; } {
hive = hiveController.hiveName; hive = hiveController.hiveName;
host = atticCfg.domain; host = atticCfg.domain;
inherit dir; inherit dir url confFile;
}; };
}; };

4
nix/host-modules/hive-priv.nix
View file

@ -124,8 +124,8 @@ in
# hive-priv runs as root. Root nix defaults to store=auto which # hive-priv runs as root. Root nix defaults to store=auto which
# resolves to the LOCAL store — bypassing the host daemon, its # resolves to the LOCAL store — bypassing the host daemon, its
# remote builders, and prebuilt derivation outputs. Force daemon # remote builders, and prebuilt derivation outputs. Force daemon
# routing so the closure `RealiseClosure` downloads is in the same # routing so the closure hive-c0re pulls through the daemon is in
# store the container is then pointed at. # the same store the container is then pointed at.
NIX_REMOTE = "daemon"; NIX_REMOTE = "daemon";
# The owner a namespaced agent's data moves to. Which agents are # The owner a namespaced agent's data moves to. Which agents are
# namespaced is not set here: hive-c0re sends it with each start. # namespaced is not set here: hive-c0re sends it with each start.

47
nix/host-modules/lib/attic-root-facts.nix
View file

@ -1,22 +1,26 @@
# The root-side copy of the two cache facts hive-priv trusts: the key a # The root-side copy of the cache facts the host's nix daemon and hive-priv
# downloaded closure must be signed by and the netrc the pull token goes out # trust: the key a downloaded closure must be signed by and the netrc the pull
# in. The script runs as root on the hive host under the hive's own store # token goes out in. The script runs as root on the hive host under the hive's
# identity and writes both into `dir`, which only root can read. # own store identity and writes both into `dir`, which only root can read, and
# a nix.conf fragment at `confFile` that the host's nix.conf includes.
# #
# Everything hive-priv decides on is here or in the host's configuration, and # What the daemon and hive-priv decide on is here or in the host's
# nothing is read from a file hive-c0re (unprivileged) writes: the key and the # configuration, and nothing is read from a file hive-c0re (unprivileged)
# token come from the store, the cache's address from # writes: the key and the token come from the store, the cache's address from
# `services.hyperhive.swarm.attic.{domain,cache}`. # `services.hyperhive.swarm.attic.{domain,cache}`. The fragment holds no secret
# (the key is public, the netrc stays in `dir`) and hive-c0re can read it: it
# takes the cache's address from there.
# #
# A store that holds no facts yet, or cannot be reached, leaves what is in `dir` # A store that holds no facts yet, or cannot be reached, leaves what is in `dir`
# as it is. Nothing here removes a file. # and `confFile` as they are. Nothing here removes a file.
# #
# Pure function — NOT a NixOS module. Call it from a unit's `script`, with # Pure function — NOT a NixOS module. Call it from a unit's `script`, with
# `bao`, `coreutils` and `gnugrep` on its `path` and the `BAO_*` client # `bao`, `coreutils`, `gnugrep` and `systemd` on its `path` and the `BAO_*`
# environment set: # client environment set:
# #
# script = import ./lib/attic-root-facts.nix { inherit lib; } { # script = import ./lib/attic-root-facts.nix { inherit lib; } {
# hive = "alpha"; host = "attic.example.org"; dir = "/run/hyperhive-priv/attic"; # hive = "alpha"; host = "attic.example.org"; dir = "/run/hyperhive-priv/attic";
# url = "https://attic.example.org/swarm"; confFile = "/run/hyperhive-nix/cache.conf";
# }; # };
{ lib }: { lib }:
{ {
@ -25,8 +29,12 @@
hive, hive,
# The cache server's host name, which the netrc line names. # The cache server's host name, which the netrc line names.
host, host,
# Where the two files go. # Where the key and the netrc go.
dir, dir,
# The cache's address as the daemon lists it in `trusted-substituters`.
url,
# Where the nix.conf fragment goes; its directory is root's and readable.
confFile,
}: }:
'' ''
set -euo pipefail set -euo pipefail
@ -35,7 +43,9 @@
${import ./refresh-consumer.nix { }} ${import ./refresh-consumer.nix { }}
dir=${lib.escapeShellArg dir} dir=${lib.escapeShellArg dir}
conf=${lib.escapeShellArg confFile}
install -d -m 0700 "$dir" install -d -m 0700 "$dir"
install -d -m 0755 "$(dirname -- "$conf")"
store_init store_init
store_login store_login
@ -84,4 +94,17 @@
if secret_differs "$dir/netrc" "$netrc"; then if secret_differs "$dir/netrc" "$netrc"; then
atomic_write_secret 0600 "" "$dir/netrc" "$netrc" atomic_write_secret 0600 "" "$dir/netrc" "$netrc"
fi fi
# Last, so a daemon that reads the fragment finds the netrc in place. The
# address is the one hive-c0re may name as the substituter. The mtime moves
# only with the content. Lix re-reads nix.conf for every connection, CppNix
# only at start, so a daemon that started before this write is restarted.
printf -v fragment '%s\n%s\n%s' \
${lib.escapeShellArg "extra-trusted-substituters = ${url}"} \
"extra-trusted-public-keys = $key" \
"netrc-file = $dir/netrc"
if secret_differs "$conf" "$fragment"; then
atomic_write_secret 0644 "" "$conf" "$fragment"
fi
refresh_unit nix-daemon.service "$conf"
'' ''

19
nix/module-eval/hive-tls.nix
View file

@ -878,10 +878,23 @@ let
inner.ProcSubset == "all" && inner.ProtectProc == "invisible"; inner.ProcSubset == "all" && inner.ProtectProc == "invisible";
} }
{ {
name = "hive-priv is told the cache's address by the host's configuration"; # The daemon only learns the cache from the fragment the root unit
# writes, and lists as trusted the one address from the host's
# configuration. A host with no unit has no include.
name = "the host's nix.conf includes the cache fragment the root unit writes, with the cache's address";
ok = ok =
allLocal.systemd.services.hive-priv.environment.HIVE_PRIV_CACHE_URL == "https://attic.t.local/swarm" let
&& !(noIdentity.systemd.services.hive-priv.environment ? HIVE_PRIV_CACHE_URL); u = units.hive-priv-cache-facts;
in
lib.hasInfix "!include /run/hyperhive-nix/cache.conf" allLocal.nix.extraOptions
&& !(lib.hasInfix "hyperhive-nix" noIdentity.nix.extraOptions)
&& lib.hasInfix "conf=/run/hyperhive-nix/cache.conf" u.script
&& lib.hasInfix "extra-trusted-substituters = https://attic.t.local/swarm" u.script
&& lib.hasInfix ''atomic_write_secret 0644 "" "$conf"'' u.script
&& lib.elem "d /run/hyperhive-nix 0755 root root - -" allLocal.systemd.tmpfiles.rules
# Control for the absence: the lookup finds the unit's other variable.
&& allLocal.systemd.services.hive-priv.environment ? NIX_REMOTE
&& !(allLocal.systemd.services.hive-priv.environment ? HIVE_PRIV_CACHE_URL);
} }
]; ];
in in

2
nix/script-tests/attic-root-facts.nix
View file

@ -15,6 +15,8 @@ let
hive = "alpha"; hive = "alpha";
host = "cache.example"; host = "cache.example";
dir = "@DIR@/attic"; dir = "@DIR@/attic";
url = "https://cache.example/swarm";
confFile = "@DIR@/nix/cache.conf";
} }
); );
in in

Some files were not shown because too many files have changed in this diff Show more