mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

scope the cache access to hive-core: client-side nix copy, daemon trusts only the key

Browse source
This commit is contained in:
atlas 2026-11-07 15:25:30 +01:00 • committed by mara
commit dab7768869
3 changed files with 365 additions and 271 deletions
Download patch file Download diff file Expand all files Collapse all files

23
docs/4-implementation/swarm/builder.md
View file

@ -539,9 +539,13 @@ unit that reads the hive's own `attic/public-key` and `attic/pull-token` from
the store under the hive's own store identity every ten minutes and writes them
to `/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). It also writes
the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
includes: the cache's address (the host's `services.hyperhive.swarm.attic.domain`
and `cache`) as a trusted substituter, the key as a trusted key and the netrc's
path. It restarts `nix-daemon` only when the fragment changed after the daemon
includes, with the key as `extra-trusted-public-keys` and nothing else: no
substituter and no netrc, so the daemon never contacts the cache and no other
client of the daemon, such as an agent on `nixBuilds = host`, can fetch from it.
A second copy of the netrc, `/run/hyperhive-nix/netrc` (`root:hive-core 0640`),
is the pull token `hive-c0re` uses. The cache's address is the host's
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re`. The
unit restarts `nix-daemon` only when the fragment changed after the daemon
started, which matters for CppNix; Lix re-reads nix.conf on every connection. A
file `hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
@ -587,13 +591,12 @@ means check attic's fingerprint format; it says nothing about an attacker. A
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and has `hive-c0re` pull
the record's store path. `hive-c0re` isn't root and needs no root: it runs
`nix-store --realise <path> --max-jobs 0 --option substituters <cache address>`
through the host's nix daemon. The address is the one in the fragment above, the
only substituter the daemon lets an unprivileged client name. The daemon
ignores a client's keys, `require-sigs` and netrc, so it stores the path only
when a key it trusts signed it. When the fragment is missing, `hive-c0re` first
asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s) instead of waiting for
its timer. A host that still has no facts, a signature refusal and a path the
`nix copy --from <cache address> --option netrc-file /run/hyperhive-nix/netrc <path>`
as `hive-core`. The nix client fetches from the cache with the pull token and
hands the NARs to the host's nix daemon, which stores each path only when a key
it trusts signed it; nothing builds. When the netrc or the key is missing,
`hive-c0re` first asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s)
instead of waiting for its timer. A host that still has no facts, a signature refusal and a path the
cache lacks each fail with their own message.
`Swap` then does the same lookup and pull, a no-op once the path is in the

67
docs/4-implementation/trust-boundary/hive-c0re-privilege.md
View file

@ -35,7 +35,7 @@ known operations; there is no arbitrary command pass-through:
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its cache fragment is missing |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its netrc or the swarm key in the fragment is missing |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
@ -70,42 +70,59 @@ bind or credential path is absolute with no whitespace, control byte, `\`,
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
built through the host's nix daemon, and `hive-priv` applies it only when the
swarm's key signed it. Pulling needs no root: `nix-store --realise` of a store
path is a request to the daemon. Root-owned sources alone decide what the
daemon accepts:
built with `nix copy --from <cache address>`, and `hive-priv` applies it only
when the swarm's key signed it. Pulling needs no root. The nix client does the
HTTP fetch itself, as `hive-core`, with a netrc only `hive-core` can read, and
hands the NARs to the host's nix daemon, which stores a path from an
unprivileged client only if a key in its `trusted-public-keys` signed it.
Root-owned sources alone decide what the daemon accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key.
- The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache`. The root unit
`hive-priv-cache-facts` lists it as `extra-trusted-substituters` in a nix.conf
fragment, `/run/hyperhive-nix/cache.conf`, that the host's nix.conf includes
(`!include`, so a host whose unit hasn't run yet still starts).
- The key and the pull token come from the secret store. The unit reads the
hive's own copies under the hive's own store identity and writes the key to
`/run/hyperhive-priv/attic/public-key` and the token to `netrc` there
(directory `0700`, files `0600`, root), and the key into the fragment as
`extra-trusted-public-keys` and the netrc's path as `netrc-file`. The fragment
holds no secret. `hive-c0re` reads its cache address, and nothing else, from
it; `hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
writes for the probe.
- The daemon enforces. `hive-core` is an unprivileged client: the daemon takes
its `--option substituters` only for an address in `trusted-substituters`,
ignores its `trusted-public-keys`, `require-sigs` and `netrc-file`, and
refuses a path no trusted key signed. `--max-jobs 0` means no answer, whatever
the cache sends, starts a build.
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re` in
`HIVE_C0RE_CACHE_URL`. The daemon has no address: it never contacts the
cache.
- The key and the pull token come from the secret store. The root unit
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
and the token to `netrc` there (directory `0700`, files `0600`, root). It
also writes the key into a nix.conf fragment, `/run/hyperhive-nix/cache.conf`
(`0644`), as `extra-trusted-public-keys`, which the host's nix.conf includes
(`!include`, so a host whose unit hasn't run yet still starts). The fragment
holds no secret, names no substituter and no netrc. `hive-priv` reads nothing
from `/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
- `hive-core` holds the pull token. The unit writes a second copy of the netrc,
`/run/hyperhive-nix/netrc`, `root:hive-core 0640`, and `hive-c0re` passes it
to `nix copy` as the client's `--option netrc-file`. This is a deliberate
grant, and no new capability: `hive-c0re` could already have `hive-priv`
download any swarm-signed path for any agent and apply it (the
`CreateContainer` row above), and the token reads only the cache the swarm
signs. The daemon ignores a client's `netrc-file`.
- The daemon enforces. It refuses a path from an unprivileged client unless a
trusted key signed it; a client can't lower `require-sigs` or add keys.
`nix copy` imports and never builds.
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
`trusted-public-keys` set to the key in the root file, and refuses a path that
is absent or that the key didn't sign. The daemon's trusted keys also include
the host's other caches, so this check is the one that counts only the swarm's.
- When the fragment is missing, `hive-c0re` asks `hive-priv` for `RefreshCacheFacts`,
which starts the facts unit and waits up to 60 s, instead of waiting for the
unit's timer. A host the swarm hasn't given facts to answers
- When the netrc or the key is missing, `hive-c0re` asks `hive-priv` for
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
`this hive has no cache facts yet`.
**What an agent with daemon access can fetch.** Agent containers on
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). They
get nothing from the private cache: the daemon has no trusted substituter and no
netrc for it, an agent can't read `/run/hyperhive-nix/netrc` (`hive-core`'s
group) or `/run/hyperhive-priv/attic/`, and an unprivileged client can't make
the daemon use a substituter or a netrc it didn't configure. The one new thing
every client of the daemon shares is the swarm's public key in
`trusted-public-keys`: the daemon takes a path the swarm key signed from any source,
and only the swarm can produce one.
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
signed, which fills the store with paths the swarm vouches for, and then ask
`hive-priv` to apply any such path to any agent. Before this change it could

8
docs/4-implementation/trust-boundary/nix-builds.md
View file

@ -56,6 +56,14 @@ next start.
`none` cuts the daemon only. It leaves the agent's egress alone.
Under `host` an agent shares the daemon with the hive's own closure pull, and
gains nothing from it. The daemon trusts the swarm cache's public key, but it
has no cache address and no pull token (hive-c0re fetches with a netrc readable
only by `hive-core`), so an agent can't fetch from the private cache. It can
still ask the daemon to substitute from the public caches the host configures, as
before. See
[hive-c0re-privilege.md](hive-c0re-privilege.md#closures-from-the-swarm-cache).
## Threat model
Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up).