mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

pull swarm closures through the nix daemon: substituters in the fragment, merged root-only netrc

Browse source
This commit is contained in:
atlas 2026-11-08 10:29:46 +01:00 • committed by mara
commit ef68baa00f
2 changed files with 582 additions and 382 deletions
Download patch file Download diff file Expand all files Collapse all files

72
docs/4-implementation/trust-boundary/hive-c0re-privilege.md
View file

@ -35,7 +35,7 @@ known operations; there is no arbitrary command pass-through:
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its netrc or the swarm key in the fragment is missing |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when the fragment is missing or lacks a setting |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
@ -70,58 +70,62 @@ bind or credential path is absolute with no whitespace, control byte, `\`,
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
built with `nix copy --from <cache address>`, and `hive-priv` applies it only
when the swarm's key signed it. Pulling needs no root. The nix client does the
HTTP fetch itself, as `hive-core`, with a netrc only `hive-core` can read, and
hands the NARs to the host's nix daemon, which stores a path from an
unprivileged client only if a key in its `trusted-public-keys` signed it.
Root-owned sources alone decide what the daemon accepts:
built with `nix-store --realise <path> --max-jobs 0`, and `hive-priv` applies it
only when the swarm's key signed it. Pulling needs no root. The command is a
request to the host's nix daemon, which substitutes each path of the closure
from the swarm cache or from its other substituters (`cache.nixos.org`),
whichever has it, and stores a path from an unprivileged client only if a key
in its `trusted-public-keys` signed it. Root-owned sources alone decide what
the daemon accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key.
- The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re` in
`HIVE_C0RE_CACHE_URL`. The daemon has no address: it never contacts the
cache.
`services.hyperhive.swarm.attic.domain` and `cache`. The daemon holds it as a
substituter; `hive-c0re` has no address and names no source.
- The key and the pull token come from the secret store. The root unit
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
and the token to `netrc` there (directory `0700`, files `0600`, root). It
also writes the key into a nix.conf fragment, `/run/hyperhive-nix/cache.conf`
(`0644`), as `extra-trusted-public-keys`, which the host's nix.conf includes
(`!include`, so a host whose unit hasn't run yet still starts). The fragment
holds no secret, names no substituter and no netrc. `hive-priv` reads nothing
from `/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
- `hive-core` holds the pull token. The unit writes a second copy of the netrc,
`/run/hyperhive-nix/netrc`, `root:hive-core 0640`, and `hive-c0re` passes it
to `nix copy` as the client's `--option netrc-file`. This is a deliberate
grant, and no new capability: `hive-c0re` could already have `hive-priv`
download any swarm-signed path for any agent and apply it (the
`CreateContainer` row above), and the token reads only the cache the swarm
signs. The daemon ignores a client's `netrc-file`.
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the
host's nix.conf includes (`!include`, so a host whose unit hasn't run yet
still starts). The fragment holds no secret. `hive-priv` reads nothing from
`/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
- Only root reads the token. The netrc the fragment names,
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
default `/etc/nix/netrc`), because `netrc-file` takes one path and would
replace the host's. The unit merges again on every run, every ten minutes.
The daemon is root and reads the file itself, so `hive-core` has no copy of
the token for the pull. The probe's copy under `/run/hyperhive/attic/` is
separate.
- The daemon enforces. It refuses a path from an unprivileged client unless a
trusted key signed it; a client can't lower `require-sigs` or add keys.
`nix copy` imports and never builds.
trusted key signed it; a client can't lower `require-sigs` or add keys or
substituters. `--max-jobs 0` forbids a build, so a path no substituter has
fails.
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
`trusted-public-keys` set to the key in the root file, and refuses a path that
is absent or that the key didn't sign. The daemon's trusted keys also include
the host's other caches, so this check is the one that counts only the swarm's.
- When the netrc or the key is missing, `hive-c0re` asks `hive-priv` for
- When the fragment is missing or lacks a setting, `hive-c0re` asks `hive-priv` for
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
`this hive has no cache facts yet`.
**What an agent with daemon access can fetch.** Agent containers on
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). They
get nothing from the private cache: the daemon has no trusted substituter and no
netrc for it, an agent can't read `/run/hyperhive-nix/netrc` (`hive-core`'s
group) or `/run/hyperhive-priv/attic/`, and an unprivileged client can't make
the daemon use a substituter or a netrc it didn't configure. The one new thing
every client of the daemon shares is the swarm's public key in
`trusted-public-keys`: the daemon takes a path the swarm key signed from any source,
and only the swarm can produce one.
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). The
daemon holds the swarm cache as a substituter with the pull token, so any client
of it can have a swarm closure substituted, as it can have a path from
`cache.nixos.org`. That gives an agent nothing it can't read already: a closure
is world-readable in `/nix/store` once it's on the host, it holds no secret
(every secret comes from the store at run time), and the agent never sees the
token, which stays in root-only `/run/hyperhive-nix/netrc`. What the agent gains
is the chance to pull another hive's closure that isn't on this host yet. The
daemon takes a path the swarm key signed from any source, and only the swarm can
produce one.
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
signed, which fills the store with paths the swarm vouches for, and then ask

12
docs/4-implementation/trust-boundary/nix-builds.md
View file

@ -56,12 +56,12 @@ next start.
`none` cuts the daemon only. It leaves the agent's egress alone.
Under `host` an agent shares the daemon with the hive's own closure pull, and
gains nothing from it. The daemon trusts the swarm cache's public key, but it
has no cache address and no pull token (hive-c0re fetches with a netrc readable
only by `hive-core`), so an agent can't fetch from the private cache. It can
still ask the daemon to substitute from the public caches the host configures, as
before. See
Under `host` an agent shares the daemon with the hive's own closure pull. The
daemon holds the swarm cache as a substituter, with the pull token in a netrc
only root reads, so an agent can ask it to substitute a swarm closure as well as
a path from the public caches the host configures. The agent never sees the
token, and a closure is world-readable once it's on the host and holds no
secret. See
[hive-c0re-privilege.md](hive-c0re-privilege.md#closures-from-the-swarm-cache).
## Threat model