pull swarm closures through the nix daemon: substituters in the fragment, merged root-only netrc
This commit is contained in:
atlas
mara
parent
63ae5d6cbf
commit
ef68baa00f
3 changed files with 582 additions and 382 deletions
|
|
@ -536,18 +536,24 @@ writes nothing.
|
|||
`hive-priv` and the nix daemon don't read those files. The key that must have
|
||||
signed a download and the pull token come from `hive-priv-cache-facts`, a root
|
||||
unit that reads the hive's own `attic/public-key` and `attic/pull-token` from
|
||||
the store under the hive's own store identity every ten minutes and writes them
|
||||
to `/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). It also writes
|
||||
the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
|
||||
includes, with the key as `extra-trusted-public-keys` and nothing else: no
|
||||
substituter and no netrc, so the daemon never contacts the cache and no other
|
||||
client of the daemon, such as an agent on `nixBuilds = host`, can fetch from it.
|
||||
A second copy of the netrc, `/run/hyperhive-nix/netrc` (`root:hive-core 0640`),
|
||||
is the pull token `hive-c0re` uses. The cache's address is the host's
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re`. The
|
||||
unit restarts `nix-daemon` only when the fragment changed after the daemon
|
||||
started, which matters for CppNix; Lix re-reads nix.conf on every connection. A
|
||||
file `hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
|
||||
the store under the hive's own store identity every ten minutes. It writes the
|
||||
key to `/run/hyperhive-priv/attic/public-key` (`0700` directory, `0600` file)
|
||||
and the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's
|
||||
nix.conf includes, with the cache's address as `extra-substituters`, the key as
|
||||
`extra-trusted-public-keys` and `netrc-file = /run/hyperhive-nix/netrc`. The
|
||||
netrc (`root:root 0600`) holds the pull token's entry for the cache's host and
|
||||
then the entries of the host's own netrc, which is `nix.settings.netrc-file`
|
||||
when the host sets it and nix's default `/etc/nix/netrc` otherwise, because
|
||||
`netrc-file` takes one path and would replace the host's. The unit merges again
|
||||
on every run, so an edit to the host's netrc reaches the daemon within ten
|
||||
minutes. An entry for the cache's own host in the host's netrc gives way to the
|
||||
pull token's. The unit doesn't see a netrc that the host names only in
|
||||
`nix.extraOptions`. The cache's address is the host's
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`. The unit restarts
|
||||
`nix-daemon` only when the fragment changed after the daemon started, which
|
||||
matters for CppNix; Lix re-reads nix.conf on every connection and runs no
|
||||
`nix-daemon.service`, so the unit logs and restarts nothing there. A file
|
||||
`hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
|
||||
|
||||
The same pass checks each of the hive's agents. It reads the agent's record in
|
||||
`agent-closures`, then fetches the closure's narinfo from
|
||||
|
|
@ -591,13 +597,15 @@ means check attic's fingerprint format; it says nothing about an attacker. A
|
|||
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
|
||||
still up, reads the agent's record in `agent-closures` and has `hive-c0re` pull
|
||||
the record's store path. `hive-c0re` isn't root and needs no root: it runs
|
||||
`nix copy --from <cache address> --option netrc-file /run/hyperhive-nix/netrc <path>`
|
||||
as `hive-core`. The nix client fetches from the cache with the pull token and
|
||||
hands the NARs to the host's nix daemon, which stores each path only when a key
|
||||
it trusts signed it; nothing builds. When the netrc or the key is missing,
|
||||
`nix-store --realise <path> --max-jobs 0` as `hive-core`, which asks the host's
|
||||
nix daemon to substitute the path. The daemon takes each path of the closure
|
||||
from the swarm cache or from its other substituters (`cache.nixos.org`),
|
||||
whichever has it, which matters because the swarm cache leaves out the paths
|
||||
`cache.nixos.org` serves. It stores a path only when a key it trusts signed it,
|
||||
and nothing builds. When the fragment is missing or lacks a setting,
|
||||
`hive-c0re` first asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s)
|
||||
instead of waiting for its timer. A host that still has no facts, a signature refusal and a path the
|
||||
cache lacks each fail with their own message.
|
||||
instead of waiting for its timer. A host that still has no facts, a signature
|
||||
refusal and a path no substituter has each fail with their own message.
|
||||
|
||||
`Swap` then does the same lookup and pull, a no-op once the path is in the
|
||||
store, and gives `hive-priv` the store path to apply. `hive-priv` applies it only
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
|
||||
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
|
||||
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its netrc or the swarm key in the fragment is missing |
|
||||
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when the fragment is missing or lacks a setting |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
|
|
@ -70,58 +70,62 @@ bind or credential path is absolute with no whitespace, control byte, `\`,
|
|||
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
||||
|
||||
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
|
||||
built with `nix copy --from <cache address>`, and `hive-priv` applies it only
|
||||
when the swarm's key signed it. Pulling needs no root. The nix client does the
|
||||
HTTP fetch itself, as `hive-core`, with a netrc only `hive-core` can read, and
|
||||
hands the NARs to the host's nix daemon, which stores a path from an
|
||||
unprivileged client only if a key in its `trusted-public-keys` signed it.
|
||||
Root-owned sources alone decide what the daemon accepts:
|
||||
built with `nix-store --realise <path> --max-jobs 0`, and `hive-priv` applies it
|
||||
only when the swarm's key signed it. Pulling needs no root. The command is a
|
||||
request to the host's nix daemon, which substitutes each path of the closure
|
||||
from the swarm cache or from its other substituters (`cache.nixos.org`),
|
||||
whichever has it, and stores a path from an unprivileged client only if a key
|
||||
in its `trusted-public-keys` signed it. Root-owned sources alone decide what
|
||||
the daemon accepts:
|
||||
|
||||
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
||||
each narinfo it serves; the private key never leaves it. The builder pushes
|
||||
with a push token and holds no signing key.
|
||||
- The address is the host's configuration:
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re` in
|
||||
`HIVE_C0RE_CACHE_URL`. The daemon has no address: it never contacts the
|
||||
cache.
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`. The daemon holds it as a
|
||||
substituter; `hive-c0re` has no address and names no source.
|
||||
- The key and the pull token come from the secret store. The root unit
|
||||
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
|
||||
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
|
||||
and the token to `netrc` there (directory `0700`, files `0600`, root). It
|
||||
also writes the key into a nix.conf fragment, `/run/hyperhive-nix/cache.conf`
|
||||
(`0644`), as `extra-trusted-public-keys`, which the host's nix.conf includes
|
||||
(`!include`, so a host whose unit hasn't run yet still starts). The fragment
|
||||
holds no secret, names no substituter and no netrc. `hive-priv` reads nothing
|
||||
from `/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
|
||||
- `hive-core` holds the pull token. The unit writes a second copy of the netrc,
|
||||
`/run/hyperhive-nix/netrc`, `root:hive-core 0640`, and `hive-c0re` passes it
|
||||
to `nix copy` as the client's `--option netrc-file`. This is a deliberate
|
||||
grant, and no new capability: `hive-c0re` could already have `hive-priv`
|
||||
download any swarm-signed path for any agent and apply it (the
|
||||
`CreateContainer` row above), and the token reads only the cache the swarm
|
||||
signs. The daemon ignores a client's `netrc-file`.
|
||||
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
|
||||
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
|
||||
address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the
|
||||
host's nix.conf includes (`!include`, so a host whose unit hasn't run yet
|
||||
still starts). The fragment holds no secret. `hive-priv` reads nothing from
|
||||
`/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
|
||||
- Only root reads the token. The netrc the fragment names,
|
||||
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
|
||||
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
|
||||
default `/etc/nix/netrc`), because `netrc-file` takes one path and would
|
||||
replace the host's. The unit merges again on every run, every ten minutes.
|
||||
The daemon is root and reads the file itself, so `hive-core` has no copy of
|
||||
the token for the pull. The probe's copy under `/run/hyperhive/attic/` is
|
||||
separate.
|
||||
- The daemon enforces. It refuses a path from an unprivileged client unless a
|
||||
trusted key signed it; a client can't lower `require-sigs` or add keys.
|
||||
`nix copy` imports and never builds.
|
||||
trusted key signed it; a client can't lower `require-sigs` or add keys or
|
||||
substituters. `--max-jobs 0` forbids a build, so a path no substituter has
|
||||
fails.
|
||||
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
|
||||
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
|
||||
`trusted-public-keys` set to the key in the root file, and refuses a path that
|
||||
is absent or that the key didn't sign. The daemon's trusted keys also include
|
||||
the host's other caches, so this check is the one that counts only the swarm's.
|
||||
- When the netrc or the key is missing, `hive-c0re` asks `hive-priv` for
|
||||
- When the fragment is missing or lacks a setting, `hive-c0re` asks `hive-priv` for
|
||||
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
|
||||
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
|
||||
`this hive has no cache facts yet`.
|
||||
|
||||
**What an agent with daemon access can fetch.** Agent containers on
|
||||
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). They
|
||||
get nothing from the private cache: the daemon has no trusted substituter and no
|
||||
netrc for it, an agent can't read `/run/hyperhive-nix/netrc` (`hive-core`'s
|
||||
group) or `/run/hyperhive-priv/attic/`, and an unprivileged client can't make
|
||||
the daemon use a substituter or a netrc it didn't configure. The one new thing
|
||||
every client of the daemon shares is the swarm's public key in
|
||||
`trusted-public-keys`: the daemon takes a path the swarm key signed from any source,
|
||||
and only the swarm can produce one.
|
||||
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). The
|
||||
daemon holds the swarm cache as a substituter with the pull token, so any client
|
||||
of it can have a swarm closure substituted, as it can have a path from
|
||||
`cache.nixos.org`. That gives an agent nothing it can't read already: a closure
|
||||
is world-readable in `/nix/store` once it's on the host, it holds no secret
|
||||
(every secret comes from the store at run time), and the agent never sees the
|
||||
token, which stays in root-only `/run/hyperhive-nix/netrc`. What the agent gains
|
||||
is the chance to pull another hive's closure that isn't on this host yet. The
|
||||
daemon takes a path the swarm key signed from any source, and only the swarm can
|
||||
produce one.
|
||||
|
||||
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
|
||||
signed, which fills the store with paths the swarm vouches for, and then ask
|
||||
|
|
|
|||
|
|
@ -56,12 +56,12 @@ next start.
|
|||
|
||||
`none` cuts the daemon only. It leaves the agent's egress alone.
|
||||
|
||||
Under `host` an agent shares the daemon with the hive's own closure pull, and
|
||||
gains nothing from it. The daemon trusts the swarm cache's public key, but it
|
||||
has no cache address and no pull token (hive-c0re fetches with a netrc readable
|
||||
only by `hive-core`), so an agent can't fetch from the private cache. It can
|
||||
still ask the daemon to substitute from the public caches the host configures, as
|
||||
before. See
|
||||
Under `host` an agent shares the daemon with the hive's own closure pull. The
|
||||
daemon holds the swarm cache as a substituter, with the pull token in a netrc
|
||||
only root reads, so an agent can ask it to substitute a swarm closure as well as
|
||||
a path from the public caches the host configures. The agent never sees the
|
||||
token, and a closure is world-readable once it's on the host and holds no
|
||||
secret. See
|
||||
[hive-c0re-privilege.md](hive-c0re-privilege.md#closures-from-the-swarm-cache).
|
||||
|
||||
## Threat model
|
||||
|
|
|
|||
Loading…
Reference in a new issue