Watch
0
0
Fork
You've already forked hyperhive
0

pull swarm closures through the nix daemon: substituters in the fragment, merged root-only netrc

This commit is contained in:
atlas 2026-11-08 10:29:46 +01:00 • committed by mara
commit ef68baa00f
4 changed files with 582 additions and 382 deletions

View file

@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the key as `extra-trusted-public-keys`, which the host's nix.conf includes, with no substituter and no netrc; and `/run/hyperhive-nix/netrc` (`root:hive-core 0640`), the token as a netrc that `hive-c0re` passes to `nix copy --from` as the client's own setting. `hive-core` is the one unprivileged reader of the token, and no other client of the host's nix daemon gets it. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
| secret | generated by | lives at |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys` and `netrc-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal

View file

@ -536,18 +536,24 @@ writes nothing.
`hive-priv` and the nix daemon don't read those files. The key that must have
signed a download and the pull token come from `hive-priv-cache-facts`, a root
unit that reads the hive's own `attic/public-key` and `attic/pull-token` from
the store under the hive's own store identity every ten minutes and writes them
to `/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). It also writes
the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
includes, with the key as `extra-trusted-public-keys` and nothing else: no
substituter and no netrc, so the daemon never contacts the cache and no other
client of the daemon, such as an agent on `nixBuilds = host`, can fetch from it.
A second copy of the netrc, `/run/hyperhive-nix/netrc` (`root:hive-core 0640`),
is the pull token `hive-c0re` uses. The cache's address is the host's
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re`. The
unit restarts `nix-daemon` only when the fragment changed after the daemon
started, which matters for CppNix; Lix re-reads nix.conf on every connection. A
file `hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
the store under the hive's own store identity every ten minutes. It writes the
key to `/run/hyperhive-priv/attic/public-key` (`0700` directory, `0600` file)
and the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's
nix.conf includes, with the cache's address as `extra-substituters`, the key as
`extra-trusted-public-keys` and `netrc-file = /run/hyperhive-nix/netrc`. The
netrc (`root:root 0600`) holds the pull token's entry for the cache's host and
then the entries of the host's own netrc, which is `nix.settings.netrc-file`
when the host sets it and nix's default `/etc/nix/netrc` otherwise, because
`netrc-file` takes one path and would replace the host's. The unit merges again
on every run, so an edit to the host's netrc reaches the daemon within ten
minutes. An entry for the cache's own host in the host's netrc gives way to the
pull token's. The unit doesn't see a netrc that the host names only in
`nix.extraOptions`. The cache's address is the host's
`services.hyperhive.swarm.attic.domain` and `cache`. The unit restarts
`nix-daemon` only when the fragment changed after the daemon started, which
matters for CppNix; Lix re-reads nix.conf on every connection and runs no
`nix-daemon.service`, so the unit logs and restarts nothing there. A file
`hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from
@ -591,13 +597,15 @@ means check attic's fingerprint format; it says nothing about an attacker. A
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
still up, reads the agent's record in `agent-closures` and has `hive-c0re` pull
the record's store path. `hive-c0re` isn't root and needs no root: it runs
`nix copy --from <cache address> --option netrc-file /run/hyperhive-nix/netrc <path>`
as `hive-core`. The nix client fetches from the cache with the pull token and
hands the NARs to the host's nix daemon, which stores each path only when a key
it trusts signed it; nothing builds. When the netrc or the key is missing,
`nix-store --realise <path> --max-jobs 0` as `hive-core`, which asks the host's
nix daemon to substitute the path. The daemon takes each path of the closure
from the swarm cache or from its other substituters (`cache.nixos.org`),
whichever has it, which matters because the swarm cache leaves out the paths
`cache.nixos.org` serves. It stores a path only when a key it trusts signed it,
and nothing builds. When the fragment is missing or lacks a setting,
`hive-c0re` first asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s)
instead of waiting for its timer. A host that still has no facts, a signature refusal and a path the
cache lacks each fail with their own message.
instead of waiting for its timer. A host that still has no facts, a signature
refusal and a path no substituter has each fail with their own message.
`Swap` then does the same lookup and pull, a no-op once the path is in the
store, and gives `hive-priv` the store path to apply. `hive-priv` applies it only

View file

@ -35,7 +35,7 @@ known operations; there is no arbitrary command pass-through:
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its netrc or the swarm key in the fragment is missing |
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when the fragment is missing or lacks a setting |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
@ -70,58 +70,62 @@ bind or credential path is absolute with no whitespace, control byte, `\`,
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
built with `nix copy --from <cache address>`, and `hive-priv` applies it only
when the swarm's key signed it. Pulling needs no root. The nix client does the
HTTP fetch itself, as `hive-core`, with a netrc only `hive-core` can read, and
hands the NARs to the host's nix daemon, which stores a path from an
unprivileged client only if a key in its `trusted-public-keys` signed it.
Root-owned sources alone decide what the daemon accepts:
built with `nix-store --realise <path> --max-jobs 0`, and `hive-priv` applies it
only when the swarm's key signed it. Pulling needs no root. The command is a
request to the host's nix daemon, which substitutes each path of the closure
from the swarm cache or from its other substituters (`cache.nixos.org`),
whichever has it, and stores a path from an unprivileged client only if a key
in its `trusted-public-keys` signed it. Root-owned sources alone decide what
the daemon accepts:
- The cache signs. The swarm's cache (attic) holds the signing key and signs
each narinfo it serves; the private key never leaves it. The builder pushes
with a push token and holds no signing key.
- The address is the host's configuration:
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re` in
`HIVE_C0RE_CACHE_URL`. The daemon has no address: it never contacts the
cache.
`services.hyperhive.swarm.attic.domain` and `cache`. The daemon holds it as a
substituter; `hive-c0re` has no address and names no source.
- The key and the pull token come from the secret store. The root unit
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
and the token to `netrc` there (directory `0700`, files `0600`, root). It
also writes the key into a nix.conf fragment, `/run/hyperhive-nix/cache.conf`
(`0644`), as `extra-trusted-public-keys`, which the host's nix.conf includes
(`!include`, so a host whose unit hasn't run yet still starts). The fragment
holds no secret, names no substituter and no netrc. `hive-priv` reads nothing
from `/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
- `hive-core` holds the pull token. The unit writes a second copy of the netrc,
`/run/hyperhive-nix/netrc`, `root:hive-core 0640`, and `hive-c0re` passes it
to `nix copy` as the client's `--option netrc-file`. This is a deliberate
grant, and no new capability: `hive-c0re` could already have `hive-priv`
download any swarm-signed path for any agent and apply it (the
`CreateContainer` row above), and the token reads only the cache the swarm
signs. The daemon ignores a client's `netrc-file`.
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the
host's nix.conf includes (`!include`, so a host whose unit hasn't run yet
still starts). The fragment holds no secret. `hive-priv` reads nothing from
`/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
- Only root reads the token. The netrc the fragment names,
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
default `/etc/nix/netrc`), because `netrc-file` takes one path and would
replace the host's. The unit merges again on every run, every ten minutes.
The daemon is root and reads the file itself, so `hive-core` has no copy of
the token for the pull. The probe's copy under `/run/hyperhive/attic/` is
separate.
- The daemon enforces. It refuses a path from an unprivileged client unless a
trusted key signed it; a client can't lower `require-sigs` or add keys.
`nix copy` imports and never builds.
trusted key signed it; a client can't lower `require-sigs` or add keys or
substituters. `--max-jobs 0` forbids a build, so a path no substituter has
fails.
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
`trusted-public-keys` set to the key in the root file, and refuses a path that
is absent or that the key didn't sign. The daemon's trusted keys also include
the host's other caches, so this check is the one that counts only the swarm's.
- When the netrc or the key is missing, `hive-c0re` asks `hive-priv` for
- When the fragment is missing or lacks a setting, `hive-c0re` asks `hive-priv` for
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
`this hive has no cache facts yet`.
**What an agent with daemon access can fetch.** Agent containers on
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). They
get nothing from the private cache: the daemon has no trusted substituter and no
netrc for it, an agent can't read `/run/hyperhive-nix/netrc` (`hive-core`'s
group) or `/run/hyperhive-priv/attic/`, and an unprivileged client can't make
the daemon use a substituter or a netrc it didn't configure. The one new thing
every client of the daemon shares is the swarm's public key in
`trusted-public-keys`: the daemon takes a path the swarm key signed from any source,
and only the swarm can produce one.
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). The
daemon holds the swarm cache as a substituter with the pull token, so any client
of it can have a swarm closure substituted, as it can have a path from
`cache.nixos.org`. That gives an agent nothing it can't read already: a closure
is world-readable in `/nix/store` once it's on the host, it holds no secret
(every secret comes from the store at run time), and the agent never sees the
token, which stays in root-only `/run/hyperhive-nix/netrc`. What the agent gains
is the chance to pull another hive's closure that isn't on this host yet. The
daemon takes a path the swarm key signed from any source, and only the swarm can
produce one.
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
signed, which fills the store with paths the swarm vouches for, and then ask

View file

@ -56,12 +56,12 @@ next start.
`none` cuts the daemon only. It leaves the agent's egress alone.
Under `host` an agent shares the daemon with the hive's own closure pull, and
gains nothing from it. The daemon trusts the swarm cache's public key, but it
has no cache address and no pull token (hive-c0re fetches with a netrc readable
only by `hive-core`), so an agent can't fetch from the private cache. It can
still ask the daemon to substitute from the public caches the host configures, as
before. See
Under `host` an agent shares the daemon with the hive's own closure pull. The
daemon holds the swarm cache as a substituter, with the pull token in a netrc
only root reads, so an agent can ask it to substitute a swarm closure as well as
a path from the public caches the host configures. The agent never sees the
token, and a closure is world-readable once it's on the host and holds no
secret. See
[hive-c0re-privilege.md](hive-c0re-privilege.md#closures-from-the-swarm-cache).
## Threat model