scope the cache access to hive-core: client-side nix copy, daemon trusts only the key
This commit is contained in:
atlas
mara
parent
ecd4445680
commit
dab7768869
4 changed files with 365 additions and 271 deletions
|
|
@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
|
||||
### Hive-level — one of each per hive
|
||||
|
||||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address, the key and the netrc's path as nix.conf settings that the host's nix.conf includes. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. `hive-core` reads only the fragment, for the address. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the key as `extra-trusted-public-keys`, which the host's nix.conf includes, with no substituter and no netrc; and `/run/hyperhive-nix/netrc` (`root:hive-core 0640`), the token as a netrc that `hive-c0re` passes to `nix copy --from` as the client's own setting. `hive-core` is the one unprivileged reader of the token, and no other client of the host's nix daemon gets it. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
|
||||
### Store identities, one per principal
|
||||
|
||||
|
|
|
|||
|
|
@ -539,9 +539,13 @@ unit that reads the hive's own `attic/public-key` and `attic/pull-token` from
|
|||
the store under the hive's own store identity every ten minutes and writes them
|
||||
to `/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). It also writes
|
||||
the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf
|
||||
includes: the cache's address (the host's `services.hyperhive.swarm.attic.domain`
|
||||
and `cache`) as a trusted substituter, the key as a trusted key and the netrc's
|
||||
path. It restarts `nix-daemon` only when the fragment changed after the daemon
|
||||
includes, with the key as `extra-trusted-public-keys` and nothing else: no
|
||||
substituter and no netrc, so the daemon never contacts the cache and no other
|
||||
client of the daemon, such as an agent on `nixBuilds = host`, can fetch from it.
|
||||
A second copy of the netrc, `/run/hyperhive-nix/netrc` (`root:hive-core 0640`),
|
||||
is the pull token `hive-c0re` uses. The cache's address is the host's
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re`. The
|
||||
unit restarts `nix-daemon` only when the fragment changed after the daemon
|
||||
started, which matters for CppNix; Lix re-reads nix.conf on every connection. A
|
||||
file `hive-c0re` writes decides nothing the daemon or `hive-priv` trusts.
|
||||
|
||||
|
|
@ -587,13 +591,12 @@ means check attic's fingerprint format; it says nothing about an attacker. A
|
|||
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
|
||||
still up, reads the agent's record in `agent-closures` and has `hive-c0re` pull
|
||||
the record's store path. `hive-c0re` isn't root and needs no root: it runs
|
||||
`nix-store --realise <path> --max-jobs 0 --option substituters <cache address>`
|
||||
through the host's nix daemon. The address is the one in the fragment above, the
|
||||
only substituter the daemon lets an unprivileged client name. The daemon
|
||||
ignores a client's keys, `require-sigs` and netrc, so it stores the path only
|
||||
when a key it trusts signed it. When the fragment is missing, `hive-c0re` first
|
||||
asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s) instead of waiting for
|
||||
its timer. A host that still has no facts, a signature refusal and a path the
|
||||
`nix copy --from <cache address> --option netrc-file /run/hyperhive-nix/netrc <path>`
|
||||
as `hive-core`. The nix client fetches from the cache with the pull token and
|
||||
hands the NARs to the host's nix daemon, which stores each path only when a key
|
||||
it trusts signed it; nothing builds. When the netrc or the key is missing,
|
||||
`hive-c0re` first asks `hive-priv` to run `hive-priv-cache-facts` (up to 60 s)
|
||||
instead of waiting for its timer. A host that still has no facts, a signature refusal and a path the
|
||||
cache lacks each fail with their own message.
|
||||
|
||||
`Swap` then does the same lookup and pull, a no-op once the path is in the
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ known operations; there is no arbitrary command pass-through:
|
|||
| `StartFailureLog` | `journalctl _SYSTEMD_UNIT=container@h-<name>.service --since @<t> -n 30 -o cat --no-pager`, fixed argv, 5 s timeout, output capped at 8 KiB; hive-priv clamps `<t>` to the last hour. hive-c0re appends the lines to the failed start's build log, which only the dashboard serves. |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create <name> --system-path <toplevel>`; for an update, `nix-env -p <profile> --set <toplevel>` and a reload of the unit. `<toplevel>` must already be in the store with a signature by the swarm cache's key: hive-priv runs `nix store verify --no-contents --sigs-needed 1` with `trusted-public-keys` set to the key in root-owned `/run/hyperhive-priv/attic/public-key`, and refuses a path that's absent or that the key didn't sign. hive-priv builds nothing. The check doesn't bind the path to the agent whose pull produced it: hive-c0re could already pull any swarm-signed path and apply it to any agent, so the check bounds the same risk (one agent gets another's closure), not an arbitrary-path one |
|
||||
| `RealiseClosure` | Kept for one release, for an older hive-c0re that still asks hive-priv to download. Downloads nothing: answers `Ok` when the path is in the store and signed by the swarm key (what `CreateContainer` needs), else `not available: <reason>`. The path must be `/nix/store/<32 chars>-<name>` and the agent name valid |
|
||||
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its cache fragment is missing |
|
||||
| `RefreshCacheFacts` | `systemctl start hive-priv-cache-facts.service`, waited for up to 60 s. The request names no unit and has no field. hive-c0re asks for it when its netrc or the swarm key in the fragment is missing |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||
|
|
@ -70,42 +70,59 @@ bind or credential path is absolute with no whitespace, control byte, `\`,
|
|||
`$`, backtick, `"` or `:`. The size limits of a value stay with systemd.
|
||||
|
||||
**Closures from the swarm cache** — `hive-c0re` pulls the closure the swarm
|
||||
built through the host's nix daemon, and `hive-priv` applies it only when the
|
||||
swarm's key signed it. Pulling needs no root: `nix-store --realise` of a store
|
||||
path is a request to the daemon. Root-owned sources alone decide what the
|
||||
daemon accepts:
|
||||
built with `nix copy --from <cache address>`, and `hive-priv` applies it only
|
||||
when the swarm's key signed it. Pulling needs no root. The nix client does the
|
||||
HTTP fetch itself, as `hive-core`, with a netrc only `hive-core` can read, and
|
||||
hands the NARs to the host's nix daemon, which stores a path from an
|
||||
unprivileged client only if a key in its `trusted-public-keys` signed it.
|
||||
Root-owned sources alone decide what the daemon accepts:
|
||||
|
||||
- The cache signs. The swarm's cache (attic) holds the signing key and signs
|
||||
each narinfo it serves; the private key never leaves it. The builder pushes
|
||||
with a push token and holds no signing key.
|
||||
- The address is the host's configuration:
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`. The root unit
|
||||
`hive-priv-cache-facts` lists it as `extra-trusted-substituters` in a nix.conf
|
||||
fragment, `/run/hyperhive-nix/cache.conf`, that the host's nix.conf includes
|
||||
(`!include`, so a host whose unit hasn't run yet still starts).
|
||||
- The key and the pull token come from the secret store. The unit reads the
|
||||
hive's own copies under the hive's own store identity and writes the key to
|
||||
`/run/hyperhive-priv/attic/public-key` and the token to `netrc` there
|
||||
(directory `0700`, files `0600`, root), and the key into the fragment as
|
||||
`extra-trusted-public-keys` and the netrc's path as `netrc-file`. The fragment
|
||||
holds no secret. `hive-c0re` reads its cache address, and nothing else, from
|
||||
it; `hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
|
||||
writes for the probe.
|
||||
- The daemon enforces. `hive-core` is an unprivileged client: the daemon takes
|
||||
its `--option substituters` only for an address in `trusted-substituters`,
|
||||
ignores its `trusted-public-keys`, `require-sigs` and `netrc-file`, and
|
||||
refuses a path no trusted key signed. `--max-jobs 0` means no answer, whatever
|
||||
the cache sends, starts a build.
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`, handed to `hive-c0re` in
|
||||
`HIVE_C0RE_CACHE_URL`. The daemon has no address: it never contacts the
|
||||
cache.
|
||||
- The key and the pull token come from the secret store. The root unit
|
||||
`hive-priv-cache-facts` reads the hive's own copies under the hive's own
|
||||
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
|
||||
and the token to `netrc` there (directory `0700`, files `0600`, root). It
|
||||
also writes the key into a nix.conf fragment, `/run/hyperhive-nix/cache.conf`
|
||||
(`0644`), as `extra-trusted-public-keys`, which the host's nix.conf includes
|
||||
(`!include`, so a host whose unit hasn't run yet still starts). The fragment
|
||||
holds no secret, names no substituter and no netrc. `hive-priv` reads nothing
|
||||
from `/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
|
||||
- `hive-core` holds the pull token. The unit writes a second copy of the netrc,
|
||||
`/run/hyperhive-nix/netrc`, `root:hive-core 0640`, and `hive-c0re` passes it
|
||||
to `nix copy` as the client's `--option netrc-file`. This is a deliberate
|
||||
grant, and no new capability: `hive-c0re` could already have `hive-priv`
|
||||
download any swarm-signed path for any agent and apply it (the
|
||||
`CreateContainer` row above), and the token reads only the cache the swarm
|
||||
signs. The daemon ignores a client's `netrc-file`.
|
||||
- The daemon enforces. It refuses a path from an unprivileged client unless a
|
||||
trusted key signed it; a client can't lower `require-sigs` or add keys.
|
||||
`nix copy` imports and never builds.
|
||||
- `hive-priv` verifies. Before `CreateContainer` or `UpdateContainer` applies a
|
||||
toplevel, it runs `nix store verify --no-contents --sigs-needed 1` with
|
||||
`trusted-public-keys` set to the key in the root file, and refuses a path that
|
||||
is absent or that the key didn't sign. The daemon's trusted keys also include
|
||||
the host's other caches, so this check is the one that counts only the swarm's.
|
||||
- When the fragment is missing, `hive-c0re` asks `hive-priv` for `RefreshCacheFacts`,
|
||||
which starts the facts unit and waits up to 60 s, instead of waiting for the
|
||||
unit's timer. A host the swarm hasn't given facts to answers
|
||||
- When the netrc or the key is missing, `hive-c0re` asks `hive-priv` for
|
||||
`RefreshCacheFacts`, which starts the facts unit and waits up to 60 s, instead
|
||||
of waiting for the unit's timer. A host the swarm hasn't given facts to answers
|
||||
`this hive has no cache facts yet`.
|
||||
|
||||
**What an agent with daemon access can fetch.** Agent containers on
|
||||
`nixBuilds = host` reach the same daemon ([nix-builds.md](nix-builds.md)). They
|
||||
get nothing from the private cache: the daemon has no trusted substituter and no
|
||||
netrc for it, an agent can't read `/run/hyperhive-nix/netrc` (`hive-core`'s
|
||||
group) or `/run/hyperhive-priv/attic/`, and an unprivileged client can't make
|
||||
the daemon use a substituter or a netrc it didn't configure. The one new thing
|
||||
every client of the daemon shares is the swarm's public key in
|
||||
`trusted-public-keys`: the daemon takes a path the swarm key signed from any source,
|
||||
and only the swarm can produce one.
|
||||
|
||||
What a compromised `hive-c0re` can do: pull any store path that the swarm cache
|
||||
signed, which fills the store with paths the swarm vouches for, and then ask
|
||||
`hive-priv` to apply any such path to any agent. Before this change it could
|
||||
|
|
|
|||
|
|
@ -56,6 +56,14 @@ next start.
|
|||
|
||||
`none` cuts the daemon only. It leaves the agent's egress alone.
|
||||
|
||||
Under `host` an agent shares the daemon with the hive's own closure pull, and
|
||||
gains nothing from it. The daemon trusts the swarm cache's public key, but it
|
||||
has no cache address and no pull token (hive-c0re fetches with a netrc readable
|
||||
only by `hive-core`), so an agent can't fetch from the private cache. It can
|
||||
still ask the daemon to substitute from the public caches the host configures, as
|
||||
before. See
|
||||
[hive-c0re-privilege.md](hive-c0re-privilege.md#closures-from-the-swarm-cache).
|
||||
|
||||
## Threat model
|
||||
|
||||
Unsandboxed nix builds run as `nixbld` users (non-root; `nrBuildUsers` is at least 32, so UIDs 30001 and up).
|
||||
|
|
|
|||
Loading…
Reference in a new issue