nix trusts the swarm's root through its own CA bundle, so root's nix clients reach the cache
This commit is contained in:
atlas
parent
526cfbec1b
commit
9d62ef44b5
3 changed files with 299 additions and 79 deletions
|
|
@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
|
||||
### Hive-level — one of each per hive
|
||||
|
||||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys` and `netrc-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. `ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`, public), the host's own CA bundle followed by the swarm's root from `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem`, so that only nix trusts the root. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
|
||||
### Store identities, one per principal
|
||||
|
||||
|
|
|
|||
|
|
@ -540,7 +540,8 @@ the store under the hive's own store identity every ten minutes. It writes the
|
|||
key to `/run/hyperhive-priv/attic/public-key` (`0700` directory, `0600` file)
|
||||
and the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's
|
||||
nix.conf includes, with the cache's address as `extra-substituters`, the key as
|
||||
`extra-trusted-public-keys` and `netrc-file = /run/hyperhive-nix/netrc`. The
|
||||
`extra-trusted-public-keys`, `netrc-file = /run/hyperhive-nix/netrc` and
|
||||
`ssl-cert-file = /run/hyperhive-nix/ca-bundle.crt`. The
|
||||
netrc (`root:root 0600`) holds the pull token's entry for the cache's host and
|
||||
then the entries of the host's own netrc, which is `nix.settings.netrc-file`
|
||||
when the host sets it and nix's default `/etc/nix/netrc` otherwise, because
|
||||
|
|
@ -548,6 +549,15 @@ when the host sets it and nix's default `/etc/nix/netrc` otherwise, because
|
|||
on every run, so an edit to the host's netrc reaches the daemon within ten
|
||||
minutes. An entry for the cache's own host in the host's netrc gives way to the
|
||||
pull token's. The unit doesn't see a netrc that the host names only in
|
||||
`nix.extraOptions`. The CA bundle (`root:root 0644`) is the host's own bundle,
|
||||
which is `nix.settings.ssl-cert-file` when the host sets it and
|
||||
`security.pki.caBundle` otherwise, followed by the swarm's root from
|
||||
`<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem`. The cache's
|
||||
certificate chains to that root, which no build-time bundle holds, and every
|
||||
nix client of the host (the daemon, `hive-priv`, root's `nixos-rebuild`) reads
|
||||
the fragment. Only nix trusts the root; the host's system trust stays as it was.
|
||||
A host without the root yet gets its own bundle alone until a later run finds
|
||||
it. The unit doesn't see an `ssl-cert-file` that the host names only in
|
||||
`nix.extraOptions`. The cache's address is the host's
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`. The unit restarts
|
||||
`nix-daemon` only when the fragment changed after the daemon started, which
|
||||
|
|
|
|||
|
|
@ -89,10 +89,14 @@ the daemon accepts:
|
|||
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
|
||||
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
|
||||
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
|
||||
address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the
|
||||
host's nix.conf includes (`!include`, so a host whose unit hasn't run yet
|
||||
still starts). The fragment holds no secret. `hive-priv` reads nothing from
|
||||
`/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
|
||||
address), `extra-trusted-public-keys` (the key), `netrc-file` and
|
||||
`ssl-cert-file`, which the host's nix.conf includes (`!include`, so a host
|
||||
whose unit hasn't run yet still starts). The fragment holds no secret.
|
||||
`ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`): the host's
|
||||
own CA bundle plus the swarm's root, so nix clients on the host trust the
|
||||
cache's certificate while the host's system trust stays without the root.
|
||||
`hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
|
||||
writes for the probe.
|
||||
- Only root reads the token. The netrc the fragment names,
|
||||
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
|
||||
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
|
||||
|
|
|
|||
Loading…
Reference in a new issue