diff --git a/docs/2-handbook/swarm/credentials-reference.md b/docs/2-handbook/swarm/credentials-reference.md index ea1b536db..282c3de3d 100644 --- a/docs/2-handbook/swarm/credentials-reference.md +++ b/docs/2-handbook/swarm/credentials-reference.md @@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. ### Hive-level — one of each per hive -| secret | generated by | lives at | -| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `/.pem` | -| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | -| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives//attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys` and `netrc-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. | -| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | -| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | -| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives//forge/token`, for the `hive-` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk | -| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `:` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct | -| the squid self-check's password | `swarm-controller` mints `swarm/hives//squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | +| secret | generated by | lives at | +| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `/.pem` | +| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | +| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives//attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. `ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`, public), the host's own CA bundle followed by the swarm's root from `/swarm-services-root.pem`, so that only nix trusts the root. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. | +| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | +| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | +| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives//forge/token`, for the `hive-` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk | +| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `:` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct | +| the squid self-check's password | `swarm-controller` mints `swarm/hives//squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | ### Store identities, one per principal diff --git a/docs/4-implementation/swarm/builder.md b/docs/4-implementation/swarm/builder.md index 39ed6639a..9307431e5 100644 --- a/docs/4-implementation/swarm/builder.md +++ b/docs/4-implementation/swarm/builder.md @@ -540,7 +540,8 @@ the store under the hive's own store identity every ten minutes. It writes the key to `/run/hyperhive-priv/attic/public-key` (`0700` directory, `0600` file) and the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's nix.conf includes, with the cache's address as `extra-substituters`, the key as -`extra-trusted-public-keys` and `netrc-file = /run/hyperhive-nix/netrc`. The +`extra-trusted-public-keys`, `netrc-file = /run/hyperhive-nix/netrc` and +`ssl-cert-file = /run/hyperhive-nix/ca-bundle.crt`. The netrc (`root:root 0600`) holds the pull token's entry for the cache's host and then the entries of the host's own netrc, which is `nix.settings.netrc-file` when the host sets it and nix's default `/etc/nix/netrc` otherwise, because @@ -548,6 +549,15 @@ when the host sets it and nix's default `/etc/nix/netrc` otherwise, because on every run, so an edit to the host's netrc reaches the daemon within ten minutes. An entry for the cache's own host in the host's netrc gives way to the pull token's. The unit doesn't see a netrc that the host names only in +`nix.extraOptions`. The CA bundle (`root:root 0644`) is the host's own bundle, +which is `nix.settings.ssl-cert-file` when the host sets it and +`security.pki.caBundle` otherwise, followed by the swarm's root from +`/swarm-services-root.pem`. The cache's +certificate chains to that root, which no build-time bundle holds, and every +nix client of the host (the daemon, `hive-priv`, root's `nixos-rebuild`) reads +the fragment. Only nix trusts the root; the host's system trust stays as it was. +A host without the root yet gets its own bundle alone until a later run finds +it. The unit doesn't see an `ssl-cert-file` that the host names only in `nix.extraOptions`. The cache's address is the host's `services.hyperhive.swarm.attic.domain` and `cache`. The unit restarts `nix-daemon` only when the fragment changed after the daemon started, which diff --git a/docs/4-implementation/trust-boundary/hive-c0re-privilege.md b/docs/4-implementation/trust-boundary/hive-c0re-privilege.md index 73113a235..4ead68973 100644 --- a/docs/4-implementation/trust-boundary/hive-c0re-privilege.md +++ b/docs/4-implementation/trust-boundary/hive-c0re-privilege.md @@ -89,10 +89,14 @@ the daemon accepts: store identity and writes the key to `/run/hyperhive-priv/attic/public-key` (directory `0700`, file `0600`, root). It also writes a nix.conf fragment, `/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the - address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the - host's nix.conf includes (`!include`, so a host whose unit hasn't run yet - still starts). The fragment holds no secret. `hive-priv` reads nothing from - `/run/hyperhive/attic/`, which `hive-c0re` writes for the probe. + address), `extra-trusted-public-keys` (the key), `netrc-file` and + `ssl-cert-file`, which the host's nix.conf includes (`!include`, so a host + whose unit hasn't run yet still starts). The fragment holds no secret. + `ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`): the host's + own CA bundle plus the swarm's root, so nix clients on the host trust the + cache's certificate while the host's system trust stays without the root. + `hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re` + writes for the probe. - Only root reads the token. The netrc the fragment names, `/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's diff --git a/nix/host-modules/hive-priv-cache.nix b/nix/host-modules/hive-priv-cache.nix index df1ef3d83..7cc58b79d 100644 --- a/nix/host-modules/hive-priv-cache.nix +++ b/nix/host-modules/hive-priv-cache.nix @@ -12,20 +12,18 @@ # - the cache's public key and the pull token are read from the store by a root # oneshot, `hive-priv-cache-facts`, under this hive's own store identity. The # key lands in `/run/hyperhive-priv/attic/public-key` (directory `0700`, file -# `0600`). -# The cache's address, the key and a netrc also land in a nix.conf fragment, -# `/run/hyperhive-nix/cache.conf`, as `extra-substituters`, -# `extra-trusted-public-keys` and `netrc-file`, which the host's nix.conf -# includes (`!include`, so a host whose unit has not run yet still starts). -# Every client of the daemon (hive-c0re, agent containers on `nixBuilds = -# host`) can have a swarm closure substituted; closures hold no secret; -# - that netrc, `/run/hyperhive-nix/netrc` (`root:root 0600`), is the pull -# token's entry plus the entries of the host's own netrc, which -# `netrc-file` would otherwise replace. The host's netrc is -# `nix.settings.netrc-file` when the host sets it, else nix's default -# `/etc/nix/netrc`. The unit merges again on every run, so an edit reaches the -# daemon within the timer's ten minutes. A netrc named only in -# `nix.extraOptions` is not seen. +# `0600`). The address, the key, a netrc and a CA bundle also land in a +# nix.conf fragment, `/run/hyperhive-nix/cache.conf`, as `extra-substituters`, +# `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the +# host's nix.conf includes (`!include`, so a host whose unit has not run yet +# still starts). Every nix client of the host can have a swarm closure +# substituted; the fragment holds no secret; +# - `netrc-file` and `ssl-cert-file` take one path and would replace the host's +# own, so the files they name merge it in: the pull token's entry plus the +# host's netrc entries, and the host's CA bundle plus the swarm's root, which +# the cache's certificate chains to. The host's are `nix.settings.netrc-file` +# else `/etc/nix/netrc`, and `nix.settings.ssl-cert-file` else +# `security.pki.caBundle`; one named only in `nix.extraOptions` is not seen. # # hive-c0re keeps its own copies under `/run/hyperhive/attic/` for the probe. { @@ -60,6 +58,13 @@ let netrcFile = "${confDir}/netrc"; # `netrc-file` is not additive: the fragment's setting replaces the host's. hostNetrc = toString (config.nix.settings.netrc-file or "/etc/nix/netrc"); + # `ssl-cert-file` is not additive either. The host's bundle is the one nix + # would use without the fragment: `nix.settings.ssl-cert-file` when the host + # sets it, else the system bundle the nix daemon's `CURL_CA_BUNDLE` names. + caBundleFile = "${confDir}/ca-bundle.crt"; + baseBundle = toString (config.nix.settings.ssl-cert-file or config.security.pki.caBundle); + # Written by `swarm-services-cert` on a host whose gateway is self-signed. + swarmCa = "${tls.stateDir}/swarm-services-root.pem"; # Where the store runs on this host, the first login waits for its container. # No `wants`: a pull would start the container mid-switch on the old closure. @@ -118,6 +123,9 @@ in confFile netrcFile hostNetrc + caBundleFile + baseBundle + swarmCa ; url = cacheUrl; }; diff --git a/nix/host-modules/lib/attic-root-facts.nix b/nix/host-modules/lib/attic-root-facts.nix index 4734e2e8a..bfce4540c 100644 --- a/nix/host-modules/lib/attic-root-facts.nix +++ b/nix/host-modules/lib/attic-root-facts.nix @@ -1,32 +1,32 @@ -# The root-side copy of the cache facts the host's nix daemon and hive-priv -# use: the key a downloaded closure must be signed by and the netrc the pull -# token goes out in. The script runs as root on the hive host under the hive's -# own store identity and writes: +# The root-side copy of the cache facts the host's nix daemon and hive-priv use: +# the key a closure must be signed by, the netrc the pull token goes out in and +# the CA bundle that trusts the cache. The script runs as root on the hive host +# under the hive's own store identity and writes: # # - the key into `dir`, which only root can read; -# - the netrc at `netrcFile`, root 0600: the pull token's entry for `host` -# followed by the entries of the host's own netrc (`hostNetrc`) for other -# machines. `netrc-file` is one path, so a fragment that names this file must -# carry what the host already keeps there; +# - the netrc at `netrcFile`, root 0600: the pull token's entry for `host` then +# the entries of the host's own netrc (`hostNetrc`), as `netrc-file` is one path; +# - the CA bundle at `caBundleFile`, root 0644: the host's own bundle +# (`baseBundle`) then the swarm's root (`swarmCa`), when the host holds it, +# which the cache's certificate chains to; `ssl-cert-file` is one path too; # - a nix.conf fragment at `confFile` that the host's nix.conf includes, with -# `extra-substituters` (the cache's address), `extra-trusted-public-keys` and -# `netrc-file` (that merged netrc), so the daemon -# substitutes from the cache and its other substituters. It holds no secret. +# `extra-substituters` (the cache's address), `extra-trusted-public-keys`, +# `netrc-file` and `ssl-cert-file`. It holds no secret. # # Nothing is read from a file hive-c0re (unprivileged) writes: the key and the -# token come from the store. +# token come from the store. A store that holds no facts yet, or cannot be +# reached, leaves what is in `dir`, `netrcFile`, `caBundleFile` and `confFile` +# as they are. Nothing here removes a file. # -# A store that holds no facts yet, or cannot be reached, leaves what is in -# `dir`, `netrcFile` and `confFile` as they are. Nothing here removes a file. -# -# Pure function — NOT a NixOS module. Call it from a unit's `script`, with -# `bao`, `coreutils`, `gnugrep` and `systemd` on its `path` and the `BAO_*` -# client environment set: +# Pure function — NOT a NixOS module. Call it from a unit's `script`, with `bao`, +# `coreutils`, `gnugrep`, `systemd` on its `path` and the `BAO_*` client set: # # script = import ./lib/attic-root-facts.nix { inherit lib; } { # hive = "alpha"; host = "attic.example.org"; dir = "/run/hyperhive-priv/attic"; # url = "https://attic.example.org/swarm"; confFile = "/run/hyperhive-nix/cache.conf"; # netrcFile = "/run/hyperhive-nix/netrc"; hostNetrc = "/etc/nix/netrc"; +# caBundleFile = "/run/hyperhive-nix/ca-bundle.crt"; baseBundle = "/etc/ssl/certs/ca-certificates.crt"; +# swarmCa = "/var/lib/hive-tls/swarm-services-root.pem"; # }; { lib }: { @@ -45,6 +45,15 @@ # module, which is merged in and never written. netrcFile, hostNetrc, + # Where the CA bundle goes. + caBundleFile, + # The CA bundle nix uses without this module, copied in and never written: + # `ssl-cert-file` is one path, so a fragment that names the new bundle must + # carry the host's CAs. + baseBundle, + # The swarm's root certificate. The file may be absent: the bundle then holds + # the host's CAs alone until a later run finds it. + swarmCa, }: '' set -euo pipefail @@ -56,8 +65,11 @@ conf=${lib.escapeShellArg confFile} netrc_file=${lib.escapeShellArg netrcFile} host_netrc=${lib.escapeShellArg hostNetrc} + ca_bundle_file=${lib.escapeShellArg caBundleFile} + base_bundle=${lib.escapeShellArg baseBundle} + swarm_ca=${lib.escapeShellArg swarmCa} install -d -m 0700 "$dir" - install -d -m 0755 "$(dirname -- "$conf")" "$(dirname -- "$netrc_file")" + install -d -m 0755 "$(dirname -- "$conf")" "$(dirname -- "$netrc_file")" "$(dirname -- "$ca_bundle_file")" store_init store_login @@ -128,14 +140,33 @@ atomic_write_secret 0600 "" "$netrc_file" "$netrc" fi - # Last, so a daemon that reads the fragment finds the netrc in place. The - # mtime moves only with the content. Lix re-reads nix.conf for every - # connection, CppNix only at start, so a CppNix daemon that started before - # this write is restarted. - printf -v fragment '%s\n%s\n%s' \ + # The host's CAs, then the swarm's root. Nix reads the file for each + # download, so a root that lands after the fragment reaches the next one + # without a restart. A file without a certificate is not appended: the bundle + # serves every https fetch of every nix client. + if [ ! -r "$base_bundle" ]; then + echo "cannot read the host's CA bundle $base_bundle; $ca_bundle_file stays as it is." >&2 + exit 1 + fi + bundle="$(< "$base_bundle")" + if [ -s "$swarm_ca" ] && grep -q -- '-----BEGIN CERTIFICATE-----' "$swarm_ca"; then + bundle+=$'\n'"$(< "$swarm_ca")" + else + echo "the swarm's root $swarm_ca is not on this host yet; $ca_bundle_file holds the host's CAs only." + fi + if secret_differs "$ca_bundle_file" "$bundle" || [ "$(stat -c %a "$ca_bundle_file")" != 644 ]; then + atomic_write_secret 0644 "" "$ca_bundle_file" "$bundle" + fi + + # Last, so a daemon that reads the fragment finds the netrc and the bundle in + # place. The mtime moves only with the content. Lix re-reads nix.conf for + # every connection, CppNix only at start, so a CppNix daemon that started + # before this write is restarted. + printf -v fragment '%s\n%s\n%s\n%s' \ ${lib.escapeShellArg "extra-substituters = ${url}"} \ "extra-trusted-public-keys = $key" \ - "netrc-file = $netrc_file" + "netrc-file = $netrc_file" \ + "ssl-cert-file = $ca_bundle_file" if secret_differs "$conf" "$fragment"; then atomic_write_secret 0644 "" "$conf" "$fragment" fi diff --git a/nix/module-eval/hive-tls.nix b/nix/module-eval/hive-tls.nix index 4659b928b..b8ab9aa29 100644 --- a/nix/module-eval/hive-tls.nix +++ b/nix/module-eval/hive-tls.nix @@ -24,6 +24,7 @@ let ; }) hive + hiveWith hiveWithController runGroup ; @@ -41,16 +42,18 @@ let # Fronting none of the swarm's names, reading a store it does not run, # with the reader identity, the services-issuer pair and the hive's own - # identity placed by hand. - noService = hiveWithController { - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem"; - deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; - deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; - deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem"; - deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem"; - }; + # identity placed by hand, and its own `ssl-cert-file` in nix.conf. + noService = hiveWith [ { nix.settings.ssl-cert-file = "/etc/pki/host-ca.pem"; } ] ( + lib.recursiveUpdate { deploy.hive-controller.enable = true; } { + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem"; + deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; + deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; + deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem"; + deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem"; + } + ); noSvcFetch = noService.systemd.services.swarm-services-cert.script; # The same hive holding its own identity and no services-issuer pair: the @@ -898,6 +901,15 @@ let && lib.hasInfix ''atomic_write_secret 0644 "" "$conf"'' u.script # The host's own netrc is merged in: nix's default path when the host sets none. && lib.hasInfix "host_netrc=/etc/nix/netrc" u.script + # The fragment names the bundle: the host's own CAs plus the swarm's root, in a file of its own. + && lib.hasInfix "ssl-cert-file = $ca_bundle_file" u.script + && lib.hasInfix "ca_bundle_file=/run/hyperhive-nix/ca-bundle.crt" u.script + && lib.hasInfix "swarm_ca=/var/lib/hive-tls/swarm-services-root.pem" u.script + && lib.hasInfix ''atomic_write_secret 0644 "" "$ca_bundle_file"'' u.script + # No ssl-cert-file on the host: the system bundle is the base. A host that sets one is merged, not replaced. + && lib.hasInfix "base_bundle=${allLocal.security.pki.caBundle}" u.script + && lib.hasInfix "base_bundle=/etc/pki/host-ca.pem" noService.systemd.services.hive-priv-cache-facts.script + && !(lib.hasInfix "base_bundle=${allLocal.security.pki.caBundle}" noService.systemd.services.hive-priv-cache-facts.script) # No client copy of the token for hive-core, and hive-c0re is not told the cache. && !(lib.hasInfix "client_netrc" u.script) && !(c0re ? HIVE_C0RE_CACHE_URL) diff --git a/nix/script-tests/attic-root-facts.nix b/nix/script-tests/attic-root-facts.nix index dea81c444..73476d9c4 100644 --- a/nix/script-tests/attic-root-facts.nix +++ b/nix/script-tests/attic-root-facts.nix @@ -4,30 +4,39 @@ # ./attic-root-facts.sh. # # What runs is the rendered script, with the directory it writes moved into the -# test's scratch space. +# test's scratch space. `SCRIPT_HOSTCA` is the same script for a host whose nix +# sets its own `ssl-cert-file`, which becomes the base of the CA bundle. { pkgs, ... }: let - script = pkgs.writeText "attic-root-facts.sh" ( - import ../host-modules/lib/attic-root-facts.nix { inherit (pkgs) lib; } { - hive = "alpha"; - host = "cache.example"; - dir = "@DIR@/attic"; - url = "https://cache.example/swarm"; - confFile = "@DIR@/nix/cache.conf"; - netrcFile = "@DIR@/nix/netrc"; - hostNetrc = "@DIR@/host-netrc"; - } - ); + render = + baseBundle: + pkgs.writeText "attic-root-facts.sh" ( + import ../host-modules/lib/attic-root-facts.nix { inherit (pkgs) lib; } { + hive = "alpha"; + host = "cache.example"; + dir = "@DIR@/attic"; + url = "https://cache.example/swarm"; + confFile = "@DIR@/nix/cache.conf"; + netrcFile = "@DIR@/nix/netrc"; + hostNetrc = "@DIR@/host-netrc"; + caBundleFile = "@DIR@/nix/ca-bundle.crt"; + inherit baseBundle; + swarmCa = "@DIR@/swarm-root.pem"; + } + ); in pkgs.runCommand "hyperhive-script-test-attic-root-facts" { - SCRIPT = script; + SCRIPT = render "@DIR@/system-bundle.crt"; + SCRIPT_HOSTCA = render "@DIR@/host-ca.pem"; nativeBuildInputs = [ pkgs.coreutils + pkgs.curl pkgs.gnugrep + pkgs.openssl ]; } '' diff --git a/nix/script-tests/attic-root-facts.sh b/nix/script-tests/attic-root-facts.sh index 9179c12bf..54ace8e60 100644 --- a/nix/script-tests/attic-root-facts.sh +++ b/nix/script-tests/attic-root-facts.sh @@ -62,11 +62,39 @@ STUB chmod +x "$dir/bin/systemctl" export PATH="$dir/bin:$PATH" +# Certificates made here, with fixed names and subjects: `system-ca` stands for +# a CA the host's own bundle holds, `swarm-ca` for the swarm's root, `host-ca` +# for a bundle the host's nix.conf names. `swarm-leaf` and `system-leaf` are +# servers for localhost signed by the first two. +pki=$dir/pki +mkdir -p "$pki" +newca() { # + openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes \ + -keyout "$pki/$1.key" -out "$pki/$1.pem" -subj "/CN=$1" -days 2 \ + -addext basicConstraints=critical,CA:TRUE -addext keyUsage=keyCertSign 2>/dev/null +} +newleaf() { # + openssl req -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes \ + -keyout "$pki/$2.key" -out "$pki/$2.csr" -subj "/CN=localhost" 2>/dev/null + printf 'subjectAltName=DNS:localhost\nbasicConstraints=CA:FALSE\n' >"$pki/$2.ext" + openssl x509 -req -in "$pki/$2.csr" -CA "$pki/$1.pem" -CAkey "$pki/$1.key" \ + -CAcreateserial -out "$pki/$2.pem" -days 2 -extfile "$pki/$2.ext" 2>/dev/null +} +newca system-ca +newca swarm-ca +newca host-ca +newleaf swarm-ca swarm-leaf +newleaf system-ca system-leaf +certs() { grep -c -- '-----BEGIN CERTIFICATE-----' "$1" 2>/dev/null || echo 0; } + KEYLEAF=secret_swarm_hives_alpha_attic_public-key TOKENLEAF=secret_swarm_hives_alpha_attic_pull-token reset() { rm -rf "$dir/store" "$dir/attic" "$dir/nix" "$dir/host-netrc" "$calls" "$dir/systemctl" + cp "$pki/system-ca.pem" "$dir/system-bundle.crt" + cp "$pki/swarm-ca.pem" "$dir/swarm-root.pem" + cp "$pki/host-ca.pem" "$dir/host-ca.pem" mkdir -p "$dir/store" : >"$calls" : >"$dir/systemctl" @@ -97,9 +125,10 @@ expect "netrc" "$(body "$dir/nix/netrc")" "machine cache.example password tok-1" expect "key mode" "$(mode "$dir/attic/public-key")" 600 expect "netrc mode" "$(mode "$dir/nix/netrc")" 600 expect "dir mode" "$(mode "$dir/attic")" 700 -expect "fragment names the cache, the key and the root-only netrc" "$(body "$dir/nix/cache.conf")" "extra-substituters = https://cache.example/swarm +expect "fragment names the cache, the key, the root-only netrc and the CA bundle" "$(body "$dir/nix/cache.conf")" "extra-substituters = https://cache.example/swarm extra-trusted-public-keys = swarm:S0VZ -netrc-file = $dir/nix/netrc" +netrc-file = $dir/nix/netrc +ssl-cert-file = $dir/nix/ca-bundle.crt" expect "fragment mode" "$(mode "$dir/nix/cache.conf")" 644 expect "fragment dir mode" "$(mode "$dir/nix")" 755 expect "no secret in the fragment" "$(grep -c tok-1 "$dir/nix/cache.conf")" 0 @@ -314,6 +343,123 @@ run expect "exit" $? 1 expect "no fragment" "$(body "$dir/nix/cache.conf")" missing +# The CA bundle the fragment names: the host's own CAs, then the swarm's root. +bundle=$dir/nix/ca-bundle.crt + +case="the bundle holds the host's CAs and the swarm's root, 0644, and the fragment names it" +reset +seed "swarm:S0VZ" "tok-1" +run +expect "exit" $? 0 +expect "bundle" "$(body "$bundle")" "$(cat "$pki/system-ca.pem") +$(cat "$pki/swarm-ca.pem")" +expect "two certificates" "$(certs "$bundle")" 2 +expect "mode" "$(mode "$bundle")" 644 +expect "fragment names it" "$(grep -c "^ssl-cert-file = $bundle\$" "$dir/nix/cache.conf")" 1 +expect "control: the host's bundle held one certificate" "$(certs "$dir/system-bundle.crt")" 1 +expect "the host's bundle is not written" "$(body "$dir/system-bundle.crt")" "$(cat "$pki/system-ca.pem")" + +case="the swarm and host servers verify against the bundle, and the swarm one does not against the host CAs alone" +expect "swarm leaf, bundle" "$(openssl verify -CAfile "$bundle" "$pki/swarm-leaf.pem" >/dev/null 2>&1 && echo ok || echo bad)" ok +expect "system leaf, bundle" "$(openssl verify -CAfile "$bundle" "$pki/system-leaf.pem" >/dev/null 2>&1 && echo ok || echo bad)" ok +expect "control: swarm leaf, host's CAs alone" "$(openssl verify -CAfile "$dir/system-bundle.crt" "$pki/swarm-leaf.pem" >/dev/null 2>&1 && echo ok || echo bad)" bad + +# curl takes the file as nix does. A TLS server for localhost with the swarm's +# leaf, on a port picked at random until one binds. +case="curl trusts the swarm's server with the bundle and refuses it with the host's CAs alone" +port= +for _ in 1 2 3 4 5 6 7 8; do + p=$((20000 + RANDOM % 20000)) + openssl s_server -www -accept "$p" -cert "$pki/swarm-leaf.pem" -key "$pki/swarm-leaf.key" \ + >"$dir/s_server.log" 2>&1 & + server=$! + for _ in 1 2 3 4 5 6 7 8 9 10; do + if curl -s --cacert "$pki/swarm-ca.pem" "https://localhost:$p/" >/dev/null 2>&1; then port=$p; break; fi + kill -0 "$server" 2>/dev/null || break + sleep 0.3 + done + [ -n "$port" ] && break + kill "$server" 2>/dev/null +done +if [ -z "$port" ]; then + fail "no TLS server for the curl check came up: $(cat "$dir/s_server.log")" +else + curl -s --cacert "$bundle" "https://localhost:$port/" >/dev/null 2>&1 + expect "with the bundle" $? 0 + curl -s --cacert "$dir/system-bundle.crt" "https://localhost:$port/" >/dev/null 2>&1 + expect "with the host's CAs alone" $? 60 + kill "$server" 2>/dev/null +fi + +case="a second run with the same CAs rewrites nothing" +b=$(ino "$bundle") +run +expect "exit" $? 0 +expect "bundle untouched" "$(ino "$bundle")" "$b" + +case="a bundle left 0640 is rewritten 0644 though its content is current" +chmod 640 "$bundle" +run +expect "exit" $? 0 +expect "mode" "$(mode "$bundle")" 644 + +case="a host that sets its own ssl-cert-file has that file merged in, not replaced" +reset +seed "swarm:S0VZ" "tok-1" +sed -e "s|@DIR@|$dir|g" "$SCRIPT_HOSTCA" >"$dir/script-hostca.sh" +cp "$dir/script.sh" "$dir/script-default.sh" +cp "$dir/script-hostca.sh" "$dir/script.sh" +run +expect "exit" $? 0 +expect "the host's file and the swarm's root" "$(body "$bundle")" "$(cat "$pki/host-ca.pem") +$(cat "$pki/swarm-ca.pem")" +line=$(sed -n 2p "$pki/system-ca.pem") +expect "the default bundle's CA is not in it" "$(grep -c -x -F -- "$line" "$bundle")" 0 +expect "control: the same search finds it in its own file" "$(grep -c -x -F -- "$line" "$pki/system-ca.pem")" 1 +expect "the host's file is not written" "$(body "$dir/host-ca.pem")" "$(cat "$pki/host-ca.pem")" +cat "$pki/system-ca.pem" >>"$dir/host-ca.pem" +run +expect "an edit to the host's file reaches the bundle" "$(certs "$bundle")" 3 +cp "$dir/script-default.sh" "$dir/script.sh" + +case="a swarm root that is not on the host yet leaves the host's CAs alone, and a later run adds it" +reset +seed "swarm:S0VZ" "tok-1" +rm "$dir/swarm-root.pem" +run DAEMON_STARTED="Sat 2000-01-01 00:00:00.000000 UTC" +expect "exit" $? 0 +expect "bundle" "$(body "$bundle")" "$(cat "$pki/system-ca.pem")" +expect "the fragment still names it" "$(grep -c "^ssl-cert-file = $bundle\$" "$dir/nix/cache.conf")" 1 +expect "said so" "$(grep -c "swarm-root.pem is not on this host yet" "$dir/out")" 1 +started=$(now) +c=$(ino "$dir/nix/cache.conf") +: >"$dir/systemctl" +cp "$pki/swarm-ca.pem" "$dir/swarm-root.pem" +run DAEMON_STARTED="$started" +expect "exit" $? 0 +expect "root added" "$(certs "$bundle")" 2 +expect "fragment untouched" "$(ino "$dir/nix/cache.conf")" "$c" +expect "no restart: nix reads the bundle per download" "$(restarts)" 0 + +case="a swarm root file holding no certificate is not appended" +reset +seed "swarm:S0VZ" "tok-1" +echo "garbage" >"$dir/swarm-root.pem" +run +expect "exit" $? 0 +expect "bundle" "$(body "$bundle")" "$(cat "$pki/system-ca.pem")" + +case="a host bundle that cannot be read fails the run and leaves the fragment and bundle" +good +cp "$bundle" "$dir/bundle-before" +rm "$dir/system-bundle.crt" +seed "swarm:NEW" "tok-new" +run +expect "exit" $? 1 +expect "says why" "$(grep -c "cannot read the host.s CA bundle" "$dir/err")" 1 +expect "bundle kept" "$(body "$bundle")" "$(body "$dir/bundle-before")" +expect "fragment kept" "$(grep -c 'extra-trusted-public-keys = swarm:S0VZ' "$dir/nix/cache.conf")" 1 + if [ "$failures" -ne 0 ]; then echo "$failures of $count checks failed" >&2 exit 1