nix trusts the swarm's root through its own CA bundle, so root's nix clients reach the cache
This commit is contained in:
atlas
parent
526cfbec1b
commit
9d62ef44b5
8 changed files with 299 additions and 79 deletions
|
|
@ -151,16 +151,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
|
||||
### Hive-level — one of each per hive
|
||||
|
||||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys` and `netrc-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. `ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`, public), the host's own CA bundle followed by the swarm's root from `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem`, so that only nix trusts the root. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity for each forge read, and hands it to the `git` commands it runs. It keeps no copy on disk |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.deploy.hive-controller.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
|
||||
### Store identities, one per principal
|
||||
|
||||
|
|
|
|||
|
|
@ -540,7 +540,8 @@ the store under the hive's own store identity every ten minutes. It writes the
|
|||
key to `/run/hyperhive-priv/attic/public-key` (`0700` directory, `0600` file)
|
||||
and the nix.conf fragment `/run/hyperhive-nix/cache.conf`, which the host's
|
||||
nix.conf includes, with the cache's address as `extra-substituters`, the key as
|
||||
`extra-trusted-public-keys` and `netrc-file = /run/hyperhive-nix/netrc`. The
|
||||
`extra-trusted-public-keys`, `netrc-file = /run/hyperhive-nix/netrc` and
|
||||
`ssl-cert-file = /run/hyperhive-nix/ca-bundle.crt`. The
|
||||
netrc (`root:root 0600`) holds the pull token's entry for the cache's host and
|
||||
then the entries of the host's own netrc, which is `nix.settings.netrc-file`
|
||||
when the host sets it and nix's default `/etc/nix/netrc` otherwise, because
|
||||
|
|
@ -548,6 +549,15 @@ when the host sets it and nix's default `/etc/nix/netrc` otherwise, because
|
|||
on every run, so an edit to the host's netrc reaches the daemon within ten
|
||||
minutes. An entry for the cache's own host in the host's netrc gives way to the
|
||||
pull token's. The unit doesn't see a netrc that the host names only in
|
||||
`nix.extraOptions`. The CA bundle (`root:root 0644`) is the host's own bundle,
|
||||
which is `nix.settings.ssl-cert-file` when the host sets it and
|
||||
`security.pki.caBundle` otherwise, followed by the swarm's root from
|
||||
`<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem`. The cache's
|
||||
certificate chains to that root, which no build-time bundle holds, and every
|
||||
nix client of the host (the daemon, `hive-priv`, root's `nixos-rebuild`) reads
|
||||
the fragment. Only nix trusts the root; the host's system trust stays as it was.
|
||||
A host without the root yet gets its own bundle alone until a later run finds
|
||||
it. The unit doesn't see an `ssl-cert-file` that the host names only in
|
||||
`nix.extraOptions`. The cache's address is the host's
|
||||
`services.hyperhive.swarm.attic.domain` and `cache`. The unit restarts
|
||||
`nix-daemon` only when the fragment changed after the daemon started, which
|
||||
|
|
|
|||
|
|
@ -89,10 +89,14 @@ the daemon accepts:
|
|||
store identity and writes the key to `/run/hyperhive-priv/attic/public-key`
|
||||
(directory `0700`, file `0600`, root). It also writes a nix.conf fragment,
|
||||
`/run/hyperhive-nix/cache.conf` (`0644`), with `extra-substituters` (the
|
||||
address), `extra-trusted-public-keys` (the key) and `netrc-file`, which the
|
||||
host's nix.conf includes (`!include`, so a host whose unit hasn't run yet
|
||||
still starts). The fragment holds no secret. `hive-priv` reads nothing from
|
||||
`/run/hyperhive/attic/`, which `hive-c0re` writes for the probe.
|
||||
address), `extra-trusted-public-keys` (the key), `netrc-file` and
|
||||
`ssl-cert-file`, which the host's nix.conf includes (`!include`, so a host
|
||||
whose unit hasn't run yet still starts). The fragment holds no secret.
|
||||
`ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`): the host's
|
||||
own CA bundle plus the swarm's root, so nix clients on the host trust the
|
||||
cache's certificate while the host's system trust stays without the root.
|
||||
`hive-priv` reads nothing from `/run/hyperhive/attic/`, which `hive-c0re`
|
||||
writes for the probe.
|
||||
- Only root reads the token. The netrc the fragment names,
|
||||
`/run/hyperhive-nix/netrc` (`root:root 0600`), holds the token's entry plus
|
||||
the entries of the host's own netrc (`nix.settings.netrc-file`, else nix's
|
||||
|
|
|
|||
|
|
@ -12,20 +12,18 @@
|
|||
# - the cache's public key and the pull token are read from the store by a root
|
||||
# oneshot, `hive-priv-cache-facts`, under this hive's own store identity. The
|
||||
# key lands in `/run/hyperhive-priv/attic/public-key` (directory `0700`, file
|
||||
# `0600`).
|
||||
# The cache's address, the key and a netrc also land in a nix.conf fragment,
|
||||
# `/run/hyperhive-nix/cache.conf`, as `extra-substituters`,
|
||||
# `extra-trusted-public-keys` and `netrc-file`, which the host's nix.conf
|
||||
# includes (`!include`, so a host whose unit has not run yet still starts).
|
||||
# Every client of the daemon (hive-c0re, agent containers on `nixBuilds =
|
||||
# host`) can have a swarm closure substituted; closures hold no secret;
|
||||
# - that netrc, `/run/hyperhive-nix/netrc` (`root:root 0600`), is the pull
|
||||
# token's entry plus the entries of the host's own netrc, which
|
||||
# `netrc-file` would otherwise replace. The host's netrc is
|
||||
# `nix.settings.netrc-file` when the host sets it, else nix's default
|
||||
# `/etc/nix/netrc`. The unit merges again on every run, so an edit reaches the
|
||||
# daemon within the timer's ten minutes. A netrc named only in
|
||||
# `nix.extraOptions` is not seen.
|
||||
# `0600`). The address, the key, a netrc and a CA bundle also land in a
|
||||
# nix.conf fragment, `/run/hyperhive-nix/cache.conf`, as `extra-substituters`,
|
||||
# `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the
|
||||
# host's nix.conf includes (`!include`, so a host whose unit has not run yet
|
||||
# still starts). Every nix client of the host can have a swarm closure
|
||||
# substituted; the fragment holds no secret;
|
||||
# - `netrc-file` and `ssl-cert-file` take one path and would replace the host's
|
||||
# own, so the files they name merge it in: the pull token's entry plus the
|
||||
# host's netrc entries, and the host's CA bundle plus the swarm's root, which
|
||||
# the cache's certificate chains to. The host's are `nix.settings.netrc-file`
|
||||
# else `/etc/nix/netrc`, and `nix.settings.ssl-cert-file` else
|
||||
# `security.pki.caBundle`; one named only in `nix.extraOptions` is not seen.
|
||||
#
|
||||
# hive-c0re keeps its own copies under `/run/hyperhive/attic/` for the probe.
|
||||
{
|
||||
|
|
@ -60,6 +58,13 @@ let
|
|||
netrcFile = "${confDir}/netrc";
|
||||
# `netrc-file` is not additive: the fragment's setting replaces the host's.
|
||||
hostNetrc = toString (config.nix.settings.netrc-file or "/etc/nix/netrc");
|
||||
# `ssl-cert-file` is not additive either. The host's bundle is the one nix
|
||||
# would use without the fragment: `nix.settings.ssl-cert-file` when the host
|
||||
# sets it, else the system bundle the nix daemon's `CURL_CA_BUNDLE` names.
|
||||
caBundleFile = "${confDir}/ca-bundle.crt";
|
||||
baseBundle = toString (config.nix.settings.ssl-cert-file or config.security.pki.caBundle);
|
||||
# Written by `swarm-services-cert` on a host whose gateway is self-signed.
|
||||
swarmCa = "${tls.stateDir}/swarm-services-root.pem";
|
||||
|
||||
# Where the store runs on this host, the first login waits for its container.
|
||||
# No `wants`: a pull would start the container mid-switch on the old closure.
|
||||
|
|
@ -118,6 +123,9 @@ in
|
|||
confFile
|
||||
netrcFile
|
||||
hostNetrc
|
||||
caBundleFile
|
||||
baseBundle
|
||||
swarmCa
|
||||
;
|
||||
url = cacheUrl;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,32 +1,32 @@
|
|||
# The root-side copy of the cache facts the host's nix daemon and hive-priv
|
||||
# use: the key a downloaded closure must be signed by and the netrc the pull
|
||||
# token goes out in. The script runs as root on the hive host under the hive's
|
||||
# own store identity and writes:
|
||||
# The root-side copy of the cache facts the host's nix daemon and hive-priv use:
|
||||
# the key a closure must be signed by, the netrc the pull token goes out in and
|
||||
# the CA bundle that trusts the cache. The script runs as root on the hive host
|
||||
# under the hive's own store identity and writes:
|
||||
#
|
||||
# - the key into `dir`, which only root can read;
|
||||
# - the netrc at `netrcFile`, root 0600: the pull token's entry for `host`
|
||||
# followed by the entries of the host's own netrc (`hostNetrc`) for other
|
||||
# machines. `netrc-file` is one path, so a fragment that names this file must
|
||||
# carry what the host already keeps there;
|
||||
# - the netrc at `netrcFile`, root 0600: the pull token's entry for `host` then
|
||||
# the entries of the host's own netrc (`hostNetrc`), as `netrc-file` is one path;
|
||||
# - the CA bundle at `caBundleFile`, root 0644: the host's own bundle
|
||||
# (`baseBundle`) then the swarm's root (`swarmCa`), when the host holds it,
|
||||
# which the cache's certificate chains to; `ssl-cert-file` is one path too;
|
||||
# - a nix.conf fragment at `confFile` that the host's nix.conf includes, with
|
||||
# `extra-substituters` (the cache's address), `extra-trusted-public-keys` and
|
||||
# `netrc-file` (that merged netrc), so the daemon
|
||||
# substitutes from the cache and its other substituters. It holds no secret.
|
||||
# `extra-substituters` (the cache's address), `extra-trusted-public-keys`,
|
||||
# `netrc-file` and `ssl-cert-file`. It holds no secret.
|
||||
#
|
||||
# Nothing is read from a file hive-c0re (unprivileged) writes: the key and the
|
||||
# token come from the store.
|
||||
# token come from the store. A store that holds no facts yet, or cannot be
|
||||
# reached, leaves what is in `dir`, `netrcFile`, `caBundleFile` and `confFile`
|
||||
# as they are. Nothing here removes a file.
|
||||
#
|
||||
# A store that holds no facts yet, or cannot be reached, leaves what is in
|
||||
# `dir`, `netrcFile` and `confFile` as they are. Nothing here removes a file.
|
||||
#
|
||||
# Pure function — NOT a NixOS module. Call it from a unit's `script`, with
|
||||
# `bao`, `coreutils`, `gnugrep` and `systemd` on its `path` and the `BAO_*`
|
||||
# client environment set:
|
||||
# Pure function — NOT a NixOS module. Call it from a unit's `script`, with `bao`,
|
||||
# `coreutils`, `gnugrep`, `systemd` on its `path` and the `BAO_*` client set:
|
||||
#
|
||||
# script = import ./lib/attic-root-facts.nix { inherit lib; } {
|
||||
# hive = "alpha"; host = "attic.example.org"; dir = "/run/hyperhive-priv/attic";
|
||||
# url = "https://attic.example.org/swarm"; confFile = "/run/hyperhive-nix/cache.conf";
|
||||
# netrcFile = "/run/hyperhive-nix/netrc"; hostNetrc = "/etc/nix/netrc";
|
||||
# caBundleFile = "/run/hyperhive-nix/ca-bundle.crt"; baseBundle = "/etc/ssl/certs/ca-certificates.crt";
|
||||
# swarmCa = "/var/lib/hive-tls/swarm-services-root.pem";
|
||||
# };
|
||||
{ lib }:
|
||||
{
|
||||
|
|
@ -45,6 +45,15 @@
|
|||
# module, which is merged in and never written.
|
||||
netrcFile,
|
||||
hostNetrc,
|
||||
# Where the CA bundle goes.
|
||||
caBundleFile,
|
||||
# The CA bundle nix uses without this module, copied in and never written:
|
||||
# `ssl-cert-file` is one path, so a fragment that names the new bundle must
|
||||
# carry the host's CAs.
|
||||
baseBundle,
|
||||
# The swarm's root certificate. The file may be absent: the bundle then holds
|
||||
# the host's CAs alone until a later run finds it.
|
||||
swarmCa,
|
||||
}:
|
||||
''
|
||||
set -euo pipefail
|
||||
|
|
@ -56,8 +65,11 @@
|
|||
conf=${lib.escapeShellArg confFile}
|
||||
netrc_file=${lib.escapeShellArg netrcFile}
|
||||
host_netrc=${lib.escapeShellArg hostNetrc}
|
||||
ca_bundle_file=${lib.escapeShellArg caBundleFile}
|
||||
base_bundle=${lib.escapeShellArg baseBundle}
|
||||
swarm_ca=${lib.escapeShellArg swarmCa}
|
||||
install -d -m 0700 "$dir"
|
||||
install -d -m 0755 "$(dirname -- "$conf")" "$(dirname -- "$netrc_file")"
|
||||
install -d -m 0755 "$(dirname -- "$conf")" "$(dirname -- "$netrc_file")" "$(dirname -- "$ca_bundle_file")"
|
||||
|
||||
store_init
|
||||
store_login
|
||||
|
|
@ -128,14 +140,33 @@
|
|||
atomic_write_secret 0600 "" "$netrc_file" "$netrc"
|
||||
fi
|
||||
|
||||
# Last, so a daemon that reads the fragment finds the netrc in place. The
|
||||
# mtime moves only with the content. Lix re-reads nix.conf for every
|
||||
# connection, CppNix only at start, so a CppNix daemon that started before
|
||||
# this write is restarted.
|
||||
printf -v fragment '%s\n%s\n%s' \
|
||||
# The host's CAs, then the swarm's root. Nix reads the file for each
|
||||
# download, so a root that lands after the fragment reaches the next one
|
||||
# without a restart. A file without a certificate is not appended: the bundle
|
||||
# serves every https fetch of every nix client.
|
||||
if [ ! -r "$base_bundle" ]; then
|
||||
echo "cannot read the host's CA bundle $base_bundle; $ca_bundle_file stays as it is." >&2
|
||||
exit 1
|
||||
fi
|
||||
bundle="$(< "$base_bundle")"
|
||||
if [ -s "$swarm_ca" ] && grep -q -- '-----BEGIN CERTIFICATE-----' "$swarm_ca"; then
|
||||
bundle+=$'\n'"$(< "$swarm_ca")"
|
||||
else
|
||||
echo "the swarm's root $swarm_ca is not on this host yet; $ca_bundle_file holds the host's CAs only."
|
||||
fi
|
||||
if secret_differs "$ca_bundle_file" "$bundle" || [ "$(stat -c %a "$ca_bundle_file")" != 644 ]; then
|
||||
atomic_write_secret 0644 "" "$ca_bundle_file" "$bundle"
|
||||
fi
|
||||
|
||||
# Last, so a daemon that reads the fragment finds the netrc and the bundle in
|
||||
# place. The mtime moves only with the content. Lix re-reads nix.conf for
|
||||
# every connection, CppNix only at start, so a CppNix daemon that started
|
||||
# before this write is restarted.
|
||||
printf -v fragment '%s\n%s\n%s\n%s' \
|
||||
${lib.escapeShellArg "extra-substituters = ${url}"} \
|
||||
"extra-trusted-public-keys = $key" \
|
||||
"netrc-file = $netrc_file"
|
||||
"netrc-file = $netrc_file" \
|
||||
"ssl-cert-file = $ca_bundle_file"
|
||||
if secret_differs "$conf" "$fragment"; then
|
||||
atomic_write_secret 0644 "" "$conf" "$fragment"
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ let
|
|||
;
|
||||
})
|
||||
hive
|
||||
hiveWith
|
||||
hiveWithController
|
||||
runGroup
|
||||
;
|
||||
|
|
@ -41,16 +42,18 @@ let
|
|||
|
||||
# Fronting none of the swarm's names, reading a store it does not run,
|
||||
# with the reader identity, the services-issuer pair and the hive's own
|
||||
# identity placed by hand.
|
||||
noService = hiveWithController {
|
||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem";
|
||||
deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem";
|
||||
deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem";
|
||||
deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem";
|
||||
deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem";
|
||||
};
|
||||
# identity placed by hand, and its own `ssl-cert-file` in nix.conf.
|
||||
noService = hiveWith [ { nix.settings.ssl-cert-file = "/etc/pki/host-ca.pem"; } ] (
|
||||
lib.recursiveUpdate { deploy.hive-controller.enable = true; } {
|
||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem";
|
||||
deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem";
|
||||
deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem";
|
||||
deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem";
|
||||
deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem";
|
||||
}
|
||||
);
|
||||
noSvcFetch = noService.systemd.services.swarm-services-cert.script;
|
||||
|
||||
# The same hive holding its own identity and no services-issuer pair: the
|
||||
|
|
@ -898,6 +901,15 @@ let
|
|||
&& lib.hasInfix ''atomic_write_secret 0644 "" "$conf"'' u.script
|
||||
# The host's own netrc is merged in: nix's default path when the host sets none.
|
||||
&& lib.hasInfix "host_netrc=/etc/nix/netrc" u.script
|
||||
# The fragment names the bundle: the host's own CAs plus the swarm's root, in a file of its own.
|
||||
&& lib.hasInfix "ssl-cert-file = $ca_bundle_file" u.script
|
||||
&& lib.hasInfix "ca_bundle_file=/run/hyperhive-nix/ca-bundle.crt" u.script
|
||||
&& lib.hasInfix "swarm_ca=/var/lib/hive-tls/swarm-services-root.pem" u.script
|
||||
&& lib.hasInfix ''atomic_write_secret 0644 "" "$ca_bundle_file"'' u.script
|
||||
# No ssl-cert-file on the host: the system bundle is the base. A host that sets one is merged, not replaced.
|
||||
&& lib.hasInfix "base_bundle=${allLocal.security.pki.caBundle}" u.script
|
||||
&& lib.hasInfix "base_bundle=/etc/pki/host-ca.pem" noService.systemd.services.hive-priv-cache-facts.script
|
||||
&& !(lib.hasInfix "base_bundle=${allLocal.security.pki.caBundle}" noService.systemd.services.hive-priv-cache-facts.script)
|
||||
# No client copy of the token for hive-core, and hive-c0re is not told the cache.
|
||||
&& !(lib.hasInfix "client_netrc" u.script)
|
||||
&& !(c0re ? HIVE_C0RE_CACHE_URL)
|
||||
|
|
|
|||
|
|
@ -4,30 +4,39 @@
|
|||
# ./attic-root-facts.sh.
|
||||
#
|
||||
# What runs is the rendered script, with the directory it writes moved into the
|
||||
# test's scratch space.
|
||||
# test's scratch space. `SCRIPT_HOSTCA` is the same script for a host whose nix
|
||||
# sets its own `ssl-cert-file`, which becomes the base of the CA bundle.
|
||||
{
|
||||
pkgs,
|
||||
...
|
||||
}:
|
||||
let
|
||||
script = pkgs.writeText "attic-root-facts.sh" (
|
||||
import ../host-modules/lib/attic-root-facts.nix { inherit (pkgs) lib; } {
|
||||
hive = "alpha";
|
||||
host = "cache.example";
|
||||
dir = "@DIR@/attic";
|
||||
url = "https://cache.example/swarm";
|
||||
confFile = "@DIR@/nix/cache.conf";
|
||||
netrcFile = "@DIR@/nix/netrc";
|
||||
hostNetrc = "@DIR@/host-netrc";
|
||||
}
|
||||
);
|
||||
render =
|
||||
baseBundle:
|
||||
pkgs.writeText "attic-root-facts.sh" (
|
||||
import ../host-modules/lib/attic-root-facts.nix { inherit (pkgs) lib; } {
|
||||
hive = "alpha";
|
||||
host = "cache.example";
|
||||
dir = "@DIR@/attic";
|
||||
url = "https://cache.example/swarm";
|
||||
confFile = "@DIR@/nix/cache.conf";
|
||||
netrcFile = "@DIR@/nix/netrc";
|
||||
hostNetrc = "@DIR@/host-netrc";
|
||||
caBundleFile = "@DIR@/nix/ca-bundle.crt";
|
||||
inherit baseBundle;
|
||||
swarmCa = "@DIR@/swarm-root.pem";
|
||||
}
|
||||
);
|
||||
in
|
||||
pkgs.runCommand "hyperhive-script-test-attic-root-facts"
|
||||
{
|
||||
SCRIPT = script;
|
||||
SCRIPT = render "@DIR@/system-bundle.crt";
|
||||
SCRIPT_HOSTCA = render "@DIR@/host-ca.pem";
|
||||
nativeBuildInputs = [
|
||||
pkgs.coreutils
|
||||
pkgs.curl
|
||||
pkgs.gnugrep
|
||||
pkgs.openssl
|
||||
];
|
||||
}
|
||||
''
|
||||
|
|
|
|||
|
|
@ -62,11 +62,39 @@ STUB
|
|||
chmod +x "$dir/bin/systemctl"
|
||||
export PATH="$dir/bin:$PATH"
|
||||
|
||||
# Certificates made here, with fixed names and subjects: `system-ca` stands for
|
||||
# a CA the host's own bundle holds, `swarm-ca` for the swarm's root, `host-ca`
|
||||
# for a bundle the host's nix.conf names. `swarm-leaf` and `system-leaf` are
|
||||
# servers for localhost signed by the first two.
|
||||
pki=$dir/pki
|
||||
mkdir -p "$pki"
|
||||
newca() { # <name>
|
||||
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes \
|
||||
-keyout "$pki/$1.key" -out "$pki/$1.pem" -subj "/CN=$1" -days 2 \
|
||||
-addext basicConstraints=critical,CA:TRUE -addext keyUsage=keyCertSign 2>/dev/null
|
||||
}
|
||||
newleaf() { # <ca> <name>
|
||||
openssl req -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes \
|
||||
-keyout "$pki/$2.key" -out "$pki/$2.csr" -subj "/CN=localhost" 2>/dev/null
|
||||
printf 'subjectAltName=DNS:localhost\nbasicConstraints=CA:FALSE\n' >"$pki/$2.ext"
|
||||
openssl x509 -req -in "$pki/$2.csr" -CA "$pki/$1.pem" -CAkey "$pki/$1.key" \
|
||||
-CAcreateserial -out "$pki/$2.pem" -days 2 -extfile "$pki/$2.ext" 2>/dev/null
|
||||
}
|
||||
newca system-ca
|
||||
newca swarm-ca
|
||||
newca host-ca
|
||||
newleaf swarm-ca swarm-leaf
|
||||
newleaf system-ca system-leaf
|
||||
certs() { grep -c -- '-----BEGIN CERTIFICATE-----' "$1" 2>/dev/null || echo 0; }
|
||||
|
||||
KEYLEAF=secret_swarm_hives_alpha_attic_public-key
|
||||
TOKENLEAF=secret_swarm_hives_alpha_attic_pull-token
|
||||
|
||||
reset() {
|
||||
rm -rf "$dir/store" "$dir/attic" "$dir/nix" "$dir/host-netrc" "$calls" "$dir/systemctl"
|
||||
cp "$pki/system-ca.pem" "$dir/system-bundle.crt"
|
||||
cp "$pki/swarm-ca.pem" "$dir/swarm-root.pem"
|
||||
cp "$pki/host-ca.pem" "$dir/host-ca.pem"
|
||||
mkdir -p "$dir/store"
|
||||
: >"$calls"
|
||||
: >"$dir/systemctl"
|
||||
|
|
@ -97,9 +125,10 @@ expect "netrc" "$(body "$dir/nix/netrc")" "machine cache.example password tok-1"
|
|||
expect "key mode" "$(mode "$dir/attic/public-key")" 600
|
||||
expect "netrc mode" "$(mode "$dir/nix/netrc")" 600
|
||||
expect "dir mode" "$(mode "$dir/attic")" 700
|
||||
expect "fragment names the cache, the key and the root-only netrc" "$(body "$dir/nix/cache.conf")" "extra-substituters = https://cache.example/swarm
|
||||
expect "fragment names the cache, the key, the root-only netrc and the CA bundle" "$(body "$dir/nix/cache.conf")" "extra-substituters = https://cache.example/swarm
|
||||
extra-trusted-public-keys = swarm:S0VZ
|
||||
netrc-file = $dir/nix/netrc"
|
||||
netrc-file = $dir/nix/netrc
|
||||
ssl-cert-file = $dir/nix/ca-bundle.crt"
|
||||
expect "fragment mode" "$(mode "$dir/nix/cache.conf")" 644
|
||||
expect "fragment dir mode" "$(mode "$dir/nix")" 755
|
||||
expect "no secret in the fragment" "$(grep -c tok-1 "$dir/nix/cache.conf")" 0
|
||||
|
|
@ -314,6 +343,123 @@ run
|
|||
expect "exit" $? 1
|
||||
expect "no fragment" "$(body "$dir/nix/cache.conf")" missing
|
||||
|
||||
# The CA bundle the fragment names: the host's own CAs, then the swarm's root.
|
||||
bundle=$dir/nix/ca-bundle.crt
|
||||
|
||||
case="the bundle holds the host's CAs and the swarm's root, 0644, and the fragment names it"
|
||||
reset
|
||||
seed "swarm:S0VZ" "tok-1"
|
||||
run
|
||||
expect "exit" $? 0
|
||||
expect "bundle" "$(body "$bundle")" "$(cat "$pki/system-ca.pem")
|
||||
$(cat "$pki/swarm-ca.pem")"
|
||||
expect "two certificates" "$(certs "$bundle")" 2
|
||||
expect "mode" "$(mode "$bundle")" 644
|
||||
expect "fragment names it" "$(grep -c "^ssl-cert-file = $bundle\$" "$dir/nix/cache.conf")" 1
|
||||
expect "control: the host's bundle held one certificate" "$(certs "$dir/system-bundle.crt")" 1
|
||||
expect "the host's bundle is not written" "$(body "$dir/system-bundle.crt")" "$(cat "$pki/system-ca.pem")"
|
||||
|
||||
case="the swarm and host servers verify against the bundle, and the swarm one does not against the host CAs alone"
|
||||
expect "swarm leaf, bundle" "$(openssl verify -CAfile "$bundle" "$pki/swarm-leaf.pem" >/dev/null 2>&1 && echo ok || echo bad)" ok
|
||||
expect "system leaf, bundle" "$(openssl verify -CAfile "$bundle" "$pki/system-leaf.pem" >/dev/null 2>&1 && echo ok || echo bad)" ok
|
||||
expect "control: swarm leaf, host's CAs alone" "$(openssl verify -CAfile "$dir/system-bundle.crt" "$pki/swarm-leaf.pem" >/dev/null 2>&1 && echo ok || echo bad)" bad
|
||||
|
||||
# curl takes the file as nix does. A TLS server for localhost with the swarm's
|
||||
# leaf, on a port picked at random until one binds.
|
||||
case="curl trusts the swarm's server with the bundle and refuses it with the host's CAs alone"
|
||||
port=
|
||||
for _ in 1 2 3 4 5 6 7 8; do
|
||||
p=$((20000 + RANDOM % 20000))
|
||||
openssl s_server -www -accept "$p" -cert "$pki/swarm-leaf.pem" -key "$pki/swarm-leaf.key" \
|
||||
>"$dir/s_server.log" 2>&1 &
|
||||
server=$!
|
||||
for _ in 1 2 3 4 5 6 7 8 9 10; do
|
||||
if curl -s --cacert "$pki/swarm-ca.pem" "https://localhost:$p/" >/dev/null 2>&1; then port=$p; break; fi
|
||||
kill -0 "$server" 2>/dev/null || break
|
||||
sleep 0.3
|
||||
done
|
||||
[ -n "$port" ] && break
|
||||
kill "$server" 2>/dev/null
|
||||
done
|
||||
if [ -z "$port" ]; then
|
||||
fail "no TLS server for the curl check came up: $(cat "$dir/s_server.log")"
|
||||
else
|
||||
curl -s --cacert "$bundle" "https://localhost:$port/" >/dev/null 2>&1
|
||||
expect "with the bundle" $? 0
|
||||
curl -s --cacert "$dir/system-bundle.crt" "https://localhost:$port/" >/dev/null 2>&1
|
||||
expect "with the host's CAs alone" $? 60
|
||||
kill "$server" 2>/dev/null
|
||||
fi
|
||||
|
||||
case="a second run with the same CAs rewrites nothing"
|
||||
b=$(ino "$bundle")
|
||||
run
|
||||
expect "exit" $? 0
|
||||
expect "bundle untouched" "$(ino "$bundle")" "$b"
|
||||
|
||||
case="a bundle left 0640 is rewritten 0644 though its content is current"
|
||||
chmod 640 "$bundle"
|
||||
run
|
||||
expect "exit" $? 0
|
||||
expect "mode" "$(mode "$bundle")" 644
|
||||
|
||||
case="a host that sets its own ssl-cert-file has that file merged in, not replaced"
|
||||
reset
|
||||
seed "swarm:S0VZ" "tok-1"
|
||||
sed -e "s|@DIR@|$dir|g" "$SCRIPT_HOSTCA" >"$dir/script-hostca.sh"
|
||||
cp "$dir/script.sh" "$dir/script-default.sh"
|
||||
cp "$dir/script-hostca.sh" "$dir/script.sh"
|
||||
run
|
||||
expect "exit" $? 0
|
||||
expect "the host's file and the swarm's root" "$(body "$bundle")" "$(cat "$pki/host-ca.pem")
|
||||
$(cat "$pki/swarm-ca.pem")"
|
||||
line=$(sed -n 2p "$pki/system-ca.pem")
|
||||
expect "the default bundle's CA is not in it" "$(grep -c -x -F -- "$line" "$bundle")" 0
|
||||
expect "control: the same search finds it in its own file" "$(grep -c -x -F -- "$line" "$pki/system-ca.pem")" 1
|
||||
expect "the host's file is not written" "$(body "$dir/host-ca.pem")" "$(cat "$pki/host-ca.pem")"
|
||||
cat "$pki/system-ca.pem" >>"$dir/host-ca.pem"
|
||||
run
|
||||
expect "an edit to the host's file reaches the bundle" "$(certs "$bundle")" 3
|
||||
cp "$dir/script-default.sh" "$dir/script.sh"
|
||||
|
||||
case="a swarm root that is not on the host yet leaves the host's CAs alone, and a later run adds it"
|
||||
reset
|
||||
seed "swarm:S0VZ" "tok-1"
|
||||
rm "$dir/swarm-root.pem"
|
||||
run DAEMON_STARTED="Sat 2000-01-01 00:00:00.000000 UTC"
|
||||
expect "exit" $? 0
|
||||
expect "bundle" "$(body "$bundle")" "$(cat "$pki/system-ca.pem")"
|
||||
expect "the fragment still names it" "$(grep -c "^ssl-cert-file = $bundle\$" "$dir/nix/cache.conf")" 1
|
||||
expect "said so" "$(grep -c "swarm-root.pem is not on this host yet" "$dir/out")" 1
|
||||
started=$(now)
|
||||
c=$(ino "$dir/nix/cache.conf")
|
||||
: >"$dir/systemctl"
|
||||
cp "$pki/swarm-ca.pem" "$dir/swarm-root.pem"
|
||||
run DAEMON_STARTED="$started"
|
||||
expect "exit" $? 0
|
||||
expect "root added" "$(certs "$bundle")" 2
|
||||
expect "fragment untouched" "$(ino "$dir/nix/cache.conf")" "$c"
|
||||
expect "no restart: nix reads the bundle per download" "$(restarts)" 0
|
||||
|
||||
case="a swarm root file holding no certificate is not appended"
|
||||
reset
|
||||
seed "swarm:S0VZ" "tok-1"
|
||||
echo "garbage" >"$dir/swarm-root.pem"
|
||||
run
|
||||
expect "exit" $? 0
|
||||
expect "bundle" "$(body "$bundle")" "$(cat "$pki/system-ca.pem")"
|
||||
|
||||
case="a host bundle that cannot be read fails the run and leaves the fragment and bundle"
|
||||
good
|
||||
cp "$bundle" "$dir/bundle-before"
|
||||
rm "$dir/system-bundle.crt"
|
||||
seed "swarm:NEW" "tok-new"
|
||||
run
|
||||
expect "exit" $? 1
|
||||
expect "says why" "$(grep -c "cannot read the host.s CA bundle" "$dir/err")" 1
|
||||
expect "bundle kept" "$(body "$bundle")" "$(body "$dir/bundle-before")"
|
||||
expect "fragment kept" "$(grep -c 'extra-trusted-public-keys = swarm:S0VZ' "$dir/nix/cache.conf")" 1
|
||||
|
||||
if [ "$failures" -ne 0 ]; then
|
||||
echo "$failures of $count checks failed" >&2
|
||||
exit 1
|
||||
|
|
|
|||
Loading…
Reference in a new issue