diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 7166f536..a2a12ee9 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -113,7 +113,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | secret | generated by | lives at | hive elsewhere | | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | +| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store is down, refuses the login, or fails the read of a leaf (a refusal, a timeout, an error), the unit runs on the files, logs the store's error, and writes nothing to the store; the unit seeds or mints only a leaf the store reports as absent. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | | authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives//oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services//oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) | diff --git a/nix/host-modules/lib/store-seeded-secret.nix b/nix/host-modules/lib/store-seeded-secret.nix index 04a93a7e..e03a99fc 100644 --- a/nix/host-modules/lib/store-seeded-secret.nix +++ b/nix/host-modules/lib/store-seeded-secret.nix @@ -5,7 +5,8 @@ # # `store_seeded` prints the value and never rotates one: the store is written # with `-cas=0`, so only a leaf that holds nothing takes a value, and a lost -# race reads the winner's back. +# race reads the winner's back. A read of the leaf that fails for any reason +# but "No value found" prints the store's error and neither mints nor writes. # # Pure function — NOT a NixOS module. Call it from a module's `let`: # @@ -17,7 +18,7 @@ # key="$(store_seeded secret/swarm/services/foo/key /var/lib/foo/key mint_key)" || exit 1 # ''; # -# Needs `bao` and `coreutils` on the caller's `path` and the `BAO_*` client +# Needs `bao`, `coreutils` and `gnugrep` on the caller's `path` and the `BAO_*` client # environment set. Takes the EXIT trap, to remove its scratch file. { }: '' @@ -48,15 +49,31 @@ } # . The value on stdout; status 1 when - # neither the store nor the file can give one. + # neither the store nor the file can give one. `kv get` exits 2 both for a + # leaf that holds nothing ("No value found at ") and for a refused + # read, so only that message means absent. Any other failure of the first + # read, the `timeout` (124) included, neither mints nor writes: the file's + # value is used when it has one, and the call fails when it has none. store_seeded() { - local leaf="$1" earlier="$2" value candidate + local leaf="$1" earlier="$2" value candidate rc=0 shift 2 if [ "$STORE_UP" = 1 ]; then - if value="$(store_bao kv get -field=value "$leaf" 2>"$STORE_ERR")" && [ -n "$value" ]; then + value="$(store_bao kv get -field=value "$leaf" 2>"$STORE_ERR")" || rc=$? + if [ "$rc" = 0 ] && [ -n "$value" ]; then printf '%s' "$value" return 0 fi + if [ "$rc" != 0 ] && ! grep -q '^No value found at ' "$STORE_ERR"; then + echo "could not read $leaf from the swarm secret store (exit status $rc):" >&2 + cat "$STORE_ERR" >&2 + if [ -s "$earlier" ]; then + echo "using the value of $earlier for $leaf; nothing is written to the store." >&2 + printf '%s' "$(< "$earlier")" + return 0 + fi + echo "no value for $leaf in the store or at $earlier." >&2 + return 1 + fi if [ -s "$earlier" ]; then candidate="$(< "$earlier")" echo "storing the value of $earlier in $leaf." >&2 diff --git a/nix/script-tests/store-seeded-secret.nix b/nix/script-tests/store-seeded-secret.nix index 494383f9..1f5c72dc 100644 --- a/nix/script-tests/store-seeded-secret.nix +++ b/nix/script-tests/store-seeded-secret.nix @@ -11,7 +11,10 @@ let # `login` answers from `FAKE_BAO_LOGIN_RC`. `kv get` and `kv put -cas=0` act # on a file per leaf under `FAKE_BAO_KV`; `FAKE_BAO_DENY_READ` and - # `FAKE_BAO_DENY_WRITE` answer 403 as a policy that grants neither does. + # `FAKE_BAO_DENY_WRITE` answer 403 as a policy that grants neither does, and + # `FAKE_BAO_READ_TIMEOUT` is `timeout` ending the read: status 124, silent. + # `FAKE_BAO_READ_5XX` is a server error on the read. A leaf nobody wrote is + # "No value found at ", status 2, as `bao kv get` answers it. # `FAKE_BAO_RACE` has another run create the leaf between this one's read and # its write. Every call is logged without the value `put` reads from stdin. fakeBao = pkgs.writeShellScriptBin "bao" '' @@ -24,14 +27,16 @@ let "kv get") [ "''${BAO_TOKEN-}" = s.fake ] || { echo "fake bao: kv without the login's token" >&2; exit 97; } leaf="$FAKE_BAO_KV/$(printf '%s' "$4" | tr / _)" + [ -z "''${FAKE_BAO_READ_TIMEOUT-}" ] || exit 124 + [ -z "''${FAKE_BAO_READ_5XX-}" ] || { echo "Error reading secret/data/x: Error making API request. Code: 503. Errors: * Vault is sealed" >&2; exit 2; } [ -z "''${FAKE_BAO_DENY_READ-}" ] || { echo "fake bao: permission denied" >&2; exit 2; } if [ -n "''${FAKE_BAO_RACE-}" ] && [ ! -e "$FAKE_BAO_KV/.raced" ]; then touch "$FAKE_BAO_KV/.raced" printf '%s' "$FAKE_BAO_RACE" > "$leaf" - echo "fake bao: no value found" >&2 + echo "No value found at secret/data/x" >&2 exit 2 fi - [ -e "$leaf" ] || { echo "fake bao: no value found" >&2; exit 2; } + [ -e "$leaf" ] || { echo "No value found at secret/data/x" >&2; exit 2; } cat "$leaf" ;; "kv put") @@ -55,7 +60,10 @@ pkgs.runCommand "hyperhive-script-test-store-seeded-secret" import ../host-modules/lib/store-seeded-secret.nix { } ); FAKE_BAO_BIN = "${fakeBao}/bin"; - nativeBuildInputs = [ pkgs.coreutils ]; + nativeBuildInputs = [ + pkgs.coreutils + pkgs.gnugrep + ]; } '' ${pkgs.bash}/bin/bash ${./store-seeded-secret.sh} diff --git a/nix/script-tests/store-seeded-secret.sh b/nix/script-tests/store-seeded-secret.sh index 35082e60..35825fe0 100644 --- a/nix/script-tests/store-seeded-secret.sh +++ b/nix/script-tests/store-seeded-secret.sh @@ -22,7 +22,7 @@ setup() { dir=$(mktemp -d) mkdir "$dir/kv" export FAKE_BAO_KV=$dir/kv FAKE_BAO_LOG=$dir/bao.log FAKE_BAO_LOGIN_RC=0 - unset FAKE_BAO_DENY_READ FAKE_BAO_DENY_WRITE FAKE_BAO_RACE + unset FAKE_BAO_DENY_READ FAKE_BAO_DENY_WRITE FAKE_BAO_RACE FAKE_BAO_READ_TIMEOUT FAKE_BAO_READ_5XX : >"$FAKE_BAO_LOG" earlier=$dir/earlier minted=$dir/minted @@ -69,6 +69,19 @@ ran_mint() { [ -s "$minted" ] } +no_write() { + if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to the store"; fi +} + +no_mint() { + if ran_mint; then fail "ran the mint command"; fi +} + +# err_has : the run's stderr contains it. +err_has() { + if ! grep -qF -- "$1" "$dir/err"; then fail "stderr lacks '$1': $(cat "$dir/err")"; fi +} + setup "an empty store is seeded from the file, unchanged, and nothing is minted" printf 'old-value\n' >"$earlier" run @@ -111,19 +124,80 @@ run expect 0 winner-value held winner-value -setup "a store that grants neither read nor write leaves the file's value in use" +setup "a refused read uses the file's value, writes nothing, mints nothing and prints the refusal" printf 'old-value\n' >"$earlier" export FAKE_BAO_DENY_READ=1 FAKE_BAO_DENY_WRITE=1 run expect 0 old-value held "" -if ran_mint; then fail "minted although the file held a value"; fi +no_write +no_mint +err_has "could not read $leaf from the swarm secret store" +err_has "permission denied" +err_has "using the value of $earlier" -setup "a store that grants neither, and no file, gives no value and mints nothing into it" +setup "a refused read, and no file, fails without writing or minting and prints the refusal" export FAKE_BAO_DENY_READ=1 FAKE_BAO_DENY_WRITE=1 run expect 1 "" held "" +no_write +no_mint +err_has "could not read $leaf from the swarm secret store" +err_has "permission denied" + +setup "a read that times out uses the file's value without writing, and says why" +printf 'old-value\n' >"$earlier" +export FAKE_BAO_READ_TIMEOUT=1 +run +expect 0 old-value +held "" +no_write +no_mint +err_has "could not read $leaf from the swarm secret store (exit status 124)" + +setup "a read that times out, and no file, fails without writing or minting" +export FAKE_BAO_READ_TIMEOUT=1 +run +expect 1 "" +held "" +no_write +no_mint +err_has "(exit status 124)" + +setup "a read that times out on a leaf the store holds leaves the leaf as it is" +printf 'stored-value' >"$stored" +printf 'old-value\n' >"$earlier" +export FAKE_BAO_READ_TIMEOUT=1 +run +expect 0 old-value +held stored-value +no_write +no_mint + +setup "a read that gets a server error, and no file, fails without writing and prints the error" +export FAKE_BAO_READ_5XX=1 +run +expect 1 "" +held "" +no_write +no_mint +err_has "Code: 503" + +setup "a leaf the store does not hold is the only read failure that mints, and no error is printed" +run +expect 0 minted-value +held minted-value +if ! ran_mint; then fail "did not mint"; fi +if grep -q 'could not read' "$dir/err"; then fail "reported a missing leaf as a failed read"; fi + +setup "a store that takes no write leaves the file's value in use" +printf 'old-value\n' >"$earlier" +export FAKE_BAO_DENY_WRITE=1 +run +expect 0 old-value +held "" +if ran_mint; then fail "minted although the file held a value"; fi setup "a store that refuses the login leaves the file's value in use and is not called again" printf 'old-value\n' >"$earlier"