report the real error when the granter role's bootstrap step fails, and retry on a standby store

nix/host-modules/lib/bootstrap-auth-list.nix

@@ -0,0 +1,57 @@
+# The bootstrap token's first use in `swarm-bao-granter-role`, as shell text
+# that leaves `mounted` set to the store's auth listing, or ends the script.
+# Same split as ./granter-login.nix, applied to `bao auth list`:
+#
+# - A refusal ("permission denied", HTTP 403: openbao answers an expired,
+#   revoked or underprivileged token alike) is the token's. If the granter's
+#   own login works the role is already in place and the dead token is
+#   deleted; otherwise the unit prints the one-time steps and exits 4, which
+#   `RestartPreventExitStatus` keeps from retrying.
+# - Any other failure from a store that answers `status` is not the token's:
+#   a node that is not yet active answers with a 500 for the first moments
+#   after an unseal. The message says so and exits 1, which `Restart=` retries.
+# - A store that does not answer `status` is sealed or unreachable; exit 1.
+#
+# In every case bao's own error follows.
+#
+# Pure function — NOT a NixOS module. `baoStoreUp` is as for
+# ./granter-login.nix. Needs `bao`, `coreutils` and `gnugrep` on the caller's
+# `path`, and `$err` to be a scratch file.
+{
+  lib,
+  bootstrapTokenFile,
+  granterClientCertFile,
+  granterClientKeyFile,
+  baoStoreUp,
+  setupSteps,
+}:
+''
+  if ! mounted="$(bao auth list -format=json 2>"$err")"; then
+    if grep -qF 'permission denied' "$err"; then
+      # A dead token beside a working granter is the steady state of
+      # a host that finished bootstrap and kept the file. The
+      # granter's own login is the proof. A fresh token still takes
+      # the full path below, since the granter cannot read back its
+      # own policy to compare it. The refused token is dead,
+      # so it is deleted here and the unit does not start again.
+      if BAO_CLIENT_CERT=${lib.escapeShellArg granterClientCertFile} \
+        BAO_CLIENT_KEY=${lib.escapeShellArg granterClientKeyFile} \
+        bao login -method=cert -token-only >/dev/null 2>&1; then
+        echo "granter role already in place; bootstrap token not needed"
+        rm -f ${lib.escapeShellArg bootstrapTokenFile}
+        exit 0
+      fi
+      echo ${lib.escapeShellArg "the store refused the bootstrap token in ${bootstrapTokenFile}: it has expired, been revoked, or does not carry the bao-bootstrap policy."} >&2
+      echo "one-time step, as root on this host (docs/getting-started/setup.md):" >&2
+      ${lib.concatMapStringsSep "\n" (l: "echo ${lib.escapeShellArg "  ${l}"} >&2") setupSteps}
+      cat "$err" >&2
+      exit 4
+    elif ${baoStoreUp}; then
+      echo "the store is unsealed but listing its auth mounts failed without a refused token, as a node that is not active yet answers; retrying." >&2
+    else
+      echo "the store is sealed or unreachable; retrying." >&2
+    fi
+    cat "$err" >&2
+    exit 1
+  fi
+''