diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 3c22e53e..bdb4b61b 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -41,27 +41,28 @@ neither is a renaming of the other. ## Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | -| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | -| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | -| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | +| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | +| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | +| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value | `/var/lib/grafana-admin/password` inside the `swarm-grafana` container: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. Grafana doesn't start until the file exists, so it stays down rather than falling back to the upstream default `admin` | +| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | +| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. **That's the test worth @@ -392,17 +393,17 @@ next controller restart. Bao matches a cert-auth role on the certificate's subject, so a certificate is an identity and sharing one merges the identities. Each of these seven leaves has a role and a policy of its own, and reaches one path, except the review -classifier's and the forge host's, which reach two: +classifier's, the forge host's and Grafana's, which reach two each: -| unit | option pair under `deploy.bao.` | reads, or writes | -| --------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | -| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | -| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | -| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | -| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services//oidc/client` | -| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` | -| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key` and `swarm/services//oidc/client` | -| `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` | +| unit | option pair under `deploy.bao.` | reads, or writes | +| --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | +| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | +| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services//oidc/client`; creates and reads `swarm/services//admin/password` | +| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | +| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services//oidc/client` | +| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` | +| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key` and `swarm/services//oidc/client` | +| `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` | The first exists **per hive**, because the path it reads carries a hive name and every hive runs its own reader. Its subject is diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 38ce804b..4847771c 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -153,10 +153,11 @@ See `retentionPeriod` below before leaving it at its default. **Logging in.** Grafana is behind swarm SSO, so the accounts are the -authelia ones — there is no separate Grafana password, and this module -switches off the local login form whenever SSO is configured. If you enable -Grafana on a host with no authelia, the form stays on and Grafana's -default `admin`/`admin` applies; change it before exposing that host. +authelia ones, and the local login form is always off. Grafana's API still +accepts HTTP basic auth for its built-in `admin` account, so that account's +password is a random one minted into the swarm secret store +(`swarm/services//admin/password`) and applied on every +Grafana start — see [`secrets.md`](secrets.md). diff --git a/nix/host-modules/glue-bao-readers-policy-order.nix b/nix/host-modules/glue-bao-readers-policy-order.nix index 1f82dc1a..2657e961 100644 --- a/nix/host-modules/glue-bao-readers-policy-order.nix +++ b/nix/host-modules/glue-bao-readers-policy-order.nix @@ -66,6 +66,12 @@ let "swarm-bao-secret-publisher-policy" ]; }; + # ./swarm-grafana.nix, under Grafana's leaf, which may create the admin + # password only once that policy is written. + swarm-bao-grafana-admin = { + here = deployCfg.grafana.enable; + policies = [ "swarm-bao-grafana-oidc-policy" ]; + }; # ./hive-matrix.nix, under matrix-ctl's leaf. swarm-bao-matrix-oidc = { here = deployCfg.matrix.enable; diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index d6e27ec1..f51c9077 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -850,11 +850,17 @@ let # Singletons, so a list of one rather than a second shape: the granting unit # below takes a list either way and the units stay identical apart from # the objects they write. + # + # Grafana's leaf also creates its admin password (`swarm-bao-grafana-admin` + # in ./swarm-grafana.nix). `create` without `update`, so a stored password is + # never replaced. grafanaOidcReaders = [ { name = "swarm-grafana-oidc"; cn = baoDeploy.grafanaOidcCommonName; - policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"; + policyText = + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client" + + createStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/admin/password"; } ]; diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 2e1a074a..ad0bd74f 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -253,6 +253,15 @@ let # the OIDC secret above, whose other reader is authelia's container. secretKeyPath = "/var/lib/grafana-secret/secret_key"; + # The built-in `admin` account's password. Grafana's API takes basic auth + # whatever the login form says, so this account is reachable from the public + # vhost and its password cannot be the upstream default. Minted once into the + # store by `swarm-bao-grafana-admin` below, which `swarm-bao.nix`'s + # `grafanaOidcReaders` grants create and read on this one leaf. + adminPasswordStorePath = "secret/swarm/services/${cfg.oidc.clientId}/admin/password"; + adminPasswordPath = "/var/lib/grafana-admin/password"; + hostAdminPasswordPath = "/var/lib/nixos-containers/${cfg.machine}${adminPasswordPath}"; + socketPath = "${deployCfg.grafana.socketDir}/grafana.sock"; # Both static NixOS ids, and both checked rather than assumed: nginx has @@ -656,6 +665,97 @@ in ''; }; + # The admin password: created in the store if it holds none, then copied + # into the container, under Grafana's own leaf. Same shape as + # `swarm-bao-nats-callout-xkey` in ./swarm-nats.nix. The grant is + # `create` without `update`, so the first password stored is the one every + # later run reads, and a race between two runs leaves one value. + # + # Grafana does not start without the file (`$__file{}` on a missing path is + # a startup error), so a store that is unreachable on first boot keeps + # Grafana down rather than up with the upstream default. + systemd.services.swarm-bao-grafana-admin = { + description = "create Grafana's admin password in the swarm secret store if absent and fetch it"; + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + before = [ "container@${cfg.machine}.service" ]; + wantedBy = [ + "multi-user.target" + "container@${cfg.machine}.service" + ]; + path = [ + baoDeploy.package + pkgs.coreutils + pkgs.openssl + pkgs.systemd + ]; + inherit (storeRetry) startLimitBurst startLimitIntervalSec; + serviceConfig = storeRetry.serviceConfig // { + Type = "oneshot"; + RemainAfterExit = true; + SyslogIdentifier = "swarm-bao-grafana-admin"; + TimeoutStartSec = 30; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile; + } + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + ${atomicWriteSecret} + ${refreshConsumer} + + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with Grafana's certificate; leaving Grafana's admin password file as it is." >&2 + cat "$err" >&2 + exit 1 + fi + export BAO_TOKEN + + # `bao kv get` exits 2 both for "nothing stored" and for a refusal, so + # a failed read always tries the create, and the second read decides. + # The value reaches `bao` on stdin, never in an argument. + if ! password="$(bao kv get -field=value ${lib.escapeShellArg adminPasswordStorePath} 2>"$err")"; then + minted="$(openssl rand -hex 32)" + if ! printf '%s' "$minted" | bao kv put -cas=0 ${lib.escapeShellArg adminPasswordStorePath} value=- >/dev/null 2>"$err"; then + echo "could not create ${adminPasswordStorePath}; reading it back in case another run created it." >&2 + cat "$err" >&2 + fi + if ! password="$(bao kv get -field=value ${lib.escapeShellArg adminPasswordStorePath} 2>"$err")"; then + echo "swarm-bao did not return ${adminPasswordStorePath}; Grafana does not start without it." >&2 + cat "$err" >&2 + exit 1 + fi + fi + if [ -z "$password" ]; then + echo "swarm-bao returned an empty ${adminPasswordStorePath}; leaving the file as it is." >&2 + exit 1 + fi + + # Owner and mode as for the OIDC secret above, for its reasons. + install -d -m 0755 ${lib.escapeShellArg (builtins.dirOf hostAdminPasswordPath)} + if secret_differs ${lib.escapeShellArg hostAdminPasswordPath} "$password"; then + atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostAdminPasswordPath} "$password" + fi + + # Grafana's `preStart` applies the file to the database, so a file that + # changed after it started needs a restart. + refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostAdminPasswordPath} + ''; + }; + containers.${cfg.machine} = { autoStart = true; ephemeral = false; @@ -681,7 +781,7 @@ in // caTrust.bindMount; config = - { ... }: + { config, ... }: { imports = [ ../container-modules/swarm-container.nix @@ -752,6 +852,32 @@ in ''; }; + # Upstream applies `security.admin_password` only when it creates the + # database, so a database created before that setting keeps the + # password it was created with. This resets it from the delivered + # file on every start, before the server listens. + # + # `grafana cli` must open the database the server opens, which the + # server's config.ini names. nixpkgs keeps that file private, so it + # is rendered again from the same settings with nixpkgs' formatter. + systemd.services.grafana.preStart = + let + grafana = config.services.grafana; + configFile = + (pkgs.formats.ini { + listToValue = lib.concatMapStringsSep " " (lib.generators.mkValueStringDefault { }); + mkKeyValue = lib.generators.mkKeyValueDefault { + mkValueString = v: if v == null then "" else lib.generators.mkValueStringDefault { } v; + } "="; + }).generate + "config.ini" + grafana.settings; + in + lib.mkAfter '' + ${lib.getExe' grafana.package "grafana"} cli --homepath ${grafana.dataDir} --config ${configFile} \ + admin reset-admin-password --password-from-stdin < ${adminPasswordPath} + ''; + # The group Grafana chgrps its socket to. Declared here because a # container has its own user database — the host knowing gid 60 as # `nginx` means nothing in here, only the number crosses. @@ -837,10 +963,14 @@ in # whole host fails to build. The unit below generates it. security.secret_key = "$__file{${secretKeyPath}}"; - # No local password path, ever. This is not tidiness: Grafana - # ships an `admin`/`admin` account and this vhost is on the - # public gateway, so SSO is the only way in — not merely the - # preferred one. + # The password a newly created database gets; the `preStart` + # above gives an existing one the same. Written by + # `swarm-bao-grafana-admin` on the host. + security.admin_password = "$__file{${adminPasswordPath}}"; + + # No password box on the public login page: people log in through + # SSO. This hides the form only; the API still takes basic auth, + # which is why the admin password above is minted. # # Not conditional on anything, and the block below is not # either, which is what keeps the two from ever disagreeing. diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index ca166b00..019a9318 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -557,13 +557,28 @@ let # for this one is the OTHER service's: `services/*` would have granted # both, and the two are separate principals precisely because a # dashboard is not entitled to a collector's credential. - name = "the Grafana OIDC reader's grant is Grafana's own client secret and nothing else"; + # + # The admin password is the one leaf it may create, and without `update`: + # a grant that could replace it would let a second run rotate the + # password Grafana's database already holds. + name = "the Grafana reader reads its own client secret and creates its admin password, and nothing else"; ok = let s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; in - lib.hasInfix "path \"secret/data/swarm/services/swarm-grafana/oidc/client\" {" s - && lib.hasInfix "capabilities = [\"read\"]" s + grantsIn s == [ + { + path = "secret/data/swarm/services/swarm-grafana/oidc/client"; + caps = [ "read" ]; + } + { + path = "secret/data/swarm/services/swarm-grafana/admin/password"; + caps = [ + "create" + "read" + ]; + } + ] && !(lib.hasInfix "secret/data/swarm/agents" s) && !(lib.hasInfix "secret/data/swarm/hives" s) && !(lib.hasInfix "secret/data/swarm/services/*" s) @@ -827,6 +842,20 @@ let && !(noReaders ? swarm-bao-matrix-oidc) && !(noReaders ? swarm-bao-nats-callout-xkey); } + { + # The admin password is created under Grafana's leaf, so its unit waits + # for the unit writing that leaf's role and policy; before it, the create + # is refused. The second arm is the absence control. + name = "grafana's admin password unit is ordered after grafana's policy unit"; + ok = + let + u = baoGrantWithConsumers.systemd.services.swarm-bao-grafana-admin; + in + lib.elem "swarm-bao-grafana-oidc-policy.service" u.after + && lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants + && !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires) + && !(baoGrantNoReaders.systemd.services ? swarm-bao-grafana-admin); + } { # The ordering is set apart from each reader's own definition, so it can # define a reader by itself: `after` on a unit nothing else declares is a @@ -959,6 +988,7 @@ let [ "swarm-services-cert" "swarm-bao-forwarder-oidc" + "swarm-bao-grafana-admin" ] ++ policyReaders ); diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index ec538e98..33f20ca3 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -123,9 +123,8 @@ let { # This fixture enables grafana and NOT authelia, so the login-form toggle # has to follow "grafana requires SSO" rather than "both services are on - # this host". Grafana ships an - # `admin`/`admin` account and its vhost is on the public gateway, so a - # password box there is a way in whatever the topology. + # this host". Its vhost is on the public gateway, so a password box there + # is a way in whatever the topology. name = "grafana disables its local login form even where authelia is not on this host"; ok = grafanaOldPath.containers.swarm-grafana.config.services.grafana.settings.auth.disable_login_form; @@ -196,6 +195,36 @@ let && !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url") && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.grafanaOidcClientCertFile"); } + { + # Grafana's API takes basic auth whatever the login form says, so the + # built-in admin's password is the only thing between the public vhost + # and server-admin. Upstream sets it only when it creates the database, + # so both halves are pinned: the setting a new database reads, and the + # reset every start applies to whichever database exists. Without the + # reset, a database created before the setting keeps `admin`. + # + # The reset's `--config` must be the file the server reads, or it resets + # a different database: compared by store path against the server's own + # command line. + name = "grafana's admin password is minted into the store and applied to new and existing databases"; + ok = + let + unit = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-admin; + grafana = grafanaRemoteAuthelia.containers.swarm-grafana.config.services.grafana; + service = grafanaRemoteAuthelia.containers.swarm-grafana.config.systemd.services.grafana; + resetConfig = lib.head ( + lib.head (lib.filter lib.isList (builtins.split "--config ([^ ]+)" service.preStart)) + ); + in + lib.hasInfix "bao kv put -cas=0 secret/swarm/services/swarm-grafana/admin/password value=-" unit.script + && lib.hasInfix "openssl rand -hex 32" unit.script + && lib.hasInfix "atomic_write_secret 0400 196:0 /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-admin/password" unit.script + && lib.elem "container@swarm-grafana.service" unit.before + && unit.environment.BAO_CLIENT_CERT == "/etc/pki/bao-grafana-oidc.pem" + && grafana.settings.security.admin_password == "$__file{/var/lib/grafana-admin/password}" + && lib.hasInfix "admin reset-admin-password --password-from-stdin < /var/lib/grafana-admin/password" service.preStart + && lib.hasInfix "-config ${resetConfig}" service.script; + } { # Same 403-not-a-miss reason as the matrix and queue arms below: the # reader's grant covers the `services` prefix, so a path outside it is