From cbab23273a8fa2d8e7a4463a11b816ea4ee5d217 Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 15 Oct 2026 22:09:03 +0200 Subject: [PATCH] swarm-grafana: keep Grafana's OIDC secret, admin password and metrics bearer on the host's /run, not in the container's root The three delivery units still read from bao under Grafana's own leaf, but now write to /run/swarm-grafana-secrets on the host (tmpfs), which the container bind-mounts read-only at the same path. Before, they wrote to /var/lib/nixos-containers/swarm-grafana/var/lib/{grafana-oidc,grafana-admin, grafana-metrics}/ on persistent disk, where a superseded value stayed after rotation and went into backups of the container root. Host tmpfiles now removes the three old files. Rotation is unchanged: refresh_consumer restarts Grafana (or starts it if it failed) when a file changes, since Grafana reads each $__file{} once as it starts. After a reboot the files are fetched again; until the store answers Grafana stays down, where before it came up on the last copy from disk. Part of #4998 (slice 2b). --- docs/swarm/secrets.md | 6 +-- docs/swarm/sso.md | 2 +- nix/host-modules/swarm-grafana.nix | 81 +++++++++++++++++++----------- nix/module-eval/grafana.nix | 43 ++++++++++++++-- 4 files changed, 95 insertions(+), 37 deletions(-) diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 32ac8276..072605f9 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -52,9 +52,9 @@ neither is a renaming of the other. | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | -| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/var/lib/grafana-admin/password` inside the `swarm-grafana` container: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. Grafana doesn't start until the file exists, so it stays down rather than falling back to the upstream default `admin` | -| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/var/lib/grafana-metrics/bearer` inside the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | +| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | diff --git a/docs/swarm/sso.md b/docs/swarm/sso.md index 728bcf7c..133a3067 100644 --- a/docs/swarm/sso.md +++ b/docs/swarm/sso.md @@ -312,7 +312,7 @@ gateway refuses its token too. See [`swarmctl/README.md`](../../swarmctl/README. dead provider locks out the SSO accounts but not that one: an operator reads the password from `swarm/services//admin/password` in the store (or, as root on the Grafana host, from - `/var/lib/grafana-admin/password` inside the `swarm-grafana` container) + `/run/swarm-grafana-secrets/admin-password`) and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana) and [`secrets.md`](secrets.md). SSO logins are the everyday way in, which is why `swarm.grafana.oidc.role` defaults to `Admin`. diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index d3a16c05..8e878a19 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -226,18 +226,26 @@ let autheliaUrl = toString hyperhiveCfg.swarm.authelia.url; - # Where the plaintext lands inside the container. Under /var/lib rather - # than /run: Grafana may start before the delivery unit on a later boot, - # and a secret that evaporates on reboot turns a working login into an - # intermittent one. - secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret"; - - # The same file seen from the host, which is where the delivery unit writes - # it. Spelled once: the unit that writes it and the config that names it are + # The three secrets below land on the host's /run (tmpfs), which the + # container bind-mounts read-only at the same path: none reaches persistent + # disk, and a reboot fetches them again. Each file is absent until its + # delivery unit has run, and Grafana does not start without them — `$__file{}` + # on a missing path is a startup error, and an empty admin password would + # fall back to the upstream default. A delivery unit that writes a file + # restarts or starts Grafana (`refresh_consumer`). + # + # Spelled once: the unit that writes a file and the config that names it are # a few hundred lines apart, and a Grafana reading a path nothing writes is # a login that fails with nothing in any log about the file. - hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}"; - hostSecretDir = builtins.dirOf hostSecretPath; + secretDir = "/run/swarm-grafana-secrets"; + secretPath = "${secretDir}/${cfg.oidc.clientId}.secret"; + + # Where the three files sat on persistent disk before they moved to /run. + legacyHostSecretPaths = map (p: "/var/lib/nixos-containers/${cfg.machine}${p}") [ + "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret" + "/var/lib/grafana-admin/password" + "/var/lib/grafana-metrics/bearer" + ]; # Where the publisher on authelia's host leaves this client's secret. A # swarm service's client is registered once for the whole swarm, so its @@ -260,8 +268,7 @@ let # store by `swarm-bao-grafana-admin` below, which `swarm-bao.nix`'s # `grafanaOidcReaders` grants create and read on this one leaf. adminPasswordStorePath = "secret/swarm/services/${cfg.oidc.clientId}/admin/password"; - adminPasswordPath = "/var/lib/grafana-admin/password"; - hostAdminPasswordPath = "/var/lib/nixos-containers/${cfg.machine}${adminPasswordPath}"; + adminPasswordPath = "${secretDir}/admin-password"; # The bearer the metrics datasource presents at the metrics store's query # API, which `swarm-metrics-auth` on the store's host checks against the @@ -270,8 +277,7 @@ let # header value, `Bearer `: nixpkgs accepts a `secureJsonData` entry # only when it is exactly one `$__file{}`. metricsBearerStorePath = "secret/swarm/services/${cfg.oidc.clientId}/metrics/bearer"; - metricsBearerPath = "/var/lib/grafana-metrics/bearer"; - hostMetricsBearerPath = "/var/lib/nixos-containers/${cfg.machine}${metricsBearerPath}"; + metricsBearerPath = "${secretDir}/metrics-bearer"; # swarm-controller's token for Grafana's API, minted by # `swarm-bao-grafana-controller-token` below for a Viewer service account of @@ -545,9 +551,18 @@ in # gateway can traverse; `0750` keeps everything else out. Numeric ids # because the host has no `grafana` account — Grafana lives in the # container, and only the number crosses that boundary. + # + # The secrets directory is the container's read-only bind source, so it + # has to exist before the container starts even when the store does not + # answer; the delivery units create it too, but unordered with the + # container on a switch. 0755 because Grafana reads the files as its own + # uid; each file is 0400 and owned by it. The legacy files are removed so + # no superseded secret stays behind on the host's disk. systemd.tmpfiles.rules = [ "d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -" - ]; + "d ${secretDir} 0755 root root -" + ] + ++ map (p: "r ${p}") legacyHostSecretPaths; # THE delivery unit — one route, in every deployment. The secret authelia # minted arrives out of the swarm secret store, which the publisher on @@ -678,13 +693,13 @@ in # the host cannot know it at eval time. Harmless here because 0400 # grants the group nothing; if this mode ever widens, the gid has to be # discovered at runtime rather than assumed. - install -d -m 0755 ${lib.escapeShellArg hostSecretDir} - if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then - atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostSecretPath} "$secret" + install -d -m 0755 ${lib.escapeShellArg secretDir} + if secret_differs ${lib.escapeShellArg secretPath} "$secret"; then + atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg secretPath} "$secret" fi # `$__file{}` is expanded once, while Grafana parses its config. - refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostSecretPath} + refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg secretPath} ''; }; @@ -766,14 +781,14 @@ in fi # Owner and mode as for the OIDC secret above, for its reasons. - install -d -m 0755 ${lib.escapeShellArg (builtins.dirOf hostAdminPasswordPath)} - if secret_differs ${lib.escapeShellArg hostAdminPasswordPath} "$password"; then - atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostAdminPasswordPath} "$password" + install -d -m 0755 ${lib.escapeShellArg secretDir} + if secret_differs ${lib.escapeShellArg adminPasswordPath} "$password"; then + atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg adminPasswordPath} "$password" fi # Grafana's `preStart` applies the file to the database, so a file that # changed after it started needs a restart. - refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostAdminPasswordPath} + refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg adminPasswordPath} ''; }; @@ -847,13 +862,13 @@ in fi # Owner and mode as for the OIDC secret above, for its reasons. - install -d -m 0755 ${lib.escapeShellArg (builtins.dirOf hostMetricsBearerPath)} - if secret_differs ${lib.escapeShellArg hostMetricsBearerPath} "Bearer $bearer"; then - atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostMetricsBearerPath} "Bearer $bearer" + install -d -m 0755 ${lib.escapeShellArg secretDir} + if secret_differs ${lib.escapeShellArg metricsBearerPath} "Bearer $bearer"; then + atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg metricsBearerPath} "Bearer $bearer" fi # `$__file{}` in a provisioned datasource is read as Grafana starts. - refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostMetricsBearerPath} + refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg metricsBearerPath} ''; }; @@ -935,11 +950,11 @@ in exit 1 } - if [ ! -s ${lib.escapeShellArg hostAdminPasswordPath} ]; then - echo "no Grafana admin password at ${hostAdminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2 + if [ ! -s ${lib.escapeShellArg adminPasswordPath} ]; then + echo "no Grafana admin password at ${adminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2 exit 1 fi - admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg hostAdminPasswordPath})" | base64 -w0)" + admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg adminPasswordPath})" | base64 -w0)" status="$(grafana "$admin" GET '/api/serviceaccounts/search?perpage=1000&query=${controllerAccountName}')" expect "the service account search" "$status" 200 @@ -1020,6 +1035,12 @@ in hostPath = deployCfg.grafana.socketDir; isReadOnly = false; }; + # The three delivered secrets, read-only: the container reads them and + # never writes one. + ${secretDir} = { + hostPath = secretDir; + isReadOnly = true; + }; } // caTrust.bindMount; diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index 9f6e712e..325db82e 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -242,11 +242,11 @@ let in lib.hasInfix "bao kv put -cas=0 secret/swarm/services/swarm-grafana/admin/password value=-" unit.script && lib.hasInfix "openssl rand -hex 32" unit.script - && lib.hasInfix "atomic_write_secret 0400 196:0 /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-admin/password" unit.script + && lib.hasInfix "atomic_write_secret 0400 196:0 /run/swarm-grafana-secrets/admin-password" unit.script && lib.elem "container@swarm-grafana.service" unit.before && unit.environment.BAO_CLIENT_CERT == "/etc/pki/bao-grafana-oidc.pem" - && grafana.settings.security.admin_password == "$__file{/var/lib/grafana-admin/password}" - && lib.hasInfix "admin reset-admin-password --password-from-stdin < /var/lib/grafana-admin/password" service.preStart + && grafana.settings.security.admin_password == "$__file{/run/swarm-grafana-secrets/admin-password}" + && lib.hasInfix "admin reset-admin-password --password-from-stdin < /run/swarm-grafana-secrets/admin-password" service.preStart && lib.hasInfix "-config ${resetConfig}" service.script; } { @@ -290,6 +290,43 @@ let lib.hasInfix "secret/swarm/services/swarm-grafana/oidc/client" s && !(lib.hasInfix "secret/swarm/hives/" s); } + { + # A copy on persistent disk outlives the secret's rotation and lands in + # backups of the container's root. + name = "grafana's OIDC secret, admin password and metrics bearer are written to the host's /run only, and the old copies removed"; + ok = + let + m = grafanaRemoteAuthelia; + services = m.systemd.services; + mount = m.containers.swarm-grafana.bindMounts."/run/swarm-grafana-secrets"; + rules = m.systemd.tmpfiles.rules; + guest = m.containers.swarm-grafana.config.services.grafana; + persistent = s: lib.hasInfix "/var/lib/nixos-containers" s || lib.hasInfix "/var/lib/grafana-" s; + oidc = services.swarm-bao-grafana-oidc.script; + admin = services.swarm-bao-grafana-admin.script; + bearer = services.swarm-bao-grafana-metrics-bearer.script; + in + lib.hasInfix "atomic_write_secret 0400 196:0 /run/swarm-grafana-secrets/swarm-grafana.secret" oidc + && lib.hasInfix "atomic_write_secret 0400 196:0 /run/swarm-grafana-secrets/metrics-bearer" bearer + && !(persistent oidc) + && !(persistent admin) + && !(persistent bearer) + && !(persistent services.swarm-bao-grafana-controller-token.script) + && lib.hasInfix "< /run/swarm-grafana-secrets/admin-password" services.swarm-bao-grafana-controller-token.script + && mount.hostPath == "/run/swarm-grafana-secrets" + && mount.isReadOnly + && + guest.settings."auth.generic_oauth".client_secret + == "$__file{/run/swarm-grafana-secrets/swarm-grafana.secret}" + && lib.any ( + d: d.secureJsonData.httpHeaderValue1 or null == "$__file{/run/swarm-grafana-secrets/metrics-bearer}" + ) guest.provision.datasources.settings.datasources + && lib.elem "d /run/swarm-grafana-secrets 0755 root root -" rules + && lib.elem "r /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-oidc/swarm-grafana.secret" rules + && lib.elem "r /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-admin/password" rules + && lib.elem "r /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-metrics/bearer" rules + && !(lib.any (lib.hasPrefix "f /run/swarm-grafana-secrets") rules); + } { # Both halves of the co-location assumption, which was one host's # `deploy.*` answering a question about the whole swarm: the identities