swarm-grafana: keep Grafana's OIDC secret, admin password and metrics bearer on the host's /run, not in the container's root
The three delivery units still read from bao under Grafana's own leaf, but
now write to /run/swarm-grafana-secrets on the host (tmpfs), which the
container bind-mounts read-only at the same path. Before, they wrote to
/var/lib/nixos-containers/swarm-grafana/var/lib/{grafana-oidc,grafana-admin,
grafana-metrics}/ on persistent disk, where a superseded value stayed after
rotation and went into backups of the container root. Host tmpfiles now
removes the three old files.
Rotation is unchanged: refresh_consumer restarts Grafana (or starts it if it
failed) when a file changes, since Grafana reads each $__file{} once as it
starts. After a reboot the files are fetched again; until the store answers
Grafana stays down, where before it came up on the last copy from disk.
Part of #4998 (slice 2b).
This commit is contained in:
atlas
mara
parent
a169947f15
commit
cbab23273a
4 changed files with 95 additions and 37 deletions
|
|
@ -52,9 +52,9 @@ neither is a renaming of the other.
|
|||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/var/lib/grafana-admin/password` inside the `swarm-grafana` container: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. Grafana doesn't start until the file exists, so it stays down rather than falling back to the upstream default `admin` |
|
||||
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/var/lib/grafana-metrics/bearer` inside the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
|
||||
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
|
||||
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
|
||||
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
||||
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
|
|
|
|||
|
|
@ -312,7 +312,7 @@ gateway refuses its token too. See [`swarmctl/README.md`](../../swarmctl/README.
|
|||
dead provider locks out the SSO accounts but not that one: an operator
|
||||
reads the password from `swarm/services/<grafana client id>/admin/password`
|
||||
in the store (or, as root on the Grafana host, from
|
||||
`/var/lib/grafana-admin/password` inside the `swarm-grafana` container)
|
||||
`/run/swarm-grafana-secrets/admin-password`)
|
||||
and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana)
|
||||
and [`secrets.md`](secrets.md). SSO logins are the everyday way in, which
|
||||
is why `swarm.grafana.oidc.role` defaults to `Admin`.
|
||||
|
|
|
|||
|
|
@ -226,18 +226,26 @@ let
|
|||
|
||||
autheliaUrl = toString hyperhiveCfg.swarm.authelia.url;
|
||||
|
||||
# Where the plaintext lands inside the container. Under /var/lib rather
|
||||
# than /run: Grafana may start before the delivery unit on a later boot,
|
||||
# and a secret that evaporates on reboot turns a working login into an
|
||||
# intermittent one.
|
||||
secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret";
|
||||
|
||||
# The same file seen from the host, which is where the delivery unit writes
|
||||
# it. Spelled once: the unit that writes it and the config that names it are
|
||||
# The three secrets below land on the host's /run (tmpfs), which the
|
||||
# container bind-mounts read-only at the same path: none reaches persistent
|
||||
# disk, and a reboot fetches them again. Each file is absent until its
|
||||
# delivery unit has run, and Grafana does not start without them — `$__file{}`
|
||||
# on a missing path is a startup error, and an empty admin password would
|
||||
# fall back to the upstream default. A delivery unit that writes a file
|
||||
# restarts or starts Grafana (`refresh_consumer`).
|
||||
#
|
||||
# Spelled once: the unit that writes a file and the config that names it are
|
||||
# a few hundred lines apart, and a Grafana reading a path nothing writes is
|
||||
# a login that fails with nothing in any log about the file.
|
||||
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
|
||||
hostSecretDir = builtins.dirOf hostSecretPath;
|
||||
secretDir = "/run/swarm-grafana-secrets";
|
||||
secretPath = "${secretDir}/${cfg.oidc.clientId}.secret";
|
||||
|
||||
# Where the three files sat on persistent disk before they moved to /run.
|
||||
legacyHostSecretPaths = map (p: "/var/lib/nixos-containers/${cfg.machine}${p}") [
|
||||
"/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret"
|
||||
"/var/lib/grafana-admin/password"
|
||||
"/var/lib/grafana-metrics/bearer"
|
||||
];
|
||||
|
||||
# Where the publisher on authelia's host leaves this client's secret. A
|
||||
# swarm service's client is registered once for the whole swarm, so its
|
||||
|
|
@ -260,8 +268,7 @@ let
|
|||
# store by `swarm-bao-grafana-admin` below, which `swarm-bao.nix`'s
|
||||
# `grafanaOidcReaders` grants create and read on this one leaf.
|
||||
adminPasswordStorePath = "secret/swarm/services/${cfg.oidc.clientId}/admin/password";
|
||||
adminPasswordPath = "/var/lib/grafana-admin/password";
|
||||
hostAdminPasswordPath = "/var/lib/nixos-containers/${cfg.machine}${adminPasswordPath}";
|
||||
adminPasswordPath = "${secretDir}/admin-password";
|
||||
|
||||
# The bearer the metrics datasource presents at the metrics store's query
|
||||
# API, which `swarm-metrics-auth` on the store's host checks against the
|
||||
|
|
@ -270,8 +277,7 @@ let
|
|||
# header value, `Bearer <value>`: nixpkgs accepts a `secureJsonData` entry
|
||||
# only when it is exactly one `$__file{}`.
|
||||
metricsBearerStorePath = "secret/swarm/services/${cfg.oidc.clientId}/metrics/bearer";
|
||||
metricsBearerPath = "/var/lib/grafana-metrics/bearer";
|
||||
hostMetricsBearerPath = "/var/lib/nixos-containers/${cfg.machine}${metricsBearerPath}";
|
||||
metricsBearerPath = "${secretDir}/metrics-bearer";
|
||||
|
||||
# swarm-controller's token for Grafana's API, minted by
|
||||
# `swarm-bao-grafana-controller-token` below for a Viewer service account of
|
||||
|
|
@ -545,9 +551,18 @@ in
|
|||
# gateway can traverse; `0750` keeps everything else out. Numeric ids
|
||||
# because the host has no `grafana` account — Grafana lives in the
|
||||
# container, and only the number crosses that boundary.
|
||||
#
|
||||
# The secrets directory is the container's read-only bind source, so it
|
||||
# has to exist before the container starts even when the store does not
|
||||
# answer; the delivery units create it too, but unordered with the
|
||||
# container on a switch. 0755 because Grafana reads the files as its own
|
||||
# uid; each file is 0400 and owned by it. The legacy files are removed so
|
||||
# no superseded secret stays behind on the host's disk.
|
||||
systemd.tmpfiles.rules = [
|
||||
"d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -"
|
||||
];
|
||||
"d ${secretDir} 0755 root root -"
|
||||
]
|
||||
++ map (p: "r ${p}") legacyHostSecretPaths;
|
||||
|
||||
# THE delivery unit — one route, in every deployment. The secret authelia
|
||||
# minted arrives out of the swarm secret store, which the publisher on
|
||||
|
|
@ -678,13 +693,13 @@ in
|
|||
# the host cannot know it at eval time. Harmless here because 0400
|
||||
# grants the group nothing; if this mode ever widens, the gid has to be
|
||||
# discovered at runtime rather than assumed.
|
||||
install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
|
||||
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
|
||||
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostSecretPath} "$secret"
|
||||
install -d -m 0755 ${lib.escapeShellArg secretDir}
|
||||
if secret_differs ${lib.escapeShellArg secretPath} "$secret"; then
|
||||
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg secretPath} "$secret"
|
||||
fi
|
||||
|
||||
# `$__file{}` is expanded once, while Grafana parses its config.
|
||||
refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostSecretPath}
|
||||
refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg secretPath}
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -766,14 +781,14 @@ in
|
|||
fi
|
||||
|
||||
# Owner and mode as for the OIDC secret above, for its reasons.
|
||||
install -d -m 0755 ${lib.escapeShellArg (builtins.dirOf hostAdminPasswordPath)}
|
||||
if secret_differs ${lib.escapeShellArg hostAdminPasswordPath} "$password"; then
|
||||
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostAdminPasswordPath} "$password"
|
||||
install -d -m 0755 ${lib.escapeShellArg secretDir}
|
||||
if secret_differs ${lib.escapeShellArg adminPasswordPath} "$password"; then
|
||||
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg adminPasswordPath} "$password"
|
||||
fi
|
||||
|
||||
# Grafana's `preStart` applies the file to the database, so a file that
|
||||
# changed after it started needs a restart.
|
||||
refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostAdminPasswordPath}
|
||||
refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg adminPasswordPath}
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -847,13 +862,13 @@ in
|
|||
fi
|
||||
|
||||
# Owner and mode as for the OIDC secret above, for its reasons.
|
||||
install -d -m 0755 ${lib.escapeShellArg (builtins.dirOf hostMetricsBearerPath)}
|
||||
if secret_differs ${lib.escapeShellArg hostMetricsBearerPath} "Bearer $bearer"; then
|
||||
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostMetricsBearerPath} "Bearer $bearer"
|
||||
install -d -m 0755 ${lib.escapeShellArg secretDir}
|
||||
if secret_differs ${lib.escapeShellArg metricsBearerPath} "Bearer $bearer"; then
|
||||
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg metricsBearerPath} "Bearer $bearer"
|
||||
fi
|
||||
|
||||
# `$__file{}` in a provisioned datasource is read as Grafana starts.
|
||||
refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg hostMetricsBearerPath}
|
||||
refresh_consumer ${lib.escapeShellArg cfg.machine} grafana.service ${lib.escapeShellArg metricsBearerPath}
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -935,11 +950,11 @@ in
|
|||
exit 1
|
||||
}
|
||||
|
||||
if [ ! -s ${lib.escapeShellArg hostAdminPasswordPath} ]; then
|
||||
echo "no Grafana admin password at ${hostAdminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2
|
||||
if [ ! -s ${lib.escapeShellArg adminPasswordPath} ]; then
|
||||
echo "no Grafana admin password at ${adminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2
|
||||
exit 1
|
||||
fi
|
||||
admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg hostAdminPasswordPath})" | base64 -w0)"
|
||||
admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg adminPasswordPath})" | base64 -w0)"
|
||||
|
||||
status="$(grafana "$admin" GET '/api/serviceaccounts/search?perpage=1000&query=${controllerAccountName}')"
|
||||
expect "the service account search" "$status" 200
|
||||
|
|
@ -1020,6 +1035,12 @@ in
|
|||
hostPath = deployCfg.grafana.socketDir;
|
||||
isReadOnly = false;
|
||||
};
|
||||
# The three delivered secrets, read-only: the container reads them and
|
||||
# never writes one.
|
||||
${secretDir} = {
|
||||
hostPath = secretDir;
|
||||
isReadOnly = true;
|
||||
};
|
||||
}
|
||||
// caTrust.bindMount;
|
||||
|
||||
|
|
|
|||
|
|
@ -242,11 +242,11 @@ let
|
|||
in
|
||||
lib.hasInfix "bao kv put -cas=0 secret/swarm/services/swarm-grafana/admin/password value=-" unit.script
|
||||
&& lib.hasInfix "openssl rand -hex 32" unit.script
|
||||
&& lib.hasInfix "atomic_write_secret 0400 196:0 /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-admin/password" unit.script
|
||||
&& lib.hasInfix "atomic_write_secret 0400 196:0 /run/swarm-grafana-secrets/admin-password" unit.script
|
||||
&& lib.elem "container@swarm-grafana.service" unit.before
|
||||
&& unit.environment.BAO_CLIENT_CERT == "/etc/pki/bao-grafana-oidc.pem"
|
||||
&& grafana.settings.security.admin_password == "$__file{/var/lib/grafana-admin/password}"
|
||||
&& lib.hasInfix "admin reset-admin-password --password-from-stdin < /var/lib/grafana-admin/password" service.preStart
|
||||
&& grafana.settings.security.admin_password == "$__file{/run/swarm-grafana-secrets/admin-password}"
|
||||
&& lib.hasInfix "admin reset-admin-password --password-from-stdin < /run/swarm-grafana-secrets/admin-password" service.preStart
|
||||
&& lib.hasInfix "-config ${resetConfig}" service.script;
|
||||
}
|
||||
{
|
||||
|
|
@ -290,6 +290,43 @@ let
|
|||
lib.hasInfix "secret/swarm/services/swarm-grafana/oidc/client" s
|
||||
&& !(lib.hasInfix "secret/swarm/hives/" s);
|
||||
}
|
||||
{
|
||||
# A copy on persistent disk outlives the secret's rotation and lands in
|
||||
# backups of the container's root.
|
||||
name = "grafana's OIDC secret, admin password and metrics bearer are written to the host's /run only, and the old copies removed";
|
||||
ok =
|
||||
let
|
||||
m = grafanaRemoteAuthelia;
|
||||
services = m.systemd.services;
|
||||
mount = m.containers.swarm-grafana.bindMounts."/run/swarm-grafana-secrets";
|
||||
rules = m.systemd.tmpfiles.rules;
|
||||
guest = m.containers.swarm-grafana.config.services.grafana;
|
||||
persistent = s: lib.hasInfix "/var/lib/nixos-containers" s || lib.hasInfix "/var/lib/grafana-" s;
|
||||
oidc = services.swarm-bao-grafana-oidc.script;
|
||||
admin = services.swarm-bao-grafana-admin.script;
|
||||
bearer = services.swarm-bao-grafana-metrics-bearer.script;
|
||||
in
|
||||
lib.hasInfix "atomic_write_secret 0400 196:0 /run/swarm-grafana-secrets/swarm-grafana.secret" oidc
|
||||
&& lib.hasInfix "atomic_write_secret 0400 196:0 /run/swarm-grafana-secrets/metrics-bearer" bearer
|
||||
&& !(persistent oidc)
|
||||
&& !(persistent admin)
|
||||
&& !(persistent bearer)
|
||||
&& !(persistent services.swarm-bao-grafana-controller-token.script)
|
||||
&& lib.hasInfix "< /run/swarm-grafana-secrets/admin-password" services.swarm-bao-grafana-controller-token.script
|
||||
&& mount.hostPath == "/run/swarm-grafana-secrets"
|
||||
&& mount.isReadOnly
|
||||
&&
|
||||
guest.settings."auth.generic_oauth".client_secret
|
||||
== "$__file{/run/swarm-grafana-secrets/swarm-grafana.secret}"
|
||||
&& lib.any (
|
||||
d: d.secureJsonData.httpHeaderValue1 or null == "$__file{/run/swarm-grafana-secrets/metrics-bearer}"
|
||||
) guest.provision.datasources.settings.datasources
|
||||
&& lib.elem "d /run/swarm-grafana-secrets 0755 root root -" rules
|
||||
&& lib.elem "r /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-oidc/swarm-grafana.secret" rules
|
||||
&& lib.elem "r /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-admin/password" rules
|
||||
&& lib.elem "r /var/lib/nixos-containers/swarm-grafana/var/lib/grafana-metrics/bearer" rules
|
||||
&& !(lib.any (lib.hasPrefix "f /run/swarm-grafana-secrets") rules);
|
||||
}
|
||||
{
|
||||
# Both halves of the co-location assumption, which was one host's
|
||||
# `deploy.*` answering a question about the whole swarm: the identities
|
||||
|
|
|
|||
Loading…
Reference in a new issue