vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm-grafana: deliver the OIDC client secret through the secret store

Browse source 
Grafana's OIDC client secret only existed where authelia did. One
`ssoLocal` gate — `grafana.enable && authelia.enable` — decided the
client registration, the minted secret's delivery and the whole
`auth.generic_oauth` block, so a swarm whose authelia runs on another
host got Grafana with no SSO wiring at all. The local login form is
disabled unconditionally, so that is no way in.

Split the one gate into the two questions it was conflating:

- `ssoConfigured` — does this SWARM have an identity provider
  (`swarm.authelia.url`, which is swarm-wide and whose own description
  makes null mean "no SSO configured"). With a delivery route present
  this is what emits Grafana's OIDC block.
- `ssoLocal` — is authelia on THIS host, now spelled as the forge and
  matrix modules spell it. It decides only which unit delivers the
  secret.

Where authelia is elsewhere, `swarm-bao-grafana-oidc.service` reads the
secret from the swarm secret store, shaped after
glue-queue-agent-credential.nix: cert login fails loudly because a retry
fixes every state it fails on, the read degrades quietly because no
retry turns "no value there" into a value, and nothing writes a
stand-in. The producer is the publisher that already runs on authelia's
host, which gains the swarm's service clients beside the per-hive ones
at `swarm/services/<id>/oidc/client` — with the write grant in
swarm-bao.nix and the hive read grant in `policy::render` to match.

Registration moved to glue-grafana-oidc-client.nix. It has to be
declared where authelia's config is rendered, and swarm-grafana.nix's
config block hangs off this host running Grafana.

Two judgement calls stated rather than buried: a hive's read policy now
grants the whole `services` prefix, because a service's path names the
service and nothing swarm-wide records which hive runs it (cost recorded
in docs/trust-boundary/security.md); and the client is registered on any
authelia host, because no swarm-wide "this swarm has a Grafana" fact
exists to gate it on.

Refs #4234

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas 2026-09-13 18:14:11 +02:00 • committed by mara
commit 4aa982cc2a
9 changed files with 577 additions and 85 deletions
Download patch file Download diff file Expand all files Collapse all files

288
nix/host-modules/swarm-grafana.nix
View file

@ -21,6 +21,8 @@ let
hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway;
autheliaCfg = hyperhiveCfg.swarm.authelia;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
vmCfg = hyperhiveCfg.swarm.victoriametrics;
vlCfg = hyperhiveCfg.swarm.victorialogs;
swarmDomain = hyperhiveCfg.swarm.domain;
@ -78,10 +80,41 @@ let
# operator sees, not a coercion error from here.
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
# The all-local case: this host runs BOTH Grafana and the swarm's authelia,
# so the minted secret can be moved without an operator. Same split the
# forge and matrix modules document.
ssoLocal = deployCfg.grafana.enable && deployCfg.authelia.enable;
# Is SSO configured for this SWARM. Swarm-wide by construction — `swarm.*` is
# identical on every host — and the option's own description is what makes
# this the right question to ask: a null URL means "no SSO configured".
ssoConfigured = autheliaCfg.url != null;
# Is authelia on THIS host, spelled exactly as the forge and matrix modules
# spell it. All locality decides is where the minted plaintext is copied
# FROM: authelia's own tree here, the swarm secret store otherwise.
#
# 🩸 This used to be `grafana.enable && authelia.enable` and was the ONLY
# gate: registration, the mint's delivery and the OIDC block all hung off
# it, so a swarm whose authelia ran elsewhere got Grafana with no SSO wiring
# — and the login form is disabled whatever happens, so with nothing else.
ssoLocal = deployCfg.authelia.enable;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Whether Grafana gets an OIDC block: the swarm-wide question, AND that
# something on this host delivers the secret that block names. The second
# half is not redundant — a block naming a file no unit here writes cannot
# produce a working login either way, and what Grafana does with an
# unreadable `$__file{}` target (refuse to start, or start and fail every
# login) is not worth betting the container on when not emitting the block is
# available and leaves this host as it was.
ssoWired = ssoConfigured && (ssoLocal || haveClientIdentity);
# The delivery that is not a copy: authelia is elsewhere, so the same minted
# plaintext arrives out of the store that the publisher on authelia's host
# wrote it into. Exclusive with the copy unit by construction — two units
# writing one file is a race over which secret Grafana parses.
storeDelivery = ssoWired && !ssoLocal;
autheliaUrl = toString autheliaCfg.url;
# Where the plaintext lands inside the container. Under /var/lib rather
@ -90,6 +123,23 @@ let
# intermittent one.
secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret";
# The same file seen from the host, which is where both delivery units write
# it. Spelled once: two units landing a secret at two spellings of one path
# is a Grafana that reads whichever of them it was configured with.
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
hostSecretDir = builtins.dirOf hostSecretPath;
# Where the publisher on authelia's host leaves this client's secret. The
# `services` segment is `swarm-secret-client`'s `path::Kind::Service` — a
# swarm service's client is registered once for the whole swarm, so its
# secret is one value and not one per hive. Both ends compose this from the
# same swarm-wide client id, so a rename cannot leave one of them behind.
#
# ⚠️ The prefix is not decoration: it is what this host's read grant covers,
# so a path outside it answers 403 rather than a miss, however correct it
# looks.
storeSecretPath = "secret/swarm/services/${cfg.oidc.clientId}/oidc/client";
# Grafana's own datasource-encryption key. Generated in-container (see the
# unit below) because nothing outside the container ever reads it — unlike
# the OIDC secret above, whose other reader is authelia's container.
@ -104,11 +154,6 @@ let
nginxGid = config.ids.gids.nginx;
grafanaUid = config.ids.uids.grafana;
# Format-locked by Grafana: the generic OAuth callback is always
# `<root_url>/login/generic_oauth`. Declared once here and read by both
# the authelia client and Grafana itself.
redirectUri = "https://${cfg.domain}/login/generic_oauth";
in
{
# `enable` moved to `services.hyperhive.deploy.grafana.enable` — see
@ -186,6 +231,25 @@ in
'';
};
redirectUri = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "https://${cfg.domain}/login/generic_oauth";
defaultText = lib.literalExpression ''"https://''${services.hyperhive.swarm.grafana.domain}/login/generic_oauth"'';
description = ''
OAuth callback authelia sends the browser back to, and the URI it
matches **exactly**.
Read-only, like {option}`services.hyperhive.swarm.grafana.machine`
and for the same reason: Grafana derives it from its own
`root_url` (`<root_url>/login/generic_oauth`), so it is a fact
other modules may read rather than a knob. The glue that registers
this client wherever authelia runs reads it from here instead of
restating the format — a second spelling of it is a silently
rejected login.
'';
};
role = lib.mkOption {
type = lib.types.enum [
"Viewer"
@ -336,11 +400,16 @@ in
# The secret oneshots as well as grafana itself: each runs before it and
# fails in ways grafana then reports only as a login that does not work.
#
# Each delivery unit is listed only where it exists, the way
# ./hive-matrix.nix lists its own: a unit name that never renders is a
# journald scrape target matching nothing, which reads as a quiet unit.
services.hyperhive.swarm.otel.journaldUnits = [
"grafana"
"swarm-grafana-oidc-secret"
"swarm-grafana-secret-key"
];
]
++ lib.optional ssoLocal "swarm-grafana-oidc-secret"
++ lib.optional storeDelivery "swarm-bao-grafana-oidc";
services.hyperhive.swarm.controller.links = [
{
@ -350,35 +419,40 @@ in
}
];
# One declaration, two readers. Grafana's callback URL is format-locked
# to its own root URL; making the operator restate it in authelia's
# client list would be a second source of truth for a string whose
# mismatch is a silently rejected login.
#
# `kind` is left at its `interactive` default: a person logs in here.
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf ssoLocal [
{
id = cfg.oidc.clientId;
description = "HyperHive swarm metrics";
redirectUris = [ redirectUri ];
}
];
# Registering the client is NOT here any more: it has to happen on the
# host that runs authelia, and this whole block is gated on the host that
# runs Grafana. ./glue-grafana-oidc-client.nix is where it moved to.
warnings = lib.optional (ssoConfigured && !ssoLocal && !haveClientIdentity) ''
services.hyperhive.swarm.grafana: this swarm has an IdP
(services.hyperhive.swarm.authelia.url) but authelia is not on this
host and this host holds no swarm-secret-store identity, so nothing
can deliver Grafana's OIDC client secret. Grafana will come up with
no OIDC login and no local login form, which is no way in at all.
Name this host's store client leaf in
services.hyperhive.deploy.bao.clientCertFile and
services.hyperhive.deploy.bao.clientKeyFile — the same identity every
other reader of the store uses. See docs/swarm/secrets.md.
'';
assertions = [
{
# Grafana reaches the token endpoint server-to-server, so a null URL
# would become a request to `null/api/oidc/token` — a DNS failure
# several layers from its cause. Only reachable by enabling authelia
# and clearing its `url`, which is why it is an assertion and not a
# fallback.
assertion = !ssoLocal || autheliaCfg.url != null;
# The URL is what `ssoConfigured` above reads, so with it null this
# host mints and delivers a secret for an OIDC block it then does not
# emit — SSO silently absent on the one deployment that has every
# other piece of it. Only reachable by enabling authelia and clearing
# its `url`, which is why it is an assertion and not a fallback.
assertion = !ssoLocal || ssoConfigured;
message = ''
services.hyperhive.swarm.grafana requires
services.hyperhive.swarm.authelia.url when authelia is enabled.
Grafana exchanges its authorization code at
`''${url}/api/oidc/token` from inside its container. With the URL
null there is no endpoint to name.
`''${url}/api/oidc/token` from inside its container, so with the URL
null there is no endpoint to name — and a null URL is also how this
module reads "no SSO configured for this swarm", which would leave
Grafana with no login of any kind on the host that mints its secret.
'';
}
];
@ -450,11 +524,11 @@ in
"d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -"
];
# The secret delivery. It runs on the HOST because that is the only place
# both container trees are addressable: they share this host's network
# namespace, which makes them feel co-located, but their filesystem roots
# are separate — Grafana cannot open a path inside authelia's tree
# however local the port looks.
# The local delivery, and the ONLY thing locality still decides. It runs on
# the HOST because that is the only place both container trees are
# addressable: they share this host's network namespace, which makes them
# feel co-located, but their filesystem roots are separate — Grafana cannot
# open a path inside authelia's tree however local the port looks.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
# refuses to start when a bind source is missing, and this secret does
@ -482,7 +556,7 @@ in
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.oidc.clientId}.secret"}
dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${secretPath}"}
dst=${lib.escapeShellArg hostSecretPath}
# authelia's container is up, but its first-boot generator may still
# be minting. Bounded wait, then fail: a silent skip here produces a
@ -513,6 +587,137 @@ in
'';
};
# The same secret, the same destination, for the deployment where authelia
# is NOT here: it arrives out of the swarm secret store, which the
# publisher on authelia's host wrote it into. Nothing above changes — this
# is the second source for one file, never a second file.
#
# Shaped after ./glue-queue-agent-credential.nix rather than after the copy
# above, because it is a store reader and those are the store's readers: a
# cert login that fails LOUDLY, since every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY, since no retry turns "no
# value there" into a value.
#
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs on the authelia host and authelia mints on its first
# boot, so early in a swarm's life there is legitimately nothing to read.
# Nothing here writes a stand-in: there is no such thing as a locally valid
# OIDC client secret, and a placeholder would turn a Grafana that cannot
# log anyone in into one whose token exchange is refused.
#
# ⇒ So the absence surfaces at Grafana rather than here, and this unit's job
# is to leave the journal line that says which of the three it was. That is
# the same division the copy unit above makes by failing rather than
# skipping: the secret not arriving is a Grafana nobody can log into, and a
# named cause is the only thing separating that from a mystery.
systemd.services.swarm-bao-grafana-oidc = lib.mkIf storeDelivery {
description = "fetch Grafana's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering
# is conditional even though the read is not: off-host there is nothing
# local to wait for, and the timeout below bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${cfg.machine}.service" ];
wantedBy = [
"multi-user.target"
"container@${cfg.machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it. An
# hours-long window would be a bet on a sealed store, and the degrade
# below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-grafana-oidc";
# What actually bounds the read below. Stated here rather than left to
# systemd's default, so the number a boot waits on is in the file that
# waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving Grafana's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; Grafana has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# Written with a shell builtin, never handed to a program: `printf` is
# bash's own, so the plaintext never becomes an argument in /proc the
# way `install <<<"$secret"` or an `echo` from `path` would.
#
# Same uid, group and mode as the copy above, for the reasons stated
# there — this is the same file arriving by a different route.
install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath}
chown ${toString config.ids.uids.grafana}:0 ${lib.escapeShellArg hostSecretPath}
chmod 0400 ${lib.escapeShellArg hostSecretPath}
'';
};
containers.${cfg.machine} = {
autoStart = true;
ephemeral = false;
@ -710,12 +915,11 @@ in
#
# Deliberately NOT `ssoLocal`: whether a password box sits on a
# public login page cannot depend on which host happens to run
# authelia. The OIDC block below is still gated on locality;
# making that follow the same swarm-wide question is a larger
# change tracked on its own.
# authelia. The OIDC block below now answers the same swarm-wide
# question, so the two no longer disagree.
auth.disable_login_form = true;
}
// lib.optionalAttrs ssoLocal {
// lib.optionalAttrs ssoWired {
"auth.generic_oauth" = {
enabled = true;
name = "HyperHive";