vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm-grafana: one delivery route for the OIDC client secret

Browse source 
The previous commit left two delivery paths and a three-way gate:
`swarm-grafana-oidc-secret.service` copied authelia's minted plaintext
out of its host tree wherever the two were co-located,
`swarm-bao-grafana-oidc.service` read the same value from the swarm
secret store wherever they were not, and `ssoConfigured && (ssoLocal ||
haveClientIdentity)` decided whether Grafana got an OIDC block at all.

Delete the co-located path. The store reader is now THE delivery unit,
in every deployment — the publisher on authelia's host writes
`swarm/services/<id>/oidc/client` whether the reader is a network away
or in the container next door. The ruling behind it: the store exists so
a host holds ONE out-of-band secret, its client certificate, and reads
everything else with it. Skipping the store when the producer happens to
be local saves a round trip and costs a second delivery unit, a second
way for the file to be wrong, and a gate to choose between them.

The gate goes too, and both of its questions become assertions, scoped
to hosts that run Grafana:

- `swarm.authelia.url` must be set. `auth.disable_login_form` is
  unconditional — Grafana ships an admin/admin account on a public
  vhost — so dropping the OIDC block when the swarm names no IdP
  produced a container with no SSO and no password box, silently. An
  eval-time refusal naming the option is the only report that reaches
  anyone, the shape swarm-nats.nix already uses for the same option.
- `deploy.bao.clientCertFile` / `clientKeyFile` must be set. This
  replaces a warning that nothing reads back, and its message names both
  options and where the leaf comes from.

Fixtures follow. `grafanaWithAuthelia` gains the cert pair, because a
co-located host is a store reader like any other. The old
`grafanaRemoteAutheliaNoIdentity` is kept rather than deleted, renamed
`grafanaNoIdentity`: the shape is still reachable, only its deliverable
changed from silence to a refusal, and an arm now reads that refusal
back. Its mirror `grafanaNoSso` covers the other assertion, each fixture
wrong in exactly one way so an arm can name which refusal fired. Every
positive keeps an explicit negative — the one-delivery-unit arm asserts
the deleted unit is absent in both topologies rather than merely that
the store reader is present.

Refs #4234

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas 2026-09-13 19:27:33 +02:00 • committed by mara
commit 815f977d7c
5 changed files with 295 additions and 233 deletions
Download patch file Download diff file Expand all files Collapse all files

266
nix/host-modules/swarm-grafana.nix
View file

@ -83,38 +83,27 @@ let
# Is SSO configured for this SWARM. Swarm-wide by construction — `swarm.*` is
# identical on every host — and the option's own description is what makes
# this the right question to ask: a null URL means "no SSO configured".
ssoConfigured = autheliaCfg.url != null;
# Is authelia on THIS host, spelled exactly as the forge and matrix modules
# spell it. All locality decides is where the minted plaintext is copied
# FROM: authelia's own tree here, the swarm secret store otherwise.
#
# 🩸 This used to be `grafana.enable && authelia.enable` and was the ONLY
# gate: registration, the mint's delivery and the OIDC block all hung off
# it, so a swarm whose authelia ran elsewhere got Grafana with no SSO wiring
# — and the login form is disabled whatever happens, so with nothing else.
ssoLocal = deployCfg.authelia.enable;
# 🩸 A subject of an assertion below, NOT a gate. Dropping the OIDC block when
# this is false looks conservative and is the outage: `disable_login_form` is
# unconditional a few hundred lines down, so a Grafana with no OIDC settings
# is a Grafana with no login of any kind, arrived at silently. SSO is a
# requirement of running this service, so an unconfigured swarm fails to
# build and says which option to set.
ssoConfigured = autheliaCfg.url != null;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full.
#
# Also asserted rather than gating anything, and that is the ruling this
# module's second delivery route was deleted under: the store exists so that
# ONE secret is placed out of band per host — this certificate — and every
# other secret is read with it. A Grafana host without it has not been given
# its identity yet, which is a thing to say out loud rather than to route
# around by reaching into authelia's tree whenever it happens to be local.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Whether Grafana gets an OIDC block: the swarm-wide question, AND that
# something on this host delivers the secret that block names. The second
# half is not redundant — a block naming a file no unit here writes cannot
# produce a working login either way, and what Grafana does with an
# unreadable `$__file{}` target (refuse to start, or start and fail every
# login) is not worth betting the container on when not emitting the block is
# available and leaves this host as it was.
ssoWired = ssoConfigured && (ssoLocal || haveClientIdentity);
# The delivery that is not a copy: authelia is elsewhere, so the same minted
# plaintext arrives out of the store that the publisher on authelia's host
# wrote it into. Exclusive with the copy unit by construction — two units
# writing one file is a race over which secret Grafana parses.
storeDelivery = ssoWired && !ssoLocal;
autheliaUrl = toString autheliaCfg.url;
# Where the plaintext lands inside the container. Under /var/lib rather
@ -123,9 +112,10 @@ let
# intermittent one.
secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret";
# The same file seen from the host, which is where both delivery units write
# it. Spelled once: two units landing a secret at two spellings of one path
# is a Grafana that reads whichever of them it was configured with.
# The same file seen from the host, which is where the delivery unit writes
# it. Spelled once: the unit that writes it and the config that names it are
# a few hundred lines apart, and a Grafana reading a path nothing writes is
# a login that fails with nothing in any log about the file.
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
hostSecretDir = builtins.dirOf hostSecretPath;
@ -401,15 +391,16 @@ in
# The secret oneshots as well as grafana itself: each runs before it and
# fails in ways grafana then reports only as a login that does not work.
#
# Each delivery unit is listed only where it exists, the way
# ./hive-matrix.nix lists its own: a unit name that never renders is a
# journald scrape target matching nothing, which reads as a quiet unit.
# Unconditional, because every unit named here now renders in every
# deployment. This list used to be assembled with `lib.optional` per
# delivery route, which was the right shape while there were two — a unit
# name that never renders is a journald scrape target matching nothing,
# which reads as a quiet unit rather than an absent one.
services.hyperhive.swarm.otel.journaldUnits = [
"grafana"
"swarm-grafana-secret-key"
]
++ lib.optional ssoLocal "swarm-grafana-oidc-secret"
++ lib.optional storeDelivery "swarm-bao-grafana-oidc";
"swarm-bao-grafana-oidc"
];
services.hyperhive.swarm.controller.links = [
{
@ -423,36 +414,57 @@ in
# host that runs authelia, and this whole block is gated on the host that
# runs Grafana. ./glue-grafana-oidc-client.nix is where it moved to.
warnings = lib.optional (ssoConfigured && !ssoLocal && !haveClientIdentity) ''
services.hyperhive.swarm.grafana: this swarm has an IdP
(services.hyperhive.swarm.authelia.url) but authelia is not on this
host and this host holds no swarm-secret-store identity, so nothing
can deliver Grafana's OIDC client secret. Grafana will come up with
no OIDC login and no local login form, which is no way in at all.
Name this host's store client leaf in
services.hyperhive.deploy.bao.clientCertFile and
services.hyperhive.deploy.bao.clientKeyFile — the same identity every
other reader of the store uses. See docs/swarm/secrets.md.
'';
# Both arms are what used to be a silent gate, and both fire only where
# Grafana runs — this whole block is under `deploy.grafana.enable`. The
# binding itself is asserted rather than a copy of its formula, the way
# ./swarm-nats.nix's own `autheliaUrl` arm does it: two spellings of one
# boolean is two places for a future edit to land in only one.
assertions = [
{
# The URL is what `ssoConfigured` above reads, so with it null this
# host mints and delivers a secret for an OIDC block it then does not
# emit — SSO silently absent on the one deployment that has every
# other piece of it. Only reachable by enabling authelia and clearing
# its `url`, which is why it is an assertion and not a fallback.
assertion = !ssoLocal || ssoConfigured;
# SSO is not optional for this service, and the reason is a hundred
# lines below in `auth.disable_login_form = true`: Grafana ships an
# `admin`/`admin` account on a public vhost, so the password box is
# off whatever the topology. Emitting no OIDC block when the swarm
# names no IdP therefore produces a container with no way in at all —
# a state no log names, since nothing failed. Failing the build and
# naming the option is the only report that reaches anyone.
assertion = ssoConfigured;
message = ''
services.hyperhive.swarm.grafana requires
services.hyperhive.swarm.authelia.url when authelia is enabled.
services.hyperhive.deploy.grafana.enable requires
services.hyperhive.swarm.authelia.url — Grafana's only login is SSO,
because its local login form is disabled unconditionally (it ships
an admin/admin account and its vhost is on the public gateway).
Grafana exchanges its authorization code at
`''${url}/api/oidc/token` from inside its container, so with the URL
null there is no endpoint to name — and a null URL is also how this
module reads "no SSO configured for this swarm", which would leave
Grafana with no login of any kind on the host that mints its secret.
It defaults to this host's own instance only when this host runs
authelia. A hive that federates with a swarm sets it explicitly to
wherever that provider lives. Grafana exchanges its authorization
code at `''${url}/api/oidc/token` from inside its container, so a
null URL leaves no endpoint to name.
'';
}
{
# The other half of one login: the OIDC block names a `$__file{}` that
# `swarm-bao-grafana-oidc.service` below writes, and that unit reads
# the store with this host's client certificate. No certificate, no
# secret, and the same no-way-in Grafana as the arm above.
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.grafana.enable requires this host to hold a
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.clientCertFile
services.hyperhive.deploy.bao.clientKeyFile
Grafana's OIDC client secret is minted by authelia and read out of
the store, on every host that runs Grafana — including the host that
runs authelia. That is one delivery route rather than two, and it is
what the store is for: this certificate is the single credential
placed out of band, and every other secret comes from the store with
it.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
'';
}
];
@ -524,79 +536,24 @@ in
"d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -"
];
# The local delivery, and the ONLY thing locality still decides. It runs on
# the HOST because that is the only place both container trees are
# addressable: they share this host's network namespace, which makes them
# feel co-located, but their filesystem roots are separate — Grafana cannot
# open a path inside authelia's tree however local the port looks.
# THE delivery unit — one route, in every deployment. The secret authelia
# minted arrives out of the swarm secret store, which the publisher on
# authelia's host wrote it into, and that is true whether authelia is a
# network away or in the container next door.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
# refuses to start when a bind source is missing, and this secret does
# not exist until authelia's first boot has minted it — so binding it
# would make Grafana wait on a file that waits on a container that starts
# after it.
systemd.services.swarm-grafana-oidc-secret = lib.mkIf ssoLocal {
description = "deliver Grafana's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ];
requires = [ "container@${autheliaCfg.machine}.service" ];
before = [ "container@${cfg.machine}.service" ];
wantedBy = [ "container@${cfg.machine}.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-grafana-oidc-secret";
# ⚠️ Longer than the wait below, and that is the whole point:
# `DefaultTimeoutStartSec` is 90s, so a 120s bounded wait is killed
# by systemd at 90 — before it can emit the error naming the file it
# waited for. The timeout has to outlive the thing it is timing.
TimeoutStartSec = "180s";
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.oidc.clientId}.secret"}
dst=${lib.escapeShellArg hostSecretPath}
# authelia's container is up, but its first-boot generator may still
# be minting. Bounded wait, then fail: a silent skip here produces a
# Grafana whose only login path dead-ends.
deadline=$(( SECONDS + 120 ))
while [ ! -s "$src" ]; do
if [ "$SECONDS" -ge "$deadline" ]; then
echo "authelia has not minted $src after 120s" >&2
exit 1
fi
sleep 2
done
# Owned by Grafana's own uid, unlike the matrix sibling which lands
# root-owned: tuwunel's secret is read by `LoadCredential` as root
# before the sandbox exists, whereas Grafana expands `$__file{}`
# itself, as itself, while parsing its config. These containers set
# no `privateUsers`, so the host uid is the container uid, and both
# sides take it from the same static NixOS id.
#
# Group is root, not grafana, and that is forced rather than chosen:
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
# — the group's gid is allocated at activation inside the container,
# so the host cannot know it at eval time. Harmless here because 0400
# grants the group nothing; if this mode ever widens, the gid has to
# be discovered at runtime rather than assumed.
install -D -m 0400 -o ${toString config.ids.uids.grafana} -g 0 "$src" "$dst"
'';
};
# The same secret, the same destination, for the deployment where authelia
# is NOT here: it arrives out of the swarm secret store, which the
# publisher on authelia's host wrote it into. Nothing above changes — this
# is the second source for one file, never a second file.
# 🩸 A second unit here used to copy the plaintext out of authelia's host
# tree wherever the two were co-located, with the OIDC block gated on
# either route being available. The ruling that deleted it: the store
# exists so a host holds ONE out-of-band secret — its client certificate —
# and reads everything else with it, so skipping the store when the
# producer is local buys a round trip and costs a second delivery path, a
# second way for the file to be wrong, and a gate whose false arm was a
# Grafana with no login. Recorded in docs/swarm/secrets.md.
#
# Shaped after ./glue-queue-agent-credential.nix rather than after the copy
# above, because it is a store reader and those are the store's readers: a
# cert login that fails LOUDLY, since every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY, since no retry turns "no
# value there" into a value.
# Shaped after ./glue-queue-agent-credential.nix, because it is a store
# reader and this is what the store's readers do: a cert login that fails
# LOUDLY, since every state it fails on is one a retry fixes, then a read
# that degrades QUIETLY, since no retry turns "no value there" into a value.
#
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs on the authelia host and authelia mints on its first
@ -606,11 +563,10 @@ in
# log anyone in into one whose token exchange is refused.
#
# ⇒ So the absence surfaces at Grafana rather than here, and this unit's job
# is to leave the journal line that says which of the three it was. That is
# the same division the copy unit above makes by failing rather than
# skipping: the secret not arriving is a Grafana nobody can log into, and a
# named cause is the only thing separating that from a mystery.
systemd.services.swarm-bao-grafana-oidc = lib.mkIf storeDelivery {
# is to leave the journal line that says which of the three it was: the
# secret not arriving is a Grafana nobody can log into, and a named cause is
# the only thing separating that from a mystery.
systemd.services.swarm-bao-grafana-oidc = {
description = "fetch Grafana's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering
@ -708,8 +664,19 @@ in
# bash's own, so the plaintext never becomes an argument in /proc the
# way `install <<<"$secret"` or an `echo` from `path` would.
#
# Same uid, group and mode as the copy above, for the reasons stated
# there — this is the same file arriving by a different route.
# Owned by Grafana's own uid, unlike the matrix sibling which lands
# root-owned: tuwunel's secret is read by `LoadCredential` as root
# before the sandbox exists, whereas Grafana expands `$__file{}`
# itself, as itself, while parsing its config. This container sets no
# `privateUsers`, so the host uid is the container uid, and both sides
# take it from the same static NixOS id.
#
# Group is root, not grafana, and that is forced rather than chosen:
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
# — the group's gid is allocated at activation inside the container, so
# the host cannot know it at eval time. Harmless here because 0400
# grants the group nothing; if this mode ever widens, the gid has to be
# discovered at runtime rather than assumed.
install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath}
@ -913,13 +880,21 @@ in
# public gateway, so SSO is the only way in — not merely the
# preferred one.
#
# Deliberately NOT `ssoLocal`: whether a password box sits on a
# public login page cannot depend on which host happens to run
# authelia. The OIDC block below now answers the same swarm-wide
# question, so the two no longer disagree.
# Not conditional on anything, and the block below is not
# either, which is what keeps the two from ever disagreeing.
# Whether a password box sits on a public login page cannot
# depend on which host happens to run authelia — and it used to,
# in a module where the OIDC block was gated and this line was
# not.
auth.disable_login_form = true;
}
// lib.optionalAttrs ssoWired {
# Emitted in every deployment, because SSO is a requirement of
# running this service rather than a feature of some topologies:
# the assertions above refuse to build a Grafana that lacks an
# IdP to point at or the identity to fetch its secret with. A
# `lib.optionalAttrs` here would be the silent half of that pair,
# dropping the only login this container has whenever a value
# went null.
"auth.generic_oauth" = {
enabled = true;
name = "HyperHive";
@ -930,6 +905,9 @@ in
# `database.password`, `security.admin_password` and
# datasource `secureJsonData`), so nothing but this comment
# stands between a literal and the store.
#
# `swarm-bao-grafana-oidc.service` above is what writes this
# path, on every host that runs Grafana.
client_secret = "$__file{${secretPath}}";
scopes = "openid profile email groups";
auth_url = "${autheliaUrl}/api/oidc/authorization";

23
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,6 +1,13 @@
# The unit that copies authelia's minted OIDC client secrets into the swarm's
# secret store, so whoever needs one without hosting authelia can read it: a
# hive its agents' credential, a swarm service its own.
# secret store, so whoever needs one can read it there: a hive its agents'
# credential, a swarm service its own.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
# reader fetches it from the store in every deployment — ./swarm-grafana.nix
# states the ruling that made that the only route. Publishing "only when the
# reader is elsewhere" would be a second shape of this unit, gated on a fact
# about another host, to save a round trip on the one host that can afford it.
#
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
@ -70,7 +77,8 @@ in
description = ''
Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service elsewhere can read its own.
agents' credential and a swarm service can read its own — from
wherever it runs, this host included.
Defaults to whether this host mints them, which is the only half of
the question that is a property of *this* host.
@ -82,10 +90,11 @@ in
publishing nothing. Whether the wiring is complete is the client
identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every non-co-located hive without a delivery
path, which is the state this exists to end — so the honest reason
to set it false is a deployment delivering those secrets by some
other mechanism it owns.
Turning it off leaves every hive but this one without its agents'
credential, and the swarm's Grafana without any login at all — its
secret has exactly one route and this is the producer's end of it. So
the honest reason to set it false is a deployment delivering those
secrets by some other mechanism it owns.
'';
};