vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm-grafana: one delivery route for the OIDC client secret

Browse source 
The previous commit left two delivery paths and a three-way gate:
`swarm-grafana-oidc-secret.service` copied authelia's minted plaintext
out of its host tree wherever the two were co-located,
`swarm-bao-grafana-oidc.service` read the same value from the swarm
secret store wherever they were not, and `ssoConfigured && (ssoLocal ||
haveClientIdentity)` decided whether Grafana got an OIDC block at all.

Delete the co-located path. The store reader is now THE delivery unit,
in every deployment — the publisher on authelia's host writes
`swarm/services/<id>/oidc/client` whether the reader is a network away
or in the container next door. The ruling behind it: the store exists so
a host holds ONE out-of-band secret, its client certificate, and reads
everything else with it. Skipping the store when the producer happens to
be local saves a round trip and costs a second delivery unit, a second
way for the file to be wrong, and a gate to choose between them.

The gate goes too, and both of its questions become assertions, scoped
to hosts that run Grafana:

- `swarm.authelia.url` must be set. `auth.disable_login_form` is
  unconditional — Grafana ships an admin/admin account on a public
  vhost — so dropping the OIDC block when the swarm names no IdP
  produced a container with no SSO and no password box, silently. An
  eval-time refusal naming the option is the only report that reaches
  anyone, the shape swarm-nats.nix already uses for the same option.
- `deploy.bao.clientCertFile` / `clientKeyFile` must be set. This
  replaces a warning that nothing reads back, and its message names both
  options and where the leaf comes from.

Fixtures follow. `grafanaWithAuthelia` gains the cert pair, because a
co-located host is a store reader like any other. The old
`grafanaRemoteAutheliaNoIdentity` is kept rather than deleted, renamed
`grafanaNoIdentity`: the shape is still reachable, only its deliverable
changed from silence to a refusal, and an arm now reads that refusal
back. Its mirror `grafanaNoSso` covers the other assertion, each fixture
wrong in exactly one way so an arm can name which refusal fired. Every
positive keeps an explicit negative — the one-delivery-unit arm asserts
the deleted unit is absent in both topologies rather than merely that
the store reader is present.

Refs #4234

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas 2026-09-13 19:27:33 +02:00 • committed by mara
commit 815f977d7c
5 changed files with 295 additions and 233 deletions
Download patch file Download diff file Expand all files Collapse all files

72
docs/swarm/secrets.md
View file

@ -41,23 +41,23 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | | swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | | swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | | swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia | | the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
| Grafana's copy of its OIDC secret | `swarm-grafana-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store instead — nothing for an operator to place, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | | the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | | the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
Authelia mints the three keys for itself, in-container, precisely Authelia mints the three keys for itself, in-container, precisely
because nothing outside that container ever reads them. **That's the test worth because nothing outside that container ever reads them. **That's the test worth
@ -185,18 +185,32 @@ the harness logs that it has no queue coordinates — the same absent-and-legal
rather than a container that refuses to start. rather than a container that refuses to start.
The **third reader** is Grafana's OIDC client secret, and it's the first one The **third reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. that's a _swarm service's_ own credential rather than a hive's. It has exactly
`swarm-grafana.nix` asks two questions where it used to ask one: whether this one delivery route: `swarm-bao-grafana-oidc.service` reads
_swarm_ has an IdP (`swarm.authelia.url`) decides whether Grafana gets SSO `swarm/services/<client-id>/oidc/client` out of the store, on **every** host
settings at all, and whether authelia is on _this_ host decides only which of that runs Grafana. Where authelia is doesn't enter into it — the publisher on
two units delivers the secret — the host copy out of authelia's tree, or authelia's host writes that path whether the reader is a network away or in the
`swarm-bao-grafana-oidc.service` reading `swarm/services/<client-id>/oidc/client` container next door.
out of the store. Exactly one of the two exists in any deployment, and where
neither can (no authelia here, no store identity here) the module warns and That's a deliberate refusal of the shortcut. Copying the plaintext
emits no OIDC settings rather than pointing Grafana at a file nothing writes. straight out of authelia's tree when the two happen to share a host is one
Registration moved to `glue-grafana-oidc-client.nix` for a related reason: a round trip cheaper and costs a second delivery unit, a second way for the file
client is a row in authelia's own config, so it has to be declared on the host to be wrong, and a gate deciding between them — and the gate is the expensive
that runs authelia — which is not necessarily the host that runs Grafana. part, because whatever it's wrong about is an outage nobody can read. The store
exists so a host holds **one** out-of-band secret, its client certificate, and
reads everything else with it.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (its local login form is disabled
unconditionally, so SSO isn't a feature of some topologies — it's the only way
in), and it requires this host's `deploy.bao.clientCertFile` /
`clientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana.
A service's secret is one value for the whole swarm rather than one per hive, so A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix it lives under the `services` prefix, and a hive's read policy grants that prefix

10
docs/trust-boundary/security.md
View file

@ -71,11 +71,11 @@ for the whole swarm, so its credential's path names the service and never the
host — and which hive runs a given service is a `deploy.*` fact, per-host by host — and which hive runs a given service is a `deploy.*` fact, per-host by
definition, so nothing swarm-wide exists to scope the grant to. The host running definition, so nothing swarm-wide exists to scope the grant to. The host running
such a service has no store identity of its own either; it reads with the such a service has no store identity of its own either; it reads with the
certificate of the hive it is. So **any hive can read any swarm service's client certificate of the hive it is. **Any hive can therefore read any swarm service's
secret**, which is worth what it buys: a swarm service whose identity provider client secret**, which is worth what it buys: every swarm service gets its
lives on another machine can get its secret at all, instead of only working credential the same way from anywhere, instead of only where the identity
co-located. Giving such a service its own store identity is what would remove provider happens to sit. Giving such a service its own store identity is what
this rather than re-scope it. would remove this rather than re-scope it.
**A tracked follow-up** narrows this, with the two candidate directions: scope **A tracked follow-up** narrows this, with the two candidate directions: scope
the grant per hive (and pay for the re-emission), or give each agent container the grant per hive (and pay for the re-emission), or give each agent container

266
nix/host-modules/swarm-grafana.nix
View file

@ -83,38 +83,27 @@ let
# Is SSO configured for this SWARM. Swarm-wide by construction — `swarm.*` is # Is SSO configured for this SWARM. Swarm-wide by construction — `swarm.*` is
# identical on every host — and the option's own description is what makes # identical on every host — and the option's own description is what makes
# this the right question to ask: a null URL means "no SSO configured". # this the right question to ask: a null URL means "no SSO configured".
ssoConfigured = autheliaCfg.url != null;
# Is authelia on THIS host, spelled exactly as the forge and matrix modules
# spell it. All locality decides is where the minted plaintext is copied
# FROM: authelia's own tree here, the swarm secret store otherwise.
# #
# 🩸 This used to be `grafana.enable && authelia.enable` and was the ONLY # 🩸 A subject of an assertion below, NOT a gate. Dropping the OIDC block when
# gate: registration, the mint's delivery and the OIDC block all hung off # this is false looks conservative and is the outage: `disable_login_form` is
# it, so a swarm whose authelia ran elsewhere got Grafana with no SSO wiring # unconditional a few hundred lines down, so a Grafana with no OIDC settings
# — and the login form is disabled whatever happens, so with nothing else. # is a Grafana with no login of any kind, arrived at silently. SSO is a
ssoLocal = deployCfg.authelia.enable; # requirement of running this service, so an unconfigured swarm fails to
# build and says which option to set.
ssoConfigured = autheliaCfg.url != null;
# A reader of the store is defined by holding a certificate the store # A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule # accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. # ./glue-matrix-bao-token.nix states in full.
#
# Also asserted rather than gating anything, and that is the ruling this
# module's second delivery route was deleted under: the store exists so that
# ONE secret is placed out of band per host — this certificate — and every
# other secret is read with it. A Grafana host without it has not been given
# its identity yet, which is a thing to say out loud rather than to route
# around by reaching into authelia's tree whenever it happens to be local.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Whether Grafana gets an OIDC block: the swarm-wide question, AND that
# something on this host delivers the secret that block names. The second
# half is not redundant — a block naming a file no unit here writes cannot
# produce a working login either way, and what Grafana does with an
# unreadable `$__file{}` target (refuse to start, or start and fail every
# login) is not worth betting the container on when not emitting the block is
# available and leaves this host as it was.
ssoWired = ssoConfigured && (ssoLocal || haveClientIdentity);
# The delivery that is not a copy: authelia is elsewhere, so the same minted
# plaintext arrives out of the store that the publisher on authelia's host
# wrote it into. Exclusive with the copy unit by construction — two units
# writing one file is a race over which secret Grafana parses.
storeDelivery = ssoWired && !ssoLocal;
autheliaUrl = toString autheliaCfg.url; autheliaUrl = toString autheliaCfg.url;
# Where the plaintext lands inside the container. Under /var/lib rather # Where the plaintext lands inside the container. Under /var/lib rather
@ -123,9 +112,10 @@ let
# intermittent one. # intermittent one.
secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret"; secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret";
# The same file seen from the host, which is where both delivery units write # The same file seen from the host, which is where the delivery unit writes
# it. Spelled once: two units landing a secret at two spellings of one path # it. Spelled once: the unit that writes it and the config that names it are
# is a Grafana that reads whichever of them it was configured with. # a few hundred lines apart, and a Grafana reading a path nothing writes is
# a login that fails with nothing in any log about the file.
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}"; hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
hostSecretDir = builtins.dirOf hostSecretPath; hostSecretDir = builtins.dirOf hostSecretPath;
@ -401,15 +391,16 @@ in
# The secret oneshots as well as grafana itself: each runs before it and # The secret oneshots as well as grafana itself: each runs before it and
# fails in ways grafana then reports only as a login that does not work. # fails in ways grafana then reports only as a login that does not work.
# #
# Each delivery unit is listed only where it exists, the way # Unconditional, because every unit named here now renders in every
# ./hive-matrix.nix lists its own: a unit name that never renders is a # deployment. This list used to be assembled with `lib.optional` per
# journald scrape target matching nothing, which reads as a quiet unit. # delivery route, which was the right shape while there were two — a unit
# name that never renders is a journald scrape target matching nothing,
# which reads as a quiet unit rather than an absent one.
services.hyperhive.swarm.otel.journaldUnits = [ services.hyperhive.swarm.otel.journaldUnits = [
"grafana" "grafana"
"swarm-grafana-secret-key" "swarm-grafana-secret-key"
] "swarm-bao-grafana-oidc"
++ lib.optional ssoLocal "swarm-grafana-oidc-secret" ];
++ lib.optional storeDelivery "swarm-bao-grafana-oidc";
services.hyperhive.swarm.controller.links = [ services.hyperhive.swarm.controller.links = [
{ {
@ -423,36 +414,57 @@ in
# host that runs authelia, and this whole block is gated on the host that # host that runs authelia, and this whole block is gated on the host that
# runs Grafana. ./glue-grafana-oidc-client.nix is where it moved to. # runs Grafana. ./glue-grafana-oidc-client.nix is where it moved to.
warnings = lib.optional (ssoConfigured && !ssoLocal && !haveClientIdentity) '' # Both arms are what used to be a silent gate, and both fire only where
services.hyperhive.swarm.grafana: this swarm has an IdP # Grafana runs — this whole block is under `deploy.grafana.enable`. The
(services.hyperhive.swarm.authelia.url) but authelia is not on this # binding itself is asserted rather than a copy of its formula, the way
host and this host holds no swarm-secret-store identity, so nothing # ./swarm-nats.nix's own `autheliaUrl` arm does it: two spellings of one
can deliver Grafana's OIDC client secret. Grafana will come up with # boolean is two places for a future edit to land in only one.
no OIDC login and no local login form, which is no way in at all.
Name this host's store client leaf in
services.hyperhive.deploy.bao.clientCertFile and
services.hyperhive.deploy.bao.clientKeyFile — the same identity every
other reader of the store uses. See docs/swarm/secrets.md.
'';
assertions = [ assertions = [
{ {
# The URL is what `ssoConfigured` above reads, so with it null this # SSO is not optional for this service, and the reason is a hundred
# host mints and delivers a secret for an OIDC block it then does not # lines below in `auth.disable_login_form = true`: Grafana ships an
# emit — SSO silently absent on the one deployment that has every # `admin`/`admin` account on a public vhost, so the password box is
# other piece of it. Only reachable by enabling authelia and clearing # off whatever the topology. Emitting no OIDC block when the swarm
# its `url`, which is why it is an assertion and not a fallback. # names no IdP therefore produces a container with no way in at all —
assertion = !ssoLocal || ssoConfigured; # a state no log names, since nothing failed. Failing the build and
# naming the option is the only report that reaches anyone.
assertion = ssoConfigured;
message = '' message = ''
services.hyperhive.swarm.grafana requires services.hyperhive.deploy.grafana.enable requires
services.hyperhive.swarm.authelia.url when authelia is enabled. services.hyperhive.swarm.authelia.url — Grafana's only login is SSO,
because its local login form is disabled unconditionally (it ships
an admin/admin account and its vhost is on the public gateway).
Grafana exchanges its authorization code at It defaults to this host's own instance only when this host runs
`''${url}/api/oidc/token` from inside its container, so with the URL authelia. A hive that federates with a swarm sets it explicitly to
null there is no endpoint to name — and a null URL is also how this wherever that provider lives. Grafana exchanges its authorization
module reads "no SSO configured for this swarm", which would leave code at `''${url}/api/oidc/token` from inside its container, so a
Grafana with no login of any kind on the host that mints its secret. null URL leaves no endpoint to name.
'';
}
{
# The other half of one login: the OIDC block names a `$__file{}` that
# `swarm-bao-grafana-oidc.service` below writes, and that unit reads
# the store with this host's client certificate. No certificate, no
# secret, and the same no-way-in Grafana as the arm above.
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.grafana.enable requires this host to hold a
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.clientCertFile
services.hyperhive.deploy.bao.clientKeyFile
Grafana's OIDC client secret is minted by authelia and read out of
the store, on every host that runs Grafana — including the host that
runs authelia. That is one delivery route rather than two, and it is
what the store is for: this certificate is the single credential
placed out of band, and every other secret comes from the store with
it.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
''; '';
} }
]; ];
@ -524,79 +536,24 @@ in
"d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -" "d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -"
]; ];
# The local delivery, and the ONLY thing locality still decides. It runs on # THE delivery unit — one route, in every deployment. The secret authelia
# the HOST because that is the only place both container trees are # minted arrives out of the swarm secret store, which the publisher on
# addressable: they share this host's network namespace, which makes them # authelia's host wrote it into, and that is true whether authelia is a
# feel co-located, but their filesystem roots are separate — Grafana cannot # network away or in the container next door.
# open a path inside authelia's tree however local the port looks.
# #
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container # 🩸 A second unit here used to copy the plaintext out of authelia's host
# refuses to start when a bind source is missing, and this secret does # tree wherever the two were co-located, with the OIDC block gated on
# not exist until authelia's first boot has minted it — so binding it # either route being available. The ruling that deleted it: the store
# would make Grafana wait on a file that waits on a container that starts # exists so a host holds ONE out-of-band secret — its client certificate —
# after it. # and reads everything else with it, so skipping the store when the
systemd.services.swarm-grafana-oidc-secret = lib.mkIf ssoLocal { # producer is local buys a round trip and costs a second delivery path, a
description = "deliver Grafana's OIDC client secret from authelia"; # second way for the file to be wrong, and a gate whose false arm was a
after = [ "container@${autheliaCfg.machine}.service" ]; # Grafana with no login. Recorded in docs/swarm/secrets.md.
requires = [ "container@${autheliaCfg.machine}.service" ];
before = [ "container@${cfg.machine}.service" ];
wantedBy = [ "container@${cfg.machine}.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-grafana-oidc-secret";
# ⚠️ Longer than the wait below, and that is the whole point:
# `DefaultTimeoutStartSec` is 90s, so a 120s bounded wait is killed
# by systemd at 90 — before it can emit the error naming the file it
# waited for. The timeout has to outlive the thing it is timing.
TimeoutStartSec = "180s";
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.oidc.clientId}.secret"}
dst=${lib.escapeShellArg hostSecretPath}
# authelia's container is up, but its first-boot generator may still
# be minting. Bounded wait, then fail: a silent skip here produces a
# Grafana whose only login path dead-ends.
deadline=$(( SECONDS + 120 ))
while [ ! -s "$src" ]; do
if [ "$SECONDS" -ge "$deadline" ]; then
echo "authelia has not minted $src after 120s" >&2
exit 1
fi
sleep 2
done
# Owned by Grafana's own uid, unlike the matrix sibling which lands
# root-owned: tuwunel's secret is read by `LoadCredential` as root
# before the sandbox exists, whereas Grafana expands `$__file{}`
# itself, as itself, while parsing its config. These containers set
# no `privateUsers`, so the host uid is the container uid, and both
# sides take it from the same static NixOS id.
#
# Group is root, not grafana, and that is forced rather than chosen:
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
# — the group's gid is allocated at activation inside the container,
# so the host cannot know it at eval time. Harmless here because 0400
# grants the group nothing; if this mode ever widens, the gid has to
# be discovered at runtime rather than assumed.
install -D -m 0400 -o ${toString config.ids.uids.grafana} -g 0 "$src" "$dst"
'';
};
# The same secret, the same destination, for the deployment where authelia
# is NOT here: it arrives out of the swarm secret store, which the
# publisher on authelia's host wrote it into. Nothing above changes — this
# is the second source for one file, never a second file.
# #
# Shaped after ./glue-queue-agent-credential.nix rather than after the copy # Shaped after ./glue-queue-agent-credential.nix, because it is a store
# above, because it is a store reader and those are the store's readers: a # reader and this is what the store's readers do: a cert login that fails
# cert login that fails LOUDLY, since every state it fails on is one a # LOUDLY, since every state it fails on is one a retry fixes, then a read
# retry fixes, then a read that degrades QUIETLY, since no retry turns "no # that degrades QUIETLY, since no retry turns "no value there" into a value.
# value there" into a value.
# #
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's. # ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs on the authelia host and authelia mints on its first # The publisher runs on the authelia host and authelia mints on its first
@ -606,11 +563,10 @@ in
# log anyone in into one whose token exchange is refused. # log anyone in into one whose token exchange is refused.
# #
# ⇒ So the absence surfaces at Grafana rather than here, and this unit's job # ⇒ So the absence surfaces at Grafana rather than here, and this unit's job
# is to leave the journal line that says which of the three it was. That is # is to leave the journal line that says which of the three it was: the
# the same division the copy unit above makes by failing rather than # secret not arriving is a Grafana nobody can log into, and a named cause is
# skipping: the secret not arriving is a Grafana nobody can log into, and a # the only thing separating that from a mystery.
# named cause is the only thing separating that from a mystery. systemd.services.swarm-bao-grafana-oidc = {
systemd.services.swarm-bao-grafana-oidc = lib.mkIf storeDelivery {
description = "fetch Grafana's OIDC client secret from the swarm secret store"; description = "fetch Grafana's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs. # Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering # `Requires=` on an absent unit fails the job outright, so the ordering
@ -708,8 +664,19 @@ in
# bash's own, so the plaintext never becomes an argument in /proc the # bash's own, so the plaintext never becomes an argument in /proc the
# way `install <<<"$secret"` or an `echo` from `path` would. # way `install <<<"$secret"` or an `echo` from `path` would.
# #
# Same uid, group and mode as the copy above, for the reasons stated # Owned by Grafana's own uid, unlike the matrix sibling which lands
# there — this is the same file arriving by a different route. # root-owned: tuwunel's secret is read by `LoadCredential` as root
# before the sandbox exists, whereas Grafana expands `$__file{}`
# itself, as itself, while parsing its config. This container sets no
# `privateUsers`, so the host uid is the container uid, and both sides
# take it from the same static NixOS id.
#
# Group is root, not grafana, and that is forced rather than chosen:
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
# — the group's gid is allocated at activation inside the container, so
# the host cannot know it at eval time. Harmless here because 0400
# grants the group nothing; if this mode ever widens, the gid has to be
# discovered at runtime rather than assumed.
install -d -m 0755 ${lib.escapeShellArg hostSecretDir} install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
umask 077 umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath} printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath}
@ -913,13 +880,21 @@ in
# public gateway, so SSO is the only way in — not merely the # public gateway, so SSO is the only way in — not merely the
# preferred one. # preferred one.
# #
# Deliberately NOT `ssoLocal`: whether a password box sits on a # Not conditional on anything, and the block below is not
# public login page cannot depend on which host happens to run # either, which is what keeps the two from ever disagreeing.
# authelia. The OIDC block below now answers the same swarm-wide # Whether a password box sits on a public login page cannot
# question, so the two no longer disagree. # depend on which host happens to run authelia — and it used to,
# in a module where the OIDC block was gated and this line was
# not.
auth.disable_login_form = true; auth.disable_login_form = true;
}
// lib.optionalAttrs ssoWired { # Emitted in every deployment, because SSO is a requirement of
# running this service rather than a feature of some topologies:
# the assertions above refuse to build a Grafana that lacks an
# IdP to point at or the identity to fetch its secret with. A
# `lib.optionalAttrs` here would be the silent half of that pair,
# dropping the only login this container has whenever a value
# went null.
"auth.generic_oauth" = { "auth.generic_oauth" = {
enabled = true; enabled = true;
name = "HyperHive"; name = "HyperHive";
@ -930,6 +905,9 @@ in
# `database.password`, `security.admin_password` and # `database.password`, `security.admin_password` and
# datasource `secureJsonData`), so nothing but this comment # datasource `secureJsonData`), so nothing but this comment
# stands between a literal and the store. # stands between a literal and the store.
#
# `swarm-bao-grafana-oidc.service` above is what writes this
# path, on every host that runs Grafana.
client_secret = "$__file{${secretPath}}"; client_secret = "$__file{${secretPath}}";
scopes = "openid profile email groups"; scopes = "openid profile email groups";
auth_url = "${autheliaUrl}/api/oidc/authorization"; auth_url = "${autheliaUrl}/api/oidc/authorization";

23
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,6 +1,13 @@
# The unit that copies authelia's minted OIDC client secrets into the swarm's # The unit that copies authelia's minted OIDC client secrets into the swarm's
# secret store, so whoever needs one without hosting authelia can read it: a # secret store, so whoever needs one can read it there: a hive its agents'
# hive its agents' credential, a swarm service its own. # credential, a swarm service its own.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
# reader fetches it from the store in every deployment — ./swarm-grafana.nix
# states the ruling that made that the only route. Publishing "only when the
# reader is elsewhere" would be a second shape of this unit, gated on a fact
# about another host, to save a round trip on the one host that can afford it.
# #
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a # ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says # separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
@ -70,7 +77,8 @@ in
description = '' description = ''
Publish the OIDC client secrets this host mints into the swarm's Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service elsewhere can read its own. agents' credential and a swarm service can read its own — from
wherever it runs, this host included.
Defaults to whether this host mints them, which is the only half of Defaults to whether this host mints them, which is the only half of
the question that is a property of *this* host. the question that is a property of *this* host.
@ -82,10 +90,11 @@ in
publishing nothing. Whether the wiring is complete is the client publishing nothing. Whether the wiring is complete is the client
identity's job (see `baoClientCertFile`), not this option's. identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every non-co-located hive without a delivery Turning it off leaves every hive but this one without its agents'
path, which is the state this exists to end — so the honest reason credential, and the swarm's Grafana without any login at all — its
to set it false is a deployment delivering those secrets by some secret has exactly one route and this is the producer's end of it. So
other mechanism it owns. the honest reason to set it false is a deployment delivering those
secrets by some other mechanism it owns.
''; '';
}; };

155
nix/module-eval.nix
View file

@ -223,18 +223,22 @@ let
swarm.grafana.package = pkgs.emptyDirectory; swarm.grafana.package = pkgs.emptyDirectory;
}; };
# The metrics UI beside the IdP, which is the deployment whose secret was # The metrics UI beside the IdP. It reads its secret out of the store like
# already delivered: a host copy out of authelia's own tree. # every other Grafana host, so it needs a store identity like every other
# Grafana host — the cert pair here is not scenery, it is the arm that would
# have caught the deleted co-located copy unit coming back.
grafanaWithAuthelia = hive { grafanaWithAuthelia = hive {
deploy.grafana.enable = true; deploy.grafana.enable = true;
deploy.grafana.plugins = [ ]; deploy.grafana.plugins = [ ];
deploy.grafana.package = pkgs.emptyDirectory; deploy.grafana.package = pkgs.emptyDirectory;
deploy.authelia.enable = true; deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
}; };
# The same UI with the IdP on ANOTHER host and a store leaf placed by hand — # The same UI with the IdP on ANOTHER host and a store leaf placed by hand.
# the deployment that had no delivery path at all. Knowing an IdP is not # Knowing an IdP is not running one: `swarm.authelia.url` is what says this
# running one: `swarm.authelia.url` is what says this swarm has SSO, and # swarm has SSO, and nothing about this host does. Identical to the fixture
# nothing about this host does. # above in everything the delivery path reads, which is the point.
grafanaRemoteAuthelia = hive { grafanaRemoteAuthelia = hive {
deploy.grafana.enable = true; deploy.grafana.enable = true;
deploy.grafana.plugins = [ ]; deploy.grafana.plugins = [ ];
@ -243,16 +247,48 @@ let
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
}; };
# The same again with the store identity taken away, which is the one shape # A Grafana host holding no store identity. This used to be the shape the
# that can deliver this secret by NO route. It separates "the swarm has an # module went QUIET on — no OIDC block, a warning, and a container whose
# IdP" from "something here can fetch what it minted", and it is what keeps # login form is off regardless, so no way in and nothing failed. It is kept
# Grafana's config from naming a file nothing writes. # rather than deleted because the shape is still reachable by an operator;
grafanaRemoteAutheliaNoIdentity = hive { # what changed is the deliverable, from a warning nothing reads back to a
# refusal naming the two options to set. Only the identity is missing, so an
# arm below can name which refusal fired.
grafanaNoIdentity = hive {
deploy.grafana.enable = true; deploy.grafana.enable = true;
deploy.grafana.plugins = [ ]; deploy.grafana.plugins = [ ];
deploy.grafana.package = pkgs.emptyDirectory; deploy.grafana.package = pkgs.emptyDirectory;
swarm.authelia.url = "https://auth.example.invalid"; swarm.authelia.url = "https://auth.example.invalid";
}; };
# The mirror image: the identity is placed, and the swarm names no IdP. The
# other half of "SSO must always be configured", and isolated the same way —
# exactly one thing wrong, so the arm reads one refusal.
grafanaNoSso = hive {
deploy.grafana.enable = true;
deploy.grafana.plugins = [ ];
deploy.grafana.package = pkgs.emptyDirectory;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# Did ./host-modules/swarm-grafana.nix refuse this host, and for which of its
# two reasons. An assertion is a config VALUE until something forces it —
# `.config` never throws — so a fixture in a state the module refuses is
# evaluable and the refusal is readable as data. That is what lets a case
# check that a misconfiguration is REPORTED, rather than only that it is not
# silently accepted.
#
# Matched on the option name the message names, not on its prose, so the
# wording stays rewordable: the option name is the part an operator has to
# act on, and a message that stopped naming it would be the actual defect.
grafanaRefusedFor =
m: option:
lib.any (
a:
!a.assertion
&& lib.hasInfix "services.hyperhive.deploy.grafana.enable requires" a.message
&& lib.hasInfix option a.message
) m.assertions;
# The first slice to leave options on BOTH sides of the split, so the # The first slice to leave options on BOTH sides of the split, so the
# fixture sets all three of them through the paths an existing config uses: # fixture sets all three of them through the paths an existing config uses:
@ -871,55 +907,80 @@ let
# The absence class this whole file is for, and the reported defect in one # The absence class this whole file is for, and the reported defect in one
# arm: the OIDC block hung off "authelia is on this host", so the split # arm: the OIDC block hung off "authelia is on this host", so the split
# deployment got a Grafana with no SSO settings and no login form — no way # deployment got a Grafana with no SSO settings and no login form — no way
# in at all. The question it asks now is whether the SWARM has an IdP. # in at all. The block is emitted in every deployment now, so the negative
name = "grafana is wired for SSO against an IdP on another host"; # arm is not "no block elsewhere" but "the two do not name the same IdP":
# each host's block has to point at the URL the SWARM names, and a block
# built from `deploy.authelia` rather than `swarm.authelia.url` would pass
# a presence check on both fixtures while sending one of them nowhere.
name = "grafana's OIDC block names the swarm's IdP, wherever that IdP runs";
ok = ok =
let let
s = oauth = m: m.containers.swarm-grafana.config.services.grafana.settings."auth.generic_oauth";
grafanaRemoteAuthelia.containers.swarm-grafana.config.services.grafana.settings."auth.generic_oauth"; remote = oauth grafanaRemoteAuthelia;
local = oauth grafanaWithAuthelia;
in in
s.enabled && lib.hasInfix "https://auth.example.invalid/api/oidc/token" s.token_url; remote.enabled
&& lib.hasInfix "https://auth.example.invalid/api/oidc/token" remote.token_url
&& local.enabled
&& lib.hasInfix "https://auth.t.local/api/oidc/token" local.token_url
&& !(lib.hasInfix "auth.example.invalid" local.token_url);
} }
{ {
# The control for the arm above, and the reason the gate is a conjunction: # 🩸 The arm that guards the ruling this slice landed under. There is ONE
# a block naming a file no unit on this host writes cannot produce a # delivery route: the store reader, on every host that runs Grafana. The
# working login, so it is emitted only where a delivery route exists — # negative names the deleted unit rather than a generic absence, because
# leaving a host with neither exactly as it was rather than pointing # the way this regresses is someone re-adding the co-located copy as an
# Grafana's config at a secret that never arrives. # optimisation — a second writer of one path, and a second shape of "the
name = "grafana emits no OIDC block when nothing on this host can deliver the secret"; # secret is wrong" to debug.
ok = name = "grafana's OIDC secret has exactly one delivery unit, the store reader, in both topologies";
!(
grafanaRemoteAutheliaNoIdentity.containers.swarm-grafana.config.services.grafana.settings
? "auth.generic_oauth"
);
}
{
# The other half of the arm above, and the only deliverable that shape
# has: silence there is a Grafana nobody can log into for a reason no log
# names. A warning is read back by nothing, so an unevaluated one is a
# warning nobody sees — the same reason the peers fixture exists.
name = "the undeliverable-secret shape warns instead of going quiet";
ok =
lib.any (
w: lib.hasInfix "no swarm-secret-store identity" w
) grafanaRemoteAutheliaNoIdentity.warnings
&& !(lib.any (w: lib.hasInfix "no swarm-secret-store identity" w) grafanaRemoteAuthelia.warnings);
}
{
# Exactly one delivery unit per deployment, and which one is the only
# thing locality decides. Two units writing one path is a race over which
# secret Grafana parses; zero is the defect this pair replaced.
name = "grafana's OIDC secret has one delivery unit, chosen by where authelia is";
ok = ok =
let let
local = grafanaWithAuthelia.systemd.services; local = grafanaWithAuthelia.systemd.services;
remote = grafanaRemoteAuthelia.systemd.services; remote = grafanaRemoteAuthelia.systemd.services;
in in
local ? swarm-grafana-oidc-secret local ? swarm-bao-grafana-oidc
&& !(local ? swarm-bao-grafana-oidc)
&& remote ? swarm-bao-grafana-oidc && remote ? swarm-bao-grafana-oidc
&& !(local ? swarm-grafana-oidc-secret)
&& !(remote ? swarm-grafana-oidc-secret); && !(remote ? swarm-grafana-oidc-secret);
} }
{
# What the deleted warning became. The shape is unchanged — a Grafana host
# holding no store leaf — but silence there is a container nobody can log
# into for a reason no log names, and a warning is read back by nothing.
# The second arm is what makes this a refusal about the IDENTITY: this
# fixture names an IdP, so a message about `swarm.authelia.url` here would
# mean the two assertions had been collapsed into one conjunction.
name = "a grafana host with no store identity is refused, naming the options to set";
ok =
grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile"
&& grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile"
&& !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url");
}
{
# "SSO must always be configured", as an eval-time refusal rather than a
# gate. A null URL used to drop the OIDC block silently, and
# `disable_login_form` is unconditional a hundred lines below it, so that
# combination produced a Grafana with no SSO and no password box — an
# outage whose cause is a boolean that evaluated to false at build time
# and left no trace. Same isolation as the arm above, mirrored.
name = "a grafana host in a swarm with no IdP is refused, naming swarm.authelia.url";
ok =
grafanaRefusedFor grafanaNoSso "services.hyperhive.swarm.authelia.url"
&& !(grafanaRefusedFor grafanaNoSso "deploy.bao.clientCertFile");
}
{
# Without this the two arms above prove nothing: a refusal that fires on
# every host is not a check, and both of these are hosts a swarm is
# expected to have. Read through the same helper, so a message that
# stopped naming its option would fail the arms above rather than pass
# this one by accident.
name = "neither grafana refusal fires on a correctly configured host, co-located or not";
ok =
!(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url")
&& !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile")
&& !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url")
&& !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile");
}
{ {
# Same 403-not-a-miss reason as the matrix and queue arms below: the # Same 403-not-a-miss reason as the matrix and queue arms below: the
# reader's grant covers the `services` prefix, so a path outside it is # reader's grant covers the `services` prefix, so a path outside it is