vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
hyperhive/nix/host-modules/swarm-grafana.nix
atlas 4aa982cc2a swarm-grafana: deliver the OIDC client secret through the secret store 
Grafana's OIDC client secret only existed where authelia did. One
`ssoLocal` gate — `grafana.enable && authelia.enable` — decided the
client registration, the minted secret's delivery and the whole
`auth.generic_oauth` block, so a swarm whose authelia runs on another
host got Grafana with no SSO wiring at all. The local login form is
disabled unconditionally, so that is no way in.

Split the one gate into the two questions it was conflating:

- `ssoConfigured` — does this SWARM have an identity provider
  (`swarm.authelia.url`, which is swarm-wide and whose own description
  makes null mean "no SSO configured"). With a delivery route present
  this is what emits Grafana's OIDC block.
- `ssoLocal` — is authelia on THIS host, now spelled as the forge and
  matrix modules spell it. It decides only which unit delivers the
  secret.

Where authelia is elsewhere, `swarm-bao-grafana-oidc.service` reads the
secret from the swarm secret store, shaped after
glue-queue-agent-credential.nix: cert login fails loudly because a retry
fixes every state it fails on, the read degrades quietly because no
retry turns "no value there" into a value, and nothing writes a
stand-in. The producer is the publisher that already runs on authelia's
host, which gains the swarm's service clients beside the per-hive ones
at `swarm/services/<id>/oidc/client` — with the write grant in
swarm-bao.nix and the hive read grant in `policy::render` to match.

Registration moved to glue-grafana-oidc-client.nix. It has to be
declared where authelia's config is rendered, and swarm-grafana.nix's
config block hangs off this host running Grafana.

Two judgement calls stated rather than buried: a hive's read policy now
grants the whole `services` prefix, because a service's path names the
service and nothing swarm-wide records which hive runs it (cost recorded
in docs/trust-boundary/security.md); and the client is registered on any
authelia host, because no swarm-wide "this swarm has a Grafana" fact
exists to gate it on.

Refs #4234

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 19:57:28 +02:00

989 lines
45 KiB
Nix
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# The swarm's metrics UI: one Grafana for the whole swarm, in a
# `swarm-grafana` nixos-container next to the VictoriaMetrics store it reads.
#
# Two containers rather than one, on the operator's call: Grafana can be
# restarted, reconfigured or broken without taking the TSDB down with it.
# They are a pair, not a unit.
#
# Dashboards are provisioned from files in this repo, deliberately not created
# in the UI: the store path is immutable, so Grafana is told not to accept UI
# edits it would lose on the next rebuild.
{
pkgs,
lib,
config,
...
}:
let
cfg = config.services.hyperhive.swarm.grafana;
deployCfg = config.services.hyperhive.deploy;
networkCfg = config.services.hyperhive.network;
hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway;
autheliaCfg = hyperhiveCfg.swarm.authelia;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
vmCfg = hyperhiveCfg.swarm.victoriametrics;
vlCfg = hyperhiveCfg.swarm.victorialogs;
swarmDomain = hyperhiveCfg.swarm.domain;
caTrust = import ./lib/hive-ca-trust.nix {
inherit lib gatewayCfg;
tlsCfg = deployCfg.hive-controller.tls;
};
# Spelled ONCE. The provisioned datasource declares it and every panel in the
# provisioned dashboards references it; a second literal would be free to
# drift, and the failure is a dashboard that renders with empty panels rather
# than an error.
datasourceUid = "swarm-victoriametrics";
# The logs store's own uid. Separate from the metrics one and spelled once for
# the same reason: a dashboard panel names it, and a second literal would be
# free to drift into a panel that renders empty rather than erroring.
logsDatasourceUid = "swarm-victorialogs";
# The shipped dashboards carry `@datasourceUid@` / `@logsDatasourceUid@` where
# a real deployment needs the uids above. They are substituted here rather
# than committed with the literals so the single binding stays single.
renderDashboard =
name:
pkgs.writeText name (
builtins.replaceStrings
[ "@datasourceUid@" "@logsDatasourceUid@" ]
[ datasourceUid logsDatasourceUid ]
(builtins.readFile (./swarm-grafana/dashboards + "/${name}"))
);
# Grafana's file provider wants a DIRECTORY to scan, so the rendered files are
# collected into one.
dashboardDir = pkgs.linkFarm "hyperhive-grafana-dashboards" (
map
(name: {
inherit name;
path = renderDashboard name;
})
[
"agents.json"
"authelia.json"
"bao.json"
"claude-usage.json"
"forge.json"
"logstore.json"
"metricstore.json"
"queue.json"
]
);
# Total on a null swarm domain for the same reason every sibling module is:
# the required-domain assertion in hive-network.nix should be what an
# operator sees, not a coercion error from here.
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
# Is SSO configured for this SWARM. Swarm-wide by construction — `swarm.*` is
# identical on every host — and the option's own description is what makes
# this the right question to ask: a null URL means "no SSO configured".
ssoConfigured = autheliaCfg.url != null;
# Is authelia on THIS host, spelled exactly as the forge and matrix modules
# spell it. All locality decides is where the minted plaintext is copied
# FROM: authelia's own tree here, the swarm secret store otherwise.
#
# 🩸 This used to be `grafana.enable && authelia.enable` and was the ONLY
# gate: registration, the mint's delivery and the OIDC block all hung off
# it, so a swarm whose authelia ran elsewhere got Grafana with no SSO wiring
# — and the login form is disabled whatever happens, so with nothing else.
ssoLocal = deployCfg.authelia.enable;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Whether Grafana gets an OIDC block: the swarm-wide question, AND that
# something on this host delivers the secret that block names. The second
# half is not redundant — a block naming a file no unit here writes cannot
# produce a working login either way, and what Grafana does with an
# unreadable `$__file{}` target (refuse to start, or start and fail every
# login) is not worth betting the container on when not emitting the block is
# available and leaves this host as it was.
ssoWired = ssoConfigured && (ssoLocal || haveClientIdentity);
# The delivery that is not a copy: authelia is elsewhere, so the same minted
# plaintext arrives out of the store that the publisher on authelia's host
# wrote it into. Exclusive with the copy unit by construction — two units
# writing one file is a race over which secret Grafana parses.
storeDelivery = ssoWired && !ssoLocal;
autheliaUrl = toString autheliaCfg.url;
# Where the plaintext lands inside the container. Under /var/lib rather
# than /run: Grafana may start before the delivery unit on a later boot,
# and a secret that evaporates on reboot turns a working login into an
# intermittent one.
secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret";
# The same file seen from the host, which is where both delivery units write
# it. Spelled once: two units landing a secret at two spellings of one path
# is a Grafana that reads whichever of them it was configured with.
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
hostSecretDir = builtins.dirOf hostSecretPath;
# Where the publisher on authelia's host leaves this client's secret. The
# `services` segment is `swarm-secret-client`'s `path::Kind::Service` — a
# swarm service's client is registered once for the whole swarm, so its
# secret is one value and not one per hive. Both ends compose this from the
# same swarm-wide client id, so a rename cannot leave one of them behind.
#
# ⚠️ The prefix is not decoration: it is what this host's read grant covers,
# so a path outside it answers 403 rather than a miss, however correct it
# looks.
storeSecretPath = "secret/swarm/services/${cfg.oidc.clientId}/oidc/client";
# Grafana's own datasource-encryption key. Generated in-container (see the
# unit below) because nothing outside the container ever reads it — unlike
# the OIDC secret above, whose other reader is authelia's container.
secretKeyPath = "/var/lib/grafana-secret/secret_key";
socketPath = "${deployCfg.grafana.socketDir}/grafana.sock";
# Both static NixOS ids, and both checked rather than assumed: nginx has
# `ids.gids.nginx = 60`, Grafana has `ids.uids.grafana = 196` — but there is
# deliberately no `ids.gids.grafana`, which is why the OIDC-secret unit
# below lands its file root-group at 0400 instead of reaching for one.
nginxGid = config.ids.gids.nginx;
grafanaUid = config.ids.uids.grafana;
in
{
# `enable` moved to `services.hyperhive.deploy.grafana.enable` — see
# ./deploy.nix. Whether this host runs the swarm's Grafana is a
# deployment decision, and `swarm.*` has to be identical on every host.
# What stays here is what the service IS to every hive: the name it is
# served under, the port its `/metrics` is re-served on, and the OIDC
# client it is registered as. What the host running it decides — which
# build it runs, where its datasources point, which plugins are in its
# store path, and the socket directory it shares with nginx — is below,
# under `deploy.grafana`.
options.services.hyperhive.swarm.grafana = {
machine = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-grafana";
description = ''
Container name. Read-only: the name appears in host paths and in
`machinectl`, so it is a fact other modules may read rather than a
knob.
'';
};
domain = lib.mkOption {
type = lib.types.str;
default = "grafana.${domainBase}";
defaultText = lib.literalExpression ''"grafana.''${services.hyperhive.swarm.domain}"'';
description = ''
Name the gateway serves this on. A sibling of the swarm's other
service names, so the swarm-services sub-CA can issue for it — see
`hive-tls.nix` for why a service name being a sibling rather than a
child decides which CA may sign it.
⚠️ Changing this changes the OAuth redirect URI, which authelia
matches exactly. Both sides move together because both derive from
this option; an operator who pins one by hand breaks the login.
'';
};
metricsPort = lib.mkOption {
type = lib.types.port;
default = 9095;
description = ''
Loopback port on which the gateway's nginx re-serves Grafana's
`/metrics`, and nothing else, so the swarm's collector can scrape it.
⚠️ **This is nginx's port, not Grafana's.** Grafana still claims none —
see {option}`services.hyperhive.deploy.grafana.socketDir` for why that
matters. A prometheus scrape target is a `host:port`, and it cannot
address a unix socket; rather than undo the socket decision, the one
endpoint a scraper needs gets a listener of its own.
Bound to loopback and unauthenticated, which is the same posture every
other entry in
{option}`services.hyperhive.swarm.otel.scrapeTargets` has: those
targets are trusted by *proximity* rather than by credential.
Deliberately **not** the published `grafana.<domain>` vhost, which
would put an authorization decision in front of a scrape.
The number itself is arbitrary and free today;
`state/eval-port-collisions.sh` is what keeps it that way, since a
second claim on a port in this shared namespace produces no bind error
and nothing in any log.
'';
};
oidc = {
clientId = lib.mkOption {
type = lib.types.str;
default = "swarm-grafana";
description = ''
The authelia OIDC client id. Names the application rather than
the protocol, per the convention in
{option}`services.hyperhive.swarm.authelia.oidc.clients`.
'';
};
redirectUri = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "https://${cfg.domain}/login/generic_oauth";
defaultText = lib.literalExpression ''"https://''${services.hyperhive.swarm.grafana.domain}/login/generic_oauth"'';
description = ''
OAuth callback authelia sends the browser back to, and the URI it
matches **exactly**.
Read-only, like {option}`services.hyperhive.swarm.grafana.machine`
and for the same reason: Grafana derives it from its own
`root_url` (`<root_url>/login/generic_oauth`), so it is a fact
other modules may read rather than a knob. The glue that registers
this client wherever authelia runs reads it from here instead of
restating the format — a second spelling of it is a silently
rejected login.
'';
};
role = lib.mkOption {
type = lib.types.enum [
"Viewer"
"Editor"
"Admin"
];
default = "Admin";
example = "Editor";
description = ''
Grafana org role every SSO user is assigned.
`Admin` by default, and that is a considered default rather than
a permissive one: the login form is disabled whenever SSO is
configured, so this is the *only* way anyone reaches Grafana —
a `Viewer` default would produce a swarm nobody can administer.
Passing authelia already means being an operator of this swarm;
its user store is the small, `swarmctl`-managed one.
Lower it if a swarm ever grows read-only operators, which is a
one-line change here.
'';
};
};
};
# What stays above is what the service IS to every hive. What the host
# running it decides is here: which build it runs, where its datasources
# point, which plugins sit in its store path, and the directory it shares a
# socket with nginx through. `enable` already lives in ./deploy.nix, which
# also carries the renames.
#
# ⚠️ Both datasource URLs are wiring and still move. A URL's scope is the
# scope of what it ADDRESSES, not the fact that it is a URL: both stores
# bind loopback, so these can only ever mean "on this host".
options.services.hyperhive.deploy.grafana = {
package = lib.mkOption {
type = lib.types.package;
default = pkgs.grafana;
defaultText = lib.literalExpression "pkgs.grafana";
description = "Grafana package to run.";
};
socketDir = lib.mkOption {
type = lib.types.str;
default = "/run/swarm-grafana";
description = ''
Directory holding the unix socket Grafana serves on, shared between
the host (where nginx runs) and the container (where Grafana runs).
⚠️ **Grafana takes no TCP port at all, and that is the point.** Every
swarm service container shares the host's network namespace, so a
port is a swarm-wide resource that two modules can silently both
claim — which is exactly what happened: Grafana defaulted to
upstream's 3000, so does the forge, and `grafana.<swarm-domain>`
served the forge with no bind error and nothing in any log.
A socket has a path, and a path collision is a build-time conflict
rather than a runtime coin toss.
'';
};
datasourceUrl = lib.mkOption {
type = lib.types.str;
default = "http://127.0.0.1:${toString vmCfg.port}";
defaultText = lib.literalExpression ''"http://127.0.0.1:''${toString services.hyperhive.swarm.victoriametrics.port}"'';
description = ''
Where the provisioned datasource points. Defaults to the metrics
store on this host, which is the only place it can be: that store
binds loopback, so a Grafana somewhere else could not reach it
anyway. Set explicitly if a deployment fronts VictoriaMetrics with
something that does listen wider.
'';
};
logsDatasourceUrl = lib.mkOption {
type = lib.types.str;
default = "http://127.0.0.1:${toString vlCfg.port}";
defaultText = lib.literalExpression ''"http://127.0.0.1:''${toString services.hyperhive.swarm.victorialogs.port}"'';
description = ''
Where the provisioned logs datasource points. Same reasoning as
{option}`services.hyperhive.deploy.grafana.datasourceUrl`: the store
binds loopback, so a Grafana elsewhere could not reach it anyway.
Provisioned unconditionally, like the metrics datasource — the store
being off is a deployment choice rather than a reason to withhold the
connection, and an operator whose logs live elsewhere sets this.
'';
};
plugins = lib.mkOption {
type = lib.types.listOf lib.types.package;
default = [
pkgs.grafanaPlugins.grafana-metricsdrilldown-app
# Reading the swarm's logs needs the datasource that speaks LogsQL.
# Nothing is paired with it to browse them: Explore is that browser.
pkgs.grafanaPlugins.victoriametrics-logs-datasource
];
defaultText = lib.literalExpression ''
[
pkgs.grafanaPlugins.grafana-metricsdrilldown-app
pkgs.grafanaPlugins.victoriametrics-logs-datasource
]'';
example = lib.literalExpression "[ pkgs.grafanaPlugins.grafana-piechart-panel ]";
description = ''
Grafana plugins to install, as packages. Declarative rather than
installed through the UI, which is the only shape that works here:
plugin management is **server-admin** scoped, and on an SSO hive
nobody holds that role — `users.auto_assign_org_role` grants an
*org* role, and the built-in local admin that does hold server
admin cannot log in because the login form is disabled whenever
SSO is configured.
That is a deliberate pair of decisions rather than an oversight,
and this option is the way through it: plugins live in the store
and in git, so they survive a container rebuild and a state reset,
and the container needs no runtime egress to grafana.com.
⚠️ This list is the ONLY way a plugin reaches this Grafana.
Setting it at all disables upstream's `preinstall` mechanism (see
the `declarativePlugins` comment below), which is what normally
fetches the Drilldown apps on first start — so a plugin left off
this list is absent, not merely un-pinned.
Metrics Drilldown is on by default because this swarm's Grafana
exists to read VictoriaMetrics, and the queryless metrics browser
is the part of that people actually reach for.
Its Logs / Traces / Profiles siblings are deliberately NOT here,
and for two different reasons. Traces and Profiles front Tempo
and Pyroscope, which this swarm does not run — a UI for a backend
that does not answer. Logs Drilldown is the subtler one: this
swarm *does* run a log store, so that argument no longer applies
to it, but the app is Loki-only and no configuration here changes
that. Its volume views call `/loki/api/v1/index/volume`, which
VictoriaLogs does not implement and answers "unsupported path
requested". Logs are browsed through Explore instead.
'';
};
};
config = lib.mkIf (hyperhiveCfg.enable && deployCfg.grafana.enable) {
# The gateway name and the quick-link, both inside `deploy.grafana` — that
# guard is the load-bearing part. Every hive in a swarm may know this UI
# exists, but only the host that RUNS it may claim the name; a client
# hive declaring the vhost would answer for a service it does not have.
services.hyperhive.gateway.localNames = [ cfg.domain ];
# The secret oneshots as well as grafana itself: each runs before it and
# fails in ways grafana then reports only as a login that does not work.
#
# Each delivery unit is listed only where it exists, the way
# ./hive-matrix.nix lists its own: a unit name that never renders is a
# journald scrape target matching nothing, which reads as a quiet unit.
services.hyperhive.swarm.otel.journaldUnits = [
"grafana"
"swarm-grafana-secret-key"
]
++ lib.optional ssoLocal "swarm-grafana-oidc-secret"
++ lib.optional storeDelivery "swarm-bao-grafana-oidc";
services.hyperhive.swarm.controller.links = [
{
label = "Grafana";
icon = "📊";
url = "https://${cfg.domain}/";
}
];
# Registering the client is NOT here any more: it has to happen on the
# host that runs authelia, and this whole block is gated on the host that
# runs Grafana. ./glue-grafana-oidc-client.nix is where it moved to.
warnings = lib.optional (ssoConfigured && !ssoLocal && !haveClientIdentity) ''
services.hyperhive.swarm.grafana: this swarm has an IdP
(services.hyperhive.swarm.authelia.url) but authelia is not on this
host and this host holds no swarm-secret-store identity, so nothing
can deliver Grafana's OIDC client secret. Grafana will come up with
no OIDC login and no local login form, which is no way in at all.
Name this host's store client leaf in
services.hyperhive.deploy.bao.clientCertFile and
services.hyperhive.deploy.bao.clientKeyFile — the same identity every
other reader of the store uses. See docs/swarm/secrets.md.
'';
assertions = [
{
# The URL is what `ssoConfigured` above reads, so with it null this
# host mints and delivers a secret for an OIDC block it then does not
# emit — SSO silently absent on the one deployment that has every
# other piece of it. Only reachable by enabling authelia and clearing
# its `url`, which is why it is an assertion and not a fallback.
assertion = !ssoLocal || ssoConfigured;
message = ''
services.hyperhive.swarm.grafana requires
services.hyperhive.swarm.authelia.url when authelia is enabled.
Grafana exchanges its authorization code at
`''${url}/api/oidc/token` from inside its container, so with the URL
null there is no endpoint to name — and a null URL is also how this
module reads "no SSO configured for this swarm", which would leave
Grafana with no login of any kind on the host that mints its secret.
'';
}
];
# Websockets: Grafana Live streams panel updates over one, and without
# the upgrade headers dashboards load and then never refresh — which
# reads as stale data rather than as a proxy fault.
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
listen = gatewayCfg.lib.listen;
extraConfig = gatewayCfg.lib.securityHeaders;
locations."/" = {
proxyPass = "http://unix:${socketPath}:/";
proxyWebsockets = true;
extraConfig = ''
# Grafana builds its OAuth redirect from the ORIGINAL request.
# Without these every request looks like it arrived at 127.0.0.1
# over plain http, and the redirect sent to authelia names a
# host the browser cannot reach.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
'';
};
};
# The scrape listener: Grafana's `/metrics` and nothing else, on loopback.
#
# No `serverName` and no TLS on purpose — this listener exists for exactly
# one client on the same host, so there is no name to match and nothing to
# verify. It is the only vhost here that is not reachable from outside the
# machine, and that is what lets it skip the authorization the published
# one carries.
#
# ⚠️ `= /metrics` is an EXACT match, not a prefix: it is the difference
# between exposing one endpoint and re-serving the whole of Grafana on an
# unauthenticated port. Widening it would be a silent authorization bypass,
# since nothing else about this block would have to change.
services.nginx.virtualHosts."grafana-metrics" = {
listen = [
{
addr = "127.0.0.1";
port = cfg.metricsPort;
}
];
locations."= /metrics".proxyPass = "http://unix:${socketPath}:/metrics";
};
# Declared here rather than in the collector's module, per that option's
# rule: an entry exists where the service that named it runs.
services.hyperhive.swarm.otel.scrapeTargets =
lib.mkIf config.services.hyperhive.deploy.swarm-otel.enable
{
grafana = "127.0.0.1:${toString cfg.metricsPort}";
};
# Order the container after the host CA service so the bind source below
# exists before nspawn sets the mount up.
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
# The socket directory, created host-side before the container starts.
# nixos-container refuses to start when a bind source is missing, so this
# rule is a prerequisite of the mount rather than a tidiness measure.
#
# Owned by Grafana's uid so it can create the socket, group nginx so the
# gateway can traverse; `0750` keeps everything else out. Numeric ids
# because the host has no `grafana` account — Grafana lives in the
# container, and only the number crosses that boundary.
systemd.tmpfiles.rules = [
"d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -"
];
# The local delivery, and the ONLY thing locality still decides. It runs on
# the HOST because that is the only place both container trees are
# addressable: they share this host's network namespace, which makes them
# feel co-located, but their filesystem roots are separate — Grafana cannot
# open a path inside authelia's tree however local the port looks.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
# refuses to start when a bind source is missing, and this secret does
# not exist until authelia's first boot has minted it — so binding it
# would make Grafana wait on a file that waits on a container that starts
# after it.
systemd.services.swarm-grafana-oidc-secret = lib.mkIf ssoLocal {
description = "deliver Grafana's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ];
requires = [ "container@${autheliaCfg.machine}.service" ];
before = [ "container@${cfg.machine}.service" ];
wantedBy = [ "container@${cfg.machine}.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-grafana-oidc-secret";
# ⚠️ Longer than the wait below, and that is the whole point:
# `DefaultTimeoutStartSec` is 90s, so a 120s bounded wait is killed
# by systemd at 90 — before it can emit the error naming the file it
# waited for. The timeout has to outlive the thing it is timing.
TimeoutStartSec = "180s";
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.oidc.clientId}.secret"}
dst=${lib.escapeShellArg hostSecretPath}
# authelia's container is up, but its first-boot generator may still
# be minting. Bounded wait, then fail: a silent skip here produces a
# Grafana whose only login path dead-ends.
deadline=$(( SECONDS + 120 ))
while [ ! -s "$src" ]; do
if [ "$SECONDS" -ge "$deadline" ]; then
echo "authelia has not minted $src after 120s" >&2
exit 1
fi
sleep 2
done
# Owned by Grafana's own uid, unlike the matrix sibling which lands
# root-owned: tuwunel's secret is read by `LoadCredential` as root
# before the sandbox exists, whereas Grafana expands `$__file{}`
# itself, as itself, while parsing its config. These containers set
# no `privateUsers`, so the host uid is the container uid, and both
# sides take it from the same static NixOS id.
#
# Group is root, not grafana, and that is forced rather than chosen:
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
# — the group's gid is allocated at activation inside the container,
# so the host cannot know it at eval time. Harmless here because 0400
# grants the group nothing; if this mode ever widens, the gid has to
# be discovered at runtime rather than assumed.
install -D -m 0400 -o ${toString config.ids.uids.grafana} -g 0 "$src" "$dst"
'';
};
# The same secret, the same destination, for the deployment where authelia
# is NOT here: it arrives out of the swarm secret store, which the
# publisher on authelia's host wrote it into. Nothing above changes — this
# is the second source for one file, never a second file.
#
# Shaped after ./glue-queue-agent-credential.nix rather than after the copy
# above, because it is a store reader and those are the store's readers: a
# cert login that fails LOUDLY, since every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY, since no retry turns "no
# value there" into a value.
#
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs on the authelia host and authelia mints on its first
# boot, so early in a swarm's life there is legitimately nothing to read.
# Nothing here writes a stand-in: there is no such thing as a locally valid
# OIDC client secret, and a placeholder would turn a Grafana that cannot
# log anyone in into one whose token exchange is refused.
#
# ⇒ So the absence surfaces at Grafana rather than here, and this unit's job
# is to leave the journal line that says which of the three it was. That is
# the same division the copy unit above makes by failing rather than
# skipping: the secret not arriving is a Grafana nobody can log into, and a
# named cause is the only thing separating that from a mystery.
systemd.services.swarm-bao-grafana-oidc = lib.mkIf storeDelivery {
description = "fetch Grafana's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering
# is conditional even though the read is not: off-host there is nothing
# local to wait for, and the timeout below bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${cfg.machine}.service" ];
wantedBy = [
"multi-user.target"
"container@${cfg.machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it. An
# hours-long window would be a bet on a sealed store, and the degrade
# below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-grafana-oidc";
# What actually bounds the read below. Stated here rather than left to
# systemd's default, so the number a boot waits on is in the file that
# waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving Grafana's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; Grafana has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# Written with a shell builtin, never handed to a program: `printf` is
# bash's own, so the plaintext never becomes an argument in /proc the
# way `install <<<"$secret"` or an `echo` from `path` would.
#
# Same uid, group and mode as the copy above, for the reasons stated
# there — this is the same file arriving by a different route.
install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath}
chown ${toString config.ids.uids.grafana}:0 ${lib.escapeShellArg hostSecretPath}
chmod 0400 ${lib.escapeShellArg hostSecretPath}
'';
};
containers.${cfg.machine} = {
autoStart = true;
ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [ "--link-journal=host" ];
# Shared host netns, like every sibling swarm container: the gateway
# reaches this at 127.0.0.1:<port>.
privateNetwork = false;
# The socket directory, shared with the host so nginx can reach in.
#
# ⚠️ It has to be a bind mount rather than a path both sides happen to
# know: a container's `/run` is an nspawn tmpfs, so it is NOT visible
# from the host at `/var/lib/nixos-containers/<name>/run`. The host
# creates the directory (tmpfiles rule below, which also fixes
# ownership) and nspawn mounts it in — the same shape the per-agent
# `/run/hive-agent/<name>/web.sock` already uses.
bindMounts = {
${deployCfg.grafana.socketDir} = {
hostPath = deployCfg.grafana.socketDir;
isReadOnly = false;
};
}
// caTrust.bindMount;
config =
{ ... }:
{
imports = [
(import ./swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp;
dnsConsumers = [ "grafana.service" ];
})
# Grafana is Go, and `SSL_CERT_FILE` *replaces* the default store
# rather than adding to it — so it needs the system CAs and the
# hive CA concatenated, not the CA alone. Without it the browser
# half of the login succeeds and the server-to-server token
# exchange fails `x509: unknown authority`.
(caTrust.trustBundle {
inherit pkgs;
name = cfg.machine;
consumers = [ "grafana" ];
})
];
system.stateVersion = "26.05";
# This container shares the host netns, so its own firewall.service
# would rewrite the HOST ruleset at every boot. The host firewall
# owns all filtering.
networking.firewall.enable = false;
# resolvconf stays off because the resolver unit imported above
# owns /etc/resolv.conf. Leaving it on would let host-tracking
# regenerate the file empty, since the host's copy doesn't cross
# the boundary after start.
networking.resolvconf.enable = lib.mkForce false;
# Grafana's `secret_key` has **no default in nixpkgs** and an
# assertion refuses the build without one — which is how the first
# version of this module broke a deploy. It signs the datasource
# secrets in Grafana's own database.
#
# Generated in-container on first boot, like authelia's own keys and
# for the same reason: nothing outside this container ever reads it,
# which makes in-container generation right rather than merely
# easier. (The OIDC client secret is delivered host-side precisely
# because it has a second reader.)
#
# ⚠️ Generated ONCE and kept — the `-s` guard is load-bearing.
# Rotating this key does not re-encrypt what it already encrypted, so
# a fresh key on every boot would leave Grafana unable to decrypt its
# own stored datasource secrets. Under /var/lib, never /run.
systemd.services.swarm-grafana-secret-key = {
description = "generate Grafana's datasource encryption key on first boot";
wantedBy = [ "grafana.service" ];
before = [ "grafana.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# Same User/Group/StateDirectory as grafana itself, so systemd
# creates the directory owned by the account that has to read
# the file — no chown, no mode juggling.
User = "grafana";
Group = "grafana";
StateDirectory = "grafana-secret";
StateDirectoryMode = "0700";
SyslogIdentifier = "swarm-grafana-secret-key";
};
path = [
pkgs.openssl
pkgs.coreutils
];
script = ''
set -euo pipefail
key=${lib.escapeShellArg secretKeyPath}
if [ ! -s "$key" ]; then
umask 077
openssl rand -hex 32 > "$key"
echo "generated Grafana's datasource encryption key"
fi
'';
};
# The group Grafana chgrps its socket to. Declared here because a
# container has its own user database — the host knowing gid 60 as
# `nginx` means nothing in here, only the number crosses.
#
# The name is local and arbitrary; the gid is the contract.
users.groups.gateway-nginx.gid = nginxGid;
users.users.grafana.extraGroups = [ "gateway-nginx" ];
services.grafana = {
enable = true;
package = deployCfg.grafana.package;
# Passed through unconditionally, empty default included.
# Upstream distinguishes `null` from `[ ]`, and both differences
# favour always handing it a list:
#
# - `null` points the plugin path at grafana's mutable
# `<dataDir>/plugins`; any list points it at a store path.
# Switching on the day someone adds their first plugin would
# bury that change inside an unrelated one.
# - upstream defaults its plugin update-check to
# `declarativePlugins == null`, so a list also stops the
# container phoning grafana.com. That is the no-runtime-egress
# property this option exists for — it should not arrive only
# once a plugin happens to be listed.
#
# - 🩸 and the one this comment used to miss: it also sets
# upstream's `plugins.preinstall_disabled` (nixpkgs'
# grafana.nix defaults it to `declarativePlugins != null`,
# saying preinstall "causes issues with declarativePlugins").
# Preinstall is what fetches the Drilldown apps on first
# start, so claiming this directory silently removes them.
# That is why the option's default is not empty.
#
# The claim that used to sit here — "nothing is taken over on a
# hive with no plugins, manual installation is impossible anyway"
# — reasoned only about plugins a PERSON installs. Upstream
# installs some itself, and those are the ones that went missing.
declarativePlugins = deployCfg.grafana.plugins;
settings = {
server = {
# Already upstream's default (measured), but pinned rather
# than inherited: the gateway is the only intended client,
# and this being loopback is what keeps the UI from being
# published on whatever else the host is reachable on.
# A unix socket, not a port. `socket_gid` is nginx's, so the
# gateway can connect; `0660` keeps everyone else out.
#
# ⚠️ Grafana chowns the socket to `socket_gid`, and a
# non-root process may only chgrp to a group it is a MEMBER
# of — hence the group declared for `grafana` in this
# container's own user database below. Without that
# membership Grafana starts, fails the chown, and the socket
# is unreachable by nginx with nothing obviously wrong.
protocol = "socket";
socket = socketPath;
socket_gid = nginxGid;
socket_mode = "0660";
domain = cfg.domain;
# Grafana builds its own OAuth redirect from this. Left at
# upstream's `%(protocol)s://%(domain)s:%(http_port)s/` it
# would name `http://<domain>:3000/`, which authelia has
# never heard of.
root_url = "https://${cfg.domain}/";
};
analytics = {
reporting_enabled = false;
check_for_updates = false;
};
# Stated rather than inherited. This is upstream's default, but
# a scrape target now depends on it, and a default that changes
# silently takes the target with it — the endpoint disappears,
# nginx keeps answering, and the scrape fails with a 404 far
# from anything that mentions Grafana.
metrics.enabled = true;
users.auto_assign_org_role = cfg.oidc.role;
# ⚠️ `$__file{}`, and required rather than optional: nixpkgs
# dropped the default and asserts on null, so without this the
# whole host fails to build. The unit below generates it.
security.secret_key = "$__file{${secretKeyPath}}";
# No local password path, ever. This is not tidiness: Grafana
# ships an `admin`/`admin` account and this vhost is on the
# public gateway, so SSO is the only way in — not merely the
# preferred one.
#
# Deliberately NOT `ssoLocal`: whether a password box sits on a
# public login page cannot depend on which host happens to run
# authelia. The OIDC block below now answers the same swarm-wide
# question, so the two no longer disagree.
auth.disable_login_form = true;
}
// lib.optionalAttrs ssoWired {
"auth.generic_oauth" = {
enabled = true;
name = "HyperHive";
client_id = cfg.oidc.clientId;
# ⚠️ `$__file{}`, never the secret itself — anything else
# here is world-readable in the nix store. Note the module's
# own leak assertion does NOT cover this key (it checks
# `database.password`, `security.admin_password` and
# datasource `secureJsonData`), so nothing but this comment
# stands between a literal and the store.
client_secret = "$__file{${secretPath}}";
scopes = "openid profile email groups";
auth_url = "${autheliaUrl}/api/oidc/authorization";
token_url = "${autheliaUrl}/api/oidc/token";
api_url = "${autheliaUrl}/api/oidc/userinfo";
use_pkce = true;
};
};
provision.datasources.settings = {
apiVersion = 1;
datasources = [
{
name = "VictoriaMetrics";
type = "prometheus";
uid = datasourceUid;
url = deployCfg.grafana.datasourceUrl;
access = "proxy";
isDefault = true;
}
{
name = "VictoriaLogs";
# The plugin's own id, read from its `plugin.json` rather
# than guessed. A `type` Grafana does not recognise is
# provisioned without complaint and every query against it
# fails at use, far from anything naming this file.
type = "victoriametrics-logs-datasource";
uid = logsDatasourceUid;
url = deployCfg.grafana.logsDatasourceUrl;
access = "proxy";
# Exactly one datasource may claim this, and metrics has it:
# two defaults is a coin toss over which one an untyped panel
# gets.
isDefault = false;
}
];
};
provision.dashboards.settings = {
apiVersion = 1;
providers = [
{
name = "hyperhive";
type = "file";
# Read-only in the UI: the store path is immutable, so an edit
# saved here would be silently discarded on the next rebuild.
# Better to refuse the edit than to lose it.
allowUiUpdates = false;
options.path = dashboardDir;
}
];
};
};
};
};
};
}
Reference in a new issue View git blame Copy permalink