vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm-otel: deliver the OIDC client secret through the secret store

Browse source 
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.

Mirrors PR #4361 (Grafana) almost exactly:

- `swarm-bao-otel-oidc.service` reads
  `swarm/services/<client-id>/oidc/client` out of the store, in every
  deployment, replacing the co-located copy unit outright — one
  delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
  block (gated on this host running the collector) into
  `glue-swarm-otel-oidc-client.nix` (gated on this host running
  authelia), the same split `glue-grafana-oidc-client.nix` made. It
  was broken the same way: a split deployment registered the client
  nowhere at all, so authelia never minted a secret for the publisher
  to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
  hive read grant in `policy::render`) already covers any service's
  path — nothing to add there. `swarm-secret-publisher.nix` only grew
  `serviceClientIds` by one entry.

One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.

Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.

Refs #4258
This commit is contained in:
atlas 2026-09-13 20:23:39 +02:00 • committed by mara
commit 0ff5c8110b
7 changed files with 409 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

271
nix/host-modules/swarm-otel.nix
View file

@ -23,6 +23,8 @@ let
vlCfg = config.services.hyperhive.swarm.victorialogs;
hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
swarmDomain = hyperhiveCfg.swarm.domain;
# Total on a null swarm domain for the same reason every sibling module is:
@ -132,6 +134,13 @@ let
# delivery unit on a later boot, and a secret that evaporates on reboot
# turns a working scrape into an intermittent one.
collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret";
# The same file seen from the host, which is where the delivery unit
# writes it — spelled once for the same reason ./swarm-grafana.nix spells
# its own pair once: the unit that writes it and the option that names it
# are a few hundred lines apart, and a collector reading a path nothing
# writes is an export that fails with nothing in any log about the file.
collectorHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}";
collectorHostSecretDir = builtins.dirOf collectorHostSecretPath;
collectorCredentialId = "oidc-client-secret";
# What the scrape config points at. ⚠️ This path and the `LoadCredential`
# id below are one fact spelled twice by systemd's design — both derive from
@ -206,11 +215,32 @@ let
pushAuthenticator = name: "oauth2client/${name}";
# Whether this collector holds a credential — a property of the credential,
# not of where any other service runs. Not an assertion: a collector on a
# host of its own is a supported shape, and refusing to build it would make
# this fix illegal where the bug bites hardest.
# not of where any other service runs. Not an assertion: a collector that
# pushes nowhere authenticated still receives from every hive, so refusing
# to build one would make a supported shape illegal for want of a value
# that only degrades what it can push.
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical-
# looking flag, this one stays outside `assertions`: the store-reading unit
# below simply does not render without it, the same choice
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for
# their own optional readers, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the publisher on authelia's host leaves this client's secret —
# composed from the same swarm-wide `clientId` the registration in
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
# them behind. The `services` segment is `swarm-secret-client`'s
# `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own
# client secret under — one write grant in ./swarm-bao.nix and one hive read
# grant in `policy::render` already cover it.
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
# The operator-configured upstream, named once: the same exporter carries
# every signal, so metrics and logs both reach it without a second
# definition.
@ -493,6 +523,21 @@ in
after the change that caused it evaluated cleanly.
'';
};
audience = lib.mkOption {
type = lib.types.listOf lib.types.str;
readOnly = true;
default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
description = ''
Every audience this collector's OAuth2 client is permitted to
present a token for — the scrape targets it reads with a credential,
plus the stores it pushes to. Published read-only so
./glue-swarm-otel-oidc-client.nix can register the client wherever
authelia runs without restating the derivation: this option and
that registration are the same fact seen from two hosts, and a
second formula for it would be free to drift from this one.
'';
};
};
# What stays above is what the collector IS to the swarm — the client it is
@ -529,7 +574,10 @@ in
# delivering says so in the store it stopped delivering to. That is
# less circular than it sounds: the failure that matters here is a
# single hive's receiver refusing pushes, not the process dying.
services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ];
services.hyperhive.swarm.otel.journaldUnits = [
"opentelemetry-collector"
"swarm-bao-otel-oidc"
];
# The metrics counterpart to the journal line above, closing the same
# gap from the other side: the journal says the process is alive, these
@ -584,109 +632,152 @@ in
# this is how it authenticates *itself* to a service published behind
# the gateway.
#
# Only where authelia is co-located. A client is a row in this host's
# provider config, so declaring one against a remote provider would
# render nothing while reading as done; a swarm whose authelia lives
# elsewhere registers it there.
#
# ⚠️ NO LONGER conditional on a published scrape target. Authelia refuses
# a bearer-authz client with no audience, which is what that guard was
# for — and the push audiences below are unconditional, so there is now
# always at least one. Keeping the old guard would have left a collector
# that scrapes nothing pushing to the stores with no client to get a
# token from.
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [
{
id = cfg.clientId;
description = "HyperHive swarm collector";
kind = "machine";
# Grants `authelia.bearer.authz`, without which the authz
# endpoint refuses an otherwise valid token and blames the
# token rather than the missing grant.
bearerAuthz = true;
# DERIVED from the targets rather than contributed alongside
# them. A service declares a URL once and this is the
# permission to reach it; two lists that had to agree would be
# a drift to maintain, and the failure mode is the quiet one —
# a target whose audience was forgotten authenticates against
# nothing and looks like a broken scrape.
#
# Both directions land in one list because authelia has one: what
# this collector may SCRAPE and what it may PUSH TO are the same
# kind of permission, differing only in who initiates.
audience = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
# Stated rather than left on authelia's default, because the
# two agreeing today is not the same as this being the
# required value: authelia permits only basic / JWT methods
# for a confidential client holding that scope, and enforces
# it in the startup validator.
tokenEndpointAuthMethod = "client_secret_basic";
}
];
# Registering the client is NOT here any more: it has to happen on the
# host that runs authelia, and this whole block is gated on the host that
# runs the collector. ./glue-swarm-otel-oidc-client.nix is where it moved
# to, the same split ./swarm-grafana.nix made for its own client.
# Deliver the collector's client secret from authelia's container into
# this one. On the HOST because that is the only place both container
# trees are addressable: they share this host's network namespace, which
# makes them feel co-located, but their filesystem roots are separate.
# THE delivery unit — one route, in every deployment. The secret authelia
# minted arrives out of the swarm secret store, which the publisher on
# authelia's host wrote it into, whether authelia is a network away or in
# the container next door.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
# refuses to start when a bind source is missing, and this secret does not
# exist until authelia's first boot has minted it — so binding it would
# make the collector wait on a file that waits on a container that starts
# after it. On a fresh swarm that is a permanent stall presenting as
# "metrics are broken", several layers from its cause.
# The delivery unit below is the one thing here that may know where
# authelia runs — it copies out of its container — so it is also what
# names the file. `mkDefault`, so a deployment that delivers the secret
# some other way just sets the option.
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable (
lib.mkDefault collectorSecretInContainer
);
systemd.services.swarm-otel-oidc-secret = lib.mkIf deployCfg.authelia.enable {
description = "deliver the swarm collector's OIDC client secret from authelia";
after = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
requires = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
# 🩸 A unit here used to copy the plaintext directly out of authelia's
# host tree, reachable only because they share this host's network
# namespace — which stopped working the moment authelia moved to another
# host, which is the defect this whole change exists to fix. The ruling
# that deleted it rather than gave it a remote sibling: the store exists
# so a host holds ONE out-of-band secret — its client certificate — and
# reads everything else with it. Recorded in docs/swarm/secrets.md.
#
# Shaped after ./glue-queue-agent-credential.nix: a cert login that fails
# LOUDLY, since every state it fails on is one a retry fixes, then a read
# that degrades QUIETLY, since no retry turns "no value there" into a
# value.
#
# ⚠️ Renders only where `haveClientIdentity` holds, unlike
# ./swarm-grafana.nix's equivalent unit. That module asserts the identity
# because a Grafana with none has no way in at all; this collector without
# one is `haveCollectorSecret = false` above — already a supported,
# merely degraded shape, so the unit that would fetch a credential simply
# does not exist rather than refusing the build for want of one.
systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity {
description = "fetch the swarm collector's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering
# is conditional even though the read is not: off-host there is nothing
# local to wait for, and the timeout below bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${cfg.machine}.service" ];
wantedBy = [ "container@${cfg.machine}.service" ];
wantedBy = [
"multi-user.target"
"container@${cfg.machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it. An
# hours-long window would be a bet on a sealed store, and the degrade
# below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-otel-oidc-secret";
# Longer than the bounded wait below, and that is the point:
# `DefaultTimeoutStartSec` is 90s, so without this systemd kills
# the unit before it can emit the message naming the file it was
# waiting for — the failure then reads as a timeout with no cause.
TimeoutStartSec = "180s";
SyslogIdentifier = "swarm-bao-otel-oidc";
# What actually bounds the read below. Stated here rather than left to
# systemd's default, so the number a boot waits on is in the file that
# waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"}
dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"}
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# authelia's container is up, but its first-boot generator may
# still be minting. Bounded wait, then fail: skipping silently
# produces a collector whose scrape gets a 401 forever, which is
# the failure this whole design exists to make impossible.
for _ in $(seq 1 60); do
[ -s "$src" ] && break
sleep 2
done
if [ ! -s "$src" ]; then
echo "authelia has not minted $src after 120s" >&2
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the collector's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# root-owned 0400. The collector runs under `DynamicUser`, so
# there is no uid to give it to — `LoadCredential` reads this as
# root before the sandbox exists and re-exposes it to whichever
# uid the unit got.
install -D -m 0400 -o root -g root "$src" "$dst"
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the collector has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# root-owned 0400, written with a shell builtin and never handed to a
# program: `printf` is bash's own, so the plaintext never becomes an
# argument in /proc the way `install <<<"$secret"` or an `echo` from
# `path` would. The collector runs under `DynamicUser`, so there is
# no uid to give this to — `LoadCredential` reads it as root before
# the sandbox exists and re-exposes it to whichever uid the unit got.
install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath}
chown root:root ${lib.escapeShellArg collectorHostSecretPath}
chmod 0400 ${lib.escapeShellArg collectorHostSecretPath}
'';
};
# Where the delivery unit above lands the secret. `mkDefault`, so a
# deployment delivering it some other way just sets the option — and
# `mkIf haveClientIdentity` so a host with no store identity is left with
# `clientSecretFile == null`, the already-supported degrade rather than a
# path nothing ever writes.
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf haveClientIdentity (
lib.mkDefault collectorSecretInContainer
);
# The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet.