vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive telemetry collector: fetch its OIDC secret from the store under the hive's own leaf

Browse source 
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.

`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.

otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.

nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.

Refs #5015
This commit is contained in:
atlas 2026-10-10 01:09:20 +02:00
commit 7c50568db1
7 changed files with 282 additions and 114 deletions
Download patch file Download diff file Expand all files Collapse all files

194
nix/host-modules/otel.nix
View file

@ -37,6 +37,16 @@ let
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
gatewayCfg = config.services.hyperhive.gateway;
};
# The hive's own store identity. The collector authenticates to the swarm
# as the hive, so it reads its secret as the hive: the hive leaf's policy
# (`swarm-secret-client`'s `policy::render`) grants `swarm/hives/<hive>/*`.
hiveLeaf =
config.services.hyperhive.deploy.bao.clientCertFile != null
&& config.services.hyperhive.deploy.bao.clientKeyFile != null;
# Where `swarm-bao-hive-otel-oidc` lands the secret on this host.
fetchedSecretPath = "/var/lib/hive-otel-oidc/${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${toString config.services.hyperhive.hiveName}.secret";
in
{
imports = [
@ -250,14 +260,8 @@ in
clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if
config.services.hyperhive.deploy.authelia.enable && config.services.hyperhive.hiveName != null
then
"${config.services.hyperhive.deploy.authelia.hostClientSecretDir}/"
+ "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret"
else
null;
defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null'';
if hiveLeaf && config.services.hyperhive.hiveName != null then fetchedSecretPath else null;
defaultText = lib.literalExpression ''"/var/lib/hive-otel-oidc/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when deploy.bao.clientCertFile and clientKeyFile are set, else null'';
example = "/var/lib/secrets/hive-telemetry.secret";
description = ''
Absolute path to this hive's OAuth2 client secret, used to
@ -268,13 +272,11 @@ in
authenticated egress possible, so its presence is the condition
rather than a second switch that could disagree with it.
Defaults to the secret this host's own authelia minted, which is
correct exactly when the swarm's IdP runs here. On a hive that is
not that host, the file has to arrive some other way and this
option names wherever it landed — the same manual-copy shape
`services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents,
where delivering a secret to a hive that is not this host is
deliberately not solved.
Defaults, on a host holding the hive's store identity
({option}`services.hyperhive.deploy.bao.clientCertFile`), to the file
`swarm-bao-hive-otel-oidc.service` fetches from the swarm secret
store under that identity. Set it to deliver the secret some other
way; that unit then does not exist.
Read by `LoadCredential`, so it is never evaluated by nix, never
copied into the store and never passed in argv.
@ -329,7 +331,19 @@ in
# renders neither the value nor the directory, and nothing has to
# hardcode `/run/credentials/<unit>`.
credName = "swarm-client.secret";
secretWaitUnit = "hive-otel-client-secret";
baoCfg = config.services.hyperhive.swarm.bao;
fetchUnit = "swarm-bao-hive-otel-oidc";
# Only when the collector reads the file the unit writes: a secret
# delivered some other way is the operator's, and a unit restarting the
# collector on a file it never loads would be noise.
fetches = senderAuth && hiveLeaf && otel.clientSecretFile == fetchedSecretPath;
# Written by swarm-secret-publish on authelia's host. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
# Reachable from agent containers and nowhere else: this opens
@ -646,69 +660,102 @@ in
# what the config above names.
(lib.mkIf senderAuth {
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
# `requires`, not merely `after`: ordering without gating leaves the
# collector starting anyway and failing on its own `LoadCredential`,
# which is the failure the wait exists to prevent. Same reasoning
# `lib/hive-ca-trust.nix` records for its bundle consumers.
requires = [ "${secretWaitUnit}.service" ];
after = [ "${secretWaitUnit}.service" ];
})
];
# Wait for the credential rather than racing it.
# This hive's OIDC client secret, out of the swarm secret store under the
# hive's own leaf. Same shape as ./hive-matrix.nix's
# `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and
# `storeRetry` retries until the publisher on authelia's host has written
# the secret, because the collector does not start without it.
#
# `LoadCredential` naming a missing path is FATAL AT UNIT START, and the
# secret is minted by authelia's first-boot generator inside its own
# container — nothing orders a host unit against that. Worse, nixpkgs
# sets `Restart = "always"` with no `RestartSec`, so the failure is
# instant and the unit burns systemd's 5-starts-in-10s allowance in well
# under a second, landing in `start-limit-hit` where it stops retrying
# entirely and needs `systemctl reset-failed` by hand.
#
# 🔑 `Restart = always` reads like it makes this self-healing and does
# the opposite: a SLOW-failing unit retries until the secret appears, a
# FAST-failing one exhausts its limit before the thing it waits for can
# exist. This oneshot converts the fast failure into a slow one, which
# is what that restart policy is actually good at.
#
# Copied from `hive-forge-oidc-secret.service`, which solves exactly
# this for the forge — see `docs/swarm/secrets.md`.
systemd.services.${secretWaitUnit} = lib.mkIf senderAuth {
description = "wait for this hive's telemetry client secret";
# Deliberately says nothing about the collector: the CONSUMER declares
# `requires` + `after` above, which both pulls this into the
# transaction and gates on it. Stating the order from both sides too
# would be one more thing to keep consistent for no added guarantee.
#
# Only when the minting container is on THIS host. Elsewhere the file
# is operator-provided and there is no local unit to order against —
# naming one that does not exist orders nothing, silently.
after = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service";
requires = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service";
serviceConfig = {
# The collector re-reads `client_secret_file` on every token fetch, but
# the file it reads is `LoadCredential`'s copy, made at start: a changed
# secret reaches it only by a restart, which `refresh_unit` does.
systemd.services.${fetchUnit} = lib.mkIf fetches {
description = "fetch this hive's telemetry OIDC client secret from the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = lib.optionals deployCfg.bao.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
before = [ "opentelemetry-collector.service" ];
wantedBy = [
"multi-user.target"
"opentelemetry-collector.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. The collector waits for one attempt; a secret
# a later attempt lands restarts it (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = secretWaitUnit;
# ⚠️ Longer than the bounded wait below. `DefaultTimeoutStartSec` is
# 90s, so without this systemd kills the unit at 90 — before it can
# emit the message naming the file it was waiting for.
TimeoutStartSec = "180s";
SyslogIdentifier = fetchUnit;
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.bao.clientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.clientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
secret=${lib.escapeShellArg (toString otel.clientSecretFile)}
# Bounded, then FAIL — never skip. A silent skip produces a
# collector that starts and ships nothing, which is the failure
# mode this whole path is trying not to have.
for _ in $(seq 1 60); do
[ -s "$secret" ] && exit 0
sleep 2
done
echo "telemetry client secret $secret has not appeared after 120s" >&2
exit 1
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this hive's certificate; leaving the telemetry collector's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2
exit 1
fi
# Root-owned 0400: the collector runs under `DynamicUser`, so there is
# no stable uid to hand the file to, and `LoadCredential` reads it as
# root before the sandbox exists.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf fetchedSecretPath)}
if secret_differs ${lib.escapeShellArg fetchedSecretPath} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg fetchedSecretPath} "$secret"
fi
refresh_unit opentelemetry-collector.service ${lib.escapeShellArg fetchedSecretPath}
'';
};
@ -733,9 +780,12 @@ in
`hive` label on its metrics mean anything — so both are
required.
On a host that runs the swarm's identity provider, the default
already points at the secret authelia minted. On a hive that
does not, copy that hive's secret across and name it here.
With the hive's store identity set
(services.hyperhive.deploy.bao.clientCertFile and clientKeyFile),
the default is the file swarm-bao-hive-otel-oidc.service fetches
under it. On a hive that runs the store both are set already;
elsewhere place the hive's leaf and name it there — see
docs/swarm/secrets.md.
'';
}
];