vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive telemetry collector: fetch its OIDC secret from the store under the hive's own leaf

Browse source 
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.

`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.

otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.

nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.

Refs #5015
This commit is contained in:
atlas 2026-10-10 01:09:20 +02:00
commit 7c50568db1
7 changed files with 282 additions and 114 deletions
Download patch file Download diff file Expand all files Collapse all files

58
nix/host-modules/lib/refresh-consumer.nix
View file

@ -1,7 +1,7 @@
# Shared shell steps for a host oneshot that fetches a credential into a
# file a service inside a container loads at start (`LoadCredential`, or a
# config file expanded once while parsing). Such a service never sees a
# value that lands after it started — late or rotated — until it restarts.
# file a service loads at start (`LoadCredential`, or a config file expanded
# once while parsing). Such a service never sees a value that lands after it
# started — late or rotated — until it restarts.
#
# The file's mtime is the record of a change, so write the file only when
# `secret_differs` says so. The restart decision is then re-derived from
@ -16,7 +16,8 @@
# if secret_differs "$path" "$secret"; then
# atomic_write_secret 0400 root:root "$path" "$secret"
# fi
# refresh_consumer my-machine my.service "$path"
# refresh_consumer my-machine my.service "$path" # a unit in a container
# refresh_unit my.service "$path" # a host unit
# '';
#
# Requires `systemd` and `coreutils` on the caller's `path`.
@ -30,22 +31,20 @@
[ ! -f "$1" ] || [ "$(< "$1")" != "$2" ]
}
# <machine> <unit> <path>. Nothing while `container@<machine>` is not
# active (a container that starts later loads the file as it starts), or
# when <unit> last entered `active` after <path> was last written.
# Otherwise a running consumer is restarted and a failed one — it may
# have hit its start limit without the credential — is reset and started;
# one stopped on purpose stays stopped. `--no-block` because a caller
# ordered `Before=` its consumer must not wait on a job that waits on the
# caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix).
refresh_consumer() {
local machine="$1" unit="$2" path="$3" started started_us=0 written_us
if ! systemctl is-active --quiet "container@$machine.service"; then
return 0
fi
# <unit> <path> [systemctl option…]. Nothing when <unit> last entered
# `active` after <path> was last written. Otherwise a running <unit> is
# restarted and a failed one — it may have hit its start limit without the
# credential — is reset and started; one stopped on purpose stays stopped.
# The options are passed to every `systemctl` call, which is how
# `refresh_consumer` reaches into a container. `--no-block` because a
# caller ordered `Before=` its consumer must not wait on a job that waits
# on the caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix).
refresh_unit() {
local unit="$1" path="$2" started started_us=0 written_us
shift 2
# Empty for a unit that has never been active, which `date` would
# otherwise read as today's midnight.
started="$(systemctl --machine="$machine" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")"
started="$(systemctl "$@" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")"
if [ -n "$started" ]; then
started_us="$(date -u -d "$started" +%s%6N)"
fi
@ -53,13 +52,24 @@
if [ "$written_us" -le "$started_us" ]; then
return 0
fi
if systemctl --machine="$machine" is-failed --quiet "$unit"; then
echo "$path changed after $unit in $machine last started, and $unit had failed — starting it"
systemctl --machine="$machine" reset-failed "$unit"
systemctl --machine="$machine" start --no-block "$unit"
if systemctl "$@" is-failed --quiet "$unit"; then
echo "$path changed after $unit last started, and $unit had failed — starting it"
systemctl "$@" reset-failed "$unit"
systemctl "$@" start --no-block "$unit"
else
echo "$path changed after $unit in $machine last started — restarting it if it runs"
systemctl --machine="$machine" try-restart --no-block "$unit"
echo "$path changed after $unit last started — restarting it if it runs"
systemctl "$@" try-restart --no-block "$unit"
fi
}
# <machine> <unit> <path>. `refresh_unit` inside `container@<machine>`, and
# nothing while that container is not active: a container that starts
# later loads the file as it starts.
refresh_consumer() {
local machine="$1" unit="$2" path="$3"
if ! systemctl is-active --quiet "container@$machine.service"; then
return 0
fi
refresh_unit "$unit" "$path" --machine="$machine"
}
''

194
nix/host-modules/otel.nix
View file

@ -37,6 +37,16 @@ let
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
gatewayCfg = config.services.hyperhive.gateway;
};
# The hive's own store identity. The collector authenticates to the swarm
# as the hive, so it reads its secret as the hive: the hive leaf's policy
# (`swarm-secret-client`'s `policy::render`) grants `swarm/hives/<hive>/*`.
hiveLeaf =
config.services.hyperhive.deploy.bao.clientCertFile != null
&& config.services.hyperhive.deploy.bao.clientKeyFile != null;
# Where `swarm-bao-hive-otel-oidc` lands the secret on this host.
fetchedSecretPath = "/var/lib/hive-otel-oidc/${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${toString config.services.hyperhive.hiveName}.secret";
in
{
imports = [
@ -250,14 +260,8 @@ in
clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if
config.services.hyperhive.deploy.authelia.enable && config.services.hyperhive.hiveName != null
then
"${config.services.hyperhive.deploy.authelia.hostClientSecretDir}/"
+ "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret"
else
null;
defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null'';
if hiveLeaf && config.services.hyperhive.hiveName != null then fetchedSecretPath else null;
defaultText = lib.literalExpression ''"/var/lib/hive-otel-oidc/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when deploy.bao.clientCertFile and clientKeyFile are set, else null'';
example = "/var/lib/secrets/hive-telemetry.secret";
description = ''
Absolute path to this hive's OAuth2 client secret, used to
@ -268,13 +272,11 @@ in
authenticated egress possible, so its presence is the condition
rather than a second switch that could disagree with it.
Defaults to the secret this host's own authelia minted, which is
correct exactly when the swarm's IdP runs here. On a hive that is
not that host, the file has to arrive some other way and this
option names wherever it landed — the same manual-copy shape
`services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents,
where delivering a secret to a hive that is not this host is
deliberately not solved.
Defaults, on a host holding the hive's store identity
({option}`services.hyperhive.deploy.bao.clientCertFile`), to the file
`swarm-bao-hive-otel-oidc.service` fetches from the swarm secret
store under that identity. Set it to deliver the secret some other
way; that unit then does not exist.
Read by `LoadCredential`, so it is never evaluated by nix, never
copied into the store and never passed in argv.
@ -329,7 +331,19 @@ in
# renders neither the value nor the directory, and nothing has to
# hardcode `/run/credentials/<unit>`.
credName = "swarm-client.secret";
secretWaitUnit = "hive-otel-client-secret";
baoCfg = config.services.hyperhive.swarm.bao;
fetchUnit = "swarm-bao-hive-otel-oidc";
# Only when the collector reads the file the unit writes: a secret
# delivered some other way is the operator's, and a unit restarting the
# collector on a file it never loads would be noise.
fetches = senderAuth && hiveLeaf && otel.clientSecretFile == fetchedSecretPath;
# Written by swarm-secret-publish on authelia's host. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
# Reachable from agent containers and nowhere else: this opens
@ -646,69 +660,102 @@ in
# what the config above names.
(lib.mkIf senderAuth {
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
# `requires`, not merely `after`: ordering without gating leaves the
# collector starting anyway and failing on its own `LoadCredential`,
# which is the failure the wait exists to prevent. Same reasoning
# `lib/hive-ca-trust.nix` records for its bundle consumers.
requires = [ "${secretWaitUnit}.service" ];
after = [ "${secretWaitUnit}.service" ];
})
];
# Wait for the credential rather than racing it.
# This hive's OIDC client secret, out of the swarm secret store under the
# hive's own leaf. Same shape as ./hive-matrix.nix's
# `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and
# `storeRetry` retries until the publisher on authelia's host has written
# the secret, because the collector does not start without it.
#
# `LoadCredential` naming a missing path is FATAL AT UNIT START, and the
# secret is minted by authelia's first-boot generator inside its own
# container — nothing orders a host unit against that. Worse, nixpkgs
# sets `Restart = "always"` with no `RestartSec`, so the failure is
# instant and the unit burns systemd's 5-starts-in-10s allowance in well
# under a second, landing in `start-limit-hit` where it stops retrying
# entirely and needs `systemctl reset-failed` by hand.
#
# 🔑 `Restart = always` reads like it makes this self-healing and does
# the opposite: a SLOW-failing unit retries until the secret appears, a
# FAST-failing one exhausts its limit before the thing it waits for can
# exist. This oneshot converts the fast failure into a slow one, which
# is what that restart policy is actually good at.
#
# Copied from `hive-forge-oidc-secret.service`, which solves exactly
# this for the forge — see `docs/swarm/secrets.md`.
systemd.services.${secretWaitUnit} = lib.mkIf senderAuth {
description = "wait for this hive's telemetry client secret";
# Deliberately says nothing about the collector: the CONSUMER declares
# `requires` + `after` above, which both pulls this into the
# transaction and gates on it. Stating the order from both sides too
# would be one more thing to keep consistent for no added guarantee.
#
# Only when the minting container is on THIS host. Elsewhere the file
# is operator-provided and there is no local unit to order against —
# naming one that does not exist orders nothing, silently.
after = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service";
requires = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service";
serviceConfig = {
# The collector re-reads `client_secret_file` on every token fetch, but
# the file it reads is `LoadCredential`'s copy, made at start: a changed
# secret reaches it only by a restart, which `refresh_unit` does.
systemd.services.${fetchUnit} = lib.mkIf fetches {
description = "fetch this hive's telemetry OIDC client secret from the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = lib.optionals deployCfg.bao.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
before = [ "opentelemetry-collector.service" ];
wantedBy = [
"multi-user.target"
"opentelemetry-collector.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. The collector waits for one attempt; a secret
# a later attempt lands restarts it (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = secretWaitUnit;
# ⚠️ Longer than the bounded wait below. `DefaultTimeoutStartSec` is
# 90s, so without this systemd kills the unit at 90 — before it can
# emit the message naming the file it was waiting for.
TimeoutStartSec = "180s";
SyslogIdentifier = fetchUnit;
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.bao.clientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.clientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
secret=${lib.escapeShellArg (toString otel.clientSecretFile)}
# Bounded, then FAIL — never skip. A silent skip produces a
# collector that starts and ships nothing, which is the failure
# mode this whole path is trying not to have.
for _ in $(seq 1 60); do
[ -s "$secret" ] && exit 0
sleep 2
done
echo "telemetry client secret $secret has not appeared after 120s" >&2
exit 1
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this hive's certificate; leaving the telemetry collector's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2
exit 1
fi
# Root-owned 0400: the collector runs under `DynamicUser`, so there is
# no stable uid to hand the file to, and `LoadCredential` reads it as
# root before the sandbox exists.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf fetchedSecretPath)}
if secret_differs ${lib.escapeShellArg fetchedSecretPath} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg fetchedSecretPath} "$secret"
fi
refresh_unit opentelemetry-collector.service ${lib.escapeShellArg fetchedSecretPath}
'';
};
@ -733,9 +780,12 @@ in
`hive` label on its metrics mean anything — so both are
required.
On a host that runs the swarm's identity provider, the default
already points at the secret authelia minted. On a hive that
does not, copy that hive's secret across and name it here.
With the hive's store identity set
(services.hyperhive.deploy.bao.clientCertFile and clientKeyFile),
the default is the file swarm-bao-hive-otel-oidc.service fetches
under it. On a hive that runs the store both are set already;
elsewhere place the hive's leaf and name it there — see
docs/swarm/secrets.md.
'';
}
];