vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive telemetry collector: fetch its OIDC secret from the store under the hive's own leaf

Browse source 
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.

`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.

otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.

nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.

Refs #5015
This commit is contained in:
atlas 2026-10-10 01:09:20 +02:00
commit 7c50568db1
7 changed files with 282 additions and 114 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/getting-started/setup.md
View file

@ -9,8 +9,7 @@ Every command runs as **root on the host**.
> **Not all-local?** Each step says which host it runs on. A split swarm > **Not all-local?** Each step says which host it runs on. A split swarm
> also has credentials you can't generate where they're read — each > also has credentials you can't generate where they're read — each
> host's mTLS identity at the secret store, and each hive's telemetry > host's mTLS identity at the secret store. Read [`swarm/secrets.md`](../swarm/secrets.md) first; it
> ingest secret. Read [`swarm/secrets.md`](../swarm/secrets.md) first; it
> says which files you place, and where. > says which files you place, and where.
## 1 · Secret store ## 1 · Secret store

15
docs/scheduler/observability.md
View file

@ -129,11 +129,18 @@ from whichever receiver accepted a sample. A hive therefore can't report
metrics as another hive, and can't relabel its own by editing what it sends — metrics as another hive, and can't relabel its own by editing what it sends —
the label isn't taken from the payload at all. the label isn't taken from the payload at all.
**On an all-local swarm there is nothing to set.** Each hive already has an **Nothing to set where the hive holds its store identity.** The hive's collector
identity, and its collector reads the secret that host's own authelia minted. reads its secret from the swarm's secret store under the hive's own leaf
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client`, which the publisher on authelia's host
writes. It runs at boot and retries every 30s until the store answers; when a
run lands a changed value it restarts the collector. It doesn't run on a timer,
so a secret rotated later arrives the next time the unit runs. On a hive that
runs the store the leaf is already there; elsewhere it's the credential you
place — see [secret placement](../swarm/secrets.md).
**On a hive that doesn't host the swarm's services**, the secret has to arrive To deliver the secret some other way, name the file, and the fetch unit isn't
somehow — copy it across and name it: rendered:
```nix ```nix
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret"; services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";

20
docs/swarm/secrets.md
View file

@ -47,12 +47,13 @@ neither is a renaming of the other.
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
@ -67,16 +68,17 @@ because nothing outside that container ever reads them. **That's the test worth
applying to any secret added here** — and the client secret's plaintext half is applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists. the one row that fails it, which is the entire reason a delivery step exists.
Three telemetry collectors exist, and they don't land on the same side of Three telemetry collectors exist, and each one fails that test: each reads its
that test. own secret out of the store, under its own identity.
The **hive's** collector needs no delivery step. It authenticates to the swarm's The **hive's** collector authenticates to the swarm's collector as its own hive,
collector as its own hive, and it's a host unit rather than a container, so on so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
an all-local swarm it reads authelia's file where it lies and nothing makes a `swarm/hives/<hive>/oidc/client` under the hive's own leaf
second copy. On any other topology it's an ordinary "operator provides the file" (`deploy.bao.clientCertFile`), on every topology. A hive with no leaf has no
case — see `services.hyperhive.otel.clientSecretFile`. default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector does need one, and its delivery route is the fourth The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short: store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client` `swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store out of the store on every host that runs the collector and holds a store

58
nix/host-modules/lib/refresh-consumer.nix
View file

@ -1,7 +1,7 @@
# Shared shell steps for a host oneshot that fetches a credential into a # Shared shell steps for a host oneshot that fetches a credential into a
# file a service inside a container loads at start (`LoadCredential`, or a # file a service loads at start (`LoadCredential`, or a config file expanded
# config file expanded once while parsing). Such a service never sees a # once while parsing). Such a service never sees a value that lands after it
# value that lands after it started — late or rotated — until it restarts. # started — late or rotated — until it restarts.
# #
# The file's mtime is the record of a change, so write the file only when # The file's mtime is the record of a change, so write the file only when
# `secret_differs` says so. The restart decision is then re-derived from # `secret_differs` says so. The restart decision is then re-derived from
@ -16,7 +16,8 @@
# if secret_differs "$path" "$secret"; then # if secret_differs "$path" "$secret"; then
# atomic_write_secret 0400 root:root "$path" "$secret" # atomic_write_secret 0400 root:root "$path" "$secret"
# fi # fi
# refresh_consumer my-machine my.service "$path" # refresh_consumer my-machine my.service "$path" # a unit in a container
# refresh_unit my.service "$path" # a host unit
# ''; # '';
# #
# Requires `systemd` and `coreutils` on the caller's `path`. # Requires `systemd` and `coreutils` on the caller's `path`.
@ -30,22 +31,20 @@
[ ! -f "$1" ] || [ "$(< "$1")" != "$2" ] [ ! -f "$1" ] || [ "$(< "$1")" != "$2" ]
} }
# <machine> <unit> <path>. Nothing while `container@<machine>` is not # <unit> <path> [systemctl option…]. Nothing when <unit> last entered
# active (a container that starts later loads the file as it starts), or # `active` after <path> was last written. Otherwise a running <unit> is
# when <unit> last entered `active` after <path> was last written. # restarted and a failed one — it may have hit its start limit without the
# Otherwise a running consumer is restarted and a failed one — it may # credential — is reset and started; one stopped on purpose stays stopped.
# have hit its start limit without the credential — is reset and started; # The options are passed to every `systemctl` call, which is how
# one stopped on purpose stays stopped. `--no-block` because a caller # `refresh_consumer` reaches into a container. `--no-block` because a
# ordered `Before=` its consumer must not wait on a job that waits on the # caller ordered `Before=` its consumer must not wait on a job that waits
# caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix). # on the caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix).
refresh_consumer() { refresh_unit() {
local machine="$1" unit="$2" path="$3" started started_us=0 written_us local unit="$1" path="$2" started started_us=0 written_us
if ! systemctl is-active --quiet "container@$machine.service"; then shift 2
return 0
fi
# Empty for a unit that has never been active, which `date` would # Empty for a unit that has never been active, which `date` would
# otherwise read as today's midnight. # otherwise read as today's midnight.
started="$(systemctl --machine="$machine" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")" started="$(systemctl "$@" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")"
if [ -n "$started" ]; then if [ -n "$started" ]; then
started_us="$(date -u -d "$started" +%s%6N)" started_us="$(date -u -d "$started" +%s%6N)"
fi fi
@ -53,13 +52,24 @@
if [ "$written_us" -le "$started_us" ]; then if [ "$written_us" -le "$started_us" ]; then
return 0 return 0
fi fi
if systemctl --machine="$machine" is-failed --quiet "$unit"; then if systemctl "$@" is-failed --quiet "$unit"; then
echo "$path changed after $unit in $machine last started, and $unit had failed — starting it" echo "$path changed after $unit last started, and $unit had failed — starting it"
systemctl --machine="$machine" reset-failed "$unit" systemctl "$@" reset-failed "$unit"
systemctl --machine="$machine" start --no-block "$unit" systemctl "$@" start --no-block "$unit"
else else
echo "$path changed after $unit in $machine last started — restarting it if it runs" echo "$path changed after $unit last started — restarting it if it runs"
systemctl --machine="$machine" try-restart --no-block "$unit" systemctl "$@" try-restart --no-block "$unit"
fi fi
} }
# <machine> <unit> <path>. `refresh_unit` inside `container@<machine>`, and
# nothing while that container is not active: a container that starts
# later loads the file as it starts.
refresh_consumer() {
local machine="$1" unit="$2" path="$3"
if ! systemctl is-active --quiet "container@$machine.service"; then
return 0
fi
refresh_unit "$unit" "$path" --machine="$machine"
}
'' ''

194
nix/host-modules/otel.nix
View file

@ -37,6 +37,16 @@ let
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls; tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
gatewayCfg = config.services.hyperhive.gateway; gatewayCfg = config.services.hyperhive.gateway;
}; };
# The hive's own store identity. The collector authenticates to the swarm
# as the hive, so it reads its secret as the hive: the hive leaf's policy
# (`swarm-secret-client`'s `policy::render`) grants `swarm/hives/<hive>/*`.
hiveLeaf =
config.services.hyperhive.deploy.bao.clientCertFile != null
&& config.services.hyperhive.deploy.bao.clientKeyFile != null;
# Where `swarm-bao-hive-otel-oidc` lands the secret on this host.
fetchedSecretPath = "/var/lib/hive-otel-oidc/${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${toString config.services.hyperhive.hiveName}.secret";
in in
{ {
imports = [ imports = [
@ -250,14 +260,8 @@ in
clientSecretFile = lib.mkOption { clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = default =
if if hiveLeaf && config.services.hyperhive.hiveName != null then fetchedSecretPath else null;
config.services.hyperhive.deploy.authelia.enable && config.services.hyperhive.hiveName != null defaultText = lib.literalExpression ''"/var/lib/hive-otel-oidc/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when deploy.bao.clientCertFile and clientKeyFile are set, else null'';
then
"${config.services.hyperhive.deploy.authelia.hostClientSecretDir}/"
+ "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret"
else
null;
defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null'';
example = "/var/lib/secrets/hive-telemetry.secret"; example = "/var/lib/secrets/hive-telemetry.secret";
description = '' description = ''
Absolute path to this hive's OAuth2 client secret, used to Absolute path to this hive's OAuth2 client secret, used to
@ -268,13 +272,11 @@ in
authenticated egress possible, so its presence is the condition authenticated egress possible, so its presence is the condition
rather than a second switch that could disagree with it. rather than a second switch that could disagree with it.
Defaults to the secret this host's own authelia minted, which is Defaults, on a host holding the hive's store identity
correct exactly when the swarm's IdP runs here. On a hive that is ({option}`services.hyperhive.deploy.bao.clientCertFile`), to the file
not that host, the file has to arrive some other way and this `swarm-bao-hive-otel-oidc.service` fetches from the swarm secret
option names wherever it landed — the same manual-copy shape store under that identity. Set it to deliver the secret some other
`services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents, way; that unit then does not exist.
where delivering a secret to a hive that is not this host is
deliberately not solved.
Read by `LoadCredential`, so it is never evaluated by nix, never Read by `LoadCredential`, so it is never evaluated by nix, never
copied into the store and never passed in argv. copied into the store and never passed in argv.
@ -329,7 +331,19 @@ in
# renders neither the value nor the directory, and nothing has to # renders neither the value nor the directory, and nothing has to
# hardcode `/run/credentials/<unit>`. # hardcode `/run/credentials/<unit>`.
credName = "swarm-client.secret"; credName = "swarm-client.secret";
secretWaitUnit = "hive-otel-client-secret";
baoCfg = config.services.hyperhive.swarm.bao;
fetchUnit = "swarm-bao-hive-otel-oidc";
# Only when the collector reads the file the unit writes: a secret
# delivered some other way is the operator's, and a unit restarting the
# collector on a file it never loads would be noise.
fetches = senderAuth && hiveLeaf && otel.clientSecretFile == fetchedSecretPath;
# Written by swarm-secret-publish on authelia's host. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
in in
{ {
# Reachable from agent containers and nowhere else: this opens # Reachable from agent containers and nowhere else: this opens
@ -646,69 +660,102 @@ in
# what the config above names. # what the config above names.
(lib.mkIf senderAuth { (lib.mkIf senderAuth {
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ]; serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
# `requires`, not merely `after`: ordering without gating leaves the
# collector starting anyway and failing on its own `LoadCredential`,
# which is the failure the wait exists to prevent. Same reasoning
# `lib/hive-ca-trust.nix` records for its bundle consumers.
requires = [ "${secretWaitUnit}.service" ];
after = [ "${secretWaitUnit}.service" ];
}) })
]; ];
# Wait for the credential rather than racing it. # This hive's OIDC client secret, out of the swarm secret store under the
# hive's own leaf. Same shape as ./hive-matrix.nix's
# `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and
# `storeRetry` retries until the publisher on authelia's host has written
# the secret, because the collector does not start without it.
# #
# `LoadCredential` naming a missing path is FATAL AT UNIT START, and the # The collector re-reads `client_secret_file` on every token fetch, but
# secret is minted by authelia's first-boot generator inside its own # the file it reads is `LoadCredential`'s copy, made at start: a changed
# container — nothing orders a host unit against that. Worse, nixpkgs # secret reaches it only by a restart, which `refresh_unit` does.
# sets `Restart = "always"` with no `RestartSec`, so the failure is systemd.services.${fetchUnit} = lib.mkIf fetches {
# instant and the unit burns systemd's 5-starts-in-10s allowance in well description = "fetch this hive's telemetry OIDC client secret from the swarm secret store";
# under a second, landing in `start-limit-hit` where it stops retrying # The store's units exist only where the store runs; `Requires=` on an
# entirely and needs `systemctl reset-failed` by hand. # absent unit fails the job, so off-host the timeout below bounds the
# # attempt instead.
# 🔑 `Restart = always` reads like it makes this self-healing and does after = lib.optionals deployCfg.bao.enable [
# the opposite: a SLOW-failing unit retries until the secret appears, a "swarm-bao-pki.service"
# FAST-failing one exhausts its limit before the thing it waits for can "container@${baoCfg.machine}.service"
# exist. This oneshot converts the fast failure into a slow one, which ];
# is what that restart policy is actually good at. wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
# requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
# Copied from `hive-forge-oidc-secret.service`, which solves exactly before = [ "opentelemetry-collector.service" ];
# this for the forge — see `docs/swarm/secrets.md`. wantedBy = [
systemd.services.${secretWaitUnit} = lib.mkIf senderAuth { "multi-user.target"
description = "wait for this hive's telemetry client secret"; "opentelemetry-collector.service"
# Deliberately says nothing about the collector: the CONSUMER declares ];
# `requires` + `after` above, which both pulls this into the path = [
# transaction and gates on it. Stating the order from both sides too deployCfg.bao.package
# would be one more thing to keep consistent for no added guarantee. pkgs.coreutils
# pkgs.systemd
# Only when the minting container is on THIS host. Elsewhere the file ];
# is operator-provided and there is no local unit to order against — # ./lib/store-retry.nix. The collector waits for one attempt; a secret
# naming one that does not exist orders nothing, silently. # a later attempt lands restarts it (below).
after = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service"; inherit (storeRetry) startLimitBurst startLimitIntervalSec;
requires = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service"; serviceConfig = storeRetry.serviceConfig // {
serviceConfig = {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
SyslogIdentifier = secretWaitUnit; SyslogIdentifier = fetchUnit;
# ⚠️ Longer than the bounded wait below. `DefaultTimeoutStartSec` is TimeoutStartSec = 30;
# 90s, so without this systemd kills the unit at 90 — before it can };
# emit the message naming the file it was waiting for. environment = {
TimeoutStartSec = "180s"; BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.bao.clientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.clientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
}; };
path = [ pkgs.coreutils ];
script = '' script = ''
set -euo pipefail set -euo pipefail
secret=${lib.escapeShellArg (toString otel.clientSecretFile)}
# Bounded, then FAIL — never skip. A silent skip produces a ${atomicWriteSecret}
# collector that starts and ships nothing, which is the failure ${refreshConsumer}
# mode this whole path is trying not to have.
for _ in $(seq 1 60); do err="$(mktemp)"
[ -s "$secret" ] && exit 0 trap 'rm -f "$err"' EXIT
sleep 2
done # `-token-only` answers on stdout and skips the token helper, a
echo "telemetry client secret $secret has not appeared after 120s" >&2 # `sh` this unit's `path` does not carry.
exit 1 if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this hive's certificate; leaving the telemetry collector's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2
exit 1
fi
# Root-owned 0400: the collector runs under `DynamicUser`, so there is
# no stable uid to hand the file to, and `LoadCredential` reads it as
# root before the sandbox exists.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf fetchedSecretPath)}
if secret_differs ${lib.escapeShellArg fetchedSecretPath} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg fetchedSecretPath} "$secret"
fi
refresh_unit opentelemetry-collector.service ${lib.escapeShellArg fetchedSecretPath}
''; '';
}; };
@ -733,9 +780,12 @@ in
`hive` label on its metrics mean anything — so both are `hive` label on its metrics mean anything — so both are
required. required.
On a host that runs the swarm's identity provider, the default With the hive's store identity set
already points at the secret authelia minted. On a hive that (services.hyperhive.deploy.bao.clientCertFile and clientKeyFile),
does not, copy that hive's secret across and name it here. the default is the file swarm-bao-hive-otel-oidc.service fetches
under it. On a hive that runs the store both are set already;
elsewhere place the hive's leaf and name it there — see
docs/swarm/secrets.md.
''; '';
} }
]; ];

95
nix/module-eval/hive-otel.nix
View file

@ -29,6 +29,24 @@ let
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret"; otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
}; };
# A hive holding its own store leaf, placed by hand, and naming no secret
# file: the collector's secret comes from the store under that leaf.
hiveOtelLeaf = hive {
otel.enable = true;
deploy.bao.clientCertFile = "/etc/pki/hive.pem";
deploy.bao.clientKeyFile = "/etc/pki/hive-key.pem";
};
# The IdP, the store and the collector on one host: the publisher that
# writes the secret and the unit that reads it, in one config.
hiveOtelAllLocal = hive {
otel.enable = true;
deploy.bao.enable = true;
deploy.authelia.enable = true;
};
hiveOtelFetch = machine: machine.systemd.services.swarm-bao-hive-otel-oidc;
hiveOtelPipelines = hiveOtel.services.opentelemetry-collector.settings.service.pipelines; hiveOtelPipelines = hiveOtel.services.opentelemetry-collector.settings.service.pipelines;
hiveOtelSettings = hiveOtel.services.opentelemetry-collector.settings; hiveOtelSettings = hiveOtel.services.opentelemetry-collector.settings;
@ -173,6 +191,83 @@ let
in in
hivePort != swarmPort && hivePort != baoPort && swarmPort != baoPort; hivePort != swarmPort && hivePort != baoPort && swarmPort != baoPort;
} }
{
# The collector authenticates as the hive, so it reads its secret as
# the hive: the leaf `deploy.bao.clientCertFile` names, whose policy
# grants `swarm/hives/<hive>/*`. The last two clauses tie the unit's
# output to what the collector loads.
name = "the hive collector's secret is fetched from the store under the hive's own leaf";
ok =
let
u = hiveOtelFetch hiveOtelLeaf;
b = hiveOtelLeaf.services.hyperhive.deploy.bao;
file = hiveOtelLeaf.services.hyperhive.otel.clientSecretFile;
s = builtins.unsafeDiscardStringContext u.script;
in
hiveOtelLeaf.systemd.services ? swarm-bao-hive-otel-oidc
&& u.environment.BAO_CLIENT_CERT == b.clientCertFile
&& u.environment.BAO_CLIENT_KEY == b.clientKeyFile
&& lib.hasInfix "bao login -method=cert" s
&& lib.hasInfix "bao kv get -field=value secret/swarm/hives/h1/oidc/client" s
&& lib.hasInfix "atomic_write_secret 0400 \"\" ${file} " s
&& builtins.elem "swarm-client.secret:${file}" hiveOtelLeaf.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential
&& builtins.elem "opentelemetry-collector.service" u.wantedBy
&& builtins.elem "opentelemetry-collector.service" u.before;
}
{
# `LoadCredential` copies the file at the collector's start, so a value
# landed later reaches it only through a restart. Retries every 30s
# (./lib/store-retry.nix) while the store is down or the secret
# unpublished.
name = "the hive collector's fetch retries and restarts the collector when the file is newer";
ok =
let
u = hiveOtelFetch hiveOtelLeaf;
s = builtins.unsafeDiscardStringContext u.script;
in
u.serviceConfig.Restart == "on-failure"
&& u.serviceConfig.RestartSec == 30
&& toString u.unitConfig.StartLimitBurst == "2880"
&& lib.hasInfix "refresh_unit opentelemetry-collector.service /var/lib/hive-otel-oidc/hive-h1.secret" s
&& lib.hasInfix "ActiveEnterTimestamp" s
&& lib.hasInfix ''if [ "$written_us" -le "$started_us" ]; then'' s;
}
{
# The store is the one route, on the IdP's own host too: no default into
# authelia's tree and no unit waiting for a file to appear there.
name = "the hive collector reads no secret out of authelia's tree and waits on no copied file";
ok =
let
file = hiveOtelAllLocal.services.hyperhive.otel.clientSecretFile;
c = hiveOtelAllLocal.systemd.services.opentelemetry-collector;
in
!(hiveOtelAllLocal.systemd.services ? hive-otel-client-secret)
&& !(hiveOtelLeaf.systemd.services ? hive-otel-client-secret)
&& !(lib.hasInfix "authelia" file)
&& hiveOtelAllLocal.systemd.services ? swarm-bao-hive-otel-oidc
&& !(builtins.elem "hive-otel-client-secret.service" (c.requires or [ ]));
}
{
# The writer and the reader spell the path independently; this is the
# one place that sees both rendered.
name = "the hive collector reads the path the secret publisher writes";
ok =
let
fetch = builtins.unsafeDiscardStringContext (hiveOtelFetch hiveOtelAllLocal).script;
publish = builtins.unsafeDiscardStringContext hiveOtelAllLocal.systemd.services.swarm-secret-publish.script;
path = "secret/swarm/hives/h1/oidc/client";
in
lib.hasInfix "bao kv get -field=value ${path}" fetch && lib.hasInfix "bao kv put ${path} " publish;
}
{
# A secret delivered some other way is the operator's: no unit fetches
# into a file the collector does not load, or restarts it on one.
name = "a hand-named collector secret renders no store fetch";
ok =
!(hiveOtel.systemd.services ? swarm-bao-hive-otel-oidc)
&& !(baoWithHiveOtel.systemd.services ? swarm-bao-hive-otel-oidc)
&& builtins.elem "swarm-client.secret:/var/lib/hive-otel-oidc/client.secret" baoWithHiveOtel.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential;
}
]; ];
in in
runGroup "hive-otel" cases runGroup "hive-otel" cases

7
nix/module-eval/nats-authelia.nix
View file

@ -44,6 +44,9 @@ let
swarm.authelia.usersFile = "/var/lib/test-authelia/users.yml"; swarm.authelia.usersFile = "/var/lib/test-authelia/users.yml";
swarm.authelia.package = pkgs.emptyDirectory; swarm.authelia.package = pkgs.emptyDirectory;
swarm.authelia.bridgePackage = pkgs.emptyDirectory; swarm.authelia.bridgePackage = pkgs.emptyDirectory;
# Renders the publisher, which reads the host client-secret dir.
deploy.swarm-secret-publisher.baoClientCertFile = "/etc/pki/publisher.pem";
deploy.swarm-secret-publisher.baoClientKeyFile = "/etc/pki/publisher-key.pem";
}; };
# The queue with a store identity for its responder, placed by hand: the # The queue with a store identity for its responder, placed by hand: the
@ -467,7 +470,9 @@ let
# `swarm.authelia` instead of `deploy.authelia`. Several modules read this # `swarm.authelia` instead of `deploy.authelia`. Several modules read this
# through an `autheliaCfg` alias, where a path-shaped grep does not see it. # through an `autheliaCfg` alias, where a path-shaped grep does not see it.
name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace"; name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace";
ok = lib.hasPrefix "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients/" autheliaOldPath.services.hyperhive.otel.clientSecretFile; ok =
autheliaOldPath.systemd.paths.swarm-secret-publish.pathConfig.PathChanged
== "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients";
} }
{ {
# authelia exits at startup on `users: {}`, so an empty database, and a # authelia exits at startup on `users: {}`, so an empty database, and a