From 7c50568db1b2e873a8d07ee2ecea202204c7590b Mon Sep 17 00:00:00 2001 From: atlas Date: Sat, 10 Oct 2026 01:09:20 +0200 Subject: [PATCH] hive telemetry collector: fetch its OIDC secret from the store under the hive's own leaf The hive collector's `oauth2client` secret came from a `clientSecretFile` defaulting to authelia's own minted file on the IdP's host, with a `hive-otel-client-secret` oneshot that only waited up to 120s for that file to exist; any other hive had to copy the file across by hand. `swarm-bao-hive-otel-oidc` now logs in with the hive's leaf (`deploy.bao.clientCertFile`), reads `swarm/hives//oidc/client` (written by the publisher, granted by the hive leaf's `swarm/hives//*` read), writes it root 0400 when it differs, and restarts the collector when the file is newer than its last start. Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and forge fetches. `clientSecretFile` defaults to that file wherever the hive leaf is set and stays as an override; an overridden file renders no fetch. otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on every token fetch, but the path is `LoadCredential`'s start-time copy, so a changed secret still needs a restart. `lib/refresh-consumer.nix` gains `refresh_unit` for a host unit; `refresh_consumer` now calls it with `--machine`. nats-authelia's "consumer of the host client-secret dir" case pointed at the collector's old default; it now reads the publisher's path unit. Refs #5015 --- docs/getting-started/setup.md | 3 +- docs/scheduler/observability.md | 15 +- docs/swarm/secrets.md | 20 ++- nix/host-modules/lib/refresh-consumer.nix | 58 ++++--- nix/host-modules/otel.nix | 194 ++++++++++++++-------- nix/module-eval/hive-otel.nix | 95 +++++++++++ nix/module-eval/nats-authelia.nix | 7 +- 7 files changed, 280 insertions(+), 112 deletions(-) diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index e852963a..6ccc60b3 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -9,8 +9,7 @@ Every command runs as **root on the host**. > **Not all-local?** Each step says which host it runs on. A split swarm > also has credentials you can't generate where they're read — each -> host's mTLS identity at the secret store, and each hive's telemetry -> ingest secret. Read [`swarm/secrets.md`](../swarm/secrets.md) first; it +> host's mTLS identity at the secret store. Read [`swarm/secrets.md`](../swarm/secrets.md) first; it > says which files you place, and where. ## 1 · Secret store diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index 28a982e9..e3f47fe8 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -129,11 +129,18 @@ from whichever receiver accepted a sample. A hive therefore can't report metrics as another hive, and can't relabel its own by editing what it sends — the label isn't taken from the payload at all. -**On an all-local swarm there is nothing to set.** Each hive already has an -identity, and its collector reads the secret that host's own authelia minted. +**Nothing to set where the hive holds its store identity.** The hive's collector +reads its secret from the swarm's secret store under the hive's own leaf +(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches +`swarm/hives//oidc/client`, which the publisher on authelia's host +writes. It runs at boot and retries every 30s until the store answers; when a +run lands a changed value it restarts the collector. It doesn't run on a timer, +so a secret rotated later arrives the next time the unit runs. On a hive that +runs the store the leaf is already there; elsewhere it's the credential you +place — see [secret placement](../swarm/secrets.md). -**On a hive that doesn't host the swarm's services**, the secret has to arrive -somehow — copy it across and name it: +To deliver the secret some other way, name the file, and the fetch unit isn't +rendered: ```nix services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret"; diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index f1114b99..8059aefe 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -47,12 +47,13 @@ neither is a renaming of the other. | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | @@ -67,16 +68,17 @@ because nothing outside that container ever reads them. **That's the test worth applying to any secret added here** — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists. -Three telemetry collectors exist, and they don't land on the same side of -that test. +Three telemetry collectors exist, and each one fails that test: each reads its +own secret out of the store, under its own identity. -The **hive's** collector needs no delivery step. It authenticates to the swarm's -collector as its own hive, and it's a host unit rather than a container, so on -an all-local swarm it reads authelia's file where it lies and nothing makes a -second copy. On any other topology it's an ordinary "operator provides the file" -case — see `services.hyperhive.otel.clientSecretFile`. +The **hive's** collector authenticates to the swarm's collector as its own hive, +so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches +`swarm/hives//oidc/client` under the hive's own leaf +(`deploy.bao.clientCertFile`), on every topology. A hive with no leaf has no +default, and the build stops at the collector's identity assertion unless +`services.hyperhive.otel.clientSecretFile` names a file delivered some other way. -The **swarm's** collector does need one, and its delivery route is the fourth +The **swarm's** collector reads its own under a leaf of its own, the fourth store reader — the same shape as Grafana's, described in full below. In short: `swarm-bao-otel-oidc.service` reads `swarm/services//oidc/client` out of the store on every host that runs the collector and holds a store diff --git a/nix/host-modules/lib/refresh-consumer.nix b/nix/host-modules/lib/refresh-consumer.nix index 5248bfd6..c27923df 100644 --- a/nix/host-modules/lib/refresh-consumer.nix +++ b/nix/host-modules/lib/refresh-consumer.nix @@ -1,7 +1,7 @@ # Shared shell steps for a host oneshot that fetches a credential into a -# file a service inside a container loads at start (`LoadCredential`, or a -# config file expanded once while parsing). Such a service never sees a -# value that lands after it started — late or rotated — until it restarts. +# file a service loads at start (`LoadCredential`, or a config file expanded +# once while parsing). Such a service never sees a value that lands after it +# started — late or rotated — until it restarts. # # The file's mtime is the record of a change, so write the file only when # `secret_differs` says so. The restart decision is then re-derived from @@ -16,7 +16,8 @@ # if secret_differs "$path" "$secret"; then # atomic_write_secret 0400 root:root "$path" "$secret" # fi -# refresh_consumer my-machine my.service "$path" +# refresh_consumer my-machine my.service "$path" # a unit in a container +# refresh_unit my.service "$path" # a host unit # ''; # # Requires `systemd` and `coreutils` on the caller's `path`. @@ -30,22 +31,20 @@ [ ! -f "$1" ] || [ "$(< "$1")" != "$2" ] } - # . Nothing while `container@` is not - # active (a container that starts later loads the file as it starts), or - # when last entered `active` after was last written. - # Otherwise a running consumer is restarted and a failed one — it may - # have hit its start limit without the credential — is reset and started; - # one stopped on purpose stays stopped. `--no-block` because a caller - # ordered `Before=` its consumer must not wait on a job that waits on the - # caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix). - refresh_consumer() { - local machine="$1" unit="$2" path="$3" started started_us=0 written_us - if ! systemctl is-active --quiet "container@$machine.service"; then - return 0 - fi + # [systemctl option…]. Nothing when last entered + # `active` after was last written. Otherwise a running is + # restarted and a failed one — it may have hit its start limit without the + # credential — is reset and started; one stopped on purpose stays stopped. + # The options are passed to every `systemctl` call, which is how + # `refresh_consumer` reaches into a container. `--no-block` because a + # caller ordered `Before=` its consumer must not wait on a job that waits + # on the caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix). + refresh_unit() { + local unit="$1" path="$2" started started_us=0 written_us + shift 2 # Empty for a unit that has never been active, which `date` would # otherwise read as today's midnight. - started="$(systemctl --machine="$machine" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")" + started="$(systemctl "$@" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")" if [ -n "$started" ]; then started_us="$(date -u -d "$started" +%s%6N)" fi @@ -53,13 +52,24 @@ if [ "$written_us" -le "$started_us" ]; then return 0 fi - if systemctl --machine="$machine" is-failed --quiet "$unit"; then - echo "$path changed after $unit in $machine last started, and $unit had failed — starting it" - systemctl --machine="$machine" reset-failed "$unit" - systemctl --machine="$machine" start --no-block "$unit" + if systemctl "$@" is-failed --quiet "$unit"; then + echo "$path changed after $unit last started, and $unit had failed — starting it" + systemctl "$@" reset-failed "$unit" + systemctl "$@" start --no-block "$unit" else - echo "$path changed after $unit in $machine last started — restarting it if it runs" - systemctl --machine="$machine" try-restart --no-block "$unit" + echo "$path changed after $unit last started — restarting it if it runs" + systemctl "$@" try-restart --no-block "$unit" fi } + + # . `refresh_unit` inside `container@`, and + # nothing while that container is not active: a container that starts + # later loads the file as it starts. + refresh_consumer() { + local machine="$1" unit="$2" path="$3" + if ! systemctl is-active --quiet "container@$machine.service"; then + return 0 + fi + refresh_unit "$unit" "$path" --machine="$machine" + } '' diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index 78e670d3..52ac5b9a 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -37,6 +37,16 @@ let tlsCfg = config.services.hyperhive.deploy.hive-controller.tls; gatewayCfg = config.services.hyperhive.gateway; }; + + # The hive's own store identity. The collector authenticates to the swarm + # as the hive, so it reads its secret as the hive: the hive leaf's policy + # (`swarm-secret-client`'s `policy::render`) grants `swarm/hives//*`. + hiveLeaf = + config.services.hyperhive.deploy.bao.clientCertFile != null + && config.services.hyperhive.deploy.bao.clientKeyFile != null; + + # Where `swarm-bao-hive-otel-oidc` lands the secret on this host. + fetchedSecretPath = "/var/lib/hive-otel-oidc/${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${toString config.services.hyperhive.hiveName}.secret"; in { imports = [ @@ -250,14 +260,8 @@ in clientSecretFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = - if - config.services.hyperhive.deploy.authelia.enable && config.services.hyperhive.hiveName != null - then - "${config.services.hyperhive.deploy.authelia.hostClientSecretDir}/" - + "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret" - else - null; - defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null''; + if hiveLeaf && config.services.hyperhive.hiveName != null then fetchedSecretPath else null; + defaultText = lib.literalExpression ''"/var/lib/hive-otel-oidc/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when deploy.bao.clientCertFile and clientKeyFile are set, else null''; example = "/var/lib/secrets/hive-telemetry.secret"; description = '' Absolute path to this hive's OAuth2 client secret, used to @@ -268,13 +272,11 @@ in authenticated egress possible, so its presence is the condition rather than a second switch that could disagree with it. - Defaults to the secret this host's own authelia minted, which is - correct exactly when the swarm's IdP runs here. On a hive that is - not that host, the file has to arrive some other way and this - option names wherever it landed — the same manual-copy shape - `services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents, - where delivering a secret to a hive that is not this host is - deliberately not solved. + Defaults, on a host holding the hive's store identity + ({option}`services.hyperhive.deploy.bao.clientCertFile`), to the file + `swarm-bao-hive-otel-oidc.service` fetches from the swarm secret + store under that identity. Set it to deliver the secret some other + way; that unit then does not exist. Read by `LoadCredential`, so it is never evaluated by nix, never copied into the store and never passed in argv. @@ -329,7 +331,19 @@ in # renders neither the value nor the directory, and nothing has to # hardcode `/run/credentials/`. credName = "swarm-client.secret"; - secretWaitUnit = "hive-otel-client-secret"; + + baoCfg = config.services.hyperhive.swarm.bao; + fetchUnit = "swarm-bao-hive-otel-oidc"; + # Only when the collector reads the file the unit writes: a secret + # delivered some other way is the operator's, and a unit restarting the + # collector on a file it never loads would be noise. + fetches = senderAuth && hiveLeaf && otel.clientSecretFile == fetchedSecretPath; + # Written by swarm-secret-publish on authelia's host. The nix half of + # `swarm_secret_client::queue::hive_client_path`. + storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client"; + atomicWriteSecret = import ./lib/atomic-write-secret.nix { }; + refreshConsumer = import ./lib/refresh-consumer.nix { }; + storeRetry = import ./lib/store-retry.nix { }; in { # Reachable from agent containers and nowhere else: this opens @@ -646,69 +660,102 @@ in # what the config above names. (lib.mkIf senderAuth { serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ]; - - # `requires`, not merely `after`: ordering without gating leaves the - # collector starting anyway and failing on its own `LoadCredential`, - # which is the failure the wait exists to prevent. Same reasoning - # `lib/hive-ca-trust.nix` records for its bundle consumers. - requires = [ "${secretWaitUnit}.service" ]; - after = [ "${secretWaitUnit}.service" ]; }) ]; - # Wait for the credential rather than racing it. + # This hive's OIDC client secret, out of the swarm secret store under the + # hive's own leaf. Same shape as ./hive-matrix.nix's + # `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and + # `storeRetry` retries until the publisher on authelia's host has written + # the secret, because the collector does not start without it. # - # `LoadCredential` naming a missing path is FATAL AT UNIT START, and the - # secret is minted by authelia's first-boot generator inside its own - # container — nothing orders a host unit against that. Worse, nixpkgs - # sets `Restart = "always"` with no `RestartSec`, so the failure is - # instant and the unit burns systemd's 5-starts-in-10s allowance in well - # under a second, landing in `start-limit-hit` where it stops retrying - # entirely and needs `systemctl reset-failed` by hand. - # - # 🔑 `Restart = always` reads like it makes this self-healing and does - # the opposite: a SLOW-failing unit retries until the secret appears, a - # FAST-failing one exhausts its limit before the thing it waits for can - # exist. This oneshot converts the fast failure into a slow one, which - # is what that restart policy is actually good at. - # - # Copied from `hive-forge-oidc-secret.service`, which solves exactly - # this for the forge — see `docs/swarm/secrets.md`. - systemd.services.${secretWaitUnit} = lib.mkIf senderAuth { - description = "wait for this hive's telemetry client secret"; - # Deliberately says nothing about the collector: the CONSUMER declares - # `requires` + `after` above, which both pulls this into the - # transaction and gates on it. Stating the order from both sides too - # would be one more thing to keep consistent for no added guarantee. - # - # Only when the minting container is on THIS host. Elsewhere the file - # is operator-provided and there is no local unit to order against — - # naming one that does not exist orders nothing, silently. - after = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service"; - requires = lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service"; - serviceConfig = { + # The collector re-reads `client_secret_file` on every token fetch, but + # the file it reads is `LoadCredential`'s copy, made at start: a changed + # secret reaches it only by a restart, which `refresh_unit` does. + systemd.services.${fetchUnit} = lib.mkIf fetches { + description = "fetch this hive's telemetry OIDC client secret from the swarm secret store"; + # The store's units exist only where the store runs; `Requires=` on an + # absent unit fails the job, so off-host the timeout below bounds the + # attempt instead. + after = lib.optionals deployCfg.bao.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ]; + before = [ "opentelemetry-collector.service" ]; + wantedBy = [ + "multi-user.target" + "opentelemetry-collector.service" + ]; + path = [ + deployCfg.bao.package + pkgs.coreutils + pkgs.systemd + ]; + # ./lib/store-retry.nix. The collector waits for one attempt; a secret + # a later attempt lands restarts it (below). + inherit (storeRetry) startLimitBurst startLimitIntervalSec; + serviceConfig = storeRetry.serviceConfig // { Type = "oneshot"; RemainAfterExit = true; - SyslogIdentifier = secretWaitUnit; - # ⚠️ Longer than the bounded wait below. `DefaultTimeoutStartSec` is - # 90s, so without this systemd kills the unit at 90 — before it can - # emit the message naming the file it was waiting for. - TimeoutStartSec = "180s"; + SyslogIdentifier = fetchUnit; + TimeoutStartSec = 30; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = deployCfg.bao.clientCertFile; + BAO_CLIENT_KEY = deployCfg.bao.clientKeyFile; + } + // lib.optionalAttrs (deployCfg.bao.serverCaFile != null) { + BAO_CACERT = deployCfg.bao.serverCaFile; }; - path = [ pkgs.coreutils ]; script = '' set -euo pipefail - secret=${lib.escapeShellArg (toString otel.clientSecretFile)} - # Bounded, then FAIL — never skip. A silent skip produces a - # collector that starts and ships nothing, which is the failure - # mode this whole path is trying not to have. - for _ in $(seq 1 60); do - [ -s "$secret" ] && exit 0 - sleep 2 - done - echo "telemetry client secret $secret has not appeared after 120s" >&2 - exit 1 + ${atomicWriteSecret} + ${refreshConsumer} + + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # `-token-only` answers on stdout and skips the token helper, a + # `sh` this unit's `path` does not carry. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this hive's certificate; leaving the telemetry collector's OIDC client secret as it is." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + fi + export BAO_TOKEN + + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then + echo "swarm-bao did not return ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + fi + + if [ -z "$secret" ]; then + echo "swarm-bao returned an empty ${storeSecretPath}; the telemetry collector has no OIDC client secret yet." >&2 + exit 1 + fi + + # Root-owned 0400: the collector runs under `DynamicUser`, so there is + # no stable uid to hand the file to, and `LoadCredential` reads it as + # root before the sandbox exists. + install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf fetchedSecretPath)} + if secret_differs ${lib.escapeShellArg fetchedSecretPath} "$secret"; then + atomic_write_secret 0400 "" ${lib.escapeShellArg fetchedSecretPath} "$secret" + fi + + refresh_unit opentelemetry-collector.service ${lib.escapeShellArg fetchedSecretPath} ''; }; @@ -733,9 +780,12 @@ in `hive` label on its metrics mean anything — so both are required. - On a host that runs the swarm's identity provider, the default - already points at the secret authelia minted. On a hive that - does not, copy that hive's secret across and name it here. + With the hive's store identity set + (services.hyperhive.deploy.bao.clientCertFile and clientKeyFile), + the default is the file swarm-bao-hive-otel-oidc.service fetches + under it. On a hive that runs the store both are set already; + elsewhere place the hive's leaf and name it there — see + docs/swarm/secrets.md. ''; } ]; diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index 8fb51d65..ff45247c 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -29,6 +29,24 @@ let otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret"; }; + # A hive holding its own store leaf, placed by hand, and naming no secret + # file: the collector's secret comes from the store under that leaf. + hiveOtelLeaf = hive { + otel.enable = true; + deploy.bao.clientCertFile = "/etc/pki/hive.pem"; + deploy.bao.clientKeyFile = "/etc/pki/hive-key.pem"; + }; + + # The IdP, the store and the collector on one host: the publisher that + # writes the secret and the unit that reads it, in one config. + hiveOtelAllLocal = hive { + otel.enable = true; + deploy.bao.enable = true; + deploy.authelia.enable = true; + }; + + hiveOtelFetch = machine: machine.systemd.services.swarm-bao-hive-otel-oidc; + hiveOtelPipelines = hiveOtel.services.opentelemetry-collector.settings.service.pipelines; hiveOtelSettings = hiveOtel.services.opentelemetry-collector.settings; @@ -173,6 +191,83 @@ let in hivePort != swarmPort && hivePort != baoPort && swarmPort != baoPort; } + { + # The collector authenticates as the hive, so it reads its secret as + # the hive: the leaf `deploy.bao.clientCertFile` names, whose policy + # grants `swarm/hives//*`. The last two clauses tie the unit's + # output to what the collector loads. + name = "the hive collector's secret is fetched from the store under the hive's own leaf"; + ok = + let + u = hiveOtelFetch hiveOtelLeaf; + b = hiveOtelLeaf.services.hyperhive.deploy.bao; + file = hiveOtelLeaf.services.hyperhive.otel.clientSecretFile; + s = builtins.unsafeDiscardStringContext u.script; + in + hiveOtelLeaf.systemd.services ? swarm-bao-hive-otel-oidc + && u.environment.BAO_CLIENT_CERT == b.clientCertFile + && u.environment.BAO_CLIENT_KEY == b.clientKeyFile + && lib.hasInfix "bao login -method=cert" s + && lib.hasInfix "bao kv get -field=value secret/swarm/hives/h1/oidc/client" s + && lib.hasInfix "atomic_write_secret 0400 \"\" ${file} " s + && builtins.elem "swarm-client.secret:${file}" hiveOtelLeaf.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential + && builtins.elem "opentelemetry-collector.service" u.wantedBy + && builtins.elem "opentelemetry-collector.service" u.before; + } + { + # `LoadCredential` copies the file at the collector's start, so a value + # landed later reaches it only through a restart. Retries every 30s + # (./lib/store-retry.nix) while the store is down or the secret + # unpublished. + name = "the hive collector's fetch retries and restarts the collector when the file is newer"; + ok = + let + u = hiveOtelFetch hiveOtelLeaf; + s = builtins.unsafeDiscardStringContext u.script; + in + u.serviceConfig.Restart == "on-failure" + && u.serviceConfig.RestartSec == 30 + && toString u.unitConfig.StartLimitBurst == "2880" + && lib.hasInfix "refresh_unit opentelemetry-collector.service /var/lib/hive-otel-oidc/hive-h1.secret" s + && lib.hasInfix "ActiveEnterTimestamp" s + && lib.hasInfix ''if [ "$written_us" -le "$started_us" ]; then'' s; + } + { + # The store is the one route, on the IdP's own host too: no default into + # authelia's tree and no unit waiting for a file to appear there. + name = "the hive collector reads no secret out of authelia's tree and waits on no copied file"; + ok = + let + file = hiveOtelAllLocal.services.hyperhive.otel.clientSecretFile; + c = hiveOtelAllLocal.systemd.services.opentelemetry-collector; + in + !(hiveOtelAllLocal.systemd.services ? hive-otel-client-secret) + && !(hiveOtelLeaf.systemd.services ? hive-otel-client-secret) + && !(lib.hasInfix "authelia" file) + && hiveOtelAllLocal.systemd.services ? swarm-bao-hive-otel-oidc + && !(builtins.elem "hive-otel-client-secret.service" (c.requires or [ ])); + } + { + # The writer and the reader spell the path independently; this is the + # one place that sees both rendered. + name = "the hive collector reads the path the secret publisher writes"; + ok = + let + fetch = builtins.unsafeDiscardStringContext (hiveOtelFetch hiveOtelAllLocal).script; + publish = builtins.unsafeDiscardStringContext hiveOtelAllLocal.systemd.services.swarm-secret-publish.script; + path = "secret/swarm/hives/h1/oidc/client"; + in + lib.hasInfix "bao kv get -field=value ${path}" fetch && lib.hasInfix "bao kv put ${path} " publish; + } + { + # A secret delivered some other way is the operator's: no unit fetches + # into a file the collector does not load, or restarts it on one. + name = "a hand-named collector secret renders no store fetch"; + ok = + !(hiveOtel.systemd.services ? swarm-bao-hive-otel-oidc) + && !(baoWithHiveOtel.systemd.services ? swarm-bao-hive-otel-oidc) + && builtins.elem "swarm-client.secret:/var/lib/hive-otel-oidc/client.secret" baoWithHiveOtel.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential; + } ]; in runGroup "hive-otel" cases diff --git a/nix/module-eval/nats-authelia.nix b/nix/module-eval/nats-authelia.nix index 04f9506f..cc9783f7 100644 --- a/nix/module-eval/nats-authelia.nix +++ b/nix/module-eval/nats-authelia.nix @@ -44,6 +44,9 @@ let swarm.authelia.usersFile = "/var/lib/test-authelia/users.yml"; swarm.authelia.package = pkgs.emptyDirectory; swarm.authelia.bridgePackage = pkgs.emptyDirectory; + # Renders the publisher, which reads the host client-secret dir. + deploy.swarm-secret-publisher.baoClientCertFile = "/etc/pki/publisher.pem"; + deploy.swarm-secret-publisher.baoClientKeyFile = "/etc/pki/publisher-key.pem"; }; # The queue with a store identity for its responder, placed by hand: the @@ -467,7 +470,9 @@ let # `swarm.authelia` instead of `deploy.authelia`. Several modules read this # through an `autheliaCfg` alias, where a path-shaped grep does not see it. name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace"; - ok = lib.hasPrefix "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients/" autheliaOldPath.services.hyperhive.otel.clientSecretFile; + ok = + autheliaOldPath.systemd.paths.swarm-secret-publish.pathConfig.PathChanged + == "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients"; } { # authelia exits at startup on `users: {}`, so an empty database, and a