From 01ce968fb6dce19ec6a21d873d9a6523b4bc2ba3 Mon Sep 17 00:00:00 2001 From: atlas Date: Mon, 7 Sep 2026 16:20:46 +0200 Subject: [PATCH] nix: split statusPublish and the otel secret into deploy.* MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slices 8 and 9 of the swarm/deploy split, and the last two. statusPublish had three coordinates under one namespace. Two of them are this machine's — where the queue listens *as seen from here*, and where its client secret sits on this disk — so they move to `deploy.hive-controller.statusPublish.*`, the namespace of the daemon that is their only reader. `tokenEndpoint` is the swarm's one address, so it stays. That leaves `swarm.statusPublish` holding a single option: a legitimate split, not a botched move. The all-or-nothing assertion now spans both namespaces. It is repointed in both its condition and its message, and the message spells all three paths in full so an operator is never told to set two options under a path that only has one. `environment.nix`'s guard and the value beside it likewise read different namespaces on purpose. The collector's secret moves the same way, for the same reason, to `deploy.swarm-otel.*` — `enable` already lives there. That also retires one of the eight cross-namespace assignments tracked in #4048: the delivery unit set a `swarm.*` value under a `deploy.*` gate, and now sets a `deploy.*` value under one. module-eval gets a fixture per slice. `otelRemoteAuthelia` already set the collector secret through its pre-rename path, so it becomes slice 9's old-path case as it stands — left spelled that way deliberately, with a comment, so it is not read later as a missed site. That fixture also turned out to be describing an impossible hive: it said authelia lives elsewhere without saying where, so the authenticator interpolated a null `swarm.authelia.url` into its `token_url`. Nothing to do with the rename, and invisible to the existing case over the same fixture, which reads `? auth` and `elem` — both stop at names and never force the extension's value. Given the address a remote-IdP deployment has. Verified: 49 -> 51 properties, all holding. --- docs/swarm/README.md | 19 +++--- docs/swarm/secrets.md | 4 +- nix/host-modules/deploy.nix | 20 ++++++ nix/host-modules/hive-c0re/default.nix | 9 +-- nix/host-modules/hive-c0re/environment.nix | 9 ++- nix/host-modules/swarm-otel.nix | 12 +++- nix/host-modules/swarm.nix | 75 ++++++++++++++-------- nix/module-eval.nix | 52 +++++++++++++++ 8 files changed, 153 insertions(+), 47 deletions(-) diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 0e5f43da..a0d5e1b3 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -301,8 +301,8 @@ it said. Hives publish upward through the swarm queue; the controller never reaches down to collect, so a hive that can't reach the swarm still knows its own state — you just can't see it from here. -A hive publishes only once it has been given the three -`swarm.statusPublish` coordinates below. A hive that has not reads +A hive publishes only once it holds all three status-publish coordinates +below. A hive without them reads `never_reported` — it's not broken, it just has nothing to say upward. | freshness | what to do about it | @@ -322,17 +322,18 @@ between reports. Hives publish once a minute, so the default tolerates one missed report and flags two. It takes effect on the next request; nothing has to re-publish. -### Making a hive report (`swarm.statusPublish`) +### Making a hive report Three options, on the **hive**, set together or not at all — a half-configured hive is an eval error rather than one that quietly never -reports: +reports. They sit in two namespaces, because two of them are facts about +_this machine_ and one is the swarm's single address: -| option | what to set it to | -| ------------------ | ------------------------------------------------------ | -| `natsUrl` | where the swarm queue listens, as this hive reaches it | -| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` | -| `clientSecretFile` | path to this hive's client secret, plaintext | +| option | what to set it to | +| ------------------------------------------------------- | ------------------------------------------------------ | +| `deploy.hive-controller.statusPublish.natsUrl` | where the swarm queue listens, as this hive reaches it | +| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` | +| `deploy.hive-controller.statusPublish.clientSecretFile` | path to this hive's client secret, plaintext | On a host that runs the queue and the IdP itself, all three default to the local ones and there is nothing to set. Any other hive needs them diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 1ac8cb51..73cd8705 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -51,7 +51,7 @@ neither is a renaming of the other. | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | operator provides the file and names it in `swarm.otel.clientSecretFile` — the collector need not share a host with authelia | +| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | @@ -84,7 +84,7 @@ published is being scraped; otherwise no secret is needed and none is placed. Whether it authenticates follows the _credential_, never another service's placement: a swarm collector may run on a host that holds neither store and no authelia, and then the secret is an ordinary operator-provided file named in -`services.hyperhive.swarm.otel.clientSecretFile` — the same shape as the hive +`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive collector's row above. The copy unit is the convenience for the co-located case, not the definition of the case. diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index b50032e5..a4e23af7 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -110,10 +110,30 @@ in [ "services" "hyperhive" "swarm" "nats" "enable" ] [ "services" "hyperhive" "deploy" "nats" "enable" ] ) + # Two of statusPublish's three coordinates. `tokenEndpoint` stays under + # `swarm.*`: it is the swarm IdP's one address, identical on every hive, + # where these two are "the queue as seen from here" and "the secret on + # this disk". The all-or-nothing assertion in ./swarm.nix spans the split + # deliberately and names all three paths in its message. + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "swarm" "statusPublish" "natsUrl" ] + [ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "natsUrl" ] + ) + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "swarm" "statusPublish" "clientSecretFile" ] + [ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "clientSecretFile" ] + ) (lib.mkRenamedOptionModule [ "services" "hyperhive" "swarm" "otel" "enable" ] [ "services" "hyperhive" "deploy" "swarm-otel" "enable" ] ) + # ⚠️ Not to be confused with `services.hyperhive.otel.clientSecretFile`, + # a different option on the hive tier that is not part of this move. A + # bare `clientSecretFile` matches five distinct options across the tree. + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "swarm" "otel" "clientSecretFile" ] + [ "services" "hyperhive" "deploy" "swarm-otel" "clientSecretFile" ] + ) # The CI runner, and the only entry here that renames more than an # `enable`: every knob under it describes the runner THIS host would run, diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index ee8a9f1c..4d71426e 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -123,7 +123,8 @@ in # plain http or loopback — so this changes nothing on an all-local # hive. It matters for the split-host shape the options invite: # `swarm.matrix.apiUrl`'s own example is `https://matrix.example.com`, - # and pointing it (or `statusPublish.natsUrl`) at another hive's + # and pointing it (or `deploy.hive-controller.statusPublish.natsUrl`) + # at another hive's # gateway means verifying a leaf signed by a CA generated at runtime, # which no build-time trust store can contain. # @@ -314,9 +315,9 @@ in # and the daemon never needs read access to wherever it lives. # (The callout responder copies instead only because it # delivers into a container, across a filesystem boundary.) - lib.optional ( - config.services.hyperhive.swarm.statusPublish.clientSecretFile != null - ) "swarm-status-client.secret:${config.services.hyperhive.swarm.statusPublish.clientSecretFile}" + lib.optional + (config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null) + "swarm-status-client.secret:${config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile}" # The secret store's client identity, on the same reasoning one # paragraph up — with a sharper edge: ./glue-bao-tls.nix mints the # key `0600` inside a `0700` directory, so hive-core cannot read it diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 9dbecbe3..8ff76562 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -259,8 +259,13 @@ in # produce is a hive that comes up fine and silently never reports. # The three-option version of that same rule is asserted at eval in # ./../swarm.nix, so this can only ever emit a complete set. - lib.optionalAttrs (config.services.hyperhive.swarm.statusPublish.natsUrl != null) { - HIVE_C0RE_NATS_URL = config.services.hyperhive.swarm.statusPublish.natsUrl; + # + # ⚠️ The guard and the value beside it read different namespaces on + # purpose: where the queue is and where its secret sits are this + # machine's, the token endpoint is the swarm's one address. The + # assertion covers all three, which is what keeps the guard honest. + lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null) { + HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl; HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint; # The identity swarm-authelia.nix already declares for every entry in # `swarm.hives` — the hive does not choose its own name here, it uses diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 9b8c0adf..090b158c 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -213,7 +213,7 @@ let # not of where any other service runs. Not an assertion: a collector on a # host of its own is a supported shape, and refusing to build it would make # this fix illegal where the bug bites hardest. - haveCollectorSecret = cfg.clientSecretFile != null; + haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; # The operator-configured upstream, named once: the same exporter carries # every signal, so metrics and logs both reach it without a second @@ -497,7 +497,13 @@ in after the change that caused it evaluated cleanly. ''; }; + }; + # What stays above is what the collector IS to the swarm — the client it is + # registered as, where it exports. The secret is a path on the machine that + # runs it, so it hangs off the deployment. `enable` already lives in + # ./deploy.nix, which also carries the rename. + options.services.hyperhive.deploy.swarm-otel = { clientSecretFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; @@ -630,7 +636,7 @@ in # authelia runs — it copies out of its container — so it is also what # names the file. `mkDefault`, so a deployment that delivers the secret # some other way just sets the option. - services.hyperhive.swarm.otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable ( + services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable ( lib.mkDefault collectorSecretInContainer ); @@ -1428,7 +1434,7 @@ in # no authenticator is rendered either, so the collector starts and # is refused by the stores rather than failing to start. // lib.optionalAttrs haveCollectorSecret { - LoadCredential = [ "${collectorCredentialId}:${cfg.clientSecretFile}" ]; + LoadCredential = [ "${collectorCredentialId}:${deployCfg.swarm-otel.clientSecretFile}" ]; }; }; }; diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 8867cc85..cfb577f4 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -356,24 +356,32 @@ in # genuinely cannot publish with two of three coordinates — which # is the distinction the `serviceDomains'` comment at the top of # this file was written about. + # + # The three coordinates live in two namespaces now: the token + # endpoint is the swarm's one address, the other two are this + # machine's. The message spells all three paths out, because an + # operator told only the option names would look for them under + # one prefix and find one of them. assertion = let set = lib.filter (v: v != null) [ - swarmCfg.statusPublish.natsUrl + deployCfg.hive-controller.statusPublish.natsUrl swarmCfg.statusPublish.tokenEndpoint - swarmCfg.statusPublish.clientSecretFile + deployCfg.hive-controller.statusPublish.clientSecretFile ]; in builtins.length set == 0 || builtins.length set == 3; message = '' - services.hyperhive.swarm.statusPublish needs natsUrl, - tokenEndpoint and clientSecretFile set together or not at all - — this hive has only some of them. + This hive's status-publishing coordinates have to be set + together or not at all — it has only some of them. Currently: - natsUrl = ${toString swarmCfg.statusPublish.natsUrl} - tokenEndpoint = ${toString swarmCfg.statusPublish.tokenEndpoint} - clientSecretFile = ${toString swarmCfg.statusPublish.clientSecretFile} + deploy.hive-controller.statusPublish.natsUrl + = ${toString deployCfg.hive-controller.statusPublish.natsUrl} + swarm.statusPublish.tokenEndpoint + = ${toString swarmCfg.statusPublish.tokenEndpoint} + deploy.hive-controller.statusPublish.clientSecretFile + = ${toString deployCfg.hive-controller.statusPublish.clientSecretFile} Set the missing ones to publish this hive's status to the swarm, or set all three to null to turn publishing off. @@ -440,6 +448,38 @@ in # enable. An extra flag would let a hive be configured-but-off, which # is one more state to explain and one more way to be silently quiet. options.services.hyperhive.swarm.statusPublish = { + tokenEndpoint = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = + if queueLocal && swarmCfg.authelia.url != null then + "${swarmCfg.authelia.url}/api/oidc/token" + else + null; + defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null''; + example = "https://auth.example.com/api/oidc/token"; + description = '' + The swarm IdP's OAuth2 token endpoint. This hive mints a + `client_credentials` access token there and presents it when + connecting to the queue, which authenticates it as + `hive-` — the client + {file}`nix/host-modules/swarm-authelia.nix` already declares for + every entry in {option}`services.hyperhive.swarm.hives`. + ''; + }; + }; + + # What stays above is the one thing every hive shares: the address of the + # swarm's token endpoint. The other two coordinates are this machine's — + # where the queue is *as seen from here*, and where its secret sits on this + # disk — so they hang off the daemon that uses them. `hive-controller` is + # hive-c0re's new name, which is why the namespace is the daemon's rather + # than a `deploy.statusPublish` of its own. + # + # ⚠️ The all-or-nothing assertion above now spans both namespaces. That is + # one service's own options either side of the split, not a service reaching + # into a foreign `deploy.*`; its message names all three paths so an operator + # is not told to set two options under a path that has one. + options.services.hyperhive.deploy.hive-controller.statusPublish = { natsUrl = lib.mkOption { type = lib.types.nullOr lib.types.str; default = if queueLocal then "nats://127.0.0.1:${toString swarmCfg.nats.port}" else null; @@ -461,25 +501,6 @@ in ''; }; - tokenEndpoint = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = - if queueLocal && swarmCfg.authelia.url != null then - "${swarmCfg.authelia.url}/api/oidc/token" - else - null; - defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null''; - example = "https://auth.example.com/api/oidc/token"; - description = '' - The swarm IdP's OAuth2 token endpoint. This hive mints a - `client_credentials` access token there and presents it when - connecting to the queue, which authenticates it as - `hive-` — the client - {file}`nix/host-modules/swarm-authelia.nix` already declares for - every entry in {option}`services.hyperhive.swarm.hives`. - ''; - }; - clientSecretFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = diff --git a/nix/module-eval.nix b/nix/module-eval.nix index d95f42a5..44b1561e 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -159,6 +159,18 @@ let swarm.grafana.plugins = [ ]; }; + # The first slice to leave options on BOTH sides of the split, so the + # fixture sets all three of them through the paths an existing config uses: + # the two movers via their rename entries, `tokenEndpoint` via the path it + # kept. That is also what the all-or-nothing assertion wants, so this hive + # is a valid one rather than one that only evaluates because nothing forced + # the assertion. + statusPublishOldPath = hive { + swarm.statusPublish.natsUrl = "nats://10.0.0.9:4222"; + swarm.statusPublish.tokenEndpoint = "https://auth.example.invalid/api/oidc/token"; + swarm.statusPublish.clientSecretFile = "/run/secrets/status-client.secret"; + }; + baoPkcs11 = hive { deploy.bao.enable = true; deploy.bao.seal = "pkcs11"; @@ -249,9 +261,19 @@ let # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. + # + # The `swarm.otel.clientSecretFile` below is the PRE-RENAME path. It predates + # the split and is deliberately left spelled that way: it makes this fixture + # the old-path case for that option too, so dropping its rename entry fails + # the eval here rather than only in a real operator's config. otelRemoteAuthelia = hive { deploy.swarm-otel.enable = true; deploy.authelia.enable = false; + # Where that elsewhere IS. Running no IdP does not mean knowing no IdP: + # the authenticator this fixture exists to render puts this address in its + # `token_url`, so a hive with a secret and no URL has a credential it can + # present nowhere. + swarm.authelia.url = "https://auth.example.invalid"; swarm.otel.clientSecretFile = "/var/lib/swarm-otel-oidc/by-hand.secret"; }; @@ -376,6 +398,36 @@ let rule: lib.hasInfix "/run/test-grafana-sock" rule ) grafanaOldPath.systemd.tmpfiles.rules; } + { + # Reads the daemon's rendered unit, not the options: the queue address + # arrives as an env var whose whole attrset is guarded on `natsUrl`, and + # the secret as a systemd credential. A rename that resolved but stopped + # reaching the module leaves hive-c0re coming up perfectly and reporting + # to nobody, which is the one failure this option set exists to prevent. + # The third arm is the option that did NOT move, read out of the same + # attrset: the guard and the value beside it now come from different + # namespaces, so a hive that renders one and drops the other is exactly + # what a split can silently produce. + name = "a config written against the pre-rename statusPublish paths still reaches the daemon"; + ok = + let + u = statusPublishOldPath.systemd.services.hive-c0re; + in + u.environment.HIVE_C0RE_NATS_URL == "nats://10.0.0.9:4222" + && u.environment.HIVE_C0RE_OIDC_TOKEN_ENDPOINT == "https://auth.example.invalid/api/oidc/token" + && lib.any (c: lib.hasInfix "/run/secrets/status-client.secret" c) u.serviceConfig.LoadCredential; + } + { + # The collector's half of the same split, and a different arm from the + # authenticator case below: this one reads the PATH the unit loads, so a + # reader left on a source that is non-null but wrong still fails. The + # fixture spells the option its pre-rename way, so it covers the rename + # entry at the same time. + name = "a config written against the pre-rename otel secret path still loads it as a credential"; + ok = + lib.any (c: lib.hasInfix "/var/lib/swarm-otel-oidc/by-hand.secret" c) + otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential; + } { name = "a config written against the pre-rename authelia usersFile still reaches the bridge"; ok =