diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 0e5f43da..a0d5e1b3 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -301,8 +301,8 @@ it said. Hives publish upward through the swarm queue; the controller never reaches down to collect, so a hive that can't reach the swarm still knows its own state — you just can't see it from here. -A hive publishes only once it has been given the three -`swarm.statusPublish` coordinates below. A hive that has not reads +A hive publishes only once it holds all three status-publish coordinates +below. A hive without them reads `never_reported` — it's not broken, it just has nothing to say upward. | freshness | what to do about it | @@ -322,17 +322,18 @@ between reports. Hives publish once a minute, so the default tolerates one missed report and flags two. It takes effect on the next request; nothing has to re-publish. -### Making a hive report (`swarm.statusPublish`) +### Making a hive report Three options, on the **hive**, set together or not at all — a half-configured hive is an eval error rather than one that quietly never -reports: +reports. They sit in two namespaces, because two of them are facts about +_this machine_ and one is the swarm's single address: -| option | what to set it to | -| ------------------ | ------------------------------------------------------ | -| `natsUrl` | where the swarm queue listens, as this hive reaches it | -| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` | -| `clientSecretFile` | path to this hive's client secret, plaintext | +| option | what to set it to | +| ------------------------------------------------------- | ------------------------------------------------------ | +| `deploy.hive-controller.statusPublish.natsUrl` | where the swarm queue listens, as this hive reaches it | +| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` | +| `deploy.hive-controller.statusPublish.clientSecretFile` | path to this hive's client secret, plaintext | On a host that runs the queue and the IdP itself, all three default to the local ones and there is nothing to set. Any other hive needs them diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 1ac8cb51..73cd8705 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -51,7 +51,7 @@ neither is a renaming of the other. | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | operator provides the file and names it in `swarm.otel.clientSecretFile` — the collector need not share a host with authelia | +| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | @@ -84,7 +84,7 @@ published is being scraped; otherwise no secret is needed and none is placed. Whether it authenticates follows the _credential_, never another service's placement: a swarm collector may run on a host that holds neither store and no authelia, and then the secret is an ordinary operator-provided file named in -`services.hyperhive.swarm.otel.clientSecretFile` — the same shape as the hive +`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive collector's row above. The copy unit is the convenience for the co-located case, not the definition of the case. diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index b50032e5..a4e23af7 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -110,10 +110,30 @@ in [ "services" "hyperhive" "swarm" "nats" "enable" ] [ "services" "hyperhive" "deploy" "nats" "enable" ] ) + # Two of statusPublish's three coordinates. `tokenEndpoint` stays under + # `swarm.*`: it is the swarm IdP's one address, identical on every hive, + # where these two are "the queue as seen from here" and "the secret on + # this disk". The all-or-nothing assertion in ./swarm.nix spans the split + # deliberately and names all three paths in its message. + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "swarm" "statusPublish" "natsUrl" ] + [ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "natsUrl" ] + ) + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "swarm" "statusPublish" "clientSecretFile" ] + [ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "clientSecretFile" ] + ) (lib.mkRenamedOptionModule [ "services" "hyperhive" "swarm" "otel" "enable" ] [ "services" "hyperhive" "deploy" "swarm-otel" "enable" ] ) + # ⚠️ Not to be confused with `services.hyperhive.otel.clientSecretFile`, + # a different option on the hive tier that is not part of this move. A + # bare `clientSecretFile` matches five distinct options across the tree. + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "swarm" "otel" "clientSecretFile" ] + [ "services" "hyperhive" "deploy" "swarm-otel" "clientSecretFile" ] + ) # The CI runner, and the only entry here that renames more than an # `enable`: every knob under it describes the runner THIS host would run, diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index ee8a9f1c..4d71426e 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -123,7 +123,8 @@ in # plain http or loopback — so this changes nothing on an all-local # hive. It matters for the split-host shape the options invite: # `swarm.matrix.apiUrl`'s own example is `https://matrix.example.com`, - # and pointing it (or `statusPublish.natsUrl`) at another hive's + # and pointing it (or `deploy.hive-controller.statusPublish.natsUrl`) + # at another hive's # gateway means verifying a leaf signed by a CA generated at runtime, # which no build-time trust store can contain. # @@ -314,9 +315,9 @@ in # and the daemon never needs read access to wherever it lives. # (The callout responder copies instead only because it # delivers into a container, across a filesystem boundary.) - lib.optional ( - config.services.hyperhive.swarm.statusPublish.clientSecretFile != null - ) "swarm-status-client.secret:${config.services.hyperhive.swarm.statusPublish.clientSecretFile}" + lib.optional + (config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null) + "swarm-status-client.secret:${config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile}" # The secret store's client identity, on the same reasoning one # paragraph up — with a sharper edge: ./glue-bao-tls.nix mints the # key `0600` inside a `0700` directory, so hive-core cannot read it diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 9dbecbe3..8ff76562 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -259,8 +259,13 @@ in # produce is a hive that comes up fine and silently never reports. # The three-option version of that same rule is asserted at eval in # ./../swarm.nix, so this can only ever emit a complete set. - lib.optionalAttrs (config.services.hyperhive.swarm.statusPublish.natsUrl != null) { - HIVE_C0RE_NATS_URL = config.services.hyperhive.swarm.statusPublish.natsUrl; + # + # ⚠️ The guard and the value beside it read different namespaces on + # purpose: where the queue is and where its secret sits are this + # machine's, the token endpoint is the swarm's one address. The + # assertion covers all three, which is what keeps the guard honest. + lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null) { + HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl; HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint; # The identity swarm-authelia.nix already declares for every entry in # `swarm.hives` — the hive does not choose its own name here, it uses diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 9b8c0adf..090b158c 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -213,7 +213,7 @@ let # not of where any other service runs. Not an assertion: a collector on a # host of its own is a supported shape, and refusing to build it would make # this fix illegal where the bug bites hardest. - haveCollectorSecret = cfg.clientSecretFile != null; + haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; # The operator-configured upstream, named once: the same exporter carries # every signal, so metrics and logs both reach it without a second @@ -497,7 +497,13 @@ in after the change that caused it evaluated cleanly. ''; }; + }; + # What stays above is what the collector IS to the swarm — the client it is + # registered as, where it exports. The secret is a path on the machine that + # runs it, so it hangs off the deployment. `enable` already lives in + # ./deploy.nix, which also carries the rename. + options.services.hyperhive.deploy.swarm-otel = { clientSecretFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; @@ -630,7 +636,7 @@ in # authelia runs — it copies out of its container — so it is also what # names the file. `mkDefault`, so a deployment that delivers the secret # some other way just sets the option. - services.hyperhive.swarm.otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable ( + services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable ( lib.mkDefault collectorSecretInContainer ); @@ -1428,7 +1434,7 @@ in # no authenticator is rendered either, so the collector starts and # is refused by the stores rather than failing to start. // lib.optionalAttrs haveCollectorSecret { - LoadCredential = [ "${collectorCredentialId}:${cfg.clientSecretFile}" ]; + LoadCredential = [ "${collectorCredentialId}:${deployCfg.swarm-otel.clientSecretFile}" ]; }; }; }; diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 8867cc85..cfb577f4 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -356,24 +356,32 @@ in # genuinely cannot publish with two of three coordinates — which # is the distinction the `serviceDomains'` comment at the top of # this file was written about. + # + # The three coordinates live in two namespaces now: the token + # endpoint is the swarm's one address, the other two are this + # machine's. The message spells all three paths out, because an + # operator told only the option names would look for them under + # one prefix and find one of them. assertion = let set = lib.filter (v: v != null) [ - swarmCfg.statusPublish.natsUrl + deployCfg.hive-controller.statusPublish.natsUrl swarmCfg.statusPublish.tokenEndpoint - swarmCfg.statusPublish.clientSecretFile + deployCfg.hive-controller.statusPublish.clientSecretFile ]; in builtins.length set == 0 || builtins.length set == 3; message = '' - services.hyperhive.swarm.statusPublish needs natsUrl, - tokenEndpoint and clientSecretFile set together or not at all - — this hive has only some of them. + This hive's status-publishing coordinates have to be set + together or not at all — it has only some of them. Currently: - natsUrl = ${toString swarmCfg.statusPublish.natsUrl} - tokenEndpoint = ${toString swarmCfg.statusPublish.tokenEndpoint} - clientSecretFile = ${toString swarmCfg.statusPublish.clientSecretFile} + deploy.hive-controller.statusPublish.natsUrl + = ${toString deployCfg.hive-controller.statusPublish.natsUrl} + swarm.statusPublish.tokenEndpoint + = ${toString swarmCfg.statusPublish.tokenEndpoint} + deploy.hive-controller.statusPublish.clientSecretFile + = ${toString deployCfg.hive-controller.statusPublish.clientSecretFile} Set the missing ones to publish this hive's status to the swarm, or set all three to null to turn publishing off. @@ -440,6 +448,38 @@ in # enable. An extra flag would let a hive be configured-but-off, which # is one more state to explain and one more way to be silently quiet. options.services.hyperhive.swarm.statusPublish = { + tokenEndpoint = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = + if queueLocal && swarmCfg.authelia.url != null then + "${swarmCfg.authelia.url}/api/oidc/token" + else + null; + defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null''; + example = "https://auth.example.com/api/oidc/token"; + description = '' + The swarm IdP's OAuth2 token endpoint. This hive mints a + `client_credentials` access token there and presents it when + connecting to the queue, which authenticates it as + `hive-` — the client + {file}`nix/host-modules/swarm-authelia.nix` already declares for + every entry in {option}`services.hyperhive.swarm.hives`. + ''; + }; + }; + + # What stays above is the one thing every hive shares: the address of the + # swarm's token endpoint. The other two coordinates are this machine's — + # where the queue is *as seen from here*, and where its secret sits on this + # disk — so they hang off the daemon that uses them. `hive-controller` is + # hive-c0re's new name, which is why the namespace is the daemon's rather + # than a `deploy.statusPublish` of its own. + # + # ⚠️ The all-or-nothing assertion above now spans both namespaces. That is + # one service's own options either side of the split, not a service reaching + # into a foreign `deploy.*`; its message names all three paths so an operator + # is not told to set two options under a path that has one. + options.services.hyperhive.deploy.hive-controller.statusPublish = { natsUrl = lib.mkOption { type = lib.types.nullOr lib.types.str; default = if queueLocal then "nats://127.0.0.1:${toString swarmCfg.nats.port}" else null; @@ -461,25 +501,6 @@ in ''; }; - tokenEndpoint = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = - if queueLocal && swarmCfg.authelia.url != null then - "${swarmCfg.authelia.url}/api/oidc/token" - else - null; - defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null''; - example = "https://auth.example.com/api/oidc/token"; - description = '' - The swarm IdP's OAuth2 token endpoint. This hive mints a - `client_credentials` access token there and presents it when - connecting to the queue, which authenticates it as - `hive-` — the client - {file}`nix/host-modules/swarm-authelia.nix` already declares for - every entry in {option}`services.hyperhive.swarm.hives`. - ''; - }; - clientSecretFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = diff --git a/nix/module-eval.nix b/nix/module-eval.nix index d95f42a5..44b1561e 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -159,6 +159,18 @@ let swarm.grafana.plugins = [ ]; }; + # The first slice to leave options on BOTH sides of the split, so the + # fixture sets all three of them through the paths an existing config uses: + # the two movers via their rename entries, `tokenEndpoint` via the path it + # kept. That is also what the all-or-nothing assertion wants, so this hive + # is a valid one rather than one that only evaluates because nothing forced + # the assertion. + statusPublishOldPath = hive { + swarm.statusPublish.natsUrl = "nats://10.0.0.9:4222"; + swarm.statusPublish.tokenEndpoint = "https://auth.example.invalid/api/oidc/token"; + swarm.statusPublish.clientSecretFile = "/run/secrets/status-client.secret"; + }; + baoPkcs11 = hive { deploy.bao.enable = true; deploy.bao.seal = "pkcs11"; @@ -249,9 +261,19 @@ let # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. + # + # The `swarm.otel.clientSecretFile` below is the PRE-RENAME path. It predates + # the split and is deliberately left spelled that way: it makes this fixture + # the old-path case for that option too, so dropping its rename entry fails + # the eval here rather than only in a real operator's config. otelRemoteAuthelia = hive { deploy.swarm-otel.enable = true; deploy.authelia.enable = false; + # Where that elsewhere IS. Running no IdP does not mean knowing no IdP: + # the authenticator this fixture exists to render puts this address in its + # `token_url`, so a hive with a secret and no URL has a credential it can + # present nowhere. + swarm.authelia.url = "https://auth.example.invalid"; swarm.otel.clientSecretFile = "/var/lib/swarm-otel-oidc/by-hand.secret"; }; @@ -376,6 +398,36 @@ let rule: lib.hasInfix "/run/test-grafana-sock" rule ) grafanaOldPath.systemd.tmpfiles.rules; } + { + # Reads the daemon's rendered unit, not the options: the queue address + # arrives as an env var whose whole attrset is guarded on `natsUrl`, and + # the secret as a systemd credential. A rename that resolved but stopped + # reaching the module leaves hive-c0re coming up perfectly and reporting + # to nobody, which is the one failure this option set exists to prevent. + # The third arm is the option that did NOT move, read out of the same + # attrset: the guard and the value beside it now come from different + # namespaces, so a hive that renders one and drops the other is exactly + # what a split can silently produce. + name = "a config written against the pre-rename statusPublish paths still reaches the daemon"; + ok = + let + u = statusPublishOldPath.systemd.services.hive-c0re; + in + u.environment.HIVE_C0RE_NATS_URL == "nats://10.0.0.9:4222" + && u.environment.HIVE_C0RE_OIDC_TOKEN_ENDPOINT == "https://auth.example.invalid/api/oidc/token" + && lib.any (c: lib.hasInfix "/run/secrets/status-client.secret" c) u.serviceConfig.LoadCredential; + } + { + # The collector's half of the same split, and a different arm from the + # authenticator case below: this one reads the PATH the unit loads, so a + # reader left on a source that is non-null but wrong still fails. The + # fixture spells the option its pre-rename way, so it covers the rename + # entry at the same time. + name = "a config written against the pre-rename otel secret path still loads it as a credential"; + ok = + lib.any (c: lib.hasInfix "/var/lib/swarm-otel-oidc/by-hand.secret" c) + otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential; + } { name = "a config written against the pre-rename authelia usersFile still reaches the bridge"; ok =