diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index ec69baea..23893245 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -41,23 +41,23 @@ neither is a renaming of the other. ## Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | -| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | -| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | -| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | -| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | -| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | +| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | +| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | +| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | +| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | +| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. **That's the test worth @@ -72,27 +72,27 @@ an all-local swarm it reads authelia's file where it lies and no second copy is made. On any other topology it's an ordinary "operator provides the file" case — see `services.hyperhive.otel.clientSecretFile`. -The **swarm's** collector does need one. It runs in a container, so -`swarm-otel-oidc-secret.service` places its copy, landing at -`/var/lib/swarm-otel-oidc/.secret` — the same shape as the forge and -homeserver rows below, and for the same reason: the container that mints the -secret isn't the container that reads it. +The **swarm's** collector does need one, and its delivery route is the fourth +store reader — the same shape as Grafana's, described in full below. In short: +`swarm-bao-otel-oidc.service` reads `swarm/services//oidc/client` +out of the store on every host that runs the collector and holds a store +identity, whether authelia is a network away or in the container next door. +The client it reads for is registered unconditionally, the same call +`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a +bearer-authz client with no audience, and the push audiences supply one even +for a collector with nothing published to scrape. -The copy is made whenever authelia is enabled **on this host**, whatever this -collector scrapes. The client it delivers for is registered unconditionally: -authelia refuses a bearer-authz client with no audience, and the push audiences -supply one even for a collector with nothing published to scrape. Like the -hive-level delivery units below, this one waits a bounded 120s for authelia's -mint and then fails rather than skipping, so a secret that never arrives takes -the collector's container start with it instead of surfacing later as a 401. - -⚠️ **Don't read that delivery unit as the only way this collector is fed.** -Whether it authenticates follows the _credential_, never another service's -placement: a swarm collector may run on a host that holds neither store and no -authelia, and then the secret is an ordinary operator-provided file named in -`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive -collector's row above. The copy unit is the convenience for the co-located case, -not the definition of the case. +⚠️ **A collector with no store identity is not refused.** Unlike Grafana, +where an absent OIDC block is a container with no way in at all, a collector +with nothing to authenticate with still receives every hive's telemetry — +only its own pushes to the stores go out unauthenticated and get refused +there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without +`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice +`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for +their own optional readers, rather than the hard assertion `swarm-grafana.nix` +uses for its non-optional one. The secret can still be delivered by hand, at +`services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this +unit never reaches. ### Minting the queue's callout nkeys @@ -212,6 +212,18 @@ nothing said anything. Registration is separate and stays where authelia is so it's declared on the host running authelia, which isn't necessarily the host running Grafana. +The **fourth reader** is the swarm collector's own OIDC client secret, the +same shape one level down: `swarm-bao-otel-oidc.service` reads +`swarm/services//oidc/client` out of the store on every host that +runs the collector, and registration lives separately in +`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it +differs is what an absent credential means: the collector still receives +telemetry with none, so `swarm-otel.nix` doesn't assert +`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it simply +doesn't render the reading unit without them, the shape +`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for +their own optional reads. + A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs @@ -220,7 +232,7 @@ there is nothing to scope the grant to. Every hive can therefore read every swar service's client secret; that's the cost of a shape where the service's host has no store identity of its own to present, and it's stated in `swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves -as it does for the other two readers: a store that says "not here" leaves the +as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index 4d71ee7c..14007901 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -66,16 +66,17 @@ _own_ entry under the hive namespace, which names the hive asking and so widens nothing between them, and every **swarm service's** OIDC client secret. That third prefix has the same shape of reason as the first, and the same honest -cost. A swarm service (Grafana is the one there today) registers **one** client -for the whole swarm, so its credential's path names the service and never the -host — and which hive runs a given service is a `deploy.*` fact, per-host by -definition, so nothing swarm-wide exists to scope the grant to. The host running -such a service has no store identity of its own either; it reads with the -certificate of the hive it is. **Any hive can therefore read any swarm service's -client secret**, which is worth what it buys: every swarm service gets its -credential the same way from anywhere, instead of only where the identity -provider happens to sit. Giving such a service its own store identity is what -would remove this rather than re-scope it. +cost. A swarm service (Grafana and the swarm collector are the two there today) +registers **one** client for the whole swarm, so its credential's path names +the service and never the host — and which hive runs a given service is a +`deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope +the grant to. The host running such a service has no store identity of its own +either; it reads with the certificate of the hive it is. **Any hive can +therefore read any swarm service's client secret**, which is worth what it +buys: every swarm service gets its credential the same way from anywhere, +instead of only where the identity provider happens to sit. Giving such a +service its own store identity is what would remove this rather than re-scope +it. **A tracked follow-up** narrows this, with the two candidate directions: scope the grant per hive (and pay for the re-emission), or give each agent container diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 5c626d84..0161ea2d 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -29,6 +29,7 @@ ./glue-matrix-bao-token.nix ./glue-queue-agent-credential.nix ./glue-secret-publisher-bao-identity.nix + ./glue-swarm-otel-oidc-client.nix ./swarm-authelia.nix ./swarm-bao.nix ./swarm-ca.nix diff --git a/nix/host-modules/glue-swarm-otel-oidc-client.nix b/nix/host-modules/glue-swarm-otel-oidc-client.nix new file mode 100644 index 00000000..a10eee4b --- /dev/null +++ b/nix/host-modules/glue-swarm-otel-oidc-client.nix @@ -0,0 +1,58 @@ +# Glue: register the swarm's collector as an OIDC client wherever authelia +# runs. +# +# ONE PAIRING PER FILE — swarm-otel ← authelia, and nothing else. Deleting +# this leaves a swarm whose collector is not a client authelia has ever heard +# of, so no token it presents is ever accepted and authelia mints no secret +# for the publisher to send on. +# +# ⚠️ Gated on authelia being HERE, and deliberately NOT on this host running +# the collector. A client is a row in THIS host's provider config, so it can +# only be declared where that config is rendered — and ./swarm-otel.nix's +# whole `config` block hangs off `deploy.swarm-otel.enable`, so a swarm with +# the collector and authelia on different hosts registered the client +# nowhere at all. Same bug, same fix, as ./glue-grafana-oidc-client.nix one +# module over — read that file's own comment for the property this one +# shares with it. +# +# ⚠️ Registered whether or not the swarm has a collector, for the same +# reason as Grafana's: nothing in `swarm.*` records that one exists, +# `deploy.swarm-otel.enable` only answers "does THIS host run it". The cost +# is one unused client and one unused minted secret in a swarm with no +# collector — the same trade ./swarm-otel.nix already made for this exact +# client when it dropped the published-scrape-target guard on registering it. +{ + lib, + config, + ... +}: +let + hyperhiveCfg = config.services.hyperhive; + deployCfg = hyperhiveCfg.deploy; + otelCfg = hyperhiveCfg.swarm.otel; +in +{ + config = lib.mkIf (hyperhiveCfg.enable && deployCfg.authelia.enable) { + # One declaration, two readers: `clientId` and `audience` are read-only + # options ./swarm-otel.nix derives from the scrape/push targets it owns, + # so this file states neither formula a second time. + services.hyperhive.swarm.authelia.oidc.clients = [ + { + id = otelCfg.clientId; + description = "HyperHive swarm collector"; + kind = "machine"; + # Grants `authelia.bearer.authz`, without which the authz endpoint + # refuses an otherwise valid token and blames the token rather than + # the missing grant. + bearerAuthz = true; + audience = otelCfg.audience; + # Stated rather than left on authelia's default, because the two + # agreeing today is not the same as this being the required value: + # authelia permits only basic / JWT methods for a confidential + # client holding this scope, and enforces it in the startup + # validator. + tokenEndpointAuthMethod = "client_secret_basic"; + } + ]; + }; +} diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 51a6dd4b..dff944ea 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -23,6 +23,8 @@ let vlCfg = config.services.hyperhive.swarm.victorialogs; hyperhiveCfg = config.services.hyperhive; gatewayCfg = hyperhiveCfg.gateway; + baoCfg = hyperhiveCfg.swarm.bao; + baoDeploy = deployCfg.bao; swarmDomain = hyperhiveCfg.swarm.domain; # Total on a null swarm domain for the same reason every sibling module is: @@ -132,6 +134,13 @@ let # delivery unit on a later boot, and a secret that evaporates on reboot # turns a working scrape into an intermittent one. collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret"; + # The same file seen from the host, which is where the delivery unit + # writes it — spelled once for the same reason ./swarm-grafana.nix spells + # its own pair once: the unit that writes it and the option that names it + # are a few hundred lines apart, and a collector reading a path nothing + # writes is an export that fails with nothing in any log about the file. + collectorHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"; + collectorHostSecretDir = builtins.dirOf collectorHostSecretPath; collectorCredentialId = "oidc-client-secret"; # What the scrape config points at. ⚠️ This path and the `LoadCredential` # id below are one fact spelled twice by systemd's design — both derive from @@ -206,11 +215,32 @@ let pushAuthenticator = name: "oauth2client/${name}"; # Whether this collector holds a credential — a property of the credential, - # not of where any other service runs. Not an assertion: a collector on a - # host of its own is a supported shape, and refusing to build it would make - # this fix illegal where the bug bites hardest. + # not of where any other service runs. Not an assertion: a collector that + # pushes nowhere authenticated still receives from every hive, so refusing + # to build one would make a supported shape illegal for want of a value + # that only degrades what it can push. haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; + # A reader of the store is defined by holding a certificate the store + # accepts, never by standing next to it — the rule + # ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- + # looking flag, this one stays outside `assertions`: the store-reading unit + # below simply does not render without it, the same choice + # ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for + # their own optional readers, because a collector with no client identity is + # `haveCollectorSecret = false` above, and that is already a supported, + # merely degraded shape rather than a service with no way in at all. + haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + + # Where the publisher on authelia's host leaves this client's secret — + # composed from the same swarm-wide `clientId` the registration in + # ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of + # them behind. The `services` segment is `swarm-secret-client`'s + # `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own + # client secret under — one write grant in ./swarm-bao.nix and one hive read + # grant in `policy::render` already cover it. + storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client"; + # The operator-configured upstream, named once: the same exporter carries # every signal, so metrics and logs both reach it without a second # definition. @@ -493,6 +523,21 @@ in after the change that caused it evaluated cleanly. ''; }; + + audience = lib.mkOption { + type = lib.types.listOf lib.types.str; + readOnly = true; + default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences; + description = '' + Every audience this collector's OAuth2 client is permitted to + present a token for — the scrape targets it reads with a credential, + plus the stores it pushes to. Published read-only so + ./glue-swarm-otel-oidc-client.nix can register the client wherever + authelia runs without restating the derivation: this option and + that registration are the same fact seen from two hosts, and a + second formula for it would be free to drift from this one. + ''; + }; }; # What stays above is what the collector IS to the swarm — the client it is @@ -529,7 +574,10 @@ in # delivering says so in the store it stopped delivering to. That is # less circular than it sounds: the failure that matters here is a # single hive's receiver refusing pushes, not the process dying. - services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ]; + services.hyperhive.swarm.otel.journaldUnits = [ + "opentelemetry-collector" + "swarm-bao-otel-oidc" + ]; # The metrics counterpart to the journal line above, closing the same # gap from the other side: the journal says the process is alive, these @@ -584,109 +632,152 @@ in # this is how it authenticates *itself* to a service published behind # the gateway. # - # Only where authelia is co-located. A client is a row in this host's - # provider config, so declaring one against a remote provider would - # render nothing while reading as done; a swarm whose authelia lives - # elsewhere registers it there. - # - # ⚠️ NO LONGER conditional on a published scrape target. Authelia refuses - # a bearer-authz client with no audience, which is what that guard was - # for — and the push audiences below are unconditional, so there is now - # always at least one. Keeping the old guard would have left a collector - # that scrapes nothing pushing to the stores with no client to get a - # token from. - services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [ - { - id = cfg.clientId; - description = "HyperHive swarm collector"; - kind = "machine"; - # Grants `authelia.bearer.authz`, without which the authz - # endpoint refuses an otherwise valid token and blames the - # token rather than the missing grant. - bearerAuthz = true; - # DERIVED from the targets rather than contributed alongside - # them. A service declares a URL once and this is the - # permission to reach it; two lists that had to agree would be - # a drift to maintain, and the failure mode is the quiet one — - # a target whose audience was forgotten authenticates against - # nothing and looks like a broken scrape. - # - # Both directions land in one list because authelia has one: what - # this collector may SCRAPE and what it may PUSH TO are the same - # kind of permission, differing only in who initiates. - audience = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences; - # Stated rather than left on authelia's default, because the - # two agreeing today is not the same as this being the - # required value: authelia permits only basic / JWT methods - # for a confidential client holding that scope, and enforces - # it in the startup validator. - tokenEndpointAuthMethod = "client_secret_basic"; - } - ]; + # Registering the client is NOT here any more: it has to happen on the + # host that runs authelia, and this whole block is gated on the host that + # runs the collector. ./glue-swarm-otel-oidc-client.nix is where it moved + # to, the same split ./swarm-grafana.nix made for its own client. - # Deliver the collector's client secret from authelia's container into - # this one. On the HOST because that is the only place both container - # trees are addressable: they share this host's network namespace, which - # makes them feel co-located, but their filesystem roots are separate. + # THE delivery unit — one route, in every deployment. The secret authelia + # minted arrives out of the swarm secret store, which the publisher on + # authelia's host wrote it into, whether authelia is a network away or in + # the container next door. # - # ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container - # refuses to start when a bind source is missing, and this secret does not - # exist until authelia's first boot has minted it — so binding it would - # make the collector wait on a file that waits on a container that starts - # after it. On a fresh swarm that is a permanent stall presenting as - # "metrics are broken", several layers from its cause. - # The delivery unit below is the one thing here that may know where - # authelia runs — it copies out of its container — so it is also what - # names the file. `mkDefault`, so a deployment that delivers the secret - # some other way just sets the option. - services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable ( - lib.mkDefault collectorSecretInContainer - ); - - systemd.services.swarm-otel-oidc-secret = lib.mkIf deployCfg.authelia.enable { - description = "deliver the swarm collector's OIDC client secret from authelia"; - after = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ]; - requires = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ]; + # 🩸 A unit here used to copy the plaintext directly out of authelia's + # host tree, reachable only because they share this host's network + # namespace — which stopped working the moment authelia moved to another + # host, which is the defect this whole change exists to fix. The ruling + # that deleted it rather than gave it a remote sibling: the store exists + # so a host holds ONE out-of-band secret — its client certificate — and + # reads everything else with it. Recorded in docs/swarm/secrets.md. + # + # Shaped after ./glue-queue-agent-credential.nix: a cert login that fails + # LOUDLY, since every state it fails on is one a retry fixes, then a read + # that degrades QUIETLY, since no retry turns "no value there" into a + # value. + # + # ⚠️ Renders only where `haveClientIdentity` holds, unlike + # ./swarm-grafana.nix's equivalent unit. That module asserts the identity + # because a Grafana with none has no way in at all; this collector without + # one is `haveCollectorSecret = false` above — already a supported, + # merely degraded shape, so the unit that would fetch a credential simply + # does not exist rather than refusing the build for want of one. + systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity { + description = "fetch the swarm collector's OIDC client secret from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering + # is conditional even though the read is not: off-host there is nothing + # local to wait for, and the timeout below bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; before = [ "container@${cfg.machine}.service" ]; - wantedBy = [ "container@${cfg.machine}.service" ]; + wantedBy = [ + "multi-user.target" + "container@${cfg.machine}.service" + ]; + path = [ + baoDeploy.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal: `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after, so a few short attempts cover it. An + # hours-long window would be a bet on a sealed store, and the degrade + # below is already correct for that. + # + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; serviceConfig = { Type = "oneshot"; RemainAfterExit = true; - SyslogIdentifier = "swarm-otel-oidc-secret"; - # Longer than the bounded wait below, and that is the point: - # `DefaultTimeoutStartSec` is 90s, so without this systemd kills - # the unit before it can emit the message naming the file it was - # waiting for — the failure then reads as a timeout with no cause. - TimeoutStartSec = "180s"; + SyslogIdentifier = "swarm-bao-otel-oidc"; + # What actually bounds the read below. Stated here rather than left to + # systemd's default, so the number a boot waits on is in the file that + # waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.clientCertFile; + BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; }; - path = [ pkgs.coreutils ]; script = '' set -euo pipefail - src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"} - dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"} + # `bao`'s own message is the only thing separating a missing value + # from a refused identity from an unreachable host. This unit's + # degraded mode is correct for all three, so it reports which one + # rather than asserting all three in a sentence of ours. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT - # authelia's container is up, but its first-boot generator may - # still be minting. Bounded wait, then fail: skipping silently - # produces a collector whose scrape gets a 401 forever, which is - # the failure this whole design exists to make impossible. - for _ in $(seq 1 60); do - [ -s "$src" ] && break - sleep 2 - done - if [ ! -s "$src" ]; then - echo "authelia has not minted $src after 120s" >&2 + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; leaving the collector's OIDC client secret as it is." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi exit 1 fi + export BAO_TOKEN - # root-owned 0400. The collector runs under `DynamicUser`, so - # there is no uid to give it to — `LoadCredential` reads this as - # root before the sandbox exists and re-exposes it to whichever - # uid the unit got. - install -D -m 0400 -o root -g root "$src" "$dst" + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then + echo "swarm-bao did not return ${storeSecretPath}; the collector has no OIDC client secret yet." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 + fi + + if [ -z "$secret" ]; then + echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2 + exit 0 + fi + + # root-owned 0400, written with a shell builtin and never handed to a + # program: `printf` is bash's own, so the plaintext never becomes an + # argument in /proc the way `install <<<"$secret"` or an `echo` from + # `path` would. The collector runs under `DynamicUser`, so there is + # no uid to give this to — `LoadCredential` reads it as root before + # the sandbox exists and re-exposes it to whichever uid the unit got. + install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir} + umask 077 + printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath} + chown root:root ${lib.escapeShellArg collectorHostSecretPath} + chmod 0400 ${lib.escapeShellArg collectorHostSecretPath} ''; }; + # Where the delivery unit above lands the secret. `mkDefault`, so a + # deployment delivering it some other way just sets the option — and + # `mkIf haveClientIdentity` so a host with no store identity is left with + # `clientSecretFile == null`, the already-supported degrade rather than a + # path nothing ever writes. + services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf haveClientIdentity ( + lib.mkDefault collectorSecretInContainer + ); + # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-secret-publisher.nix b/nix/host-modules/swarm-secret-publisher.nix index 4ac1dafe..b373bf58 100644 --- a/nix/host-modules/swarm-secret-publisher.nix +++ b/nix/host-modules/swarm-secret-publisher.nix @@ -64,10 +64,13 @@ let # # The ids come from `swarm.*`, which is identical on every host — that is what # lets this host name a service's client while running none of them, and it is - # the same read the service's own module registers the client with. A swarm - # that runs no Grafana mints no secret for it, so its entry skips below rather - # than needing a condition here. - serviceClientIds = [ hyperhiveCfg.swarm.grafana.oidc.clientId ]; + # the same read each service's own module registers its client with. A swarm + # that runs no Grafana, or no collector, mints no secret for the one it lacks, + # so its entry skips below rather than needing a condition here. + serviceClientIds = [ + hyperhiveCfg.swarm.grafana.oidc.clientId + hyperhiveCfg.swarm.otel.clientId + ]; in { options.services.hyperhive.deploy.swarm-secret-publisher = { @@ -92,10 +95,11 @@ in identity's job (see `baoClientCertFile`), not this option's. Turning it off leaves every hive but this one without its agents' - credential, and the swarm's Grafana without any login at all — its - secret has exactly one route and this is the producer's end of it. So - the honest reason to set it false is a deployment delivering those - secrets by some other mechanism it owns. + credential, the swarm's Grafana without any login at all, and its + collector pushing unauthenticated — each secret has exactly one route + and this is the producer's end of it. So the honest reason to set it + false is a deployment delivering those secrets by some other + mechanism it owns. ''; }; diff --git a/nix/module-eval.nix b/nix/module-eval.nix index d893217d..59b5db5a 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -504,6 +504,35 @@ let otelSettings = machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings; + # The collector beside authelia, reading its own OIDC secret out of the + # store like every other collector — the cert pair here is not scenery, it + # is the arm that would catch the deleted co-located copy unit coming back. + otelBaoWithAuthelia = hive { + deploy.swarm-otel.enable = true; + deploy.authelia.enable = true; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + # The same collector with the IdP on ANOTHER host and a store leaf placed by + # hand. Identical to the fixture above in everything the delivery path + # reads, which is the point. + otelBaoRemoteAuthelia = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + # A collector holding no store identity at all. Unlike Grafana's mirror + # image, this is not a refused shape: the collector still receives every + # hive's telemetry with nothing to push authenticated with, which is the + # already-supported degrade `haveCollectorSecret` names above the module's + # `let`. What this fixture is for is checking the reading unit itself does + # not render, rather than rendering with an env var nothing filled in. + otelNoIdentity = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + }; + # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. @@ -1031,6 +1060,71 @@ let lib.elem "swarm-grafana" (clients secretPublisherHere) && !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia)); } + { + # The collector's half of the same defect and the same fix: this used to + # be gated on `deploy.swarm-otel.enable`, so a split deployment + # registered the client nowhere and authelia minted nothing to publish. + name = "the swarm's collector client is registered wherever authelia runs"; + ok = + let + clients = m: map (c: c.id) m.services.hyperhive.swarm.authelia.oidc.clients; + in + lib.elem "swarm-collector" (clients secretPublisherHere) + && !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia)); + } + { + # 🩸 The arm that guards the ruling this slice landed under, the + # collector's half of ./swarm-grafana.nix's own. There is ONE delivery + # route: the store reader, on every host that runs the collector and + # holds a store identity. The negative names the deleted unit rather + # than a generic absence, because the way this regresses is someone + # re-adding the co-located copy as an optimisation. + name = "the collector's OIDC secret has exactly one delivery unit, the store reader, in both topologies"; + ok = + let + local = otelBaoWithAuthelia.systemd.services; + remote = otelBaoRemoteAuthelia.systemd.services; + in + local ? swarm-bao-otel-oidc + && remote ? swarm-bao-otel-oidc + && !(local ? swarm-otel-oidc-secret) + && !(remote ? swarm-otel-oidc-secret); + } + { + # Same 403-not-a-miss reason as grafana's arm above: the reader's grant + # covers the `services` prefix, so a path outside it is refused rather + # than empty, however correct it reads. + name = "the collector's OIDC secret is read from the prefix the publisher writes"; + ok = + let + s = otelBaoRemoteAuthelia.systemd.services.swarm-bao-otel-oidc.script; + in + lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" s + && !(lib.hasInfix "secret/swarm/hives/" s); + } + { + # Both ends of a wire nothing at eval time carries end to end: the + # publisher on authelia's host writes the path the reader on the + # collector's host reads, and the two files agree only because both + # compose it from the same swarm-wide client id. `secretPublisherHere` + # already grew this client when `serviceClientIds` did. + name = "the publisher writes the swarm service path the collector reads"; + ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" ( + secretPublisherHere.systemd.services.swarm-secret-publish.script + ); + } + { + # The collector's non-assertion, the deliberate mirror of Grafana's + # assertion two cases up: a host with no store identity is a supported, + # merely degraded shape here, so the reading unit simply does not exist + # rather than refusing the build. `haveCollectorSecret` is what the + # degrade already reads, unchanged by this slice. + name = "a collector with no store identity renders no reading unit, and is not refused"; + ok = + !(otelNoIdentity.systemd.services ? swarm-bao-otel-oidc) + && otelNoIdentity.services.hyperhive.deploy.swarm-otel.clientSecretFile == null + && !(lib.any (a: !a.assertion) otelNoIdentity.assertions); + } { # Reads the daemon's rendered unit, not the options: the queue address # arrives as an env var whose whole attrset is guarded on `natsUrl`, and