diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 24481b57..ec69baea 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -41,23 +41,23 @@ neither is a renaming of the other. ## Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | -| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | -| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | -| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | -| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | -| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | +| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | +| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | +| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | +| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | +| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. **That's the test worth @@ -72,27 +72,27 @@ an all-local swarm it reads authelia's file where it lies and no second copy is made. On any other topology it's an ordinary "operator provides the file" case — see `services.hyperhive.otel.clientSecretFile`. -The **swarm's** collector does need one, and its delivery route is the fourth -store reader — the same shape as Grafana's, described in full below. In short: -`swarm-bao-otel-oidc.service` reads `swarm/services//oidc/client` -out of the store on every host that runs the collector and holds a store -identity, whether authelia is a network away or in the container next door. -The client it reads for is registered unconditionally, the same call -`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a -bearer-authz client with no audience, and the push audiences supply one even -for a collector with nothing published to scrape. +The **swarm's** collector does need one. It runs in a container, so +`swarm-otel-oidc-secret.service` places its copy, landing at +`/var/lib/swarm-otel-oidc/.secret` — the same shape as the forge and +homeserver rows below, and for the same reason: the container that mints the +secret isn't the container that reads it. -⚠️ **A collector with no store identity isn't refused.** Unlike Grafana, -where an absent OIDC block is a container with no way in at all, a collector -with nothing to authenticate with still receives every hive's telemetry — -only its own pushes to the stores go out unauthenticated and get refused -there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without -`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice -`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for -their own optional readers, rather than the hard assertion `swarm-grafana.nix` -uses for its non-optional one. The secret can still be delivered by hand, at -`services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this -unit never reaches. +The copy is made whenever authelia is enabled **on this host**, whatever this +collector scrapes. The client it delivers for is registered unconditionally: +authelia refuses a bearer-authz client with no audience, and the push audiences +supply one even for a collector with nothing published to scrape. Like the +hive-level delivery units below, this one waits a bounded 120s for authelia's +mint and then fails rather than skipping, so a secret that never arrives takes +the collector's container start with it instead of surfacing later as a 401. + +⚠️ **Don't read that delivery unit as the only way this collector is fed.** +Whether it authenticates follows the _credential_, never another service's +placement: a swarm collector may run on a host that holds neither store and no +authelia, and then the secret is an ordinary operator-provided file named in +`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive +collector's row above. The copy unit is the convenience for the co-located case, +not the definition of the case. ### Minting the queue's callout nkeys @@ -212,18 +212,6 @@ nothing said anything. Registration is separate and stays where authelia is so it's declared on the host running authelia, which isn't necessarily the host running Grafana. -The **fourth reader** is the swarm collector's own OIDC client secret, the -same shape one level down: `swarm-bao-otel-oidc.service` reads -`swarm/services//oidc/client` out of the store on every host that -runs the collector, and registration lives separately in -`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it -differs is what an absent credential means: the collector still receives -telemetry with none, so `swarm-otel.nix` doesn't assert -`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it -doesn't render the reading unit without them, the shape -`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for -their own optional reads. - A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs @@ -232,7 +220,7 @@ there is nothing to scope the grant to. Every hive can therefore read every swar service's client secret; that's the cost of a shape where the service's host has no store identity of its own to present, and it's stated in `swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves -as it does for the other readers: a store that says "not here" leaves the +as it does for the other two readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index 14007901..4d71ee7c 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -66,17 +66,16 @@ _own_ entry under the hive namespace, which names the hive asking and so widens nothing between them, and every **swarm service's** OIDC client secret. That third prefix has the same shape of reason as the first, and the same honest -cost. A swarm service (Grafana and the swarm collector are the two there today) -registers **one** client for the whole swarm, so its credential's path names -the service and never the host — and which hive runs a given service is a -`deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope -the grant to. The host running such a service has no store identity of its own -either; it reads with the certificate of the hive it is. **Any hive can -therefore read any swarm service's client secret**, which is worth what it -buys: every swarm service gets its credential the same way from anywhere, -instead of only where the identity provider happens to sit. Giving such a -service its own store identity is what would remove this rather than re-scope -it. +cost. A swarm service (Grafana is the one there today) registers **one** client +for the whole swarm, so its credential's path names the service and never the +host — and which hive runs a given service is a `deploy.*` fact, per-host by +definition, so nothing swarm-wide exists to scope the grant to. The host running +such a service has no store identity of its own either; it reads with the +certificate of the hive it is. **Any hive can therefore read any swarm service's +client secret**, which is worth what it buys: every swarm service gets its +credential the same way from anywhere, instead of only where the identity +provider happens to sit. Giving such a service its own store identity is what +would remove this rather than re-scope it. **A tracked follow-up** narrows this, with the two candidate directions: scope the grant per hive (and pay for the re-emission), or give each agent container diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 0161ea2d..5c626d84 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -29,7 +29,6 @@ ./glue-matrix-bao-token.nix ./glue-queue-agent-credential.nix ./glue-secret-publisher-bao-identity.nix - ./glue-swarm-otel-oidc-client.nix ./swarm-authelia.nix ./swarm-bao.nix ./swarm-ca.nix diff --git a/nix/host-modules/glue-swarm-otel-oidc-client.nix b/nix/host-modules/glue-swarm-otel-oidc-client.nix deleted file mode 100644 index a10eee4b..00000000 --- a/nix/host-modules/glue-swarm-otel-oidc-client.nix +++ /dev/null @@ -1,58 +0,0 @@ -# Glue: register the swarm's collector as an OIDC client wherever authelia -# runs. -# -# ONE PAIRING PER FILE — swarm-otel ← authelia, and nothing else. Deleting -# this leaves a swarm whose collector is not a client authelia has ever heard -# of, so no token it presents is ever accepted and authelia mints no secret -# for the publisher to send on. -# -# ⚠️ Gated on authelia being HERE, and deliberately NOT on this host running -# the collector. A client is a row in THIS host's provider config, so it can -# only be declared where that config is rendered — and ./swarm-otel.nix's -# whole `config` block hangs off `deploy.swarm-otel.enable`, so a swarm with -# the collector and authelia on different hosts registered the client -# nowhere at all. Same bug, same fix, as ./glue-grafana-oidc-client.nix one -# module over — read that file's own comment for the property this one -# shares with it. -# -# ⚠️ Registered whether or not the swarm has a collector, for the same -# reason as Grafana's: nothing in `swarm.*` records that one exists, -# `deploy.swarm-otel.enable` only answers "does THIS host run it". The cost -# is one unused client and one unused minted secret in a swarm with no -# collector — the same trade ./swarm-otel.nix already made for this exact -# client when it dropped the published-scrape-target guard on registering it. -{ - lib, - config, - ... -}: -let - hyperhiveCfg = config.services.hyperhive; - deployCfg = hyperhiveCfg.deploy; - otelCfg = hyperhiveCfg.swarm.otel; -in -{ - config = lib.mkIf (hyperhiveCfg.enable && deployCfg.authelia.enable) { - # One declaration, two readers: `clientId` and `audience` are read-only - # options ./swarm-otel.nix derives from the scrape/push targets it owns, - # so this file states neither formula a second time. - services.hyperhive.swarm.authelia.oidc.clients = [ - { - id = otelCfg.clientId; - description = "HyperHive swarm collector"; - kind = "machine"; - # Grants `authelia.bearer.authz`, without which the authz endpoint - # refuses an otherwise valid token and blames the token rather than - # the missing grant. - bearerAuthz = true; - audience = otelCfg.audience; - # Stated rather than left on authelia's default, because the two - # agreeing today is not the same as this being the required value: - # authelia permits only basic / JWT methods for a confidential - # client holding this scope, and enforces it in the startup - # validator. - tokenEndpointAuthMethod = "client_secret_basic"; - } - ]; - }; -} diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index dff944ea..51a6dd4b 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -23,8 +23,6 @@ let vlCfg = config.services.hyperhive.swarm.victorialogs; hyperhiveCfg = config.services.hyperhive; gatewayCfg = hyperhiveCfg.gateway; - baoCfg = hyperhiveCfg.swarm.bao; - baoDeploy = deployCfg.bao; swarmDomain = hyperhiveCfg.swarm.domain; # Total on a null swarm domain for the same reason every sibling module is: @@ -134,13 +132,6 @@ let # delivery unit on a later boot, and a secret that evaporates on reboot # turns a working scrape into an intermittent one. collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret"; - # The same file seen from the host, which is where the delivery unit - # writes it — spelled once for the same reason ./swarm-grafana.nix spells - # its own pair once: the unit that writes it and the option that names it - # are a few hundred lines apart, and a collector reading a path nothing - # writes is an export that fails with nothing in any log about the file. - collectorHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"; - collectorHostSecretDir = builtins.dirOf collectorHostSecretPath; collectorCredentialId = "oidc-client-secret"; # What the scrape config points at. ⚠️ This path and the `LoadCredential` # id below are one fact spelled twice by systemd's design — both derive from @@ -215,32 +206,11 @@ let pushAuthenticator = name: "oauth2client/${name}"; # Whether this collector holds a credential — a property of the credential, - # not of where any other service runs. Not an assertion: a collector that - # pushes nowhere authenticated still receives from every hive, so refusing - # to build one would make a supported shape illegal for want of a value - # that only degrades what it can push. + # not of where any other service runs. Not an assertion: a collector on a + # host of its own is a supported shape, and refusing to build it would make + # this fix illegal where the bug bites hardest. haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; - # A reader of the store is defined by holding a certificate the store - # accepts, never by standing next to it — the rule - # ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- - # looking flag, this one stays outside `assertions`: the store-reading unit - # below simply does not render without it, the same choice - # ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for - # their own optional readers, because a collector with no client identity is - # `haveCollectorSecret = false` above, and that is already a supported, - # merely degraded shape rather than a service with no way in at all. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; - - # Where the publisher on authelia's host leaves this client's secret — - # composed from the same swarm-wide `clientId` the registration in - # ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of - # them behind. The `services` segment is `swarm-secret-client`'s - # `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own - # client secret under — one write grant in ./swarm-bao.nix and one hive read - # grant in `policy::render` already cover it. - storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client"; - # The operator-configured upstream, named once: the same exporter carries # every signal, so metrics and logs both reach it without a second # definition. @@ -523,21 +493,6 @@ in after the change that caused it evaluated cleanly. ''; }; - - audience = lib.mkOption { - type = lib.types.listOf lib.types.str; - readOnly = true; - default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences; - description = '' - Every audience this collector's OAuth2 client is permitted to - present a token for — the scrape targets it reads with a credential, - plus the stores it pushes to. Published read-only so - ./glue-swarm-otel-oidc-client.nix can register the client wherever - authelia runs without restating the derivation: this option and - that registration are the same fact seen from two hosts, and a - second formula for it would be free to drift from this one. - ''; - }; }; # What stays above is what the collector IS to the swarm — the client it is @@ -574,10 +529,7 @@ in # delivering says so in the store it stopped delivering to. That is # less circular than it sounds: the failure that matters here is a # single hive's receiver refusing pushes, not the process dying. - services.hyperhive.swarm.otel.journaldUnits = [ - "opentelemetry-collector" - "swarm-bao-otel-oidc" - ]; + services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ]; # The metrics counterpart to the journal line above, closing the same # gap from the other side: the journal says the process is alive, these @@ -632,152 +584,109 @@ in # this is how it authenticates *itself* to a service published behind # the gateway. # - # Registering the client is NOT here any more: it has to happen on the - # host that runs authelia, and this whole block is gated on the host that - # runs the collector. ./glue-swarm-otel-oidc-client.nix is where it moved - # to, the same split ./swarm-grafana.nix made for its own client. + # Only where authelia is co-located. A client is a row in this host's + # provider config, so declaring one against a remote provider would + # render nothing while reading as done; a swarm whose authelia lives + # elsewhere registers it there. + # + # ⚠️ NO LONGER conditional on a published scrape target. Authelia refuses + # a bearer-authz client with no audience, which is what that guard was + # for — and the push audiences below are unconditional, so there is now + # always at least one. Keeping the old guard would have left a collector + # that scrapes nothing pushing to the stores with no client to get a + # token from. + services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [ + { + id = cfg.clientId; + description = "HyperHive swarm collector"; + kind = "machine"; + # Grants `authelia.bearer.authz`, without which the authz + # endpoint refuses an otherwise valid token and blames the + # token rather than the missing grant. + bearerAuthz = true; + # DERIVED from the targets rather than contributed alongside + # them. A service declares a URL once and this is the + # permission to reach it; two lists that had to agree would be + # a drift to maintain, and the failure mode is the quiet one — + # a target whose audience was forgotten authenticates against + # nothing and looks like a broken scrape. + # + # Both directions land in one list because authelia has one: what + # this collector may SCRAPE and what it may PUSH TO are the same + # kind of permission, differing only in who initiates. + audience = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences; + # Stated rather than left on authelia's default, because the + # two agreeing today is not the same as this being the + # required value: authelia permits only basic / JWT methods + # for a confidential client holding that scope, and enforces + # it in the startup validator. + tokenEndpointAuthMethod = "client_secret_basic"; + } + ]; - # THE delivery unit — one route, in every deployment. The secret authelia - # minted arrives out of the swarm secret store, which the publisher on - # authelia's host wrote it into, whether authelia is a network away or in - # the container next door. + # Deliver the collector's client secret from authelia's container into + # this one. On the HOST because that is the only place both container + # trees are addressable: they share this host's network namespace, which + # makes them feel co-located, but their filesystem roots are separate. # - # 🩸 A unit here used to copy the plaintext directly out of authelia's - # host tree, reachable only because they share this host's network - # namespace — which stopped working the moment authelia moved to another - # host, which is the defect this whole change exists to fix. The ruling - # that deleted it rather than gave it a remote sibling: the store exists - # so a host holds ONE out-of-band secret — its client certificate — and - # reads everything else with it. Recorded in docs/swarm/secrets.md. - # - # Shaped after ./glue-queue-agent-credential.nix: a cert login that fails - # LOUDLY, since every state it fails on is one a retry fixes, then a read - # that degrades QUIETLY, since no retry turns "no value there" into a - # value. - # - # ⚠️ Renders only where `haveClientIdentity` holds, unlike - # ./swarm-grafana.nix's equivalent unit. That module asserts the identity - # because a Grafana with none has no way in at all; this collector without - # one is `haveCollectorSecret = false` above — already a supported, - # merely degraded shape, so the unit that would fetch a credential simply - # does not exist rather than refusing the build for want of one. - systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity { - description = "fetch the swarm collector's OIDC client secret from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering - # is conditional even though the read is not: off-host there is nothing - # local to wait for, and the timeout below bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" - ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + # ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container + # refuses to start when a bind source is missing, and this secret does not + # exist until authelia's first boot has minted it — so binding it would + # make the collector wait on a file that waits on a container that starts + # after it. On a fresh swarm that is a permanent stall presenting as + # "metrics are broken", several layers from its cause. + # The delivery unit below is the one thing here that may know where + # authelia runs — it copies out of its container — so it is also what + # names the file. `mkDefault`, so a deployment that delivers the secret + # some other way just sets the option. + services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable ( + lib.mkDefault collectorSecretInContainer + ); + + systemd.services.swarm-otel-oidc-secret = lib.mkIf deployCfg.authelia.enable { + description = "deliver the swarm collector's OIDC client secret from authelia"; + after = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ]; + requires = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ]; before = [ "container@${cfg.machine}.service" ]; - wantedBy = [ - "multi-user.target" - "container@${cfg.machine}.service" - ]; - path = [ - baoDeploy.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal: `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after, so a few short attempts cover it. An - # hours-long window would be a bet on a sealed store, and the degrade - # below is already correct for that. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; + wantedBy = [ "container@${cfg.machine}.service" ]; serviceConfig = { Type = "oneshot"; RemainAfterExit = true; - SyslogIdentifier = "swarm-bao-otel-oidc"; - # What actually bounds the read below. Stated here rather than left to - # systemd's default, so the number a boot waits on is in the file that - # waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; + SyslogIdentifier = "swarm-otel-oidc-secret"; + # Longer than the bounded wait below, and that is the point: + # `DefaultTimeoutStartSec` is 90s, so without this systemd kills + # the unit before it can emit the message naming the file it was + # waiting for — the failure then reads as a timeout with no cause. + TimeoutStartSec = "180s"; }; + path = [ pkgs.coreutils ]; script = '' set -euo pipefail - # `bao`'s own message is the only thing separating a missing value - # from a refused identity from an unreachable host. This unit's - # degraded mode is correct for all three, so it reports which one - # rather than asserting all three in a sentence of ours. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"} + dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"} - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; leaving the collector's OIDC client secret as it is." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi + # authelia's container is up, but its first-boot generator may + # still be minting. Bounded wait, then fail: skipping silently + # produces a collector whose scrape gets a 401 forever, which is + # the failure this whole design exists to make impossible. + for _ in $(seq 1 60); do + [ -s "$src" ] && break + sleep 2 + done + if [ ! -s "$src" ]; then + echo "authelia has not minted $src after 120s" >&2 exit 1 fi - export BAO_TOKEN - if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then - echo "swarm-bao did not return ${storeSecretPath}; the collector has no OIDC client secret yet." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 0 - fi - - if [ -z "$secret" ]; then - echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2 - exit 0 - fi - - # root-owned 0400, written with a shell builtin and never handed to a - # program: `printf` is bash's own, so the plaintext never becomes an - # argument in /proc the way `install <<<"$secret"` or an `echo` from - # `path` would. The collector runs under `DynamicUser`, so there is - # no uid to give this to — `LoadCredential` reads it as root before - # the sandbox exists and re-exposes it to whichever uid the unit got. - install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir} - umask 077 - printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath} - chown root:root ${lib.escapeShellArg collectorHostSecretPath} - chmod 0400 ${lib.escapeShellArg collectorHostSecretPath} + # root-owned 0400. The collector runs under `DynamicUser`, so + # there is no uid to give it to — `LoadCredential` reads this as + # root before the sandbox exists and re-exposes it to whichever + # uid the unit got. + install -D -m 0400 -o root -g root "$src" "$dst" ''; }; - # Where the delivery unit above lands the secret. `mkDefault`, so a - # deployment delivering it some other way just sets the option — and - # `mkIf haveClientIdentity` so a host with no store identity is left with - # `clientSecretFile == null`, the already-supported degrade rather than a - # path nothing ever writes. - services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf haveClientIdentity ( - lib.mkDefault collectorSecretInContainer - ); - # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-secret-publisher.nix b/nix/host-modules/swarm-secret-publisher.nix index b373bf58..4ac1dafe 100644 --- a/nix/host-modules/swarm-secret-publisher.nix +++ b/nix/host-modules/swarm-secret-publisher.nix @@ -64,13 +64,10 @@ let # # The ids come from `swarm.*`, which is identical on every host — that is what # lets this host name a service's client while running none of them, and it is - # the same read each service's own module registers its client with. A swarm - # that runs no Grafana, or no collector, mints no secret for the one it lacks, - # so its entry skips below rather than needing a condition here. - serviceClientIds = [ - hyperhiveCfg.swarm.grafana.oidc.clientId - hyperhiveCfg.swarm.otel.clientId - ]; + # the same read the service's own module registers the client with. A swarm + # that runs no Grafana mints no secret for it, so its entry skips below rather + # than needing a condition here. + serviceClientIds = [ hyperhiveCfg.swarm.grafana.oidc.clientId ]; in { options.services.hyperhive.deploy.swarm-secret-publisher = { @@ -95,11 +92,10 @@ in identity's job (see `baoClientCertFile`), not this option's. Turning it off leaves every hive but this one without its agents' - credential, the swarm's Grafana without any login at all, and its - collector pushing unauthenticated — each secret has exactly one route - and this is the producer's end of it. So the honest reason to set it - false is a deployment delivering those secrets by some other - mechanism it owns. + credential, and the swarm's Grafana without any login at all — its + secret has exactly one route and this is the producer's end of it. So + the honest reason to set it false is a deployment delivering those + secrets by some other mechanism it owns. ''; }; diff --git a/nix/module-eval.nix b/nix/module-eval.nix index e764d29f..d893217d 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -498,44 +498,12 @@ let otelNoStores = hive { deploy.swarm-otel.enable = true; deploy.authelia.enable = true; - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.victoriametrics.enable = false; deploy.victorialogs.enable = false; }; otelSettings = machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings; - # The collector beside authelia, reading its own OIDC secret out of the - # store like every other collector — the cert pair here is not scenery, it - # is the arm that would catch the deleted co-located copy unit coming back. - otelBaoWithAuthelia = hive { - deploy.swarm-otel.enable = true; - deploy.authelia.enable = true; - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - }; - # The same collector with the IdP on ANOTHER host and a store leaf placed by - # hand. Identical to the fixture above in everything the delivery path - # reads, which is the point. - otelBaoRemoteAuthelia = hive { - deploy.swarm-otel.enable = true; - swarm.authelia.url = "https://auth.example.invalid"; - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - }; - # A collector holding no store identity at all. Unlike Grafana's mirror - # image, this is not a refused shape: the collector still receives every - # hive's telemetry with nothing to push authenticated with, which is the - # already-supported degrade `haveCollectorSecret` names above the module's - # `let`. What this fixture is for is checking the reading unit itself does - # not render, rather than rendering with an env var nothing filled in. - otelNoIdentity = hive { - deploy.swarm-otel.enable = true; - swarm.authelia.url = "https://auth.example.invalid"; - deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; - }; - # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. @@ -1063,71 +1031,6 @@ let lib.elem "swarm-grafana" (clients secretPublisherHere) && !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia)); } - { - # The collector's half of the same defect and the same fix: this used to - # be gated on `deploy.swarm-otel.enable`, so a split deployment - # registered the client nowhere and authelia minted nothing to publish. - name = "the swarm's collector client is registered wherever authelia runs"; - ok = - let - clients = m: map (c: c.id) m.services.hyperhive.swarm.authelia.oidc.clients; - in - lib.elem "swarm-collector" (clients secretPublisherHere) - && !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia)); - } - { - # 🩸 The arm that guards the ruling this slice landed under, the - # collector's half of ./swarm-grafana.nix's own. There is ONE delivery - # route: the store reader, on every host that runs the collector and - # holds a store identity. The negative names the deleted unit rather - # than a generic absence, because the way this regresses is someone - # re-adding the co-located copy as an optimisation. - name = "the collector's OIDC secret has exactly one delivery unit, the store reader, in both topologies"; - ok = - let - local = otelBaoWithAuthelia.systemd.services; - remote = otelBaoRemoteAuthelia.systemd.services; - in - local ? swarm-bao-otel-oidc - && remote ? swarm-bao-otel-oidc - && !(local ? swarm-otel-oidc-secret) - && !(remote ? swarm-otel-oidc-secret); - } - { - # Same 403-not-a-miss reason as grafana's arm above: the reader's grant - # covers the `services` prefix, so a path outside it is refused rather - # than empty, however correct it reads. - name = "the collector's OIDC secret is read from the prefix the publisher writes"; - ok = - let - s = otelBaoRemoteAuthelia.systemd.services.swarm-bao-otel-oidc.script; - in - lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" s - && !(lib.hasInfix "secret/swarm/hives/" s); - } - { - # Both ends of a wire nothing at eval time carries end to end: the - # publisher on authelia's host writes the path the reader on the - # collector's host reads, and the two files agree only because both - # compose it from the same swarm-wide client id. `secretPublisherHere` - # already grew this client when `serviceClientIds` did. - name = "the publisher writes the swarm service path the collector reads"; - ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" ( - secretPublisherHere.systemd.services.swarm-secret-publish.script - ); - } - { - # The collector's non-assertion, the deliberate mirror of Grafana's - # assertion two cases up: a host with no store identity is a supported, - # merely degraded shape here, so the reading unit simply does not exist - # rather than refusing the build. `haveCollectorSecret` is what the - # degrade already reads, unchanged by this slice. - name = "a collector with no store identity renders no reading unit, and is not refused"; - ok = - !(otelNoIdentity.systemd.services ? swarm-bao-otel-oidc) - && otelNoIdentity.services.hyperhive.deploy.swarm-otel.clientSecretFile == null - && !(lib.any (a: !a.assertion) otelNoIdentity.assertions); - } { # Reads the daemon's rendered unit, not the options: the queue address # arrives as an env var whose whole attrset is guarded on `natsUrl`, and