diff --git a/docs/observability.md b/docs/observability.md index 4b23a4bb..a1f16a7c 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -186,6 +186,31 @@ redefines it — the agent-facing value is *derived* keeps working unchanged. The bridge port is contributed to `exposeHostPorts` automatically; there is nothing to open by hand. +### Authenticated ingest + +The swarm tier gives **each hive its own receiver**, and stamps the `hive` label +from whichever receiver accepted a sample. A hive therefore cannot report +metrics as another hive, and cannot relabel its own by editing what it sends — +the label is not taken from the payload at all. + +**On an all-local swarm there is nothing to set.** Each hive already has an +identity, and its collector reads the secret that host's own authelia minted. + +**On a hive that does not host the swarm's services**, the secret has to arrive +somehow — copy it across and name it: + +```nix +services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret"; +``` + +**There is no unauthenticated mode.** A hive always presents an identity, so a +missing credential is a build error rather than a quieter fallback — the +collector has no anonymous route to accept samples on, and every path it serves +belongs to exactly one hive. + +Getting the secret wrong shows up as the hive's collector logging 401s from the +swarm tier and no metrics appearing for that hive. + ### `services.hyperhive.otel.collector.port` — port, default `4318` The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The diff --git a/docs/setup.md b/docs/setup.md index d5ec447c..cdebe400 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -156,6 +156,11 @@ See [`tools/hivectl.md`](tools/hivectl.md) for every `hivectl` verb. - All config changes (forge PRs on `agent-configs/`) go through operator approval — agents can't unilaterally rebuild containers, by design. See [`boundary.md`](boundary.md) and [`security.md`](security.md). +- **Telemetry ingest is authenticated per hive**, and the `hive` label comes + from which hive authenticated rather than from the payload — so no hive can + report metrics as another. A first-run all-local hive gets this with nothing + to configure; joining a swarm you don't host needs one secret copied across. + See [`observability.md`](observability.md#authenticated-ingest). Once the hive is running, ruth records anything it needs to remember across restarts in `/agents/ruth/state/notes.md`. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 23547b7c..4d324944 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -45,7 +45,7 @@ Every row below is read against one of these. | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in the service's `sso.clientSecretFile` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | @@ -56,6 +56,13 @@ because nothing outside that container ever reads them. **That is the test worth applying to any secret added here** — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists. +One reader needs no delivery step: the **hive's telemetry collector**, which +authenticates to the swarm's collector as its own hive. It is a host unit rather +than a container, so on an all-local swarm it reads authelia's file where it +lies (through `LoadCredential`) and no second copy is made. On any other +topology it is an ordinary "operator provides the file" case — see +`services.hyperhive.otel.clientSecretFile`. + ### Minting the queue's callout nkeys `nats.autoGenerateCallout` mints both keypairs on the host before the queue diff --git a/nix/host-modules/lib/hive-ca-trust.nix b/nix/host-modules/lib/hive-ca-trust.nix index 509730c3..0ad563ea 100644 --- a/nix/host-modules/lib/hive-ca-trust.nix +++ b/nix/host-modules/lib/hive-ca-trust.nix @@ -39,10 +39,26 @@ let # the bundle next to the CA and explains the split. caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem"; caContainerPath = "/run/hive-ca/trust-bundle.pem"; + + # One definition, used by `trustBundle` to WRITE the bundle and published + # below so a caller can NAME it. Two copies of this path would be two + # things to keep in step, and the one that drifts is the reader. + bundleDirFor = name: "/run/${name}-ca"; + bundlePathFor = name: "${bundleDirFor name}/trust-bundle.pem"; in { inherit useSelfSigned caContainerPath; + # Where `trustBundle` below puts the assembled bundle, for the callers + # that must NAME it rather than just have it exported. `SSL_CERT_FILE` is + # set for you and needs no path here; a consumer that takes its own CA + # argument (an OIDC verifier's `issuer_ca_path`, a client's `--cacert`) + # does, and the alternative is copying `/run/-ca/…` to the call + # site. That copy breaks silently: the bundle keeps being written, the + # consumer keeps reading a path that no longer exists, and the failure + # surfaces as a TLS error naming the peer rather than the file. + inherit bundlePathFor; + # Fold into the container's `bindMounts` via `//`. Binds ONLY the public # CA cert (never the `hive-tls` state dir — it holds the CA + leaf private # keys), read-only. Empty when not self-signed, so the whole trust path @@ -106,8 +122,8 @@ in enable ? true, }: let - dir = "/run/${name}-ca"; - bundlePath = "${dir}/trust-bundle.pem"; + dir = bundleDirFor name; + bundlePath = bundlePathFor name; unit = "${name}-ca-bundle"; source = if hostUnit then caHostPath else caContainerPath; in diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index 53d43329..2800717c 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -193,6 +193,40 @@ in ''; }; + clientSecretFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = + if + config.services.hyperhive.swarm.authelia.enable && config.services.hyperhive.hiveName != null + then + "${config.services.hyperhive.swarm.authelia.hostClientSecretDir}/" + + "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret" + else + null; + defaultText = lib.literalExpression ''"''${swarm.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null''; + example = "/var/lib/secrets/hive-telemetry.secret"; + description = '' + Absolute path to this hive's OAuth2 client secret, used to + authenticate to the swarm's collector as this hive. + + **Whether this is set is what decides whether this tier + authenticates at all.** A credential is the only thing that makes + authenticated egress possible, so its presence is the condition + rather than a second switch that could disagree with it. + + Defaults to the secret this host's own authelia minted, which is + correct exactly when the swarm's IdP runs here. On a hive that is + not that host, the file has to arrive some other way and this + option names wherever it landed — the same manual-copy shape + `services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents, + where delivering a secret to a hive that is not this host is + deliberately not solved. + + Read by `LoadCredential`, so it is never evaluated by nix, never + copied into the store and never passed in argv. + ''; + }; + metricIntervalMs = lib.mkOption { type = lib.types.nullOr lib.types.ints.positive; default = null; @@ -214,8 +248,32 @@ in config = lib.mkIf config.services.hyperhive.otel.enable ( let otel = config.services.hyperhive.otel; + autheliaCfg = config.services.hyperhive.swarm.authelia; + swarmOtelCfg = config.services.hyperhive.swarm.otel; + hiveName = config.services.hyperhive.hiveName; listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}"; swarmName = "otlphttp/swarm"; + authName = "oauth2client/swarm"; + + # A hive always authenticates to the swarm's collector as itself, so + # this is not a mode — it is a precondition, and the assertion below + # is what enforces it. Kept as a name because several places have to + # read "do we have what it takes", and an eval error from a null path + # names this file rather than the option an operator has to set. + senderAuth = otel.clientSecretFile != null && hiveName != null; + + # This hive's client id, and also the audience it must ASK for. Both + # are `hiveClientPrefix` + the hive's name because that is the one + # name the swarm already agrees on; the receiver one tier up derives + # the same string. + hiveClient = "${autheliaCfg.hiveClientPrefix}${toString hiveName}"; + + # systemd exports `CREDENTIALS_DIRECTORY` to any unit with + # `LoadCredential`, and the collector expands `${env:…}` at load. So + # the secret reaches the process as a PATH resolved at runtime — nix + # renders neither the value nor the directory, and nothing has to + # hardcode `/run/credentials/`. + credName = "swarm-client.secret"; in { # Reachable from agent containers and nowhere else: this opens @@ -224,6 +282,22 @@ in services.opentelemetry-collector = { enable = true; + # Contrib, matching the swarm tier (./swarm-otel.nix). The upstream + # default build has no auth extensions at all, and this tier has to + # *present* a credential to the swarm tier — `oauth2client` lives + # only in contrib, so the package choice is what makes authenticated + # egress expressible rather than a preference. + # + # Not a build-farm cost: contrib is fetched, not compiled. + # + # ⚠️ Read the note directly below before adding any extension here. + # It describes precisely the trap this package unlocks: naming an + # extension the build lacks passes `validate` and then kills the + # collector at startup. With contrib the extensions exist — but the + # gap it warns about (a green build proving nothing about whether + # the process starts) is exactly why this module's auth wiring is + # gated by a probe that runs both collectors, not by eval. + package = pkgs.opentelemetry-collector-contrib; # `validateConfigFile` defaults to `isStorePath configFile`, # and `configFile` is null on the `settings` path — so the # upstream default is OFF for exactly the way this module @@ -263,15 +337,80 @@ in # change. `https://` because that name resolves through the # gateway even on a co-located host — see `caTrust` above for # the trust half that makes this verify. - endpoint = "https://${config.services.hyperhive.swarm.otel.domain}"; - }; + # The hive's own path under the collector's single name. The + # swarm tier gives each hive its own authenticated receiver and + # routes to it by this prefix, so the path is not decoration — + # it selects WHICH receiver, and therefore which hive the + # samples get labelled as. + endpoint = "https://${swarmOtelCfg.domain}" + lib.optionalString senderAuth "/${toString hiveName}"; + } + // lib.optionalAttrs senderAuth { auth.authenticator = authName; }; + + # ⚠️ An extension not listed here is INERT: the collector starts + # clean and the exporter naming it sends nothing authenticated. + service.extensions = lib.optional senderAuth authName; service.pipelines.metrics = { receivers = [ "otlp" ]; exporters = [ swarmName ]; }; + } + // lib.optionalAttrs senderAuth { + extensions.${authName} = { + client_id = hiveClient; + # A real key, measured against this collector version rather + # than assumed — with a deliberate typo rejected in the same + # run, so "accepted" is distinguishable from "ignores + # everything". Keeps the secret out of nix entirely: the + # collector opens the file itself. + client_secret_file = "\${env:CREDENTIALS_DIRECTORY}/${credName}"; + token_url = "${toString autheliaCfg.url}/api/oidc/token"; + # ⚠️ THE AUDIENCE HAS TO BE REQUESTED, not merely granted. + # Registering it on the client only makes it permissible; a + # token minted without asking carries `aud: []` and every + # receiver refuses it — with a config that reads perfectly at + # both ends. Measured against authelia 4.39.20. + endpoint_params.audience = hiveClient; + }; }; }; + + # `LoadCredential` and not a copy-oneshot: this collector is a HOST + # unit, so there is no container boundary to cross and therefore no + # reason for a second on-disk copy of the secret. systemd hands it to + # the process in a private tmpfs and exports the directory, which is + # what the config above names. + systemd.services.opentelemetry-collector.serviceConfig = lib.optionalAttrs senderAuth { + LoadCredential = [ "${credName}:${otel.clientSecretFile}" ]; + }; + + assertions = [ + { + # A hive authenticates to the swarm's collector as itself — there + # is no unauthenticated path to fall back to, so a missing + # credential is a broken deployment rather than a quieter mode. + # Caught here because the alternative is a collector that starts + # cleanly, retries forever, and reports nothing to anyone. + assertion = senderAuth; + message = '' + services.hyperhive.otel.enable is true but this hive has no + identity to present to the swarm's collector: + + services.hyperhive.otel.clientSecretFile = ${ + if otel.clientSecretFile == null then "null" else otel.clientSecretFile + } + services.hyperhive.hiveName = ${if hiveName == null then "null" else hiveName} + + Every hive authenticates as itself — that is what makes the + `hive` label on its metrics mean anything — so both are + required. + + On a host that runs the swarm's identity provider, the default + already points at the secret authelia minted. On a hive that + does not, copy that hive's secret across and name it here. + ''; + } + ]; } ); } diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index c25cf13a..6a419cf4 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -102,11 +102,26 @@ let # and a field later added to the submodule then existed on the # declared entries and not on these, which is an eval error reachable # only once hive identities are on. + # + # `audience` is the hive's own client id rather than a second per-hive + # string invented here. A swarm service that has to tell hives apart + # needs one name per hive that both sides already agree on, and the + # client id is that name — published as `hiveClientPrefix` for exactly + # this reason. Minting a parallel naming scheme would be a second thing + # to keep in step, and the one that drifts is the one nobody tests. + # + # `RS256` because a resource server that cannot call this provider back + # is a real case here: the swarm's telemetry collector verifies tokens + # offline against `/jwks.json`, and an opaque token gives it nothing to + # verify. The queue's auth-callout responder introspects instead, which + # is a different question asked of the same token. hiveClients = lib.mapAttrsToList (name: _: { id = "${cfg.hiveClientPrefix}${name}"; description = "HyperHive hive ${name}"; kind = "machine"; redirectUris = [ ]; + audience = [ "${cfg.hiveClientPrefix}${name}" ]; + accessTokenSignedResponseAlg = "RS256"; }) hyperhiveCfg.swarm.hives; # `swarm-authelia-bridge`'s own identity — distinct from @@ -188,6 +203,15 @@ let + lib.optionalString (c.tokenEndpointAuthMethod != null) '' printf -- ' token_endpoint_auth_method: %s\n' ${lib.escapeShellArg c.tokenEndpointAuthMethod} '' + # Flow-style YAML, matching `scopes` below. The values are client ids + # and hive names, which `Ident` already constrains to `[a-z0-9-]` — no + # character in that set needs quoting in a YAML flow sequence. + + lib.optionalString (c.audience != [ ]) '' + printf -- ' audience: [%s]\n' ${lib.escapeShellArg (lib.concatStringsSep ", " c.audience)} + '' + + lib.optionalString (c.accessTokenSignedResponseAlg != null) '' + printf -- ' access_token_signed_response_alg: %s\n' ${lib.escapeShellArg c.accessTokenSignedResponseAlg} + '' + ( if c.kind == "machine" then '' @@ -493,6 +517,63 @@ in the secret is at fault. ''; }; + + audience = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ ]; + example = [ "hive-alpha" ]; + description = '' + Audiences (`aud`) this client is permitted to request a + token for. Empty means it asks for none, which is the + right answer for a client whose resource server does not + distinguish callers. + + ⚠️ Registering an audience only *permits* it — the value + lands in a token when the client **asks** for it at the + token endpoint, and a client that does not send + `audience=` receives a token with `aud: []` however + complete this list looks. Measured against authelia + 4.39.20: the config reads exactly right and the resource + server rejects every token, because a config that grants + and a request that claims are two separate acts. + + Requesting an audience that is *not* listed here is + refused with `invalid_target`, which is what makes this + usable as a boundary rather than a label: a client cannot + mint a token for a resource slot that is not its own. + ''; + }; + + accessTokenSignedResponseAlg = lib.mkOption { + type = lib.types.nullOr ( + lib.types.enum [ + "none" + "RS256" + ] + ); + default = null; + example = "RS256"; + description = '' + Signing algorithm for this client's **access** tokens. + `null` leaves authelia on its default, which issues an + opaque token (`authelia_at_…`) — a database handle that + carries no claims and means nothing to anyone but this + provider. + + Set `RS256` when the resource server verifies the token + *itself* rather than asking this provider about it: that + yields an RFC 9068 JWT (`at+jwt`) carrying `aud`, `iss` + and `client_id`, verifiable against `/jwks.json` with no + round trip. + + ⚠️ This is what makes a token readable by an + OIDC-verifying consumer at all. A resource server given + an opaque token is not *misconfigured* — it is + structurally unable to verify it, and says so in terms + that point at the verifier rather than at the token's + format. + ''; + }; }; } ); diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 0eacc0de..525bac27 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -40,6 +40,43 @@ let # the required-domain assertion in hive-network.nix should be what an # operator sees, not a coercion error from here. domainBase = if swarmDomain == null then "invalid" else swarmDomain; + + autheliaCfg = hyperhiveCfg.swarm.authelia; + + # `attrNames` is sorted, so this is a function of the hive SET and not of + # the order anyone wrote it in. + # + # These ports are internal and appear in no URL: a hive addresses its own + # receiver as a PATH on this collector's single gateway name, and nginx — + # rendered from this same evaluation — is the only thing that ever names + # the port. That is what makes deriving them safe here and unsafe in the + # obvious other place: were a hive told a port, inserting a hive would + # renumber the ones after it and silently move a port a running hive was + # already sending to. + hivePorts = lib.listToAttrs ( + lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives) + ); + + # The swarm's authelia is reached by its gateway name, whose leaf is + # issued by the swarm services sub-CA — so this container needs the same + # runtime CA trust every other consumer of a swarm-service name needs. + # The CA is generated at runtime and cannot be baked into a derivation, + # which is why it arrives as a bind mount rather than + # `security.pki.certificateFiles`. + caTrust = import ./lib/hive-ca-trust.nix { + inherit lib; + tlsCfg = hyperhiveCfg.tls; + inherit gatewayCfg; + }; + caBundle = caTrust.bundlePathFor cfg.machine; + + # One list, read by every pipeline: the per-hive pipelines fan out to + # exactly the same destinations as the single pipeline they replace. + # Written once because "which exporters" is a property of this tier, not + # of which hive a sample came from. + exporterNames = + lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp") + ++ lib.optional vmCfg.enable "otlphttp/victoriametrics"; in { options.services.hyperhive.swarm.otel = { @@ -75,7 +112,17 @@ in type = lib.types.port; default = 4319; description = '' - Port this collector's OTLP/HTTP receiver listens on. + First port of this collector's receiver range. Every hive in + {option}`services.hyperhive.swarm.hives` gets its **own** + authenticated receiver — that is what makes the `hive` label + unforgeable — so the range is one port per hive, starting here, in + sorted-name order. + + ⚠️ Internal. No client is ever told a port: a hive reaches its own + receiver as `https://''${domain}/`, and the gateway routes on + that path. So adding a hive, which renumbers the ones after it, is + harmless — nginx is rendered from this same evaluation and moves + with it. ⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the hive tier already uses it (`services.hyperhive.otel.collector.port`) @@ -83,7 +130,9 @@ in listeners claiming one port on one host is not a build failure — it is a runtime coin toss over which one gets it, with nothing in any log saying so. The same collision cost a release when grafana - and the forge both defaulted to 3000. + and the forge both defaulted to 3000. The assertions below check + the whole derived range against every port this module and the hive + tier declare, which is as far as a module can see. ''; }; @@ -143,11 +192,39 @@ in services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // { listen = gatewayCfg.lib.listen; extraConfig = gatewayCfg.lib.securityHeaders; - locations."/" = { - proxyPass = "http://127.0.0.1:${toString cfg.port}"; - }; + locations = + # One name for the whole collector, and the hive is a path under + # it. The alternative — a vhost per hive — needs a certificate, + # a DNS name and a `localNames` entry per hive to express the + # same routing the gateway already does for free. + # + # ⚠️ The trailing slash on both sides is load-bearing: it is what + # strips `/` before the request reaches the receiver, which + # serves `/v1/metrics` and knows nothing about hives. Without it + # the receiver sees `//v1/metrics` and answers 404 to a + # request that authenticated perfectly. + lib.mapAttrs' ( + h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; } + ) hivePorts + // { + # There is no swarm-wide inbox, and a closed door is the honest + # description of that. Every route into this collector belongs to + # exactly one hive. + "/".return = "404"; + }; }; + # Turning on the identities this tier authenticates against, which the + # option exists to allow: its own description names this module as the + # second consumer, so the queue is not a prerequisite for authenticated + # telemetry. + services.hyperhive.swarm.authelia.oidc.hiveIdentities = true; + + # The CA bind source is written at runtime by a host unit, so the + # container has to start after it — otherwise nspawn sets up a mount + # over a file that does not exist yet. + systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; + assertions = [ { # The tier exists to hold the upstream credential and to write the @@ -164,6 +241,67 @@ in metrics store. ''; } + { + # Without a roster there are no receivers at all, so this + # collector would listen on nothing while looking configured. + assertion = hyperhiveCfg.swarm.hives != { }; + message = '' + services.hyperhive.swarm.otel.enable is true but + services.hyperhive.swarm.hives is empty: ingest is authenticated + per hive, so an empty roster means this collector accepts nothing + from anyone. + + List the swarm's hives. + ''; + } + { + # A hive proves who it is with a token this provider mints, so + # there is no version of this collector that runs without one. + # Stated as an assertion rather than a fallback because a guessed + # issuer URL evaluates cleanly and refuses every hive at runtime. + assertion = autheliaCfg.url != null; + message = '' + services.hyperhive.swarm.otel.enable is true but + services.hyperhive.swarm.authelia.url is null: every hive + authenticates to this collector as itself, and the token comes + from the swarm's identity provider. + + Point authelia.url at the swarm's provider, or enable + services.hyperhive.swarm.authelia on the host that runs it. + ''; + } + { + # A port collision between two listeners on one host is a runtime + # coin toss with nothing in any log — the failure this whole + # comment budget exists to prevent. Checked against every port + # reachable from here; a port some other module picks is not. + # + # ⚠️ `cfg.port` is deliberately absent from `others`: it is the + # FIRST element of the derived range, so listing it would make this + # assertion fire on every config. + assertion = + let + derived = lib.attrValues hivePorts; + others = [ + cfg.telemetryPort + otelCfg.collector.port + ] + ++ lib.optional vmCfg.enable vmCfg.port; + all = derived ++ others; + in + lib.length (lib.unique all) == lib.length all; + message = '' + services.hyperhive.swarm.otel: the receiver range starting at + port (${toString cfg.port}, one port per hive in + services.hyperhive.swarm.hives) overlaps another port on this + host. + + Every swarm container shares the host's network namespace, so + two listeners claiming one port is not a build failure — it is + whichever process started first, silently. Move + services.hyperhive.swarm.otel.port to a free range. + ''; + } ]; containers.${cfg.machine} = { @@ -179,12 +317,16 @@ in # Read-only, and only when one is configured — binding a path that # does not exist makes nixos-container refuse to start the container, # which is a stall several layers from its cause. - bindMounts = lib.optionalAttrs (otelCfg.headersCredential != null) { - ${otelCfg.headersCredential} = { - hostPath = otelCfg.headersCredential; - isReadOnly = true; - }; - }; + bindMounts = + lib.optionalAttrs (otelCfg.headersCredential != null) { + ${otelCfg.headersCredential} = { + hostPath = otelCfg.headersCredential; + isReadOnly = true; + }; + } + # The public hive CA, read-only — only when something in here + # actually verifies a swarm-service name. + // caTrust.bindMount; config = { ... }: @@ -200,6 +342,18 @@ in inherit (config.services.hyperhive.network) bridgeIp; dnsConsumers = [ "opentelemetry-collector.service" ]; }) + ] + # `SSL_CERT_FILE` REPLACES the trust store rather than adding to + # it, so a failed assembly yields an empty pool and every TLS + # call fails while the unit looks healthy. That is why this is + # the shared helper — it carries the `Requires` and the + # non-empty check — and not a local `cat`. + ++ [ + (caTrust.trustBundle { + inherit pkgs; + name = cfg.machine; + consumers = [ "opentelemetry-collector" ]; + }) ]; system.stateVersion = config.system.stateVersion; @@ -219,7 +373,24 @@ in # real sample through both tiers into the store. validateConfigFile = true; settings = { - receivers.otlp.protocols.http.endpoint = "127.0.0.1:${toString cfg.port}"; + # One receiver per hive, and that multiplicity is forced + # rather than chosen. The `hive` + # label has to come from something the sender cannot write, + # and the only such thing here is WHICH RECEIVER accepted + # the sample: a processor cannot read the token's claims + # (`from_context` reads request metadata, and asking it for + # an auth claim yields nothing — silently, with a healthy + # startup), and one receiver holding many credentials never + # reveals which one matched. + receivers = lib.mapAttrs' ( + h: p: + lib.nameValuePair "otlp/${h}" { + protocols.http = { + endpoint = "127.0.0.1:${toString p}"; + auth.authenticator = "oidc/${h}"; + }; + } + ) hivePorts; exporters = lib.optionalAttrs vmCfg.enable { @@ -263,16 +434,64 @@ in } ]; - service.pipelines.metrics = { - receivers = [ "otlp" ]; - # Fan-out, not a choice: with both configured the same - # samples go upstream AND into the swarm's store. The store - # is for looking at this swarm; the upstream is for whoever - # aggregates across swarms, and neither replaces the other. - exporters = - lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp") - ++ lib.optional vmCfg.enable "otlphttp/victoriametrics"; - }; + # ⚠️ An extension that is configured but not listed here is + # INERT — the collector starts clean and the receiver + # naming it authenticates nothing. Derived from the same + # attrset as the receivers so the two cannot disagree. + service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts); + + # Fan-out, not a choice: with both configured the same + # samples go upstream AND into the swarm's store. The store + # is for looking at this swarm; the upstream is for whoever + # aggregates across swarms, and neither replaces the other. + # `exporterNames` is shared by every pipeline — where a + # sample goes is a property of this tier, not of the hive + # that sent it. + service.pipelines = lib.mapAttrs' ( + h: _: + lib.nameValuePair "metrics/${h}" { + receivers = [ "otlp/${h}" ]; + processors = [ "resource/${h}" ]; + exporters = exporterNames; + } + ) hivePorts; + } + // { + extensions = lib.mapAttrs' ( + h: _: + lib.nameValuePair "oidc/${h}" { + issuer_url = autheliaCfg.url; + # The audience this hive's client is registered to + # request, and the reason one hive's token is refused by + # another hive's receiver. Same expression authelia + # registers it under — a second spelling here would deny + # every hive, as a 401 that blames the token. + audience = "${autheliaCfg.hiveClientPrefix}${h}"; + # ⚠️ `issuer_ca_path`. `issuer_ca_file`, `ca_file` and + # `tls.ca_file` are all INVALID KEYS for this extension + # — measured, and the failure is a startup error naming + # the key rather than anything about certificates. + issuer_ca_path = caBundle; + } + ) hivePorts; + + # `upsert`, not `insert`: a sender that stamps its own + # `hive` must be OVERWRITTEN, not deferred to. This + # processor is the whole attribution boundary — the value + # is a constant per receiver, so it says which hive + # authenticated, not which hive claimed to be sending. + processors = lib.mapAttrs' ( + h: _: + lib.nameValuePair "resource/${h}" { + attributes = [ + { + key = "hive"; + value = h; + action = "upsert"; + } + ]; + } + ) hivePorts; }; };