diff --git a/docs/observability.md b/docs/observability.md index a1f16a7c..4b23a4bb 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -186,31 +186,6 @@ redefines it — the agent-facing value is *derived* keeps working unchanged. The bridge port is contributed to `exposeHostPorts` automatically; there is nothing to open by hand. -### Authenticated ingest - -The swarm tier gives **each hive its own receiver**, and stamps the `hive` label -from whichever receiver accepted a sample. A hive therefore cannot report -metrics as another hive, and cannot relabel its own by editing what it sends — -the label is not taken from the payload at all. - -**On an all-local swarm there is nothing to set.** Each hive already has an -identity, and its collector reads the secret that host's own authelia minted. - -**On a hive that does not host the swarm's services**, the secret has to arrive -somehow — copy it across and name it: - -```nix -services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret"; -``` - -**There is no unauthenticated mode.** A hive always presents an identity, so a -missing credential is a build error rather than a quieter fallback — the -collector has no anonymous route to accept samples on, and every path it serves -belongs to exactly one hive. - -Getting the secret wrong shows up as the hive's collector logging 401s from the -swarm tier and no metrics appearing for that hive. - ### `services.hyperhive.otel.collector.port` — port, default `4318` The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The diff --git a/docs/setup.md b/docs/setup.md index cdebe400..d5ec447c 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -156,11 +156,6 @@ See [`tools/hivectl.md`](tools/hivectl.md) for every `hivectl` verb. - All config changes (forge PRs on `agent-configs/`) go through operator approval — agents can't unilaterally rebuild containers, by design. See [`boundary.md`](boundary.md) and [`security.md`](security.md). -- **Telemetry ingest is authenticated per hive**, and the `hive` label comes - from which hive authenticated rather than from the payload — so no hive can - report metrics as another. A first-run all-local hive gets this with nothing - to configure; joining a swarm you don't host needs one secret copied across. - See [`observability.md`](observability.md#authenticated-ingest). Once the hive is running, ruth records anything it needs to remember across restarts in `/agents/ruth/state/notes.md`. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 4d324944..23547b7c 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -45,7 +45,7 @@ Every row below is read against one of these. | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in the service's `sso.clientSecretFile` | | OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | @@ -56,13 +56,6 @@ because nothing outside that container ever reads them. **That is the test worth applying to any secret added here** — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists. -One reader needs no delivery step: the **hive's telemetry collector**, which -authenticates to the swarm's collector as its own hive. It is a host unit rather -than a container, so on an all-local swarm it reads authelia's file where it -lies (through `LoadCredential`) and no second copy is made. On any other -topology it is an ordinary "operator provides the file" case — see -`services.hyperhive.otel.clientSecretFile`. - ### Minting the queue's callout nkeys `nats.autoGenerateCallout` mints both keypairs on the host before the queue diff --git a/nix/host-modules/lib/hive-ca-trust.nix b/nix/host-modules/lib/hive-ca-trust.nix index 0ad563ea..509730c3 100644 --- a/nix/host-modules/lib/hive-ca-trust.nix +++ b/nix/host-modules/lib/hive-ca-trust.nix @@ -39,26 +39,10 @@ let # the bundle next to the CA and explains the split. caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem"; caContainerPath = "/run/hive-ca/trust-bundle.pem"; - - # One definition, used by `trustBundle` to WRITE the bundle and published - # below so a caller can NAME it. Two copies of this path would be two - # things to keep in step, and the one that drifts is the reader. - bundleDirFor = name: "/run/${name}-ca"; - bundlePathFor = name: "${bundleDirFor name}/trust-bundle.pem"; in { inherit useSelfSigned caContainerPath; - # Where `trustBundle` below puts the assembled bundle, for the callers - # that must NAME it rather than just have it exported. `SSL_CERT_FILE` is - # set for you and needs no path here; a consumer that takes its own CA - # argument (an OIDC verifier's `issuer_ca_path`, a client's `--cacert`) - # does, and the alternative is copying `/run/-ca/…` to the call - # site. That copy breaks silently: the bundle keeps being written, the - # consumer keeps reading a path that no longer exists, and the failure - # surfaces as a TLS error naming the peer rather than the file. - inherit bundlePathFor; - # Fold into the container's `bindMounts` via `//`. Binds ONLY the public # CA cert (never the `hive-tls` state dir — it holds the CA + leaf private # keys), read-only. Empty when not self-signed, so the whole trust path @@ -122,8 +106,8 @@ in enable ? true, }: let - dir = bundleDirFor name; - bundlePath = bundlePathFor name; + dir = "/run/${name}-ca"; + bundlePath = "${dir}/trust-bundle.pem"; unit = "${name}-ca-bundle"; source = if hostUnit then caHostPath else caContainerPath; in diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index 2800717c..53d43329 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -193,40 +193,6 @@ in ''; }; - clientSecretFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = - if - config.services.hyperhive.swarm.authelia.enable && config.services.hyperhive.hiveName != null - then - "${config.services.hyperhive.swarm.authelia.hostClientSecretDir}/" - + "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret" - else - null; - defaultText = lib.literalExpression ''"''${swarm.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null''; - example = "/var/lib/secrets/hive-telemetry.secret"; - description = '' - Absolute path to this hive's OAuth2 client secret, used to - authenticate to the swarm's collector as this hive. - - **Whether this is set is what decides whether this tier - authenticates at all.** A credential is the only thing that makes - authenticated egress possible, so its presence is the condition - rather than a second switch that could disagree with it. - - Defaults to the secret this host's own authelia minted, which is - correct exactly when the swarm's IdP runs here. On a hive that is - not that host, the file has to arrive some other way and this - option names wherever it landed — the same manual-copy shape - `services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents, - where delivering a secret to a hive that is not this host is - deliberately not solved. - - Read by `LoadCredential`, so it is never evaluated by nix, never - copied into the store and never passed in argv. - ''; - }; - metricIntervalMs = lib.mkOption { type = lib.types.nullOr lib.types.ints.positive; default = null; @@ -248,32 +214,8 @@ in config = lib.mkIf config.services.hyperhive.otel.enable ( let otel = config.services.hyperhive.otel; - autheliaCfg = config.services.hyperhive.swarm.authelia; - swarmOtelCfg = config.services.hyperhive.swarm.otel; - hiveName = config.services.hyperhive.hiveName; listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}"; swarmName = "otlphttp/swarm"; - authName = "oauth2client/swarm"; - - # A hive always authenticates to the swarm's collector as itself, so - # this is not a mode — it is a precondition, and the assertion below - # is what enforces it. Kept as a name because several places have to - # read "do we have what it takes", and an eval error from a null path - # names this file rather than the option an operator has to set. - senderAuth = otel.clientSecretFile != null && hiveName != null; - - # This hive's client id, and also the audience it must ASK for. Both - # are `hiveClientPrefix` + the hive's name because that is the one - # name the swarm already agrees on; the receiver one tier up derives - # the same string. - hiveClient = "${autheliaCfg.hiveClientPrefix}${toString hiveName}"; - - # systemd exports `CREDENTIALS_DIRECTORY` to any unit with - # `LoadCredential`, and the collector expands `${env:…}` at load. So - # the secret reaches the process as a PATH resolved at runtime — nix - # renders neither the value nor the directory, and nothing has to - # hardcode `/run/credentials/`. - credName = "swarm-client.secret"; in { # Reachable from agent containers and nowhere else: this opens @@ -282,22 +224,6 @@ in services.opentelemetry-collector = { enable = true; - # Contrib, matching the swarm tier (./swarm-otel.nix). The upstream - # default build has no auth extensions at all, and this tier has to - # *present* a credential to the swarm tier — `oauth2client` lives - # only in contrib, so the package choice is what makes authenticated - # egress expressible rather than a preference. - # - # Not a build-farm cost: contrib is fetched, not compiled. - # - # ⚠️ Read the note directly below before adding any extension here. - # It describes precisely the trap this package unlocks: naming an - # extension the build lacks passes `validate` and then kills the - # collector at startup. With contrib the extensions exist — but the - # gap it warns about (a green build proving nothing about whether - # the process starts) is exactly why this module's auth wiring is - # gated by a probe that runs both collectors, not by eval. - package = pkgs.opentelemetry-collector-contrib; # `validateConfigFile` defaults to `isStorePath configFile`, # and `configFile` is null on the `settings` path — so the # upstream default is OFF for exactly the way this module @@ -337,80 +263,15 @@ in # change. `https://` because that name resolves through the # gateway even on a co-located host — see `caTrust` above for # the trust half that makes this verify. - # The hive's own path under the collector's single name. The - # swarm tier gives each hive its own authenticated receiver and - # routes to it by this prefix, so the path is not decoration — - # it selects WHICH receiver, and therefore which hive the - # samples get labelled as. - endpoint = "https://${swarmOtelCfg.domain}" + lib.optionalString senderAuth "/${toString hiveName}"; - } - // lib.optionalAttrs senderAuth { auth.authenticator = authName; }; - - # ⚠️ An extension not listed here is INERT: the collector starts - # clean and the exporter naming it sends nothing authenticated. - service.extensions = lib.optional senderAuth authName; + endpoint = "https://${config.services.hyperhive.swarm.otel.domain}"; + }; service.pipelines.metrics = { receivers = [ "otlp" ]; exporters = [ swarmName ]; }; - } - // lib.optionalAttrs senderAuth { - extensions.${authName} = { - client_id = hiveClient; - # A real key, measured against this collector version rather - # than assumed — with a deliberate typo rejected in the same - # run, so "accepted" is distinguishable from "ignores - # everything". Keeps the secret out of nix entirely: the - # collector opens the file itself. - client_secret_file = "\${env:CREDENTIALS_DIRECTORY}/${credName}"; - token_url = "${toString autheliaCfg.url}/api/oidc/token"; - # ⚠️ THE AUDIENCE HAS TO BE REQUESTED, not merely granted. - # Registering it on the client only makes it permissible; a - # token minted without asking carries `aud: []` and every - # receiver refuses it — with a config that reads perfectly at - # both ends. Measured against authelia 4.39.20. - endpoint_params.audience = hiveClient; - }; }; }; - - # `LoadCredential` and not a copy-oneshot: this collector is a HOST - # unit, so there is no container boundary to cross and therefore no - # reason for a second on-disk copy of the secret. systemd hands it to - # the process in a private tmpfs and exports the directory, which is - # what the config above names. - systemd.services.opentelemetry-collector.serviceConfig = lib.optionalAttrs senderAuth { - LoadCredential = [ "${credName}:${otel.clientSecretFile}" ]; - }; - - assertions = [ - { - # A hive authenticates to the swarm's collector as itself — there - # is no unauthenticated path to fall back to, so a missing - # credential is a broken deployment rather than a quieter mode. - # Caught here because the alternative is a collector that starts - # cleanly, retries forever, and reports nothing to anyone. - assertion = senderAuth; - message = '' - services.hyperhive.otel.enable is true but this hive has no - identity to present to the swarm's collector: - - services.hyperhive.otel.clientSecretFile = ${ - if otel.clientSecretFile == null then "null" else otel.clientSecretFile - } - services.hyperhive.hiveName = ${if hiveName == null then "null" else hiveName} - - Every hive authenticates as itself — that is what makes the - `hive` label on its metrics mean anything — so both are - required. - - On a host that runs the swarm's identity provider, the default - already points at the secret authelia minted. On a hive that - does not, copy that hive's secret across and name it here. - ''; - } - ]; } ); } diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index 6a419cf4..c25cf13a 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -102,26 +102,11 @@ let # and a field later added to the submodule then existed on the # declared entries and not on these, which is an eval error reachable # only once hive identities are on. - # - # `audience` is the hive's own client id rather than a second per-hive - # string invented here. A swarm service that has to tell hives apart - # needs one name per hive that both sides already agree on, and the - # client id is that name — published as `hiveClientPrefix` for exactly - # this reason. Minting a parallel naming scheme would be a second thing - # to keep in step, and the one that drifts is the one nobody tests. - # - # `RS256` because a resource server that cannot call this provider back - # is a real case here: the swarm's telemetry collector verifies tokens - # offline against `/jwks.json`, and an opaque token gives it nothing to - # verify. The queue's auth-callout responder introspects instead, which - # is a different question asked of the same token. hiveClients = lib.mapAttrsToList (name: _: { id = "${cfg.hiveClientPrefix}${name}"; description = "HyperHive hive ${name}"; kind = "machine"; redirectUris = [ ]; - audience = [ "${cfg.hiveClientPrefix}${name}" ]; - accessTokenSignedResponseAlg = "RS256"; }) hyperhiveCfg.swarm.hives; # `swarm-authelia-bridge`'s own identity — distinct from @@ -203,15 +188,6 @@ let + lib.optionalString (c.tokenEndpointAuthMethod != null) '' printf -- ' token_endpoint_auth_method: %s\n' ${lib.escapeShellArg c.tokenEndpointAuthMethod} '' - # Flow-style YAML, matching `scopes` below. The values are client ids - # and hive names, which `Ident` already constrains to `[a-z0-9-]` — no - # character in that set needs quoting in a YAML flow sequence. - + lib.optionalString (c.audience != [ ]) '' - printf -- ' audience: [%s]\n' ${lib.escapeShellArg (lib.concatStringsSep ", " c.audience)} - '' - + lib.optionalString (c.accessTokenSignedResponseAlg != null) '' - printf -- ' access_token_signed_response_alg: %s\n' ${lib.escapeShellArg c.accessTokenSignedResponseAlg} - '' + ( if c.kind == "machine" then '' @@ -517,63 +493,6 @@ in the secret is at fault. ''; }; - - audience = lib.mkOption { - type = lib.types.listOf lib.types.str; - default = [ ]; - example = [ "hive-alpha" ]; - description = '' - Audiences (`aud`) this client is permitted to request a - token for. Empty means it asks for none, which is the - right answer for a client whose resource server does not - distinguish callers. - - ⚠️ Registering an audience only *permits* it — the value - lands in a token when the client **asks** for it at the - token endpoint, and a client that does not send - `audience=` receives a token with `aud: []` however - complete this list looks. Measured against authelia - 4.39.20: the config reads exactly right and the resource - server rejects every token, because a config that grants - and a request that claims are two separate acts. - - Requesting an audience that is *not* listed here is - refused with `invalid_target`, which is what makes this - usable as a boundary rather than a label: a client cannot - mint a token for a resource slot that is not its own. - ''; - }; - - accessTokenSignedResponseAlg = lib.mkOption { - type = lib.types.nullOr ( - lib.types.enum [ - "none" - "RS256" - ] - ); - default = null; - example = "RS256"; - description = '' - Signing algorithm for this client's **access** tokens. - `null` leaves authelia on its default, which issues an - opaque token (`authelia_at_…`) — a database handle that - carries no claims and means nothing to anyone but this - provider. - - Set `RS256` when the resource server verifies the token - *itself* rather than asking this provider about it: that - yields an RFC 9068 JWT (`at+jwt`) carrying `aud`, `iss` - and `client_id`, verifiable against `/jwks.json` with no - round trip. - - ⚠️ This is what makes a token readable by an - OIDC-verifying consumer at all. A resource server given - an opaque token is not *misconfigured* — it is - structurally unable to verify it, and says so in terms - that point at the verifier rather than at the token's - format. - ''; - }; }; } ); diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 525bac27..0eacc0de 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -40,43 +40,6 @@ let # the required-domain assertion in hive-network.nix should be what an # operator sees, not a coercion error from here. domainBase = if swarmDomain == null then "invalid" else swarmDomain; - - autheliaCfg = hyperhiveCfg.swarm.authelia; - - # `attrNames` is sorted, so this is a function of the hive SET and not of - # the order anyone wrote it in. - # - # These ports are internal and appear in no URL: a hive addresses its own - # receiver as a PATH on this collector's single gateway name, and nginx — - # rendered from this same evaluation — is the only thing that ever names - # the port. That is what makes deriving them safe here and unsafe in the - # obvious other place: were a hive told a port, inserting a hive would - # renumber the ones after it and silently move a port a running hive was - # already sending to. - hivePorts = lib.listToAttrs ( - lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives) - ); - - # The swarm's authelia is reached by its gateway name, whose leaf is - # issued by the swarm services sub-CA — so this container needs the same - # runtime CA trust every other consumer of a swarm-service name needs. - # The CA is generated at runtime and cannot be baked into a derivation, - # which is why it arrives as a bind mount rather than - # `security.pki.certificateFiles`. - caTrust = import ./lib/hive-ca-trust.nix { - inherit lib; - tlsCfg = hyperhiveCfg.tls; - inherit gatewayCfg; - }; - caBundle = caTrust.bundlePathFor cfg.machine; - - # One list, read by every pipeline: the per-hive pipelines fan out to - # exactly the same destinations as the single pipeline they replace. - # Written once because "which exporters" is a property of this tier, not - # of which hive a sample came from. - exporterNames = - lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp") - ++ lib.optional vmCfg.enable "otlphttp/victoriametrics"; in { options.services.hyperhive.swarm.otel = { @@ -112,17 +75,7 @@ in type = lib.types.port; default = 4319; description = '' - First port of this collector's receiver range. Every hive in - {option}`services.hyperhive.swarm.hives` gets its **own** - authenticated receiver — that is what makes the `hive` label - unforgeable — so the range is one port per hive, starting here, in - sorted-name order. - - ⚠️ Internal. No client is ever told a port: a hive reaches its own - receiver as `https://''${domain}/`, and the gateway routes on - that path. So adding a hive, which renumbers the ones after it, is - harmless — nginx is rendered from this same evaluation and moves - with it. + Port this collector's OTLP/HTTP receiver listens on. ⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the hive tier already uses it (`services.hyperhive.otel.collector.port`) @@ -130,9 +83,7 @@ in listeners claiming one port on one host is not a build failure — it is a runtime coin toss over which one gets it, with nothing in any log saying so. The same collision cost a release when grafana - and the forge both defaulted to 3000. The assertions below check - the whole derived range against every port this module and the hive - tier declare, which is as far as a module can see. + and the forge both defaulted to 3000. ''; }; @@ -192,39 +143,11 @@ in services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // { listen = gatewayCfg.lib.listen; extraConfig = gatewayCfg.lib.securityHeaders; - locations = - # One name for the whole collector, and the hive is a path under - # it. The alternative — a vhost per hive — needs a certificate, - # a DNS name and a `localNames` entry per hive to express the - # same routing the gateway already does for free. - # - # ⚠️ The trailing slash on both sides is load-bearing: it is what - # strips `/` before the request reaches the receiver, which - # serves `/v1/metrics` and knows nothing about hives. Without it - # the receiver sees `//v1/metrics` and answers 404 to a - # request that authenticated perfectly. - lib.mapAttrs' ( - h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; } - ) hivePorts - // { - # There is no swarm-wide inbox, and a closed door is the honest - # description of that. Every route into this collector belongs to - # exactly one hive. - "/".return = "404"; - }; + locations."/" = { + proxyPass = "http://127.0.0.1:${toString cfg.port}"; + }; }; - # Turning on the identities this tier authenticates against, which the - # option exists to allow: its own description names this module as the - # second consumer, so the queue is not a prerequisite for authenticated - # telemetry. - services.hyperhive.swarm.authelia.oidc.hiveIdentities = true; - - # The CA bind source is written at runtime by a host unit, so the - # container has to start after it — otherwise nspawn sets up a mount - # over a file that does not exist yet. - systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; - assertions = [ { # The tier exists to hold the upstream credential and to write the @@ -241,67 +164,6 @@ in metrics store. ''; } - { - # Without a roster there are no receivers at all, so this - # collector would listen on nothing while looking configured. - assertion = hyperhiveCfg.swarm.hives != { }; - message = '' - services.hyperhive.swarm.otel.enable is true but - services.hyperhive.swarm.hives is empty: ingest is authenticated - per hive, so an empty roster means this collector accepts nothing - from anyone. - - List the swarm's hives. - ''; - } - { - # A hive proves who it is with a token this provider mints, so - # there is no version of this collector that runs without one. - # Stated as an assertion rather than a fallback because a guessed - # issuer URL evaluates cleanly and refuses every hive at runtime. - assertion = autheliaCfg.url != null; - message = '' - services.hyperhive.swarm.otel.enable is true but - services.hyperhive.swarm.authelia.url is null: every hive - authenticates to this collector as itself, and the token comes - from the swarm's identity provider. - - Point authelia.url at the swarm's provider, or enable - services.hyperhive.swarm.authelia on the host that runs it. - ''; - } - { - # A port collision between two listeners on one host is a runtime - # coin toss with nothing in any log — the failure this whole - # comment budget exists to prevent. Checked against every port - # reachable from here; a port some other module picks is not. - # - # ⚠️ `cfg.port` is deliberately absent from `others`: it is the - # FIRST element of the derived range, so listing it would make this - # assertion fire on every config. - assertion = - let - derived = lib.attrValues hivePorts; - others = [ - cfg.telemetryPort - otelCfg.collector.port - ] - ++ lib.optional vmCfg.enable vmCfg.port; - all = derived ++ others; - in - lib.length (lib.unique all) == lib.length all; - message = '' - services.hyperhive.swarm.otel: the receiver range starting at - port (${toString cfg.port}, one port per hive in - services.hyperhive.swarm.hives) overlaps another port on this - host. - - Every swarm container shares the host's network namespace, so - two listeners claiming one port is not a build failure — it is - whichever process started first, silently. Move - services.hyperhive.swarm.otel.port to a free range. - ''; - } ]; containers.${cfg.machine} = { @@ -317,16 +179,12 @@ in # Read-only, and only when one is configured — binding a path that # does not exist makes nixos-container refuse to start the container, # which is a stall several layers from its cause. - bindMounts = - lib.optionalAttrs (otelCfg.headersCredential != null) { - ${otelCfg.headersCredential} = { - hostPath = otelCfg.headersCredential; - isReadOnly = true; - }; - } - # The public hive CA, read-only — only when something in here - # actually verifies a swarm-service name. - // caTrust.bindMount; + bindMounts = lib.optionalAttrs (otelCfg.headersCredential != null) { + ${otelCfg.headersCredential} = { + hostPath = otelCfg.headersCredential; + isReadOnly = true; + }; + }; config = { ... }: @@ -342,18 +200,6 @@ in inherit (config.services.hyperhive.network) bridgeIp; dnsConsumers = [ "opentelemetry-collector.service" ]; }) - ] - # `SSL_CERT_FILE` REPLACES the trust store rather than adding to - # it, so a failed assembly yields an empty pool and every TLS - # call fails while the unit looks healthy. That is why this is - # the shared helper — it carries the `Requires` and the - # non-empty check — and not a local `cat`. - ++ [ - (caTrust.trustBundle { - inherit pkgs; - name = cfg.machine; - consumers = [ "opentelemetry-collector" ]; - }) ]; system.stateVersion = config.system.stateVersion; @@ -373,24 +219,7 @@ in # real sample through both tiers into the store. validateConfigFile = true; settings = { - # One receiver per hive, and that multiplicity is forced - # rather than chosen. The `hive` - # label has to come from something the sender cannot write, - # and the only such thing here is WHICH RECEIVER accepted - # the sample: a processor cannot read the token's claims - # (`from_context` reads request metadata, and asking it for - # an auth claim yields nothing — silently, with a healthy - # startup), and one receiver holding many credentials never - # reveals which one matched. - receivers = lib.mapAttrs' ( - h: p: - lib.nameValuePair "otlp/${h}" { - protocols.http = { - endpoint = "127.0.0.1:${toString p}"; - auth.authenticator = "oidc/${h}"; - }; - } - ) hivePorts; + receivers.otlp.protocols.http.endpoint = "127.0.0.1:${toString cfg.port}"; exporters = lib.optionalAttrs vmCfg.enable { @@ -434,64 +263,16 @@ in } ]; - # ⚠️ An extension that is configured but not listed here is - # INERT — the collector starts clean and the receiver - # naming it authenticates nothing. Derived from the same - # attrset as the receivers so the two cannot disagree. - service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts); - - # Fan-out, not a choice: with both configured the same - # samples go upstream AND into the swarm's store. The store - # is for looking at this swarm; the upstream is for whoever - # aggregates across swarms, and neither replaces the other. - # `exporterNames` is shared by every pipeline — where a - # sample goes is a property of this tier, not of the hive - # that sent it. - service.pipelines = lib.mapAttrs' ( - h: _: - lib.nameValuePair "metrics/${h}" { - receivers = [ "otlp/${h}" ]; - processors = [ "resource/${h}" ]; - exporters = exporterNames; - } - ) hivePorts; - } - // { - extensions = lib.mapAttrs' ( - h: _: - lib.nameValuePair "oidc/${h}" { - issuer_url = autheliaCfg.url; - # The audience this hive's client is registered to - # request, and the reason one hive's token is refused by - # another hive's receiver. Same expression authelia - # registers it under — a second spelling here would deny - # every hive, as a 401 that blames the token. - audience = "${autheliaCfg.hiveClientPrefix}${h}"; - # ⚠️ `issuer_ca_path`. `issuer_ca_file`, `ca_file` and - # `tls.ca_file` are all INVALID KEYS for this extension - # — measured, and the failure is a startup error naming - # the key rather than anything about certificates. - issuer_ca_path = caBundle; - } - ) hivePorts; - - # `upsert`, not `insert`: a sender that stamps its own - # `hive` must be OVERWRITTEN, not deferred to. This - # processor is the whole attribution boundary — the value - # is a constant per receiver, so it says which hive - # authenticated, not which hive claimed to be sending. - processors = lib.mapAttrs' ( - h: _: - lib.nameValuePair "resource/${h}" { - attributes = [ - { - key = "hive"; - value = h; - action = "upsert"; - } - ]; - } - ) hivePorts; + service.pipelines.metrics = { + receivers = [ "otlp" ]; + # Fan-out, not a choice: with both configured the same + # samples go upstream AND into the swarm's store. The store + # is for looking at this swarm; the upstream is for whoever + # aggregates across swarms, and neither replaces the other. + exporters = + lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp") + ++ lib.optional vmCfg.enable "otlphttp/victoriametrics"; + }; }; };