From b8b571fea0818b64e8a926c375e5cea5d4ea3f03 Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 19 Aug 2026 17:54:26 +0200 Subject: [PATCH] fix(#3524): stop naming the trust bundle as the oidc issuer anchor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The swarm collector could never verify its OIDC issuer, so it exited at startup on every boot and the hive tier dropped every metric. `issuer_ca_path` loads only the FIRST certificate in the file it names. The bundle assembled for this container is `system CAs ++ hive anchors`, so the swarm CA sits ~123rd and was never in the pool: the extension got whichever public CA sorts first, could verify nothing of ours, and failed `x509: certificate signed by unknown authority` — with 125 valid certificates in the file. Leaving the option unset makes the extension use the process trust store, which `trustBundle` already populates via `SSL_CERT_FILE`, and that consumer reads every certificate regardless of order. One file, two consumers, opposite parsing; the fix is to stop naming it twice rather than to reorder the bundle. Measured with the deployed binary against the deployed config, varying only the CA source: root-only OK, root-first OK, root-last FAILS, root-second FAILS, and unset-with-SSL_CERT_FILE OK against a control that correctly fails when the anchor is absent. --- nix/host-modules/lib/hive-ca-trust.nix | 26 ++++++++++++++++---- nix/host-modules/swarm-otel.nix | 33 +++++++++++++++++++++----- 2 files changed, 48 insertions(+), 11 deletions(-) diff --git a/nix/host-modules/lib/hive-ca-trust.nix b/nix/host-modules/lib/hive-ca-trust.nix index 0ad563ea..e88065c8 100644 --- a/nix/host-modules/lib/hive-ca-trust.nix +++ b/nix/host-modules/lib/hive-ca-trust.nix @@ -52,11 +52,27 @@ in # Where `trustBundle` below puts the assembled bundle, for the callers # that must NAME it rather than just have it exported. `SSL_CERT_FILE` is # set for you and needs no path here; a consumer that takes its own CA - # argument (an OIDC verifier's `issuer_ca_path`, a client's `--cacert`) - # does, and the alternative is copying `/run/-ca/…` to the call - # site. That copy breaks silently: the bundle keeps being written, the - # consumer keeps reading a path that no longer exists, and the failure - # surfaces as a TLS error naming the peer rather than the file. + # argument (a client's `--cacert`) does, and the alternative is copying + # `/run/-ca/…` to the call site. That copy breaks silently: the + # bundle keeps being written, the consumer keeps reading a path that no + # longer exists, and the failure surfaces as a TLS error naming the peer + # rather than the file. + # + # ⛔ **Do not hand this path to a consumer that reads only ONE + # certificate from it.** This comment used to offer an OIDC verifier's + # `issuer_ca_path` as the motivating example; that is exactly what took + # the swarm collector down for forty minutes once. The bundle is + # `system CAs ++ hive anchors`, so the anchor is ~123rd, and a + # first-certificate-only reader gets whichever public CA sorts first and + # can verify nothing of ours — with a full, valid, 125-certificate file + # on disk and no check able to see it. + # + # 🔑 The general trust store (`SSL_CERT_FILE`, set by `trustBundle`) + # reads every certificate and is order-independent, which is why a + # consumer that can use the process trust store should simply be left to + # do so. Before naming this path, check how that consumer parses it — + # the requirement is a property of the *reader*, and nothing here can + # enforce it. inherit bundlePathFor; # Fold into the container's `bindMounts` via `//`. Binds ONLY the public diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 8f23d3ca..bcbfee5a 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -68,7 +68,6 @@ let tlsCfg = hyperhiveCfg.tls; inherit gatewayCfg; }; - caBundle = caTrust.bundlePathFor cfg.machine; # `unknown` rather than omitting the label, copying `agent-modules/otel.nix` # deliberately: a producer that cannot name its swarm should say so in the @@ -473,11 +472,33 @@ in # registers it under — a second spelling here would deny # every hive, as a 401 that blames the token. audience = "${autheliaCfg.hiveClientPrefix}${h}"; - # ⚠️ `issuer_ca_path`. `issuer_ca_file`, `ca_file` and - # `tls.ca_file` are all INVALID KEYS for this extension - # — measured, and the failure is a startup error naming - # the key rather than anything about certificates. - issuer_ca_path = caBundle; + # ⛔ DO NOT ADD `issuer_ca_path` HERE. It took this + # collector down for forty minutes once, and the failure + # is invisible to every check we have. + # + # It loads only the FIRST certificate in the file it + # names. The bundle assembled for this container is + # `system CAs ++ hive trust bundle`, so the anchor sits + # ~123rd and is never in the pool: the extension then + # cannot verify authelia and the whole collector exits + # `x509: certificate signed by unknown authority`, on + # every start, with 125 valid certificates in the file. + # + # Leaving it unset makes the extension use the process + # trust store, which `trustBundle` already populates via + # `SSL_CERT_FILE` — and *that* consumer reads every + # certificate regardless of order. One file, two + # consumers, opposite parsing: the fix is to stop naming + # it twice, not to reorder the bundle. + # + # ⚠️ Nor is pointing it at the hive trust bundle a fix: + # `hive-tls.nix` writes that leading with the *hive* CA + # (`nameConstraints` = this hive's domain), which cannot + # issue a swarm-level name at all. + # + # (Kept from the original note, still true and still + # worth not re-deriving: `issuer_ca_file`, `ca_file` and + # `tls.ca_file` are INVALID KEYS for this extension.) } ) hivePorts;