argus, reviewing, ran the guard against a source path that does not
exist rather than one that is empty. grep writes nothing to stdout in
that case, so `|| true` left the count variable empty and the -eq test
died with "integer expected" instead of reporting.
The unit still failed — cat hits the same missing file and set -e stops
it — but with a generic "no such file" rather than the message naming
which half is absent, which is the only thing this guard is for.
`|| echo 0` on all three counts. The gate gained the arm that was
missing: an absent source, asserted to fail THROUGH the guard rather
than merely to fail.
The guard inspected the assembled file for any certificate. The system
store always holds certificates, so it passed unconditionally — including
in the one case it was written to catch, where the hive CA half
contributed nothing.
That half is the only one that matters here: every name these consumers
verify is issued by our own CA, so a bundle of nothing but public CAs is,
for this purpose, an empty bundle that measures as full. The failure is
silent and total — the unit reports success and every egress TLS call to
a swarm service then fails.
Counts the source on its own before assembling, and checks the result
carries what both halves brought, so a source truncated between the count
and the copy is caught too.
Scope is stated at the guard: it proves the anchor was contributed, not
that it is usable. A consumer reading only the first certificate ignores
it regardless, which is what took the swarm collector down, and no check
on this file can see that. Only a handshake can.
The swarm collector could never verify its OIDC issuer, so it exited at
startup on every boot and the hive tier dropped every metric.
`issuer_ca_path` loads only the FIRST certificate in the file it names.
The bundle assembled for this container is `system CAs ++ hive anchors`,
so the swarm CA sits ~123rd and was never in the pool: the extension got
whichever public CA sorts first, could verify nothing of ours, and failed
`x509: certificate signed by unknown authority` — with 125 valid
certificates in the file.
Leaving the option unset makes the extension use the process trust store,
which `trustBundle` already populates via `SSL_CERT_FILE`, and that
consumer reads every certificate regardless of order. One file, two
consumers, opposite parsing; the fix is to stop naming it twice rather
than to reorder the bundle.
Measured with the deployed binary against the deployed config, varying
only the CA source: root-only OK, root-first OK, root-last FAILS,
root-second FAILS, and unset-with-SSL_CERT_FILE OK against a control that
correctly fails when the anchor is absent.
The swarm collector accepted OTLP from anyone who could reach it, and took
the `hive` resource attribute from the payload. So any writer on the swarm
network could attribute metrics to any hive, and nothing downstream could
tell.
The label now comes from which receiver accepted the sample: one receiver
per hive, each behind an `oidc` extension verifying a token minted for that
hive's audience, each feeding a pipeline whose `resource` processor upserts
a constant. A sender cannot influence it, because the only input is which
authenticated port the bytes arrived on.
That multiplicity is forced rather than preferred. A processor cannot read
the token's claims — `from_context` reads request metadata, and asking it
for an auth claim yields nothing, silently, with a healthy startup — and
one receiver holding several credentials never reveals which one matched.
The per-hive ports are internal: a hive reaches its receiver as a path
under this collector's existing gateway name, so nginx (rendered from this
same evaluation) is the only thing that names a port. Fronting each hive
with its own vhost would need a certificate, a DNS name and a gateway entry
per hive to express routing the gateway already does.
Turning this on removes the unauthenticated receiver. While an open port
still accepts samples the per-hive receivers are decoration, so this is the
switch itself rather than a hardening layer beside it; a swarm that wants
the open receiver says so.
`hive-ca-trust.nix` grows `bundlePathFor`, because a consumer taking its own
CA argument has to name the bundle rather than just have `SSL_CERT_FILE`
exported at it.
The controller's forge client speaks TLS to `https://<forge domain>`,
which the gateway serves with a leaf signed by the hive CA. That CA is
generated at runtime, so nothing build-time can name it and it is not in
the system store -- and `reqwest`/`rustls` resolves roots through
`rustls-native-certs`, whose `SSL_CERT_FILE` *replaces* the store rather
than adding to it. With no bundle wired, every forge call failed
`invalid peer certificate: UnknownIssuer` and agent creation died at its
first step.
`lib/hive-ca-trust.nix` already solved this, but only for containers: it
sources the CA through `/run/hive-ca/trust-bundle.pem`, a bind mount that
does not exist on the host. `hostUnit` makes it read the host copy and
wait on `hive-tls-ca.service` itself -- one flag driving both, because a
host source without that ordering is a race.
`enable` is the other half, and it is the sharp edge: a container caller
imports this into the container's module set, so it disappears with the
container. A host caller imports it at the host's top level, where
`imports` is unconditional -- without the flag, a hive with the
controller turned off would get a bundle oneshot and a `swarm-controller`
service conjured by `genAttrs`, holding an `SSL_CERT_FILE` and no
`ExecStart`.
Last of the three modules that hand-rolled the same concat with wantedBy +
before and no requires, so a failed assembly left the consumer running against
a missing file and trusting nothing -- every outbound TLS call fails while the
unit looks healthy.
The forge is the one with two consumers: forgejo-sso-source fetches the
issuer's discovery document over the swarm CA and once shipped without the
trust its sibling had. It only exists when SSO is on, so the consumer list is
conditional -- naming an absent unit would define a serviceless one and order
nothing.
Removes the now-dead useSelfSigned and caContainerPath bindings (nix does not
warn) and retargets three comments the deletion orphaned, including the helper
header that still named this module as the per-call-site concat.
The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
Cross-hive trust was O(n²) hand-pinning: every hive had to name every
peer's CA. A swarm root makes it O(1) — trust the root once and every
present and future peer validates.
The root is generated by a new `swarm-ca` unit on a single-host swarm
and operator-provided otherwise; `swarm.ca.autoConfigure` picks between
them and derives its default from `swarm.peers` being empty, so "all on
one host" is read off the deployment rather than remembered. Both modes
produce the same artifacts in the same places, so splitting hosts later
is moving the service dirs, not switching code paths. The root key never
enters the nix store, and the root is never regenerated automatically —
replacing it invalidates every peer at once.
Each hive CA carries `nameConstraints` pinned to that hive's domain, so
a leaked hive CA can only mint names inside its own subdomain, enforced
by verifiers rather than by convention.
`ca.pem` was serving as both the issuer and the anchor consumers trust;
those are the same file only while it is self-signed. openssl will not
terminate a chain at a trusted cert that isn't self-signed (rustls and
Go will), so the promotion would have broken some consumers and not
others. `hive-tls-ca` now also writes `trust-bundle.pem` — the hive CA
plus whatever it is rooted at — and every anchor consumer reads that:
agents, the CI and forge containers, and the peer-config recipe. On a
hive with no swarm root the bundle is just that CA, so nothing consuming
it needs a mode to branch on.
hive-ci and hive-forge both bind the runtime-generated hive CA cert
read-only and order their container@ unit after hive-tls-ca.service so
the bind source exists before nspawn sets the mount up — the same
bind-mount + ordering + rationale duplicated verbatim in two modules.
Extract that language-agnostic half into a pure helper,
nix/host-modules/lib/hive-ca-trust.nix, taking a container name and
returning { useSelfSigned, caContainerPath, bindMount, containerOrdering }.
The per-runtime consumption stays at each call site (hive-ci's additive
NODE_EXTRA_CA_CERTS, hive-forge's Go SSL_CERT_FILE concat). hive-ci folds
containerOrdering into its existing mkMerge alongside the TimeoutStartSec
bump.
The helper is a pure function, not a module: host-modules/default.nix is
an explicit aggregator (not a glob) and the docs eval imports that same
aggregator, so the lib/ file is never picked up as a module. A third
outbound-TLS-trusting container no longer means a third copy-paste.
