Compare commits

...
Author SHA1 Message Date
atlas
41f0d7e036 fix(#3462): apply the name check in the unit that runs on the deploy
hive-tls-ca re-signs at service activation -- the rebuild itself --
while hive-tls-resign only fires from a weekly timer. The previous commit
put the coverage check in the timer unit, so a corrected serviceDomains
would not have taken effect until up to a week after the deploy that
changed it. Same bug one level along: found a trigger, not the trigger.

Two guards now share one definition rather than each carrying their own,
because a rule enforced in one and not the other is worse than one
enforced in neither -- it looks fixed and only fires on whichever path
you did not take.

Also widens hive-tls-ca's condition to the services leaf. Both leaves are
signed inside that block but only the hive leaf gated it, so a fresh
gateway.pem suppressed the re-signing of a swarm-services.pem that was
missing or stale.
2026-08-18 21:54:38 +02:00
atlas
2d2f16406f fix(#3462): re-sign a leaf when it stops covering the configured names
Expiry was the only re-sign trigger, so a leaf signed when the name set
was smaller stayed valid -- and wrong -- for its whole lifetime. Adding a
service to swarm.serviceDomains reissues the sub-CA (its own .names
reconciliation) but nothing regenerated the leaf nginx actually serves,
which left the previous commit's config change unable to fix anything on
a hive whose gateway leaf was not near expiry.

covers() reads the DNS names back out of the certificate rather than a
sidecar file: the pem is what nginx serves, and a bookkeeping file drifts
from it the moment a leaf is replaced by hand. Applied to both leaves --
the hive leaf has the same defect if the hive domain ever changes.
2026-08-18 21:54:38 +02:00
atlas
2dea5798e4 fix(#3462): the swarm-services leaf never covered grafana, metrics or otel
swarm.serviceDomains is what gateway.lib.tlsFor consults to pick the
services leaf over the hive leaf. grafana, victoriametrics and otel each
claim a gateway name under the swarm apex but were absent from that list,
so their vhosts were served the HIVE certificate -- which cannot cover a
name under a different apex.

Invisible until a machine client hit it: a name mismatch is a
click-through warning in a browser and an outright refusal in an OTLP
exporter. The metrics UI and store looked healthy while the collector
failed every POST and dropped the samples.

  tls: failed to verify certificate: x509: certificate is valid for
  probe.example, *.probe.example, not otel.swarm.example
2026-08-18 21:54:38 +02:00
2 changed files with 111 additions and 19 deletions

View file

@ -202,6 +202,67 @@ let
echo "no swarm-services sub-CA at $servicesCa skipping the services leaf"
fi
'';
# Shared by BOTH units that decide whether to re-sign. It lives here
# rather than in one of them because the two guards have to agree: they
# answer the same question at different times (`hive-tls-ca` at service
# activation, i.e. on the deploy; `hive-tls-resign` from a weekly timer
# for a host that stays up long enough to drift). A rule implemented in
# one and not the other is worse than one implemented in neither — it
# looks fixed and only fires on whichever path you did not take, which
# is exactly how a corrected `serviceDomains` still served a stale leaf.
#
# Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and
# `covers`.
leafCoverage = ''
svcleaf="$d/swarm-services.pem"
hiveNames=${lib.escapeShellArg "${domain} *.${domain}"}
svcNames=${lib.escapeShellArg (lib.concatStringsSep " " swarmServiceDomains)}
# The services leaf is only expected where the sub-CA exists;
# elsewhere its absence is the correct state, not a stale leaf.
want_svc=${if swarmServiceDomains == [ ] then "0" else "1"}
if [ ! -s ${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} ]; then
want_svc=0
fi
# Expiry is not the only way a leaf goes wrong. One signed when the
# configured name set was smaller stays valid for its whole lifetime
# while omitting every name added since — so a config change can
# evaluate, build and deploy cleanly while the gateway keeps serving a
# certificate that does not cover the new service. The services sub-CA
# already reconciles this way (its `.names` comparison in
# ../swarm-ca.nix); the leaves did not.
#
# Reads the names out of the CERTIFICATE, not a sidecar file: the pem
# is what nginx serves, and a bookkeeping file drifts from it the
# moment anyone replaces a leaf by hand.
covers() { # $1 = leaf, $2 = space-separated names it must carry
[ -s "$1" ] || return 1
_have="$(openssl x509 -in "$1" -noout -ext subjectAltName 2>/dev/null \
| tr ',' '\n' | sed -n 's/^[[:space:]]*DNS:\(.*\)$/\1/p' || true)"
[ -n "$_have" ] || return 1
_missing=0
# ⚠️ The hive leaf carries `*.<domain>`; without this the shell
# expands it against the cwd and the comparison silently tests a
# filename instead of a name.
set -f
for _n in $2; do
printf '%s\n' "$_have" | grep -qxF "$_n" || _missing=1
done
set +f
[ "$_missing" = 0 ]
}
# True when every leaf this host is supposed to hold is present and
# carries its configured names. Expiry is the callers' own business —
# they use different windows.
leavesCoverNames() {
covers "$leaf" "$hiveNames" || return 1
[ "$want_svc" = 0 ] && return 0
covers "$svcleaf" "$svcNames"
}
'';
in
{
# Host-side TLS trust root for the self-signed gateway mode.
@ -425,11 +486,31 @@ in
rm -f "$leaf" "$leafk"
fi
# --- Leaf: (re)sign when missing or within 30 days of expiry,
# always under the current (stable) CA.
if [ ! -s "$leaf" ] || [ ! -s "$leafk" ] \
|| ! openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1; then
echo "signing fresh gateway leaf at $leaf"
# --- Leaf: (re)sign when missing, within 30 days of expiry, or no
# longer covering the configured names, always under the current
# (stable) CA.
${leafCoverage}
# ⚠️ This is the guard that runs ON THE DEPLOY — `hive-tls-resign`
# only fires from a weekly timer, so a rule enforced only there is
# up to a week late and does nothing for the rebuild that changed
# the names in the first place.
#
# The name check has to include the SERVICES leaf even though the
# condition is written around the hive one, because both are signed
# in this block: a fresh `gateway.pem` otherwise suppresses the
# re-sign of a `swarm-services.pem` that is missing or stale, which
# is what left a corrected `serviceDomains` still mis-served.
resign=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1
openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1
leavesCoverNames || resign=1
if [ "$want_svc" = 1 ] && [ ! -s "$svcleaf" ]; then
resign=1
fi
if [ "$resign" = 1 ]; then
echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)"
${signHiveLeaf}
${signServicesLeaf}
fi
@ -500,11 +581,14 @@ in
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
leaf="$d/gateway.pem"
# ⚠️ EVERY leaf this host issues must be listed here. A leaf that
# ⚠️ EVERY leaf this host issues must be covered here. A leaf that
# first-boot issuance creates and this unit does not know about
# looks perfect for its entire validity and then expires with no
# warning — the failure is invisible until it is total.
svcleaf="$d/swarm-services.pem"
# warning — the failure is invisible until it is total. `svcleaf`
# and the name checks come from the shared snippet, so this unit
# and `hive-tls-ca` cannot disagree about what a good leaf is.
${leafCoverage}
# Re-sign only when a leaf is within half its validity of expiry.
# The weekly cadence catches this window well before one lapses.
@ -514,19 +598,13 @@ in
[ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1
}
# The services leaf is only expected where the sub-CA exists;
# elsewhere its absence is the correct state, not a stale leaf.
want_svc=${if swarmServiceDomains == [ ] then "0" else "1"}
if [ ! -s ${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} ]; then
want_svc=0
fi
if fresh "$leaf" && { [ "$want_svc" = 0 ] || fresh "$svcleaf"; }; then
echo "leaves valid for more than half their lifetime no resign needed"
if fresh "$leaf" && { [ "$want_svc" = 0 ] || fresh "$svcleaf"; } \
&& leavesCoverNames; then
echo "leaves valid, and covering the configured names no resign needed"
exit 0
fi
echo "a leaf is missing or near expiry re-signing under the current CAs"
echo "a leaf is missing, near expiry, or missing a configured name re-signing"
before="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
${signHiveLeaf}

View file

@ -45,7 +45,21 @@ let
# `forge.<apex>` is, and no CA in the hierarchy issues for it
# implicitly. Left out, its vhost falls back to the hive leaf and the
# swarm's front page opens with a name mismatch.
++ lib.optional swarmCfg.ui.enable swarmCfg.ui.domain;
++ lib.optional swarmCfg.ui.enable swarmCfg.ui.domain
# Every swarm service that claims a gateway name belongs here, and
# these three were missing it. Membership is what `gateway.lib.tlsFor`
# consults to pick the services leaf over the hive one, so a name
# absent from this list is served the HIVE certificate — which cannot
# cover a name under a different apex however the sub-CA is set up.
#
# ⚠️ How that stayed invisible: a mismatch a browser shows as a
# click-through warning is one a machine client rejects outright. The
# metrics UI and store looked fine for as long as only people opened
# them; the collector's exporter — same defect, no human in the loop —
# failed every POST and dropped the samples.
++ lib.optional swarmCfg.grafana.enable swarmCfg.grafana.domain
++ lib.optional swarmCfg.victoriametrics.enable swarmCfg.victoriametrics.domain
++ lib.optional swarmCfg.otel.enable swarmCfg.otel.domain;
# Hives whose entry still carries the removed `certFingerprint`. Scanned
# here, at top level, because that is the only place an assertion about a