vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

nix: issue the swarm-services leaf from bao's pki mount

Browse source 
The `pki` mount had no issuer and no principal could log in to it, so the
swarm's service certificates were still minted by two openssl hops from a
root key on disk. Close both halves and retire the openssl path with them.

The mount now generates its own root, once. The granting unit asks bao
whether an issuer already exists (`bao list pki/issuers`) before calling
`pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role
and the grant without touching the anchor — a root that changed per boot
would invalidate every certificate issued under it and every browser
taught to trust it. The guard asks the store rather than looking for a
marker file on this host's disk: a file is a claim about a mount that may
have been restored from a snapshot or disabled and re-enabled underneath
it.

`swarm-services-issuer` stops being an inert policy. A fourth cert-auth
role attaches it, following the shape the controller, the publisher and
matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its
CN — that credential is what opens the mount, so it cannot come out of it.

`swarm-services-cert.service` logs in with that leaf, calls
`pki/issue/swarm-services`, and writes the result to the path
hive-tls.nix already wrote and the gateway already copies from. The
sub-CA layer does not move; it stops existing. The role's
`allowed_domains`, read from the same `swarm.serviceDomains` the SANs
come from, enforces at issue time what the sub-CA encoded in x509
`nameConstraints`, and with the root inside the mount there is nothing
left for an intermediate to be an intermediate of.

Not a flag day: the issuing root is published beside the leaf as
`swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the
swarm root still sits. A leaf chaining to the old sub-CA and one issued
by the store both verify against the same bundle, so hives can be
rebuilt in any order. The same file is what an operator hands a browser
— readable without a store login, which matters because every listener
demands a client certificate.

The eval-time warning about uncovered service names is gone rather than
reworded. It fired on "this host does not hold the swarm root key", which
was the reason a hive could end up serving its own leaf on a
swarm-service name. Every hive now asks the store with its own identity,
so that stopped being the thing that decides.

Closes #4586
This commit is contained in:
atlas 2026-09-21 17:59:22 +02:00 • committed by mara
commit f4df4fc4a9
10 changed files with 700 additions and 315 deletions
Download patch file Download diff file Expand all files Collapse all files

512
nix/host-modules/hive-tls.nix
View file

@ -10,26 +10,22 @@ let
gatewayCfg = config.services.hyperhive.gateway;
swarmCaCfg = config.services.hyperhive.swarm.ca;
domain = hyperhiveCfg.domain;
# Derived once in ./swarm.nix and read here + in ./swarm-ca.nix, so
# the names this leaf carries as SANs and the names the sub-CA is
# constrained to cannot disagree.
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
# The endpoint the services leaf comes out of. Read off the store's own
# options rather than spelled here: ./swarm-bao.nix enables the mount,
# generates its root and writes the grant that names this exact path, and a
# second module composing its own is how the grant and the caller stop
# agreeing.
baoServicesPkiMount = baoDeploy.servicesPkiMountPath;
baoServicesPkiRole = baoDeploy.servicesPkiRoleName;
# Derived once in ./swarm.nix and read here + in ./swarm-bao.nix, so
# the names this leaf carries as SANs and the names the store's
# `pki/roles/swarm-services` is narrowed to cannot disagree.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# True for the names `signHiveLeaf` below actually covers: the hive
# domain itself, or ONE label under it. `*.<domain>` is a single-label
# wildcard — `a.b.<domain>` does not match it — so the depth check is
# the whole point rather than a nicety.
coveredByHiveLeaf =
n:
n == domain
|| (lib.hasSuffix ".${domain}" n && !lib.hasInfix "." (lib.removeSuffix ".${domain}" n));
# Service names this host can serve a *matching* certificate for, and
# the ones it can't. A name is issuable here when the hive leaf covers
# it, or when this host issues the swarm-services leaf — which needs
# the swarm root's private key, i.e. `swarm.ca.autoConfigure`.
uncoveredServiceDomains = lib.filter (n: !coveredByHiveLeaf n) swarmServiceDomains;
# The host-managed hive CA is the trust anchor for self-signed mode.
# It is only stood up when the gateway actually serves a self-signed
# cert: the gateway must be in self-signed mode. `domain` is required
@ -182,57 +178,38 @@ let
"$d/ca.pem" "$d/ca-key.pem"
'';
# The swarm-services leaf: signed by the services sub-CA, covering the
# swarm's service names. Those are *siblings* of the hive domain, not
# children, so the hive CA is name-constrained out of them and cannot
# sign this however its SAN list is written.
#
# Skipped when the sub-CA isn't on disk: it only exists where the
# swarm CA is autoconfigured, and a hive that gets its certs from its
# operator has nothing for this to do.
signServicesLeaf = lib.optionalString (swarmServiceDomains != [ ]) ''
servicesCa=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"}
servicesCaKey=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca-key.pem"}
if [ -s "$servicesCa" ] && [ -s "$servicesCaKey" ]; then
${signLeafScript} "$d" swarm-services \
${lib.escapeShellArg (builtins.head swarmServiceDomains)} \
${lib.escapeShellArg (lib.concatMapStringsSep "," (n: "DNS:${n}") swarmServiceDomains)} \
"$servicesCa" "$servicesCaKey"
else
echo "no swarm-services sub-CA at $servicesCa — skipping the services leaf"
fi
'';
# Shared by BOTH units that decide whether to re-sign. It lives here
# rather than in one of them because the two guards have to agree: they
# answer the same question at different times (`hive-tls-ca` at service
# activation, i.e. on the deploy; `hive-tls-resign` from a weekly timer
# for a host that stays up long enough to drift). A rule implemented in
# one and not the other is worse than one implemented in neither — it
# looks fixed and only fires on whichever path you did not take, which
# is exactly how a corrected `serviceDomains` still served a stale leaf.
# Shared by every unit that decides whether to re-issue a leaf. It
# lives here rather than in one of them because the guards have to
# agree: they answer the same question at different times
# (`hive-tls-ca` at service activation, i.e. on the deploy;
# `hive-tls-resign` from a weekly timer for a host that stays up long
# enough to drift; `swarm-services-cert` for the leaf the store
# issues). A rule implemented in one and not the others is worse than
# one implemented in none — it looks fixed and only fires on whichever
# path you did not take, which is exactly how a corrected
# `serviceDomains` still served a stale leaf.
#
# Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and
# `covers`.
leafCoverage = ''
svcleaf="$d/swarm-services.pem"
svcroot="$d/swarm-services-root.pem"
hiveNames=${lib.escapeShellArg "${domain} *.${domain}"}
svcNames=${lib.escapeShellArg (lib.concatStringsSep " " swarmServiceDomains)}
# The services leaf is only expected where the sub-CA exists;
# elsewhere its absence is the correct state, not a stale leaf.
# A hive with no configured service names has no services leaf to
# hold, and its absence there is the correct state rather than a
# stale one. Anywhere else it is expected: the leaf comes from the
# secret store's `pki` mount now, not from a sub-CA that exists on
# one host in the swarm, so "this host cannot issue it" is no longer
# one of the answers.
want_svc=${if swarmServiceDomains == [ ] then "0" else "1"}
if [ ! -s ${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} ]; then
want_svc=0
fi
# Expiry is not the only way a leaf goes wrong. One signed when the
# configured name set was smaller stays valid for its whole lifetime
# while omitting every name added since — so a config change can
# evaluate, build and deploy cleanly while the gateway keeps serving a
# certificate that does not cover the new service. The services sub-CA
# already reconciles this way (its `.names` comparison in
# ../swarm-ca.nix); the leaves did not.
# certificate that does not cover the new service.
#
# Reads the names out of the CERTIFICATE, not a sidecar file: the pem
# is what nginx serves, and a bookkeeping file drifts from it the
@ -254,13 +231,20 @@ let
[ "$_missing" = 0 ]
}
# True when every leaf this host is supposed to hold is present and
# carries its configured names. Expiry is the callers' own business —
# they use different windows.
# True when the leaf this host SIGNS is present and carries its
# configured names. Expiry is the callers' own business — they use
# different windows.
#
# ⚠️ The services leaf is deliberately NOT part of this any more, and
# the omission is load-bearing rather than a simplification. Its two
# callers sign under the hive CA's key; the services leaf comes out
# of the secret store, which neither of them can reach. A guard that
# went on reporting it stale would make `hive-tls-ca` re-sign the
# hive leaf forever over a file it has no way to fix. The
# `swarm-services-cert` unit below holds the same rule for that leaf,
# using the `covers` helper above so the two cannot drift.
leavesCoverNames() {
covers "$leaf" "$hiveNames" || return 1
[ "$want_svc" = 0 ] && return 0
covers "$svcleaf" "$svcNames"
covers "$leaf" "$hiveNames"
}
'';
in
@ -313,6 +297,44 @@ in
'';
};
baoClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer.pem";
description = ''
Certificate `swarm-services-cert` presents to the secret store when it
asks for the swarm-services leaf the gateway serves.
Its subject has to be
{option}`services.hyperhive.deploy.bao.servicesIssuerCommonName` — cert
auth matches on the common name, and this is the one role whose policy
permits `pki/issue/swarm-services`.
A **separate** credential from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is this
hive's own reader identity and whose policy reads every secret the
hive is entitled to. A unit that renews one server certificate has no
business holding that.
On a hive that mints its own store PKI a glue module supplies the leaf
as a `mkDefault`; everywhere else it is the credential an operator
places, for the reason the store's own client certificate is.
A path, never a value.
'';
};
baoClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.hive-controller.tls.baoClientCertFile`.
Both or neither — a certificate with no key authenticates nothing.
'';
};
leafValidityDays = lib.mkOption {
type = lib.types.int;
default = 30;
@ -333,40 +355,17 @@ in
};
config = lib.mkIf active {
# A swarm service name that is a sibling of the hive domain rather
# than a child needs the swarm-services leaf, and this host only
# signs that one when it holds the swarm root key. Without it nginx
# falls back to the hive leaf on those names and every client sees a
# name mismatch — on a config that evaluates and deploys cleanly.
#
# ⚠️ A WARNING, NOT AN ASSERTION, and the distinction is the point:
# this module can see what *it* can issue; it cannot see an
# operator-installed services sub-CA, an external ACME setup, or a
# cert delivered by any other means. A rebuild must not be blocked
# by a conclusion this host is not in a position to reach. Report
# the observation and let the operator judge it.
#
# (`certDir` / ACME modes don't reach here at all: `active` is
# self-signed-only, and there the operator's cert decides.)
warnings = lib.optional (uncoveredServiceDomains != [ ] && !swarmCaCfg.autoConfigure) ''
services.hyperhive: these swarm service names are outside this
hive's domain (${domain}), so the hive CA's leaf does not cover
them:
${lib.concatMapStringsSep "\n" (n: " ${n}") uncoveredServiceDomains}
services.hyperhive.swarm.ca.autoConfigure is false, so this host
does not sign the swarm-services leaf either, and the gateway will
serve the hive leaf on those names — a certificate-name mismatch
for browsers and for agents' git-over-https.
If you have already arranged certificates for them — an
operator-installed services sub-CA in ${swarmCaCfg.stateDir}, or
services.hyperhive.gateway.tls.{certDir,acme} — this is expected
and you can ignore it. Otherwise pin the names back under
${domain} (services.hyperhive.swarm.{forge.domain,
matrix.gatewayHost, authelia.domain}) or install the sub-CA.
'';
# 🚫 The "these service names are outside this hive's domain and this
# host cannot sign for them" warning is GONE, and its absence is the
# deliverable rather than a tidy-up. It fired on
# `!swarm.ca.autoConfigure` — i.e. "this host does not hold the swarm
# root key" — which was the whole reason a hive could end up serving
# its own leaf on a swarm-service name. The services leaf now comes
# from the secret store's `pki` mount, which every hive reaches over
# the network with its own identity, so holding the root key stopped
# being the thing that decides. Left in place it would warn, on every
# rebuild of every hive that is not the CA host, about a fallback
# that no longer happens.
# Generate (and rotate) the hive CA + gateway leaf before anything
# serves it. Idempotent: the CA is created once and reused; the leaf
@ -381,26 +380,22 @@ in
# against the unit that reads the file.
before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The issuance below needs the swarm root key on disk, and (for the
# services leaf) the services sub-CA it signs under. When this host
# generates them (single-host swarm) both units must have run first;
# when the operator provides the material there is no unit to wait
# for, so the dependency is conditional rather than a unit that
# exists and does nothing.
# The issuance below needs the swarm root key on disk. When this
# host generates it (single-host swarm) that unit must have run
# first; when the operator provides the material there is no unit
# to wait for, so the dependency is conditional rather than a unit
# that exists and does nothing.
#
# Without waiting for swarm-services-ca specifically, this unit races
# it: if hive-tls-ca finishes first, it finds no services-ca.pem yet,
# silently skips signing the services leaf (the same as "operator
# hasn't set one up"), and the gateway comes up with a vhost pointed
# at a cert that was never written.
after = lib.optionals swarmCaCfg.autoConfigure [
"swarm-ca.service"
"swarm-services-ca.service"
];
requires = lib.optionals swarmCaCfg.autoConfigure [
"swarm-ca.service"
"swarm-services-ca.service"
];
# `swarm-services-cert` is ordered before this one rather than the
# other way round, and not because this unit needs the leaf — it
# never touches it. The trust bundle written at the end of this
# script anchors on every root a consumer must hold, and the
# services root is one of them, so the file has to be on disk
# before the `cat`. Get it wrong and the bundle is a boot behind:
# correct-looking, and missing the anchor for exactly the names the
# gateway serves with it.
after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
requires = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
@ -496,23 +491,14 @@ in
# up to a week late and does nothing for the rebuild that changed
# the names in the first place.
#
# The name check has to include the SERVICES leaf even though the
# condition is written around the hive one, because both are signed
# in this block: a fresh `gateway.pem` otherwise suppresses the
# re-sign of a `swarm-services.pem` that is missing or stale, which
# is what left a corrected `serviceDomains` still mis-served.
resign=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1
openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1
leavesCoverNames || resign=1
if [ "$want_svc" = 1 ] && [ ! -s "$svcleaf" ]; then
resign=1
fi
if [ "$resign" = 1 ]; then
echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)"
${signHiveLeaf}
${signServicesLeaf}
fi
# --- Trust bundle: what a consumer must TRUST, as opposed to
@ -534,10 +520,19 @@ in
# hive where the optional file is legitimately absent. The
# reflexive `|| true` is worse; it also swallows a real failure to
# read the hive CA.
#
# `swarm-services-root.pem` joins the set for the reason the swarm
# root is in it: it is an ANCHOR a consumer terminates at, not an
# intermediate. It is also what keeps this change from being a
# flag day — the swarm root stays in the bundle beside it, so a
# leaf still chaining to the old services sub-CA and one issued by
# the store both verify against the same file, and a hive can be
# rebuilt before or after its peers.
bundle="$d/trust-bundle.pem"
anchors=("$ca")
if [ -s "$prev" ]; then anchors+=("$prev"); fi
if [ -s "$root" ]; then anchors+=("$root"); fi
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
cat "''${anchors[@]}" > "$bundle"
chmod 0644 "$bundle"
'';
@ -582,12 +577,17 @@ in
d=${lib.escapeShellArg cfg.stateDir}
leaf="$d/gateway.pem"
# ⚠️ EVERY leaf this host issues must be covered here. A leaf that
# ⚠️ EVERY leaf this host SIGNS must be covered here. A leaf that
# first-boot issuance creates and this unit does not know about
# looks perfect for its entire validity and then expires with no
# warning — the failure is invisible until it is total. `svcleaf`
# and the name checks come from the shared snippet, so this unit
# and `hive-tls-ca` cannot disagree about what a good leaf is.
# warning — the failure is invisible until it is total. The name
# checks come from the shared snippet, so this unit and
# `hive-tls-ca` cannot disagree about what a good leaf is.
#
# The services leaf is not one of them any longer: it is issued
# by the secret store, whose key this unit does not hold and
# cannot re-sign under. Renewing it is `swarm-services-cert`'s
# job, at boot, which is a cadence its own issue owns.
${leafCoverage}
# Re-sign only when a leaf is within half its validity of expiry.
@ -598,19 +598,17 @@ in
[ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1
}
if fresh "$leaf" && { [ "$want_svc" = 0 ] || fresh "$svcleaf"; } \
&& leavesCoverNames; then
if fresh "$leaf" && leavesCoverNames; then
echo "leaves valid, and covering the configured names — no resign needed"
exit 0
fi
echo "a leaf is missing, near expiry, or missing a configured name — re-signing"
before="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
before="$(sha256sum "$leaf" 2>/dev/null || true)"
${signHiveLeaf}
${signServicesLeaf}
after="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
after="$(sha256sum "$leaf" 2>/dev/null || true)"
if [ "$before" != "$after" ]; then
echo "gateway leaf rotated — re-importing and reloading nginx"
systemctl restart hive-gateway-self-signed-cert.service
@ -621,6 +619,256 @@ in
'';
};
# The swarm-services leaf, issued by the secret store's `pki` mount.
#
# ⚠️ This unit replaces a chain, not a command. What used to happen
# was two openssl hops on whichever host held the swarm root key: a
# name-constrained `swarm-services` sub-CA (../swarm-ca.nix), then a
# leaf signed under it here. Both are gone. The store's
# `pki/roles/swarm-services` carries the same narrowing as an
# `allowed_domains` list enforced at issue time, so the intermediate
# that used to encode it in x509 `nameConstraints` has nothing left
# to express — and with the root inside the mount there is nothing
# for it to be an intermediate OF.
#
# The names it may carry are not this unit's to assert: the role
# refuses anything outside `allowed_domains`, which is read from the
# same `swarm.serviceDomains` the SANs below are built from. A
# mismatch is a refusal from the store naming the offending name,
# not a certificate quietly issued for something nobody configured.
systemd.services.swarm-services-cert = {
description = "Issue the swarm-services TLS leaf from the secret store's PKI";
wantedBy = [ "multi-user.target" ];
# ⚠️ The ordering the old shape got by ACCIDENT, written down.
# Nothing used to declare "no TLS until the store is up" — it held
# because `swarm-services-ca.service` happened to block the gateway.
# Now it is a real edge: the gateway's certificate comes out of the
# store, so the store being up is a precondition rather than a
# coincidence, and `requiredBy` is what makes a store that never
# comes up an outage that says so instead of a gateway serving the
# wrong name.
before = [
"hive-tls-ca.service"
"hive-gateway-self-signed-cert.service"
];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The store's container, where it runs here. On a hive that reads a
# store hosted elsewhere no such unit exists and systemd ignores
# the name, which is the correct behaviour rather than a gap: what
# this unit actually needs is the store reachable, and on a remote
# store nothing local can order against that.
after = [
"container@${baoCfg.machine}.service"
"swarm-bao-services-issuer-policy.service"
];
wants = [ "container@${baoCfg.machine}.service" ];
path = [
baoDeploy.package
pkgs.jq
pkgs.openssl
pkgs.coreutils
pkgs.systemd
];
# Sized like the store's own granting units, and for the same
# reason: under `seal = "shamir"` an operator unseals BY HAND, and
# the login below fails for as long as that takes. 2880 × 30s is
# 24h inside a 25h window — `StartLimit*` are `[Unit]` settings, so
# the window must exceed `RestartSec × burst` or it closes between
# attempts and the burst is never reached.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
UMask = "0077";
SyslogIdentifier = "swarm-services-cert";
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs (cfg.baoClientCertFile != null) {
BAO_CLIENT_CERT = cfg.baoClientCertFile;
}
// lib.optionalAttrs (cfg.baoClientKeyFile != null) {
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
}
# Absent means the system trust store, which is what a deployment
# with a real CA wants and what a self-signed one must not be left
# with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
install -d -m 0755 "$d"
${leafCoverage}
if [ "$want_svc" = 0 ]; then
echo "no swarm service domains configured — nothing to issue"
exit 0
fi
# Same rule as the hive leaf's, at a different cadence: re-issue
# when the file is missing, within 30 days of expiry, or no
# longer carrying every configured name. The name check is what
# makes adding a swarm service take effect on the rebuild that
# added it rather than whenever the certificate happens to lapse.
reissue=0
{ [ -s "$svcleaf" ] && [ -s "$d/swarm-services-key.pem" ]; } || reissue=1
openssl x509 -in "$svcleaf" -noout -checkend 2592000 >/dev/null 2>&1 || reissue=1
covers "$svcleaf" "$svcNames" || reissue=1
[ -s "$svcroot" ] || reissue=1
if [ "$reissue" = 0 ]; then
echo "swarm-services leaf valid and covering the configured names — leaving it alone"
exit 0
fi
${
if cfg.baoClientCertFile == null || cfg.baoClientKeyFile == null then
''
echo "no bao client certificate configured for this host, so the gateway's" >&2
echo "swarm-services certificate cannot be requested from the store." >&2
echo "Set services.hyperhive.deploy.hive-controller.tls.baoClient{Cert,Key}File" >&2
echo "to a leaf the store's CA signed with CN=${baoDeploy.servicesIssuerCommonName}." >&2
exit 1''
else
""
}
# `bao`'s own message is the only thing separating an unreachable
# store from a refused identity from a role that would not issue
# these names, and this unit retries on all three — so it reports
# which one rather than asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting: the
# `BAO_CLIENT_*` variables only decide which certificate the
# handshake presents. Without a token `bao` asks its token
# helper, and that is a `sh` this unit's `path` does not carry —
# `-token-only` answers on stdout and skips the helper.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to the swarm secret store with this host's services-issuer certificate." >&2
cat "$err" >&2
exit 1
fi
export BAO_TOKEN
echo "requesting the swarm-services leaf from the store"
resp="$(mktemp "$d/swarm-services.json.XXXXXX")"
trap 'rm -f "$err" "$resp"' EXIT
# ONE call, and the response carries all three artefacts —
# certificate, private key and issuing CA. Splitting it into a
# local CSR plus `pki/sign` would buy a key that never crosses
# the wire, and cost the thing that makes this unit worth having:
# the store, not this host, decides what the certificate says.
if ! bao write -format=json \
${lib.escapeShellArg "${baoServicesPkiMount}/issue/${baoServicesPkiRole}"} \
common_name=${lib.escapeShellArg (builtins.head swarmServiceDomains)} \
alt_names=${lib.escapeShellArg (lib.concatStringsSep "," swarmServiceDomains)} \
> "$resp" 2>"$err"; then
echo "the store refused to issue the swarm-services certificate." >&2
cat "$err" >&2
exit 1
fi
# Three fields out of ONE issuance, which is why the response is
# captured and then split rather than asked for three times with
# `bao write -field=…`: each of those calls would mint a separate
# certificate, and the key would belong to a different one than
# the leaf.
#
# `jq -r` into a file the shell opened, never into a variable and
# never into an argv — the private key does not become visible in
# /proc on its way to disk.
#
# `// empty` on every one of them: `jq -r` renders a missing
# field as the four characters `null`, which is a non-empty file
# and would sail past the check below into a certificate slot.
umask 077
jq -r '.data.private_key // empty' < "$resp" > "$d/swarm-services-key.pem.new"
jq -r '.data.certificate // empty' < "$resp" > "$d/swarm-services-leaf.new"
jq -r '.data.issuing_ca // empty' < "$resp" > "$svcroot.new"
for f in "$d/swarm-services-key.pem.new" "$d/swarm-services-leaf.new" "$svcroot.new"; do
if [ ! -s "$f" ]; then
echo "the store's response was missing a field: $f is empty" >&2
exit 1
fi
done
# nginx serves this file verbatim. It carries the leaf ALONE and
# not leaf-plus-issuer, which is the one visible consequence of
# flattening the chain: the issuer here is a self-signed root, so
# appending it would ship an anchor a client either already holds
# — in which case it is redundant — or does not, in which case
# shipping it proves nothing. The old file appended the services
# sub-CA because that WAS an intermediate, and a client anchored
# on the swarm root could not build the middle of the chain by
# itself.
mv -f "$d/swarm-services-leaf.new" "$svcleaf"
mv -f "$d/swarm-services-key.pem.new" "$d/swarm-services-key.pem"
chmod 0644 "$svcleaf"
chmod 0600 "$d/swarm-services-key.pem"
# The anchor, world-readable, beside the leaf it signed. This is
# the file an operator adds to a browser's trust store, and the
# one `hive-tls-ca` folds into `trust-bundle.pem`. Public
# certificate material only — the private half never leaves the
# store's `pki` mount, which is what `root/generate/internal`
# buys.
rootchanged=0
if ! cmp -s "$svcroot.new" "$svcroot"; then
rootchanged=1
fi
mv -f "$svcroot.new" "$svcroot"
chmod 0644 "$svcroot"
# Propagation, for the RETRY path only. Ordered before both of
# these, so on a normal boot they have not run yet, `is-active`
# is false, and ordering alone does the work. What this covers is
# the store coming up hours after the gateway did: nginx serves a
# *copy* of the leaf, so re-issuing the source changes nothing
# until the copy is remade.
#
# ⚠️ `--no-block`, and it is not a preference. This unit declares
# `Before=` both of these, so a blocking `systemctl restart`
# enqueues a job that systemd will not start until this unit is
# active — and this unit is not active until its ExecStart
# returns, which is waiting on that job. A deadlock, held until
# the 24h retry window's `TimeoutStartSec` fires. Queueing the
# job and letting it run once we exit is the only ordering that
# terminates.
#
# The cost is real and is the reason `hive-tls-resign` does NOT
# do this: a propagation that fails after we return does not fail
# us, so the leaf can rotate on disk while nginx keeps the old
# copy. That unit blocks because nothing orders it before its
# consumers; this one cannot.
if systemctl is-active --quiet nginx.service; then
echo "swarm-services leaf rotated — re-importing and reloading nginx"
systemctl restart --no-block hive-gateway-self-signed-cert.service
systemctl reload --no-block nginx.service
fi
# The bundle is assembled by `hive-tls-ca`, which is ordered
# after this unit and therefore already holds the current root on
# any normal boot. On the retry path it ran a long time ago, so a
# CHANGED root has to reach it — and only a changed one, or every
# boot would bounce a unit with nothing to do.
if [ "$rootchanged" = 1 ] && systemctl is-active --quiet hive-tls-ca.service; then
echo "the services root changed — rebuilding the trust bundle"
systemctl restart --no-block hive-tls-ca.service
fi
'';
};
systemd.timers.hive-tls-resign = {
description = "Weekly gateway-leaf re-sign and propagation";
wantedBy = [ "timers.target" ];