vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

nix: issue the swarm-services leaf from bao's pki mount

Browse source 
The `pki` mount had no issuer and no principal could log in to it, so the
swarm's service certificates were still minted by two openssl hops from a
root key on disk. Close both halves and retire the openssl path with them.

The mount now generates its own root, once. The granting unit asks bao
whether an issuer already exists (`bao list pki/issuers`) before calling
`pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role
and the grant without touching the anchor — a root that changed per boot
would invalidate every certificate issued under it and every browser
taught to trust it. The guard asks the store rather than looking for a
marker file on this host's disk: a file is a claim about a mount that may
have been restored from a snapshot or disabled and re-enabled underneath
it.

`swarm-services-issuer` stops being an inert policy. A fourth cert-auth
role attaches it, following the shape the controller, the publisher and
matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its
CN — that credential is what opens the mount, so it cannot come out of it.

`swarm-services-cert.service` logs in with that leaf, calls
`pki/issue/swarm-services`, and writes the result to the path
hive-tls.nix already wrote and the gateway already copies from. The
sub-CA layer does not move; it stops existing. The role's
`allowed_domains`, read from the same `swarm.serviceDomains` the SANs
come from, enforces at issue time what the sub-CA encoded in x509
`nameConstraints`, and with the root inside the mount there is nothing
left for an intermediate to be an intermediate of.

Not a flag day: the issuing root is published beside the leaf as
`swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the
swarm root still sits. A leaf chaining to the old sub-CA and one issued
by the store both verify against the same bundle, so hives can be
rebuilt in any order. The same file is what an operator hands a browser
— readable without a store login, which matters because every listener
demands a client certificate.

The eval-time warning about uncovered service names is gone rather than
reworded. It fired on "this host does not hold the swarm root key", which
was the reason a hive could end up serving its own leaf on a
swarm-service name. Every hive now asks the store with its own identity,
so that stopped being the thing that decides.

Closes #4586
This commit is contained in:
atlas 2026-09-21 17:59:22 +02:00 • committed by mara
commit f4df4fc4a9
10 changed files with 700 additions and 315 deletions
Download patch file Download diff file Expand all files Collapse all files

1
nix/host-modules/default.nix
View file

@ -30,6 +30,7 @@
./glue-matrix-ctl-bao-identity.nix
./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix
./glue-services-issuer-bao-identity.nix
./glue-swarm-bao-otel-oidc-client.nix
./glue-swarm-otel-oidc-client.nix
./swarm-authelia.nix

15
nix/host-modules/glue-bao-tls.nix
View file

@ -208,6 +208,21 @@ in
[ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \
${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth
# The identity a hive presents to ask the store's `pki` mount for the
# swarm-services certificate its gateway serves. Minted here like the
# three above, and the reason is the sharpest of the four: this leaf
# is what OPENS the mint, so it cannot come out of it. A certificate
# authority that issues the credential admitting you to it is the
# cycle this file's header exists to keep out — which is exactly why
# the swarm-services root moved into the store and this one did not.
#
# Its own CN, not the reader's: the hive's own leaf reads every secret
# its policy names, while this principal may do one thing — `update`
# on `pki/issue/swarm-services` — and giving a renewal unit the wider
# credential would undo the separation the role was created for.
[ -s ${pkiDir}/services-issuer.pem ] || ${signLeaf} ${pkiDir} services-issuer \
${lib.escapeShellArg deployCfg.bao.servicesIssuerCommonName} "" clientAuth
'';
};
};

43
nix/host-modules/glue-services-issuer-bao-identity.nix Normal file
View file

@ -0,0 +1,43 @@
# Glue: point the swarm-services certificate unit at the bao leaf minted for
# it.
#
# ONE PAIRING PER FILE — the services-issuer principal ← bao, and nothing
# else. Deleting this leaves a hive that takes an operator-provided path to
# that credential, which is what any deployment not minting its own already
# does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, because the thing that owns a private key owns issuing from it. What
# belongs here is the pairing: which paths `swarm-services-cert` reads.
#
# ⚠️ Gated on the leaf existing, not on the store running here — the same rule
# ./glue-controller-bao-identity.nix states, and it matters more for this
# principal than for any of its siblings: every hive in a swarm serves the
# swarm's service names and therefore needs this certificate, while exactly
# one of them hosts the store.
#
# Everything is `mkDefault`. An operator naming their own paths wins.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path rather than repeating that file's directory literal: an operator who
# moves the PKI moves both, and the two cannot drift apart.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in
{
config = lib.mkIf (hyperhiveCfg.enable && haveMintedPki) {
services.hyperhive.deploy.hive-controller.tls = {
baoClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
baoClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem";
};
};
}

512
nix/host-modules/hive-tls.nix
View file

@ -10,26 +10,22 @@ let
gatewayCfg = config.services.hyperhive.gateway;
swarmCaCfg = config.services.hyperhive.swarm.ca;
domain = hyperhiveCfg.domain;
# Derived once in ./swarm.nix and read here + in ./swarm-ca.nix, so
# the names this leaf carries as SANs and the names the sub-CA is
# constrained to cannot disagree.
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
# The endpoint the services leaf comes out of. Read off the store's own
# options rather than spelled here: ./swarm-bao.nix enables the mount,
# generates its root and writes the grant that names this exact path, and a
# second module composing its own is how the grant and the caller stop
# agreeing.
baoServicesPkiMount = baoDeploy.servicesPkiMountPath;
baoServicesPkiRole = baoDeploy.servicesPkiRoleName;
# Derived once in ./swarm.nix and read here + in ./swarm-bao.nix, so
# the names this leaf carries as SANs and the names the store's
# `pki/roles/swarm-services` is narrowed to cannot disagree.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# True for the names `signHiveLeaf` below actually covers: the hive
# domain itself, or ONE label under it. `*.<domain>` is a single-label
# wildcard — `a.b.<domain>` does not match it — so the depth check is
# the whole point rather than a nicety.
coveredByHiveLeaf =
n:
n == domain
|| (lib.hasSuffix ".${domain}" n && !lib.hasInfix "." (lib.removeSuffix ".${domain}" n));
# Service names this host can serve a *matching* certificate for, and
# the ones it can't. A name is issuable here when the hive leaf covers
# it, or when this host issues the swarm-services leaf — which needs
# the swarm root's private key, i.e. `swarm.ca.autoConfigure`.
uncoveredServiceDomains = lib.filter (n: !coveredByHiveLeaf n) swarmServiceDomains;
# The host-managed hive CA is the trust anchor for self-signed mode.
# It is only stood up when the gateway actually serves a self-signed
# cert: the gateway must be in self-signed mode. `domain` is required
@ -182,57 +178,38 @@ let
"$d/ca.pem" "$d/ca-key.pem"
'';
# The swarm-services leaf: signed by the services sub-CA, covering the
# swarm's service names. Those are *siblings* of the hive domain, not
# children, so the hive CA is name-constrained out of them and cannot
# sign this however its SAN list is written.
#
# Skipped when the sub-CA isn't on disk: it only exists where the
# swarm CA is autoconfigured, and a hive that gets its certs from its
# operator has nothing for this to do.
signServicesLeaf = lib.optionalString (swarmServiceDomains != [ ]) ''
servicesCa=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"}
servicesCaKey=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca-key.pem"}
if [ -s "$servicesCa" ] && [ -s "$servicesCaKey" ]; then
${signLeafScript} "$d" swarm-services \
${lib.escapeShellArg (builtins.head swarmServiceDomains)} \
${lib.escapeShellArg (lib.concatMapStringsSep "," (n: "DNS:${n}") swarmServiceDomains)} \
"$servicesCa" "$servicesCaKey"
else
echo "no swarm-services sub-CA at $servicesCa — skipping the services leaf"
fi
'';
# Shared by BOTH units that decide whether to re-sign. It lives here
# rather than in one of them because the two guards have to agree: they
# answer the same question at different times (`hive-tls-ca` at service
# activation, i.e. on the deploy; `hive-tls-resign` from a weekly timer
# for a host that stays up long enough to drift). A rule implemented in
# one and not the other is worse than one implemented in neither — it
# looks fixed and only fires on whichever path you did not take, which
# is exactly how a corrected `serviceDomains` still served a stale leaf.
# Shared by every unit that decides whether to re-issue a leaf. It
# lives here rather than in one of them because the guards have to
# agree: they answer the same question at different times
# (`hive-tls-ca` at service activation, i.e. on the deploy;
# `hive-tls-resign` from a weekly timer for a host that stays up long
# enough to drift; `swarm-services-cert` for the leaf the store
# issues). A rule implemented in one and not the others is worse than
# one implemented in none — it looks fixed and only fires on whichever
# path you did not take, which is exactly how a corrected
# `serviceDomains` still served a stale leaf.
#
# Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and
# `covers`.
leafCoverage = ''
svcleaf="$d/swarm-services.pem"
svcroot="$d/swarm-services-root.pem"
hiveNames=${lib.escapeShellArg "${domain} *.${domain}"}
svcNames=${lib.escapeShellArg (lib.concatStringsSep " " swarmServiceDomains)}
# The services leaf is only expected where the sub-CA exists;
# elsewhere its absence is the correct state, not a stale leaf.
# A hive with no configured service names has no services leaf to
# hold, and its absence there is the correct state rather than a
# stale one. Anywhere else it is expected: the leaf comes from the
# secret store's `pki` mount now, not from a sub-CA that exists on
# one host in the swarm, so "this host cannot issue it" is no longer
# one of the answers.
want_svc=${if swarmServiceDomains == [ ] then "0" else "1"}
if [ ! -s ${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} ]; then
want_svc=0
fi
# Expiry is not the only way a leaf goes wrong. One signed when the
# configured name set was smaller stays valid for its whole lifetime
# while omitting every name added since — so a config change can
# evaluate, build and deploy cleanly while the gateway keeps serving a
# certificate that does not cover the new service. The services sub-CA
# already reconciles this way (its `.names` comparison in
# ../swarm-ca.nix); the leaves did not.
# certificate that does not cover the new service.
#
# Reads the names out of the CERTIFICATE, not a sidecar file: the pem
# is what nginx serves, and a bookkeeping file drifts from it the
@ -254,13 +231,20 @@ let
[ "$_missing" = 0 ]
}
# True when every leaf this host is supposed to hold is present and
# carries its configured names. Expiry is the callers' own business —
# they use different windows.
# True when the leaf this host SIGNS is present and carries its
# configured names. Expiry is the callers' own business — they use
# different windows.
#
# ⚠️ The services leaf is deliberately NOT part of this any more, and
# the omission is load-bearing rather than a simplification. Its two
# callers sign under the hive CA's key; the services leaf comes out
# of the secret store, which neither of them can reach. A guard that
# went on reporting it stale would make `hive-tls-ca` re-sign the
# hive leaf forever over a file it has no way to fix. The
# `swarm-services-cert` unit below holds the same rule for that leaf,
# using the `covers` helper above so the two cannot drift.
leavesCoverNames() {
covers "$leaf" "$hiveNames" || return 1
[ "$want_svc" = 0 ] && return 0
covers "$svcleaf" "$svcNames"
covers "$leaf" "$hiveNames"
}
'';
in
@ -313,6 +297,44 @@ in
'';
};
baoClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer.pem";
description = ''
Certificate `swarm-services-cert` presents to the secret store when it
asks for the swarm-services leaf the gateway serves.
Its subject has to be
{option}`services.hyperhive.deploy.bao.servicesIssuerCommonName` — cert
auth matches on the common name, and this is the one role whose policy
permits `pki/issue/swarm-services`.
A **separate** credential from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is this
hive's own reader identity and whose policy reads every secret the
hive is entitled to. A unit that renews one server certificate has no
business holding that.
On a hive that mints its own store PKI a glue module supplies the leaf
as a `mkDefault`; everywhere else it is the credential an operator
places, for the reason the store's own client certificate is.
A path, never a value.
'';
};
baoClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.hive-controller.tls.baoClientCertFile`.
Both or neither — a certificate with no key authenticates nothing.
'';
};
leafValidityDays = lib.mkOption {
type = lib.types.int;
default = 30;
@ -333,40 +355,17 @@ in
};
config = lib.mkIf active {
# A swarm service name that is a sibling of the hive domain rather
# than a child needs the swarm-services leaf, and this host only
# signs that one when it holds the swarm root key. Without it nginx
# falls back to the hive leaf on those names and every client sees a
# name mismatch — on a config that evaluates and deploys cleanly.
#
# ⚠️ A WARNING, NOT AN ASSERTION, and the distinction is the point:
# this module can see what *it* can issue; it cannot see an
# operator-installed services sub-CA, an external ACME setup, or a
# cert delivered by any other means. A rebuild must not be blocked
# by a conclusion this host is not in a position to reach. Report
# the observation and let the operator judge it.
#
# (`certDir` / ACME modes don't reach here at all: `active` is
# self-signed-only, and there the operator's cert decides.)
warnings = lib.optional (uncoveredServiceDomains != [ ] && !swarmCaCfg.autoConfigure) ''
services.hyperhive: these swarm service names are outside this
hive's domain (${domain}), so the hive CA's leaf does not cover
them:
${lib.concatMapStringsSep "\n" (n: " ${n}") uncoveredServiceDomains}
services.hyperhive.swarm.ca.autoConfigure is false, so this host
does not sign the swarm-services leaf either, and the gateway will
serve the hive leaf on those names — a certificate-name mismatch
for browsers and for agents' git-over-https.
If you have already arranged certificates for them — an
operator-installed services sub-CA in ${swarmCaCfg.stateDir}, or
services.hyperhive.gateway.tls.{certDir,acme} — this is expected
and you can ignore it. Otherwise pin the names back under
${domain} (services.hyperhive.swarm.{forge.domain,
matrix.gatewayHost, authelia.domain}) or install the sub-CA.
'';
# 🚫 The "these service names are outside this hive's domain and this
# host cannot sign for them" warning is GONE, and its absence is the
# deliverable rather than a tidy-up. It fired on
# `!swarm.ca.autoConfigure` — i.e. "this host does not hold the swarm
# root key" — which was the whole reason a hive could end up serving
# its own leaf on a swarm-service name. The services leaf now comes
# from the secret store's `pki` mount, which every hive reaches over
# the network with its own identity, so holding the root key stopped
# being the thing that decides. Left in place it would warn, on every
# rebuild of every hive that is not the CA host, about a fallback
# that no longer happens.
# Generate (and rotate) the hive CA + gateway leaf before anything
# serves it. Idempotent: the CA is created once and reused; the leaf
@ -381,26 +380,22 @@ in
# against the unit that reads the file.
before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The issuance below needs the swarm root key on disk, and (for the
# services leaf) the services sub-CA it signs under. When this host
# generates them (single-host swarm) both units must have run first;
# when the operator provides the material there is no unit to wait
# for, so the dependency is conditional rather than a unit that
# exists and does nothing.
# The issuance below needs the swarm root key on disk. When this
# host generates it (single-host swarm) that unit must have run
# first; when the operator provides the material there is no unit
# to wait for, so the dependency is conditional rather than a unit
# that exists and does nothing.
#
# Without waiting for swarm-services-ca specifically, this unit races
# it: if hive-tls-ca finishes first, it finds no services-ca.pem yet,
# silently skips signing the services leaf (the same as "operator
# hasn't set one up"), and the gateway comes up with a vhost pointed
# at a cert that was never written.
after = lib.optionals swarmCaCfg.autoConfigure [
"swarm-ca.service"
"swarm-services-ca.service"
];
requires = lib.optionals swarmCaCfg.autoConfigure [
"swarm-ca.service"
"swarm-services-ca.service"
];
# `swarm-services-cert` is ordered before this one rather than the
# other way round, and not because this unit needs the leaf — it
# never touches it. The trust bundle written at the end of this
# script anchors on every root a consumer must hold, and the
# services root is one of them, so the file has to be on disk
# before the `cat`. Get it wrong and the bundle is a boot behind:
# correct-looking, and missing the anchor for exactly the names the
# gateway serves with it.
after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
requires = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
@ -496,23 +491,14 @@ in
# up to a week late and does nothing for the rebuild that changed
# the names in the first place.
#
# The name check has to include the SERVICES leaf even though the
# condition is written around the hive one, because both are signed
# in this block: a fresh `gateway.pem` otherwise suppresses the
# re-sign of a `swarm-services.pem` that is missing or stale, which
# is what left a corrected `serviceDomains` still mis-served.
resign=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1
openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1
leavesCoverNames || resign=1
if [ "$want_svc" = 1 ] && [ ! -s "$svcleaf" ]; then
resign=1
fi
if [ "$resign" = 1 ]; then
echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)"
${signHiveLeaf}
${signServicesLeaf}
fi
# --- Trust bundle: what a consumer must TRUST, as opposed to
@ -534,10 +520,19 @@ in
# hive where the optional file is legitimately absent. The
# reflexive `|| true` is worse; it also swallows a real failure to
# read the hive CA.
#
# `swarm-services-root.pem` joins the set for the reason the swarm
# root is in it: it is an ANCHOR a consumer terminates at, not an
# intermediate. It is also what keeps this change from being a
# flag day — the swarm root stays in the bundle beside it, so a
# leaf still chaining to the old services sub-CA and one issued by
# the store both verify against the same file, and a hive can be
# rebuilt before or after its peers.
bundle="$d/trust-bundle.pem"
anchors=("$ca")
if [ -s "$prev" ]; then anchors+=("$prev"); fi
if [ -s "$root" ]; then anchors+=("$root"); fi
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
cat "''${anchors[@]}" > "$bundle"
chmod 0644 "$bundle"
'';
@ -582,12 +577,17 @@ in
d=${lib.escapeShellArg cfg.stateDir}
leaf="$d/gateway.pem"
# ⚠️ EVERY leaf this host issues must be covered here. A leaf that
# ⚠️ EVERY leaf this host SIGNS must be covered here. A leaf that
# first-boot issuance creates and this unit does not know about
# looks perfect for its entire validity and then expires with no
# warning — the failure is invisible until it is total. `svcleaf`
# and the name checks come from the shared snippet, so this unit
# and `hive-tls-ca` cannot disagree about what a good leaf is.
# warning — the failure is invisible until it is total. The name
# checks come from the shared snippet, so this unit and
# `hive-tls-ca` cannot disagree about what a good leaf is.
#
# The services leaf is not one of them any longer: it is issued
# by the secret store, whose key this unit does not hold and
# cannot re-sign under. Renewing it is `swarm-services-cert`'s
# job, at boot, which is a cadence its own issue owns.
${leafCoverage}
# Re-sign only when a leaf is within half its validity of expiry.
@ -598,19 +598,17 @@ in
[ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1
}
if fresh "$leaf" && { [ "$want_svc" = 0 ] || fresh "$svcleaf"; } \
&& leavesCoverNames; then
if fresh "$leaf" && leavesCoverNames; then
echo "leaves valid, and covering the configured names — no resign needed"
exit 0
fi
echo "a leaf is missing, near expiry, or missing a configured name — re-signing"
before="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
before="$(sha256sum "$leaf" 2>/dev/null || true)"
${signHiveLeaf}
${signServicesLeaf}
after="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
after="$(sha256sum "$leaf" 2>/dev/null || true)"
if [ "$before" != "$after" ]; then
echo "gateway leaf rotated — re-importing and reloading nginx"
systemctl restart hive-gateway-self-signed-cert.service
@ -621,6 +619,256 @@ in
'';
};
# The swarm-services leaf, issued by the secret store's `pki` mount.
#
# ⚠️ This unit replaces a chain, not a command. What used to happen
# was two openssl hops on whichever host held the swarm root key: a
# name-constrained `swarm-services` sub-CA (../swarm-ca.nix), then a
# leaf signed under it here. Both are gone. The store's
# `pki/roles/swarm-services` carries the same narrowing as an
# `allowed_domains` list enforced at issue time, so the intermediate
# that used to encode it in x509 `nameConstraints` has nothing left
# to express — and with the root inside the mount there is nothing
# for it to be an intermediate OF.
#
# The names it may carry are not this unit's to assert: the role
# refuses anything outside `allowed_domains`, which is read from the
# same `swarm.serviceDomains` the SANs below are built from. A
# mismatch is a refusal from the store naming the offending name,
# not a certificate quietly issued for something nobody configured.
systemd.services.swarm-services-cert = {
description = "Issue the swarm-services TLS leaf from the secret store's PKI";
wantedBy = [ "multi-user.target" ];
# ⚠️ The ordering the old shape got by ACCIDENT, written down.
# Nothing used to declare "no TLS until the store is up" — it held
# because `swarm-services-ca.service` happened to block the gateway.
# Now it is a real edge: the gateway's certificate comes out of the
# store, so the store being up is a precondition rather than a
# coincidence, and `requiredBy` is what makes a store that never
# comes up an outage that says so instead of a gateway serving the
# wrong name.
before = [
"hive-tls-ca.service"
"hive-gateway-self-signed-cert.service"
];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The store's container, where it runs here. On a hive that reads a
# store hosted elsewhere no such unit exists and systemd ignores
# the name, which is the correct behaviour rather than a gap: what
# this unit actually needs is the store reachable, and on a remote
# store nothing local can order against that.
after = [
"container@${baoCfg.machine}.service"
"swarm-bao-services-issuer-policy.service"
];
wants = [ "container@${baoCfg.machine}.service" ];
path = [
baoDeploy.package
pkgs.jq
pkgs.openssl
pkgs.coreutils
pkgs.systemd
];
# Sized like the store's own granting units, and for the same
# reason: under `seal = "shamir"` an operator unseals BY HAND, and
# the login below fails for as long as that takes. 2880 × 30s is
# 24h inside a 25h window — `StartLimit*` are `[Unit]` settings, so
# the window must exceed `RestartSec × burst` or it closes between
# attempts and the burst is never reached.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
UMask = "0077";
SyslogIdentifier = "swarm-services-cert";
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs (cfg.baoClientCertFile != null) {
BAO_CLIENT_CERT = cfg.baoClientCertFile;
}
// lib.optionalAttrs (cfg.baoClientKeyFile != null) {
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
}
# Absent means the system trust store, which is what a deployment
# with a real CA wants and what a self-signed one must not be left
# with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
install -d -m 0755 "$d"
${leafCoverage}
if [ "$want_svc" = 0 ]; then
echo "no swarm service domains configured — nothing to issue"
exit 0
fi
# Same rule as the hive leaf's, at a different cadence: re-issue
# when the file is missing, within 30 days of expiry, or no
# longer carrying every configured name. The name check is what
# makes adding a swarm service take effect on the rebuild that
# added it rather than whenever the certificate happens to lapse.
reissue=0
{ [ -s "$svcleaf" ] && [ -s "$d/swarm-services-key.pem" ]; } || reissue=1
openssl x509 -in "$svcleaf" -noout -checkend 2592000 >/dev/null 2>&1 || reissue=1
covers "$svcleaf" "$svcNames" || reissue=1
[ -s "$svcroot" ] || reissue=1
if [ "$reissue" = 0 ]; then
echo "swarm-services leaf valid and covering the configured names — leaving it alone"
exit 0
fi
${
if cfg.baoClientCertFile == null || cfg.baoClientKeyFile == null then
''
echo "no bao client certificate configured for this host, so the gateway's" >&2
echo "swarm-services certificate cannot be requested from the store." >&2
echo "Set services.hyperhive.deploy.hive-controller.tls.baoClient{Cert,Key}File" >&2
echo "to a leaf the store's CA signed with CN=${baoDeploy.servicesIssuerCommonName}." >&2
exit 1''
else
""
}
# `bao`'s own message is the only thing separating an unreachable
# store from a refused identity from a role that would not issue
# these names, and this unit retries on all three — so it reports
# which one rather than asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting: the
# `BAO_CLIENT_*` variables only decide which certificate the
# handshake presents. Without a token `bao` asks its token
# helper, and that is a `sh` this unit's `path` does not carry —
# `-token-only` answers on stdout and skips the helper.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to the swarm secret store with this host's services-issuer certificate." >&2
cat "$err" >&2
exit 1
fi
export BAO_TOKEN
echo "requesting the swarm-services leaf from the store"
resp="$(mktemp "$d/swarm-services.json.XXXXXX")"
trap 'rm -f "$err" "$resp"' EXIT
# ONE call, and the response carries all three artefacts —
# certificate, private key and issuing CA. Splitting it into a
# local CSR plus `pki/sign` would buy a key that never crosses
# the wire, and cost the thing that makes this unit worth having:
# the store, not this host, decides what the certificate says.
if ! bao write -format=json \
${lib.escapeShellArg "${baoServicesPkiMount}/issue/${baoServicesPkiRole}"} \
common_name=${lib.escapeShellArg (builtins.head swarmServiceDomains)} \
alt_names=${lib.escapeShellArg (lib.concatStringsSep "," swarmServiceDomains)} \
> "$resp" 2>"$err"; then
echo "the store refused to issue the swarm-services certificate." >&2
cat "$err" >&2
exit 1
fi
# Three fields out of ONE issuance, which is why the response is
# captured and then split rather than asked for three times with
# `bao write -field=…`: each of those calls would mint a separate
# certificate, and the key would belong to a different one than
# the leaf.
#
# `jq -r` into a file the shell opened, never into a variable and
# never into an argv — the private key does not become visible in
# /proc on its way to disk.
#
# `// empty` on every one of them: `jq -r` renders a missing
# field as the four characters `null`, which is a non-empty file
# and would sail past the check below into a certificate slot.
umask 077
jq -r '.data.private_key // empty' < "$resp" > "$d/swarm-services-key.pem.new"
jq -r '.data.certificate // empty' < "$resp" > "$d/swarm-services-leaf.new"
jq -r '.data.issuing_ca // empty' < "$resp" > "$svcroot.new"
for f in "$d/swarm-services-key.pem.new" "$d/swarm-services-leaf.new" "$svcroot.new"; do
if [ ! -s "$f" ]; then
echo "the store's response was missing a field: $f is empty" >&2
exit 1
fi
done
# nginx serves this file verbatim. It carries the leaf ALONE and
# not leaf-plus-issuer, which is the one visible consequence of
# flattening the chain: the issuer here is a self-signed root, so
# appending it would ship an anchor a client either already holds
# — in which case it is redundant — or does not, in which case
# shipping it proves nothing. The old file appended the services
# sub-CA because that WAS an intermediate, and a client anchored
# on the swarm root could not build the middle of the chain by
# itself.
mv -f "$d/swarm-services-leaf.new" "$svcleaf"
mv -f "$d/swarm-services-key.pem.new" "$d/swarm-services-key.pem"
chmod 0644 "$svcleaf"
chmod 0600 "$d/swarm-services-key.pem"
# The anchor, world-readable, beside the leaf it signed. This is
# the file an operator adds to a browser's trust store, and the
# one `hive-tls-ca` folds into `trust-bundle.pem`. Public
# certificate material only — the private half never leaves the
# store's `pki` mount, which is what `root/generate/internal`
# buys.
rootchanged=0
if ! cmp -s "$svcroot.new" "$svcroot"; then
rootchanged=1
fi
mv -f "$svcroot.new" "$svcroot"
chmod 0644 "$svcroot"
# Propagation, for the RETRY path only. Ordered before both of
# these, so on a normal boot they have not run yet, `is-active`
# is false, and ordering alone does the work. What this covers is
# the store coming up hours after the gateway did: nginx serves a
# *copy* of the leaf, so re-issuing the source changes nothing
# until the copy is remade.
#
# ⚠️ `--no-block`, and it is not a preference. This unit declares
# `Before=` both of these, so a blocking `systemctl restart`
# enqueues a job that systemd will not start until this unit is
# active — and this unit is not active until its ExecStart
# returns, which is waiting on that job. A deadlock, held until
# the 24h retry window's `TimeoutStartSec` fires. Queueing the
# job and letting it run once we exit is the only ordering that
# terminates.
#
# The cost is real and is the reason `hive-tls-resign` does NOT
# do this: a propagation that fails after we return does not fail
# us, so the leaf can rotate on disk while nginx keeps the old
# copy. That unit blocks because nothing orders it before its
# consumers; this one cannot.
if systemctl is-active --quiet nginx.service; then
echo "swarm-services leaf rotated — re-importing and reloading nginx"
systemctl restart --no-block hive-gateway-self-signed-cert.service
systemctl reload --no-block nginx.service
fi
# The bundle is assembled by `hive-tls-ca`, which is ordered
# after this unit and therefore already holds the current root on
# any normal boot. On the retry path it ran a long time ago, so a
# CHANGED root has to reach it — and only a changed one, or every
# boot would bounce a unit with nothing to do.
if [ "$rootchanged" = 1 ] && systemctl is-active --quiet hive-tls-ca.service; then
echo "the services root changed — rebuilding the trust bundle"
systemctl restart --no-block hive-tls-ca.service
fi
'';
};
systemd.timers.hive-tls-resign = {
description = "Weekly gateway-leaf re-sign and propagation";
wantedBy = [ "timers.target" ];

243
nix/host-modules/swarm-bao.nix
View file

@ -300,15 +300,48 @@ let
# self-signed CA under `/var/lib/swarm-bao-pki`, and it stays outside the
# store permanently: bao cannot issue the credential that opens bao.
#
# Nothing issues through this mount yet — see the role below.
servicesPkiMountPath = "pki";
# The mount's own root is generated into it by the bootstrap unit below and
# its private key never leaves — `root/generate/internal` keeps it inside
# the store, which is the whole reason the mount exists rather than another
# key beside ./swarm-ca.nix's.
#
# An option rather than a literal because ./hive-tls.nix issues through it
# and has to spell it the same way.
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
# Subject of the root generated into that mount. A label for a human reading
# a chain, not an identity anything authenticates against — same fall-through
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
# neither a swarm name nor a domain must still evaluate.
servicesPkiRootLabel =
if hyperhiveCfg.swarm.name != null then
hyperhiveCfg.swarm.name
else if hyperhiveCfg.domain != null then
hyperhiveCfg.domain
else
"hyperhive";
# Where the store's host publishes that root's PUBLIC certificate. Its own
# directory rather than ./glue-bao-tls.nix's `/var/lib/swarm-bao-pki`: that
# one holds the CA key that opens the store, is 0700 for exactly that
# reason, and this file is the opposite kind of thing — the one piece of
# certificate material meant to be copied out and handed around.
servicesPkiRootDir = "/var/lib/swarm-bao-services-pki";
servicesPkiRootFile = "${servicesPkiRootDir}/services-root.pem";
# ~30y, matching ./swarm-ca.nix's root rather than its services sub-CA. This
# is an ANCHOR: an operator adds it to a browser's trust store by hand, and
# every rotation is a manual re-trust on every machine that holds it. The
# sub-CA's short window bought cheap rotation because the root above it
# absorbed the churn; nothing sits above this one.
servicesPkiRootTtl = "262800h";
# The role every swarm-service certificate will be issued through, and the
# whole narrowing of this mount. A role rather than a name-constrained
# sub-CA: the engine enforces the same "these names and no others" at issue
# time that ./swarm-ca.nix builds into x509 `nameConstraints` today, so a
# second CA in the chain would re-implement a check that already happens.
servicesPkiRoleName = "swarm-services";
servicesPkiRoleName = baoDeploy.servicesPkiRoleName;
# The names that role may issue for. Read out of the swarm-tier option
# rather than assembled here, exactly as ./swarm-ca.nix:45 and
@ -317,23 +350,22 @@ let
# composing its own list is how they stop agreeing.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# The identity that will ask this mount for a certificate, once something
# does. A THIRD principal rather than either existing one, for the reason
# `secretPublisherCommonName` gives for being a second: the controller may
# rewrite every hive's policy and login role, and a unit whose whole job is
# renewing one server certificate has no business holding that.
# The identity that asks this mount for a certificate. A FOURTH principal
# rather than any existing one, for the reason `secretPublisherCommonName`
# gives for being a second: the controller may rewrite every hive's policy
# and login role, and a unit whose whole job is renewing one server
# certificate has no business holding that.
#
# `update` and not `create`: bao's issue endpoint is a POST to an existing
# path, which is `update` in ACL terms — `create` here would grant nothing
# and read as though it did.
#
# ⚠️ NOTHING IS ATTACHED TO THIS POLICY. A bao policy grants only through a
# token that carries it, so until a login role names it this is an inert
# declaration of intent — which is the point: the cert-auth role that
# attaches it needs a leaf carrying its CN, and minting that is
# ./glue-bao-tls.nix's job, one step further along than this one.
servicesIssuerPolicyName = "swarm-services-issuer";
# The subject the login role accepts, and the CN ./glue-bao-tls.nix signs
# the matching leaf with. An option rather than a literal for the reason
# `controllerCn` gives: two files have to spell it identically.
servicesIssuerCn = baoDeploy.servicesIssuerCommonName;
servicesIssuerPolicyText = ''
path "${servicesPkiMountPath}/issue/${servicesPkiRoleName}" {
capabilities = ["update"]
@ -971,6 +1003,65 @@ in
'';
};
servicesPkiMountPath = lib.mkOption {
type = lib.types.str;
default = "pki";
description = ''
Mount path of the PKI engine the swarm's **service** certificates are
issued from, and whose root is the anchor those certificates chain to.
An option rather than a literal because two files have to spell it
identically: this module enables the mount, generates its root and
writes the grant that points at `<path>/issue/`, while ./hive-tls.nix
calls that endpoint. A mount path only one of them knows is a grant
naming a location nobody issues from.
⚠️ NOT the store's own client-auth PKI. That one is
./glue-bao-tls.nix's self-signed CA on disk, and it stays outside the
store permanently — bao cannot issue the credential that opens bao.
'';
};
servicesPkiRoleName = lib.mkOption {
type = lib.types.str;
default = "swarm-services";
description = ''
Role on {option}`services.hyperhive.deploy.bao.servicesPkiMountPath`
that the swarm-services certificate is issued through, and the whole
narrowing of that mount.
A role rather than a name-constrained sub-CA: the engine enforces the
same "these names and no others" at issue time, from an
`allowed_domains` read off
{option}`services.hyperhive.swarm.serviceDomains`, that an
intermediate would encode in x509 `nameConstraints` — so a second
authority in the chain would re-implement a check that has already
happened.
'';
};
servicesIssuerCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-services-issuer";
example = "swarm-services-issuer.svc";
description = ''
Subject the store's services-issuer cert-auth role accepts — the
identity a hive presents when it asks the `pki` mount for the
swarm-services certificate its gateway serves.
A **fourth** identity rather than reuse of any sibling above, and the
reasoning is theirs: this principal's whole grant is one `update` on
`pki/issue/swarm-services`, while the controller's includes rewriting
every hive's policy and login role. Handing a certificate-renewal unit
the controller's leaf would give that separation away in one line.
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
feeds this value into the guard on
{option}`services.hyperhive.swarm.hives`, so a hive named after it
fails evaluation rather than silently receiving this grant.
'';
};
matrixCtlHiveName = lib.mkOption {
type = lib.types.str;
default = toString hyperhiveCfg.hiveName;
@ -1407,6 +1498,7 @@ in
"swarm-bao-certs"
"swarm-bao-token"
"swarm-bao-forwarder-oidc"
"swarm-bao-services-issuer-policy"
];
# 🚫 No `swarm.otel.scrapeTargets.bao` entry any more, and its absence is
@ -1748,21 +1840,69 @@ in
*) bao secrets enable -path=${credentialMountPath} kv-v2 ;;
esac
# The PKI engine the swarm's service certificates are to come from.
# Asked rather than attempted for the same reason as the mount above:
# The PKI engine the swarm's service certificates come from. Asked
# rather than attempted for the same reason as the mount above:
# `secrets enable` errors on a path already in use, so a second
# rebuild would fail a unit that has nothing left to do.
#
# ⚠️ Deliberately EMPTY: nothing here generates an issuer into it, so
# the role below can be written but cannot yet issue. That is the
# whole shape of this step — the mount, its narrowing and its grant
# exist, and the binary that uses them arrives separately. Undoing it
# is `bao secrets disable ${servicesPkiMountPath}`.
case "$mounts" in
*'"${servicesPkiMountPath}/"'*) ;;
*) bao secrets enable -path=${servicesPkiMountPath} pki ;;
esac
# ⚠️ THE ROOT, AND THE ONE THING THIS UNIT MUST NEVER DO TWICE.
#
# `root/generate/internal` mints a new self-signed CA every time it
# is called. Calling it unconditionally would hand the swarm a
# different anchor on every boot: every certificate issued under the
# previous one stops verifying, and every browser an operator taught
# to trust the old root has to be re-taught. So the generation is
# guarded, and the guard asks BAO rather than looking for a marker
# file — a file on this host's disk is a claim about a store that
# may live in a container, may have been restored from a snapshot,
# and may have had the mount disabled and re-enabled underneath it.
# The mount's own issuer list is the only answer that cannot be
# stale.
#
# `bao list ${servicesPkiMountPath}/issuers` is a 404 (non-zero) on a
# mount with no issuer and a key list once one exists, so its exit
# status IS the question — no output parsing, no error string to
# recognise. The default issuer is what `${servicesPkiMountPath}/issue/`
# signs with, and there is exactly one.
if ! bao list ${lib.escapeShellArg "${servicesPkiMountPath}/issuers"} >/dev/null 2>&1; then
echo "generating the swarm services root into the ${servicesPkiMountPath} mount"
# `-field=issuing_ca` discards the rest of the response. The
# private key is not in it under `internal` and cannot be — that
# is the point of the endpoint — and the certificate is read back
# below from a path that works on every subsequent boot too.
bao write -field=issuing_ca ${lib.escapeShellArg "${servicesPkiMountPath}/root/generate/internal"} \
common_name=${lib.escapeShellArg "swarm-services-ca ${servicesPkiRootLabel}"} \
issuer_name=${lib.escapeShellArg servicesPkiRoleName} \
ttl=${servicesPkiRootTtl} \
key_type=rsa \
key_bits=4096 >/dev/null
else
echo "the ${servicesPkiMountPath} mount already has an issuer — leaving it alone"
fi
# The anchor, as a world-readable file on the store's own host.
#
# Written on EVERY run and not only after a generation, so a store
# whose root predates this unit still publishes it. It is a public
# certificate — the material a browser has to be told to trust —
# and the private half stays in the mount, so 0644 is the correct
# mode rather than a concession.
#
# Why a file at all when `bao read -field=certificate
# ${servicesPkiMountPath}/cert/ca` returns the same bytes: every
# listener sets `tls_require_and_verify_client_cert`, so reading it
# from the store costs a client certificate. An operator adding the
# root to a browser is not necessarily holding one.
install -d -m 0755 ${lib.escapeShellArg servicesPkiRootDir}
bao read -field=certificate ${lib.escapeShellArg "${servicesPkiMountPath}/cert/ca"} \
> ${lib.escapeShellArg "${servicesPkiRootFile}.new"}
chmod 0644 ${lib.escapeShellArg "${servicesPkiRootFile}.new"}
mv -f ${lib.escapeShellArg "${servicesPkiRootFile}.new"} ${lib.escapeShellArg servicesPkiRootFile}
# `bao write` is an upsert, so this re-asserts the role on every
# rebuild rather than failing on one that exists — same shape as the
# cert-auth roles below, and the reason the mount needs the `case`
@ -1799,14 +1939,6 @@ in
client_flag=false \
key_type=rsa \
key_bits=4096
# Idempotent for the same reason the controller's policy above is: a
# rebuild re-asserts it. Written here rather than in a unit of its
# own because it attaches to nothing yet — the sibling-unit rule
# below is about a second *principal* that logs in, and this policy
# has no login role until the leaf carrying its CN exists.
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
@ -1958,6 +2090,57 @@ in
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
# A FOURTH sibling, same shape and same reasons as the two above. This
# one is what turns `swarm-services-issuer` from a declaration into a
# grant: a bao policy reaches nothing until a login role hands it to a
# token, so the `bao write auth/cert/certs/…` below is the whole
# difference between a mount nobody may issue from and one the gateway's
# certificate comes out of.
#
# The policy text moved here from the controller's unit, where it sat
# while it attached to nothing — a policy and the role that carries it
# belong in one place, and now there is a principal to put them with.
systemd.services.swarm-bao-services-issuer-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm services issuer's bao policy and cert-auth role";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
# Same unseal wait as its three siblings above, for the reason stated
# there: under `seal = "shamir"` a human unseals by hand.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
bao write auth/cert/certs/${lib.escapeShellArg servicesIssuerPolicyName} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg servicesIssuerCn} \
token_policies=${lib.escapeShellArg servicesIssuerPolicyName} \
display_name=${lib.escapeShellArg servicesIssuerCn}
'';
};
# The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet.

140
nix/host-modules/swarm-ca.nix
View file

@ -14,6 +14,13 @@
# for the obvious reason; the cert as a consequence, and that one costs
# something real — nothing whose trust store is built at build time can
# name it: `docs/swarm/ca.md::Distributing the root`.
#
# ⚠️ This root no longer issues the swarm's SERVICE certificates, and the
# `swarm-services-ca` sub-CA that used to sit under it is gone rather than
# moved. Those come out of the secret store's own `pki` mount now, anchored
# on a root generated inside it (./swarm-bao.nix), and ./hive-tls.nix asks
# for them over the network instead of signing them here. What is left here
# is one job: the anchor each hive's own CA is issued under.
{
lib,
config,
@ -36,13 +43,6 @@ let
hyperhiveCfg.domain
else
"hyperhive";
# Derived once in ./swarm.nix, read here and by ./hive-tls.nix: the
# CA that name-constrains these and the leaf that carries them as SANs
# must agree exactly, and two modules each assembling the list is how
# they stop agreeing. It is also this unit's *rotation trigger* below,
# which is why the ordering is stable there rather than here.
serviceDomains = hyperhiveCfg.swarm.serviceDomains;
in
{
options.services.hyperhive.swarm.ca = {
@ -90,23 +90,6 @@ in
'';
};
servicesValidityDays = lib.mkOption {
type = lib.types.int;
default = 1825;
description = ''
Validity window of the swarm-services sub-CA in days (~5y).
Deliberately far shorter than the root's: this CA is *meant* to
be re-issued — adding a swarm service changes its name
constraints and rotates it — so a long window buys nothing, and
a short one keeps the rotation path exercised rather than
theoretical.
Rotating it is cheap in the way rotating the root is not: it
touches only the swarm-service vhosts, and no peer hive holds it
as an anchor.
'';
};
validityDays = lib.mkOption {
type = lib.types.int;
default = 10950;
@ -125,11 +108,8 @@ in
config = lib.mkIf (hyperhiveCfg.enable && cfg.autoConfigure) {
# A CA that fails to issue is invisible until something makes a TLS call
# hours later, so these two oneshots are worth more than most services.
services.hyperhive.swarm.otel.journaldUnits = [
"swarm-ca"
"swarm-services-ca"
];
# hours later, so this oneshot is worth more than most services.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-ca" ];
systemd.services.swarm-ca = {
description = "Generate the swarm root CA when absent";
@ -183,107 +163,5 @@ in
chmod 0644 "$root"
'';
};
# The swarm-services sub-CA: issues leaves for the swarm's own
# service names, which no hive CA can sign — each of those is
# name-constrained to its own hive's domain, and the service names
# are siblings of it, not children.
#
# Rotation is the point of it being separate (mara: "swarm services
# sub ca that can rotate independently of swarm root ca"): the
# constraint enumerates the exact service names, so adding a service
# re-issues *this* and never touches the root or any hive CA.
systemd.services.swarm-services-ca = {
description = "Issue the swarm-services sub-CA under the swarm root";
wantedBy = [ "multi-user.target" ];
after = [ "swarm-ca.service" ];
requires = [ "swarm-ca.service" ];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
UMask = "0077";
SyslogIdentifier = "swarm-services-ca";
};
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
root="$d/root.pem"
rootk="$d/root-key.pem"
ca="$d/services-ca.pem"
cak="$d/services-ca-key.pem"
# The name set this CA was last issued for. Comparing against it
# is what makes re-issuance happen exactly when the service
# names change — not every boot, and not never.
names="$d/services-ca.names"
want=${lib.escapeShellArg (lib.concatStringsSep "\n" serviceDomains)}
if [ -z "$want" ]; then
echo "no swarm service domains configured — nothing to issue for"
exit 0
fi
# Same half-provisioned guard as the root: a key beside a cert
# that did not sign it looks like it works and issues nothing
# anyone will trust.
if { [ -e "$ca" ] && [ ! -e "$cak" ]; } || { [ -e "$cak" ] && [ ! -e "$ca" ]; }; then
echo "services sub-CA half-provisioned ($ca / $cak) — refusing to generate over it" >&2
exit 1
fi
if [ -s "$ca" ] && [ -s "$cak" ] && [ -f "$names" ] \
&& [ "$(cat "$names")" = "$want" ]; then
echo "services sub-CA present and covers the configured names — leaving it alone"
exit 0
fi
if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then
echo "no swarm root CA at $root — cannot issue the services sub-CA under it" >&2
exit 1
fi
echo "issuing services sub-CA at $ca for: $(echo "$want" | tr '\n' ' ')"
csr="$(mktemp "$d/services-ca.csr.XXXXXX")"
ext="$(mktemp "$d/services-ca.ext.XXXXXX")"
trap 'rm -f "$csr" "$ext"' EXIT
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$cak" -out "$csr" \
-subj "/CN=swarm-services-ca ${swarmLabel}"
{
# pathlen:0 — this signs leaves and delegates no further.
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n'
printf 'keyUsage=critical,keyCertSign,cRLSign\n'
printf 'subjectKeyIdentifier=hash\n'
printf 'authorityKeyIdentifier=keyid:always\n'
# Constrained to the exact service names, not to the whole
# swarm domain: a leaked services CA should mint `forge.`,
# `chat.`, `auth.` and nothing else. The IP exclusions are not
# redundant — a DNS constraint says nothing about an
# iPAddress SAN, and an unconstrained name type is a name
# type this CA is unconstrained for.
#
# ⚠️ EVERY entry carries its own `permitted;` / `excluded;`
# prefix. openssl's parser takes the qualifier per subtree, not
# once for a run of them: `permitted;DNS:a,DNS:b` is rejected
# outright with `v2i_NAME_CONSTRAINTS: invalid syntax`, which
# fails the whole unit. The hive CA next door emits exactly one
# permitted name, so the missing-prefix form is accidentally
# valid there and does NOT generalise — this list is always
# longer than one.
printf 'nameConstraints=critical,%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \
"$(echo "$want" | sed 's/^/permitted;DNS:/' | paste -sd, -)"
} > "$ext"
openssl x509 -req -in "$csr" -CA "$root" -CAkey "$rootk" \
-CAcreateserial -days ${toString cfg.servicesValidityDays} -sha256 \
-extfile "$ext" -out "$ca"
printf '%s' "$want" > "$names"
chmod 0600 "$cak"
chmod 0644 "$ca" "$names"
'';
};
};
}

1
nix/host-modules/swarm.nix
View file

@ -49,6 +49,7 @@ let
deployCfg.bao.matrixCtlCommonName
deployCfg.bao.grafanaOidcCommonName
deployCfg.bao.otelOidcCommonName
deployCfg.bao.servicesIssuerCommonName
]
# The two per-hive readers' subjects, spelled out per hive rather than as the
# prefix. The prefix alone would reserve the wrong string: the role for hive