cf4a60b837
main is already unborked -- the constraint-syntax fix landed directly in31e47632and the hive-tls-ca ordering race inb5066ad0. What is left is the reason the syntax bug became an OUTAGE rather than a warning, and it is independent of both. nginx refuses to load a config naming a cert file that does not exist: `cannot load certificate ... no such file` fails the pre-start test, so the vhost does not degrade -- the ENTIRE proxy dies and restart-loops into start-limit-hit, taking the forge, the dashboard and matrix with it. Signing tolerated the leaf's absence and the container import tolerated it; the vhost reference did not, and that is the one that decides whether anything serves. Absence stays a normal state here: the leaf only exists where the swarm CA is autoconfigured. So the import now installs the HIVE leaf at the services path instead of removing the file. That is a name mismatch on those vhosts -- browsers warn, strict clients refuse -- and every other name keeps working. A bad certificate is a bad afternoon; a dead gateway is a dead hive. Also keeps the comment explaining why every nameConstraints entry needs its own `permitted;` prefix, so the next reader doesn't re-derive it from an outage.
296 lines
12 KiB
Nix
296 lines
12 KiB
Nix
# The swarm root CA: the anchor a whole swarm shares, and the issuer of
|
||
# each hive's own CA (which is where it gets used — see ./hive-tls.nix).
|
||
#
|
||
# Why a hierarchy at all: cross-hive trust used to be hand-pinned per
|
||
# peer, so every hive had to name every other one — O(n²) configuration
|
||
# that a new hive could only join by editing all the others. One root
|
||
# makes it O(1): trust the root once and every present *and future* peer
|
||
# validates. The per-peer CA field is gone precisely because this
|
||
# replaced it.
|
||
#
|
||
# Two provisioning modes, ONE structure — what differs is who puts the
|
||
# artifacts on disk, never what the artifacts are:
|
||
#
|
||
# - autoconfigured (`services.hyperhive.enableAllLocalDefaults`, or
|
||
# `autoConfigure` set directly): the unit below generates the root
|
||
# here on first boot.
|
||
# - operator-provided (multi-host): the operator installs the root cert
|
||
# — and, on a host that does not hold the root key, the hive CA too —
|
||
# into the state dirs, and this unit does nothing. Splitting a
|
||
# single-host swarm across hosts is then a matter of moving the
|
||
# service dirs and turning `autoConfigure` off, not of a second code
|
||
# path that has to be kept agreeing with the first.
|
||
#
|
||
# The root KEY is the reason this is a runtime file and not a nix option:
|
||
# the store is world-readable and content-addressed, so a key committed
|
||
# to a flake is a key *published* to every consumer of that flake.
|
||
#
|
||
# ⚠️ The root CERT is therefore a runtime file too, which costs something:
|
||
# nothing whose trust store is built at build time can name it. See
|
||
# docs/swarm/ca.md § "Distributing the root".
|
||
{
|
||
lib,
|
||
config,
|
||
pkgs,
|
||
...
|
||
}:
|
||
let
|
||
cfg = config.services.hyperhive.swarm.ca;
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
|
||
# The subject CN is a label for a human reading a chain, not an
|
||
# identity anything authenticates against. Fall through swarm name →
|
||
# hive domain → a constant so a hive that has set neither still
|
||
# evaluates; a missing `domain` is reported by its own assertion in
|
||
# hive-network.nix, and shouldn't also surface here as a null.
|
||
swarmLabel =
|
||
if hyperhiveCfg.swarm.name != null then
|
||
hyperhiveCfg.swarm.name
|
||
else if hyperhiveCfg.domain != null then
|
||
hyperhiveCfg.domain
|
||
else
|
||
"hyperhive";
|
||
|
||
# Derived once in ./swarm.nix, read here and by ./hive-tls.nix: the
|
||
# CA that name-constrains these and the leaf that carries them as SANs
|
||
# must agree exactly, and two modules each assembling the list is how
|
||
# they stop agreeing. It is also this unit's *rotation trigger* below,
|
||
# which is why the ordering is stable there rather than here.
|
||
serviceDomains = hyperhiveCfg.swarm.serviceDomains;
|
||
in
|
||
{
|
||
options.services.hyperhive.swarm.ca = {
|
||
autoConfigure = lib.mkOption {
|
||
type = lib.types.bool;
|
||
default = false;
|
||
example = true;
|
||
description = ''
|
||
Run the whole swarm CA on this one host: generate the swarm
|
||
root when it is missing, and issue this hive's CA under it.
|
||
|
||
`services.hyperhive.enableAllLocalDefaults` turns this on as
|
||
part of the all-on-one-box mode. Set it here directly to run the
|
||
CA on a host that is not otherwise all-local.
|
||
|
||
**Off by default, deliberately.** A swarm's services and its
|
||
hives can live on different hosts, and this host has no way to
|
||
tell whether it is the one holding the root — so the swarm CA
|
||
is something an operator sets up, not something a host decides
|
||
it is. Turn this on for an all-on-one-host deployment (dev
|
||
boxes, single-hive swarms) and get the hierarchy for free.
|
||
|
||
With it off, both artifacts are operator-provided: the root
|
||
under `stateDir`, and this hive's CA under
|
||
`services.hyperhive.tls.stateDir`. A hive given neither keeps
|
||
the self-signed CA it has always had — it simply isn't part of
|
||
a swarm's trust hierarchy, which is the correct outcome for a
|
||
hive nobody has federated yet.
|
||
'';
|
||
};
|
||
|
||
stateDir = lib.mkOption {
|
||
type = lib.types.str;
|
||
default = "/var/lib/swarm-ca";
|
||
description = ''
|
||
Host directory holding the swarm root CA: `root.pem` (the
|
||
anchor, safe to distribute — copy it to this same path on every
|
||
other host in the swarm) and `root-key.pem`
|
||
(0600, the one file that must never reach the nix store or
|
||
another host). The directory itself is 0700: nothing reads
|
||
out of it but the hive CA issuance in `hive-tls.nix`.
|
||
|
||
Moving the swarm CA to its own host is a matter of moving this
|
||
directory and setting `autoConfigure = false` here.
|
||
'';
|
||
};
|
||
|
||
servicesValidityDays = lib.mkOption {
|
||
type = lib.types.int;
|
||
default = 1825;
|
||
description = ''
|
||
Validity window of the swarm-services sub-CA in days (~5y).
|
||
Deliberately far shorter than the root's: this CA is *meant* to
|
||
be re-issued — adding a swarm service changes its name
|
||
constraints and rotates it — so a long window buys nothing, and
|
||
a short one keeps the rotation path exercised rather than
|
||
theoretical.
|
||
|
||
Rotating it is cheap in the way rotating the root is not: it
|
||
touches only the swarm-service vhosts, and no peer hive holds it
|
||
as an anchor.
|
||
'';
|
||
};
|
||
|
||
validityDays = lib.mkOption {
|
||
type = lib.types.int;
|
||
default = 10950;
|
||
description = ''
|
||
Validity window of the swarm root CA in days (default ~30y).
|
||
Deliberately longer than `services.hyperhive.tls.caValidityDays`:
|
||
the root must outlive the hive CAs it issues, or those chains
|
||
expire out from under hives that are still perfectly happy with
|
||
their own intermediate. Rotating a root is the one operation in
|
||
this system with no partial-failure mode — it invalidates every
|
||
peer at once, paced by the slowest peer's rebuild — so it is
|
||
never automatic and this window is meant to be uneventful.
|
||
'';
|
||
};
|
||
};
|
||
|
||
config = lib.mkIf (hyperhiveCfg.enable && cfg.autoConfigure) {
|
||
systemd.services.swarm-ca = {
|
||
description = "Generate the swarm root CA when absent";
|
||
wantedBy = [ "multi-user.target" ];
|
||
path = [ pkgs.openssl ];
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
UMask = "0077";
|
||
# Pin the journal identity (else it's the `script` store-path wrapper).
|
||
SyslogIdentifier = "swarm-ca";
|
||
};
|
||
script = ''
|
||
set -euo pipefail
|
||
d=${lib.escapeShellArg cfg.stateDir}
|
||
install -d -m 0700 "$d"
|
||
|
||
root="$d/root.pem"
|
||
rootk="$d/root-key.pem"
|
||
|
||
# Note the asymmetry with the hive CA in hive-tls.nix, which
|
||
# regenerates itself once expired: a root is never replaced
|
||
# automatically, not even an expired one. Consumers hold this
|
||
# cert, so replacing it is a swarm-wide flag day that wants an
|
||
# operator running it deliberately, with both roots trusted
|
||
# across the overlap.
|
||
if [ -s "$root" ] && [ -s "$rootk" ]; then
|
||
echo "swarm root CA already present at $root — leaving it alone"
|
||
exit 0
|
||
fi
|
||
|
||
# Half a root is not a root. Generating a fresh key beside an
|
||
# already-distributed cert (or the reverse) leaves every
|
||
# consumer trusting an anchor that no longer signs anything —
|
||
# and it would look like it worked.
|
||
if [ -e "$root" ] || [ -e "$rootk" ]; then
|
||
echo "swarm root CA half-provisioned ($root / $rootk) — refusing to generate over it" >&2
|
||
exit 1
|
||
fi
|
||
|
||
echo "generating swarm root CA at $root"
|
||
# pathlen:1 — the root signs hive CAs, which sign leaves. One
|
||
# intermediate below the root and no deeper.
|
||
openssl req -x509 -newkey rsa:4096 -nodes -sha256 \
|
||
-days ${toString cfg.validityDays} \
|
||
-keyout "$rootk" -out "$root" \
|
||
-subj "/CN=swarm-ca ${swarmLabel}" \
|
||
-addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
|
||
-addext "keyUsage=critical,keyCertSign,cRLSign"
|
||
chmod 0600 "$rootk"
|
||
chmod 0644 "$root"
|
||
'';
|
||
};
|
||
|
||
# The swarm-services sub-CA: issues leaves for the swarm's own
|
||
# service names, which no hive CA can sign — each of those is
|
||
# name-constrained to its own hive's domain, and the service names
|
||
# are siblings of it, not children.
|
||
#
|
||
# Rotation is the point of it being separate (mara: "swarm services
|
||
# sub ca that can rotate independently of swarm root ca"): the
|
||
# constraint enumerates the exact service names, so adding a service
|
||
# re-issues *this* and never touches the root or any hive CA.
|
||
systemd.services.swarm-services-ca = {
|
||
description = "Issue the swarm-services sub-CA under the swarm root";
|
||
wantedBy = [ "multi-user.target" ];
|
||
after = [ "swarm-ca.service" ];
|
||
requires = [ "swarm-ca.service" ];
|
||
path = [ pkgs.openssl ];
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
UMask = "0077";
|
||
SyslogIdentifier = "swarm-services-ca";
|
||
};
|
||
script = ''
|
||
set -euo pipefail
|
||
d=${lib.escapeShellArg cfg.stateDir}
|
||
root="$d/root.pem"
|
||
rootk="$d/root-key.pem"
|
||
ca="$d/services-ca.pem"
|
||
cak="$d/services-ca-key.pem"
|
||
# The name set this CA was last issued for. Comparing against it
|
||
# is what makes re-issuance happen exactly when the service
|
||
# names change — not every boot, and not never.
|
||
names="$d/services-ca.names"
|
||
want=${lib.escapeShellArg (lib.concatStringsSep "\n" serviceDomains)}
|
||
|
||
if [ -z "$want" ]; then
|
||
echo "no swarm service domains configured — nothing to issue for"
|
||
exit 0
|
||
fi
|
||
|
||
# Same half-provisioned guard as the root: a key beside a cert
|
||
# that did not sign it looks like it works and issues nothing
|
||
# anyone will trust.
|
||
if { [ -e "$ca" ] && [ ! -e "$cak" ]; } || { [ -e "$cak" ] && [ ! -e "$ca" ]; }; then
|
||
echo "services sub-CA half-provisioned ($ca / $cak) — refusing to generate over it" >&2
|
||
exit 1
|
||
fi
|
||
|
||
if [ -s "$ca" ] && [ -s "$cak" ] && [ -f "$names" ] \
|
||
&& [ "$(cat "$names")" = "$want" ]; then
|
||
echo "services sub-CA present and covers the configured names — leaving it alone"
|
||
exit 0
|
||
fi
|
||
|
||
if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then
|
||
echo "no swarm root CA at $root — cannot issue the services sub-CA under it" >&2
|
||
exit 1
|
||
fi
|
||
|
||
echo "issuing services sub-CA at $ca for: $(echo "$want" | tr '\n' ' ')"
|
||
csr="$(mktemp "$d/services-ca.csr.XXXXXX")"
|
||
ext="$(mktemp "$d/services-ca.ext.XXXXXX")"
|
||
trap 'rm -f "$csr" "$ext"' EXIT
|
||
|
||
openssl req -newkey rsa:4096 -nodes -sha256 \
|
||
-keyout "$cak" -out "$csr" \
|
||
-subj "/CN=swarm-services-ca ${swarmLabel}"
|
||
|
||
{
|
||
# pathlen:0 — this signs leaves and delegates no further.
|
||
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n'
|
||
printf 'keyUsage=critical,keyCertSign,cRLSign\n'
|
||
printf 'subjectKeyIdentifier=hash\n'
|
||
printf 'authorityKeyIdentifier=keyid:always\n'
|
||
# Constrained to the exact service names, not to the whole
|
||
# swarm domain: a leaked services CA should mint `forge.`,
|
||
# `chat.`, `auth.` and nothing else. The IP exclusions are not
|
||
# redundant — a DNS constraint says nothing about an
|
||
# iPAddress SAN, and an unconstrained name type is a name
|
||
# type this CA is unconstrained for.
|
||
#
|
||
# ⚠️ EVERY entry carries its own `permitted;` / `excluded;`
|
||
# prefix. openssl's parser takes the qualifier per subtree, not
|
||
# once for a run of them: `permitted;DNS:a,DNS:b` is rejected
|
||
# outright with `v2i_NAME_CONSTRAINTS: invalid syntax`, which
|
||
# fails the whole unit. The hive CA next door emits exactly one
|
||
# permitted name, so the missing-prefix form is accidentally
|
||
# valid there and does NOT generalise — this list is always
|
||
# longer than one.
|
||
printf 'nameConstraints=critical,%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \
|
||
"$(echo "$want" | sed 's/^/permitted;DNS:/' | paste -sd, -)"
|
||
} > "$ext"
|
||
|
||
openssl x509 -req -in "$csr" -CA "$root" -CAkey "$rootk" \
|
||
-CAcreateserial -days ${toString cfg.servicesValidityDays} -sha256 \
|
||
-extfile "$ext" -out "$ca"
|
||
|
||
printf '%s' "$want" > "$names"
|
||
chmod 0600 "$cak"
|
||
chmod 0644 "$ca" "$names"
|
||
'';
|
||
};
|
||
};
|
||
}
|