feat(nix): issue each hive's CA under a swarm root CA
Cross-hive trust was O(n²) hand-pinning: every hive had to name every peer's CA. A swarm root makes it O(1) — trust the root once and every present and future peer validates. The root is generated by a new `swarm-ca` unit on a single-host swarm and operator-provided otherwise; `swarm.ca.autoConfigure` picks between them and derives its default from `swarm.peers` being empty, so "all on one host" is read off the deployment rather than remembered. Both modes produce the same artifacts in the same places, so splitting hosts later is moving the service dirs, not switching code paths. The root key never enters the nix store, and the root is never regenerated automatically — replacing it invalidates every peer at once. Each hive CA carries `nameConstraints` pinned to that hive's domain, so a leaked hive CA can only mint names inside its own subdomain, enforced by verifiers rather than by convention. `ca.pem` was serving as both the issuer and the anchor consumers trust; those are the same file only while it is self-signed. openssl will not terminate a chain at a trusted cert that isn't self-signed (rustls and Go will), so the promotion would have broken some consumers and not others. `hive-tls-ca` now also writes `trust-bundle.pem` — the hive CA plus whatever it is rooted at — and every anchor consumer reads that: agents, the CI and forge containers, and the peer-config recipe. On a hive with no swarm root the bundle is just that CA, so nothing consuming it needs a mode to branch on.
This commit is contained in:
atlas
mara
parent
fbf3757551
commit
06710e83b4
8 changed files with 366 additions and 33 deletions
|
|
@ -8,6 +8,7 @@ let
|
|||
cfg = config.services.hyperhive.tls;
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
gatewayCfg = config.services.hyperhive.gateway;
|
||||
swarmCaCfg = config.services.hyperhive.swarm.ca;
|
||||
domain = hyperhiveCfg.domain;
|
||||
|
||||
# The host-managed hive CA is the trust anchor for self-signed mode.
|
||||
|
|
@ -35,7 +36,8 @@ let
|
|||
leafk="$d/gateway-key.pem"
|
||||
csr="$(mktemp "$d/gateway.csr.XXXXXX")"
|
||||
ext="$(mktemp "$d/leaf.ext.XXXXXX")"
|
||||
trap 'rm -f "$csr" "$ext"' EXIT
|
||||
only="$(mktemp "$d/gateway.leaf.XXXXXX")"
|
||||
trap 'rm -f "$csr" "$ext" "$only"' EXIT
|
||||
|
||||
openssl req -newkey rsa:4096 -nodes -sha256 \
|
||||
-keyout "$leafk" -out "$csr" \
|
||||
|
|
@ -54,7 +56,17 @@ let
|
|||
|
||||
openssl x509 -req -in "$csr" -CA "$ca" -CAkey "$cak" \
|
||||
-CAcreateserial -days ${toString cfg.leafValidityDays} -sha256 \
|
||||
-extfile "$ext" -out "$leaf"
|
||||
-extfile "$ext" -out "$only"
|
||||
|
||||
# nginx serves this file verbatim, so it must carry the leaf AND its
|
||||
# issuer: the hive CA is an intermediate under the swarm root, and a
|
||||
# client that anchors on the root cannot build the middle of the
|
||||
# chain by itself. Agents anchor on the hive CA directly and
|
||||
# validated either way — the appended cert is what makes a swarm
|
||||
# peer, or anything else holding only the root, work.
|
||||
# Leaf first: both `ssl_certificate` and the `openssl x509 -in`
|
||||
# expiry checks read the first cert in the file.
|
||||
cat "$only" "$ca" > "$leaf"
|
||||
chmod 0600 "$leafk"
|
||||
chmod 0644 "$leaf"
|
||||
'';
|
||||
|
|
@ -68,11 +80,18 @@ in
|
|||
# runtime-generated, in-container cert can't be wired into an agent's
|
||||
# build-time trust store at all.
|
||||
#
|
||||
# So the anchor is a long-lived **hive CA** held on the host. The
|
||||
# So the issuer is a long-lived **hive CA** held on the host. The
|
||||
# gateway serves a **leaf** signed by that CA (via the `tls.certDir`
|
||||
# bind-mount path); agents and federation peers trust the *CA* once,
|
||||
# and leaf rotation never re-breaks them. See `docs/gateway.md`
|
||||
# ("Self-signed TLS").
|
||||
#
|
||||
# The hive CA is itself an intermediate, issued under the swarm root
|
||||
# (./swarm-ca.nix, which owns the why) and name-constrained to this
|
||||
# hive's domain — so it stays the anchor agents pin, while a peer that
|
||||
# trusts only the root can still validate everything this hive serves.
|
||||
# Hives that predate the root keep their self-signed CA until an
|
||||
# operator drops it; see the issuance comment below.
|
||||
|
||||
options.services.hyperhive.tls = {
|
||||
stateDir = lib.mkOption {
|
||||
|
|
@ -131,6 +150,13 @@ in
|
|||
# instances of the `container@.service` template.
|
||||
before = [ "container@hive-gateway.service" ];
|
||||
requiredBy = [ "container@hive-gateway.service" ];
|
||||
# The issuance below needs the swarm root key on disk. When this
|
||||
# host generates it (single-host swarm) that unit must have run;
|
||||
# when the operator provides it there is no unit to wait for, so
|
||||
# the dependency is conditional rather than a unit that exists and
|
||||
# does nothing.
|
||||
after = lib.optional swarmCaCfg.autoConfigure "swarm-ca.service";
|
||||
requires = lib.optional swarmCaCfg.autoConfigure "swarm-ca.service";
|
||||
path = [ pkgs.openssl ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
|
|
@ -148,20 +174,68 @@ in
|
|||
cak="$d/ca-key.pem"
|
||||
leaf="$d/gateway.pem"
|
||||
leafk="$d/gateway-key.pem"
|
||||
root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
|
||||
rootk=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root-key.pem"}
|
||||
|
||||
# --- CA: generate once, reuse across leaf rotations. Regenerate
|
||||
# only if missing or already expired (checkend 0). A new CA means
|
||||
# every consumer must re-trust, so the leaf is dropped to force a
|
||||
# re-sign under the fresh CA.
|
||||
# --- CA: an intermediate under the swarm root, generated once
|
||||
# and reused across leaf rotations. Regenerated only if missing
|
||||
# or already expired (checkend 0). A new CA means every consumer
|
||||
# must re-trust, so the leaf is dropped to force a re-sign under
|
||||
# the fresh CA.
|
||||
#
|
||||
# An existing CA is never re-rooted here. A hive predating the
|
||||
# swarm root carries a self-signed `ca.pem`, and swapping it for
|
||||
# a swarm-issued one would break every consumer already trusting
|
||||
# it — including agents, whose trust is refreshed only when their
|
||||
# container restarts. Adopting the hierarchy on such a hive is
|
||||
# therefore an operator step (drop the CA, let this unit
|
||||
# re-issue), which is also what keeps this change non-disruptive
|
||||
# to hives that never adopt it.
|
||||
if [ ! -s "$ca" ] || [ ! -s "$cak" ] \
|
||||
|| ! openssl x509 -in "$ca" -noout -checkend 0 >/dev/null 2>&1; then
|
||||
echo "generating fresh hive CA at $ca"
|
||||
openssl req -x509 -newkey rsa:4096 -nodes -sha256 \
|
||||
-days ${toString cfg.caValidityDays} \
|
||||
-keyout "$cak" -out "$ca" \
|
||||
-subj "/CN=hive-ca ${domain}" \
|
||||
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
|
||||
-addext "keyUsage=critical,keyCertSign,cRLSign"
|
||||
# Signing needs the root's private key, which on a multi-host
|
||||
# swarm is deliberately somewhere else. There is nothing to
|
||||
# fall back to: a self-signed CA here would still serve TLS
|
||||
# and would silently not be part of the swarm's trust.
|
||||
if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then
|
||||
echo "no swarm root CA key at $rootk — cannot issue this hive's CA." >&2
|
||||
echo "Either set services.hyperhive.swarm.ca.autoConfigure (single-host swarm)," >&2
|
||||
echo "or install the operator-issued hive CA at $ca + $cak." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "issuing fresh hive CA at $ca under the swarm root"
|
||||
cacsr="$(mktemp "$d/ca.csr.XXXXXX")"
|
||||
caext="$(mktemp "$d/ca.ext.XXXXXX")"
|
||||
trap 'rm -f "$cacsr" "$caext"' EXIT
|
||||
|
||||
openssl req -newkey rsa:4096 -nodes -sha256 \
|
||||
-keyout "$cak" -out "$cacsr" \
|
||||
-subj "/CN=hive-ca ${domain}"
|
||||
|
||||
# printf (not a heredoc) so the ext-file lines carry no leading
|
||||
# whitespace once nix has stripped the indented-string indent.
|
||||
{
|
||||
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n'
|
||||
printf 'keyUsage=critical,keyCertSign,cRLSign\n'
|
||||
printf 'subjectKeyIdentifier=hash\n'
|
||||
printf 'authorityKeyIdentifier=keyid:always\n'
|
||||
# The constraint is the point of the hierarchy, not a
|
||||
# flourish: without it a leaked hive CA mints any name in
|
||||
# the swarm, and it is verifiers that enforce this, not our
|
||||
# good behaviour. The IP exclusions are not redundant — a
|
||||
# DNS constraint says nothing about an iPAddress SAN, and a
|
||||
# name type nobody constrained is a name type this CA is
|
||||
# unconstrained for.
|
||||
printf 'nameConstraints=critical,permitted;DNS:%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \
|
||||
${lib.escapeShellArg domain}
|
||||
} > "$caext"
|
||||
|
||||
openssl x509 -req -in "$cacsr" -CA "$root" -CAkey "$rootk" \
|
||||
-CAcreateserial -days ${toString cfg.caValidityDays} -sha256 \
|
||||
-extfile "$caext" -out "$ca"
|
||||
rm -f "$cacsr" "$caext"
|
||||
trap - EXIT
|
||||
chmod 0600 "$cak"
|
||||
chmod 0644 "$ca"
|
||||
rm -f "$leaf" "$leafk"
|
||||
|
|
@ -174,6 +248,34 @@ in
|
|||
echo "signing fresh gateway leaf at $leaf"
|
||||
${signLeafScript} "$d"
|
||||
fi
|
||||
|
||||
# --- Trust bundle: what a consumer must TRUST, as opposed to
|
||||
# `ca.pem`, which is what this host SIGNS with. The two were the
|
||||
# same file while the hive CA was self-signed, and stopped being
|
||||
# the same file the moment it became an intermediate: openssl
|
||||
# refuses to end a chain at a trusted cert that isn't
|
||||
# self-signed (that's what `-partial_chain` is for), so an
|
||||
# agent's curl handed only `ca.pem` fails with "unable to get
|
||||
# issuer certificate". Verified in both directions before this
|
||||
# was written — rustls and Go accept a trusted intermediate,
|
||||
# which is what makes the breakage partial and easy to miss.
|
||||
#
|
||||
# Consumers therefore trust hive CA + swarm root; on a hive that
|
||||
# still has a self-signed CA the bundle is just that CA, so the
|
||||
# consumer side needs no condition at all. Trusting the root is
|
||||
# also the point of the hierarchy — it is what lets a peer hive
|
||||
# validate without being hand-pinned.
|
||||
#
|
||||
# Written IN PLACE, never renamed into position: containers bind
|
||||
# -mount this file, and a bind mount follows the inode. A
|
||||
# rename would leave every consumer holding the old one.
|
||||
bundle="$d/trust-bundle.pem"
|
||||
if [ -s "$root" ]; then
|
||||
cat "$ca" "$root" > "$bundle"
|
||||
else
|
||||
cat "$ca" > "$bundle"
|
||||
fi
|
||||
chmod 0644 "$bundle"
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -256,11 +358,12 @@ in
|
|||
};
|
||||
|
||||
# Signal the hive-c0re lifecycle that a hive CA exists: it bind-mounts
|
||||
# this file (read-only, the CA cert ONLY — never the key) into each
|
||||
# this file (read-only, public certs ONLY — never a key) into each
|
||||
# agent container so agents + their tools can trust the gateway's
|
||||
# self-signed leaf, and the meta flake wires the per-agent trust
|
||||
# bundle. Only the `ca.pem` path is exposed; `ca-key.pem` stays on the
|
||||
# host (an agent that could read it could mint trusted certs).
|
||||
systemd.services.hive-c0re.environment.HIVE_TLS_CA_PATH = "${cfg.stateDir}/ca.pem";
|
||||
# bundle. It is the ANCHOR bundle rather than `ca.pem` for the reason
|
||||
# spelled out where the bundle is written above; no key path is ever
|
||||
# exposed (an agent that could read one could mint trusted certs).
|
||||
systemd.services.hive-c0re.environment.HIVE_TLS_CA_PATH = "${cfg.stateDir}/trust-bundle.pem";
|
||||
};
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue