vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

nix: issue the swarm-services leaf from bao's pki mount

Browse source 
The `pki` mount had no issuer and no principal could log in to it, so the
swarm's service certificates were still minted by two openssl hops from a
root key on disk. Close both halves and retire the openssl path with them.

The mount now generates its own root, once. The granting unit asks bao
whether an issuer already exists (`bao list pki/issuers`) before calling
`pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role
and the grant without touching the anchor — a root that changed per boot
would invalidate every certificate issued under it and every browser
taught to trust it. The guard asks the store rather than looking for a
marker file on this host's disk: a file is a claim about a mount that may
have been restored from a snapshot or disabled and re-enabled underneath
it.

`swarm-services-issuer` stops being an inert policy. A fourth cert-auth
role attaches it, following the shape the controller, the publisher and
matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its
CN — that credential is what opens the mount, so it cannot come out of it.

`swarm-services-cert.service` logs in with that leaf, calls
`pki/issue/swarm-services`, and writes the result to the path
hive-tls.nix already wrote and the gateway already copies from. The
sub-CA layer does not move; it stops existing. The role's
`allowed_domains`, read from the same `swarm.serviceDomains` the SANs
come from, enforces at issue time what the sub-CA encoded in x509
`nameConstraints`, and with the root inside the mount there is nothing
left for an intermediate to be an intermediate of.

Not a flag day: the issuing root is published beside the leaf as
`swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the
swarm root still sits. A leaf chaining to the old sub-CA and one issued
by the store both verify against the same bundle, so hives can be
rebuilt in any order. The same file is what an operator hands a browser
— readable without a store login, which matters because every listener
demands a client certificate.

The eval-time warning about uncovered service names is gone rather than
reworded. It fired on "this host does not hold the swarm root key", which
was the reason a hive could end up serving its own leaf on a
swarm-service name. Every hive now asks the store with its own identity,
so that stopped being the thing that decides.

Closes #4586
This commit is contained in:
atlas 2026-09-21 17:59:22 +02:00 • committed by mara
commit f4df4fc4a9
10 changed files with 700 additions and 315 deletions
Download patch file Download diff file Expand all files Collapse all files

243
nix/host-modules/swarm-bao.nix
View file

@ -300,15 +300,48 @@ let
# self-signed CA under `/var/lib/swarm-bao-pki`, and it stays outside the
# store permanently: bao cannot issue the credential that opens bao.
#
# Nothing issues through this mount yet — see the role below.
servicesPkiMountPath = "pki";
# The mount's own root is generated into it by the bootstrap unit below and
# its private key never leaves — `root/generate/internal` keeps it inside
# the store, which is the whole reason the mount exists rather than another
# key beside ./swarm-ca.nix's.
#
# An option rather than a literal because ./hive-tls.nix issues through it
# and has to spell it the same way.
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
# Subject of the root generated into that mount. A label for a human reading
# a chain, not an identity anything authenticates against — same fall-through
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
# neither a swarm name nor a domain must still evaluate.
servicesPkiRootLabel =
if hyperhiveCfg.swarm.name != null then
hyperhiveCfg.swarm.name
else if hyperhiveCfg.domain != null then
hyperhiveCfg.domain
else
"hyperhive";
# Where the store's host publishes that root's PUBLIC certificate. Its own
# directory rather than ./glue-bao-tls.nix's `/var/lib/swarm-bao-pki`: that
# one holds the CA key that opens the store, is 0700 for exactly that
# reason, and this file is the opposite kind of thing — the one piece of
# certificate material meant to be copied out and handed around.
servicesPkiRootDir = "/var/lib/swarm-bao-services-pki";
servicesPkiRootFile = "${servicesPkiRootDir}/services-root.pem";
# ~30y, matching ./swarm-ca.nix's root rather than its services sub-CA. This
# is an ANCHOR: an operator adds it to a browser's trust store by hand, and
# every rotation is a manual re-trust on every machine that holds it. The
# sub-CA's short window bought cheap rotation because the root above it
# absorbed the churn; nothing sits above this one.
servicesPkiRootTtl = "262800h";
# The role every swarm-service certificate will be issued through, and the
# whole narrowing of this mount. A role rather than a name-constrained
# sub-CA: the engine enforces the same "these names and no others" at issue
# time that ./swarm-ca.nix builds into x509 `nameConstraints` today, so a
# second CA in the chain would re-implement a check that already happens.
servicesPkiRoleName = "swarm-services";
servicesPkiRoleName = baoDeploy.servicesPkiRoleName;
# The names that role may issue for. Read out of the swarm-tier option
# rather than assembled here, exactly as ./swarm-ca.nix:45 and
@ -317,23 +350,22 @@ let
# composing its own list is how they stop agreeing.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# The identity that will ask this mount for a certificate, once something
# does. A THIRD principal rather than either existing one, for the reason
# `secretPublisherCommonName` gives for being a second: the controller may
# rewrite every hive's policy and login role, and a unit whose whole job is
# renewing one server certificate has no business holding that.
# The identity that asks this mount for a certificate. A FOURTH principal
# rather than any existing one, for the reason `secretPublisherCommonName`
# gives for being a second: the controller may rewrite every hive's policy
# and login role, and a unit whose whole job is renewing one server
# certificate has no business holding that.
#
# `update` and not `create`: bao's issue endpoint is a POST to an existing
# path, which is `update` in ACL terms — `create` here would grant nothing
# and read as though it did.
#
# ⚠️ NOTHING IS ATTACHED TO THIS POLICY. A bao policy grants only through a
# token that carries it, so until a login role names it this is an inert
# declaration of intent — which is the point: the cert-auth role that
# attaches it needs a leaf carrying its CN, and minting that is
# ./glue-bao-tls.nix's job, one step further along than this one.
servicesIssuerPolicyName = "swarm-services-issuer";
# The subject the login role accepts, and the CN ./glue-bao-tls.nix signs
# the matching leaf with. An option rather than a literal for the reason
# `controllerCn` gives: two files have to spell it identically.
servicesIssuerCn = baoDeploy.servicesIssuerCommonName;
servicesIssuerPolicyText = ''
path "${servicesPkiMountPath}/issue/${servicesPkiRoleName}" {
capabilities = ["update"]
@ -971,6 +1003,65 @@ in
'';
};
servicesPkiMountPath = lib.mkOption {
type = lib.types.str;
default = "pki";
description = ''
Mount path of the PKI engine the swarm's **service** certificates are
issued from, and whose root is the anchor those certificates chain to.
An option rather than a literal because two files have to spell it
identically: this module enables the mount, generates its root and
writes the grant that points at `<path>/issue/`, while ./hive-tls.nix
calls that endpoint. A mount path only one of them knows is a grant
naming a location nobody issues from.
⚠️ NOT the store's own client-auth PKI. That one is
./glue-bao-tls.nix's self-signed CA on disk, and it stays outside the
store permanently — bao cannot issue the credential that opens bao.
'';
};
servicesPkiRoleName = lib.mkOption {
type = lib.types.str;
default = "swarm-services";
description = ''
Role on {option}`services.hyperhive.deploy.bao.servicesPkiMountPath`
that the swarm-services certificate is issued through, and the whole
narrowing of that mount.
A role rather than a name-constrained sub-CA: the engine enforces the
same "these names and no others" at issue time, from an
`allowed_domains` read off
{option}`services.hyperhive.swarm.serviceDomains`, that an
intermediate would encode in x509 `nameConstraints` — so a second
authority in the chain would re-implement a check that has already
happened.
'';
};
servicesIssuerCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-services-issuer";
example = "swarm-services-issuer.svc";
description = ''
Subject the store's services-issuer cert-auth role accepts — the
identity a hive presents when it asks the `pki` mount for the
swarm-services certificate its gateway serves.
A **fourth** identity rather than reuse of any sibling above, and the
reasoning is theirs: this principal's whole grant is one `update` on
`pki/issue/swarm-services`, while the controller's includes rewriting
every hive's policy and login role. Handing a certificate-renewal unit
the controller's leaf would give that separation away in one line.
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
feeds this value into the guard on
{option}`services.hyperhive.swarm.hives`, so a hive named after it
fails evaluation rather than silently receiving this grant.
'';
};
matrixCtlHiveName = lib.mkOption {
type = lib.types.str;
default = toString hyperhiveCfg.hiveName;
@ -1407,6 +1498,7 @@ in
"swarm-bao-certs"
"swarm-bao-token"
"swarm-bao-forwarder-oidc"
"swarm-bao-services-issuer-policy"
];
# 🚫 No `swarm.otel.scrapeTargets.bao` entry any more, and its absence is
@ -1748,21 +1840,69 @@ in
*) bao secrets enable -path=${credentialMountPath} kv-v2 ;;
esac
# The PKI engine the swarm's service certificates are to come from.
# Asked rather than attempted for the same reason as the mount above:
# The PKI engine the swarm's service certificates come from. Asked
# rather than attempted for the same reason as the mount above:
# `secrets enable` errors on a path already in use, so a second
# rebuild would fail a unit that has nothing left to do.
#
# ⚠️ Deliberately EMPTY: nothing here generates an issuer into it, so
# the role below can be written but cannot yet issue. That is the
# whole shape of this step — the mount, its narrowing and its grant
# exist, and the binary that uses them arrives separately. Undoing it
# is `bao secrets disable ${servicesPkiMountPath}`.
case "$mounts" in
*'"${servicesPkiMountPath}/"'*) ;;
*) bao secrets enable -path=${servicesPkiMountPath} pki ;;
esac
# ⚠️ THE ROOT, AND THE ONE THING THIS UNIT MUST NEVER DO TWICE.
#
# `root/generate/internal` mints a new self-signed CA every time it
# is called. Calling it unconditionally would hand the swarm a
# different anchor on every boot: every certificate issued under the
# previous one stops verifying, and every browser an operator taught
# to trust the old root has to be re-taught. So the generation is
# guarded, and the guard asks BAO rather than looking for a marker
# file — a file on this host's disk is a claim about a store that
# may live in a container, may have been restored from a snapshot,
# and may have had the mount disabled and re-enabled underneath it.
# The mount's own issuer list is the only answer that cannot be
# stale.
#
# `bao list ${servicesPkiMountPath}/issuers` is a 404 (non-zero) on a
# mount with no issuer and a key list once one exists, so its exit
# status IS the question — no output parsing, no error string to
# recognise. The default issuer is what `${servicesPkiMountPath}/issue/`
# signs with, and there is exactly one.
if ! bao list ${lib.escapeShellArg "${servicesPkiMountPath}/issuers"} >/dev/null 2>&1; then
echo "generating the swarm services root into the ${servicesPkiMountPath} mount"
# `-field=issuing_ca` discards the rest of the response. The
# private key is not in it under `internal` and cannot be — that
# is the point of the endpoint — and the certificate is read back
# below from a path that works on every subsequent boot too.
bao write -field=issuing_ca ${lib.escapeShellArg "${servicesPkiMountPath}/root/generate/internal"} \
common_name=${lib.escapeShellArg "swarm-services-ca ${servicesPkiRootLabel}"} \
issuer_name=${lib.escapeShellArg servicesPkiRoleName} \
ttl=${servicesPkiRootTtl} \
key_type=rsa \
key_bits=4096 >/dev/null
else
echo "the ${servicesPkiMountPath} mount already has an issuer — leaving it alone"
fi
# The anchor, as a world-readable file on the store's own host.
#
# Written on EVERY run and not only after a generation, so a store
# whose root predates this unit still publishes it. It is a public
# certificate — the material a browser has to be told to trust —
# and the private half stays in the mount, so 0644 is the correct
# mode rather than a concession.
#
# Why a file at all when `bao read -field=certificate
# ${servicesPkiMountPath}/cert/ca` returns the same bytes: every
# listener sets `tls_require_and_verify_client_cert`, so reading it
# from the store costs a client certificate. An operator adding the
# root to a browser is not necessarily holding one.
install -d -m 0755 ${lib.escapeShellArg servicesPkiRootDir}
bao read -field=certificate ${lib.escapeShellArg "${servicesPkiMountPath}/cert/ca"} \
> ${lib.escapeShellArg "${servicesPkiRootFile}.new"}
chmod 0644 ${lib.escapeShellArg "${servicesPkiRootFile}.new"}
mv -f ${lib.escapeShellArg "${servicesPkiRootFile}.new"} ${lib.escapeShellArg servicesPkiRootFile}
# `bao write` is an upsert, so this re-asserts the role on every
# rebuild rather than failing on one that exists — same shape as the
# cert-auth roles below, and the reason the mount needs the `case`
@ -1799,14 +1939,6 @@ in
client_flag=false \
key_type=rsa \
key_bits=4096
# Idempotent for the same reason the controller's policy above is: a
# rebuild re-asserts it. Written here rather than in a unit of its
# own because it attaches to nothing yet — the sibling-unit rule
# below is about a second *principal* that logs in, and this policy
# has no login role until the leaf carrying its CN exists.
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
@ -1958,6 +2090,57 @@ in
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
# A FOURTH sibling, same shape and same reasons as the two above. This
# one is what turns `swarm-services-issuer` from a declaration into a
# grant: a bao policy reaches nothing until a login role hands it to a
# token, so the `bao write auth/cert/certs/…` below is the whole
# difference between a mount nobody may issue from and one the gateway's
# certificate comes out of.
#
# The policy text moved here from the controller's unit, where it sat
# while it attached to nothing — a policy and the role that carries it
# belong in one place, and now there is a principal to put them with.
systemd.services.swarm-bao-services-issuer-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm services issuer's bao policy and cert-auth role";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
# Same unseal wait as its three siblings above, for the reason stated
# there: under `seal = "shamir"` a human unseals by hand.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
bao write auth/cert/certs/${lib.escapeShellArg servicesIssuerPolicyName} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg servicesIssuerCn} \
token_policies=${lib.escapeShellArg servicesIssuerPolicyName} \
display_name=${lib.escapeShellArg servicesIssuerCn}
'';
};
# The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet.