f4df4fc4a9
The `pki` mount had no issuer and no principal could log in to it, so the swarm's service certificates were still minted by two openssl hops from a root key on disk. Close both halves and retire the openssl path with them. The mount now generates its own root, once. The granting unit asks bao whether an issuer already exists (`bao list pki/issuers`) before calling `pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role and the grant without touching the anchor — a root that changed per boot would invalidate every certificate issued under it and every browser taught to trust it. The guard asks the store rather than looking for a marker file on this host's disk: a file is a claim about a mount that may have been restored from a snapshot or disabled and re-enabled underneath it. `swarm-services-issuer` stops being an inert policy. A fourth cert-auth role attaches it, following the shape the controller, the publisher and matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its CN — that credential is what opens the mount, so it cannot come out of it. `swarm-services-cert.service` logs in with that leaf, calls `pki/issue/swarm-services`, and writes the result to the path hive-tls.nix already wrote and the gateway already copies from. The sub-CA layer does not move; it stops existing. The role's `allowed_domains`, read from the same `swarm.serviceDomains` the SANs come from, enforces at issue time what the sub-CA encoded in x509 `nameConstraints`, and with the root inside the mount there is nothing left for an intermediate to be an intermediate of. Not a flag day: the issuing root is published beside the leaf as `swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the swarm root still sits. A leaf chaining to the old sub-CA and one issued by the store both verify against the same bundle, so hives can be rebuilt in any order. The same file is what an operator hands a browser — readable without a store login, which matters because every listener demands a client certificate. The eval-time warning about uncovered service names is gone rather than reworded. It fired on "this host does not hold the swarm root key", which was the reason a hive could end up serving its own leaf on a swarm-service name. Every hive now asks the store with its own identity, so that stopped being the thing that decides. Closes #4586
229 lines
12 KiB
Nix
229 lines
12 KiB
Nix
# Glue: give the secret store a PKI of its own, and point it at it.
|
||
#
|
||
# ONE PAIRING PER FILE — `glue-<consumer>-<what>.nix`. A single module holding
|
||
# every co-location default becomes the file nobody dares change, because a
|
||
# reader cannot tell which of its rules their deployment is subject to. Each
|
||
# of these should be deletable on its own, and deleting this one leaves a
|
||
# store that takes operator-provided certificates and nothing else.
|
||
#
|
||
# ⚠️ Why the PKI lives HERE and not in ./swarm-bao.nix: the store must have no
|
||
# opinion about where its identity comes from. Minting is an opinion — the
|
||
# most consequential one available — so it belongs to the glue that decides
|
||
# this deployment self-signs, not to the service that merely serves what it is
|
||
# handed. A deployment with a real internal CA drops this file and names its
|
||
# own paths; nothing in the store changes.
|
||
#
|
||
# ⚠️ Not the hive CA and not the swarm CA. The store will eventually
|
||
# distribute both, and an authority you must already hold a certificate from
|
||
# cannot be one the store hands out — reach the store to get the CA material,
|
||
# need a cert from that CA to reach the store. This CA signs a fixed, short
|
||
# list of leaves and distributes nothing, so it cannot enter that cycle.
|
||
#
|
||
# ⚠️ Files like this are the only place a `deploy.<foo>` value may derive from
|
||
# a `deploy.<bar>.enable`. Everywhere else that is forbidden. The exception
|
||
# earns itself: the derivation happens either way, and the alternative is
|
||
# having it spread through the service modules where it is invisible.
|
||
#
|
||
# Everything is `mkDefault`. An operator naming their own paths wins.
|
||
{
|
||
pkgs,
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
deployCfg = hyperhiveCfg.deploy;
|
||
cfg = hyperhiveCfg.swarm.bao;
|
||
|
||
# Host-side, outside the container's tree, for the same reason the raft data
|
||
# is: `nixos-container destroy` must not take it. Losing the CA key means
|
||
# re-issuing every client certificate in the swarm.
|
||
pkiDir = "/var/lib/swarm-bao-pki";
|
||
|
||
# What a reader calls itself to the store. The hive's name, because a bao
|
||
# cert-auth role matches on the CN — this is an interface, not a label.
|
||
# No fallback: `hiveName` is asserted set for every hyperhive host, which is
|
||
# the same condition this file's `config` is gated on. A fallback here reads
|
||
# as a second supported spelling and there is no such thing.
|
||
clientCn = hyperhiveCfg.hiveName;
|
||
|
||
# $1 dir $2 basename $3 CN $4 SAN or "" $5 EKU
|
||
signLeaf = pkgs.writeShellScript "swarm-bao-sign-leaf" ''
|
||
set -euo pipefail
|
||
d="$1"; base="$2"; cn="$3"; sans="$4"; eku="$5"
|
||
csr="$(mktemp "$d/$base.csr.XXXXXX")"
|
||
ext="$(mktemp "$d/$base.ext.XXXXXX")"
|
||
trap 'rm -f "$csr" "$ext"' EXIT
|
||
|
||
openssl req -newkey rsa:4096 -nodes -sha256 \
|
||
-keyout "$d/$base-key.pem" -out "$csr" -subj "/CN=$cn"
|
||
{
|
||
[ -n "$sans" ] && printf 'subjectAltName=%s\n' "$sans"
|
||
printf 'basicConstraints=critical,CA:FALSE\n'
|
||
printf 'keyUsage=critical,digitalSignature,keyEncipherment\n'
|
||
printf 'extendedKeyUsage=%s\n' "$eku"
|
||
} > "$ext"
|
||
openssl x509 -req -in "$csr" -CA "$d/ca.pem" -CAkey "$d/ca-key.pem" \
|
||
-CAcreateserial -days 3650 -sha256 -extfile "$ext" -out "$d/$base.pem"
|
||
chmod 0600 "$d/$base-key.pem"
|
||
chmod 0644 "$d/$base.pem"
|
||
'';
|
||
in
|
||
{
|
||
config = lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable) {
|
||
services.hyperhive.deploy.bao = {
|
||
serverCertFile = lib.mkDefault "${pkiDir}/server.pem";
|
||
serverKeyFile = lib.mkDefault "${pkiDir}/server-key.pem";
|
||
clientCaFile = lib.mkDefault "${pkiDir}/ca.pem";
|
||
|
||
# A reader on this host, which happens to be the host that mints. Only
|
||
# these three are what a reader elsewhere needs placed by hand; that they
|
||
# collapse to the same CA file here is a property of self-signing, not of
|
||
# the pairing.
|
||
clientCertFile = lib.mkDefault "${pkiDir}/client.pem";
|
||
clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem";
|
||
serverCaFile = lib.mkDefault "${pkiDir}/ca.pem";
|
||
|
||
# 🩸 Four readers that used to present `client.pem` above, each now
|
||
# pointed at a leaf of its own. The leaf is what bao sees, so this pairing
|
||
# is the whole of what turns "one principal with the union of four grants"
|
||
# into four principals with one grant each — see ./swarm-bao.nix's
|
||
# `perHiveReaders` block for the grants themselves.
|
||
#
|
||
# Defaulted here rather than in four `glue-<consumer>-bao-identity.nix`
|
||
# files: those exist where the consumer is a CONTAINER with a
|
||
# `deploy.<service>.*` namespace of its own to point at. These four are
|
||
# host units reading the store, which is the pairing this file already is.
|
||
matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem";
|
||
matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem";
|
||
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
|
||
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
|
||
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
|
||
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
|
||
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
|
||
otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem";
|
||
};
|
||
|
||
# Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates
|
||
# every client certificate already trusting it, so a rebuild that
|
||
# "refreshed" it would lock every reader in the swarm out at once — the
|
||
# same rule the store's TPM PIN unit follows, for a sharper reason.
|
||
# Declared beside the unit it names, not in the store's module: an entry
|
||
# exists only where the unit does, and this one is minted by glue that not
|
||
# every hive runs.
|
||
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-pki" ];
|
||
|
||
systemd.services.swarm-bao-pki = {
|
||
description = "mint the swarm secret store's own CA and leaves";
|
||
before = [ "swarm-bao-certs.service" ];
|
||
requiredBy = [ "swarm-bao-certs.service" ];
|
||
path = [
|
||
pkgs.openssl
|
||
pkgs.coreutils
|
||
];
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
};
|
||
script = ''
|
||
set -euo pipefail
|
||
install -d -m 0700 ${pkiDir}
|
||
|
||
if [ ! -s ${pkiDir}/ca.pem ]; then
|
||
openssl req -x509 -newkey rsa:4096 -nodes -sha256 -days 3650 \
|
||
-keyout ${pkiDir}/ca-key.pem -out ${pkiDir}/ca.pem \
|
||
-subj "/CN=swarm-bao-ca ${cfg.domain}" \
|
||
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
|
||
-addext "keyUsage=critical,keyCertSign,cRLSign"
|
||
chmod 0600 ${pkiDir}/ca-key.pem
|
||
chmod 0644 ${pkiDir}/ca.pem
|
||
fi
|
||
|
||
# The store's own identity, and the identity of a reader on this host.
|
||
# A reader elsewhere gets its leaf from this CA out of band — that is
|
||
# what makes the store reachable from another machine at all, and why
|
||
# the CA is a file rather than a service.
|
||
[ -s ${pkiDir}/server.pem ] || ${signLeaf} ${pkiDir} server \
|
||
${lib.escapeShellArg cfg.domain} ${lib.escapeShellArg "DNS:${cfg.domain}"} serverAuth
|
||
[ -s ${pkiDir}/client.pem ] || ${signLeaf} ${pkiDir} client \
|
||
${lib.escapeShellArg clientCn} "" clientAuth
|
||
|
||
# Minted whether or not a controller runs here, because the case it
|
||
# serves is the one where it does not: a controller elsewhere needs a
|
||
# leaf from this CA and has no way to sign one. Issuing it here turns
|
||
# "obtain a certificate out of band" into "copy this file".
|
||
#
|
||
# Its own CN rather than the reader's above: the controller's policy
|
||
# lets it create roles for every hive, and the reader's leaf carries
|
||
# this hive's name.
|
||
[ -s ${pkiDir}/controller.pem ] || ${signLeaf} ${pkiDir} controller \
|
||
${lib.escapeShellArg deployCfg.bao.controllerCommonName} "" clientAuth
|
||
|
||
# The secret publisher's, minted here for the reason the controller's
|
||
# line above gives — and it serves that case more often, not less: the
|
||
# publisher runs beside AUTHELIA, which is the one host guaranteed not
|
||
# to be this one whenever the store has a host of its own.
|
||
[ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \
|
||
${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth
|
||
|
||
# The matrix container's `swarm-matrix-ctl`. Minted unconditionally like the two
|
||
# above, and for the third variant of the same reason: the homeserver
|
||
# is a swarm singleton, so on every hive but the one running it this
|
||
# leaf is the file an operator copies rather than a file anything
|
||
# local reads.
|
||
#
|
||
# Its own CN, not the reader's: the reader's leaf carries this hive's
|
||
# name and its policy reads the whole store, while this principal may
|
||
# only write one path — which is the entire point of giving the
|
||
# container an identity instead of lending it the hive's.
|
||
[ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \
|
||
${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth
|
||
|
||
# 🩸 The four readers that used to present `client.pem` above. Each
|
||
# carries its own subject, which is the entire mechanism: bao matches a
|
||
# cert-auth role on the CN, so four units sharing one leaf were one
|
||
# principal holding the union of four grants, and the union included
|
||
# every agent credential in the swarm.
|
||
#
|
||
# ⚠️ The first two carry THIS hive's name in the subject, unlike the
|
||
# three service leaves above. Their grants name one hive's path, because
|
||
# a matrix token and a queue credential live under `swarm/hives/<name>/`
|
||
# and every hive runs a reader for its own; ./swarm-bao.nix writes one
|
||
# role per hive in the swarm directory to match. The two OIDC readers
|
||
# need no such segment — a client is registered once per swarm.
|
||
#
|
||
# Minted whether or not the consumer runs here, for the reason the three
|
||
# leaves above give: on a hive that does not run the store this is the
|
||
# file an operator copies, and a leaf that only appears where its
|
||
# consumer does is one nobody can copy from anywhere.
|
||
[ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \
|
||
${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth
|
||
|
||
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
|
||
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
|
||
|
||
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
|
||
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
|
||
|
||
[ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \
|
||
${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth
|
||
|
||
# The identity a hive presents to ask the store's `pki` mount for the
|
||
# swarm-services certificate its gateway serves. Minted here like the
|
||
# three above, and the reason is the sharpest of the four: this leaf
|
||
# is what OPENS the mint, so it cannot come out of it. A certificate
|
||
# authority that issues the credential admitting you to it is the
|
||
# cycle this file's header exists to keep out — which is exactly why
|
||
# the swarm-services root moved into the store and this one did not.
|
||
#
|
||
# Its own CN, not the reader's: the hive's own leaf reads every secret
|
||
# its policy names, while this principal may do one thing — `update`
|
||
# on `pki/issue/swarm-services` — and giving a renewal unit the wider
|
||
# credential would undo the separation the role was created for.
|
||
[ -s ${pkiDir}/services-issuer.pem ] || ${signLeaf} ${pkiDir} services-issuer \
|
||
${lib.escapeShellArg deployCfg.bao.servicesIssuerCommonName} "" clientAuth
|
||
'';
|
||
};
|
||
};
|
||
}
|