5704c0c583
Three defects in the store-issued path, each of which alone kept nginx from starting at all. The gateway's cert import `Requires=` this leaf, so a leaf that is never issued is not a name mismatch — it is an empty listener, and the swarm's own forge stopped answering on :443. `swarm-services-cert` declared `Before=hive-tls-ca` for the trust bundle's sake while also being `After=` the store's container, which is itself `After=hive-tls-ca`. systemd resolved the cycle the only way it can, by deleting the job, so the leaf went unissued on every activation. The edge is gone; the bundle converges the other way round, through the restart this unit already performed when the root it wrote was new. The `pki` mount was enabled without `-max-lease-ttl`, so bao clamped the 30-year root to the 768h default and then refused every issue call, because a leaf of the mount's own default length would outlive the CA signing it. The mount is tuned on every run, the role pins a 720h leaf, and a root that can no longer cover one is replaced rather than left to refuse forever. A hive tests its own copy of that certificate against the same threshold, so both ends reach a fresh leaf without signalling. `swarm-bao-pki` mints the services-issuer leaf that opens the mount, but only `swarm-bao-certs` required it. `RemainAfterExit` plus an already-active unit means an activation that ADDS a leaf mints nothing — which is how a host whose config named `services-issuer.pem` came to have no such file. A target wants it now, like every sibling granting unit.
239 lines
13 KiB
Nix
239 lines
13 KiB
Nix
# Glue: give the secret store a PKI of its own, and point it at it.
|
||
#
|
||
# ONE PAIRING PER FILE — `glue-<consumer>-<what>.nix`. A single module holding
|
||
# every co-location default becomes the file nobody dares change, because a
|
||
# reader cannot tell which of its rules their deployment is subject to. Each
|
||
# of these should be deletable on its own, and deleting this one leaves a
|
||
# store that takes operator-provided certificates and nothing else.
|
||
#
|
||
# ⚠️ Why the PKI lives HERE and not in ./swarm-bao.nix: the store must have no
|
||
# opinion about where its identity comes from. Minting is an opinion — the
|
||
# most consequential one available — so it belongs to the glue that decides
|
||
# this deployment self-signs, not to the service that merely serves what it is
|
||
# handed. A deployment with a real internal CA drops this file and names its
|
||
# own paths; nothing in the store changes.
|
||
#
|
||
# ⚠️ Not the hive CA and not the swarm CA. The store will eventually
|
||
# distribute both, and an authority you must already hold a certificate from
|
||
# cannot be one the store hands out — reach the store to get the CA material,
|
||
# need a cert from that CA to reach the store. This CA signs a fixed, short
|
||
# list of leaves and distributes nothing, so it cannot enter that cycle.
|
||
#
|
||
# ⚠️ Files like this are the only place a `deploy.<foo>` value may derive from
|
||
# a `deploy.<bar>.enable`. Everywhere else that is forbidden. The exception
|
||
# earns itself: the derivation happens either way, and the alternative is
|
||
# having it spread through the service modules where it is invisible.
|
||
#
|
||
# Everything is `mkDefault`. An operator naming their own paths wins.
|
||
{
|
||
pkgs,
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
deployCfg = hyperhiveCfg.deploy;
|
||
cfg = hyperhiveCfg.swarm.bao;
|
||
|
||
# Host-side, outside the container's tree, for the same reason the raft data
|
||
# is: `nixos-container destroy` must not take it. Losing the CA key means
|
||
# re-issuing every client certificate in the swarm.
|
||
pkiDir = "/var/lib/swarm-bao-pki";
|
||
|
||
# What a reader calls itself to the store. The hive's name, because a bao
|
||
# cert-auth role matches on the CN — this is an interface, not a label.
|
||
# No fallback: `hiveName` is asserted set for every hyperhive host, which is
|
||
# the same condition this file's `config` is gated on. A fallback here reads
|
||
# as a second supported spelling and there is no such thing.
|
||
clientCn = hyperhiveCfg.hiveName;
|
||
|
||
# $1 dir $2 basename $3 CN $4 SAN or "" $5 EKU
|
||
signLeaf = pkgs.writeShellScript "swarm-bao-sign-leaf" ''
|
||
set -euo pipefail
|
||
d="$1"; base="$2"; cn="$3"; sans="$4"; eku="$5"
|
||
csr="$(mktemp "$d/$base.csr.XXXXXX")"
|
||
ext="$(mktemp "$d/$base.ext.XXXXXX")"
|
||
trap 'rm -f "$csr" "$ext"' EXIT
|
||
|
||
openssl req -newkey rsa:4096 -nodes -sha256 \
|
||
-keyout "$d/$base-key.pem" -out "$csr" -subj "/CN=$cn"
|
||
{
|
||
[ -n "$sans" ] && printf 'subjectAltName=%s\n' "$sans"
|
||
printf 'basicConstraints=critical,CA:FALSE\n'
|
||
printf 'keyUsage=critical,digitalSignature,keyEncipherment\n'
|
||
printf 'extendedKeyUsage=%s\n' "$eku"
|
||
} > "$ext"
|
||
openssl x509 -req -in "$csr" -CA "$d/ca.pem" -CAkey "$d/ca-key.pem" \
|
||
-CAcreateserial -days 3650 -sha256 -extfile "$ext" -out "$d/$base.pem"
|
||
chmod 0600 "$d/$base-key.pem"
|
||
chmod 0644 "$d/$base.pem"
|
||
'';
|
||
in
|
||
{
|
||
config = lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable) {
|
||
services.hyperhive.deploy.bao = {
|
||
serverCertFile = lib.mkDefault "${pkiDir}/server.pem";
|
||
serverKeyFile = lib.mkDefault "${pkiDir}/server-key.pem";
|
||
clientCaFile = lib.mkDefault "${pkiDir}/ca.pem";
|
||
|
||
# A reader on this host, which happens to be the host that mints. Only
|
||
# these three are what a reader elsewhere needs placed by hand; that they
|
||
# collapse to the same CA file here is a property of self-signing, not of
|
||
# the pairing.
|
||
clientCertFile = lib.mkDefault "${pkiDir}/client.pem";
|
||
clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem";
|
||
serverCaFile = lib.mkDefault "${pkiDir}/ca.pem";
|
||
|
||
# 🩸 Four readers that used to present `client.pem` above, each now
|
||
# pointed at a leaf of its own. The leaf is what bao sees, so this pairing
|
||
# is the whole of what turns "one principal with the union of four grants"
|
||
# into four principals with one grant each — see ./swarm-bao.nix's
|
||
# `perHiveReaders` block for the grants themselves.
|
||
#
|
||
# Defaulted here rather than in four `glue-<consumer>-bao-identity.nix`
|
||
# files: those exist where the consumer is a CONTAINER with a
|
||
# `deploy.<service>.*` namespace of its own to point at. These four are
|
||
# host units reading the store, which is the pairing this file already is.
|
||
matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem";
|
||
matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem";
|
||
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
|
||
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
|
||
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
|
||
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
|
||
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
|
||
otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem";
|
||
};
|
||
|
||
# Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates
|
||
# every client certificate already trusting it, so a rebuild that
|
||
# "refreshed" it would lock every reader in the swarm out at once — the
|
||
# same rule the store's TPM PIN unit follows, for a sharper reason.
|
||
# Declared beside the unit it names, not in the store's module: an entry
|
||
# exists only where the unit does, and this one is minted by glue that not
|
||
# every hive runs.
|
||
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-pki" ];
|
||
|
||
systemd.services.swarm-bao-pki = {
|
||
description = "mint the swarm secret store's own CA and leaves";
|
||
before = [ "swarm-bao-certs.service" ];
|
||
requiredBy = [ "swarm-bao-certs.service" ];
|
||
# 🩸 A target wants this, not only `swarm-bao-certs`, and the reason is
|
||
# what happens when the LIST of leaves grows. `requiredBy` alone is
|
||
# satisfied by a unit that already ran: this one is `RemainAfterExit`,
|
||
# so an activation that adds a leaf here finds it active, pulls nothing,
|
||
# and the new leaf is never minted — while every sibling granting unit,
|
||
# each of which carries this line, was re-run by the same activation.
|
||
# That is how `services-issuer.pem` came to be missing on a host whose
|
||
# config named it, leaving `swarm-services-cert` to fail on an absent
|
||
# file until someone restarted this by hand.
|
||
wantedBy = [ "multi-user.target" ];
|
||
path = [
|
||
pkgs.openssl
|
||
pkgs.coreutils
|
||
];
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
};
|
||
script = ''
|
||
set -euo pipefail
|
||
install -d -m 0700 ${pkiDir}
|
||
|
||
if [ ! -s ${pkiDir}/ca.pem ]; then
|
||
openssl req -x509 -newkey rsa:4096 -nodes -sha256 -days 3650 \
|
||
-keyout ${pkiDir}/ca-key.pem -out ${pkiDir}/ca.pem \
|
||
-subj "/CN=swarm-bao-ca ${cfg.domain}" \
|
||
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
|
||
-addext "keyUsage=critical,keyCertSign,cRLSign"
|
||
chmod 0600 ${pkiDir}/ca-key.pem
|
||
chmod 0644 ${pkiDir}/ca.pem
|
||
fi
|
||
|
||
# The store's own identity, and the identity of a reader on this host.
|
||
# A reader elsewhere gets its leaf from this CA out of band — that is
|
||
# what makes the store reachable from another machine at all, and why
|
||
# the CA is a file rather than a service.
|
||
[ -s ${pkiDir}/server.pem ] || ${signLeaf} ${pkiDir} server \
|
||
${lib.escapeShellArg cfg.domain} ${lib.escapeShellArg "DNS:${cfg.domain}"} serverAuth
|
||
[ -s ${pkiDir}/client.pem ] || ${signLeaf} ${pkiDir} client \
|
||
${lib.escapeShellArg clientCn} "" clientAuth
|
||
|
||
# Minted whether or not a controller runs here, because the case it
|
||
# serves is the one where it does not: a controller elsewhere needs a
|
||
# leaf from this CA and has no way to sign one. Issuing it here turns
|
||
# "obtain a certificate out of band" into "copy this file".
|
||
#
|
||
# Its own CN rather than the reader's above: the controller's policy
|
||
# lets it create roles for every hive, and the reader's leaf carries
|
||
# this hive's name.
|
||
[ -s ${pkiDir}/controller.pem ] || ${signLeaf} ${pkiDir} controller \
|
||
${lib.escapeShellArg deployCfg.bao.controllerCommonName} "" clientAuth
|
||
|
||
# The secret publisher's, minted here for the reason the controller's
|
||
# line above gives — and it serves that case more often, not less: the
|
||
# publisher runs beside AUTHELIA, which is the one host guaranteed not
|
||
# to be this one whenever the store has a host of its own.
|
||
[ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \
|
||
${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth
|
||
|
||
# The matrix container's `swarm-matrix-ctl`. Minted unconditionally like the two
|
||
# above, and for the third variant of the same reason: the homeserver
|
||
# is a swarm singleton, so on every hive but the one running it this
|
||
# leaf is the file an operator copies rather than a file anything
|
||
# local reads.
|
||
#
|
||
# Its own CN, not the reader's: the reader's leaf carries this hive's
|
||
# name and its policy reads the whole store, while this principal may
|
||
# only write one path — which is the entire point of giving the
|
||
# container an identity instead of lending it the hive's.
|
||
[ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \
|
||
${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth
|
||
|
||
# 🩸 The four readers that used to present `client.pem` above. Each
|
||
# carries its own subject, which is the entire mechanism: bao matches a
|
||
# cert-auth role on the CN, so four units sharing one leaf were one
|
||
# principal holding the union of four grants, and the union included
|
||
# every agent credential in the swarm.
|
||
#
|
||
# ⚠️ The first two carry THIS hive's name in the subject, unlike the
|
||
# three service leaves above. Their grants name one hive's path, because
|
||
# a matrix token and a queue credential live under `swarm/hives/<name>/`
|
||
# and every hive runs a reader for its own; ./swarm-bao.nix writes one
|
||
# role per hive in the swarm directory to match. The two OIDC readers
|
||
# need no such segment — a client is registered once per swarm.
|
||
#
|
||
# Minted whether or not the consumer runs here, for the reason the three
|
||
# leaves above give: on a hive that does not run the store this is the
|
||
# file an operator copies, and a leaf that only appears where its
|
||
# consumer does is one nobody can copy from anywhere.
|
||
[ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \
|
||
${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth
|
||
|
||
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
|
||
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
|
||
|
||
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
|
||
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
|
||
|
||
[ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \
|
||
${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth
|
||
|
||
# The identity a hive presents to ask the store's `pki` mount for the
|
||
# swarm-services certificate its gateway serves. Minted here like the
|
||
# three above, and the reason is the sharpest of the four: this leaf
|
||
# is what OPENS the mint, so it cannot come out of it. A certificate
|
||
# authority that issues the credential admitting you to it is the
|
||
# cycle this file's header exists to keep out — which is exactly why
|
||
# the swarm-services root moved into the store and this one did not.
|
||
#
|
||
# Its own CN, not the reader's: the hive's own leaf reads every secret
|
||
# its policy names, while this principal may do one thing — `update`
|
||
# on `pki/issue/swarm-services` — and giving a renewal unit the wider
|
||
# credential would undo the separation the role was created for.
|
||
[ -s ${pkiDir}/services-issuer.pem ] || ${signLeaf} ${pkiDir} services-issuer \
|
||
${lib.escapeShellArg deployCfg.bao.servicesIssuerCommonName} "" clientAuth
|
||
'';
|
||
};
|
||
};
|
||
}
|