hive CA: delete ca.pem, ca-previous.pem and ca-key.pem, the bundle is the store's root
The gateway leaf comes from the store's pki mount, so a hive CA's certificates anchor nothing and its key is a secret outside the store. - hive-tls-ca writes trust-bundle.pem from swarm-services-root.pem alone; the ca.pem and ca-previous.pem slots are gone. - Every host deletes ca.pem, ca-previous.pem and ca-key.pem in deploy.hive-controller.tls.stateDir with tmpfiles `r` rules that follow the option. The removal shim, now host-ca-removed.nix, holds these beside the /var/lib/swarm-ca rule and the swarm.ca.* removals. - deploy.bao.clientCaFile, deploy.bao.serverCaFile and deploy.swarm-controller.hiveClientCaFile naming /var/lib/swarm-ca or one of the deleted hive CA files fail evaluation, naming the store's mTLS CA (/var/lib/swarm-bao-pki/ca.pem on the store's host) as the replacement. module-eval hive-tls covers the single-anchor bundle, the deletion (incl. a relocated stateDir) and the assertion on all three options. Refs #4588
This commit is contained in:
atlas
mara
parent
93f1a76ea7
commit
4e5a642928
10 changed files with 170 additions and 122 deletions
|
|
@ -64,7 +64,7 @@
|
|||
./swarm-victoriametrics.nix
|
||||
./swarm-wireguard.nix
|
||||
./swarm.nix
|
||||
./swarm-ca-removed.nix
|
||||
./host-ca-removed.nix
|
||||
./swarm-peers-removed.nix
|
||||
./swarm-required-services.nix
|
||||
];
|
||||
|
|
|
|||
|
|
@ -249,9 +249,6 @@ in
|
|||
# `trust-bundle.pem`. Leaf rotation never re-breaks them, and nothing here
|
||||
# generates or signs with a CA key. See
|
||||
# `docs/networking/gateway.md` ("Self-signed TLS") and docs/swarm/ca.md.
|
||||
#
|
||||
# The trust bundle also carries a hive CA's public `ca.pem` where one
|
||||
# exists, so a leaf it signed verifies until it is replaced.
|
||||
|
||||
options.services.hyperhive.deploy.hive-controller.tls = {
|
||||
stateDir = lib.mkOption {
|
||||
|
|
@ -261,10 +258,7 @@ in
|
|||
Host directory holding the gateway leaf and trust bundle for the
|
||||
self-signed gateway mode: `gateway.pem` / `gateway-key.pem` (the
|
||||
leaf the store issues and nginx serves) and `trust-bundle.pem` (the
|
||||
anchors agents and federation peers trust). `ca.pem` and
|
||||
`ca-previous.pem`, where present, are a hive CA's public
|
||||
certificates and stay in the bundle. A `ca-key.pem` here is neither
|
||||
read nor removed.
|
||||
anchor agents and federation peers trust, the store's `pki` root).
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -363,9 +357,8 @@ in
|
|||
# resolves a loop by dropping a job.
|
||||
#
|
||||
# What that costs: the trust bundle written at the end of this script
|
||||
# anchors on every root a consumer must hold, and the services root is
|
||||
# one of them, so on a first boot the `cat` below runs before that file
|
||||
# exists and the bundle is one restart short of complete. The `if [ -s
|
||||
# is the services root, so on a first boot it is written before that
|
||||
# file exists and is empty until a restart. The `if [ -s
|
||||
# "$svcroot" ]` guard is what makes that state legal rather than fatal,
|
||||
# and `swarm-services-cert` restarts this unit once the root it wrote
|
||||
# is new — so the bundle converges instead of being a boot behind.
|
||||
|
|
@ -381,42 +374,21 @@ in
|
|||
d=${lib.escapeShellArg cfg.stateDir}
|
||||
install -d -m 0755 "$d"
|
||||
|
||||
ca="$d/ca.pem"
|
||||
prev="$d/ca-previous.pem"
|
||||
svcroot="$d/swarm-services-root.pem"
|
||||
|
||||
# --- Trust bundle: what a consumer must TRUST. Why it is a set of
|
||||
# anchors rather than one file is in docs/swarm/ca.md. Three
|
||||
# constraints the code cannot state:
|
||||
# --- Trust bundle: what a consumer must TRUST, the store's `pki`
|
||||
# root, which both the gateway leaf and the swarm-services leaf
|
||||
# chain to (docs/swarm/ca.md).
|
||||
#
|
||||
# Written IN PLACE, never renamed into position — containers
|
||||
# bind-mount this file and a bind mount follows the inode, so a
|
||||
# rename leaves every consumer holding the old one.
|
||||
#
|
||||
# Dropping `ca.pem` and `ca-previous.pem` is deliberately NOT done
|
||||
# here: "long enough" is a deployment fact, not a unit's call.
|
||||
#
|
||||
# Built as an array with `if` guards, not
|
||||
# `[ -s f ] && anchors+=(f)`: under `set -e` an && list whose test
|
||||
# fails IS a failing command and kills the unit — on exactly the
|
||||
# hive where the optional file is legitimately absent.
|
||||
#
|
||||
# `swarm-services-root.pem` is the store's `pki` root, which both
|
||||
# the gateway leaf and the swarm-services leaf chain to. `ca.pem`
|
||||
# and `ca-previous.pem` are a hive CA's public certificates, kept
|
||||
# so a leaf they signed verifies until replaced.
|
||||
bundle="$d/trust-bundle.pem"
|
||||
anchors=()
|
||||
if [ -s "$ca" ]; then anchors+=("$ca"); fi
|
||||
if [ -s "$prev" ]; then anchors+=("$prev"); fi
|
||||
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
|
||||
# `cat` with no arguments reads stdin, so no anchor at all is an
|
||||
# empty file until `swarm-services-cert` writes the root.
|
||||
if [ "''${#anchors[@]}" = 0 ]; then
|
||||
if [ -s "$svcroot" ]; then
|
||||
cat "$svcroot" > "$bundle"
|
||||
else
|
||||
echo "no trust anchor on disk yet — $bundle is empty until the store's root is fetched" >&2
|
||||
: > "$bundle"
|
||||
else
|
||||
cat "''${anchors[@]}" > "$bundle"
|
||||
fi
|
||||
chmod 0644 "$bundle"
|
||||
'';
|
||||
|
|
@ -942,9 +914,8 @@ in
|
|||
# this file (read-only, public certs ONLY — never a key) into each
|
||||
# agent container so agents + their tools can trust the gateway's
|
||||
# self-signed leaf, and the meta flake wires the per-agent trust
|
||||
# bundle. It is the ANCHOR bundle rather than `ca.pem` for the reason
|
||||
# spelled out where the bundle is written above; no key path is ever
|
||||
# exposed (an agent that could read one could mint trusted certs).
|
||||
# bundle. No key path is ever exposed (an agent that could read one
|
||||
# could mint trusted certs).
|
||||
#
|
||||
# ⚠️ Every line here that names another daemon's unit is gated on that
|
||||
# daemon's own enable: the gateway also runs on hosts with no hive and no
|
||||
|
|
|
|||
65
nix/host-modules/host-ca-removed.nix
Normal file
65
nix/host-modules/host-ca-removed.nix
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
# Removal shim for the CAs a host held outside the secret store: the
|
||||
# swarm root under `/var/lib/swarm-ca` (and its `services.hyperhive.swarm.ca.*`
|
||||
# options), and a hive CA's `ca.pem`, `ca-previous.pem` and `ca-key.pem` in
|
||||
# `deploy.hive-controller.tls.stateDir`.
|
||||
#
|
||||
# The secret store's `pki` root is the only trust anchor: it issues every
|
||||
# hive's gateway leaf and the swarm's service names, and its key never leaves
|
||||
# the store. Nothing generates, reads or bundles either CA, so a definition of
|
||||
# a removed option fails evaluation with a message, a CA-file option naming
|
||||
# one of the deleted paths fails it too, and every host deletes the files.
|
||||
#
|
||||
# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to
|
||||
# remove when the deprecation window closes.
|
||||
{ lib, config, ... }:
|
||||
let
|
||||
deployCfg = config.services.hyperhive.deploy;
|
||||
|
||||
swarmCaDir = "/var/lib/swarm-ca";
|
||||
hiveCaFiles = map (f: "${deployCfg.hive-controller.tls.stateDir}/${f}") [
|
||||
"ca.pem"
|
||||
"ca-previous.pem"
|
||||
"ca-key.pem"
|
||||
];
|
||||
deleted = p: p == swarmCaDir || lib.hasPrefix "${swarmCaDir}/" p || lib.elem p hiveCaFiles;
|
||||
|
||||
# Every option that names a CA the store's mTLS checks against. One
|
||||
# pointing at a deleted file fails the unit that reads it at start.
|
||||
caFileOptions = {
|
||||
"services.hyperhive.deploy.bao.clientCaFile" = deployCfg.bao.clientCaFile;
|
||||
"services.hyperhive.deploy.bao.serverCaFile" = deployCfg.bao.serverCaFile;
|
||||
"services.hyperhive.deploy.swarm-controller.hiveClientCaFile" =
|
||||
deployCfg.swarm-controller.hiveClientCaFile;
|
||||
};
|
||||
|
||||
removed =
|
||||
name:
|
||||
lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] ''
|
||||
The host-held swarm root CA is gone: the secret store's pki root is the
|
||||
only trust anchor, and nothing generates or reads a root under
|
||||
${swarmCaDir}. Remove this definition. See docs/swarm/ca.md.
|
||||
'';
|
||||
in
|
||||
{
|
||||
imports = map removed [
|
||||
"autoConfigure"
|
||||
"stateDir"
|
||||
"validityDays"
|
||||
];
|
||||
|
||||
assertions = lib.mapAttrsToList (option: path: {
|
||||
assertion = path == null || !(deleted path);
|
||||
message = ''
|
||||
${option} is ${toString path}, a file every host deletes along with
|
||||
the host-held CA it belonged to. Point it at the store's mTLS CA:
|
||||
/var/lib/swarm-bao-pki/ca.pem on the host that runs the store, the copy
|
||||
placed by hand on any other host. See docs/swarm/ca.md.
|
||||
'';
|
||||
}) caFileOptions;
|
||||
|
||||
# Both CAs' private keys are secrets outside the store, and the
|
||||
# certificates beside them are anchors nothing issues under. The swarm
|
||||
# root at the option's old default path only: a relocated `stateDir` is
|
||||
# the operator's to delete.
|
||||
systemd.tmpfiles.rules = [ "R ${swarmCaDir}" ] ++ map (f: "r ${f}") hiveCaFiles;
|
||||
}
|
||||
|
|
@ -33,11 +33,8 @@ let
|
|||
# `gateway.useSelfSigned` is the single source of truth for the
|
||||
# self-signed condition — no duplicated derivation.
|
||||
useSelfSigned = gatewayCfg.useSelfSigned;
|
||||
# The bundle, not `ca.pem`: the hive CA is an intermediate under the
|
||||
# swarm root, and openssl (which is what both consumers below sit on —
|
||||
# Node's `NODE_EXTRA_CA_CERTS`, Go's `SSL_CERT_FILE`) will not end a
|
||||
# chain at a trusted cert that isn't self-signed. `hive-tls.nix` writes
|
||||
# the bundle next to the CA and explains the split.
|
||||
# The bundle `hive-tls.nix` writes: the store's `pki` root, which the
|
||||
# gateway's leaf chains to.
|
||||
caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem";
|
||||
caContainerPath = "/run/hive-ca/trust-bundle.pem";
|
||||
|
||||
|
|
|
|||
|
|
@ -127,7 +127,7 @@ let
|
|||
# Where the leaf lands for openbao to read. `tlsDir` is bind-mounted at the
|
||||
# same path on both sides, so the delivery below needs no second mount, and
|
||||
# nothing has to bind `deploy.hive-controller.tls.stateDir`, which holds the
|
||||
# hive CA's private key.
|
||||
# gateway's private keys.
|
||||
#
|
||||
# The certificate and the client CA are public material and are read straight
|
||||
# off the mount; only the key takes the credential path above.
|
||||
|
|
|
|||
|
|
@ -1,33 +0,0 @@
|
|||
# Removal shim for `services.hyperhive.swarm.ca.*` and the host-held swarm
|
||||
# root it generated.
|
||||
#
|
||||
# The secret store's `pki` root is the only trust anchor: it issues every
|
||||
# hive's gateway leaf and the swarm's service names, and its key never leaves
|
||||
# the store. Nothing generates, reads or bundles a root under
|
||||
# `/var/lib/swarm-ca`, so a definition of these options fails evaluation with
|
||||
# a message, and the directory is deleted on every host.
|
||||
#
|
||||
# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to
|
||||
# remove when the deprecation window closes.
|
||||
{ lib, ... }:
|
||||
let
|
||||
removed =
|
||||
name:
|
||||
lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] ''
|
||||
The host-held swarm root CA is gone: the secret store's pki root is the
|
||||
only trust anchor, and nothing generates or reads a root under
|
||||
/var/lib/swarm-ca. Remove this definition. See docs/swarm/ca.md.
|
||||
'';
|
||||
in
|
||||
{
|
||||
imports = map removed [
|
||||
"autoConfigure"
|
||||
"stateDir"
|
||||
"validityDays"
|
||||
];
|
||||
|
||||
# The root's private key is a secret outside the store, and the
|
||||
# certificate beside it is an anchor nothing issues under. The option's
|
||||
# default path only: a relocated `stateDir` is the operator's to delete.
|
||||
systemd.tmpfiles.rules = [ "R /var/lib/swarm-ca" ];
|
||||
}
|
||||
Loading…
Reference in a new issue