From 4e5a64292833655b1c9839b77d043a21ac1ce858 Mon Sep 17 00:00:00 2001 From: atlas Date: Mon, 5 Oct 2026 19:54:55 +0200 Subject: [PATCH] hive CA: delete ca.pem, ca-previous.pem and ca-key.pem, the bundle is the store's root The gateway leaf comes from the store's pki mount, so a hive CA's certificates anchor nothing and its key is a secret outside the store. - hive-tls-ca writes trust-bundle.pem from swarm-services-root.pem alone; the ca.pem and ca-previous.pem slots are gone. - Every host deletes ca.pem, ca-previous.pem and ca-key.pem in deploy.hive-controller.tls.stateDir with tmpfiles `r` rules that follow the option. The removal shim, now host-ca-removed.nix, holds these beside the /var/lib/swarm-ca rule and the swarm.ca.* removals. - deploy.bao.clientCaFile, deploy.bao.serverCaFile and deploy.swarm-controller.hiveClientCaFile naming /var/lib/swarm-ca or one of the deleted hive CA files fail evaluation, naming the store's mTLS CA (/var/lib/swarm-bao-pki/ca.pem on the store's host) as the replacement. module-eval hive-tls covers the single-anchor bundle, the deletion (incl. a relocated stateDir) and the assertion on all three options. Refs #4588 --- docs/networking/gateway.md | 2 +- docs/swarm/README.md | 2 +- docs/swarm/ca.md | 33 ++++----- nix/host-modules/default.nix | 2 +- nix/host-modules/hive-tls.nix | 51 +++----------- nix/host-modules/host-ca-removed.nix | 65 ++++++++++++++++++ nix/host-modules/lib/hive-ca-trust.nix | 7 +- nix/host-modules/swarm-bao.nix | 2 +- nix/host-modules/swarm-ca-removed.nix | 33 --------- nix/module-eval/hive-tls.nix | 95 ++++++++++++++++++++------ 10 files changed, 170 insertions(+), 122 deletions(-) create mode 100644 nix/host-modules/host-ca-removed.nix delete mode 100644 nix/host-modules/swarm-ca-removed.nix diff --git a/docs/networking/gateway.md b/docs/networking/gateway.md index 18204a94..c87c0320 100644 --- a/docs/networking/gateway.md +++ b/docs/networking/gateway.md @@ -80,7 +80,7 @@ The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. ` **Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them. -**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root, plus a hive CA's public `ca.pem` where one exists ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle. +**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle. **Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https:///.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap. diff --git a/docs/swarm/README.md b/docs/swarm/README.md index b0a76b62..58e6933d 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -34,7 +34,7 @@ grafana, the controller, authelia, …) → contract → [`credentials.md`](credentials.md) · how the store comes up, who writes its grants and how it unseals → [`bao.md`](bao.md) - **swarm CA** — the store root every hive's gateway leaf chains to, and what to - hand a peer (`trust-bundle.pem`, never `ca.pem`). → [`ca.md`](ca.md) + hand a peer. → [`ca.md`](ca.md) - **snapshot store** — the swarm's one `btrfs receive` endpoint, `swarm.snapshotStore.{address,port}`. → [`snapshot-store.md`](../networking/snapshot-store.md) diff --git a/docs/swarm/ca.md b/docs/swarm/ca.md index 7b153d6f..6259e1b7 100644 --- a/docs/swarm/ca.md +++ b/docs/swarm/ca.md @@ -75,21 +75,10 @@ issues it. ## The trust bundle `hive-tls-ca.service` writes `/trust-bundle.pem`, -the file every consumer anchors on. It holds whichever of these exist: - -- `swarm-services-root.pem`, the store's `pki` root, which the gateway leaf - and the swarm-services leaf both chain to. -- `ca.pem` and `ca-previous.pem`: a hive CA's public certificates. Nothing - issues under that CA; they stay in the bundle so a leaf it signed verifies - until `hive-tls-resign` replaces it. Removing them is a deliberate later - step. - -No other anchor goes in: the store's `pki` root is the only root the swarm -has. - -No hive generates a CA or signs with a CA key. A -`/ca-key.pem` stays on disk, unread: -removing it comes with dropping `ca.pem` from the bundle. +the file every consumer anchors on. It holds one certificate, a copy of +`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and +the swarm-services leaf both chain to. It's the only root the swarm has. Until +the first fetch of that root lands, the bundle is empty. ⚠️ **A consumer reads its trust when it starts.** An agent or service container that last started before the store's root reached the bundle @@ -164,9 +153,17 @@ role's `allowed_domains`, enforced by the store at issue time: a hive's identity that leaks can get certificates for that hive's names and no others. -No hive holds a root CA or its key, and nothing under `/var/lib/swarm-ca` -exists: every host deletes that directory at boot and on every deploy. The -swarm distributes only certificates. +No host holds a CA or its key, except the store's mTLS CA +(`glue-bao-tls.nix`), which signs only the store's server certificate and the +certificates that log in to it. At boot and on every deploy, every host +deletes `/var/lib/swarm-ca` and the `ca.pem`, `ca-previous.pem` and +`ca-key.pem` in ``. The swarm distributes +only certificates. + +A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or +`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails +evaluation. Those options name the store's mTLS CA, +`/var/lib/swarm-bao-pki/ca.pem` on the host that runs the store. ## What to hand a peer diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 4dfac220..558ac8b9 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -64,7 +64,7 @@ ./swarm-victoriametrics.nix ./swarm-wireguard.nix ./swarm.nix - ./swarm-ca-removed.nix + ./host-ca-removed.nix ./swarm-peers-removed.nix ./swarm-required-services.nix ]; diff --git a/nix/host-modules/hive-tls.nix b/nix/host-modules/hive-tls.nix index b31609d6..14e6f6b9 100644 --- a/nix/host-modules/hive-tls.nix +++ b/nix/host-modules/hive-tls.nix @@ -249,9 +249,6 @@ in # `trust-bundle.pem`. Leaf rotation never re-breaks them, and nothing here # generates or signs with a CA key. See # `docs/networking/gateway.md` ("Self-signed TLS") and docs/swarm/ca.md. - # - # The trust bundle also carries a hive CA's public `ca.pem` where one - # exists, so a leaf it signed verifies until it is replaced. options.services.hyperhive.deploy.hive-controller.tls = { stateDir = lib.mkOption { @@ -261,10 +258,7 @@ in Host directory holding the gateway leaf and trust bundle for the self-signed gateway mode: `gateway.pem` / `gateway-key.pem` (the leaf the store issues and nginx serves) and `trust-bundle.pem` (the - anchors agents and federation peers trust). `ca.pem` and - `ca-previous.pem`, where present, are a hive CA's public - certificates and stay in the bundle. A `ca-key.pem` here is neither - read nor removed. + anchor agents and federation peers trust, the store's `pki` root). ''; }; @@ -363,9 +357,8 @@ in # resolves a loop by dropping a job. # # What that costs: the trust bundle written at the end of this script - # anchors on every root a consumer must hold, and the services root is - # one of them, so on a first boot the `cat` below runs before that file - # exists and the bundle is one restart short of complete. The `if [ -s + # is the services root, so on a first boot it is written before that + # file exists and is empty until a restart. The `if [ -s # "$svcroot" ]` guard is what makes that state legal rather than fatal, # and `swarm-services-cert` restarts this unit once the root it wrote # is new — so the bundle converges instead of being a boot behind. @@ -381,42 +374,21 @@ in d=${lib.escapeShellArg cfg.stateDir} install -d -m 0755 "$d" - ca="$d/ca.pem" - prev="$d/ca-previous.pem" svcroot="$d/swarm-services-root.pem" - # --- Trust bundle: what a consumer must TRUST. Why it is a set of - # anchors rather than one file is in docs/swarm/ca.md. Three - # constraints the code cannot state: + # --- Trust bundle: what a consumer must TRUST, the store's `pki` + # root, which both the gateway leaf and the swarm-services leaf + # chain to (docs/swarm/ca.md). # # Written IN PLACE, never renamed into position — containers # bind-mount this file and a bind mount follows the inode, so a # rename leaves every consumer holding the old one. - # - # Dropping `ca.pem` and `ca-previous.pem` is deliberately NOT done - # here: "long enough" is a deployment fact, not a unit's call. - # - # Built as an array with `if` guards, not - # `[ -s f ] && anchors+=(f)`: under `set -e` an && list whose test - # fails IS a failing command and kills the unit — on exactly the - # hive where the optional file is legitimately absent. - # - # `swarm-services-root.pem` is the store's `pki` root, which both - # the gateway leaf and the swarm-services leaf chain to. `ca.pem` - # and `ca-previous.pem` are a hive CA's public certificates, kept - # so a leaf they signed verifies until replaced. bundle="$d/trust-bundle.pem" - anchors=() - if [ -s "$ca" ]; then anchors+=("$ca"); fi - if [ -s "$prev" ]; then anchors+=("$prev"); fi - if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi - # `cat` with no arguments reads stdin, so no anchor at all is an - # empty file until `swarm-services-cert` writes the root. - if [ "''${#anchors[@]}" = 0 ]; then + if [ -s "$svcroot" ]; then + cat "$svcroot" > "$bundle" + else echo "no trust anchor on disk yet — $bundle is empty until the store's root is fetched" >&2 : > "$bundle" - else - cat "''${anchors[@]}" > "$bundle" fi chmod 0644 "$bundle" ''; @@ -942,9 +914,8 @@ in # this file (read-only, public certs ONLY — never a key) into each # agent container so agents + their tools can trust the gateway's # self-signed leaf, and the meta flake wires the per-agent trust - # bundle. It is the ANCHOR bundle rather than `ca.pem` for the reason - # spelled out where the bundle is written above; no key path is ever - # exposed (an agent that could read one could mint trusted certs). + # bundle. No key path is ever exposed (an agent that could read one + # could mint trusted certs). # # ⚠️ Every line here that names another daemon's unit is gated on that # daemon's own enable: the gateway also runs on hosts with no hive and no diff --git a/nix/host-modules/host-ca-removed.nix b/nix/host-modules/host-ca-removed.nix new file mode 100644 index 00000000..390ceddb --- /dev/null +++ b/nix/host-modules/host-ca-removed.nix @@ -0,0 +1,65 @@ +# Removal shim for the CAs a host held outside the secret store: the +# swarm root under `/var/lib/swarm-ca` (and its `services.hyperhive.swarm.ca.*` +# options), and a hive CA's `ca.pem`, `ca-previous.pem` and `ca-key.pem` in +# `deploy.hive-controller.tls.stateDir`. +# +# The secret store's `pki` root is the only trust anchor: it issues every +# hive's gateway leaf and the swarm's service names, and its key never leaves +# the store. Nothing generates, reads or bundles either CA, so a definition of +# a removed option fails evaluation with a message, a CA-file option naming +# one of the deleted paths fails it too, and every host deletes the files. +# +# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to +# remove when the deprecation window closes. +{ lib, config, ... }: +let + deployCfg = config.services.hyperhive.deploy; + + swarmCaDir = "/var/lib/swarm-ca"; + hiveCaFiles = map (f: "${deployCfg.hive-controller.tls.stateDir}/${f}") [ + "ca.pem" + "ca-previous.pem" + "ca-key.pem" + ]; + deleted = p: p == swarmCaDir || lib.hasPrefix "${swarmCaDir}/" p || lib.elem p hiveCaFiles; + + # Every option that names a CA the store's mTLS checks against. One + # pointing at a deleted file fails the unit that reads it at start. + caFileOptions = { + "services.hyperhive.deploy.bao.clientCaFile" = deployCfg.bao.clientCaFile; + "services.hyperhive.deploy.bao.serverCaFile" = deployCfg.bao.serverCaFile; + "services.hyperhive.deploy.swarm-controller.hiveClientCaFile" = + deployCfg.swarm-controller.hiveClientCaFile; + }; + + removed = + name: + lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] '' + The host-held swarm root CA is gone: the secret store's pki root is the + only trust anchor, and nothing generates or reads a root under + ${swarmCaDir}. Remove this definition. See docs/swarm/ca.md. + ''; +in +{ + imports = map removed [ + "autoConfigure" + "stateDir" + "validityDays" + ]; + + assertions = lib.mapAttrsToList (option: path: { + assertion = path == null || !(deleted path); + message = '' + ${option} is ${toString path}, a file every host deletes along with + the host-held CA it belonged to. Point it at the store's mTLS CA: + /var/lib/swarm-bao-pki/ca.pem on the host that runs the store, the copy + placed by hand on any other host. See docs/swarm/ca.md. + ''; + }) caFileOptions; + + # Both CAs' private keys are secrets outside the store, and the + # certificates beside them are anchors nothing issues under. The swarm + # root at the option's old default path only: a relocated `stateDir` is + # the operator's to delete. + systemd.tmpfiles.rules = [ "R ${swarmCaDir}" ] ++ map (f: "r ${f}") hiveCaFiles; +} diff --git a/nix/host-modules/lib/hive-ca-trust.nix b/nix/host-modules/lib/hive-ca-trust.nix index c955fa45..b4c3f33c 100644 --- a/nix/host-modules/lib/hive-ca-trust.nix +++ b/nix/host-modules/lib/hive-ca-trust.nix @@ -33,11 +33,8 @@ let # `gateway.useSelfSigned` is the single source of truth for the # self-signed condition — no duplicated derivation. useSelfSigned = gatewayCfg.useSelfSigned; - # The bundle, not `ca.pem`: the hive CA is an intermediate under the - # swarm root, and openssl (which is what both consumers below sit on — - # Node's `NODE_EXTRA_CA_CERTS`, Go's `SSL_CERT_FILE`) will not end a - # chain at a trusted cert that isn't self-signed. `hive-tls.nix` writes - # the bundle next to the CA and explains the split. + # The bundle `hive-tls.nix` writes: the store's `pki` root, which the + # gateway's leaf chains to. caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem"; caContainerPath = "/run/hive-ca/trust-bundle.pem"; diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 05d21c90..c099820e 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -127,7 +127,7 @@ let # Where the leaf lands for openbao to read. `tlsDir` is bind-mounted at the # same path on both sides, so the delivery below needs no second mount, and # nothing has to bind `deploy.hive-controller.tls.stateDir`, which holds the - # hive CA's private key. + # gateway's private keys. # # The certificate and the client CA are public material and are read straight # off the mount; only the key takes the credential path above. diff --git a/nix/host-modules/swarm-ca-removed.nix b/nix/host-modules/swarm-ca-removed.nix deleted file mode 100644 index b09e65c2..00000000 --- a/nix/host-modules/swarm-ca-removed.nix +++ /dev/null @@ -1,33 +0,0 @@ -# Removal shim for `services.hyperhive.swarm.ca.*` and the host-held swarm -# root it generated. -# -# The secret store's `pki` root is the only trust anchor: it issues every -# hive's gateway leaf and the swarm's service names, and its key never leaves -# the store. Nothing generates, reads or bundles a root under -# `/var/lib/swarm-ca`, so a definition of these options fails evaluation with -# a message, and the directory is deleted on every host. -# -# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to -# remove when the deprecation window closes. -{ lib, ... }: -let - removed = - name: - lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] '' - The host-held swarm root CA is gone: the secret store's pki root is the - only trust anchor, and nothing generates or reads a root under - /var/lib/swarm-ca. Remove this definition. See docs/swarm/ca.md. - ''; -in -{ - imports = map removed [ - "autoConfigure" - "stateDir" - "validityDays" - ]; - - # The root's private key is a secret outside the store, and the - # certificate beside it is an anchor nothing issues under. The option's - # default path only: a relocated `stateDir` is the operator's to delete. - systemd.tmpfiles.rules = [ "R /var/lib/swarm-ca" ]; -} diff --git a/nix/module-eval/hive-tls.nix b/nix/module-eval/hive-tls.nix index cf544b9b..e4e08aa5 100644 --- a/nix/module-eval/hive-tls.nix +++ b/nix/module-eval/hive-tls.nix @@ -155,7 +155,7 @@ let && lib.hasInfix ''"$BAO_ADDR/v1/pki/ca/pem"'' noSvcFetch && lib.hasInfix ''mv -f "$svcroot.new" "$svcroot"'' noSvcFetch && lib.hasInfix ''svcroot="$d/swarm-services-root.pem"'' noService.systemd.services.hive-tls-ca.script - && lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' noService.systemd.services.hive-tls-ca.script; + && lib.hasInfix ''cat "$svcroot" > "$bundle"'' noService.systemd.services.hive-tls-ca.script; } { # The hive's own identity, not its reader identity: the reader's policy @@ -258,18 +258,12 @@ let && mentions noService "swarm-services-root.pem"; } { - # Three slots and no fourth: a root slot beside them is an anchor - # nothing issues under. - name = "the trust bundle anchors on the store's root and a hive CA's public certificates only"; + # One anchor: any other slot is a root nothing issues under. + name = "the trust bundle is the store's root alone"; ok = - let - slots = lib.filter (l: lib.hasPrefix "if [ -s " l && lib.hasInfix "anchors+=(" l) ( - lib.splitString "\n" caUnit.script - ); - in - lib.length slots == 3 - && lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' caUnit.script - && !(lib.hasInfix ''anchors+=("$root")'' caUnit.script); + lib.hasInfix ''cat "$svcroot" > "$bundle"'' caUnit.script + && !(lib.hasInfix "anchors+=(" caUnit.script) + && !(lib.hasInfix "ca.pem" caUnit.script); } { # The control is the all-local host, which defines none of them. @@ -322,17 +316,74 @@ let && !(lib.hasInfix refusal (imp noService)); } { - # Nothing reads or removes a `ca-key.pem`, so a rollback finds the - # key and does not mint a new anchor. The control is the bundle - # reading `ca.pem` in the same script. - name = "a hive CA key is neither read nor removed, and its public certificate stays in the trust bundle"; + # The control is the same scan finding the store root's file, and a + # relocated state dir, whose rules follow it. + name = "no hive TLS unit reads a hive CA's files, and every host deletes them"; ok = - !(lib.hasInfix "ca-key.pem" caUnit.script) - && !(lib.hasInfix "ca-key.pem" leafUnit.script) - && lib.hasInfix ''ca="$d/ca.pem"'' caUnit.script - && lib.hasInfix ''if [ -s "$ca" ]; then anchors+=("$ca"); fi'' caUnit.script - && lib.hasInfix ''if [ -s "$prev" ]; then anchors+=("$prev"); fi'' caUnit.script - && lib.hasInfix ''cat "''${anchors[@]}" > "$bundle"'' caUnit.script; + let + texts = + m: + map (u: m.systemd.services.${u}.script) [ + "hive-tls-ca" + "hive-tls-resign" + "hive-gateway-self-signed-cert" + ]; + reads = m: needle: lib.any (lib.hasInfix needle) (texts m); + deletes = + m: dir: + lib.all (f: lib.elem "r ${dir}/${f}" m.systemd.tmpfiles.rules) [ + "ca.pem" + "ca-previous.pem" + "ca-key.pem" + ]; + relocated = hiveWithController { deploy.hive-controller.tls.stateDir = "/srv/hive-tls"; }; + in + !(lib.any (reads allLocal) [ + "/ca.pem" + "ca-previous.pem" + "ca-key.pem" + ]) + && reads allLocal "swarm-services-root.pem" + && deletes allLocal "/var/lib/hive-tls" + && deletes noService "/var/lib/hive-tls" + && deletes relocated "/srv/hive-tls" + && !(deletes relocated "/var/lib/hive-tls"); + } + { + # The store host pointing its client CA at the old root: the store's and + # the controller's options both fire, the second through the glue's + # `mkDefault`. The control is the same host on its defaults. + name = "a CA-file option naming a deleted CA file fails evaluation, naming the store's mTLS CA"; + ok = + let + refused = + m: option: + lib.any ( + a: + !a.assertion + && lib.hasInfix "services.hyperhive.deploy.${option} is" a.message + && lib.hasInfix "/var/lib/swarm-bao-pki/ca.pem" a.message + ) m.assertions; + oldRoot = hiveWithController { + deploy.singleHostSwarm = true; + deploy.bao.clientCaFile = "/var/lib/swarm-ca/root.pem"; + }; + hiveCa = hive { + deploy.bao.serverCaFile = "/var/lib/hive-tls/ca.pem"; + deploy.swarm-controller.hiveClientCaFile = "/var/lib/hive-tls/ca-previous.pem"; + }; + options = [ + "bao.clientCaFile" + "bao.serverCaFile" + "swarm-controller.hiveClientCaFile" + ]; + in + refused oldRoot "bao.clientCaFile" + && refused oldRoot "swarm-controller.hiveClientCaFile" + && refused hiveCa "bao.serverCaFile" + && refused hiveCa "swarm-controller.hiveClientCaFile" + && !(lib.any (refused allLocal) options) + && allLocal.services.hyperhive.deploy.bao.clientCaFile == "/var/lib/swarm-bao-pki/ca.pem"; } { # The store host's hive presents the leaf ./glue-bao-tls.nix mints for