hive CA: delete ca.pem, ca-previous.pem and ca-key.pem, the bundle is the store's root

The gateway leaf comes from the store's pki mount, so a hive CA's
certificates anchor nothing and its key is a secret outside the store.

- hive-tls-ca writes trust-bundle.pem from swarm-services-root.pem alone;
  the ca.pem and ca-previous.pem slots are gone.
- Every host deletes ca.pem, ca-previous.pem and ca-key.pem in
  deploy.hive-controller.tls.stateDir with tmpfiles `r` rules that follow
  the option. The removal shim, now host-ca-removed.nix, holds these
  beside the /var/lib/swarm-ca rule and the swarm.ca.* removals.
- deploy.bao.clientCaFile, deploy.bao.serverCaFile and
  deploy.swarm-controller.hiveClientCaFile naming /var/lib/swarm-ca or one
  of the deleted hive CA files fail evaluation, naming the store's mTLS CA
  (/var/lib/swarm-bao-pki/ca.pem on the store's host) as the replacement.

module-eval hive-tls covers the single-anchor bundle, the deletion (incl.
a relocated stateDir) and the assertion on all three options.

Refs #4588

docs/networking/gateway.md

@@ -80,7 +80,7 @@ The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `
 
 **Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them.
 
-**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root, plus a hive CA's public `ca.pem` where one exists ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle.
+**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle.
 
 **Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https://<host>/.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap.
 

docs/swarm/README.md

@@ -34,7 +34,7 @@ grafana, the controller, authelia, …) →
   contract → [`credentials.md`](credentials.md) · how the store comes up,
   who writes its grants and how it unseals → [`bao.md`](bao.md)
 - **swarm CA** — the store root every hive's gateway leaf chains to, and what to
-  hand a peer (`trust-bundle.pem`, never `ca.pem`). → [`ca.md`](ca.md)
+  hand a peer. → [`ca.md`](ca.md)
 - **snapshot store** — the swarm's one `btrfs receive` endpoint,
   `swarm.snapshotStore.{address,port}`. →
   [`snapshot-store.md`](../networking/snapshot-store.md)

docs/swarm/ca.md

@@ -75,21 +75,10 @@ issues it.
 ## The trust bundle
 
 `hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
-the file every consumer anchors on. It holds whichever of these exist:
+the file every consumer anchors on. It holds one certificate, a copy of
-
+`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
-- `swarm-services-root.pem`, the store's `pki` root, which the gateway leaf
+the swarm-services leaf both chain to. It's the only root the swarm has. Until
-  and the swarm-services leaf both chain to.
+the first fetch of that root lands, the bundle is empty.
-- `ca.pem` and `ca-previous.pem`: a hive CA's public certificates. Nothing
-  issues under that CA; they stay in the bundle so a leaf it signed verifies
-  until `hive-tls-resign` replaces it. Removing them is a deliberate later
-  step.
-
-No other anchor goes in: the store's `pki` root is the only root the swarm
-has.
-
-No hive generates a CA or signs with a CA key. A
-`<deploy.hive-controller.tls.stateDir>/ca-key.pem` stays on disk, unread:
-removing it comes with dropping `ca.pem` from the bundle.
 
 ⚠️ **A consumer reads its trust when it starts.** An agent or service
 container that last started before the store's root reached the bundle
@@ -164,9 +153,17 @@ role's `allowed_domains`, enforced by the store at issue time: a hive's
 identity that leaks can get certificates for that hive's names and no
 others.
 
-No hive holds a root CA or its key, and nothing under `/var/lib/swarm-ca`
+No host holds a CA or its key, except the store's mTLS CA
-exists: every host deletes that directory at boot and on every deploy. The
+(`glue-bao-tls.nix`), which signs only the store's server certificate and the
-swarm distributes only certificates.
+certificates that log in to it. At boot and on every deploy, every host
+deletes `/var/lib/swarm-ca` and the `ca.pem`, `ca-previous.pem` and
+`ca-key.pem` in `<deploy.hive-controller.tls.stateDir>`. The swarm distributes
+only certificates.
+
+A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
+`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
+evaluation. Those options name the store's mTLS CA,
+`/var/lib/swarm-bao-pki/ca.pem` on the host that runs the store.
 
 ## What to hand a peer
 

nix/host-modules/default.nix

@@ -64,7 +64,7 @@
     ./swarm-victoriametrics.nix
     ./swarm-wireguard.nix
     ./swarm.nix
-    ./swarm-ca-removed.nix
+    ./host-ca-removed.nix
     ./swarm-peers-removed.nix
     ./swarm-required-services.nix
   ];

nix/host-modules/hive-tls.nix

@@ -249,9 +249,6 @@ in
   # `trust-bundle.pem`. Leaf rotation never re-breaks them, and nothing here
   # generates or signs with a CA key. See
   # `docs/networking/gateway.md` ("Self-signed TLS") and docs/swarm/ca.md.
-  #
-  # The trust bundle also carries a hive CA's public `ca.pem` where one
-  # exists, so a leaf it signed verifies until it is replaced.
 
   options.services.hyperhive.deploy.hive-controller.tls = {
     stateDir = lib.mkOption {
@@ -261,10 +258,7 @@ in
         Host directory holding the gateway leaf and trust bundle for the
         self-signed gateway mode: `gateway.pem` / `gateway-key.pem` (the
         leaf the store issues and nginx serves) and `trust-bundle.pem` (the
-        anchors agents and federation peers trust). `ca.pem` and
+        anchor agents and federation peers trust, the store's `pki` root).
-        `ca-previous.pem`, where present, are a hive CA's public
-        certificates and stay in the bundle. A `ca-key.pem` here is neither
-        read nor removed.
       '';
     };
 
@@ -363,9 +357,8 @@ in
       # resolves a loop by dropping a job.
       #
       # What that costs: the trust bundle written at the end of this script
-      # anchors on every root a consumer must hold, and the services root is
+      # is the services root, so on a first boot it is written before that
-      # one of them, so on a first boot the `cat` below runs before that file
+      # file exists and is empty until a restart. The `if [ -s
-      # exists and the bundle is one restart short of complete. The `if [ -s
       # "$svcroot" ]` guard is what makes that state legal rather than fatal,
       # and `swarm-services-cert` restarts this unit once the root it wrote
       # is new — so the bundle converges instead of being a boot behind.
@@ -381,42 +374,21 @@ in
         d=${lib.escapeShellArg cfg.stateDir}
         install -d -m 0755 "$d"
 
-        ca="$d/ca.pem"
-        prev="$d/ca-previous.pem"
         svcroot="$d/swarm-services-root.pem"
 
-        # --- Trust bundle: what a consumer must TRUST. Why it is a set of
+        # --- Trust bundle: what a consumer must TRUST, the store's `pki`
-        # anchors rather than one file is in docs/swarm/ca.md. Three
+        # root, which both the gateway leaf and the swarm-services leaf
-        # constraints the code cannot state:
+        # chain to (docs/swarm/ca.md).
         #
         # Written IN PLACE, never renamed into position — containers
         # bind-mount this file and a bind mount follows the inode, so a
         # rename leaves every consumer holding the old one.
-        #
-        # Dropping `ca.pem` and `ca-previous.pem` is deliberately NOT done
-        # here: "long enough" is a deployment fact, not a unit's call.
-        #
-        # Built as an array with `if` guards, not
-        # `[ -s f ] && anchors+=(f)`: under `set -e` an && list whose test
-        # fails IS a failing command and kills the unit — on exactly the
-        # hive where the optional file is legitimately absent.
-        #
-        # `swarm-services-root.pem` is the store's `pki` root, which both
-        # the gateway leaf and the swarm-services leaf chain to. `ca.pem`
-        # and `ca-previous.pem` are a hive CA's public certificates, kept
-        # so a leaf they signed verifies until replaced.
         bundle="$d/trust-bundle.pem"
-        anchors=()
+        if [ -s "$svcroot" ]; then
-        if [ -s "$ca" ]; then anchors+=("$ca"); fi
+          cat "$svcroot" > "$bundle"
-        if [ -s "$prev" ]; then anchors+=("$prev"); fi
+        else
-        if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
-        # `cat` with no arguments reads stdin, so no anchor at all is an
-        # empty file until `swarm-services-cert` writes the root.
-        if [ "''${#anchors[@]}" = 0 ]; then
           echo "no trust anchor on disk yet — $bundle is empty until the store's root is fetched" >&2
           : > "$bundle"
-        else
-          cat "''${anchors[@]}" > "$bundle"
         fi
         chmod 0644 "$bundle"
       '';
@@ -942,9 +914,8 @@ in
     # this file (read-only, public certs ONLY — never a key) into each
     # agent container so agents + their tools can trust the gateway's
     # self-signed leaf, and the meta flake wires the per-agent trust
-    # bundle. It is the ANCHOR bundle rather than `ca.pem` for the reason
+    # bundle. No key path is ever exposed (an agent that could read one
-    # spelled out where the bundle is written above; no key path is ever
+    # could mint trusted certs).
-    # exposed (an agent that could read one could mint trusted certs).
     #
     # ⚠️ Every line here that names another daemon's unit is gated on that
     # daemon's own enable: the gateway also runs on hosts with no hive and no

nix/host-modules/host-ca-removed.nix

@@ -0,0 +1,65 @@
+# Removal shim for the CAs a host held outside the secret store: the
+# swarm root under `/var/lib/swarm-ca` (and its `services.hyperhive.swarm.ca.*`
+# options), and a hive CA's `ca.pem`, `ca-previous.pem` and `ca-key.pem` in
+# `deploy.hive-controller.tls.stateDir`.
+#
+# The secret store's `pki` root is the only trust anchor: it issues every
+# hive's gateway leaf and the swarm's service names, and its key never leaves
+# the store. Nothing generates, reads or bundles either CA, so a definition of
+# a removed option fails evaluation with a message, a CA-file option naming
+# one of the deleted paths fails it too, and every host deletes the files.
+#
+# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to
+# remove when the deprecation window closes.
+{ lib, config, ... }:
+let
+  deployCfg = config.services.hyperhive.deploy;
+
+  swarmCaDir = "/var/lib/swarm-ca";
+  hiveCaFiles = map (f: "${deployCfg.hive-controller.tls.stateDir}/${f}") [
+    "ca.pem"
+    "ca-previous.pem"
+    "ca-key.pem"
+  ];
+  deleted = p: p == swarmCaDir || lib.hasPrefix "${swarmCaDir}/" p || lib.elem p hiveCaFiles;
+
+  # Every option that names a CA the store's mTLS checks against. One
+  # pointing at a deleted file fails the unit that reads it at start.
+  caFileOptions = {
+    "services.hyperhive.deploy.bao.clientCaFile" = deployCfg.bao.clientCaFile;
+    "services.hyperhive.deploy.bao.serverCaFile" = deployCfg.bao.serverCaFile;
+    "services.hyperhive.deploy.swarm-controller.hiveClientCaFile" =
+      deployCfg.swarm-controller.hiveClientCaFile;
+  };
+
+  removed =
+    name:
+    lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] ''
+      The host-held swarm root CA is gone: the secret store's pki root is the
+      only trust anchor, and nothing generates or reads a root under
+      ${swarmCaDir}. Remove this definition. See docs/swarm/ca.md.
+    '';
+in
+{
+  imports = map removed [
+    "autoConfigure"
+    "stateDir"
+    "validityDays"
+  ];
+
+  assertions = lib.mapAttrsToList (option: path: {
+    assertion = path == null || !(deleted path);
+    message = ''
+      ${option} is ${toString path}, a file every host deletes along with
+      the host-held CA it belonged to. Point it at the store's mTLS CA:
+      /var/lib/swarm-bao-pki/ca.pem on the host that runs the store, the copy
+      placed by hand on any other host. See docs/swarm/ca.md.
+    '';
+  }) caFileOptions;
+
+  # Both CAs' private keys are secrets outside the store, and the
+  # certificates beside them are anchors nothing issues under. The swarm
+  # root at the option's old default path only: a relocated `stateDir` is
+  # the operator's to delete.
+  systemd.tmpfiles.rules = [ "R ${swarmCaDir}" ] ++ map (f: "r ${f}") hiveCaFiles;
+}

nix/host-modules/lib/hive-ca-trust.nix

@@ -33,11 +33,8 @@ let
   # `gateway.useSelfSigned` is the single source of truth for the
   # self-signed condition — no duplicated derivation.
   useSelfSigned = gatewayCfg.useSelfSigned;
-  # The bundle, not `ca.pem`: the hive CA is an intermediate under the
+  # The bundle `hive-tls.nix` writes: the store's `pki` root, which the
-  # swarm root, and openssl (which is what both consumers below sit on —
+  # gateway's leaf chains to.
-  # Node's `NODE_EXTRA_CA_CERTS`, Go's `SSL_CERT_FILE`) will not end a
-  # chain at a trusted cert that isn't self-signed. `hive-tls.nix` writes
-  # the bundle next to the CA and explains the split.
   caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem";
   caContainerPath = "/run/hive-ca/trust-bundle.pem";
 

nix/host-modules/swarm-bao.nix

@@ -127,7 +127,7 @@ let
   # Where the leaf lands for openbao to read. `tlsDir` is bind-mounted at the
   # same path on both sides, so the delivery below needs no second mount, and
   # nothing has to bind `deploy.hive-controller.tls.stateDir`, which holds the
-  # hive CA's private key.
+  # gateway's private keys.
   #
   # The certificate and the client CA are public material and are read straight
   # off the mount; only the key takes the credential path above.

nix/host-modules/swarm-ca-removed.nix

@@ -1,33 +0,0 @@
-# Removal shim for `services.hyperhive.swarm.ca.*` and the host-held swarm
-# root it generated.
-#
-# The secret store's `pki` root is the only trust anchor: it issues every
-# hive's gateway leaf and the swarm's service names, and its key never leaves
-# the store. Nothing generates, reads or bundles a root under
-# `/var/lib/swarm-ca`, so a definition of these options fails evaluation with
-# a message, and the directory is deleted on every host.
-#
-# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to
-# remove when the deprecation window closes.
-{ lib, ... }:
-let
-  removed =
-    name:
-    lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] ''
-      The host-held swarm root CA is gone: the secret store's pki root is the
-      only trust anchor, and nothing generates or reads a root under
-      /var/lib/swarm-ca. Remove this definition. See docs/swarm/ca.md.
-    '';
-in
-{
-  imports = map removed [
-    "autoConfigure"
-    "stateDir"
-    "validityDays"
-  ];
-
-  # The root's private key is a secret outside the store, and the
-  # certificate beside it is an anchor nothing issues under. The option's
-  # default path only: a relocated `stateDir` is the operator's to delete.
-  systemd.tmpfiles.rules = [ "R /var/lib/swarm-ca" ];
-}

nix/module-eval/hive-tls.nix

@@ -155,7 +155,7 @@ let
         && lib.hasInfix ''"$BAO_ADDR/v1/pki/ca/pem"'' noSvcFetch
         && lib.hasInfix ''mv -f "$svcroot.new" "$svcroot"'' noSvcFetch
         && lib.hasInfix ''svcroot="$d/swarm-services-root.pem"'' noService.systemd.services.hive-tls-ca.script
-        && lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' noService.systemd.services.hive-tls-ca.script;
+        && lib.hasInfix ''cat "$svcroot" > "$bundle"'' noService.systemd.services.hive-tls-ca.script;
     }
     {
       # The hive's own identity, not its reader identity: the reader's policy
@@ -258,18 +258,12 @@ let
         && mentions noService "swarm-services-root.pem";
     }
     {
-      # Three slots and no fourth: a root slot beside them is an anchor
+      # One anchor: any other slot is a root nothing issues under.
-      # nothing issues under.
+      name = "the trust bundle is the store's root alone";
-      name = "the trust bundle anchors on the store's root and a hive CA's public certificates only";
       ok =
-        let
+        lib.hasInfix ''cat "$svcroot" > "$bundle"'' caUnit.script
-          slots = lib.filter (l: lib.hasPrefix "if [ -s " l && lib.hasInfix "anchors+=(" l) (
+        && !(lib.hasInfix "anchors+=(" caUnit.script)
-            lib.splitString "\n" caUnit.script
+        && !(lib.hasInfix "ca.pem" caUnit.script);
-          );
-        in
-        lib.length slots == 3
-        && lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' caUnit.script
-        && !(lib.hasInfix ''anchors+=("$root")'' caUnit.script);
     }
     {
       # The control is the all-local host, which defines none of them.
@@ -322,17 +316,74 @@ let
         && !(lib.hasInfix refusal (imp noService));
     }
     {
-      # Nothing reads or removes a `ca-key.pem`, so a rollback finds the
+      # The control is the same scan finding the store root's file, and a
-      # key and does not mint a new anchor. The control is the bundle
+      # relocated state dir, whose rules follow it.
-      # reading `ca.pem` in the same script.
+      name = "no hive TLS unit reads a hive CA's files, and every host deletes them";
-      name = "a hive CA key is neither read nor removed, and its public certificate stays in the trust bundle";
       ok =
-        !(lib.hasInfix "ca-key.pem" caUnit.script)
+        let
-        && !(lib.hasInfix "ca-key.pem" leafUnit.script)
+          texts =
-        && lib.hasInfix ''ca="$d/ca.pem"'' caUnit.script
+            m:
-        && lib.hasInfix ''if [ -s "$ca" ]; then anchors+=("$ca"); fi'' caUnit.script
+            map (u: m.systemd.services.${u}.script) [
-        && lib.hasInfix ''if [ -s "$prev" ]; then anchors+=("$prev"); fi'' caUnit.script
+              "hive-tls-ca"
-        && lib.hasInfix ''cat "''${anchors[@]}" > "$bundle"'' caUnit.script;
+              "hive-tls-resign"
+              "hive-gateway-self-signed-cert"
+            ];
+          reads = m: needle: lib.any (lib.hasInfix needle) (texts m);
+          deletes =
+            m: dir:
+            lib.all (f: lib.elem "r ${dir}/${f}" m.systemd.tmpfiles.rules) [
+              "ca.pem"
+              "ca-previous.pem"
+              "ca-key.pem"
+            ];
+          relocated = hiveWithController { deploy.hive-controller.tls.stateDir = "/srv/hive-tls"; };
+        in
+        !(lib.any (reads allLocal) [
+          "/ca.pem"
+          "ca-previous.pem"
+          "ca-key.pem"
+        ])
+        && reads allLocal "swarm-services-root.pem"
+        && deletes allLocal "/var/lib/hive-tls"
+        && deletes noService "/var/lib/hive-tls"
+        && deletes relocated "/srv/hive-tls"
+        && !(deletes relocated "/var/lib/hive-tls");
+    }
+    {
+      # The store host pointing its client CA at the old root: the store's and
+      # the controller's options both fire, the second through the glue's
+      # `mkDefault`. The control is the same host on its defaults.
+      name = "a CA-file option naming a deleted CA file fails evaluation, naming the store's mTLS CA";
+      ok =
+        let
+          refused =
+            m: option:
+            lib.any (
+              a:
+              !a.assertion
+              && lib.hasInfix "services.hyperhive.deploy.${option} is" a.message
+              && lib.hasInfix "/var/lib/swarm-bao-pki/ca.pem" a.message
+            ) m.assertions;
+          oldRoot = hiveWithController {
+            deploy.singleHostSwarm = true;
+            deploy.bao.clientCaFile = "/var/lib/swarm-ca/root.pem";
+          };
+          hiveCa = hive {
+            deploy.bao.serverCaFile = "/var/lib/hive-tls/ca.pem";
+            deploy.swarm-controller.hiveClientCaFile = "/var/lib/hive-tls/ca-previous.pem";
+          };
+          options = [
+            "bao.clientCaFile"
+            "bao.serverCaFile"
+            "swarm-controller.hiveClientCaFile"
+          ];
+        in
+        refused oldRoot "bao.clientCaFile"
+        && refused oldRoot "swarm-controller.hiveClientCaFile"
+        && refused hiveCa "bao.serverCaFile"
+        && refused hiveCa "swarm-controller.hiveClientCaFile"
+        && !(lib.any (refused allLocal) options)
+        && allLocal.services.hyperhive.deploy.bao.clientCaFile == "/var/lib/swarm-bao-pki/ca.pem";
     }
     {
       # The store host's hive presents the leaf ./glue-bao-tls.nix mints for