hive-tls: keep ca-key.pem on disk, and re-issue a leaf that does not match its key

hive-tls-ca no longer deletes a ca-key.pem an earlier generation left:
nothing reads it, and leaving it means a rollback finds the old key instead
of minting a new anchor. Deleting it moves to PR 3, beside dropping ca.pem
from the trust bundle.

hive-tls-resign now also re-issues when the leaf's public key differs from
the key file's. The leaf and key are moved into place by two renames, and an
interrupted pair left a certificate beside the wrong key that the validity
check accepted.

Refs #4588

docs/networking/gateway.md

@@ -86,7 +86,7 @@ The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `
 
 **Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive's role allows nothing outside `<hive>`, so a swarm-service name can't go on this leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
 
-**Rotation**: `hive-tls-resign.service` runs at boot, on every deploy and once a day, and asks the store for a new leaf when the current one is missing, past half its 720-hour lifetime, no longer covers the configured names, or doesn't chain to the store's root. A leaf already on disk lets the gateway start while the store can't answer; a failed renewal leaves it in place. To force a rotation, delete `gateway.pem` under the state dir and restart the unit; it re-imports the leaf and reloads `nginx` itself.
+**Rotation**: `hive-tls-resign.service` runs at boot, on every deploy and once a day, and asks the store for a new leaf when the current one is missing, past half its 720-hour lifetime, no longer covers the configured names, doesn't match its key, or doesn't chain to the store's root. A leaf already on disk lets the gateway start while the store can't answer; a failed renewal leaves it in place. To force a rotation, delete `gateway.pem` under the state dir and restart the unit; it re-imports the leaf and reloads `nginx` itself.
 
 **Cert prompts**: browsers warn once per host until you add the hive's `trust-bundle.pem` (the anchor, not the leaf) to the browser or OS trust store. A container that needs to trust it for its own outbound TLS gets the bundle a different way, bind-mounted read-only at `/run/hive-ca/trust-bundle.pem` (`nix/host-modules/lib/hive-ca-trust.nix`).
 

docs/swarm/ca.md

@@ -48,9 +48,9 @@ identity a swarm service uses.
 **When it issues.** The unit runs at boot, on every deploy, and from
 `hive-tls-resign.timer` once a day. It asks the store for a new leaf when
 the current one is missing, past half the role's 720-hour lifetime, missing
-a configured name, or doesn't chain to the store's root
+a configured name, doesn't match the key beside it, or doesn't chain to the
-(`swarm-services-root.pem`). That last check is what replaces a leaf the
+store's root (`swarm-services-root.pem`). That last check is what replaces a
-retired hive CA signed, on the first run that reaches the store.
+leaf the retired hive CA signed, on the first run that reaches the store.
 
 **The gateway doesn't wait for the store.** Nothing orders the gateway after
 `hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
@@ -76,9 +76,10 @@ the file every consumer anchors on. It holds whichever of these exist:
 - the swarm root, `<swarm.ca.stateDir>/root.pem`, on the host that generates
   it.
 
-No hive keeps a CA key. `hive-tls-ca.service` deletes
+No hive generates a CA or signs with a CA key. A
-`<deploy.hive-controller.tls.stateDir>/ca-key.pem` where an earlier
+`<deploy.hive-controller.tls.stateDir>/ca-key.pem` an earlier generation left
-generation left one, and generates no CA.
+stays on disk, unread: removing it comes with dropping `ca.pem` from the
+bundle.
 
 ⚠️ **A consumer reads its trust when it starts.** An agent or service
 container that last started before the store's root reached the bundle

nix/host-modules/hive-tls.nix

@@ -241,8 +241,8 @@ in
   # The gateway serves a short-lived **leaf** issued by the secret store's
   # `pki` mount through this hive's own role, and consumers (agents,
   # federation peers) anchor on that mount's root, which ships in
-  # `trust-bundle.pem`. Leaf rotation never re-breaks them, and no key that
+  # `trust-bundle.pem`. Leaf rotation never re-breaks them, and nothing here
-  # signs a certificate exists on this host. See
+  # generates or signs with a CA key. See
   # `docs/networking/gateway.md` ("Self-signed TLS") and docs/swarm/ca.md.
   #
   # The trust bundle still carries the retired hive CA's public `ca.pem`
@@ -259,7 +259,8 @@ in
         leaf the store issues and nginx serves) and `trust-bundle.pem` (the
         anchors agents and federation peers trust). `ca.pem` and
         `ca-previous.pem`, where present, are the retired hive CA's public
-        certificates and stay in the bundle; no CA key is kept here.
+        certificates and stay in the bundle. A `ca-key.pem` an earlier
+        generation left is neither read nor removed.
       '';
     };
 
@@ -396,13 +397,6 @@ in
         root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
         svcroot="$d/swarm-services-root.pem"
 
-        # The gateway leaf comes from the store, so no key that signs a
-        # certificate is kept here.
-        if [ -e "$d/ca-key.pem" ]; then
-          echo "removing the retired hive CA key $d/ca-key.pem"
-          rm -f "$d/ca-key.pem"
-        fi
-
         # --- Trust bundle: what a consumer must TRUST. Why it is a set of
         # anchors rather than one file is in docs/swarm/ca.md. Three
         # constraints the code cannot state:
@@ -444,7 +438,8 @@ in
     # The gateway leaf: issued by the secret store's `pki` mount through
     # this hive's own role, at boot, on every deploy and from the daily
     # timer below, whenever it is missing, past half its lifetime, missing
-    # a configured name, or not chained to the store's root.
+    # a configured name, not chained to the store's root, or not matching
+    # its key.
     #
     # ⚠️ Nothing orders the gateway against this unit, which is what lets a
     # hive holding a valid leaf start its gateway with the store sealed or
@@ -530,9 +525,14 @@ in
         # A leaf the retired hive CA signed, or one under a store root since
         # replaced, fails this and is re-issued.
         { [ -s "$svcroot" ] && openssl verify -CAfile "$svcroot" "$leaf" >/dev/null 2>&1; } || reissue=1
+        # The leaf and the key are two renames, so an interrupted rotation can
+        # leave a certificate beside a key it does not belong to.
+        leafpub="$(openssl x509 -in "$leaf" -noout -pubkey 2>/dev/null || true)"
+        keypub="$(openssl pkey -in "$leafk" -pubout 2>/dev/null || true)"
+        { [ -n "$leafpub" ] && [ "$leafpub" = "$keypub" ]; } || reissue=1
 
         if [ "$reissue" = 0 ]; then
-          echo "gateway leaf valid, chained to the store's root and covering the configured names — leaving it alone"
+          echo "gateway leaf valid, matching its key, chained to the store's root and covering the configured names — leaving it alone"
           exit 0
         fi
 

nix/module-eval/hive-tls.nix

@@ -4,7 +4,7 @@
 # The swarm-services leaf's renewal: a timer that really re-runs the
 # issuance, at a threshold read off the store role's lifetime. And the
 # store's root on a hive that asks it for no leaf. And the gateway leaf,
-# issued by the store through the hive's own role, with no hive CA key.
+# issued by the store through the hive's own role, with no hive CA.
 {
   pkgs,
   lib,
@@ -231,9 +231,13 @@ let
         && lib.hasInfix "root-key.pem" rootScript;
     }
     {
-      name = "a retired hive CA key is deleted, and its public certificate stays in the trust bundle";
+      # Left for PR 3, beside dropping `ca.pem`, so a rollback finds the old
+      # key and does not mint a new anchor. The control is the bundle reading
+      # `ca.pem` in the same script.
+      name = "a retired hive CA key is left in place, and its public certificate stays in the trust bundle";
       ok =
-        lib.hasInfix ''rm -f "$d/ca-key.pem"'' caUnit.script
+        !(lib.hasInfix "ca-key.pem" caUnit.script)
+        && !(lib.hasInfix "ca-key.pem" leafUnit.script)
         && lib.hasInfix ''ca="$d/ca.pem"'' caUnit.script
         && lib.hasInfix ''if [ -s "$ca" ]; then anchors+=("$ca"); fi'' caUnit.script
         && lib.hasInfix ''if [ -s "$prev" ]; then anchors+=("$prev"); fi'' caUnit.script
@@ -279,9 +283,19 @@ let
     {
       # A leaf the retired hive CA signed is re-issued from the store on the
       # first run that reaches it; one that renews in time never is.
-      name = "the gateway leaf is re-issued when not chained to the store's root, and at half its role lifetime";
+      # A cert from one issuance beside the key of another, left by an
+      # interrupted pair of renames, is re-issued rather than left for
+      # `nginx -t` to refuse. Checked on the decision side of the exit.
+      name = "the gateway leaf is re-issued when not matching its key, not chained to the store's root, or at half its role lifetime";
       ok =
-        lib.hasInfix ''openssl verify -CAfile "$svcroot" "$leaf"'' leafUnit.script
+        let
+          decision = lib.head (lib.splitString ''if [ "$reissue" = 0 ]'' leafUnit.script);
+        in
+        decision != leafUnit.script
+        && lib.hasInfix ''openssl verify -CAfile "$svcroot" "$leaf"'' decision
+        && lib.hasInfix ''leafpub="$(openssl x509 -in "$leaf" -noout -pubkey'' decision
+        && lib.hasInfix ''keypub="$(openssl pkey -in "$leafk" -pubout'' decision
+        && lib.hasInfix ''{ [ -n "$leafpub" ] && [ "$leafpub" = "$keypub" ]; } || reissue=1'' decision
         && lib.hasInfix (renewAt 720) leafUnit.script
         && lib.hasInfix (renewAt 48) shortTtl.systemd.services.hive-tls-resign.script
         && lib.hasInfix ''-in "$leaf" -noout -checkend "$renewat"'' leafUnit.script;