swarm-bao: keep the raft state in the container, bind only the TLS material

The state directory was bind-mounted from the host so a nixos-container
destroy could not take the swarm's secrets with it. No sibling service does
that -- swarm-grafana keeps its sqlite database inside the container on
ephemeral = false -- and the bind is what broke the store: upstream pairs
StateDirectory= with DynamicUser=, systemd relocates the state to
/var/lib/private/openbao, and that rename fails EBUSY on an active mount
point, so the unit died at STATE_DIRECTORY before bao ever ran.

The TLS material still has to cross the boundary, because a host unit writes
it and the container reads it, so it moves to its own small bind at
/var/lib/swarm-bao-tls rather than riding along in the state directory. That
directory is 0755 and read-only inside: the certificate and client CA are
public and are read straight off the mount.

The private key is not. install -m 0600 leaves it root-owned and the service
runs as a DynamicUser, so the bind-mounted file is unreadable to it -- which
the old layout hid, because StateDirectory chowned the whole tree on the way
past. LoadCredential is systemd's mechanism for precisely this: PID 1 opens
the source as root and re-exposes it inside the unit owned by the service's
own account.

nix/host-modules/swarm-bao.nix

@@ -39,13 +39,34 @@ let
   networkCfg = hyperhiveCfg.network;
   swarmDomain = hyperhiveCfg.swarm.domain;
 
-  # Two names for one location. `stateDir` is where openbao writes inside the
+  # Upstream's own default, kept so its documentation matches. The raft data
-  # container — upstream's own default, kept so its documentation matches. The
+  # lives INSIDE the container on `ephemeral = false`, the same way
-  # host path is a sibling of the other swarm services' state rather than a
+  # ./swarm-grafana.nix keeps its sqlite database — no sibling service binds
-  # path inside the container's tree, so `nixos-container destroy` cannot take
+  # its state out to the host.
-  # the swarm's secrets with it.
+  #
+  # ⚠️ Do not bind-mount this path. Upstream pairs `StateDirectory=` with
+  # `DynamicUser=`, which makes systemd hold the state at
+  # `/var/lib/private/openbao` and symlink this to it; that relocation is a
+  # rename, and a rename of an active mount point fails `EBUSY` at
+  # `STATE_DIRECTORY` — the unit then dies before `bao` runs at all.
   stateDir = "/var/lib/openbao";
-  hostStateDir = "/var/lib/swarm-bao";
+
+  # The TLS material is its own small bind, on both sides of the boundary at
+  # the same path. Separate from `stateDir` because a HOST unit writes it and
+  # the container only reads it, and because systemd owns `stateDir` and moves
+  # it around; this directory is ours.
+  tlsDir = "/var/lib/swarm-bao-tls";
+
+  # Where the key surfaces for openbao to open. NOT `${tlsDir}/server-key.pem`:
+  # that file is root-owned 0600 on the host, and the service runs as a
+  # `DynamicUser`, so it cannot read the bind-mounted original. `LoadCredential`
+  # is systemd's answer to exactly this — PID 1 reads the source as root and
+  # re-exposes it inside the unit, owned by the service's own account.
+  # `$CREDENTIALS_DIRECTORY` is `/run/credentials/<unit>`, and the config file
+  # is rendered ahead of time, so the path is spelled out rather than read from
+  # the environment.
+  serverKeyCredential = "server-key";
+  serverKeyCredentialPath = "/run/credentials/openbao.service/${serverKeyCredential}";
 
   # The PIN is deliberately absent here. It arrives as `BAO_HSM_PIN` from an
   # EnvironmentFile the provisioning unit writes, because a value interpolated
@@ -55,8 +76,9 @@ let
   # empty is the difference between a store that unseals itself and one that
   # needs a human after every restart.
   # The PKCS11 token store and its PINs live on the HOST and are bind-mounted
-  # in. Losing them loses the sealed store, so they are a host-level fact an
+  # in. Losing them loses the sealed store — the raft data is worth nothing
-  # operator can back up — the same reasoning as the raft data below.
+  # without the key that unseals it — so unlike the state directory these are
+  # deliberately a host-level fact an operator can back up.
   tokenStoreDir = "/var/lib/swarm-bao-token";
   pinEnvFile = "${tokenStoreDir}/pin.env";
 
@@ -73,12 +95,15 @@ let
   # sees, not a coercion error from here.
   domainBase = if swarmDomain == null then "invalid" else swarmDomain;
 
-  # Where the leaf lands for openbao to read. Inside the container, because
+  # Where the leaf lands for openbao to read. `tlsDir` is bind-mounted at the
-  # `hostStateDir` is already bind-mounted at `stateDir` — so the delivery
+  # same path on both sides, so the delivery below needs no second mount, and
-  # below needs no second mount, and nothing has to bind
+  # nothing has to bind `deploy.hive-controller.tls.stateDir`, which holds the
-  # `deploy.hive-controller.tls.stateDir`, which holds the hive CA's private key.
+  # hive CA's private key.
-  serverCertPath = "${stateDir}/server.pem";
+  #
-  serverKeyPath = "${stateDir}/server-key.pem";
+  # The certificate and the client CA are public material and are read straight
+  # off the mount; only the key takes the credential path above.
+  serverCertPath = "${tlsDir}/server.pem";
+  serverKeyPath = "${tlsDir}/server-key.pem";
 
   # The host-side sources, verbatim from the options — no fallback, because a
   # fallback is exactly the CA opinion this module must not hold. The units
@@ -99,14 +124,14 @@ let
   # has said what to trust yet".
   listenerTls = {
     tls_cert_file = serverCertPath;
-    tls_key_file = serverKeyPath;
+    tls_key_file = serverKeyCredentialPath;
   }
   // lib.optionalAttrs (baoDeploy.clientCaFile != null) {
     tls_client_ca_file = clientCaPath;
     tls_require_and_verify_client_cert = true;
   };
 
-  clientCaPath = "${stateDir}/client-ca.pem";
+  clientCaPath = "${tlsDir}/client-ca.pem";
 
   extraListeners = lib.listToAttrs (
     lib.imap1 (
@@ -365,8 +390,8 @@ in
       #      the container is ordered. Same trap `hostClientSecretDir`
       #      documents in ./swarm-authelia.nix. Ordering against whatever
       #      mints it belongs with whatever named the path, not here.
-      #   2. `hostStateDir` is already mounted at `stateDir`, so a copy needs
+      #   2. `tlsDir` is already mounted at the same path inside, so a copy
-      #      no second mount.
+      #      needs no second mount.
       #   3. A directory holding a leaf usually holds the CA's private key
       #      beside it. Binding that directory to reach one file inside it
       #      would hand the container authority to mint any name that CA can —
@@ -382,7 +407,11 @@ in
         };
         script = ''
           set -euo pipefail
-          install -d -m 0700 ${hostStateDir}
+          # 0755, not 0700: the container reads the certificate and the client
+          # CA straight off this mount as a non-root user, so it has to be able
+          # to traverse the directory. The key inside stays 0600 and reaches
+          # the service through `LoadCredential` instead.
+          install -d -m 0755 ${tlsDir}
 
           # Fail loudly rather than start a store that cannot serve. The path
           # is configured, so a missing file means whatever was supposed to
@@ -397,8 +426,8 @@ in
             fi
           done
 
-          install -m 0644 ${lib.escapeShellArg serverCertSrc} ${hostStateDir}/server.pem
+          install -m 0644 ${lib.escapeShellArg serverCertSrc} ${tlsDir}/server.pem
-          install -m 0600 ${lib.escapeShellArg serverKeySrc} ${hostStateDir}/server-key.pem
+          install -m 0600 ${lib.escapeShellArg serverKeySrc} ${tlsDir}/server-key.pem
         ''
         + lib.optionalString (baoDeploy.clientCaFile != null) ''
 
@@ -406,7 +435,7 @@ in
             echo "deploy.bao.clientCaFile names ${baoDeploy.clientCaFile}, which is missing or empty." >&2
             exit 1
           fi
-          install -m 0644 ${lib.escapeShellArg baoDeploy.clientCaFile} ${hostStateDir}/client-ca.pem
+          install -m 0644 ${lib.escapeShellArg baoDeploy.clientCaFile} ${tlsDir}/client-ca.pem
         '';
       };
 
@@ -469,15 +498,16 @@ in
         # rather than a convenience for nginx.
         privateNetwork = false;
 
-        # Raft state outlives the container. `ephemeral = false` keeps the
+        # Only what a HOST unit writes and this container reads crosses the
-        # container's own /var, but a bind makes the store's data a host-level
+        # boundary. The raft state deliberately does not: `ephemeral = false`
-        # fact an operator can back up and a `nixos-container destroy` cannot
+        # keeps the container's own /var, systemd owns `${stateDir}` through
-        # take with it — which for the swarm's secrets is the difference between
+        # `StateDirectory=`, and binding over it is what breaks the unit.
-        # a rebuild and an outage.
         bindMounts = {
-          ${stateDir} = {
+          # Read-only: `swarm-bao-certs` on the host is the only writer, and
-            hostPath = hostStateDir;
+          # the store has no reason to modify its own identity.
-            isReadOnly = false;
+          ${tlsDir} = {
+            hostPath = tlsDir;
+            isReadOnly = true;
           };
         }
         // lib.optionalAttrs (baoDeploy.seal == "pkcs11") {
@@ -528,15 +558,30 @@ in
               // sealSettings;
             };
 
-            # The PIN reaches openbao as an environment variable read from a
+            # The private key crosses the user boundary here, not on the mount.
-            # 0400 file the provisioning unit wrote — never as a value in this
+            # `swarm-bao-certs` installs it 0600 root-owned, and the unit runs
-            # expression, which would render it world-readable into the store.
+            # as a `DynamicUser`, so the bind-mounted file is unreadable to it —
-            # `TPM2_PKCS11_STORE` is required because the store is not at the
+            # PID 1 opens the source as root and re-exposes it under
-            # library's default location.
+            # `${serverKeyCredentialPath}`, owned by the service's own account.
-            systemd.services.openbao.serviceConfig = lib.mkIf (baoDeploy.seal == "pkcs11") {
+            #
-              EnvironmentFile = pinEnvFile;
+            # One assignment, not two: `serviceConfig.X = …` beside a
-              Environment = [ "TPM2_PKCS11_STORE=${tokenStoreDir}" ];
+            # `serviceConfig = …` is a duplicate attribute inside a single
-            };
+            # attrset literal and does not parse. Module merging happens across
+            # `config` blocks, not within a literal — so the seal's half joins
+            # with `optionalAttrs`.
+            systemd.services.openbao.serviceConfig =
+              lib.optionalAttrs haveServerTls {
+                LoadCredential = [ "${serverKeyCredential}:${serverKeyPath}" ];
+              }
+              # The PIN reaches openbao as an environment variable read from a
+              # 0400 file the provisioning unit wrote — never as a value in this
+              # expression, which would render it world-readable into the store.
+              # `TPM2_PKCS11_STORE` is required because the store is not at the
+              # library's default location.
+              // lib.optionalAttrs (baoDeploy.seal == "pkcs11") {
+                EnvironmentFile = pinEnvFile;
+                Environment = [ "TPM2_PKCS11_STORE=${tokenStoreDir}" ];
+              };
 
             # ⚠️ Upstream sets `restartIfChanged = false` on this unit, on
             # purpose: a restart SEALS the store and disconnects every client.