swarm-bao: create the KV mount the controller writes credentials through

The bootstrap unit writes a policy granting `secret/data/swarm/agents/*` and
nothing creates that engine. A fresh OpenBao has no `secret/` — only a dev-mode
one does — so `swarm-controller`'s first credential write answers `no handler
for route "secret/data/swarm/agents/<agent>/matrix/<name>". route entry not
found.` Measured on the live host at 21:27:27Z; #4171.

`git grep` for `secrets enable`, `kv-v2`, `kv_v2` and `sys/mounts` returned zero
across the whole tree. Control, so the zero means something: `auth enable` in
this same file returns 2 — the same defect was already found and fixed once, for
the cert auth mount, with a comment that states the principle. This is the other
half of it.

The mount name is now bound once and interpolated into both the policy text and
the new step, because a grant and a mount that disagree is exactly the failure
being fixed.

Placed outside the client-CA block: the controller writes *through* this mount
regardless of whether anything can log in by certificate. `module-eval` asserts
that, since one indentation level decides it.

Grants, measured against a real openbao 2.6.2 rather than derived:
`-output-policy` asks for `sys/mounts/secret` create+update, and a token holding
exactly `sys/mounts` read + `sys/mounts/<path>` create/update enabled the engine
— **no `sudo`**, unlike `sys/auth/cert`. Negative control: the same token on an
ungranted path got 403, so the grant is what made it work. `setup.md`'s
documented policy gains those two.

Also from that session, each deciding how this is written: re-enabling an
existing path errors (exit 2), so this asks first like the auth mount does;
`secrets list -format=json` keys look like `"secret/"`, so the `case` idiom
ports over; and `kv put -mount=<p>` reports `<p>/data/...`, confirming v2 — the
prefix the policy grants and the client writes.

setup.md also drops a check that cannot work: it told the operator to confirm
with `bao read auth/cert/…`, which 403s because the host wrapper carries no
token. `systemctl status swarm-bao-controller-policy` needs no credential and
names the three success lines. The first-attempt-after-rebuild race is now
written down too — the store is still coming up, and the 30s retry is what
lands.

Refs #4171.

nix/host-modules/swarm-bao.nix

@@ -198,11 +198,18 @@ let
       capabilities = ["list"]
     }
 
-    path "secret/data/swarm/agents/*" {
+    path "${credentialMountPath}/data/swarm/agents/*" {
       capabilities = ["create", "update"]
     }
   '';
 
+  # The KV v2 engine the controller writes agent credentials through. Named
+  # once because the grant above and the `secrets enable` in the bootstrap unit
+  # have to agree: a policy pointing at a mount nobody created is precisely the
+  # failure this pair exists to avoid. `swarm-secret-client` pins the same
+  # value on the reading side.
+  credentialMountPath = "secret";
+
   # Every listener serves the same identity: they differ in which address
   # they answer on, not in who they are. Client verification is separate and
   # optional — a store with no `clientCaFile` still serves TLS, it just does
@@ -904,6 +911,23 @@ in
           # than failing on one that already exists.
           printf '%s' ${lib.escapeShellArg controllerPolicyText} |
             bao policy write ${lib.escapeShellArg controllerPolicyName} -
+
+          # The policy grants paths under `${credentialMountPath}/`, and a
+          # fresh store has no such engine — only a dev-mode one does. Writing
+          # to an unmounted path answers "route entry not found", so the
+          # controller's first credential write fails against a grant that
+          # reads as correct.
+          #
+          # Outside the client-CA block below on purpose: this mount is what
+          # the controller writes *through*, independent of who may log in.
+          #
+          # Asked rather than attempted, same as the auth mount: `secrets
+          # enable` errors on a path already in use.
+          mounts="$(bao secrets list -format=json)"
+          case "$mounts" in
+            *'"${credentialMountPath}/"'*) ;;
+            *) bao secrets enable -path=${credentialMountPath} kv-v2 ;;
+          esac
         ''
         + lib.optionalString (baoDeploy.clientCaFile != null) ''
 

nix/module-eval.nix

@@ -613,6 +613,22 @@ let
         && lib.hasInfix "auth/cert/certs/swarm-controller" s
         && lib.hasInfix "/var/lib/swarm-bao-tls/client-ca.pem" s;
     }
+    {
+      # Same shape as the cert mount above, for the engine the controller
+      # writes credentials through: a fresh store has no `secret/`, so the
+      # grant would name a mount nobody created and the first write would 404.
+      #
+      # ⚠️ Matched on the COMMAND, for the reason the no-client-CA case below
+      # spells out: the policy text is embedded in this same script and grants
+      # `secret/data/...`, so any arm keyed on the *path* is satisfied either
+      # way and could never fail.
+      name = "the granting unit creates the KV mount the controller writes through";
+      ok =
+        let
+          s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
+        in
+        lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
+    }
     {
       # Nothing asserted the PKI script before this, so a third leaf could be
       # added to it and every case still passed — measured, not assumed: the
@@ -780,7 +796,12 @@ let
         in
         lib.hasInfix "bao policy write" s
         && !(lib.hasInfix "bao auth enable cert" s)
-        && !(lib.hasInfix "client-ca.pem" s);
+        && !(lib.hasInfix "client-ca.pem" s)
+        # The KV mount is NOT part of what a missing client CA switches off:
+        # the controller writes through it whether or not anything can log in
+        # by certificate. Asserted here rather than trusted, because both
+        # steps live in the same script and one indentation level decides it.
+        && lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
     }
     {
       # Reads the vhost's rendered `root`, not the option: the UI is served