agents: re-pull the agent's own bao mTLS leaf on a timer

swarm-controller re-issues an agent's store leaf at half its validity, but
the running container kept presenting the copy its hive delivered at
start, so a re-issued leaf reached the agent only on its next restart.

hive-agent-bao-identity now seeds a live copy of the pair in its own
runtime directory from the delivered credentials, and re-runs every 10
minutes: it logs in with the live leaf, reads its own
swarm/agents/<agent>/bao-mtls (a read render_agent already grants), and
when the leaf changed, logs in with the new pair and swaps it in only if
the store accepts it.

Every unit in the container that logs in to the store presents the live
pair: hive-agent (queue-identity.nix), hive-matrix-daemon,
hive-subagent-daemon, hive-agent-forge-token, hive-agent-forge-accounts
and hive-agent-github-token. They keep loading the server CA as a
credential. The long-running daemons are ordered after the refresher so
the live pair exists when they start.

It renews only while the held leaf is still valid: the login that reads
the new leaf is made with the current one.

Refs #5017

docs/swarm/credentials.md

@@ -62,7 +62,7 @@ of the cell says how.
 | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once                                                                                                        | `swarm-controller`, under its own certificate                                                                                                                                                                                         | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs                                                                                                                                                                                                                                                        | ✅ the controller reads it on every five-minute matrix pass                                                                                                                                                                                                                               |
 | `swarm/controller/swarm-controller/oidc/client`             | authelia, at its first boot; `swarm-secret-publish` copies it in                                                                                                    | `swarm-controller`, under its own certificate, once at start                                                                                                                                                                          | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit                                                                                                                                                                                                                                                                            | ❌ read once at start; the controller holds the old value until it restarts                                                                                                                                                                                                               |
 | `swarm/controller/swarm-controller/forge/token`             | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start                                                                                                                                                                               | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file                                                                                                                                                                                           | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart                                                                                                                                        |
-| `swarm/agents/<agent>/bao-mtls`                             | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation                            | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config                                                                                                                                            | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself)                                                                                                                                                                                                            | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires                                                                                                                              |
+| `swarm/agents/<agent>/bao-mtls`                             | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation                            | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds                                                            | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself)                                                                                                                                                                                                            | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid                                      |
 | `swarm/agents/<agent>/queue`                                | `swarm-controller`, at agent creation                                                                                                                               | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive                                                      | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
 | `swarm/agents/<agent>/forge-token`                          | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity                                                          | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token`                                                                                                                                 | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes)                                                                                                                                                                                                                                        | ✅ the agent re-fetches on a 10-minute timer                                                                                                                                                                                                                                              |
 | `swarm/agents/<agent>/forge/<label>`                        | `swarm-controller`, when an operator links an external forge account in the swarm UI                                                                                | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json`                                                                                    | ❌ an operator's token; replaced only by deleting it and linking the label again                                                                                                                                                                                                                                                                                     | ✅ the agent re-fetches on a 2-minute timer                                                                                                                                                                                                                                               |
@@ -170,7 +170,10 @@ as root and re-exposes it under the consuming unit's own user.
 Inside the container, `hive-agent-bao-identity.service` logs in with that
 certificate and reads this row back before reporting success, so an agent
 locked out of its own identity says so at boot rather than at whichever pull
-needed the store first. The unit exists whenever
+needed the store first. It keeps a live copy of the pair in `/run/hive-agent-bao-identity/`,
+which every unit in the container that logs in to the store presents, and
+re-runs every 10 minutes: a leaf the controller re-issued replaces that copy
+once a login with it succeeds. The unit exists whenever
 `services.hyperhive.agent.bao.addr` has a value, which the hive's meta flake
 sets from its own store address — the same all-or-nothing gate the per-hive
 queue credential beside it uses, and the reason the delivery above never lands

nix/agent-modules/bao.nix

@@ -1,11 +1,14 @@
-# This agent's own identity at the swarm secret store, and the check that
+# This agent's own identity at the swarm secret store: the check that proves
-# proves it works.
+# it works, and the live copy every store reader in the container presents.
 #
 # `swarm-controller` mints the leaf at agent creation and publishes it; this
 # agent's hive collects it under the hive's own certificate and hands it in as
-# systemd credentials (`hive_c0re::lifecycle::agent_identity`). Nothing here
+# systemd credentials (`hive_c0re::lifecycle::agent_identity`). That copy is
-# fetches anything from the store — by the time this container boots, its
+# fixed for the life of the container, and the controller re-issues the leaf
-# identity is already inside it.
+# at half its validity, so this unit seeds a live copy from it and, on a timer,
+# reads the current leaf back from the store under the one it holds. Because
+# that login needs the held leaf to still be valid, a container whose leaf
+# expired while every pull failed recovers only at its next start.
 #
 # ⚠️ The identity arrives as credentials and NOT as a bind mount, and the mode
 # is why: the host file is `0600` to the hive daemon and this unit runs as the
@@ -46,6 +49,17 @@ let
 
   unitName = "hive-agent-bao-identity";
 
+  runtimeDir = unitName;
+  certFile = "/run/${runtimeDir}/cert";
+  keyFile = "/run/${runtimeDir}/key";
+  # Beside the live pair, so each rename that replaces it stays in one
+  # directory.
+  stagedCert = "/run/${runtimeDir}/cert.new";
+  stagedKey = "/run/${runtimeDir}/key.new";
+  # The store's whole answer, key included, in the unit's own `0700` directory.
+  rawFile = "/run/${runtimeDir}/bao-mtls.json";
+  errFile = "/run/${runtimeDir}/bao.err";
+
   # Where the identity lives in the store, spelled from the same pieces the
   # publisher uses. `swarm_secret_client::mtls::identity_path` builds
   # `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
@@ -87,16 +101,39 @@ in
         loudest possible way about a deployment that never asked for it.
       '';
     };
+
+    clientCertFile = lib.mkOption {
+      type = lib.types.str;
+      readOnly = true;
+      default = certFile;
+      description = ''
+        This agent's current store certificate, kept by `${unitName}.service`.
+        Read-only: it is where that unit writes, not a knob. Every unit that
+        logs in to the store as this agent points `BAO_CLIENT_CERT` here.
+      '';
+    };
+
+    clientKeyFile = lib.mkOption {
+      type = lib.types.str;
+      readOnly = true;
+      default = keyFile;
+      description = ''
+        The private key for `clientCertFile`, `0400` to the agent user.
+        🩸 A PATH and never a value.
+      '';
+    };
   };
 
   config = lib.mkIf configured {
     systemd.services.${unitName} = {
-      description = "prove this agent can authenticate to the swarm secret store as itself";
+      description = "keep this agent's swarm secret store identity current, and prove it works";
       after = [ "network.target" ];
       wantedBy = [ "multi-user.target" ];
       path = [
         pkgs.openbao
         pkgs.coreutils
+        pkgs.diffutils
+        pkgs.jq
       ];
       # ../host-modules/lib/store-retry.nix. From an agent the store is reached
       # through the gateway's stream passthrough, so either being down fails
@@ -104,10 +141,16 @@ in
       inherit (storeRetry) startLimitBurst startLimitIntervalSec;
       serviceConfig = storeRetry.serviceConfig // {
         Type = "oneshot";
-        RemainAfterExit = true;
+        # Not `RemainAfterExit`, so the timer below can start it again.
+        # `RuntimeDirectoryPreserve` keeps the live pair between runs.
+        RemainAfterExit = false;
         TimeoutStartSec = 30;
         User = agentName;
         Group = agentName;
+        RuntimeDirectory = runtimeDir;
+        RuntimeDirectoryMode = "0700";
+        RuntimeDirectoryPreserve = "yes";
+        UMask = "0377";
         # Bare ids, no paths: the terse `LoadCredential=` form that inherits a
         # credential the service *manager* received, which is what the
         # container manager passed in. ./queue.nix states the same shape.
@@ -119,9 +162,8 @@ in
       };
       environment = {
         BAO_ADDR = cfg.addr;
-        # `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
+        BAO_CLIENT_CERT = certFile;
-        BAO_CLIENT_CERT = "%d/${certCredential}";
+        BAO_CLIENT_KEY = keyFile;
-        BAO_CLIENT_KEY = "%d/${keyCredential}";
       };
       script = ''
         set -euo pipefail
@@ -150,8 +192,19 @@ in
           export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}"
         fi
 
-        err="$(mktemp)"
+        # `UMask=0377` makes every file this script creates `0400`, so each one
-        trap 'rm -f "$err"' EXIT
+        # is removed before the command that creates it.
+        err=${lib.escapeShellArg errFile}
+        raw=${lib.escapeShellArg rawFile}
+        trap 'rm -f "$err" "$raw" ${lib.escapeShellArg stagedCert} ${lib.escapeShellArg stagedKey}' EXIT
+
+        # The delivered pair is the start of every container's life: /run is
+        # empty at boot and kept across this unit's runs.
+        if [ ! -s "$BAO_CLIENT_CERT" ] || [ ! -s "$BAO_CLIENT_KEY" ]; then
+          rm -f "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"
+          cp "$CREDENTIALS_DIRECTORY/${keyCredential}" "$BAO_CLIENT_KEY"
+          cp "$CREDENTIALS_DIRECTORY/${certCredential}" "$BAO_CLIENT_CERT"
+        fi
 
         # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
         # variables above only decide which certificate the TLS handshake
@@ -163,6 +216,7 @@ in
         # it accepts, so this agent's leaf matches its own role and no other.
         # Naming the role here would be a third copy of a string
         # `swarm_secret_client::policy::agent_object_name` already owns.
+        rm -f "$err"
         if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
           echo "this agent's certificate was refused by the swarm secret store at $BAO_ADDR." >&2
           if [ -s "$err" ]; then
@@ -175,16 +229,13 @@ in
         export BAO_TOKEN
 
         # A token is not yet an answer: the login proves the certificate, this
-        # proves the policy attached to it. Reading this agent's own identity
+        # proves the policy attached to it, and the answer is the current
-        # back is the smallest read its document grants, and it is the same
+        # leaf. One read, so the certificate and key come from one version.
-        # check `swarm-controller` runs against the leaf before it reports the
-        # creation done — so a policy that drifted apart from the path fails
-        # here rather than in whichever pull needed it first.
         #
-        # ⚠️ Output discarded, not printed: the field is a certificate and the
+        # ⚠️ Written by redirect, never printed: the object holds a private
-        # object beside it is a private key. Nothing about this check needs a
+        # key.
-        # value, only whether the read succeeded.
+        rm -f "$err" "$raw"
-        if ! bao kv get -field=cert ${lib.escapeShellArg identityPath} >/dev/null 2>"$err"; then
+        if ! bao kv get -format=json ${lib.escapeShellArg identityPath} >"$raw" 2>"$err"; then
           echo "this agent logged in to the swarm secret store but cannot read ${identityPath}, so its policy does not cover its own path." >&2
           if [ -s "$err" ]; then
             cat "$err" >&2
@@ -194,8 +245,47 @@ in
           exit 1
         fi
 
-        echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName} and read ${identityPath}."
+        # `-j`: the bytes as the store holds them, which is what the hive
+        # delivered, so an unchanged leaf compares equal to the seeded copy.
+        rm -f ${lib.escapeShellArg stagedCert} ${lib.escapeShellArg stagedKey}
+        if ! jq -ej '.data.data.cert | strings' "$raw" >${lib.escapeShellArg stagedCert} \
+          || ! jq -ej '.data.data.key | strings' "$raw" >${lib.escapeShellArg stagedKey}; then
+          echo "${identityPath} holds no certificate and key; this agent keeps the one it has." >&2
+          exit 1
+        fi
+
+        if cmp -s ${lib.escapeShellArg stagedCert} "$BAO_CLIENT_CERT" \
+          && cmp -s ${lib.escapeShellArg stagedKey} "$BAO_CLIENT_KEY"; then
+          echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName}; its certificate at ${identityPath} is unchanged."
+        else
+          # This unit's next login uses whatever is swapped in, so a pair the
+          # store refuses must never replace the one it accepts.
+          rm -f "$err"
+          if ! BAO_CLIENT_CERT=${lib.escapeShellArg stagedCert} BAO_CLIENT_KEY=${lib.escapeShellArg stagedKey} \
+            bao login -method=cert -token-only >/dev/null 2>"$err"; then
+            echo "the swarm secret store refused the certificate at ${identityPath}; this agent keeps the one it has." >&2
+            if [ -s "$err" ]; then cat "$err" >&2; fi
+            exit 1
+          fi
+
+          # Two renames: a reader that opens the pair between them gets a
+          # mismatched cert and key and fails that one login.
+          mv -f ${lib.escapeShellArg stagedKey} "$BAO_CLIENT_KEY"
+          mv -f ${lib.escapeShellArg stagedCert} "$BAO_CLIENT_CERT"
+          echo "replaced this agent's store certificate with the one at ${identityPath}."
+        fi
       '';
     };
+
+    # The controller re-issues a leaf at half its validity, so ten minutes is
+    # small against the time the old one stays valid.
+    systemd.timers.${unitName} = {
+      description = "re-read this agent's store certificate from the secret store";
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        OnUnitInactiveSec = "10min";
+        RandomizedDelaySec = "1min";
+      };
+    };
   };
 }

nix/agent-modules/forge-accounts.nix

@@ -23,9 +23,7 @@ let
   agentName = config.services.hyperhive.agent.user.name;
   stateDir = "/agents/${agentName}/state";
 
-  # The same three ids ./bao.nix and ./forge-token.nix load.
+  # The server CA id ./bao.nix loads.
-  certCredential = "hive-agent-bao-cert";
-  keyCredential = "hive-agent-bao-key";
   serverCaCredential = "hive-agent-bao-server-ca";
 
   unitName = "hive-agent-forge-accounts";
@@ -73,23 +71,19 @@ in
         RuntimeDirectoryMode = "0700";
         # `0600`, the mode the files in the state dir have always had.
         UMask = "0077";
-        LoadCredential = [
+        LoadCredential = [ serverCaCredential ];
-          certCredential
-          keyCredential
-          serverCaCredential
-        ];
       };
       environment = {
         BAO_ADDR = cfg.addr;
-        BAO_CLIENT_CERT = "%d/${certCredential}";
+        BAO_CLIENT_CERT = cfg.clientCertFile;
-        BAO_CLIENT_KEY = "%d/${keyCredential}";
+        BAO_CLIENT_KEY = cfg.clientKeyFile;
       };
       script = ''
         set -euo pipefail
 
         # No identity delivered: ./bao.nix's check reports that.
-        for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
+        for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
-          if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
+          if [ ! -s "$f" ]; then
             echo "this agent has no store identity, so it cannot fetch its external forge accounts." >&2
             exit 0
           fi

nix/agent-modules/forge-token.nix

@@ -28,9 +28,7 @@ let
   # The name `swarm-controller` minted the token under — see ./queue-identity.nix.
   agentName = config.services.hyperhive.agent.user.name;
 
-  # The same three ids ./bao.nix and ./queue-identity.nix load.
+  # The server CA id ./bao.nix loads.
-  certCredential = "hive-agent-bao-cert";
-  keyCredential = "hive-agent-bao-key";
   serverCaCredential = "hive-agent-bao-server-ca";
 
   unitName = "hive-agent-forge-token";
@@ -108,29 +106,25 @@ in
         RuntimeDirectoryMode = "0700";
         RuntimeDirectoryPreserve = "yes";
         UMask = "0377";
-        LoadCredential = [
+        LoadCredential = [ serverCaCredential ];
-          certCredential
-          keyCredential
-          serverCaCredential
-        ];
       };
       environment = {
         BAO_ADDR = cfg.addr;
-        BAO_CLIENT_CERT = "%d/${certCredential}";
+        BAO_CLIENT_CERT = cfg.clientCertFile;
-        BAO_CLIENT_KEY = "%d/${keyCredential}";
+        BAO_CLIENT_KEY = cfg.clientKeyFile;
       };
       script = ''
         set -euo pipefail
 
         # No identity delivered: ./bao.nix's check reports that; saying it
         # twice adds nothing.
-        for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
+        for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
-          if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
+          if [ ! -s "$f" ]; then
             echo "this agent has no store identity, so it cannot fetch its own forge token." >&2
             exit 0
           fi
-          if [ ! -r "$CREDENTIALS_DIRECTORY/$id" ]; then
+          if [ ! -r "$f" ]; then
-            echo "cannot read $CREDENTIALS_DIRECTORY/$id, so this agent cannot present its store identity." >&2
+            echo "cannot read $f, so this agent cannot present its store identity." >&2
             exit 1
           fi
         done

nix/agent-modules/github-token.nix

@@ -22,9 +22,7 @@ let
   agentName = config.services.hyperhive.agent.user.name;
   stateDir = "/agents/${agentName}/state";
 
-  # The same three ids ./bao.nix and ./forge-accounts.nix load.
+  # The server CA id ./bao.nix loads.
-  certCredential = "hive-agent-bao-cert";
-  keyCredential = "hive-agent-bao-key";
   serverCaCredential = "hive-agent-bao-server-ca";
 
   unitName = "hive-agent-github-token";
@@ -76,23 +74,19 @@ in
         RuntimeDirectoryMode = "0700";
         # `0600`, the mode `github-token` has.
         UMask = "0077";
-        LoadCredential = [
+        LoadCredential = [ serverCaCredential ];
-          certCredential
-          keyCredential
-          serverCaCredential
-        ];
       };
       environment = {
         BAO_ADDR = cfg.addr;
-        BAO_CLIENT_CERT = "%d/${certCredential}";
+        BAO_CLIENT_CERT = cfg.clientCertFile;
-        BAO_CLIENT_KEY = "%d/${keyCredential}";
+        BAO_CLIENT_KEY = cfg.clientKeyFile;
       };
       script = ''
         set -euo pipefail
 
         # No identity delivered: ./bao.nix's check reports that.
-        for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
+        for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
-          if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
+          if [ ! -s "$f" ]; then
             echo "this agent has no store identity, so it cannot fetch its GitHub token." >&2
             exit 0
           fi

nix/agent-modules/matrix.nix

@@ -21,12 +21,10 @@ let
   accounts = config.services.hyperhive.agent.matrixAccounts;
   # This agent's own identity at the swarm secret store (./bao.nix). The daemon
   # reads each account's token from the store itself, `main` included (the
-  # swarm mints that one), so the store's coordinates belong on its unit. The
+  # swarm mints that one), so the store's coordinates belong on its unit: the
-  # same three credential ids ./bao.nix and ./forge-token.nix load.
+  # live certificate pair ./bao.nix keeps, and the server CA id it loads.
   baoCfg = config.services.hyperhive.agent.bao;
   storeConfigured = baoCfg.addr != null;
-  certCredential = "hive-agent-bao-cert";
-  keyCredential = "hive-agent-bao-key";
   serverCaCredential = "hive-agent-bao-server-ca";
   # **The enable signal.** Matrix is on for this agent exactly when it has at
   # least one account, because an account is the only thing the daemon has to
@@ -308,7 +306,11 @@ in
       description = "long-running matrix-sdk Client + MCP daemon";
       wantedBy = [ "multi-user.target" ];
       before = [ "hive-agent.service" ];
-      after = [ "network-online.target" ];
+      # The live certificate pair exists once ./bao.nix's unit has run.
+      after = [
+        "network-online.target"
+      ]
+      ++ lib.optional storeConfigured "hive-agent-bao-identity.service";
       wants = [ "network-online.target" ];
       environment = {
         # In-agent todo socket the harness serves (loose-ends v2): the
@@ -328,8 +330,8 @@ in
       // lib.optionalAttrs storeConfigured {
         HIVE_AGENT_NAME = userName;
         BAO_ADDR = baoCfg.addr;
-        BAO_CLIENT_CERT = "%d/${certCredential}";
+        BAO_CLIENT_CERT = baoCfg.clientCertFile;
-        BAO_CLIENT_KEY = "%d/${keyCredential}";
+        BAO_CLIENT_KEY = baoCfg.clientKeyFile;
         # Named even when no CA was delivered: a bare `LoadCredential=` is
         # non-fatal when absent, and the daemon treats a missing or empty file
         # as "use the container's own trust store".
@@ -400,14 +402,10 @@ in
         User = userName;
         Group = userName;
       }
-      # This agent's own certificate, imported by id so systemd materialises it
+      # The store's server CA, imported by id so systemd materialises it under
-      # under this unit's `User=` — the same bare form ./forge-token.nix uses.
+      # this unit's `User=` — the same bare form ./forge-token.nix uses.
       // lib.optionalAttrs storeConfigured {
-        LoadCredential = [
+        LoadCredential = [ serverCaCredential ];
-          certCredential
-          keyCredential
-          serverCaCredential
-        ];
       };
     };
 

nix/agent-modules/mcp.nix

@@ -381,6 +381,8 @@ in
       description = "subagent task runner + MCP daemon for hive-subagent";
       wantedBy = [ "multi-user.target" ];
       before = [ "hive-agent.service" ];
+      # The live certificate pair exists once ./bao.nix's unit has run.
+      after = lib.optional subagentReadsStore "hive-agent-bao-identity.service";
       # A subagent task runs its own nested `claude` invocation (via
       # `hive-claude`), which needs to resolve `claude` itself off PATH —
       # same reasoning as `hive-bash-daemon`'s `path` above, even though
@@ -442,8 +444,8 @@ in
         # Paths and a name only. `%d` is this unit's own credentials directory.
         HIVE_AGENT_NAME = userName;
         BAO_ADDR = config.services.hyperhive.agent.bao.addr;
-        BAO_CLIENT_CERT = "%d/hive-agent-bao-cert";
+        BAO_CLIENT_CERT = config.services.hyperhive.agent.bao.clientCertFile;
-        BAO_CLIENT_KEY = "%d/hive-agent-bao-key";
+        BAO_CLIENT_KEY = config.services.hyperhive.agent.bao.clientKeyFile;
         BAO_CACERT = "%d/hive-agent-bao-server-ca";
       };
       serviceConfig = {
@@ -481,13 +483,9 @@ in
         MemoryHigh = toString subagentMemoryHigh;
       }
       // lib.optionalAttrs subagentReadsStore {
-        # Bare ids: inherits the credentials the container manager passed in,
+        # Bare id: inherits the credential the container manager passed in,
         # the same form ./queue-identity.nix uses.
-        LoadCredential = [
+        LoadCredential = [ "hive-agent-bao-server-ca" ];
-          "hive-agent-bao-cert"
-          "hive-agent-bao-key"
-          "hive-agent-bao-server-ca"
-        ];
       }
       //
         lib.optionalAttrs

nix/agent-modules/queue-identity.nix

@@ -18,7 +18,7 @@
 # before its first queue connection and again on every reconnect attempt
 # (`hive-agent`'s `swarm_queue`), so a re-minted secret is picked up without a
 # restart. What this module hands the harness is the store's address and the
-# paths of its certificate.
+# paths of the live certificate pair ./bao.nix keeps.
 #
 # ⛔ The store certificate is for reaching the store and nothing else. It is
 # never presented to the queue: what goes to the queue is the secret read
@@ -37,11 +37,9 @@ let
   # role from it.
   agentName = config.services.hyperhive.agent.user.name;
 
-  # The same three ids ./bao.nix loads, because there is one store identity
+  # The server CA id ./bao.nix loads; the id is
-  # per agent; the ids are `hive_c0re::lifecycle::agent_identity`'s and a
+  # `hive_c0re::lifecycle::agent_identity`'s and a rename is a rename there
-  # rename is a rename there too.
+  # too.
-  certCredential = "hive-agent-bao-cert";
-  keyCredential = "hive-agent-bao-key";
   serverCaCredential = "hive-agent-bao-server-ca";
 
   # The store's address is the whole switch, exactly as in ./bao.nix — and
@@ -54,20 +52,18 @@ in
 {
   config = lib.mkIf configured {
     systemd.services.hive-agent = {
-      # Bare ids, no paths: the terse `LoadCredential=` form that inherits a
+      # A bare id, no path: the terse `LoadCredential=` form that inherits a
       # credential the service *manager* received, which is what the container
       # manager passed in. ./bao.nix and ./queue.nix state the same shape.
-      serviceConfig.LoadCredential = [
+      serviceConfig.LoadCredential = [ serverCaCredential ];
-        certCredential
+      # The live certificate pair exists once ./bao.nix's unit has run.
-        keyCredential
+      after = [ "hive-agent-bao-identity.service" ];
-        serverCaCredential
-      ];
       # Paths and a name only. `%d` is this unit's own credentials directory.
       environment = {
         HIVE_AGENT_NAME = agentName;
         BAO_ADDR = cfg.addr;
-        BAO_CLIENT_CERT = "%d/${certCredential}";
+        BAO_CLIENT_CERT = cfg.clientCertFile;
-        BAO_CLIENT_KEY = "%d/${keyCredential}";
+        BAO_CLIENT_KEY = cfg.clientKeyFile;
         # Named even when no CA was delivered: a bare `LoadCredential=` is
         # non-fatal when absent, and the harness treats a missing or empty file
         # as "use the container's own trust store".

nix/module-eval/agent-forge-bao.nix

@@ -75,19 +75,17 @@ let
     }
     {
       # One store identity per agent: the fetch presents the same certificate
-      # the identity check proves. Every `BAO_*` value is an address or a
+      # the identity check keeps current. Every `BAO_*` value is an address or
-      # `%d/` path, never a value.
+      # a path, never a value.
       name = "the fetch presents the agent's own store identity, by path";
       ok =
         let
           u = fetchUnit agentForgeBao;
           e = u.environment;
         in
-        builtins.elem "hive-agent-bao-cert" u.serviceConfig.LoadCredential
+        e.BAO_ADDR == baoAddr
-        && builtins.elem "hive-agent-bao-key" u.serviceConfig.LoadCredential
+        && e.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-        && e.BAO_ADDR == baoAddr
+        && e.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key";
-        && e.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
-        && e.BAO_CLIENT_KEY == "%d/hive-agent-bao-key";
     }
     {
       # The field is the secret. It goes to a file by redirect and is never
@@ -201,8 +199,7 @@ let
         in
         u.serviceConfig.User == name
         && u.serviceConfig.UMask == "0077"
-        && builtins.elem "hive-agent-bao-cert" u.serviceConfig.LoadCredential
+        && u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert";
-        && u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert";
     }
     {
       # Files a hive wrote keep working until the operator re-links them.

nix/module-eval/agent-github-bao.nix

@@ -82,11 +82,9 @@ let
         in
         u.serviceConfig.User == name
         && u.serviceConfig.UMask == "0077"
-        && builtins.elem "hive-agent-bao-cert" u.serviceConfig.LoadCredential
-        && builtins.elem "hive-agent-bao-key" u.serviceConfig.LoadCredential
         && u.environment.BAO_ADDR == baoAddr
-        && u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
+        && u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-        && u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key";
+        && u.environment.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key";
     }
     {
       # A file a hive wrote keeps working until the operator links a token in

nix/module-eval/agent-matrix.nix

@@ -116,20 +116,17 @@ let
       # The daemon reads this agent's tokens from the store ITSELF, as itself,
       # from inside this container — `main` too, since the swarm mints it and
       # no hive does. So it needs the same identity ./agent-forge-bao.nix's
-      # fetch carries, in its own credentials directory.
+      # fetch carries: the live pair, and the server CA from its credentials.
       name = "the matrix daemon carries this agent's own store identity";
       ok =
         let
           u = daemon agentMatrixBao;
         in
-        u.serviceConfig.LoadCredential == [
+        u.serviceConfig.LoadCredential == [ "hive-agent-bao-server-ca" ]
-          "hive-agent-bao-cert"
-          "hive-agent-bao-key"
-          "hive-agent-bao-server-ca"
-        ]
         && u.environment.BAO_ADDR == "https://bao.t.local:8200"
-        && u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
+        && u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-        && u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key";
+        && u.environment.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key"
+        && builtins.elem "hive-agent-bao-identity.service" u.after;
     }
     {
       # The agent's name, and nothing derived from it: the daemon builds its

nix/module-eval/agent-queue-bao.nix

@@ -87,20 +87,30 @@ let
         && builtins.elem "hive-agent-bao-server-ca" c;
     }
     {
-      # `%d` and not a path under the agent's state dir: the host file is
+      # The live pair sits in the unit's own runtime directory, which outlives
-      # `0600` to the hive daemon, so the only copy this unprivileged unit can
+      # each run so the timer's next run logs in with what the last one
-      # open is the one systemd puts in its own credentials directory. The
+      # wrote. The address is the option's value rather than a literal that
-      # address is the option's value rather than a literal that agrees with
+      # agrees with it today.
-      # it today.
+      name = "the identity check presents the live pair it keeps in its runtime directory";
-      name = "the identity check presents its certificate out of the credentials directory";
       ok =
         let
-          e = (agentBaoIdentity agentBao).environment;
+          u = agentBaoIdentity agentBao;
+          e = u.environment;
         in
-        e.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
+        e.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-        && e.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
+        && e.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key"
+        && u.serviceConfig.RuntimeDirectory == "hive-agent-bao-identity"
+        && u.serviceConfig.RuntimeDirectoryPreserve == "yes"
         && e.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr;
     }
+    {
+      # A leaf the controller re-issues reaches a running container only
+      # through this re-run, and an active unit cannot be started again.
+      name = "the identity check re-runs on a timer";
+      ok =
+        (agentBaoIdentity agentBao).serviceConfig.RemainAfterExit == false
+        && agentBao.systemd.timers.hive-agent-bao-identity.timerConfig.OnUnitInactiveSec == "10min";
+    }
     {
       # Same 403-not-a-miss reason as the hive-side readers: the path
       # `swarm_secret_client::mtls::identity_path` builds is the one this
@@ -133,22 +143,22 @@ let
     {
       # Nothing about the identity may be printed, and the read-back is where
       # that could slip: `bao kv get` on this path answers with certificate
-      # material, and the object beside it is a private key. The check needs
+      # material and a private key. The answer goes to a file in the unit's
-      # only whether the read succeeded.
+      # own `0700` directory.
       #
       # The path goes through `lib.escapeShellArg` here for the same reason the
       # module passes it through one — that helper decides whether an argument
       # needs quotes at all, and this one (only `[a-z0-9/-]`) comes back bare.
       # Spelling the quotes in by hand asserts a rendering nixpkgs chooses
       # rather than the redirect this property is about.
-      name = "the identity check discards what it reads back";
+      name = "the identity check writes what it reads back to a file";
       ok =
         let
           m = agentBao;
           name = m.services.hyperhive.agent.user.name;
           arg = lib.escapeShellArg "secret/swarm/agents/${name}/bao-mtls";
         in
-        lib.hasInfix "bao kv get -field=cert ${arg} >/dev/null" (agentBaoIdentity m).script;
+        lib.hasInfix "bao kv get -format=json ${arg} >\"$raw\"" (agentBaoIdentity m).script;
     }
     {
       # The absence arm, and what makes the four above able to fail. An agent
@@ -160,17 +170,16 @@ let
     }
     {
       # The harness reads its per-agent queue secret from the store itself,
-      # under the same store identity the check above proves, because there is
+      # under the same store identity the check above keeps current, because
-      # one identity per agent.
+      # there is one identity per agent.
       name = "the harness presents the agent's own store identity";
       ok =
         let
           u = agentHarness agentBao;
         in
-        builtins.elem "hive-agent-bao-cert" u.serviceConfig.LoadCredential
+        u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-        && builtins.elem "hive-agent-bao-key" u.serviceConfig.LoadCredential
+        && u.environment.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key"
-        && u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
+        && builtins.elem "hive-agent-bao-identity.service" u.after
-        && u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
         && u.environment.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr;
     }
     {
@@ -200,7 +209,7 @@ let
         in
         !(u.environment ? BAO_ADDR)
         && !(u.environment ? HIVE_AGENT_NAME)
-        && !(builtins.elem "hive-agent-bao-cert" (u.serviceConfig.LoadCredential or [ ]));
+        && !(builtins.elem "hive-agent-bao-server-ca" (u.serviceConfig.LoadCredential or [ ]));
     }
   ];
 in

nix/module-eval/agent-runtime.nix

@@ -113,15 +113,11 @@ let
         lib.all
           (
             u:
-            u.serviceConfig.LoadCredential == [
+            u.serviceConfig.LoadCredential == [ "hive-agent-bao-server-ca" ]
-              "hive-agent-bao-cert"
-              "hive-agent-bao-key"
-              "hive-agent-bao-server-ca"
-            ]
             && u.environment.HIVE_AGENT_NAME == "a1"
             && u.environment.BAO_ADDR == "https://bao.t.local:8200"
-            && u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
+            && u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-            && u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
+            && u.environment.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key"
           )
           [
             (subagent claudeBao)
@@ -135,15 +131,11 @@ let
         let
           u = subagent opencodeBao;
         in
-        u.serviceConfig.LoadCredential == [
+        u.serviceConfig.LoadCredential == [ "hive-agent-bao-server-ca" ]
-          "hive-agent-bao-cert"
-          "hive-agent-bao-key"
-          "hive-agent-bao-server-ca"
-        ]
         && u.environment.HIVE_AGENT_NAME == "a1"
         && u.environment.BAO_ADDR == "https://bao.t.local:8200"
-        && u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
+        && u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/cert"
-        && u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key";
+        && u.environment.BAO_CLIENT_KEY == "/run/hive-agent-bao-identity/key";
     }
     {
       name = "an opencode agent with no store hands its subagent daemon no store identity";

nix/script-tests/agent-bao-fetch.nix

@@ -1,6 +1,7 @@
-# `checks.script-test-agent-bao-fetch` — runs the agent unit that logs in to
+# `checks.script-test-agent-bao-fetch` — runs the agent units that log in to
-# the swarm secret store and fetches a secret, ../agent-modules/forge-token.nix,
+# the swarm secret store and fetch a secret, ../agent-modules/forge-token.nix
-# against a stub `bao`. The cases are in ./agent-bao-fetch.sh.
+# and ../agent-modules/bao.nix, against a stub `bao`. The cases are in
+# ./agent-bao-fetch.sh.
 #
 # What runs is the unit's rendered `ExecStart`, on the unit's own `PATH` with
 # the stub in front. The one edit is the unit's `/run/<unit>/` prefix, moved
@@ -42,11 +43,16 @@ let
 
   # Answers `login` and `kv get` from `FAKE_BAO_*` variables and logs every
   # call. A `kv` call without the token `login` printed fails, so a script that
-  # drops `BAO_TOKEN` between the two cannot pass.
+  # drops `BAO_TOKEN` between the two cannot pass. A login presenting the
+  # certificate `FAKE_BAO_REFUSE_CERT` is refused.
   fakeBao = pkgs.writeShellScriptBin "bao" ''
     echo "$*" >> "$FAKE_BAO_LOG"
     case "$1" in
       login)
+        if [ -n "''${FAKE_BAO_REFUSE_CERT-}" ] && [ "$(<"$BAO_CLIENT_CERT")" = "$FAKE_BAO_REFUSE_CERT" ]; then
+          echo "fake bao: certificate refused" >&2
+          exit 2
+        fi
         printf '%s' "$FAKE_BAO_LOGIN_ERR" >&2
         printf '%s' "$FAKE_BAO_LOGIN_OUT"
         exit "$FAKE_BAO_LOGIN_RC"
@@ -70,6 +76,7 @@ in
 pkgs.runCommand "hyperhive-script-test-agent-bao-fetch"
   (
     unitEnv "FORGE" "hive-agent-forge-token"
+    // unitEnv "IDENT" "hive-agent-bao-identity"
     // {
       FAKE_BAO_BIN = "${fakeBao}/bin";
     }

nix/script-tests/agent-bao-fetch.sh

@@ -38,6 +38,13 @@ setup() {
   mkdir "$creds"
   printf cert >"$creds/hive-agent-bao-cert"
   printf key >"$creds/hive-agent-bao-key"
+  # The identity pair the unit presents: the delivered one for the fetch, the
+  # live copy it keeps for the identity unit.
+  if [ "$prefix" = IDENT ]; then
+    client_cert=$rt/cert client_key=$rt/key
+  else
+    client_cert=$creds/hive-agent-bao-cert client_key=$creds/hive-agent-bao-key
+  fi
   : >"$log"
   if ! sed "s|/run/$unit/|$rt/|g" "${!script_var}" >"$dir/script"; then
     echo "cannot read the rendered script for $unit" >&2
@@ -53,6 +60,7 @@ setup() {
   kv_rc=0
   kv_out=the-secret
   kv_err=
+  refuse_cert=
 }
 
 go() {
@@ -63,6 +71,8 @@ go() {
     exec env -i \
       PATH="$FAKE_BAO_BIN:$unit_path" \
       BAO_ADDR="$bao_addr" \
+      BAO_CLIENT_CERT="$client_cert" \
+      BAO_CLIENT_KEY="$client_key" \
       CREDENTIALS_DIRECTORY="$creds" \
       FAKE_BAO_LOG="$log" \
       FAKE_BAO_LOGIN_RC="$login_rc" \
@@ -71,6 +81,7 @@ go() {
       FAKE_BAO_KV_RC="$kv_rc" \
       FAKE_BAO_KV_OUT="$kv_out" \
       FAKE_BAO_KV_ERR="$kv_err" \
+      FAKE_BAO_REFUSE_CERT="$refuse_cert" \
       "$dir/script"
   ) >"$dir/out" 2>"$dir/err"
   rc=$?
@@ -234,6 +245,68 @@ expect_out "fetched this agent's forge token"
 expect_file "$rt/token" the-secret
 expect_no_file "$rt/token.new"
 
+ident_path=secret/swarm/agents/a1/bao-mtls
+ident_kv_call="kv get -format=json $ident_path"
+leaf() { printf '{"data":{"data":{"cert":"%s","key":"%s"}}}' "$1" "$2"; }
+
+setup IDENT "no store identity delivered"
+: >"$creds/hive-agent-bao-cert"
+go
+expect_rc 1
+expect_err "has no store identity"
+expect_calls
+expect_no_file "$rt/cert"
+
+setup IDENT "a first run seeds the live pair from the delivered one"
+kv_out=$(leaf cert key)
+go
+expect_rc 0
+expect_out "is unchanged"
+expect_calls "$login_call" "$ident_kv_call"
+expect_file "$rt/cert" cert
+expect_file "$rt/key" key
+if [ "$(stat -c %a "$rt/key")" != 400 ]; then fail "key is mode $(stat -c %a "$rt/key"), expected 400"; fi
+expect_no_file "$rt/bao-mtls.json"
+
+setup IDENT "the live pair outlives the delivered one"
+printf cert2 >"$rt/cert"
+printf key2 >"$rt/key"
+chmod 0400 "$rt/cert" "$rt/key"
+kv_out=$(leaf cert2 key2)
+go
+expect_rc 0
+expect_out "is unchanged"
+expect_file "$rt/cert" cert2
+
+setup IDENT "a re-issued leaf replaces the live pair once a login with it succeeds"
+kv_out=$(leaf cert2 key2)
+go
+expect_rc 0
+expect_out "replaced this agent's store certificate"
+expect_calls "$login_call" "$ident_kv_call" "$login_call"
+expect_file "$rt/cert" cert2
+expect_file "$rt/key" key2
+expect_no_file "$rt/cert.new"
+expect_no_file "$rt/key.new"
+
+setup IDENT "a re-issued leaf the store refuses is not swapped in"
+kv_out=$(leaf cert2 key2)
+refuse_cert=cert2
+go
+expect_rc 1
+expect_err "refused the certificate at $ident_path"
+expect_file "$rt/cert" cert
+expect_file "$rt/key" key
+expect_no_file "$rt/cert.new"
+
+setup IDENT "an object without a key keeps the live pair"
+kv_out='{"data":{"data":{"cert":"cert2"}}}'
+go
+expect_rc 1
+expect_err "holds no certificate and key"
+expect_file "$rt/cert" cert
+expect_file "$rt/key" key
+
 if [ "$failures" -ne 0 ]; then
   echo "script-test-agent-bao-fetch: $failures failed assertions over $count runs" >&2
   exit 1

swarm-controller/src/agent_renewal.rs

@@ -8,12 +8,12 @@
 //! the controller's own record, [`queue::AgentCredential::minted_at`]; a secret
 //! without one is stamped, value unchanged ([`SecretStep::Backfill`]).
 //!
-//! Either replacement reaches the agent at its next start, when the container
+//! A running agent re-reads its certificate every ten minutes under the one it
-//! is handed the certificate and fetches the secret; nothing pulls either into
+//! holds (`nix/agent-modules/bao.nix`); the old one stays valid until it
-//! a running container. The old certificate stays valid until it expires. The
+//! expires. A new secret reaches the agent at its next start, when the
-//! old secret stops working at once: the queue checks it only at `CONNECT`, so
+//! container fetches it. The old secret stops working at once: the queue checks
-//! an open connection is unaffected, but a reconnect before that restart
+//! it only at `CONNECT`, so an open connection is unaffected, but a reconnect
-//! presents the old secret and is denied.
+//! before that restart presents the old secret and is denied.
 //!
 //! [`spawn`]'s pass, once the queue is up and every [`RECONCILE_INTERVAL`],
 //! inserts a job per agent with anything due: `MintAgentIdentity` for the

swarm-secret-client/src/mtls.rs

@@ -9,9 +9,11 @@
 //! container.
 //!
 //! 🔑 The recursion this shape looks like it has — *a credential fetched from
-//! the store that is what opens the store* — is broken by who reads it. The
+//! the store that is what opens the store* — is broken by who reads it first.
-//! reader is the hive, never the agent, and a hive already holds its own leaf.
+//! The first reader is the hive, which already holds its own leaf, so nothing
-//! Nothing has to already be an agent in order to obtain an agent's identity.
+//! has to already be an agent in order to obtain an agent's identity. After
+//! that the agent re-reads its own path under the leaf it holds, which picks
+//! up a re-issued leaf only while the held one is still valid.
 
 use serde::{Deserialize, Serialize};