From 1ac34a4ad68893985474ca79c4068182e9a7b4dd Mon Sep 17 00:00:00 2001
From: atlas <atlas@hyperhive.local>
Date: Mon, 5 Oct 2026 22:13:59 +0200
Subject: [PATCH] docs: callout xkey needs the store at every responder start;
 split-host order; replacing it

secrets.md now says that the responder reads the seed at every start, so
a store outage plus a responder restart refuses every client; that on a
split deployment the store's host deploys first; and that replacing the
key takes `bao kv metadata delete` on both paths, since a soft delete
keeps the metadata and the create is then refused. It also drops the
claim that every token reads `swarm/public/`.
---
 docs/swarm/secrets.md | 49 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 12 deletions(-)

diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md
index 46f5ad2f..3134dfb5 100644
--- a/docs/swarm/secrets.md
+++ b/docs/swarm/secrets.md
@@ -161,12 +161,12 @@ account or on the wire can read the request or the user JWT minted for it.
 You configure nothing about the key: it lives in the swarm secret store and
 comes into being on the queue's first boot.
 
-- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`. Only the
-  `swarm-nats-auth` role reads it, and it may create the path but never
-  replace it.
-- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`. Every token
-  the store issues may read `swarm/public/`; see
-  [`bao.md`](bao.md#what-every-token-may-read).
+- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
+- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
+
+Only the `swarm-nats-auth` role reads either path, and it may create each but
+never replace it. The policy of `swarm-secret-publisher`, which writes the rest
+of `swarm/services/`, denies it the seed's path.
 
 On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
 `swarm-nats` container, under the responder's store leaf:
@@ -182,15 +182,40 @@ On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
    A responder that can't read it doesn't start, rather than answering
    requests it can't open.
 
-When the store is down at boot, the fetch unit fails and retries for a day.
-The server starts with the key fetched last time, or not at all on a first
-boot. The retry that lands the file starts or restarts the server, and the
-responder comes up with it.
+The store has to be reachable each time the responder starts. When it's down,
+the fetch unit fails and retries for a day. The server starts with the key
+fetched last time, or not at all on a first boot. A responder that starts
+while the store is down exits and retries every 5 seconds, and until one
+start reads the seed the queue refuses every client, OIDC ones included. The
+retry that lands the file starts or restarts the server, and the responder
+comes up with it.
+
+Where the queue and the store run on different hosts, deploy the store's host
+first. `swarm-bao-nats-auth-policy` there writes the role's grants on the two
+paths. A queue host deployed before it can't read or create the key, so its
+fetch fails and the server stays down until you deploy the store's host. If
+that takes longer than the fetch's day of retries, run
+`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
 
 Two starts that create the key at once can't both win: each write is a
 create that fails once the path holds anything, and the start that lost reads
-the winner's key back. To replace the key, delete both paths, then restart
-`swarm-bao-nats-callout-xkey`.
+the winner's key back.
+
+To replace the key, remove both paths with their metadata and every version,
+then restart the fetch unit on the queue's host. Its restart of `nats.service`
+restarts the responder too, which then loads the new seed:
+
+```sh
+bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
+bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
+systemctl restart swarm-bao-nats-callout-xkey
+```
+
+`bao kv delete` isn't enough. It deletes the latest version but keeps the
+path's metadata, so the store still counts the path as existing and refuses
+the create that makes a new key. No principal hyperhive configures
+may delete under `secret/metadata/`, so run the two deletes with a token
+that may, such as the root token.
 
 ## Hive-level — one of each per hive