hive status publishing: read the hive's OIDC secret from the store, retry until it is there

hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.

Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.

The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.

Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.

Refs #5015

nix/host-modules/swarm-secret-publisher.nix

@@ -1,7 +1,7 @@
 # The unit that puts swarm-level secrets into the swarm's secret store, so
-# whoever needs one can read it there: a hive its agents' credential, a
-# service or the controller its own. It mints nothing: every value is an OIDC
-# client secret authelia minted, and this unit copies it.
+# whoever needs one can read it there: a hive its own and its agents'
+# credentials, a service or the controller its own. It mints nothing: every
+# value is an OIDC client secret authelia minted, and this unit copies it.
 #
 # ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
 # into the store even when the service runs beside authelia, because its
@@ -57,6 +57,10 @@ let
     hive:
     "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}${hyperhiveCfg.swarm.authelia.agentClientSuffix}";
 
+  # The hive's own client id, which hive-c0re publishes the hive's status with.
+  # Composed as ./swarm-authelia.nix composes it, for the reason above.
+  hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
+
   # The swarm's own services, as opposed to its hives. One client for the whole
   # swarm rather than one per hive, so one value in the store rather than a copy
   # each: `swarm/services/<id>/oidc/client`.
@@ -100,8 +104,9 @@ in
       description = ''
         Publish the OIDC client secrets this host mints into the swarm's
         secret store, so a hive that does not run authelia can read its
-        agents' credential and a swarm service or the controller can read its
-        own — from wherever it runs, this host included.
+        own and its agents' credentials and a swarm service or the
+        controller can read its own — from wherever it runs, this host
+        included.
 
         Defaults to whether this host mints the OIDC secrets, which is the
         only half of the question that is a property of *this* host.
@@ -114,8 +119,9 @@ in
         identity's job (see `baoClientCertFile`), not this option's.
 
         Turning it off leaves every hive but this one without its agents'
-        credential, the swarm controller unable to start, the swarm's Grafana
-        without any login at all, its collector pushing unauthenticated, the
+        credential, every hive unable to publish its status, the swarm
+        controller unable to start, the swarm's Grafana without any login at
+        all, its collector pushing unauthenticated, the
         queue's responder unable to start, and the homeserver unable to start.
         Each secret has exactly one route and this is the producer's
         end of it, so the honest reason to set it false is a deployment
@@ -223,6 +229,21 @@ in
           fi
         '') hiveNames}
 
+        # Each hive's own client, which that hive's hive-c0re reads under its
+        # own certificate. The nix half of
+        # `swarm_secret_client::queue::hive_client_path`.
+        ${lib.concatMapStringsSep "\n" (hive: ''
+          src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${hiveClientId hive}.secret"}
+          if [ -s "$src" ]; then
+            bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/oidc/client"} \
+              value=@"$src"
+            published=$((published + 1))
+          else
+            echo "no minted secret at $src yet; the path unit will re-run this" >&2
+            skipped=$((skipped + 1))
+          fi
+        '') hiveNames}
+
         ${lib.concatMapStringsSep "\n" (id: ''
           src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${id}.secret"}
           if [ -s "$src" ]; then