swarm-queue-client: re-read a held OIDC client secret when the IdP refuses it

hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.

`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.

swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.

The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.

agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.

Part of #5017 (slice Q).

hive-c0re/src/swarm_queue.rs

@@ -13,7 +13,8 @@
 //! The queue's address, the token endpoint and the client id come from the
 //! environment. The client secret comes from the swarm's secret store, at
 //! `swarm_secret_client::queue::hive_client_path`, read under this hive's own
-//! certificate and held only in memory. `swarm-secret-publish`, on the host
+//! certificate, held only in memory and read again when authelia refuses it
+//! (`swarm_queue_client::stored_secret`). `swarm-secret-publish`, on the host
 //! that runs authelia, puts it there.
 //!
 //! Connects lazily on first use rather than at boot — nothing here
@@ -27,6 +28,7 @@ use std::path::PathBuf;
 use std::time::Duration;
 
 use anyhow::{Context, Result, anyhow, bail};
+use swarm_queue_client::stored_secret::StoredSecret;
 use swarm_queue_client::{ClientSecret, QueueConfig};
 use swarm_secret_client::queue::{self, HiveClientCredential};
 use tokio::sync::Mutex;
@@ -229,11 +231,14 @@ struct Coordinates {
 
 impl Coordinates {
     fn with_secret(self, secret: String) -> QueueConfig {
+        let stored = StoredSecret::new(self.client_id.clone(), secret, || async {
+            fetch_secret().await.map_err(Into::into)
+        });
         QueueConfig {
             url: self.url,
             token_endpoint: self.token_endpoint,
             client_id: self.client_id,
-            client_secret: ClientSecret::Value(secret),
+            client_secret: ClientSecret::Stored(stored),
             ca_file: self.ca_file,
         }
     }
@@ -327,14 +332,14 @@ mod tests {
         );
     }
 
-    #[test]
-    fn a_full_environment_carries_the_stored_secret_as_a_value() {
+    #[tokio::test]
+    async fn a_full_environment_carries_the_stored_secret_in_memory() {
         let cfg = coordinates_from(env(&FULL))
             .expect("complete")
             .expect("configured")
             .with_secret("s3cret".to_owned());
         assert_eq!(cfg.client_id, "hive-alpha");
-        assert!(matches!(&cfg.client_secret, ClientSecret::Value(v) if v == "s3cret"));
+        assert!(matches!(&cfg.client_secret, ClientSecret::Stored(s) if s.current() == "s3cret"));
     }
 
     /// The secret is not in the store when this daemon first asks, and is