swarm-queue-client: re-read a held OIDC client secret when the IdP refuses it

hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.

`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.

swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.

The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.

agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.

Part of #5017 (slice Q).

swarm-controller/src/queue_identity.rs

@@ -3,9 +3,10 @@
 //!
 //! The queue's address, the token endpoint and the client id come from the
 //! environment. The client secret comes from the swarm's secret store, at
-//! `swarm_secret_client::queue::controller_client_path`, read once at start
-//! under this daemon's own certificate and held only in memory — it is never
-//! on this host's disk. `swarm-secret-publish`, on the host that runs
+//! `swarm_secret_client::queue::controller_client_path`, read at start under
+//! this daemon's own certificate, held only in memory and read again when
+//! authelia refuses it (`swarm_queue_client::stored_secret`). It is never on
+//! this host's disk. `swarm-secret-publish`, on the host that runs
 //! authelia, puts it there.
 //!
 //! [`init`] runs once in `main`, before anything calls [`get`].
@@ -16,6 +17,7 @@ use std::sync::OnceLock;
 use std::time::Duration;
 
 use anyhow::{Context, Result, bail};
+use swarm_queue_client::stored_secret::StoredSecret;
 use swarm_queue_client::{ClientSecret, QueueConfig};
 use swarm_secret_client::queue::{self, ControllerCredential};
 
@@ -40,11 +42,14 @@ struct Coordinates {
 
 impl Coordinates {
     fn with_secret(self, secret: String) -> QueueConfig {
+        let stored = StoredSecret::new(self.client_id.clone(), secret, || async {
+            fetch_secret().await.map_err(Into::into)
+        });
         QueueConfig {
             url: self.url,
             token_endpoint: self.token_endpoint,
             client_id: self.client_id,
-            client_secret: ClientSecret::Value(secret),
+            client_secret: ClientSecret::Stored(stored),
             ca_file: self.ca_file,
         }
     }
@@ -201,14 +206,14 @@ mod tests {
         assert_eq!(c.ca_file, None);
     }
 
-    #[test]
-    fn the_fetched_secret_is_held_as_a_value_not_a_path() {
+    #[tokio::test]
+    async fn the_fetched_secret_is_held_in_memory_not_a_path() {
         let cfg = coordinates_from(env(&FULL))
             .expect("complete")
             .expect("configured")
             .with_secret("s3cret".to_owned());
         assert!(
-            matches!(&cfg.client_secret, ClientSecret::Value(v) if v == "s3cret"),
+            matches!(&cfg.client_secret, ClientSecret::Stored(s) if s.current() == "s3cret"),
             "{:?}",
             cfg.client_secret
         );