swarm-controller gains forge access when the token arrives, without a restart

A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):

- `forge::enable` tries the store once at start. On failure it logs once and
  re-reads in the background with backoff (30s doubling to a 5-minute cap,
  no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
  poll and the webhook registration start from that moment, through
  `start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
  "no forge" before and work after.

A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.

Refs #4998

swarm-controller/src/queue_identity.rs

@@ -25,7 +25,7 @@ const PREFIX: &str = "SWARM_CONTROLLER";
 /// Seconds to wait after each failed fetch before the next one. About a minute
 /// in all, which rides out a store restarting alongside this daemon. A longer
 /// outage fails the start, and the unit's `Restart=` tries again from the top.
-pub(crate) const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
+const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
 
 static IDENTITY: OnceLock<Option<QueueConfig>> = OnceLock::new();
 
@@ -61,9 +61,7 @@ pub async fn init() -> Result<()> {
         None => None,
         Some(coordinates) => {
             let delays = RETRY_DELAYS_S.map(Duration::from_secs);
-            Some(coordinates.with_secret(
-                fetch_with_retry(&delays, "the queue client secret", fetch_secret).await?,
-            ))
+            Some(coordinates.with_secret(fetch_with_retry(&delays, fetch_secret).await?))
         }
     };
     IDENTITY
@@ -120,13 +118,8 @@ async fn fetch_secret() -> Result<String> {
 }
 
 /// Run `attempt`, waiting each of `delays` after a failure, then once more.
-/// Answers with the first success, or the last attempt's error. `what` names the
-/// secret in the log line and the error.
-pub(crate) async fn fetch_with_retry<F, Fut>(
-    delays: &[Duration],
-    what: &str,
-    mut attempt: F,
-) -> Result<String>
+/// Answers with the first success, or the last attempt's error.
+async fn fetch_with_retry<F, Fut>(delays: &[Duration], mut attempt: F) -> Result<String>
 where
     F: FnMut() -> Fut,
     Fut: Future<Output = Result<String>>,
@@ -138,7 +131,7 @@ where
                 tracing::warn!(
                     error = %format!("{e:#}"),
                     retry_in = ?delay,
-                    "fetching {what} from the swarm secret store failed"
+                    "fetching the queue client secret from the swarm secret store failed"
                 );
                 tokio::time::sleep(*delay).await;
             }
@@ -146,7 +139,7 @@ where
     }
     attempt().await.with_context(|| {
         format!(
-            "fetching {what} from the swarm secret store, after {} attempts",
+            "fetching the queue client secret from the swarm secret store, after {} attempts",
             delays.len() + 1
         )
     })
@@ -224,7 +217,7 @@ mod tests {
     #[tokio::test]
     async fn a_store_that_comes_back_within_the_window_is_waited_for() {
         let mut calls = 0;
-        let got = fetch_with_retry(&[Duration::ZERO; 3], "a secret", || {
+        let got = fetch_with_retry(&[Duration::ZERO; 3], || {
             calls += 1;
             let n = calls;
             async move {
@@ -243,7 +236,7 @@ mod tests {
     #[tokio::test]
     async fn a_store_that_stays_down_fails_after_one_attempt_per_delay_plus_one() {
         let mut calls = 0;
-        let err = fetch_with_retry(&[Duration::ZERO; 3], "a secret", || {
+        let err = fetch_with_retry(&[Duration::ZERO; 3], || {
             calls += 1;
             async { bail!("store sealed") }
         })