mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller: keep retrying the forge webhook secret and registration after a startup outage

Browse source 
The webhook secret slot is filled in the background, on the forge token's
capped unbounded backoff, when the store can't answer at start. The hook
registration waits for the secret and retries on the same backoff instead
of giving up after five tries.

Closes #6101
This commit is contained in:
atlas 2026-11-06 04:05:39 +01:00 • committed by mara
commit 68b38a9d44
1 changed files with 284 additions and 114 deletions
Download patch file Download diff file Expand all files Collapse all files

55
swarm-controller/src/forge.rs
View file

@ -317,40 +317,53 @@ const BACKGROUND_FIRST: std::time::Duration = std::time::Duration::from_secs(30)
/// The longest the background re-read waits between attempts.
const BACKGROUND_CAP: std::time::Duration = std::time::Duration::from_mins(5);
/// Fill `slot` with the first value `attempt` yields: after
/// [`BACKGROUND_FIRST`], then at 60s, doubling, capped at [`BACKGROUND_CAP`].
/// Never gives up. A failure is logged at `info` once the next wait is the cap,
/// so a controller that never gets its token says so once per
/// Run `attempt` until it succeeds: after `min_delay`, doubling, capped at
/// [`BACKGROUND_CAP`]. Never gives up. A failure is logged at `info` once the
/// next wait is the cap, so work that never succeeds says so once per
/// [`BACKGROUND_CAP`] without a line per early retry, and at `debug` before
/// that. The caller logs the first failure and the recovery.
async fn fill_when_ready<T, F, Fut>(slot: &OnceLock<T>, attempt: F) -> &T
/// that. The caller logs the first failure and the recovery. `what` names the
/// thing waited for in those lines.
pub(crate) async fn retry_unbounded<T, F, Fut>(
what: &str,
min_delay: std::time::Duration,
attempt: F,
) -> T
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T>>,
{
tokio::time::sleep(BACKGROUND_FIRST).await;
let value = attempt
attempt
.retry(
ExponentialBuilder::default()
.with_min_delay(BACKGROUND_FIRST * 2)
.with_min_delay(min_delay)
.with_max_delay(BACKGROUND_CAP)
.without_max_times(),
)
.notify(|e, wait| {
if wait >= BACKGROUND_CAP {
tracing::info!(
error = %format!("{e:#}"),
"still no forge access: the controller's forge token is unavailable"
);
tracing::info!(what, error = %format!("{e:#}"), "still unavailable; retrying");
} else {
tracing::debug!(
error = %format!("{e:#}"),
"the controller's forge token is still unavailable"
);
tracing::debug!(what, error = %format!("{e:#}"), "unavailable; retrying");
}
})
.await
.expect("a retry with no cap on its tries returns only once it succeeds");
.expect("a retry with no cap on its tries returns only once it succeeds")
}
/// Fill `slot` with the first value `attempt` yields: after
/// [`BACKGROUND_FIRST`], then at 60s, doubling, capped at [`BACKGROUND_CAP`],
/// as [`retry_unbounded`] does. `what` names the value in its log lines.
pub(crate) async fn fill_when_ready<'a, T, F, Fut>(
slot: &'a OnceLock<T>,
what: &str,
attempt: F,
) -> &'a T
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T>>,
{
tokio::time::sleep(BACKGROUND_FIRST).await;
let value = retry_unbounded(what, BACKGROUND_FIRST * 2, attempt).await;
slot.get_or_init(|| value)
}
@ -400,7 +413,7 @@ pub async fn enable(
);
tokio::spawn(async move {
let client = Arc::clone(
fill_when_ready(&slot, || async {
fill_when_ready(&slot, "the controller's forge token", || async {
Client::connect(&url).await.map(Arc::new)
})
.await,
@ -2330,7 +2343,7 @@ mod tests {
let slot = OnceLock::new();
let start = tokio::time::Instant::now();
let mut calls = 0;
let got = fill_when_ready(&slot, || {
let got = fill_when_ready(&slot, "a test value", || {
calls += 1;
let n = calls;
async move {
@ -2352,7 +2365,7 @@ mod tests {
let slot = OnceLock::new();
let start = tokio::time::Instant::now();
let mut calls = 0;
fill_when_ready(&slot, || {
fill_when_ready(&slot, "a test value", || {
calls += 1;
let n = calls;
async move {