swarm-bao-forge-publisher starts on the deploy that adds it
The publisher was wanted only by container@hive-forge.service. A switch starts a new unit only through the targets that want it, and the container is already running, so the deploy that introduced the publisher left it dead until the next container restart or boot; the path unit cannot help, because the mint's only-once guard never rewrites the file. - The publisher is also wanted by multi-user.target, like swarm-secret-publish. - A token file that is not there yet exits 0 instead of waiting 120s and failing: the path unit runs the publisher when the mint writes it, so a switch with the forge container down does not fail. TimeoutStartSec is 60. - swarm-controller loses its After= on the publisher: a switch issues the two jobs separately, so it ordered nothing there, and the background re-read covers the gap. - The controller's background re-read logs at info once the backoff is capped, one line per five minutes, so a forge host that never got its publisher leaf stays visible; earlier retries stay at debug. Refs #4998
This commit is contained in:
parent
2d2d7529fc
commit
597fdebaf3
4 changed files with 52 additions and 32 deletions
|
|
@ -183,21 +183,31 @@ pub fn url_from_env() -> Result<Option<url::Url>> {
|
|||
.map(Some)
|
||||
}
|
||||
|
||||
/// The longest the background re-read waits between attempts.
|
||||
const BACKGROUND_CAP: std::time::Duration = std::time::Duration::from_mins(5);
|
||||
|
||||
/// How long the background re-read waits before its `n`-th attempt: 30s,
|
||||
/// doubling, capped at 5 minutes.
|
||||
/// doubling, capped at [`BACKGROUND_CAP`].
|
||||
fn background_delay(n: u32) -> std::time::Duration {
|
||||
const FIRST_S: u64 = 30;
|
||||
const CAP_S: u64 = 300;
|
||||
std::time::Duration::from_secs(
|
||||
FIRST_S
|
||||
.saturating_mul(1u64.checked_shl(n).unwrap_or(u64::MAX))
|
||||
.min(CAP_S),
|
||||
.min(BACKGROUND_CAP.as_secs()),
|
||||
)
|
||||
}
|
||||
|
||||
/// Whether the `n`-th failed attempt is logged at `info` rather than `debug`:
|
||||
/// once the backoff is capped, so a controller that never gets its token says
|
||||
/// so once per [`BACKGROUND_CAP`] without a line per early retry.
|
||||
fn failure_logs_at_info(n: u32) -> bool {
|
||||
background_delay(n) >= BACKGROUND_CAP
|
||||
}
|
||||
|
||||
/// Fill `slot` with the first value `attempt` yields, sleeping
|
||||
/// [`background_delay`] before each attempt. Never gives up, and logs no
|
||||
/// failure: the caller logs the first one and the recovery.
|
||||
/// [`background_delay`] before each attempt. Never gives up. A failure is logged
|
||||
/// as [`failure_logs_at_info`] says; the caller logs the first one and the
|
||||
/// recovery.
|
||||
async fn fill_when_ready<T, F, Fut>(slot: &OnceLock<T>, mut attempt: F) -> &T
|
||||
where
|
||||
F: FnMut() -> Fut,
|
||||
|
|
@ -208,6 +218,10 @@ where
|
|||
tokio::time::sleep(background_delay(n)).await;
|
||||
match attempt().await {
|
||||
Ok(value) => return slot.get_or_init(|| value),
|
||||
Err(e) if failure_logs_at_info(n) => tracing::info!(
|
||||
error = %format!("{e:#}"),
|
||||
"still no forge access: the controller's forge token is unavailable"
|
||||
),
|
||||
Err(e) => tracing::debug!(
|
||||
error = %format!("{e:#}"),
|
||||
"the controller's forge token is still unavailable"
|
||||
|
|
@ -1402,6 +1416,13 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_failed_re_read_logs_at_info_only_once_the_backoff_is_capped() {
|
||||
assert!((0..4).all(|n| !failure_logs_at_info(n)));
|
||||
assert!(failure_logs_at_info(4));
|
||||
assert!(failure_logs_at_info(40));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_background_re_read_backs_off_from_30s_to_a_5_minute_cap() {
|
||||
let s = |n| background_delay(n).as_secs();
|
||||
|
|
|
|||
Loading…
Reference in a new issue