swarm-controller: bound each periodic forge pass with a deadline
forgejo-api 0.11.1 builds its reqwest client with no timeout and offers no way to hand it a caller-built one (the `client` field is private, `new`/`with_user_agent` are the only constructors, `Endpoint` is sealed). A forge that accepts a connection and then stalls therefore held each of the six periodic reconcile loops on one request forever, with no log line. Wrap each loop's per-tick work in `forge::within_pass_deadline` (`PASS_DEADLINE`, 2 min, under the 5-minute interval): on elapsed the pass is dropped, a warn is logged, and the loop waits for its next tick. Every pass is an idempotent reconcile, so abandoning one mid-way leaves nothing the next pass does not redo. Loops: config-pr poll, runner registration token, hive forge tokens, agent forge tokens (observe + legacy sweep), docs grants, swarm forge objects. Refs #5060
This commit is contained in:
parent
dd172d9cf8
commit
45f0582163
7 changed files with 85 additions and 9 deletions
|
|
@ -231,6 +231,31 @@ where
|
|||
}
|
||||
}
|
||||
|
||||
/// The longest one periodic forge pass may run before it is abandoned until
|
||||
/// its next tick. forgejo-api's client has no request timeout, so without
|
||||
/// this a forge that accepts a connection and never answers holds the pass,
|
||||
/// and the loop running it, forever. Under every pass's 5-minute interval and
|
||||
/// far over what a pass that is making progress takes.
|
||||
pub(crate) const PASS_DEADLINE: std::time::Duration = std::time::Duration::from_mins(2);
|
||||
|
||||
/// Run `fut`, one periodic pass named `pass` in the log, under
|
||||
/// [`PASS_DEADLINE`]. `None` means it was abandoned at the deadline; that is
|
||||
/// logged here, and the caller moves on to its next tick.
|
||||
pub(crate) async fn within_pass_deadline<T>(
|
||||
pass: &str,
|
||||
fut: impl std::future::Future<Output = T>,
|
||||
) -> Option<T> {
|
||||
let result = tokio::time::timeout(PASS_DEADLINE, fut).await.ok();
|
||||
if result.is_none() {
|
||||
tracing::warn!(
|
||||
pass,
|
||||
deadline_s = PASS_DEADLINE.as_secs(),
|
||||
"forge pass: no answer within the deadline; abandoned, retrying next tick"
|
||||
);
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Put a client for `url` into `slot` and hand it to `on_ready`: at once when
|
||||
/// the store has the token, otherwise from a background task that re-reads it
|
||||
/// until it does. Either way the daemon starts, and every forge-backed path
|
||||
|
|
@ -1455,6 +1480,19 @@ mod tests {
|
|||
assert_eq!(start.elapsed().as_secs(), 30 + 60 + 120);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn a_pass_that_never_answers_is_abandoned_at_the_deadline() {
|
||||
let start = tokio::time::Instant::now();
|
||||
let got = within_pass_deadline("test", std::future::pending::<()>()).await;
|
||||
assert_eq!(got, None);
|
||||
assert_eq!(start.elapsed(), PASS_DEADLINE);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn a_pass_that_answers_returns_its_result() {
|
||||
assert_eq!(within_pass_deadline("test", async { 7 }).await, Some(7));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn page_search_results_pages_past_a_full_first_page() {
|
||||
#[derive(Clone)]
|
||||
|
|
|
|||
Loading…
Reference in a new issue