Watch
0
0
Fork
You've already forked hyperhive
0

hive-matrix-mcp: bound each secondary retry, warn once per outage

Three follow-ups to argus's review of #5237:

- A down secondary's retry runs `restore` under RESTORE_TIMEOUT (2 min).
  Tries run one per tick on one future, so a restore that never resolved
  held back every other down account's retries for good; a try that
  outlasts the timeout now counts as a failure like any other.
- `client::build_and_restore` warns about a rejected token only for the
  primary, which exits right after. For a secondary it logs at debug: the
  retry loop already warns once per outage (`DownAccounts::failed` is news
  on the first failure and on a changed reason, and an account that came
  up starts afresh), so the extra warn repeated on every try.
- `a_secondary_that_comes_up_late_does_not_wipe_the_todos` runs the late
  bring-up in a child process of the test binary with HIVE_AGENT_SOCKET
  pointed at the test's listener. In-process the variable was unset, so a
  wipe placed back in the bring-up path was a no-op and the test passed.

New tests: a_second_rejection_in_a_row_does_not_warn_again (captured logs, alone
in a child process since tracing caches callsite interest process-wide;
a stub whoami answering M_UNKNOWN_TOKEN, the real build_and_restore),
a_failure_after_the_account_came_up_is_news_again, and
an_account_whose_restore_hangs_does_not_hold_back_the_others (paused time).
This commit is contained in:
atlas 2026-10-14 16:53:57 +02:00 • committed by mara
commit 7b1435eaea
3 changed files with 225 additions and 20 deletions

View file

@ -108,13 +108,12 @@ pub async fn build_and_restore(
// or a secondary one, because a single bad secondary token must
// NOT take down the whole daemon (and with it every healthy
// account).
tracing::warn!(
credential = %token.origin(),
is_primary,
"matrix token rejected (M_UNKNOWN_TOKEN); discarding stale token"
);
token.discard_stale().await;
if is_primary {
tracing::warn!(
credential = %token.origin(),
"matrix token rejected (M_UNKNOWN_TOKEN); discarding stale token"
);
// Primary: also drop the matrix-sdk sqlite state keyed to the
// now-invalid session, then exit 0; the swarm's backfill
// re-mints the token and systemd restarts us once it
@ -137,6 +136,12 @@ pub async fn build_and_restore(
// token for the same device.
// Return a PermanentBringUpError so the caller can distinguish
// "retrying now won't help" from a transient network/DNS failure.
// Every try of a down secondary lands here, so the warning is the
// caller's, once per outage.
tracing::debug!(
credential = %token.origin(),
"matrix token rejected (M_UNKNOWN_TOKEN); discarded stale token"
);
return Err(anyhow::Error::new(PermanentBringUpError(
"matrix token rejected (M_UNKNOWN_TOKEN); discarded stale token, leaving account down".into(),
)));