RESTORE_TIMEOUT bounded only retry_down's restore. The boot loop in main
(primary and secondaries) and bring_up_secondary_with_retry's backoff
retries called bring_up_account with no bound, so a restore that never
answered held up every account after it, and the retry loop that would
recover it never started. A hung primary also never reached the fatal Err
path, so systemd never restarted the daemon.
The boot loop moves out of main into bring_up_all, which takes the restore
as a closure (the same seam retry_loop already has) so a test can stand in
a restore that never resolves. Every restore, at boot, on a boot retry and
in retry_once, now goes through within_restore_timeout, so RESTORE_TIMEOUT
is applied in one place. A timeout is an Err naming the account: fatal for
the primary, retried for a secondary, as any other restore error.
bring_up_account had no caller left and is removed.
Closes#5266
Three follow-ups to argus's review of #5237:
- A down secondary's retry runs `restore` under RESTORE_TIMEOUT (2 min).
Tries run one per tick on one future, so a restore that never resolved
held back every other down account's retries for good; a try that
outlasts the timeout now counts as a failure like any other.
- `client::build_and_restore` warns about a rejected token only for the
primary, which exits right after. For a secondary it logs at debug: the
retry loop already warns once per outage (`DownAccounts::failed` is news
on the first failure and on a changed reason, and an account that came
up starts afresh), so the extra warn repeated on every try.
- `a_secondary_that_comes_up_late_does_not_wipe_the_todos` runs the late
bring-up in a child process of the test binary with HIVE_AGENT_SOCKET
pointed at the test's listener. In-process the variable was unset, so a
wipe placed back in the bring-up path was a no-op and the test passed.
New tests: a_second_rejection_in_a_row_does_not_warn_again (captured logs, alone
in a child process since tracing caches callsite interest process-wide;
a stub whoami answering M_UNKNOWN_TOKEN, the real build_and_restore),
a_failure_after_the_account_came_up_is_news_again, and
an_account_whose_restore_hangs_does_not_hold_back_the_others (paused time).
The startup todo wipe runs once in main, before any sync loop, so a
secondary that comes up later cannot clear the todos the running
accounts' sweeps already deduped. Nothing tested that.
main now reads HIVE_AGENT_SOCKET once at startup and hands the path to
wake::clear_all_todos, the only way left to wipe the whole set;
send_todo_clear clears one key only. bring_up_account is split into
restore_account (everything that talks to the network) and the
synchronous account_sync_loop, and the retry path restores with the
former and builds the loop itself. Neither the loop nor the retry path
is given the socket.
The new test runs a late secondary's retry and its sync loop against a
listener standing in for the harness socket, after startup's wipe, and
asserts the harness saw exactly one wipe. It pins that seam, not every
place in restore_account a wipe could be added.
The retry loop's body was only reachable through bring_up_account, which
needs a live homeserver, so nothing exercised what happens when a down
secondary comes up or fails, or that the Registry's new RwLock is never
held across the network call. The unit tests covered the pure decision
(DownAccounts, retry_delay) only.
retry_down now delegates to retry_loop / retry_once, which take the
bring-up step and the snapshot path as parameters; retry_down passes
bring_up_account and the real path, so behaviour is unchanged. Tests use
offline matrix-sdk clients and fake bring-ups to show that:
- a secondary that comes up is inserted, leaves the down set, has its
sync loop sent on and is published in the snapshot, while the primary
and the other accounts stay the same clients;
- one that fails (error or no token) stays down, is not published, and
waits twice as long next time; one that is not yet due is not tried;
- readers are not blocked while a bring-up is pending;
- inserts under four busy reader threads neither deadlock nor lose an
account;
- the loop, under paused time, retries at 2, 6, 14 and 30 minutes.
tokio's test-util feature is added as a dev-dependency for the last.
Audit follow-ups on the secondary-account retry.
The accounts snapshot is written by the heartbeat task and, now, by the
retry loop. Both used one fixed tmp path, so concurrent writers could
clobber it or fail the rename. Each write now names its own tmp file
(pid + counter); a test hammers it from four threads.
A permanently bad secondary logged three warnings per attempt, every 16
minutes, for as long as the daemon ran. DownAccounts::failed now reports
whether a failure is news (first, or a different reason from the last);
only that is a warning, repeats are debug. The two per-attempt warnings
in client.rs and credential.rs are covered by that line, which names the
reason, so the store-token one drops to debug.
The comments in client.rs and a credential.rs test, and persistence.md,
still described the old skip-until-restart behaviour or implied that a
store-listed account without a credential is retried here; that case is
handled by the linked-name set changing, not by this loop.
The daemon's two-minute refresh compared the store's linked account
names with the names it booted with. A secondary that was named but did
not come up - no token at bring-up, a token the homeserver rejected
(M_UNKNOWN_TOKEN), or a homeserver still down after the ~52 s boot
retries - kept its name in the set, so nothing ever noticed it again:
it stayed down until the daemon restarted for some other reason, and
an operator re-linking a rejected account changed no name.
Each such secondary is now tried again on the same two-minute period,
one per tick, after one refresh and then doubling waits capped at
sixteen minutes. One that comes up joins the registry (now behind a
lock) and its sync loop joins the running set; the primary and the
other accounts are not restarted. The store is still only read.
The startup todo wipe moves out of bring_up_account to run once before
any sync loop: run per account, a late secondary would have cleared the
todos the running accounts' sweeps had already deduped.
Part of #5017 (slice M).
An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).
swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.
hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.
Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.
Part of #5022 (S1+S2, with the S4 docs).
hive-matrix-daemon now learns which external matrix accounts it has from
the swarm secret store, under the agent's own certificate, and the hive
push chain for matrix is gone.
The daemon lists swarm/agents/<agent>/matrix/ (the `list` its policy
grants on its own metadata subtree), reads each account's homeserver
from its credential, and brings the accounts up with their tokens from
the store. Every two minutes it lists again and exits with 75 when the
set of linked accounts changed; the unit restarts on 75 without counting
a failure. A listed name whose credential reads as absent is skipped and
logged once. At start it removes the matrix-token-<a> /
matrix-account-<a>.json pairs a hive delivered (a sidecar marks a pair
as delivered; a declared tokenFile keeps its token).
Removed: CredentialNotice and the $SWARM.credential.* subject and NATS
grant, the controller's publish and its queue precondition on the PUT
route, hive-c0re's credential subscription arm and workers/credential.rs,
priv_client::write_agent_matrix_token, hive-priv's WriteAgentMatrixToken
and its helpers, and the daemon's state-dir account discovery.
Kept: WriteAgentGithubToken and the external-forge path
(WriteAgentExtraForgeAccount, extra_forges.rs) are untouched, and a
declared matrixAccounts tokenFile is still read when the store has no
token for that account.
Refs #4348
The daemon reads each account's token from `swarm/agents/<agent>/matrix/`
as the agent itself, inside its own container, and falls back to the file
only when the store has none. This is #4519's read, without its `main`
carve-out: the swarm now mints `main` there and no hive writes the file.
The daemon unit gets the agent's store identity, spelled the way
forge-token.nix spells it. A timer re-starts it while it is down: a token
the swarm mints or replaces in the store changes no file, so the path
watcher never fires for it, and a daemon that exited on a replaced token
would otherwise stay down until the container restarts.
meta.rs writes each agent's flake, and it still named the pre-move
`hyperhive.*` paths — so every agent rebuild would print a rename
deprecation warning about a line no human wrote and no operator could fix.
A warning nobody can act on trains everyone to ignore the ones that matter,
which is the whole value of the alias shims.
Repoints the FORWARDED_VAR_OPTIONS table and every other emitted option
assignment (otel.*, docs.source, claudeCodePath, github.enable, user.name,
claudeMemoryMaxBytes) to `services.hyperhive.agent.*`, with the test
expectations that pin the rendered text. The flake input named `hyperhive`
(`hyperhive.url`, `hyperhive.inputs.nixpkgs.follows`,
`hyperhive.nixosConfigurations.*`), hive-tier `services.hyperhive.*` paths,
and the `@hyperhive.local` git identity share the word and are untouched.
Also repoints the same option paths where they appear in comments, rustdoc
and runtime message strings across the other crates — a refusal message
naming `hyperhive.allowedRecipients` sends an operator to a path that will
stop existing. Prose under docs/ is deliberately not in this commit.
Refs #4473
Third and last of #2860's agent-facing URL fallbacks. The operator's
ruling was "any special casing is done on the nix side - same binaries,
no hard coded fallback", so the default is deleted rather than replaced.
Every layer guessed the same wrong thing, and each guess was only ever
correct for a process sharing the host netns:
- nix/agent-modules/matrix.nix: matrixUrlDefault = localhost:8008, both
as the option's default and as a sentinel the daemon unit compared
against to decide whether to write HIVE_MATRIX_URL. Now nullOr str,
default null, the guard is != null, and the doc says what forge.url's
already says: null means "no matrix", not "guess one".
- nix/host-modules/hive-c0re/environment.nix: forwarded
http://127.0.0.1:<port> when no gatewayHost was set. hive-c0re shares
the host netns so it reads as harmless, but the value is handed to
agents, which do not -- there it names the agent itself. Now forwarded
only when there is a gateway vhost to name, matching the guard
HIVE_MATRIX_PUBLIC_URL already uses twelve lines below.
- hive-matrix-mcp: paths::DEFAULT_HOMESERVER was the same address
compiled in, so dropping the nix defaults alone would have left the
daemon dialling loopback inside the agent's own netns -- the very bug,
one layer down. homeserver_url() is now Option, and an account with no
homeserver is skipped with a log, exactly as one with no token is.
discover_token_accounts already refused to guess for the same reason.
Two comments taught the assumption back to the next reader ("shared host
netns means every agent container resolves localhost to the same
machine"); both now say which side of the netns boundary they describe.
MATRIX_HTTP keeps its value -- hive-c0re really does share the host
netns -- but no longer claims agents do.
Gated with nix eval against the extended agent-base config, as a pair:
with no url set the daemon unit carries no HIVE_MATRIX_URL, and with one
set it carries exactly that. Either check alone passes on a broken guard.
On a transient error (network blip, DNS not ready, homeserver 5xx) a
secondary account's bring-up was immediately skipped for the entire
daemon lifetime. This bit janet's catgirl account repeatedly when the
host DNS resolver wasn't ready at daemon start — the account would
silently disappear until the next restart.
Add a PermanentBringUpError sentinel in client.rs so callers can
distinguish M_UNKNOWN_TOKEN (stale/expired token — permanent, don't
retry) from transient network/homeserver errors.
In main.rs, replace the immediate skip with a bounded retry loop for
secondary accounts: up to 4 attempts with 2s/5s/15s/30s backoffs
(~52s total wait). On a transient error the daemon now stays alive
serving the primary and any other healthy accounts while the failing
secondary gets another chance. Permanent failures (PermanentBringUpError)
still skip immediately with no retry.
The primary account keep its existing behaviour: fatal on non-permanent
error so systemd restarts the whole daemon (systemd is the right retry
mechanism for primary bring-up failure).
when the homeserver rejects the token (stale session after state wipe or
expiry), the daemon now deletes the token file + matrix-sdk state dir
and exits cleanly instead of crash-looping.
hive-c0re's matrix::ensure_all sweep now runs periodically every 30
minutes (in addition to startup) so deleted token files get
re-provisioned without requiring a hive-c0re restart. the systemd.paths
watcher on the token file path then restarts the daemon with a fresh
token.
