matrix: back up agents' room keys with a controller-minted recovery passphrase

An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).

swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.

hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.

Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.

Part of #5022 (S1+S2, with the S4 docs).

hive-matrix-mcp/src/client.rs

@@ -12,17 +12,20 @@
 //! token + user/device as its appservice and handed us the bearer. matrix-sdk's `restore_session` with a constructed
 //! `MatrixSession` skips the login flow entirely.
 //!
-//! E2EE is enabled via `with_encryption_settings(EncryptionSettings::default())`.
-//! Crypto keys are persisted in the sqlite store under `state_dir`
-//! (survives container restarts, lost on `--purge`).
+//! E2EE is enabled via `with_encryption_settings`. Crypto keys are persisted
+//! in the sqlite store under `state_dir` (survives container restarts, lost on
+//! `--purge`).
 //!
 //! Cross-signing is bootstrapped best-effort after session restore
 //! (`ensure_cross_signing`) so the agent's device stops showing as
 //! "unverified" in other users' Matrix clients: the in-hive tuwunel accepts
 //! the initial device-signing-key upload on the bearer token alone (no UIAA
 //! re-auth). A homeserver that DID require UIAA just fails the call, which is
-//! swallowed — the session stays exactly as unverified as before. Automatic
-//! key backup is still left at its default (disabled).
+//! swallowed — the session stays exactly as unverified as before.
+//!
+//! Server-side key backup is turned on or recovered by `ensure_recovery`, with
+//! the passphrase `swarm-controller` mints per account, so a crypto store lost
+//! to a purge or a token reset gets its history back.
 
 use std::path::Path;
 
@@ -30,13 +33,13 @@ use anyhow::{Context, Result, anyhow};
 use matrix_sdk::{
     Client, SessionMeta, SessionTokens,
     authentication::matrix::MatrixSession,
-    encryption::EncryptionSettings,
+    encryption::{BackupDownloadStrategy, EncryptionSettings},
     ruma::{OwnedDeviceId, OwnedUserId},
 };
 use serde::Deserialize;
 use tokio::fs;
 
-use crate::credential::Token;
+use crate::credential::{Passphrase, Token};
 
 /// Sentinel returned when `build_and_restore` detects that the token is
 /// permanently invalid (`M_UNKNOWN_TOKEN`). The token has already been
@@ -146,7 +149,10 @@ pub async fn build_and_restore(
     let client = Client::builder()
         .homeserver_url(homeserver)
         .sqlite_store(state_dir, None)
-        .with_encryption_settings(EncryptionSettings::default())
+        .with_encryption_settings(EncryptionSettings {
+            backup_download_strategy: BackupDownloadStrategy::OneShot,
+            ..EncryptionSettings::default()
+        })
         .build()
         .await
         .with_context(|| format!("build matrix client for {homeserver}"))?;
@@ -220,11 +226,11 @@ async fn whoami(homeserver: &str, token: &str) -> Result<(OwnedUserId, OwnedDevi
 /// the failure is logged and swallowed, leaving the session no worse off than
 /// before. Never breaks account bring-up.
 ///
-/// Known gap (self-verify only): this signs the *current* device against a
-/// freshly-bootstrapped identity. A device re-provisioned under an already
-/// existing identity (new `device_id`, e.g. after a token reissue) would need
-/// the cross-signing secret keys via secret storage to re-sign itself —
-/// tracked as the longer-term mutual-verification follow-up.
+/// This only signs the *current* device against a freshly-bootstrapped
+/// identity. A device under an already existing identity is a no-op here; it
+/// is signed by [`ensure_recovery`]'s `recover`, which imports the
+/// self-signing key from secret storage, and stays unverified for an account
+/// with no recovery passphrase.
 pub async fn ensure_cross_signing(client: &Client, account: &str) {
     match client
         .encryption()
@@ -247,6 +253,88 @@ pub async fn ensure_cross_signing(client: &Client, account: &str) {
     }
 }
 
+/// What [`ensure_recovery`] did.
+enum Backup {
+    /// This device was already connected to the account's backup.
+    AlreadyOn,
+    /// The server had no backup; one was created, with secret storage that
+    /// `passphrase` opens.
+    Enabled,
+    /// The server had a backup; the keys secret storage holds for it, and
+    /// for cross-signing if it holds those, were imported.
+    Restored,
+}
+
+/// Best-effort: turn on server-side key backup for this account, or recover
+/// from the backup it already has, with the account's recovery `passphrase`.
+///
+/// Run after [`ensure_cross_signing`], so a newly bootstrapped identity's
+/// private keys are local when `enable` exports them into secret storage, and
+/// an existing identity is left for `recover` to import. After a recover the
+/// room keys download in the background (`BackupDownloadStrategy::OneShot`).
+///
+/// Every failure is logged and swallowed, as in [`ensure_cross_signing`]: the
+/// account comes up as it would have without backup. That includes secret
+/// storage created with a different key, which this does not replace — the
+/// backup under it would be lost.
+pub async fn ensure_recovery(client: &Client, account: &str, passphrase: &Passphrase) {
+    match recover_or_enable(client, passphrase).await {
+        Ok(Backup::AlreadyOn) => {
+            tracing::debug!(account, "matrix key backup already on for this device");
+        }
+        Ok(Backup::Enabled) => tracing::info!(
+            account,
+            passphrase = %passphrase.path(),
+            "matrix recovery enabled: server-side key backup created"
+        ),
+        Ok(Backup::Restored) => tracing::info!(
+            account,
+            passphrase = %passphrase.path(),
+            "matrix recovery: recovered from the server-side key backup; room keys download in the background"
+        ),
+        Err(e) => tracing::warn!(
+            account,
+            passphrase = %passphrase.path(),
+            error = %format!("{e:#}"),
+            "matrix recovery failed; key backup stays off (non-fatal)"
+        ),
+    }
+}
+
+/// The body of [`ensure_recovery`], with its outcome returned rather than
+/// logged.
+async fn recover_or_enable(client: &Client, passphrase: &Passphrase) -> Result<Backup> {
+    let encryption = client.encryption();
+    // A device restored from its own store reconnects to the backup in a
+    // background task; reading `are_enabled` before it finishes says no.
+    encryption.wait_for_e2ee_initialization_tasks().await;
+    let backups = encryption.backups();
+    if backups.are_enabled().await {
+        return Ok(Backup::AlreadyOn);
+    }
+    let recovery = encryption.recovery();
+    if backups
+        .fetch_exists_on_server()
+        .await
+        .context("asking the homeserver for the account's key backup")?
+    {
+        recovery
+            .recover(passphrase.expose())
+            .await
+            .context("recovering from the key backup")?;
+        Ok(Backup::Restored)
+    } else {
+        // The recovery key `enable` returns is not kept: the passphrase opens
+        // the same secret storage.
+        recovery
+            .enable()
+            .with_passphrase(passphrase.expose())
+            .await
+            .context("enabling key backup")?;
+        Ok(Backup::Enabled)
+    }
+}
+
 /// Best-effort: set this account's matrix avatar from the rasterized icon
 /// PNG. The path comes from `HIVE_ICON_PNG` (a nix-built derivation the
 /// harness forwards into the daemon env; absent when no icon resolves).

hive-matrix-mcp/src/credential.rs

@@ -1,5 +1,6 @@
 //! The matrix access token as a value this process holds, and the two places
-//! it comes from; and the accounts the store links to this agent.
+//! it comes from; the accounts the store links to this agent; and each
+//! account's key-backup recovery passphrase.
 //!
 //! 🩸 **A secret is a path, not a value.** Nothing here writes the token
 //! anywhere, interpolates it into a command, or lets it reach a log line or an
@@ -179,6 +180,73 @@ impl Token {
     }
 }
 
+/// The passphrase that opens an account's server-side key backup, held in
+/// memory for as long as it takes to enable or recover with it.
+///
+/// ⚠️ **No `Debug` derive**, for the reason [`Token`] has none: with the token,
+/// this decrypts every message the account has backed up.
+pub struct Passphrase {
+    value: String,
+    path: String,
+}
+
+impl std::fmt::Debug for Passphrase {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("Passphrase")
+            .field("path", &self.path)
+            .field("value", &"<redacted>")
+            .finish()
+    }
+}
+
+impl Passphrase {
+    /// The bytes, for the one caller that hands them to matrix-sdk.
+    #[must_use]
+    pub fn expose(&self) -> &str {
+        &self.value
+    }
+
+    /// The store path it was read from — the only half of it that may be
+    /// printed.
+    #[must_use]
+    pub fn path(&self) -> &str {
+        &self.path
+    }
+
+    /// Read this agent's recovery passphrase for `account` out of the swarm
+    /// secret store, under this agent's own certificate.
+    ///
+    /// `Ok(None)` when this container was given no store, or the store holds
+    /// nothing at the path yet: `swarm-controller` mints it on a periodic
+    /// pass, so a freshly linked account can come up before it exists.
+    ///
+    /// # Errors
+    /// A name that is not a single path segment, an environment naming an
+    /// identity that cannot be read, a store that refuses the certificate or
+    /// the path, or a passphrase stored empty.
+    pub async fn from_store(agent: &str, account: &str) -> Result<Option<Self>> {
+        let path = matrix::recovery_path(agent, account)
+            .context("building this agent's recovery passphrase path in the store")?;
+        let Some(store) = connect(agent).await? else {
+            return Ok(None);
+        };
+        let Some(stored): Option<matrix::Recovery> = store
+            .read_optional(&path)
+            .await
+            .with_context(|| format!("reading {path} from the store"))?
+        else {
+            return Ok(None);
+        };
+        let value = stored.value.trim().to_owned();
+        if value.is_empty() {
+            return Err(anyhow!(
+                "the recovery passphrase at {path} in the store is empty"
+            ));
+        }
+        Ok(Some(Self { value, path }))
+    }
+}
+
 /// An account the store links to this agent: its name, and the homeserver
 /// stored beside its token.
 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -340,6 +408,24 @@ mod tests {
         );
     }
 
+    #[test]
+    fn formatting_a_passphrase_does_not_reveal_it() {
+        let p = Passphrase {
+            value: "SUPER_SECRET_PASSPHRASE".to_owned(),
+            path: "swarm/agents/a1/matrix-recovery/main".to_owned(),
+        };
+        let rendered = format!("{p:?}");
+        assert!(
+            !rendered.contains("SUPER_SECRET_PASSPHRASE"),
+            "got {rendered}"
+        );
+        // Control: the path is meant to survive.
+        assert!(
+            rendered.contains("swarm/agents/a1/matrix-recovery/main"),
+            "got {rendered}"
+        );
+    }
+
     /// The same property for the arm a reader actually reaches for: `Display`
     /// on the origin is what every message in this crate interpolates.
     #[test]

hive-matrix-mcp/src/main.rs

@@ -408,6 +408,28 @@ async fn account_token(cfg: &AccountCfg) -> Result<Option<Token>> {
     }
 }
 
+/// Read `account`'s recovery passphrase from the store and hand it to
+/// [`client::ensure_recovery`]. With none — no store, or `swarm-controller`
+/// has not minted it yet — key backup stays off until the next start.
+async fn ensure_recovery(client: &Client, account: &str) {
+    let found = match credential::agent_name() {
+        Some(agent) => credential::Passphrase::from_store(&agent, account).await,
+        None => Ok(None),
+    };
+    match found {
+        Ok(Some(passphrase)) => client::ensure_recovery(client, account, &passphrase).await,
+        Ok(None) => tracing::info!(
+            account,
+            "no matrix recovery passphrase in the store; key backup stays off"
+        ),
+        Err(e) => tracing::warn!(
+            account,
+            error = %format!("{e:#}"),
+            "could not read the matrix recovery passphrase; key backup stays off"
+        ),
+    }
+}
+
 /// Restore one account's client (when its token exists), install its
 /// message handler, and build its sync loop. Returns
 /// `Ok(Some((client, sync_loop)))` when the account came up, `Ok(None)`
@@ -450,6 +472,8 @@ async fn bring_up_account(
     // flagged "unverified" in other users' clients. Idempotent + swallows
     // failures (a UIAA-requiring homeserver) inside ensure_cross_signing.
     client::ensure_cross_signing(&client, &cfg.name).await;
+    // Best-effort, and after cross-signing: see `client::ensure_recovery`.
+    ensure_recovery(&client, &cfg.name).await;
 
     let sync_client = client.clone();
     let cb_client = client.clone();