matrix: back up agents' room keys with a controller-minted recovery passphrase

An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).

swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.

hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.

Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.

Part of #5022 (S1+S2, with the S4 docs).

docs/agent-lifecycle/persistence.md

@@ -16,7 +16,11 @@ keeps its state, purging it doesn't.**
   destroy --purge`) is `DESTR0Y` plus wiping all of it — config
   history, claude credentials, `/state/` notes, everything. **No
   undo.** Only reach for this when you actually want the agent gone
-  for good.
+  for good. The agent's matrix encryption keys go with the state dir,
+  but an account whose key backup is on (see
+  [matrix](../integrations/matrix.md)) gets them back from its
+  homeserver when it next comes up, so its encrypted history stays
+  readable.
 
 Beyond that:
 

docs/integrations/matrix.md

@@ -306,12 +306,25 @@ Initial rollout settings:
   `matrix-sdk` so it can read encrypted rooms it's invited to even when
   this homeserver doesn't permit room encryption. matrix-sdk stores
   crypto keys in the per-agent sqlite store under the state dir; they persist across
-  restarts (lost on `--purge`). `read_room` decrypts via
+  restarts. `read_room` decrypts via
   `room.messages()` — UTD events surface as `event_type =
   "m.room.encrypted"` with `body = "[unable to decrypt]"`.
-  Cross-signing and automatic key backup aren't enabled for the first
+  The daemon bootstraps cross-signing on the bearer token alone, which
-  pass: static bearer-token bot accounts can't bootstrap cross-signing
+  works where the homeserver skips re-auth for a first key upload
-  without MSC3967.
+  (MSC3967, as tuwunel does).
+- **Key backup.** `swarm-controller` mints a random recovery passphrase
+  for every matrix account an agent has, at
+  `swarm/agents/<agent>/matrix-recovery/<account>` in the swarm secret
+  store, and never replaces it. When `hive-matrix-daemon` brings an
+  account up it reads that passphrase: with no backup on the account's
+  homeserver it turns on server-side key backup and secret storage
+  (log line `matrix recovery enabled`); with one there it recovers from
+  it (`recovered from the server-side key backup`), which downloads the
+  room keys and signs the new device with the account's existing
+  cross-signing identity. A crypto store lost to `--purge` or to a
+  primary-token reset therefore comes back with its history. With no passphrase
+  stored yet the account comes up without backup and picks it up on the
+  next daemon start. A failure is a `WARN` and never stops the daemon.
 
 ## Hive Matrix Space
 

docs/swarm/credentials.md

@@ -59,6 +59,7 @@ of the cell says how.
 | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `swarm/agents/<agent>/matrix/main`                          | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass                                                                  | the agent container itself, under the certificate its hive passed in                                                                                                                                                                                                                                                                                                                                                | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's                                                                                                                                                                                                                                                                  | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | `swarm/agents/<agent>/matrix/<account>`                     | `swarm-controller`                                                                                                                                                  | the agent container itself, under the certificate its hive passed in                                                                                                                                                                                                                                                                                                                                                | must be stated                                                                                                                                                                                                                                                                                                                                                       | must be stated                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| `swarm/agents/<agent>/matrix-recovery/<account>`            | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass                                                 | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up                                                                                                                                                                                                                                                                                               | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path                                                                                                                                                                                                                 | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once                                                                                                        | `swarm-controller`, under its own certificate                                                                                                                                                                                                                                                                                                                                                                       | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs                                                                                                                                                                                                                                                        | ✅ the controller reads it on every five-minute matrix pass                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
 | `swarm/controller/swarm-controller/oidc/client`             | authelia, at its first boot; `swarm-secret-publish` copies it in                                                                                                    | `swarm-controller`, under its own certificate, at start and on refusal                                                                                                                                                                                                                                                                                                                                              | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit                                                                                                                                                                                                                                                                            | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | `swarm/controller/swarm-controller/forge/token`             | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start                                                                                                                                                                                                                                                                                                                                                             | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file                                                                                                                                                                                           | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart                                                                                                                                                                                                                                                                                                                                                                                                                                     |

docs/trust-boundary/security.md

@@ -62,7 +62,9 @@ public handle crosses that boundary; the token never does.
 A hive reads its agents' credentials out of the swarm's secret store with its
 own certificate. **Every hive's policy grants read on every agent's
 credentials**, not only on the agents it hosts — so a compromised hive can read
-the matrix token of an agent running on a different hive.
+the matrix token of an agent running on a different hive. That includes each
+account's key-backup recovery passphrase, which with the token decrypts the
+agent's whole encrypted history, not only messages from then on.
 
 That's deliberate and it's the interim state, not the intent. An agent's
 credential path doesn't name the hive hosting it (agents move between hives),

hive-matrix-mcp/src/client.rs

@@ -12,17 +12,20 @@
 //! token + user/device as its appservice and handed us the bearer. matrix-sdk's `restore_session` with a constructed
 //! `MatrixSession` skips the login flow entirely.
 //!
-//! E2EE is enabled via `with_encryption_settings(EncryptionSettings::default())`.
+//! E2EE is enabled via `with_encryption_settings`. Crypto keys are persisted
-//! Crypto keys are persisted in the sqlite store under `state_dir`
+//! in the sqlite store under `state_dir` (survives container restarts, lost on
-//! (survives container restarts, lost on `--purge`).
+//! `--purge`).
 //!
 //! Cross-signing is bootstrapped best-effort after session restore
 //! (`ensure_cross_signing`) so the agent's device stops showing as
 //! "unverified" in other users' Matrix clients: the in-hive tuwunel accepts
 //! the initial device-signing-key upload on the bearer token alone (no UIAA
 //! re-auth). A homeserver that DID require UIAA just fails the call, which is
-//! swallowed — the session stays exactly as unverified as before. Automatic
+//! swallowed — the session stays exactly as unverified as before.
-//! key backup is still left at its default (disabled).
+//!
+//! Server-side key backup is turned on or recovered by `ensure_recovery`, with
+//! the passphrase `swarm-controller` mints per account, so a crypto store lost
+//! to a purge or a token reset gets its history back.
 
 use std::path::Path;
 
@@ -30,13 +33,13 @@ use anyhow::{Context, Result, anyhow};
 use matrix_sdk::{
     Client, SessionMeta, SessionTokens,
     authentication::matrix::MatrixSession,
-    encryption::EncryptionSettings,
+    encryption::{BackupDownloadStrategy, EncryptionSettings},
     ruma::{OwnedDeviceId, OwnedUserId},
 };
 use serde::Deserialize;
 use tokio::fs;
 
-use crate::credential::Token;
+use crate::credential::{Passphrase, Token};
 
 /// Sentinel returned when `build_and_restore` detects that the token is
 /// permanently invalid (`M_UNKNOWN_TOKEN`). The token has already been
@@ -146,7 +149,10 @@ pub async fn build_and_restore(
     let client = Client::builder()
         .homeserver_url(homeserver)
         .sqlite_store(state_dir, None)
-        .with_encryption_settings(EncryptionSettings::default())
+        .with_encryption_settings(EncryptionSettings {
+            backup_download_strategy: BackupDownloadStrategy::OneShot,
+            ..EncryptionSettings::default()
+        })
         .build()
         .await
         .with_context(|| format!("build matrix client for {homeserver}"))?;
@@ -220,11 +226,11 @@ async fn whoami(homeserver: &str, token: &str) -> Result<(OwnedUserId, OwnedDevi
 /// the failure is logged and swallowed, leaving the session no worse off than
 /// before. Never breaks account bring-up.
 ///
-/// Known gap (self-verify only): this signs the *current* device against a
+/// This only signs the *current* device against a freshly-bootstrapped
-/// freshly-bootstrapped identity. A device re-provisioned under an already
+/// identity. A device under an already existing identity is a no-op here; it
-/// existing identity (new `device_id`, e.g. after a token reissue) would need
+/// is signed by [`ensure_recovery`]'s `recover`, which imports the
-/// the cross-signing secret keys via secret storage to re-sign itself —
+/// self-signing key from secret storage, and stays unverified for an account
-/// tracked as the longer-term mutual-verification follow-up.
+/// with no recovery passphrase.
 pub async fn ensure_cross_signing(client: &Client, account: &str) {
     match client
         .encryption()
@@ -247,6 +253,88 @@ pub async fn ensure_cross_signing(client: &Client, account: &str) {
     }
 }
 
+/// What [`ensure_recovery`] did.
+enum Backup {
+    /// This device was already connected to the account's backup.
+    AlreadyOn,
+    /// The server had no backup; one was created, with secret storage that
+    /// `passphrase` opens.
+    Enabled,
+    /// The server had a backup; the keys secret storage holds for it, and
+    /// for cross-signing if it holds those, were imported.
+    Restored,
+}
+
+/// Best-effort: turn on server-side key backup for this account, or recover
+/// from the backup it already has, with the account's recovery `passphrase`.
+///
+/// Run after [`ensure_cross_signing`], so a newly bootstrapped identity's
+/// private keys are local when `enable` exports them into secret storage, and
+/// an existing identity is left for `recover` to import. After a recover the
+/// room keys download in the background (`BackupDownloadStrategy::OneShot`).
+///
+/// Every failure is logged and swallowed, as in [`ensure_cross_signing`]: the
+/// account comes up as it would have without backup. That includes secret
+/// storage created with a different key, which this does not replace — the
+/// backup under it would be lost.
+pub async fn ensure_recovery(client: &Client, account: &str, passphrase: &Passphrase) {
+    match recover_or_enable(client, passphrase).await {
+        Ok(Backup::AlreadyOn) => {
+            tracing::debug!(account, "matrix key backup already on for this device");
+        }
+        Ok(Backup::Enabled) => tracing::info!(
+            account,
+            passphrase = %passphrase.path(),
+            "matrix recovery enabled: server-side key backup created"
+        ),
+        Ok(Backup::Restored) => tracing::info!(
+            account,
+            passphrase = %passphrase.path(),
+            "matrix recovery: recovered from the server-side key backup; room keys download in the background"
+        ),
+        Err(e) => tracing::warn!(
+            account,
+            passphrase = %passphrase.path(),
+            error = %format!("{e:#}"),
+            "matrix recovery failed; key backup stays off (non-fatal)"
+        ),
+    }
+}
+
+/// The body of [`ensure_recovery`], with its outcome returned rather than
+/// logged.
+async fn recover_or_enable(client: &Client, passphrase: &Passphrase) -> Result<Backup> {
+    let encryption = client.encryption();
+    // A device restored from its own store reconnects to the backup in a
+    // background task; reading `are_enabled` before it finishes says no.
+    encryption.wait_for_e2ee_initialization_tasks().await;
+    let backups = encryption.backups();
+    if backups.are_enabled().await {
+        return Ok(Backup::AlreadyOn);
+    }
+    let recovery = encryption.recovery();
+    if backups
+        .fetch_exists_on_server()
+        .await
+        .context("asking the homeserver for the account's key backup")?
+    {
+        recovery
+            .recover(passphrase.expose())
+            .await
+            .context("recovering from the key backup")?;
+        Ok(Backup::Restored)
+    } else {
+        // The recovery key `enable` returns is not kept: the passphrase opens
+        // the same secret storage.
+        recovery
+            .enable()
+            .with_passphrase(passphrase.expose())
+            .await
+            .context("enabling key backup")?;
+        Ok(Backup::Enabled)
+    }
+}
+
 /// Best-effort: set this account's matrix avatar from the rasterized icon
 /// PNG. The path comes from `HIVE_ICON_PNG` (a nix-built derivation the
 /// harness forwards into the daemon env; absent when no icon resolves).

hive-matrix-mcp/src/credential.rs

@@ -1,5 +1,6 @@
 //! The matrix access token as a value this process holds, and the two places
-//! it comes from; and the accounts the store links to this agent.
+//! it comes from; the accounts the store links to this agent; and each
+//! account's key-backup recovery passphrase.
 //!
 //! 🩸 **A secret is a path, not a value.** Nothing here writes the token
 //! anywhere, interpolates it into a command, or lets it reach a log line or an
@@ -179,6 +180,73 @@ impl Token {
     }
 }
 
+/// The passphrase that opens an account's server-side key backup, held in
+/// memory for as long as it takes to enable or recover with it.
+///
+/// ⚠️ **No `Debug` derive**, for the reason [`Token`] has none: with the token,
+/// this decrypts every message the account has backed up.
+pub struct Passphrase {
+    value: String,
+    path: String,
+}
+
+impl std::fmt::Debug for Passphrase {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("Passphrase")
+            .field("path", &self.path)
+            .field("value", &"<redacted>")
+            .finish()
+    }
+}
+
+impl Passphrase {
+    /// The bytes, for the one caller that hands them to matrix-sdk.
+    #[must_use]
+    pub fn expose(&self) -> &str {
+        &self.value
+    }
+
+    /// The store path it was read from — the only half of it that may be
+    /// printed.
+    #[must_use]
+    pub fn path(&self) -> &str {
+        &self.path
+    }
+
+    /// Read this agent's recovery passphrase for `account` out of the swarm
+    /// secret store, under this agent's own certificate.
+    ///
+    /// `Ok(None)` when this container was given no store, or the store holds
+    /// nothing at the path yet: `swarm-controller` mints it on a periodic
+    /// pass, so a freshly linked account can come up before it exists.
+    ///
+    /// # Errors
+    /// A name that is not a single path segment, an environment naming an
+    /// identity that cannot be read, a store that refuses the certificate or
+    /// the path, or a passphrase stored empty.
+    pub async fn from_store(agent: &str, account: &str) -> Result<Option<Self>> {
+        let path = matrix::recovery_path(agent, account)
+            .context("building this agent's recovery passphrase path in the store")?;
+        let Some(store) = connect(agent).await? else {
+            return Ok(None);
+        };
+        let Some(stored): Option<matrix::Recovery> = store
+            .read_optional(&path)
+            .await
+            .with_context(|| format!("reading {path} from the store"))?
+        else {
+            return Ok(None);
+        };
+        let value = stored.value.trim().to_owned();
+        if value.is_empty() {
+            return Err(anyhow!(
+                "the recovery passphrase at {path} in the store is empty"
+            ));
+        }
+        Ok(Some(Self { value, path }))
+    }
+}
+
 /// An account the store links to this agent: its name, and the homeserver
 /// stored beside its token.
 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -340,6 +408,24 @@ mod tests {
         );
     }
 
+    #[test]
+    fn formatting_a_passphrase_does_not_reveal_it() {
+        let p = Passphrase {
+            value: "SUPER_SECRET_PASSPHRASE".to_owned(),
+            path: "swarm/agents/a1/matrix-recovery/main".to_owned(),
+        };
+        let rendered = format!("{p:?}");
+        assert!(
+            !rendered.contains("SUPER_SECRET_PASSPHRASE"),
+            "got {rendered}"
+        );
+        // Control: the path is meant to survive.
+        assert!(
+            rendered.contains("swarm/agents/a1/matrix-recovery/main"),
+            "got {rendered}"
+        );
+    }
+
     /// The same property for the arm a reader actually reaches for: `Display`
     /// on the origin is what every message in this crate interpolates.
     #[test]

hive-matrix-mcp/src/main.rs

@@ -408,6 +408,28 @@ async fn account_token(cfg: &AccountCfg) -> Result<Option<Token>> {
     }
 }
 
+/// Read `account`'s recovery passphrase from the store and hand it to
+/// [`client::ensure_recovery`]. With none — no store, or `swarm-controller`
+/// has not minted it yet — key backup stays off until the next start.
+async fn ensure_recovery(client: &Client, account: &str) {
+    let found = match credential::agent_name() {
+        Some(agent) => credential::Passphrase::from_store(&agent, account).await,
+        None => Ok(None),
+    };
+    match found {
+        Ok(Some(passphrase)) => client::ensure_recovery(client, account, &passphrase).await,
+        Ok(None) => tracing::info!(
+            account,
+            "no matrix recovery passphrase in the store; key backup stays off"
+        ),
+        Err(e) => tracing::warn!(
+            account,
+            error = %format!("{e:#}"),
+            "could not read the matrix recovery passphrase; key backup stays off"
+        ),
+    }
+}
+
 /// Restore one account's client (when its token exists), install its
 /// message handler, and build its sync loop. Returns
 /// `Ok(Some((client, sync_loop)))` when the account came up, `Ok(None)`
@@ -450,6 +472,8 @@ async fn bring_up_account(
     // flagged "unverified" in other users' clients. Idempotent + swallows
     // failures (a UIAA-requiring homeserver) inside ensure_cross_signing.
     client::ensure_cross_signing(&client, &cfg.name).await;
+    // Best-effort, and after cross-signing: see `client::ensure_recovery`.
+    ensure_recovery(&client, &cfg.name).await;
 
     let sync_client = client.clone();
     let cb_client = client.clone();

swarm-controller/src/main.rs

@@ -2476,15 +2476,17 @@ fn configured_matrix_homeserver() -> Option<Arc<str>> {
         .map(Arc::from)
 }
 
-/// Start the matrix-account backfill and the hive sender-token pass when a
+/// Start the recovery passphrase pass, which needs only the store; the
-/// homeserver is configured, and the operator admin pass when the bridge is
+/// matrix-account backfill and the hive sender-token pass when a homeserver is
-/// too. Lifted out of `main` for `clippy::too_many_lines`.
+/// configured; and the operator admin pass when the bridge is too. Lifted out
+/// of `main` for `clippy::too_many_lines`.
 fn spawn_matrix_account_backfill(
     jobq: &Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
     homeserver: Option<Arc<str>>,
     hives: &[HiveEntry],
     auth: Option<Arc<auth::AuthBridge>>,
 ) {
+    matrix_account::recovery::spawn();
     let Some(base) = homeserver else {
         return;
     };

swarm-controller/src/matrix_account.rs

@@ -7,6 +7,8 @@
 //! [`hive_sender`] mints each **hive's** sender account the same way, sharing this module's helpers with [`agent_token`].
 //! [`operator_admin`] makes authelia's operators homeserver admins with the same
 //! appservice token.
+//! [`recovery`] mints the passphrase that opens each account's server-side key
+//! backup, for the swarm's accounts and external ones alike.
 //!
 //! **The external one** — [`put_matrix_account`] and everything under it — is
 //! an account somewhere else that an operator hands us a credential for: put it
@@ -37,6 +39,7 @@ use super::{AppState, error_problem, swarm_hive};
 pub mod agent_token;
 pub mod hive_sender;
 pub mod operator_admin;
+pub mod recovery;
 
 fn default_mode() -> String {
     "token".to_owned()

swarm-controller/src/matrix_account/recovery.rs

@@ -0,0 +1,285 @@
+//! A recovery passphrase for every matrix account of every agent, stored at
+//! `swarm_secret_client::matrix::recovery_path`: what the agent's matrix daemon
+//! turns on server-side key backup with, and recovers its keys with once its
+//! local crypto store is gone.
+//!
+//! Minted here because the agent's store policy is read-only. [`spawn`]'s pass
+//! runs at start and every [`RECONCILE_INTERVAL`] over every agent that holds a
+//! store identity — the same roster as [`super::agent_token`] — and every
+//! account listed under that agent's `matrix/` directory.
+//!
+//! ⚠️ **Never replaced.** A backup can only be opened with the passphrase it
+//! was enabled with, so a new one locks the agent out of everything backed up
+//! under the old. The write is a `cas=0` create, and a path that holds anything
+//! is left alone.
+
+use anyhow::{Context, Result};
+use serde::de::IgnoredAny;
+use swarm_secret_client::{Error, SecretStore, client::DEFAULT_CERT_MOUNT, matrix, policy};
+
+/// How often [`spawn`] re-checks every account.
+const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
+
+/// Bytes drawn from the kernel's CSPRNG per passphrase.
+const PASSPHRASE_BYTES: usize = 32;
+
+/// What [`ensure`] did at one path.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Outcome {
+    /// The path already held a passphrase.
+    Present,
+    /// This pass wrote one.
+    Minted,
+    /// Another writer's create landed between this pass's read and its own.
+    LostRace,
+}
+
+/// The two store calls [`ensure`] makes, so a test can stand in for the store.
+pub(crate) trait PassphraseStore {
+    /// Whether anything is stored at `path`, as [`SecretStore::read_optional`]
+    /// answers it. The value is decoded as [`IgnoredAny`] and never held.
+    fn exists(&self, path: &str) -> impl Future<Output = Result<bool, Error>> + Send;
+
+    /// As [`SecretStore::create`].
+    fn create(
+        &self,
+        path: &str,
+        value: &matrix::Recovery,
+    ) -> impl Future<Output = Result<(), Error>> + Send;
+}
+
+impl PassphraseStore for SecretStore {
+    async fn exists(&self, path: &str) -> Result<bool, Error> {
+        Ok(self.read_optional::<IgnoredAny>(path).await?.is_some())
+    }
+
+    async fn create(&self, path: &str, value: &matrix::Recovery) -> Result<(), Error> {
+        SecretStore::create(self, path, value).await
+    }
+}
+
+/// A passphrase: [`PASSPHRASE_BYTES`] from the kernel's CSPRNG, base64url
+/// without padding.
+///
+/// # Errors
+/// When the kernel will not supply randomness.
+fn generate() -> Result<matrix::Recovery> {
+    let mut bytes = [0u8; PASSPHRASE_BYTES];
+    getrandom::fill(&mut bytes)
+        .context("drawing a recovery passphrase from the kernel's CSPRNG")?;
+    Ok(matrix::Recovery {
+        value: base64::Engine::encode(&base64::engine::general_purpose::URL_SAFE_NO_PAD, bytes),
+    })
+}
+
+/// Make sure `path` holds a passphrase, writing one only if it holds nothing.
+///
+/// A create that fails is read back: if the path now holds something, a
+/// concurrent writer won and that is the passphrase.
+///
+/// # Errors
+/// When the store refuses the read, or the create fails and the path is still
+/// empty.
+pub(crate) async fn ensure(store: &impl PassphraseStore, path: &str) -> Result<Outcome> {
+    if store
+        .exists(path)
+        .await
+        .with_context(|| format!("reading {path}"))?
+    {
+        return Ok(Outcome::Present);
+    }
+    let Err(e) = store.create(path, &generate()?).await else {
+        return Ok(Outcome::Minted);
+    };
+    if store
+        .exists(path)
+        .await
+        .with_context(|| format!("reading {path} back after a failed create"))?
+    {
+        return Ok(Outcome::LostRace);
+    }
+    Err(anyhow::Error::new(e).context(format!("creating {path}")))
+}
+
+/// The account names listed under `agent`'s `matrix/` directory. A key ending
+/// in `/` is a directory below it, not an account.
+async fn accounts(store: &SecretStore, agent: &str) -> Result<Vec<String>> {
+    let dir = matrix::accounts_dir(agent)?;
+    let keys = store
+        .list(&dir)
+        .await
+        .with_context(|| format!("listing {dir}"))?;
+    Ok(keys.into_iter().filter(|k| !k.ends_with('/')).collect())
+}
+
+/// One pass over every agent's accounts. Returns how many paths were checked
+/// and how many were minted. A failure for one agent or account is logged and
+/// the pass carries on.
+async fn pass() -> Result<(usize, usize)> {
+    let store = crate::store::connect()
+        .await
+        .context("logging in to the swarm secret store")?;
+    let roles = store
+        .list_cert_roles(DEFAULT_CERT_MOUNT)
+        .await
+        .context("listing the store's cert-auth roles")?;
+    let (mut checked, mut minted) = (0, 0);
+    for agent in policy::agents_from_role_names(&roles) {
+        let names = match accounts(&store, &agent).await {
+            Ok(names) => names,
+            Err(e) => {
+                tracing::warn!(agent, error = %format!("{e:#}"), "matrix recovery passphrases: listing failed");
+                continue;
+            }
+        };
+        for account in names {
+            checked += 1;
+            let path = match matrix::recovery_path(&agent, &account) {
+                Ok(path) => path,
+                Err(e) => {
+                    tracing::warn!(agent, account, error = %e, "matrix recovery passphrase: unusable account name");
+                    continue;
+                }
+            };
+            match ensure(&store, &path).await {
+                Ok(Outcome::Present) => {}
+                Ok(Outcome::Minted) => {
+                    minted += 1;
+                    tracing::info!(agent, account, %path, "matrix recovery passphrase minted");
+                }
+                Ok(Outcome::LostRace) => {
+                    tracing::info!(agent, account, %path, "matrix recovery passphrase: another writer created it first");
+                }
+                Err(e) => tracing::warn!(
+                    agent,
+                    account,
+                    error = %format!("{e:#}"),
+                    "matrix recovery passphrase: mint failed"
+                ),
+            }
+        }
+    }
+    Ok((checked, minted))
+}
+
+/// Run [`pass`] now and every [`RECONCILE_INTERVAL`] after. A pass that fails
+/// is logged and retried on the next tick; it never stops the daemon.
+pub fn spawn() {
+    tokio::spawn(async move {
+        let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
+        loop {
+            ticker.tick().await;
+            match pass().await {
+                Ok((checked, minted)) => {
+                    tracing::debug!(checked, minted, "matrix recovery passphrases: pass done");
+                }
+                Err(e) => tracing::warn!(
+                    error = %format!("{e:#}"),
+                    retry_in_s = RECONCILE_INTERVAL.as_secs(),
+                    "matrix recovery passphrases: pass failed; retrying next tick"
+                ),
+            }
+        }
+    });
+}
+
+#[cfg(test)]
+mod tests {
+    use std::collections::HashMap;
+    use std::sync::Mutex;
+
+    use super::*;
+
+    /// An in-memory store. `race` makes the next create find the path taken by
+    /// someone else, the way a concurrent `cas=0` loses.
+    #[derive(Default)]
+    struct Fake {
+        held: Mutex<HashMap<String, String>>,
+        race: Option<String>,
+        refuse_reads: bool,
+    }
+
+    /// A store error with no store behind it.
+    fn refused() -> Error {
+        Error::MissingEnv("BAO_ADDR")
+    }
+
+    impl PassphraseStore for Fake {
+        async fn exists(&self, path: &str) -> Result<bool, Error> {
+            if self.refuse_reads {
+                return Err(refused());
+            }
+            Ok(self.held.lock().unwrap().contains_key(path))
+        }
+
+        async fn create(&self, path: &str, value: &matrix::Recovery) -> Result<(), Error> {
+            let mut held = self.held.lock().unwrap();
+            if let Some(winner) = &self.race {
+                held.insert(path.to_owned(), winner.clone());
+                return Err(refused());
+            }
+            if held.contains_key(path) {
+                return Err(refused());
+            }
+            held.insert(path.to_owned(), value.value.clone());
+            Ok(())
+        }
+    }
+
+    const PATH: &str = "swarm/agents/atlas/matrix-recovery/main";
+
+    #[tokio::test]
+    async fn an_empty_path_gets_a_passphrase() {
+        let store = Fake::default();
+        assert_eq!(ensure(&store, PATH).await.unwrap(), Outcome::Minted);
+        assert!(store.held.lock().unwrap().contains_key(PATH));
+    }
+
+    #[tokio::test]
+    async fn a_held_passphrase_is_never_replaced() {
+        let store = Fake::default();
+        store
+            .held
+            .lock()
+            .unwrap()
+            .insert(PATH.to_owned(), "first".to_owned());
+        assert_eq!(ensure(&store, PATH).await.unwrap(), Outcome::Present);
+        assert_eq!(store.held.lock().unwrap()[PATH], "first");
+    }
+
+    #[tokio::test]
+    async fn a_create_another_writer_beat_counts_as_done() {
+        let store = Fake {
+            race: Some("winner".to_owned()),
+            ..Fake::default()
+        };
+        assert_eq!(ensure(&store, PATH).await.unwrap(), Outcome::LostRace);
+        assert_eq!(store.held.lock().unwrap()[PATH], "winner");
+    }
+
+    #[tokio::test]
+    async fn a_store_that_cannot_be_read_gets_no_write() {
+        // Reading "absent" into a refusal would overwrite a live passphrase
+        // whenever the store was merely unreachable.
+        let store = Fake {
+            refuse_reads: true,
+            ..Fake::default()
+        };
+        assert!(ensure(&store, PATH).await.is_err());
+        assert!(store.held.lock().unwrap().is_empty());
+    }
+
+    #[test]
+    fn a_passphrase_is_32_random_bytes_in_a_plain_alphabet() {
+        let a = generate().unwrap().value;
+        let b = generate().unwrap().value;
+        // 32 bytes is 43 base64 characters without padding.
+        assert_eq!(a.len(), 43, "{a}");
+        assert!(
+            a.chars()
+                .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
+            "{a}"
+        );
+        assert_ne!(a, b);
+    }
+}

swarm-secret-client/src/matrix.rs

@@ -35,6 +35,41 @@ pub fn accounts_dir(agent: &str) -> Result<String, Error> {
     Ok(format!("{prefix}/matrix"))
 }
 
+/// The path holding the passphrase that opens the server-side key backup of
+/// `agent`'s matrix account `account`.
+///
+/// A directory of its own beside [`accounts_dir`], not a name under it: every
+/// object listed there is read as an account, so a leaf placed there would be
+/// brought up as one.
+///
+/// # Errors
+/// [`Error::PathSegment`] when either name contains anything but
+/// `[A-Za-z0-9_-]`.
+pub fn recovery_path(agent: &str, account: &str) -> Result<String, Error> {
+    checked_segment("account", account)?;
+    let prefix = principal_prefix(Kind::Agent, agent)?;
+    Ok(format!("{prefix}/matrix-recovery/{account}"))
+}
+
+/// What a [`recovery_path`] holds: the passphrase `swarm-controller` mints and
+/// the agent's matrix daemon enables or recovers key backup with.
+///
+/// No `Debug` derive: with the account's token, `value` decrypts every message
+/// the account has backed up.
+#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct Recovery {
+    /// Named `value` like [`Credential::value`].
+    pub value: String,
+}
+
+impl std::fmt::Debug for Recovery {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("Recovery")
+            .field("value", &"<redacted>")
+            .finish()
+    }
+}
+
 /// The localpart `hive` acts as on the homeserver: the account
 /// `swarm-controller` creates for it with the swarm's appservice token, and
 /// the one hive-c0re's sender token belongs to.
@@ -132,6 +167,55 @@ mod tests {
         assert_eq!(p, "swarm/agents/atlas/matrix/ops-relay");
     }
 
+    #[test]
+    fn the_recovery_passphrase_lands_beside_the_accounts_not_among_them() {
+        // Spelled out for the reason above.
+        let p = recovery_path("atlas", "ops-relay").expect("both segments are legal");
+        assert_eq!(p, "swarm/agents/atlas/matrix-recovery/ops-relay");
+        let dir = accounts_dir("atlas").expect("legal");
+        assert!(
+            !p.starts_with(&format!("{dir}/")),
+            "{p} is under {dir}, where every object is read as an account"
+        );
+        // Control: an account path is under it, so the assertion above can fail.
+        let account = account_path("atlas", "ops-relay").expect("legal");
+        assert!(account.starts_with(&format!("{dir}/")));
+    }
+
+    #[test]
+    fn the_agent_reads_its_own_recovery_passphrase_and_no_other() {
+        // The daemon reads this under the agent's own policy, so the rendered
+        // document is what has to cover it.
+        let doc = crate::policy::render_agent("atlas").expect("a plain name is legal");
+        let granted: Vec<&str> = doc
+            .lines()
+            .filter_map(|l| l.strip_prefix("path \"secret/data/"))
+            .filter_map(|p| p.strip_suffix("*\" {"))
+            .collect();
+        let reads = |path: &str| granted.iter().any(|g| path.starts_with(g));
+        let own = recovery_path("atlas", "main").expect("legal");
+        assert!(reads(&own), "{own} is not readable under {granted:?}");
+        let other = recovery_path("argus", "main").expect("legal");
+        assert!(!reads(&other), "{other} is readable under {granted:?}");
+    }
+
+    #[test]
+    fn a_recovery_passphrase_does_not_survive_formatting() {
+        let r = Recovery {
+            value: "SUPER_SECRET_PASSPHRASE".to_owned(),
+        };
+        let rendered = format!("{r:?}");
+        assert!(!rendered.contains("SUPER_SECRET_PASSPHRASE"), "{rendered}");
+        assert!(rendered.contains("<redacted>"), "{rendered}");
+    }
+
+    #[test]
+    fn a_traversal_is_refused_by_the_recovery_path_too() {
+        assert!(recovery_path("../argus", "main").is_err());
+        assert!(recovery_path("atlas", "../main").is_err());
+        assert!(recovery_path("a-b_C9", "d-e_F0").is_ok());
+    }
+
     #[test]
     fn the_sender_token_lands_under_the_hive_prefix_the_grant_covers() {
         // Spelled out rather than rebuilt from the same pieces the code uses,