matrix: back up agents' room keys with a controller-minted recovery passphrase
An agent's matrix encryption keys lived only in its local crypto store, so `destroy --purge` or a primary-account token reset lost every encrypted room's history with no way back (#5022). swarm-controller now mints a random passphrase (32 bytes from the kernel's CSPRNG, base64url) for every account listed under each agent's `matrix/` directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a five-minute pass. The write is a cas=0 create after a read; a path that holds anything is never replaced, and a create that loses a race counts as done. The controller's existing `data/swarm/agents/*` grant covers the write and the agent's own `agents/<agent>/*` read grant covers the read, so no policy changes. hive-matrix-daemon reads the passphrase when it brings an account up, after cross-signing: with no backup on the homeserver it enables recovery with it (backup + secret storage); with one there it recovers, which imports the backup key and the self-signing key, and room keys then download through `BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the account comes up without backup; any failure logs WARN and never stops the daemon. Recovery runs after `ensure_cross_signing`, not before: on a new account cross-signing must exist locally for `enable` to export it into secret storage, and on an existing identity `bootstrap_cross_signing_if_needed` is a no-op, leaving `recover` to import and sign. Part of #5022 (S1+S2, with the S4 docs).
This commit is contained in:
atlas
parent
bff605035c
commit
45568f6e60
11 changed files with 615 additions and 23 deletions
|
|
@ -16,7 +16,11 @@ keeps its state, purging it doesn't.**
|
|||
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
|
||||
history, claude credentials, `/state/` notes, everything. **No
|
||||
undo.** Only reach for this when you actually want the agent gone
|
||||
for good.
|
||||
for good. The agent's matrix encryption keys go with the state dir,
|
||||
but an account whose key backup is on (see
|
||||
[matrix](../integrations/matrix.md)) gets them back from its
|
||||
homeserver when it next comes up, so its encrypted history stays
|
||||
readable.
|
||||
|
||||
Beyond that:
|
||||
|
||||
|
|
|
|||
|
|
@ -306,12 +306,25 @@ Initial rollout settings:
|
|||
`matrix-sdk` so it can read encrypted rooms it's invited to even when
|
||||
this homeserver doesn't permit room encryption. matrix-sdk stores
|
||||
crypto keys in the per-agent sqlite store under the state dir; they persist across
|
||||
restarts (lost on `--purge`). `read_room` decrypts via
|
||||
restarts. `read_room` decrypts via
|
||||
`room.messages()` — UTD events surface as `event_type =
|
||||
"m.room.encrypted"` with `body = "[unable to decrypt]"`.
|
||||
Cross-signing and automatic key backup aren't enabled for the first
|
||||
pass: static bearer-token bot accounts can't bootstrap cross-signing
|
||||
without MSC3967.
|
||||
The daemon bootstraps cross-signing on the bearer token alone, which
|
||||
works where the homeserver skips re-auth for a first key upload
|
||||
(MSC3967, as tuwunel does).
|
||||
- **Key backup.** `swarm-controller` mints a random recovery passphrase
|
||||
for every matrix account an agent has, at
|
||||
`swarm/agents/<agent>/matrix-recovery/<account>` in the swarm secret
|
||||
store, and never replaces it. When `hive-matrix-daemon` brings an
|
||||
account up it reads that passphrase: with no backup on the account's
|
||||
homeserver it turns on server-side key backup and secret storage
|
||||
(log line `matrix recovery enabled`); with one there it recovers from
|
||||
it (`recovered from the server-side key backup`), which downloads the
|
||||
room keys and signs the new device with the account's existing
|
||||
cross-signing identity. A crypto store lost to `--purge` or to a
|
||||
primary-token reset therefore comes back with its history. With no passphrase
|
||||
stored yet the account comes up without backup and picks it up on the
|
||||
next daemon start. A failure is a `WARN` and never stops the daemon.
|
||||
|
||||
## Hive Matrix Space
|
||||
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ of the cell says how.
|
|||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
|
|
|
|||
|
|
@ -62,7 +62,9 @@ public handle crosses that boundary; the token never does.
|
|||
A hive reads its agents' credentials out of the swarm's secret store with its
|
||||
own certificate. **Every hive's policy grants read on every agent's
|
||||
credentials**, not only on the agents it hosts — so a compromised hive can read
|
||||
the matrix token of an agent running on a different hive.
|
||||
the matrix token of an agent running on a different hive. That includes each
|
||||
account's key-backup recovery passphrase, which with the token decrypts the
|
||||
agent's whole encrypted history, not only messages from then on.
|
||||
|
||||
That's deliberate and it's the interim state, not the intent. An agent's
|
||||
credential path doesn't name the hive hosting it (agents move between hives),
|
||||
|
|
|
|||
|
|
@ -12,17 +12,20 @@
|
|||
//! token + user/device as its appservice and handed us the bearer. matrix-sdk's `restore_session` with a constructed
|
||||
//! `MatrixSession` skips the login flow entirely.
|
||||
//!
|
||||
//! E2EE is enabled via `with_encryption_settings(EncryptionSettings::default())`.
|
||||
//! Crypto keys are persisted in the sqlite store under `state_dir`
|
||||
//! (survives container restarts, lost on `--purge`).
|
||||
//! E2EE is enabled via `with_encryption_settings`. Crypto keys are persisted
|
||||
//! in the sqlite store under `state_dir` (survives container restarts, lost on
|
||||
//! `--purge`).
|
||||
//!
|
||||
//! Cross-signing is bootstrapped best-effort after session restore
|
||||
//! (`ensure_cross_signing`) so the agent's device stops showing as
|
||||
//! "unverified" in other users' Matrix clients: the in-hive tuwunel accepts
|
||||
//! the initial device-signing-key upload on the bearer token alone (no UIAA
|
||||
//! re-auth). A homeserver that DID require UIAA just fails the call, which is
|
||||
//! swallowed — the session stays exactly as unverified as before. Automatic
|
||||
//! key backup is still left at its default (disabled).
|
||||
//! swallowed — the session stays exactly as unverified as before.
|
||||
//!
|
||||
//! Server-side key backup is turned on or recovered by `ensure_recovery`, with
|
||||
//! the passphrase `swarm-controller` mints per account, so a crypto store lost
|
||||
//! to a purge or a token reset gets its history back.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
|
|
@ -30,13 +33,13 @@ use anyhow::{Context, Result, anyhow};
|
|||
use matrix_sdk::{
|
||||
Client, SessionMeta, SessionTokens,
|
||||
authentication::matrix::MatrixSession,
|
||||
encryption::EncryptionSettings,
|
||||
encryption::{BackupDownloadStrategy, EncryptionSettings},
|
||||
ruma::{OwnedDeviceId, OwnedUserId},
|
||||
};
|
||||
use serde::Deserialize;
|
||||
use tokio::fs;
|
||||
|
||||
use crate::credential::Token;
|
||||
use crate::credential::{Passphrase, Token};
|
||||
|
||||
/// Sentinel returned when `build_and_restore` detects that the token is
|
||||
/// permanently invalid (`M_UNKNOWN_TOKEN`). The token has already been
|
||||
|
|
@ -146,7 +149,10 @@ pub async fn build_and_restore(
|
|||
let client = Client::builder()
|
||||
.homeserver_url(homeserver)
|
||||
.sqlite_store(state_dir, None)
|
||||
.with_encryption_settings(EncryptionSettings::default())
|
||||
.with_encryption_settings(EncryptionSettings {
|
||||
backup_download_strategy: BackupDownloadStrategy::OneShot,
|
||||
..EncryptionSettings::default()
|
||||
})
|
||||
.build()
|
||||
.await
|
||||
.with_context(|| format!("build matrix client for {homeserver}"))?;
|
||||
|
|
@ -220,11 +226,11 @@ async fn whoami(homeserver: &str, token: &str) -> Result<(OwnedUserId, OwnedDevi
|
|||
/// the failure is logged and swallowed, leaving the session no worse off than
|
||||
/// before. Never breaks account bring-up.
|
||||
///
|
||||
/// Known gap (self-verify only): this signs the *current* device against a
|
||||
/// freshly-bootstrapped identity. A device re-provisioned under an already
|
||||
/// existing identity (new `device_id`, e.g. after a token reissue) would need
|
||||
/// the cross-signing secret keys via secret storage to re-sign itself —
|
||||
/// tracked as the longer-term mutual-verification follow-up.
|
||||
/// This only signs the *current* device against a freshly-bootstrapped
|
||||
/// identity. A device under an already existing identity is a no-op here; it
|
||||
/// is signed by [`ensure_recovery`]'s `recover`, which imports the
|
||||
/// self-signing key from secret storage, and stays unverified for an account
|
||||
/// with no recovery passphrase.
|
||||
pub async fn ensure_cross_signing(client: &Client, account: &str) {
|
||||
match client
|
||||
.encryption()
|
||||
|
|
@ -247,6 +253,88 @@ pub async fn ensure_cross_signing(client: &Client, account: &str) {
|
|||
}
|
||||
}
|
||||
|
||||
/// What [`ensure_recovery`] did.
|
||||
enum Backup {
|
||||
/// This device was already connected to the account's backup.
|
||||
AlreadyOn,
|
||||
/// The server had no backup; one was created, with secret storage that
|
||||
/// `passphrase` opens.
|
||||
Enabled,
|
||||
/// The server had a backup; the keys secret storage holds for it, and
|
||||
/// for cross-signing if it holds those, were imported.
|
||||
Restored,
|
||||
}
|
||||
|
||||
/// Best-effort: turn on server-side key backup for this account, or recover
|
||||
/// from the backup it already has, with the account's recovery `passphrase`.
|
||||
///
|
||||
/// Run after [`ensure_cross_signing`], so a newly bootstrapped identity's
|
||||
/// private keys are local when `enable` exports them into secret storage, and
|
||||
/// an existing identity is left for `recover` to import. After a recover the
|
||||
/// room keys download in the background (`BackupDownloadStrategy::OneShot`).
|
||||
///
|
||||
/// Every failure is logged and swallowed, as in [`ensure_cross_signing`]: the
|
||||
/// account comes up as it would have without backup. That includes secret
|
||||
/// storage created with a different key, which this does not replace — the
|
||||
/// backup under it would be lost.
|
||||
pub async fn ensure_recovery(client: &Client, account: &str, passphrase: &Passphrase) {
|
||||
match recover_or_enable(client, passphrase).await {
|
||||
Ok(Backup::AlreadyOn) => {
|
||||
tracing::debug!(account, "matrix key backup already on for this device");
|
||||
}
|
||||
Ok(Backup::Enabled) => tracing::info!(
|
||||
account,
|
||||
passphrase = %passphrase.path(),
|
||||
"matrix recovery enabled: server-side key backup created"
|
||||
),
|
||||
Ok(Backup::Restored) => tracing::info!(
|
||||
account,
|
||||
passphrase = %passphrase.path(),
|
||||
"matrix recovery: recovered from the server-side key backup; room keys download in the background"
|
||||
),
|
||||
Err(e) => tracing::warn!(
|
||||
account,
|
||||
passphrase = %passphrase.path(),
|
||||
error = %format!("{e:#}"),
|
||||
"matrix recovery failed; key backup stays off (non-fatal)"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The body of [`ensure_recovery`], with its outcome returned rather than
|
||||
/// logged.
|
||||
async fn recover_or_enable(client: &Client, passphrase: &Passphrase) -> Result<Backup> {
|
||||
let encryption = client.encryption();
|
||||
// A device restored from its own store reconnects to the backup in a
|
||||
// background task; reading `are_enabled` before it finishes says no.
|
||||
encryption.wait_for_e2ee_initialization_tasks().await;
|
||||
let backups = encryption.backups();
|
||||
if backups.are_enabled().await {
|
||||
return Ok(Backup::AlreadyOn);
|
||||
}
|
||||
let recovery = encryption.recovery();
|
||||
if backups
|
||||
.fetch_exists_on_server()
|
||||
.await
|
||||
.context("asking the homeserver for the account's key backup")?
|
||||
{
|
||||
recovery
|
||||
.recover(passphrase.expose())
|
||||
.await
|
||||
.context("recovering from the key backup")?;
|
||||
Ok(Backup::Restored)
|
||||
} else {
|
||||
// The recovery key `enable` returns is not kept: the passphrase opens
|
||||
// the same secret storage.
|
||||
recovery
|
||||
.enable()
|
||||
.with_passphrase(passphrase.expose())
|
||||
.await
|
||||
.context("enabling key backup")?;
|
||||
Ok(Backup::Enabled)
|
||||
}
|
||||
}
|
||||
|
||||
/// Best-effort: set this account's matrix avatar from the rasterized icon
|
||||
/// PNG. The path comes from `HIVE_ICON_PNG` (a nix-built derivation the
|
||||
/// harness forwards into the daemon env; absent when no icon resolves).
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
//! The matrix access token as a value this process holds, and the two places
|
||||
//! it comes from; and the accounts the store links to this agent.
|
||||
//! it comes from; the accounts the store links to this agent; and each
|
||||
//! account's key-backup recovery passphrase.
|
||||
//!
|
||||
//! 🩸 **A secret is a path, not a value.** Nothing here writes the token
|
||||
//! anywhere, interpolates it into a command, or lets it reach a log line or an
|
||||
|
|
@ -179,6 +180,73 @@ impl Token {
|
|||
}
|
||||
}
|
||||
|
||||
/// The passphrase that opens an account's server-side key backup, held in
|
||||
/// memory for as long as it takes to enable or recover with it.
|
||||
///
|
||||
/// ⚠️ **No `Debug` derive**, for the reason [`Token`] has none: with the token,
|
||||
/// this decrypts every message the account has backed up.
|
||||
pub struct Passphrase {
|
||||
value: String,
|
||||
path: String,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Passphrase {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("Passphrase")
|
||||
.field("path", &self.path)
|
||||
.field("value", &"<redacted>")
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl Passphrase {
|
||||
/// The bytes, for the one caller that hands them to matrix-sdk.
|
||||
#[must_use]
|
||||
pub fn expose(&self) -> &str {
|
||||
&self.value
|
||||
}
|
||||
|
||||
/// The store path it was read from — the only half of it that may be
|
||||
/// printed.
|
||||
#[must_use]
|
||||
pub fn path(&self) -> &str {
|
||||
&self.path
|
||||
}
|
||||
|
||||
/// Read this agent's recovery passphrase for `account` out of the swarm
|
||||
/// secret store, under this agent's own certificate.
|
||||
///
|
||||
/// `Ok(None)` when this container was given no store, or the store holds
|
||||
/// nothing at the path yet: `swarm-controller` mints it on a periodic
|
||||
/// pass, so a freshly linked account can come up before it exists.
|
||||
///
|
||||
/// # Errors
|
||||
/// A name that is not a single path segment, an environment naming an
|
||||
/// identity that cannot be read, a store that refuses the certificate or
|
||||
/// the path, or a passphrase stored empty.
|
||||
pub async fn from_store(agent: &str, account: &str) -> Result<Option<Self>> {
|
||||
let path = matrix::recovery_path(agent, account)
|
||||
.context("building this agent's recovery passphrase path in the store")?;
|
||||
let Some(store) = connect(agent).await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let Some(stored): Option<matrix::Recovery> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path} from the store"))?
|
||||
else {
|
||||
return Ok(None);
|
||||
};
|
||||
let value = stored.value.trim().to_owned();
|
||||
if value.is_empty() {
|
||||
return Err(anyhow!(
|
||||
"the recovery passphrase at {path} in the store is empty"
|
||||
));
|
||||
}
|
||||
Ok(Some(Self { value, path }))
|
||||
}
|
||||
}
|
||||
|
||||
/// An account the store links to this agent: its name, and the homeserver
|
||||
/// stored beside its token.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
|
|
@ -340,6 +408,24 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn formatting_a_passphrase_does_not_reveal_it() {
|
||||
let p = Passphrase {
|
||||
value: "SUPER_SECRET_PASSPHRASE".to_owned(),
|
||||
path: "swarm/agents/a1/matrix-recovery/main".to_owned(),
|
||||
};
|
||||
let rendered = format!("{p:?}");
|
||||
assert!(
|
||||
!rendered.contains("SUPER_SECRET_PASSPHRASE"),
|
||||
"got {rendered}"
|
||||
);
|
||||
// Control: the path is meant to survive.
|
||||
assert!(
|
||||
rendered.contains("swarm/agents/a1/matrix-recovery/main"),
|
||||
"got {rendered}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The same property for the arm a reader actually reaches for: `Display`
|
||||
/// on the origin is what every message in this crate interpolates.
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -408,6 +408,28 @@ async fn account_token(cfg: &AccountCfg) -> Result<Option<Token>> {
|
|||
}
|
||||
}
|
||||
|
||||
/// Read `account`'s recovery passphrase from the store and hand it to
|
||||
/// [`client::ensure_recovery`]. With none — no store, or `swarm-controller`
|
||||
/// has not minted it yet — key backup stays off until the next start.
|
||||
async fn ensure_recovery(client: &Client, account: &str) {
|
||||
let found = match credential::agent_name() {
|
||||
Some(agent) => credential::Passphrase::from_store(&agent, account).await,
|
||||
None => Ok(None),
|
||||
};
|
||||
match found {
|
||||
Ok(Some(passphrase)) => client::ensure_recovery(client, account, &passphrase).await,
|
||||
Ok(None) => tracing::info!(
|
||||
account,
|
||||
"no matrix recovery passphrase in the store; key backup stays off"
|
||||
),
|
||||
Err(e) => tracing::warn!(
|
||||
account,
|
||||
error = %format!("{e:#}"),
|
||||
"could not read the matrix recovery passphrase; key backup stays off"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// Restore one account's client (when its token exists), install its
|
||||
/// message handler, and build its sync loop. Returns
|
||||
/// `Ok(Some((client, sync_loop)))` when the account came up, `Ok(None)`
|
||||
|
|
@ -450,6 +472,8 @@ async fn bring_up_account(
|
|||
// flagged "unverified" in other users' clients. Idempotent + swallows
|
||||
// failures (a UIAA-requiring homeserver) inside ensure_cross_signing.
|
||||
client::ensure_cross_signing(&client, &cfg.name).await;
|
||||
// Best-effort, and after cross-signing: see `client::ensure_recovery`.
|
||||
ensure_recovery(&client, &cfg.name).await;
|
||||
|
||||
let sync_client = client.clone();
|
||||
let cb_client = client.clone();
|
||||
|
|
|
|||
|
|
@ -2476,15 +2476,17 @@ fn configured_matrix_homeserver() -> Option<Arc<str>> {
|
|||
.map(Arc::from)
|
||||
}
|
||||
|
||||
/// Start the matrix-account backfill and the hive sender-token pass when a
|
||||
/// homeserver is configured, and the operator admin pass when the bridge is
|
||||
/// too. Lifted out of `main` for `clippy::too_many_lines`.
|
||||
/// Start the recovery passphrase pass, which needs only the store; the
|
||||
/// matrix-account backfill and the hive sender-token pass when a homeserver is
|
||||
/// configured; and the operator admin pass when the bridge is too. Lifted out
|
||||
/// of `main` for `clippy::too_many_lines`.
|
||||
fn spawn_matrix_account_backfill(
|
||||
jobq: &Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
|
||||
homeserver: Option<Arc<str>>,
|
||||
hives: &[HiveEntry],
|
||||
auth: Option<Arc<auth::AuthBridge>>,
|
||||
) {
|
||||
matrix_account::recovery::spawn();
|
||||
let Some(base) = homeserver else {
|
||||
return;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -7,6 +7,8 @@
|
|||
//! [`hive_sender`] mints each **hive's** sender account the same way, sharing this module's helpers with [`agent_token`].
|
||||
//! [`operator_admin`] makes authelia's operators homeserver admins with the same
|
||||
//! appservice token.
|
||||
//! [`recovery`] mints the passphrase that opens each account's server-side key
|
||||
//! backup, for the swarm's accounts and external ones alike.
|
||||
//!
|
||||
//! **The external one** — [`put_matrix_account`] and everything under it — is
|
||||
//! an account somewhere else that an operator hands us a credential for: put it
|
||||
|
|
@ -37,6 +39,7 @@ use super::{AppState, error_problem, swarm_hive};
|
|||
pub mod agent_token;
|
||||
pub mod hive_sender;
|
||||
pub mod operator_admin;
|
||||
pub mod recovery;
|
||||
|
||||
fn default_mode() -> String {
|
||||
"token".to_owned()
|
||||
|
|
|
|||
285
swarm-controller/src/matrix_account/recovery.rs
Normal file
285
swarm-controller/src/matrix_account/recovery.rs
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
//! A recovery passphrase for every matrix account of every agent, stored at
|
||||
//! `swarm_secret_client::matrix::recovery_path`: what the agent's matrix daemon
|
||||
//! turns on server-side key backup with, and recovers its keys with once its
|
||||
//! local crypto store is gone.
|
||||
//!
|
||||
//! Minted here because the agent's store policy is read-only. [`spawn`]'s pass
|
||||
//! runs at start and every [`RECONCILE_INTERVAL`] over every agent that holds a
|
||||
//! store identity — the same roster as [`super::agent_token`] — and every
|
||||
//! account listed under that agent's `matrix/` directory.
|
||||
//!
|
||||
//! ⚠️ **Never replaced.** A backup can only be opened with the passphrase it
|
||||
//! was enabled with, so a new one locks the agent out of everything backed up
|
||||
//! under the old. The write is a `cas=0` create, and a path that holds anything
|
||||
//! is left alone.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use serde::de::IgnoredAny;
|
||||
use swarm_secret_client::{Error, SecretStore, client::DEFAULT_CERT_MOUNT, matrix, policy};
|
||||
|
||||
/// How often [`spawn`] re-checks every account.
|
||||
const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
|
||||
|
||||
/// Bytes drawn from the kernel's CSPRNG per passphrase.
|
||||
const PASSPHRASE_BYTES: usize = 32;
|
||||
|
||||
/// What [`ensure`] did at one path.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Outcome {
|
||||
/// The path already held a passphrase.
|
||||
Present,
|
||||
/// This pass wrote one.
|
||||
Minted,
|
||||
/// Another writer's create landed between this pass's read and its own.
|
||||
LostRace,
|
||||
}
|
||||
|
||||
/// The two store calls [`ensure`] makes, so a test can stand in for the store.
|
||||
pub(crate) trait PassphraseStore {
|
||||
/// Whether anything is stored at `path`, as [`SecretStore::read_optional`]
|
||||
/// answers it. The value is decoded as [`IgnoredAny`] and never held.
|
||||
fn exists(&self, path: &str) -> impl Future<Output = Result<bool, Error>> + Send;
|
||||
|
||||
/// As [`SecretStore::create`].
|
||||
fn create(
|
||||
&self,
|
||||
path: &str,
|
||||
value: &matrix::Recovery,
|
||||
) -> impl Future<Output = Result<(), Error>> + Send;
|
||||
}
|
||||
|
||||
impl PassphraseStore for SecretStore {
|
||||
async fn exists(&self, path: &str) -> Result<bool, Error> {
|
||||
Ok(self.read_optional::<IgnoredAny>(path).await?.is_some())
|
||||
}
|
||||
|
||||
async fn create(&self, path: &str, value: &matrix::Recovery) -> Result<(), Error> {
|
||||
SecretStore::create(self, path, value).await
|
||||
}
|
||||
}
|
||||
|
||||
/// A passphrase: [`PASSPHRASE_BYTES`] from the kernel's CSPRNG, base64url
|
||||
/// without padding.
|
||||
///
|
||||
/// # Errors
|
||||
/// When the kernel will not supply randomness.
|
||||
fn generate() -> Result<matrix::Recovery> {
|
||||
let mut bytes = [0u8; PASSPHRASE_BYTES];
|
||||
getrandom::fill(&mut bytes)
|
||||
.context("drawing a recovery passphrase from the kernel's CSPRNG")?;
|
||||
Ok(matrix::Recovery {
|
||||
value: base64::Engine::encode(&base64::engine::general_purpose::URL_SAFE_NO_PAD, bytes),
|
||||
})
|
||||
}
|
||||
|
||||
/// Make sure `path` holds a passphrase, writing one only if it holds nothing.
|
||||
///
|
||||
/// A create that fails is read back: if the path now holds something, a
|
||||
/// concurrent writer won and that is the passphrase.
|
||||
///
|
||||
/// # Errors
|
||||
/// When the store refuses the read, or the create fails and the path is still
|
||||
/// empty.
|
||||
pub(crate) async fn ensure(store: &impl PassphraseStore, path: &str) -> Result<Outcome> {
|
||||
if store
|
||||
.exists(path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?
|
||||
{
|
||||
return Ok(Outcome::Present);
|
||||
}
|
||||
let Err(e) = store.create(path, &generate()?).await else {
|
||||
return Ok(Outcome::Minted);
|
||||
};
|
||||
if store
|
||||
.exists(path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path} back after a failed create"))?
|
||||
{
|
||||
return Ok(Outcome::LostRace);
|
||||
}
|
||||
Err(anyhow::Error::new(e).context(format!("creating {path}")))
|
||||
}
|
||||
|
||||
/// The account names listed under `agent`'s `matrix/` directory. A key ending
|
||||
/// in `/` is a directory below it, not an account.
|
||||
async fn accounts(store: &SecretStore, agent: &str) -> Result<Vec<String>> {
|
||||
let dir = matrix::accounts_dir(agent)?;
|
||||
let keys = store
|
||||
.list(&dir)
|
||||
.await
|
||||
.with_context(|| format!("listing {dir}"))?;
|
||||
Ok(keys.into_iter().filter(|k| !k.ends_with('/')).collect())
|
||||
}
|
||||
|
||||
/// One pass over every agent's accounts. Returns how many paths were checked
|
||||
/// and how many were minted. A failure for one agent or account is logged and
|
||||
/// the pass carries on.
|
||||
async fn pass() -> Result<(usize, usize)> {
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
let roles = store
|
||||
.list_cert_roles(DEFAULT_CERT_MOUNT)
|
||||
.await
|
||||
.context("listing the store's cert-auth roles")?;
|
||||
let (mut checked, mut minted) = (0, 0);
|
||||
for agent in policy::agents_from_role_names(&roles) {
|
||||
let names = match accounts(&store, &agent).await {
|
||||
Ok(names) => names,
|
||||
Err(e) => {
|
||||
tracing::warn!(agent, error = %format!("{e:#}"), "matrix recovery passphrases: listing failed");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
for account in names {
|
||||
checked += 1;
|
||||
let path = match matrix::recovery_path(&agent, &account) {
|
||||
Ok(path) => path,
|
||||
Err(e) => {
|
||||
tracing::warn!(agent, account, error = %e, "matrix recovery passphrase: unusable account name");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
match ensure(&store, &path).await {
|
||||
Ok(Outcome::Present) => {}
|
||||
Ok(Outcome::Minted) => {
|
||||
minted += 1;
|
||||
tracing::info!(agent, account, %path, "matrix recovery passphrase minted");
|
||||
}
|
||||
Ok(Outcome::LostRace) => {
|
||||
tracing::info!(agent, account, %path, "matrix recovery passphrase: another writer created it first");
|
||||
}
|
||||
Err(e) => tracing::warn!(
|
||||
agent,
|
||||
account,
|
||||
error = %format!("{e:#}"),
|
||||
"matrix recovery passphrase: mint failed"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok((checked, minted))
|
||||
}
|
||||
|
||||
/// Run [`pass`] now and every [`RECONCILE_INTERVAL`] after. A pass that fails
|
||||
/// is logged and retried on the next tick; it never stops the daemon.
|
||||
pub fn spawn() {
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
match pass().await {
|
||||
Ok((checked, minted)) => {
|
||||
tracing::debug!(checked, minted, "matrix recovery passphrases: pass done");
|
||||
}
|
||||
Err(e) => tracing::warn!(
|
||||
error = %format!("{e:#}"),
|
||||
retry_in_s = RECONCILE_INTERVAL.as_secs(),
|
||||
"matrix recovery passphrases: pass failed; retrying next tick"
|
||||
),
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Mutex;
|
||||
|
||||
use super::*;
|
||||
|
||||
/// An in-memory store. `race` makes the next create find the path taken by
|
||||
/// someone else, the way a concurrent `cas=0` loses.
|
||||
#[derive(Default)]
|
||||
struct Fake {
|
||||
held: Mutex<HashMap<String, String>>,
|
||||
race: Option<String>,
|
||||
refuse_reads: bool,
|
||||
}
|
||||
|
||||
/// A store error with no store behind it.
|
||||
fn refused() -> Error {
|
||||
Error::MissingEnv("BAO_ADDR")
|
||||
}
|
||||
|
||||
impl PassphraseStore for Fake {
|
||||
async fn exists(&self, path: &str) -> Result<bool, Error> {
|
||||
if self.refuse_reads {
|
||||
return Err(refused());
|
||||
}
|
||||
Ok(self.held.lock().unwrap().contains_key(path))
|
||||
}
|
||||
|
||||
async fn create(&self, path: &str, value: &matrix::Recovery) -> Result<(), Error> {
|
||||
let mut held = self.held.lock().unwrap();
|
||||
if let Some(winner) = &self.race {
|
||||
held.insert(path.to_owned(), winner.clone());
|
||||
return Err(refused());
|
||||
}
|
||||
if held.contains_key(path) {
|
||||
return Err(refused());
|
||||
}
|
||||
held.insert(path.to_owned(), value.value.clone());
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
const PATH: &str = "swarm/agents/atlas/matrix-recovery/main";
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_empty_path_gets_a_passphrase() {
|
||||
let store = Fake::default();
|
||||
assert_eq!(ensure(&store, PATH).await.unwrap(), Outcome::Minted);
|
||||
assert!(store.held.lock().unwrap().contains_key(PATH));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_held_passphrase_is_never_replaced() {
|
||||
let store = Fake::default();
|
||||
store
|
||||
.held
|
||||
.lock()
|
||||
.unwrap()
|
||||
.insert(PATH.to_owned(), "first".to_owned());
|
||||
assert_eq!(ensure(&store, PATH).await.unwrap(), Outcome::Present);
|
||||
assert_eq!(store.held.lock().unwrap()[PATH], "first");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_create_another_writer_beat_counts_as_done() {
|
||||
let store = Fake {
|
||||
race: Some("winner".to_owned()),
|
||||
..Fake::default()
|
||||
};
|
||||
assert_eq!(ensure(&store, PATH).await.unwrap(), Outcome::LostRace);
|
||||
assert_eq!(store.held.lock().unwrap()[PATH], "winner");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_store_that_cannot_be_read_gets_no_write() {
|
||||
// Reading "absent" into a refusal would overwrite a live passphrase
|
||||
// whenever the store was merely unreachable.
|
||||
let store = Fake {
|
||||
refuse_reads: true,
|
||||
..Fake::default()
|
||||
};
|
||||
assert!(ensure(&store, PATH).await.is_err());
|
||||
assert!(store.held.lock().unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_passphrase_is_32_random_bytes_in_a_plain_alphabet() {
|
||||
let a = generate().unwrap().value;
|
||||
let b = generate().unwrap().value;
|
||||
// 32 bytes is 43 base64 characters without padding.
|
||||
assert_eq!(a.len(), 43, "{a}");
|
||||
assert!(
|
||||
a.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
|
||||
"{a}"
|
||||
);
|
||||
assert_ne!(a, b);
|
||||
}
|
||||
}
|
||||
|
|
@ -35,6 +35,41 @@ pub fn accounts_dir(agent: &str) -> Result<String, Error> {
|
|||
Ok(format!("{prefix}/matrix"))
|
||||
}
|
||||
|
||||
/// The path holding the passphrase that opens the server-side key backup of
|
||||
/// `agent`'s matrix account `account`.
|
||||
///
|
||||
/// A directory of its own beside [`accounts_dir`], not a name under it: every
|
||||
/// object listed there is read as an account, so a leaf placed there would be
|
||||
/// brought up as one.
|
||||
///
|
||||
/// # Errors
|
||||
/// [`Error::PathSegment`] when either name contains anything but
|
||||
/// `[A-Za-z0-9_-]`.
|
||||
pub fn recovery_path(agent: &str, account: &str) -> Result<String, Error> {
|
||||
checked_segment("account", account)?;
|
||||
let prefix = principal_prefix(Kind::Agent, agent)?;
|
||||
Ok(format!("{prefix}/matrix-recovery/{account}"))
|
||||
}
|
||||
|
||||
/// What a [`recovery_path`] holds: the passphrase `swarm-controller` mints and
|
||||
/// the agent's matrix daemon enables or recovers key backup with.
|
||||
///
|
||||
/// No `Debug` derive: with the account's token, `value` decrypts every message
|
||||
/// the account has backed up.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct Recovery {
|
||||
/// Named `value` like [`Credential::value`].
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Recovery {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("Recovery")
|
||||
.field("value", &"<redacted>")
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
/// The localpart `hive` acts as on the homeserver: the account
|
||||
/// `swarm-controller` creates for it with the swarm's appservice token, and
|
||||
/// the one hive-c0re's sender token belongs to.
|
||||
|
|
@ -132,6 +167,55 @@ mod tests {
|
|||
assert_eq!(p, "swarm/agents/atlas/matrix/ops-relay");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_recovery_passphrase_lands_beside_the_accounts_not_among_them() {
|
||||
// Spelled out for the reason above.
|
||||
let p = recovery_path("atlas", "ops-relay").expect("both segments are legal");
|
||||
assert_eq!(p, "swarm/agents/atlas/matrix-recovery/ops-relay");
|
||||
let dir = accounts_dir("atlas").expect("legal");
|
||||
assert!(
|
||||
!p.starts_with(&format!("{dir}/")),
|
||||
"{p} is under {dir}, where every object is read as an account"
|
||||
);
|
||||
// Control: an account path is under it, so the assertion above can fail.
|
||||
let account = account_path("atlas", "ops-relay").expect("legal");
|
||||
assert!(account.starts_with(&format!("{dir}/")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_agent_reads_its_own_recovery_passphrase_and_no_other() {
|
||||
// The daemon reads this under the agent's own policy, so the rendered
|
||||
// document is what has to cover it.
|
||||
let doc = crate::policy::render_agent("atlas").expect("a plain name is legal");
|
||||
let granted: Vec<&str> = doc
|
||||
.lines()
|
||||
.filter_map(|l| l.strip_prefix("path \"secret/data/"))
|
||||
.filter_map(|p| p.strip_suffix("*\" {"))
|
||||
.collect();
|
||||
let reads = |path: &str| granted.iter().any(|g| path.starts_with(g));
|
||||
let own = recovery_path("atlas", "main").expect("legal");
|
||||
assert!(reads(&own), "{own} is not readable under {granted:?}");
|
||||
let other = recovery_path("argus", "main").expect("legal");
|
||||
assert!(!reads(&other), "{other} is readable under {granted:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_recovery_passphrase_does_not_survive_formatting() {
|
||||
let r = Recovery {
|
||||
value: "SUPER_SECRET_PASSPHRASE".to_owned(),
|
||||
};
|
||||
let rendered = format!("{r:?}");
|
||||
assert!(!rendered.contains("SUPER_SECRET_PASSPHRASE"), "{rendered}");
|
||||
assert!(rendered.contains("<redacted>"), "{rendered}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_traversal_is_refused_by_the_recovery_path_too() {
|
||||
assert!(recovery_path("../argus", "main").is_err());
|
||||
assert!(recovery_path("atlas", "../main").is_err());
|
||||
assert!(recovery_path("a-b_C9", "d-e_F0").is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_sender_token_lands_under_the_hive_prefix_the_grant_covers() {
|
||||
// Spelled out rather than rebuilt from the same pieces the code uses,
|
||||
|
|
|
|||
Loading…
Reference in a new issue