vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

render the matrix appservice registration from the store onto tmpfs

Browse source 
swarm-matrix-ctl appservice render now reads the registration's tokens
from the store and renders swarm.yaml under /run/swarm-matrix-appservice,
which tuwunel loads through LoadCredential. It mints a pair only when the
store answers and holds none; a host that has the old files stores their
pair once instead. A store that does not answer fails the unit and keeps
tuwunel down; sync starts it once the store answers.

Refs #4998 (item 3)
This commit is contained in:
atlas 2026-10-30 05:47:39 +01:00 • committed by mara
commit 3a4b0ffabf
2 changed files with 594 additions and 378 deletions
Download patch file Download diff file Expand all files Collapse all files

750
swarm-matrix-ctl/src/appservice.rs
View file

@ -1,21 +1,22 @@
//! `swarm-matrix-ctl appservice` — the **swarm's** own appservice
//! registration. The store holds it; this container keeps a copy the
//! homeserver starts from.
//! registration. The store holds it; this container renders a copy the
//! homeserver starts from onto its own tmpfs, and keeps none on disk.
//!
//! Two verbs, because they have opposite failure rules:
//! Two verbs, both reading the store:
//!
//! - [`render`] runs before tuwunel and touches nothing but this container's
//! state dir. tuwunel loads the file it writes through `LoadCredential`, and
//! a missing credential source fails the homeserver's start, so this half
//! must not be able to fail on a network. It renders from the tokens on
//! disk, which are the last copy [`sync`] took from the store, and mints
//! them only on a host that has none.
//! - [`sync`] runs beside the homeserver, never before it, and needs the store.
//! The store's tokens win: it writes the state dir only when they differ, and
//! restarts the homeserver whenever the registration is newer than the
//! homeserver's last start. A store holding none is seeded from
//! the state dir, create-only, so nothing rotates. A sealed store delays it
//! and nothing else.
//! - [`render`] runs before tuwunel. tuwunel loads the file it writes through
//! `LoadCredential`, and a missing credential source fails the homeserver's
//! start, so a store that does not answer fails this unit and keeps the
//! homeserver down until it does. It mints a pair only when the store
//! answers and holds none.
//! - [`sync`] runs beside the homeserver and follows the store: it rewrites
//! the file when the store's tokens differ, restarts the homeserver
//! whenever the registration is newer than the homeserver's last start, and
//! publishes the `as_token` for the controller.
//!
//! Where the store holds none, both first store the tokens an earlier version
//! kept in the legacy state dir (see [`resolve`]), so a host that has run for
//! months keeps its value. The files stay, unused.
//!
//! This registration's sender is promoted to homeserver admin at boot
//! (`nix/host-modules/hive-matrix.nix`), which is why its `as_token` goes to
@ -33,14 +34,17 @@ use swarm_secret_client::{
matrix::{self, Registration},
};
/// This container's state dir for the registration and its two tokens.
const ENV_DIR: &str = "MATRIX_APPSERVICE_DIR";
/// Where the homeserver's copy is rendered: a runtime directory on this
/// container's tmpfs.
const ENV_RUNTIME_DIR: &str = "MATRIX_APPSERVICE_RUNTIME_DIR";
/// The directory earlier versions kept the tokens in. Read once, never written.
const ENV_LEGACY_DIR: &str = "MATRIX_APPSERVICE_LEGACY_DIR";
/// The registration's `sender_localpart`: the account the homeserver creates
/// for it and the one `admin_execute` promotes.
const ENV_SENDER: &str = "MATRIX_APPSERVICE_SENDER";
/// The user namespace regex, rendered by nix.
const ENV_USER_REGEX: &str = "MATRIX_APPSERVICE_USER_REGEX";
/// Role on the store's `cert` auth mount that [`sync`] logs in with.
/// Role on the store's `cert` auth mount that both verbs log in with.
const ENV_CERT_ROLE: &str = "MATRIX_APPSERVICE_CERT_ROLE";
/// The unit that loads the registration, which [`sync`] restarts when the
/// registration is newer than its last start.
@ -48,7 +52,8 @@ const ENV_RESTART_UNIT: &str = "MATRIX_APPSERVICE_RESTART_UNIT";
/// The registration's `id`.
const ID: &str = "swarm";
/// File names inside [`ENV_DIR`]. `REGISTRATION` is what tuwunel loads.
/// `REGISTRATION` is what tuwunel loads, in [`ENV_RUNTIME_DIR`]. All three are
/// the names earlier versions used in [`ENV_LEGACY_DIR`].
const AS_TOKEN: &str = "as-token";
const HS_TOKEN: &str = "hs-token";
const REGISTRATION: &str = "swarm.yaml";
@ -62,41 +67,51 @@ fn required(get: &impl Fn(&str) -> Option<String>, var: &'static str) -> Result<
.with_context(|| format!("{var} is unset or empty"))
}
/// Mint the tokens if absent and render the registration from them.
/// The store, logged in as matrix-ctl.
async fn connect(cert_role: &str) -> Result<Bao> {
let settings = Settings::from_env().context("reading the store's BAO_* environment")?;
let store = SecretStore::connect(&settings, cert_role, DEFAULT_CERT_MOUNT)
.await
.with_context(|| {
format!("logging in to the swarm secret store as cert role {cert_role}")
})?;
Ok(Bao(store))
}
/// Render the registration from the store's tokens, which are minted first
/// when the store holds none.
///
/// # Errors
/// If a variable is missing, or the state dir cannot be read or written.
pub fn render() -> Result<()> {
/// If a variable is missing, or the store refuses the login, a read or a
/// write, or the runtime dir cannot be written.
pub async fn render() -> Result<()> {
let get = |k: &str| std::env::var(k).ok();
let dir = PathBuf::from(required(&get, ENV_DIR)?);
let runtime = PathBuf::from(required(&get, ENV_RUNTIME_DIR)?);
let legacy = PathBuf::from(required(&get, ENV_LEGACY_DIR)?);
let sender = required(&get, ENV_SENDER)?;
let regex = required(&get, ENV_USER_REGEX)?;
render_into(&dir, &sender, &regex)?;
tracing::info!(path = %dir.join(REGISTRATION).display(), "rendered the swarm appservice registration");
let store = connect(&required(&get, ENV_CERT_ROLE)?).await?;
render_with(&store, &runtime, &legacy, &sender, &regex, mint).await?;
tracing::info!(path = %runtime.join(REGISTRATION).display(), "rendered the swarm appservice registration");
Ok(())
}
/// [`render`] against an explicit directory, so a test can run it twice.
fn render_into(dir: &Path, sender: &str, regex: &str) -> Result<()> {
let as_token = existing_or_minted(&dir.join(AS_TOKEN))?;
let hs_token = existing_or_minted(&dir.join(HS_TOKEN))?;
/// [`render`] against an explicit store and directories.
async fn render_with(
store: &impl Store,
runtime: &Path,
legacy: &Path,
sender: &str,
regex: &str,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<()> {
let current = resolve(store, legacy, mint).await?;
write_secret(
&dir.join(REGISTRATION),
&registration_yaml(sender, regex, &as_token, &hs_token),
&runtime.join(REGISTRATION),
&registration_yaml(sender, regex, &current.as_token, &current.hs_token),
)
}
/// The token at `path`, minting and writing one first when there is none.
fn existing_or_minted(path: &Path) -> Result<String> {
if let Some(t) = read_token(path)? {
return Ok(t);
}
let token = swarm_matrix_client::random_hex(TOKEN_BYTES)?;
write_secret(path, &token)?;
tracing::info!(path = %path.display(), "minted a swarm appservice token");
Ok(token)
}
/// The trimmed token at `path`; `None` when the file is missing or blank.
fn read_token(path: &Path) -> Result<Option<String>> {
match std::fs::read_to_string(path) {
@ -107,21 +122,43 @@ fn read_token(path: &Path) -> Result<Option<String>> {
}
}
/// A temp path beside `path`, unique per call and per process, so two writers
/// of the same file never share one. Same directory, so the rename stays
/// atomic.
fn temp_path(path: &Path) -> PathBuf {
static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let mut name = path.file_name().unwrap_or_default().to_os_string();
name.push(format!(
".{}-{}.tmp",
std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
));
path.with_file_name(name)
}
/// Write `contents` to `path` as a `0600` file, through a rename so a reader
/// never sees half of it.
/// never sees half of it. A failure removes the temp file.
fn write_secret(path: &Path, contents: &str) -> Result<()> {
let tmp = path.with_extension("tmp");
let _ = std::fs::remove_file(&tmp);
let tmp = temp_path(path);
let written = write_temp(&tmp, contents).and_then(|()| {
std::fs::rename(&tmp, path).with_context(|| format!("renaming onto {}", path.display()))
});
if written.is_err() {
let _ = std::fs::remove_file(&tmp);
}
written
}
fn write_temp(tmp: &Path, contents: &str) -> Result<()> {
let mut f = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&tmp)
.open(tmp)
.with_context(|| format!("creating {}", tmp.display()))?;
f.write_all(contents.as_bytes())
.and_then(|()| f.sync_all())
.with_context(|| format!("writing {}", tmp.display()))?;
std::fs::rename(&tmp, path).with_context(|| format!("renaming onto {}", path.display()))
.with_context(|| format!("writing {}", tmp.display()))
}
/// The registration tuwunel loads.
@ -146,7 +183,7 @@ fn registration_yaml(sender: &str, regex: &str, as_token: &str, hs_token: &str)
)
}
/// What [`sync`] needs of the store, so the decisions can be tested without
/// What [`render`] and [`sync`] need of the store, so the decisions can be tested without
/// one.
trait Store {
/// The registration's tokens, or `None` when nothing is stored.
@ -201,55 +238,55 @@ impl Store for Bao {
}
}
/// Make the state dir hold the store's registration, and the controller's token
/// match it.
/// Make the runtime dir hold the store's registration, and the controller's
/// token match it.
///
/// # Errors
/// If a variable is missing, the store refuses the login, a read or a write,
/// the state dir cannot be written, or the homeserver cannot be restarted.
/// the runtime dir cannot be written, or the homeserver cannot be restarted.
pub async fn sync() -> Result<()> {
let get = |k: &str| std::env::var(k).ok();
let dir = PathBuf::from(required(&get, ENV_DIR)?);
let runtime = PathBuf::from(required(&get, ENV_RUNTIME_DIR)?);
let legacy = PathBuf::from(required(&get, ENV_LEGACY_DIR)?);
let sender = required(&get, ENV_SENDER)?;
let regex = required(&get, ENV_USER_REGEX)?;
let cert_role = required(&get, ENV_CERT_ROLE)?;
let unit = required(&get, ENV_RESTART_UNIT)?;
let settings = Settings::from_env().context("reading the store's BAO_* environment")?;
let store = SecretStore::connect(&settings, &cert_role, DEFAULT_CERT_MOUNT)
.await
.with_context(|| {
format!("logging in to the swarm secret store as cert role {cert_role}")
})?;
sync_with(&Bao(store), &dir, &sender, &regex, mint, &Systemd(unit)).await
let store = connect(&required(&get, ENV_CERT_ROLE)?).await?;
sync_with(
&store,
&runtime,
&legacy,
&sender,
&regex,
mint,
&Systemd(unit),
)
.await
}
/// [`sync`] against an explicit store, directory and homeserver unit.
/// [`sync`] against an explicit store, directories and homeserver unit.
///
/// The store's registration wins. Where it holds none, the state dir's tokens
/// seed it, and only a host with neither mints a pair. The homeserver is
/// restarted whenever the registration was written after it last started, so
/// a run that failed between the write and the restart is finished by the next.
/// The homeserver is restarted whenever the registration was written after it
/// last started, so a run that failed between the write and the restart is
/// finished by the next.
async fn sync_with(
store: &impl Store,
dir: &Path,
runtime: &Path,
legacy: &Path,
sender: &str,
regex: &str,
mint: impl FnOnce() -> Result<Registration>,
unit: &impl Homeserver,
) -> Result<()> {
let local = read_local(dir)?;
let current = stored_or_seeded(store, local.as_ref(), mint).await?;
let current = resolve(store, legacy, mint).await?;
let path = runtime.join(REGISTRATION);
let want = registration_yaml(sender, regex, &current.as_token, &current.hs_token);
let have = std::fs::read_to_string(dir.join(REGISTRATION)).ok();
if local.as_ref() != Some(&current) || have.as_deref() != Some(&want) {
write_secret(&dir.join(AS_TOKEN), &current.as_token)?;
write_secret(&dir.join(HS_TOKEN), &current.hs_token)?;
write_secret(&dir.join(REGISTRATION), &want)?;
if std::fs::read_to_string(&path).ok().as_deref() != Some(&want) {
write_secret(&path, &want)?;
tracing::info!("took the swarm appservice registration from the store");
}
restart_if_stale(unit, &dir.join(REGISTRATION))?;
restart_if_stale(unit, &path)?;
if needs_publish(store.controller_token().await?.as_ref(), &current.as_token) {
store.write_controller_token(&current.as_token).await?;
@ -258,17 +295,31 @@ async fn sync_with(
Ok(())
}
/// The store's registration; where it holds none, `local` or a minted pair is
/// created there first. A refused create is a lost race or a denial, and the
/// read-back tells which: it finds the winner's value, or nothing.
async fn stored_or_seeded(
/// The registration the store holds. Where it holds none, the tokens an
/// earlier version kept in `legacy` are stored once (create-only, so nothing
/// rotates), and only a host with neither mints a pair. A store that cannot
/// be read is an error and mints nothing. The files stay where they are,
/// unused.
async fn resolve(
store: &impl Store,
legacy: &Path,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<Registration> {
match store.registration().await? {
Some(stored) => Ok(stored),
None => seed_store(store, read_local(legacy)?.as_ref(), mint).await,
}
}
/// Create the registration in the empty store from `local`, or from a minted
/// pair when there are no files, and return what the store then holds. A
/// refused create is a lost race or a denial, and the read-back tells which:
/// it finds the winner's pair, or nothing.
async fn seed_store(
store: &impl Store,
local: Option<&Registration>,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<Registration> {
if let Some(stored) = store.registration().await? {
return Ok(stored);
}
let seed = match local {
Some(l) => l.clone(),
None => mint()?,
@ -282,7 +333,7 @@ async fn stored_or_seeded(
.context("the store holds no swarm appservice registration after creating one")
}
/// The tokens on disk, when both are there.
/// The tokens in `dir`, when both are there.
fn read_local(dir: &Path) -> Result<Option<Registration>> {
let as_token = read_token(&dir.join(AS_TOKEN))?;
let hs_token = read_token(&dir.join(HS_TOKEN))?;
@ -447,18 +498,24 @@ mod tests {
const REGEX: &str = "^@[a-z0-9._=/-]+:t\\.local$";
/// A directory of its own per call. A counter, not the clock, keeps two
/// tests that start in the same clock tick out of one directory.
fn scratch() -> PathBuf {
/// A runtime dir and a legacy dir of their own per call; the legacy one
/// is not created, as on a host that never had the files. A counter, not
/// the clock, keeps two tests that start in the same clock tick out of one
/// directory.
fn scratch() -> (PathBuf, PathBuf) {
static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let dir = std::env::temp_dir().join(format!(
let root = std::env::temp_dir().join(format!(
"swarm-appservice-{}-{}",
std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
));
std::fs::remove_dir_all(&dir).ok();
std::fs::create_dir_all(&dir).expect("temp dir");
dir
std::fs::remove_dir_all(&root).ok();
std::fs::create_dir_all(root.join("run")).expect("temp dir");
(root.join("run"), root.join("legacy"))
}
fn cleanup(runtime: &Path) {
std::fs::remove_dir_all(runtime.parent().expect("a parent")).ok();
}
fn reg(tag: &str) -> Registration {
@ -469,18 +526,34 @@ mod tests {
}
/// A store in memory. `down` fails every call, as a sealed or unreachable
/// store does; `racer` is what another writer stores between this run's
/// read and its create.
/// store does; `deny_create` refuses a create and stores nothing; `racer`
/// is what another writer stores between this run's read and its create.
#[derive(Default)]
struct Fake {
registration: RefCell<Option<Registration>>,
token: RefCell<Option<String>>,
down: bool,
deny_create: bool,
racer: RefCell<Option<Registration>>,
creates: Cell<u32>,
token_writes: Cell<u32>,
}
impl Fake {
fn holding(r: Registration) -> Self {
let f = Self::default();
*f.registration.borrow_mut() = Some(r);
f
}
fn down() -> Self {
Self {
down: true,
..Self::default()
}
}
}
impl Store for Fake {
async fn registration(&self) -> Result<Option<Registration>> {
if self.down {
@ -491,6 +564,9 @@ mod tests {
async fn create_registration(&self, r: &Registration) -> Result<()> {
self.creates.set(self.creates.get() + 1);
if self.deny_create {
bail!("permission denied");
}
if let Some(winner) = self.racer.borrow_mut().take() {
*self.registration.borrow_mut() = Some(winner);
bail!("already created");
@ -513,8 +589,9 @@ mod tests {
}
}
/// Write the state dir as `render` leaves it for `r`.
fn host_with(dir: &Path, r: &Registration) {
/// The three files an earlier version left in the legacy dir for `r`.
fn legacy_with(dir: &Path, r: &Registration) {
std::fs::create_dir_all(dir).expect("legacy dir");
write_secret(&dir.join(AS_TOKEN), &r.as_token).expect("as");
write_secret(&dir.join(HS_TOKEN), &r.hs_token).expect("hs");
write_secret(
@ -524,11 +601,25 @@ mod tests {
.expect("yaml");
}
fn files(dir: &Path) -> Vec<String> {
[AS_TOKEN, HS_TOKEN, REGISTRATION]
.iter()
.map(|f| std::fs::read_to_string(dir.join(f)).unwrap_or_default())
.collect()
/// What is left of the legacy files: the dir's file names, sorted.
fn legacy_left(dir: &Path) -> Vec<String> {
let mut names: Vec<String> = std::fs::read_dir(dir)
.map(|d| {
d.map(|e| e.expect("entry").file_name().to_string_lossy().into_owned())
.collect()
})
.unwrap_or_default();
names.sort();
names
}
/// The registration the runtime dir holds, or empty.
fn rendered(runtime: &Path) -> String {
std::fs::read_to_string(runtime.join(REGISTRATION)).unwrap_or_default()
}
fn yaml_for(r: &Registration) -> String {
registration_yaml("swarm", REGEX, &r.as_token, &r.hs_token)
}
/// The homeserver unit. `started` is when it last became active; a restart
@ -574,12 +665,24 @@ mod tests {
}
}
/// Run one render, returning how often it minted.
async fn render_run(store: &Fake, runtime: &Path, legacy: &Path) -> (Result<()>, u32) {
let mints = Cell::new(0);
let out = render_with(store, runtime, legacy, "swarm", REGEX, || {
mints.set(mints.get() + 1);
Ok(reg("minted"))
})
.await;
(out, mints.get())
}
/// Run one sync against `unit`, returning how often it minted.
async fn run(store: &Fake, dir: &Path, unit: &Unit) -> (Result<()>, u32) {
async fn run(store: &Fake, runtime: &Path, legacy: &Path, unit: &Unit) -> (Result<()>, u32) {
let mints = Cell::new(0);
let out = sync_with(
store,
dir,
runtime,
legacy,
"swarm",
REGEX,
|| {
@ -592,28 +695,154 @@ mod tests {
(out, mints.get())
}
#[test]
fn the_rendered_registration_carries_the_token_it_minted() {
let dir = scratch();
render_into(&dir, "swarm", REGEX).expect("renders");
let yaml = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
let token = std::fs::read_to_string(dir.join(AS_TOKEN)).expect("minted");
assert_eq!(token.len(), TOKEN_BYTES * 2);
assert!(yaml.contains(&format!("\nas_token: {token}\n")), "{yaml}");
std::fs::remove_dir_all(&dir).ok();
#[tokio::test]
async fn an_empty_store_and_no_files_mint_a_pair_into_the_store() {
let (runtime, legacy) = scratch();
let store = Fake::default();
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(mints, 1);
assert_eq!(*store.registration.borrow(), Some(reg("minted")));
assert_eq!(rendered(&runtime), yaml_for(&reg("minted")));
assert!(!legacy.exists(), "nothing is written to the legacy dir");
cleanup(&runtime);
}
#[tokio::test]
async fn an_empty_store_takes_the_files_value_once() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("old"));
let store = Fake::default();
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(mints, 0, "a host that has the value mints nothing");
assert_eq!(*store.registration.borrow(), Some(reg("old")));
assert_eq!(rendered(&runtime), yaml_for(&reg("old")));
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_mints_nothing_and_renders_nothing() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let (out, mints) = render_run(&Fake::down(), &runtime, &legacy).await;
assert!(out.is_err(), "the unit fails and retries");
assert_eq!(mints, 0);
assert_eq!(rendered(&runtime), "", "nothing for tuwunel to load");
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_on_a_fresh_host_mints_nothing() {
let (runtime, legacy) = scratch();
let (out, mints) = render_run(&Fake::down(), &runtime, &legacy).await;
assert!(out.is_err());
assert_eq!(
mints, 0,
"a pair minted while the store is away is a rotation"
);
assert_eq!(rendered(&runtime), "");
cleanup(&runtime);
}
#[tokio::test]
async fn a_lost_race_adopts_the_winners_pair() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let store = Fake::default();
*store.racer.borrow_mut() = Some(reg("winner"));
let (out, _) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(rendered(&runtime), yaml_for(&reg("winner")));
cleanup(&runtime);
}
#[tokio::test]
async fn a_refused_create_fails_the_run() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let store = Fake {
deny_create: true,
..Fake::default()
};
let (out, _) = render_run(&store, &runtime, &legacy).await;
assert!(out.is_err(), "no read-back finds the pair");
assert_eq!(rendered(&runtime), "");
cleanup(&runtime);
}
#[tokio::test]
async fn a_second_render_keeps_the_pair() {
// "Only once": a re-mint on every boot would hand the controller a
// token the homeserver no longer loads until sync catches up.
let (runtime, legacy) = scratch();
let store = Fake::default();
render_run(&store, &runtime, &legacy)
.await
.0
.expect("first");
let first = rendered(&runtime);
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("second");
assert_eq!(mints, 0);
assert_eq!(rendered(&runtime), first);
cleanup(&runtime);
}
#[test]
fn a_second_render_keeps_the_token() {
// "Only once": a re-mint on every boot would hand the controller a
// token the homeserver no longer loads until sync catches up.
let dir = scratch();
render_into(&dir, "swarm", REGEX).expect("first");
let first = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
render_into(&dir, "swarm", REGEX).expect("second");
let second = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
assert_eq!(first, second);
std::fs::remove_dir_all(&dir).ok();
fn two_writers_of_one_file_use_different_temp_files() {
let path = Path::new("/run/x/swarm.yaml");
let (a, b) = (temp_path(path), temp_path(path));
assert_ne!(a, b);
assert_eq!(
a.parent(),
path.parent(),
"the rename stays in one directory"
);
}
#[test]
fn overlapping_writers_all_succeed_and_leave_no_temp_file() {
let (runtime, _) = scratch();
let path = runtime.join(REGISTRATION);
let writers: Vec<_> = (0..8)
.map(|_| {
let path = path.clone();
std::thread::spawn(move || {
(0..50)
.try_for_each(|_| write_secret(&path, "same\n"))
.is_ok()
})
})
.collect();
assert!(writers.into_iter().all(|w| w.join().expect("joins")));
assert_eq!(std::fs::read_to_string(&path).expect("written"), "same\n");
assert_eq!(legacy_left(&runtime), [REGISTRATION]);
cleanup(&runtime);
}
#[test]
fn a_failed_write_removes_its_temp_file() {
let (runtime, _) = scratch();
// A directory where the file goes: the rename fails.
std::fs::create_dir(runtime.join(REGISTRATION)).expect("dir");
write_secret(&runtime.join(REGISTRATION), "x").expect_err("cannot replace a directory");
assert_eq!(legacy_left(&runtime), [REGISTRATION]);
cleanup(&runtime);
}
#[test]
@ -624,172 +853,86 @@ mod tests {
assert!(y.contains("\n - exclusive: false\n"), "{y}");
assert!(y.contains(&format!("regex: '{REGEX}'")), "{y}");
assert!(y.contains("\nurl: null\n"), "{y}");
assert!(y.contains("\nas_token: aa\n"), "{y}");
assert!(y.contains("\nhs_token: bb\n"), "{y}");
}
#[test]
fn the_registration_and_tokens_are_owner_only() {
#[tokio::test]
async fn the_rendered_registration_is_owner_only() {
use std::os::unix::fs::PermissionsExt as _;
let dir = scratch();
render_into(&dir, "swarm", REGEX).expect("renders");
for f in [AS_TOKEN, HS_TOKEN, REGISTRATION] {
let mode = std::fs::metadata(dir.join(f))
.expect("exists")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600, "{f}");
}
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn seeding_from_the_host_leaves_every_file_unchanged() {
// A host that has run for months with the store empty: the store takes
// its value, and nothing on the host moves, so nothing rotates.
let dir = scratch();
let on_disk = reg("old");
host_with(&dir, &on_disk);
let before = files(&dir);
let store = Fake::default();
let unit = Unit::just_started();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(*store.registration.borrow(), Some(on_disk));
assert_eq!(files(&dir), before);
assert_eq!(mints, 0, "a host that has the value mints nothing");
assert_eq!(restarts, 0, "an unchanged host does not restart");
assert_eq!(store.token.borrow().as_deref(), Some("as-old"));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn the_store_wins_when_both_hold_a_value() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
*store.token.borrow_mut() = Some("as-disk".to_owned());
let unit = Unit::long_running();
let (out, _) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
let want = registration_yaml("swarm", REGEX, "as-store", "hs-store");
assert_eq!(
files(&dir),
vec!["as-store".to_owned(), "hs-store".to_owned(), want]
);
assert_eq!(*store.registration.borrow(), Some(reg("store")));
assert_eq!(store.creates.get(), 0, "a held value is never rewritten");
assert_eq!(restarts, 1);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_host_without_the_files_takes_them_from_the_store() {
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unit::long_running();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(read_local(&dir).expect("reads"), Some(reg("store")));
assert!(dir.join(REGISTRATION).exists());
assert_eq!((restarts, mints), (1, 0));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_fresh_host_mints_into_the_store() {
let dir = scratch();
let store = Fake::default();
let unit = Unit::long_running();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(mints, 1);
assert_eq!(*store.registration.borrow(), Some(reg("minted")));
assert_eq!(read_local(&dir).expect("reads"), Some(reg("minted")));
assert_eq!(restarts, 1);
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_lost_race_adopts_the_winners_registration() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let store = Fake::default();
*store.racer.borrow_mut() = Some(reg("winner"));
let unit = Unit::long_running();
let (out, _) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(read_local(&dir).expect("reads"), Some(reg("winner")));
assert_eq!(restarts, 1);
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_store_that_is_down_leaves_the_host_as_it_is() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let before = files(&dir);
let store = Fake {
down: true,
..Fake::default()
};
let unit = Unit::just_started();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
assert!(out.is_err(), "the unit reports the outage");
assert_eq!(files(&dir), before, "the homeserver keeps the last copy");
assert_eq!((restarts, mints), (0, 0));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn the_start_path_renders_the_copy_sync_took_from_the_store() {
// `render` has no store: after an outage it renders what the last sync
// wrote, without minting over it.
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
run(&store, &dir, &Unit::long_running())
let (runtime, legacy) = scratch();
render_run(&Fake::default(), &runtime, &legacy)
.await
.0
.expect("syncs");
let synced = files(&dir);
.expect("renders");
let mode = std::fs::metadata(runtime.join(REGISTRATION))
.expect("exists")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600);
cleanup(&runtime);
}
render_into(&dir, "swarm", REGEX).expect("renders");
#[tokio::test]
async fn the_store_wins_when_the_rendered_copy_differs() {
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("old"))).expect("rendered");
let store = Fake::holding(reg("store"));
*store.token.borrow_mut() = Some("as-old".to_owned());
assert_eq!(files(&dir), synced);
std::fs::remove_dir_all(&dir).ok();
let unit = Unit::long_running();
let (out, _) = run(&store, &runtime, &legacy, &unit).await;
out.expect("syncs");
assert_eq!(rendered(&runtime), yaml_for(&reg("store")));
assert_eq!(*store.registration.borrow(), Some(reg("store")));
assert_eq!(store.creates.get(), 0, "a held value is never rewritten");
assert_eq!(unit.restarts.get(), 1);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
cleanup(&runtime);
}
#[tokio::test]
async fn a_sync_with_nothing_rendered_takes_the_stores_pair() {
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
let (out, mints) = run(&store, &runtime, &legacy, &unit).await;
out.expect("syncs");
assert_eq!(rendered(&runtime), yaml_for(&reg("store")));
assert_eq!((unit.restarts.get(), mints), (1, 0));
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_leaves_the_rendered_copy_as_it_is() {
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("disk"))).expect("rendered");
let unit = Unit::just_started();
let (out, mints) = run(&Fake::down(), &runtime, &legacy, &unit).await;
assert!(out.is_err(), "the unit reports the outage");
assert_eq!(rendered(&runtime), yaml_for(&reg("disk")));
assert_eq!((unit.restarts.get(), mints), (0, 0));
cleanup(&runtime);
}
#[tokio::test]
async fn a_second_sync_restarts_nothing() {
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
run(&store, &dir, &unit).await.0.expect("first");
run(&store, &runtime, &legacy, &unit)
.await
.0
.expect("first");
let first = unit.restarts.get();
let (out, _) = run(&store, &dir, &unit).await;
let (out, _) = run(&store, &runtime, &legacy, &unit).await;
let second = unit.restarts.get() - first;
out.expect("syncs");
@ -799,48 +942,48 @@ mod tests {
1,
"the controller's copy is written once"
);
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
#[tokio::test]
async fn a_restart_that_failed_is_retried_by_the_next_run() {
// The files are written and the restart is refused. The next run finds
// The file is written and the restart is refused. The next run finds
// nothing to write, and still restarts: the decision comes from the
// files' age against the unit's start, not from this run's writes.
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
// file's age against the unit's start, not from this run's writes.
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
unit.fail_restart.set(true);
let (first, _) = run(&store, &dir, &unit).await;
let (first, _) = run(&store, &runtime, &legacy, &unit).await;
assert!(first.is_err(), "the refused restart fails the run");
let written = files(&dir);
let written = rendered(&runtime);
unit.fail_restart.set(false);
let (second, _) = run(&store, &dir, &unit).await;
let (second, _) = run(&store, &runtime, &legacy, &unit).await;
second.expect("the retry succeeds");
assert_eq!(files(&dir), written, "nothing was left to write");
assert_eq!(rendered(&runtime), written, "nothing was left to write");
assert_eq!(unit.restarts.get(), 2);
assert_eq!(read_local(&dir).expect("reads"), Some(reg("store")));
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
#[tokio::test]
async fn a_unit_that_started_after_the_files_is_left_alone() {
// The control for the retry above: the same files, a unit that started
// after them, and no restart.
let dir = scratch();
host_with(&dir, &reg("store"));
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
async fn a_unit_that_started_after_the_file_is_left_alone() {
// The control for the retry above: the same file, a unit that started
// after it, and no restart.
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("store"))).expect("rendered");
let store = Fake::holding(reg("store"));
let unit = Unit::just_started();
run(&store, &dir, &unit).await.0.expect("syncs");
run(&store, &runtime, &legacy, &unit)
.await
.0
.expect("syncs");
assert_eq!(unit.restarts.get(), 0);
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
#[tokio::test]
@ -855,17 +998,25 @@ mod tests {
Ok(())
}
}
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unreadable(Cell::new(0));
let out = sync_with(&store, &dir, "swarm", REGEX, || Ok(reg("minted")), &unit).await;
let out = sync_with(
&store,
&runtime,
&legacy,
"swarm",
REGEX,
|| Ok(reg("minted")),
&unit,
)
.await;
out.expect("the run goes on");
assert_eq!(unit.0.get(), 0);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
/// `restart_via` against a `systemctl` that exits with the given codes for the
@ -939,7 +1090,8 @@ mod tests {
#[test]
fn every_variable_is_scoped_to_the_verb() {
for var in [
ENV_DIR,
ENV_RUNTIME_DIR,
ENV_LEGACY_DIR,
ENV_SENDER,
ENV_USER_REGEX,
ENV_CERT_ROLE,

17
swarm-matrix-ctl/src/main.rs
View file

@ -7,8 +7,8 @@
//! identity plumbing to add one action, so the next thing that has to run in
//! here is a verb below, not a new crate.
//!
//! [`appservice`] mints the **swarm's** own appservice registration and
//! keeps it in step with the store and publishes its token for `swarm-controller`.
//! [`appservice`] renders the **swarm's** own appservice registration from
//! the store and publishes its token for `swarm-controller`.
//!
//! 🩸 **A secret is a path, never a value.** The only identifier any verb here
//! logs is the store path; see `swarm_matrix_client`'s module doc for the same rule
@ -39,12 +39,13 @@ enum Command {
#[derive(Debug, Subcommand)]
enum Appservice {
/// Mint the tokens when absent and render the registration tuwunel loads.
/// Local only: it runs before the homeserver and must not need a network.
/// Render the registration tuwunel loads from the store, minting the tokens
/// first when the store holds none. Runs before the homeserver, which stays
/// down until the store answers.
Render,
/// Make the state dir hold the store's registration, seeding the store
/// from it when the store holds none, and restart the homeserver when the
/// registration is newer than its last start. Runs beside the homeserver, never before it.
/// Make the rendered registration match the store's, restart the homeserver
/// when the registration is newer than its last start, and publish the
/// `as_token` for `swarm-controller`. Runs beside the homeserver.
Sync,
}
@ -58,7 +59,7 @@ async fn main() -> Result<()> {
.init();
match Cli::parse().command {
Command::Appservice(Appservice::Render) => appservice::render(),
Command::Appservice(Appservice::Render) => appservice::render().await,
Command::Appservice(Appservice::Sync) => appservice::sync().await,
}
}