vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

render the matrix appservice registration from the store onto tmpfs

Browse source 
swarm-matrix-ctl appservice render now reads the registration's tokens
from the store and renders swarm.yaml under /run/swarm-matrix-appservice,
which tuwunel loads through LoadCredential. It mints a pair only when the
store answers and holds none; a host that has the old files stores their
pair once instead. A store that does not answer fails the unit and keeps
tuwunel down; sync starts it once the store answers.

Refs #4998 (item 3)
This commit is contained in:
atlas 2026-10-30 05:47:39 +01:00 • committed by mara
commit 3a4b0ffabf
3 changed files with 594 additions and 378 deletions
Download patch file Download diff file Expand all files Collapse all files

17
swarm-matrix-ctl/README.md
View file

@ -13,18 +13,19 @@ crate.
### `appservice render`
Renders the swarm appservice registration tuwunel loads from the tokens in the
state dir, minting them only on a host that has none. Runs before the
homeserver and needs no network: the tokens on disk are the last copy `sync`
took from the store.
Renders the swarm appservice registration tuwunel loads from the store's tokens
onto the container's tmpfs, minting a pair only when the store answers and holds
none. Runs before the homeserver, which stays down while the store does not
answer. Tokens an earlier version kept in the container's `/var/lib` are stored
first when the store holds none; the files stay, unused.
### `appservice sync`
The store holds the registration's tokens. This writes them into the state dir
The store holds the registration's tokens. This rewrites the rendered copy
when they differ, restarts the homeserver whenever the registration is
newer than its last start, and publishes the `as_token` for `swarm-controller`. A store holding none is seeded from the
state dir, create-only, so nothing rotates. It runs beside the homeserver, so a
store outage leaves the homeserver running on the last copy.
newer than its last start, and publishes the `as_token` for `swarm-controller`.
It runs the same reconciliation as `render`, and starts a homeserver that
failed because `render` could not reach the store.
Both are configured entirely by the `MATRIX_APPSERVICE_*` environment the units
set — no flags. A systemd `Environment=` block is what a nix module can render; a

750
swarm-matrix-ctl/src/appservice.rs
View file

@ -1,21 +1,22 @@
//! `swarm-matrix-ctl appservice` — the **swarm's** own appservice
//! registration. The store holds it; this container keeps a copy the
//! homeserver starts from.
//! registration. The store holds it; this container renders a copy the
//! homeserver starts from onto its own tmpfs, and keeps none on disk.
//!
//! Two verbs, because they have opposite failure rules:
//! Two verbs, both reading the store:
//!
//! - [`render`] runs before tuwunel and touches nothing but this container's
//! state dir. tuwunel loads the file it writes through `LoadCredential`, and
//! a missing credential source fails the homeserver's start, so this half
//! must not be able to fail on a network. It renders from the tokens on
//! disk, which are the last copy [`sync`] took from the store, and mints
//! them only on a host that has none.
//! - [`sync`] runs beside the homeserver, never before it, and needs the store.
//! The store's tokens win: it writes the state dir only when they differ, and
//! restarts the homeserver whenever the registration is newer than the
//! homeserver's last start. A store holding none is seeded from
//! the state dir, create-only, so nothing rotates. A sealed store delays it
//! and nothing else.
//! - [`render`] runs before tuwunel. tuwunel loads the file it writes through
//! `LoadCredential`, and a missing credential source fails the homeserver's
//! start, so a store that does not answer fails this unit and keeps the
//! homeserver down until it does. It mints a pair only when the store
//! answers and holds none.
//! - [`sync`] runs beside the homeserver and follows the store: it rewrites
//! the file when the store's tokens differ, restarts the homeserver
//! whenever the registration is newer than the homeserver's last start, and
//! publishes the `as_token` for the controller.
//!
//! Where the store holds none, both first store the tokens an earlier version
//! kept in the legacy state dir (see [`resolve`]), so a host that has run for
//! months keeps its value. The files stay, unused.
//!
//! This registration's sender is promoted to homeserver admin at boot
//! (`nix/host-modules/hive-matrix.nix`), which is why its `as_token` goes to
@ -33,14 +34,17 @@ use swarm_secret_client::{
matrix::{self, Registration},
};
/// This container's state dir for the registration and its two tokens.
const ENV_DIR: &str = "MATRIX_APPSERVICE_DIR";
/// Where the homeserver's copy is rendered: a runtime directory on this
/// container's tmpfs.
const ENV_RUNTIME_DIR: &str = "MATRIX_APPSERVICE_RUNTIME_DIR";
/// The directory earlier versions kept the tokens in. Read once, never written.
const ENV_LEGACY_DIR: &str = "MATRIX_APPSERVICE_LEGACY_DIR";
/// The registration's `sender_localpart`: the account the homeserver creates
/// for it and the one `admin_execute` promotes.
const ENV_SENDER: &str = "MATRIX_APPSERVICE_SENDER";
/// The user namespace regex, rendered by nix.
const ENV_USER_REGEX: &str = "MATRIX_APPSERVICE_USER_REGEX";
/// Role on the store's `cert` auth mount that [`sync`] logs in with.
/// Role on the store's `cert` auth mount that both verbs log in with.
const ENV_CERT_ROLE: &str = "MATRIX_APPSERVICE_CERT_ROLE";
/// The unit that loads the registration, which [`sync`] restarts when the
/// registration is newer than its last start.
@ -48,7 +52,8 @@ const ENV_RESTART_UNIT: &str = "MATRIX_APPSERVICE_RESTART_UNIT";
/// The registration's `id`.
const ID: &str = "swarm";
/// File names inside [`ENV_DIR`]. `REGISTRATION` is what tuwunel loads.
/// `REGISTRATION` is what tuwunel loads, in [`ENV_RUNTIME_DIR`]. All three are
/// the names earlier versions used in [`ENV_LEGACY_DIR`].
const AS_TOKEN: &str = "as-token";
const HS_TOKEN: &str = "hs-token";
const REGISTRATION: &str = "swarm.yaml";
@ -62,41 +67,51 @@ fn required(get: &impl Fn(&str) -> Option<String>, var: &'static str) -> Result<
.with_context(|| format!("{var} is unset or empty"))
}
/// Mint the tokens if absent and render the registration from them.
/// The store, logged in as matrix-ctl.
async fn connect(cert_role: &str) -> Result<Bao> {
let settings = Settings::from_env().context("reading the store's BAO_* environment")?;
let store = SecretStore::connect(&settings, cert_role, DEFAULT_CERT_MOUNT)
.await
.with_context(|| {
format!("logging in to the swarm secret store as cert role {cert_role}")
})?;
Ok(Bao(store))
}
/// Render the registration from the store's tokens, which are minted first
/// when the store holds none.
///
/// # Errors
/// If a variable is missing, or the state dir cannot be read or written.
pub fn render() -> Result<()> {
/// If a variable is missing, or the store refuses the login, a read or a
/// write, or the runtime dir cannot be written.
pub async fn render() -> Result<()> {
let get = |k: &str| std::env::var(k).ok();
let dir = PathBuf::from(required(&get, ENV_DIR)?);
let runtime = PathBuf::from(required(&get, ENV_RUNTIME_DIR)?);
let legacy = PathBuf::from(required(&get, ENV_LEGACY_DIR)?);
let sender = required(&get, ENV_SENDER)?;
let regex = required(&get, ENV_USER_REGEX)?;
render_into(&dir, &sender, &regex)?;
tracing::info!(path = %dir.join(REGISTRATION).display(), "rendered the swarm appservice registration");
let store = connect(&required(&get, ENV_CERT_ROLE)?).await?;
render_with(&store, &runtime, &legacy, &sender, &regex, mint).await?;
tracing::info!(path = %runtime.join(REGISTRATION).display(), "rendered the swarm appservice registration");
Ok(())
}
/// [`render`] against an explicit directory, so a test can run it twice.
fn render_into(dir: &Path, sender: &str, regex: &str) -> Result<()> {
let as_token = existing_or_minted(&dir.join(AS_TOKEN))?;
let hs_token = existing_or_minted(&dir.join(HS_TOKEN))?;
/// [`render`] against an explicit store and directories.
async fn render_with(
store: &impl Store,
runtime: &Path,
legacy: &Path,
sender: &str,
regex: &str,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<()> {
let current = resolve(store, legacy, mint).await?;
write_secret(
&dir.join(REGISTRATION),
&registration_yaml(sender, regex, &as_token, &hs_token),
&runtime.join(REGISTRATION),
&registration_yaml(sender, regex, &current.as_token, &current.hs_token),
)
}
/// The token at `path`, minting and writing one first when there is none.
fn existing_or_minted(path: &Path) -> Result<String> {
if let Some(t) = read_token(path)? {
return Ok(t);
}
let token = swarm_matrix_client::random_hex(TOKEN_BYTES)?;
write_secret(path, &token)?;
tracing::info!(path = %path.display(), "minted a swarm appservice token");
Ok(token)
}
/// The trimmed token at `path`; `None` when the file is missing or blank.
fn read_token(path: &Path) -> Result<Option<String>> {
match std::fs::read_to_string(path) {
@ -107,21 +122,43 @@ fn read_token(path: &Path) -> Result<Option<String>> {
}
}
/// A temp path beside `path`, unique per call and per process, so two writers
/// of the same file never share one. Same directory, so the rename stays
/// atomic.
fn temp_path(path: &Path) -> PathBuf {
static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let mut name = path.file_name().unwrap_or_default().to_os_string();
name.push(format!(
".{}-{}.tmp",
std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
));
path.with_file_name(name)
}
/// Write `contents` to `path` as a `0600` file, through a rename so a reader
/// never sees half of it.
/// never sees half of it. A failure removes the temp file.
fn write_secret(path: &Path, contents: &str) -> Result<()> {
let tmp = path.with_extension("tmp");
let _ = std::fs::remove_file(&tmp);
let tmp = temp_path(path);
let written = write_temp(&tmp, contents).and_then(|()| {
std::fs::rename(&tmp, path).with_context(|| format!("renaming onto {}", path.display()))
});
if written.is_err() {
let _ = std::fs::remove_file(&tmp);
}
written
}
fn write_temp(tmp: &Path, contents: &str) -> Result<()> {
let mut f = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&tmp)
.open(tmp)
.with_context(|| format!("creating {}", tmp.display()))?;
f.write_all(contents.as_bytes())
.and_then(|()| f.sync_all())
.with_context(|| format!("writing {}", tmp.display()))?;
std::fs::rename(&tmp, path).with_context(|| format!("renaming onto {}", path.display()))
.with_context(|| format!("writing {}", tmp.display()))
}
/// The registration tuwunel loads.
@ -146,7 +183,7 @@ fn registration_yaml(sender: &str, regex: &str, as_token: &str, hs_token: &str)
)
}
/// What [`sync`] needs of the store, so the decisions can be tested without
/// What [`render`] and [`sync`] need of the store, so the decisions can be tested without
/// one.
trait Store {
/// The registration's tokens, or `None` when nothing is stored.
@ -201,55 +238,55 @@ impl Store for Bao {
}
}
/// Make the state dir hold the store's registration, and the controller's token
/// match it.
/// Make the runtime dir hold the store's registration, and the controller's
/// token match it.
///
/// # Errors
/// If a variable is missing, the store refuses the login, a read or a write,
/// the state dir cannot be written, or the homeserver cannot be restarted.
/// the runtime dir cannot be written, or the homeserver cannot be restarted.
pub async fn sync() -> Result<()> {
let get = |k: &str| std::env::var(k).ok();
let dir = PathBuf::from(required(&get, ENV_DIR)?);
let runtime = PathBuf::from(required(&get, ENV_RUNTIME_DIR)?);
let legacy = PathBuf::from(required(&get, ENV_LEGACY_DIR)?);
let sender = required(&get, ENV_SENDER)?;
let regex = required(&get, ENV_USER_REGEX)?;
let cert_role = required(&get, ENV_CERT_ROLE)?;
let unit = required(&get, ENV_RESTART_UNIT)?;
let settings = Settings::from_env().context("reading the store's BAO_* environment")?;
let store = SecretStore::connect(&settings, &cert_role, DEFAULT_CERT_MOUNT)
.await
.with_context(|| {
format!("logging in to the swarm secret store as cert role {cert_role}")
})?;
sync_with(&Bao(store), &dir, &sender, &regex, mint, &Systemd(unit)).await
let store = connect(&required(&get, ENV_CERT_ROLE)?).await?;
sync_with(
&store,
&runtime,
&legacy,
&sender,
&regex,
mint,
&Systemd(unit),
)
.await
}
/// [`sync`] against an explicit store, directory and homeserver unit.
/// [`sync`] against an explicit store, directories and homeserver unit.
///
/// The store's registration wins. Where it holds none, the state dir's tokens
/// seed it, and only a host with neither mints a pair. The homeserver is
/// restarted whenever the registration was written after it last started, so
/// a run that failed between the write and the restart is finished by the next.
/// The homeserver is restarted whenever the registration was written after it
/// last started, so a run that failed between the write and the restart is
/// finished by the next.
async fn sync_with(
store: &impl Store,
dir: &Path,
runtime: &Path,
legacy: &Path,
sender: &str,
regex: &str,
mint: impl FnOnce() -> Result<Registration>,
unit: &impl Homeserver,
) -> Result<()> {
let local = read_local(dir)?;
let current = stored_or_seeded(store, local.as_ref(), mint).await?;
let current = resolve(store, legacy, mint).await?;
let path = runtime.join(REGISTRATION);
let want = registration_yaml(sender, regex, &current.as_token, &current.hs_token);
let have = std::fs::read_to_string(dir.join(REGISTRATION)).ok();
if local.as_ref() != Some(&current) || have.as_deref() != Some(&want) {
write_secret(&dir.join(AS_TOKEN), &current.as_token)?;
write_secret(&dir.join(HS_TOKEN), &current.hs_token)?;
write_secret(&dir.join(REGISTRATION), &want)?;
if std::fs::read_to_string(&path).ok().as_deref() != Some(&want) {
write_secret(&path, &want)?;
tracing::info!("took the swarm appservice registration from the store");
}
restart_if_stale(unit, &dir.join(REGISTRATION))?;
restart_if_stale(unit, &path)?;
if needs_publish(store.controller_token().await?.as_ref(), &current.as_token) {
store.write_controller_token(&current.as_token).await?;
@ -258,17 +295,31 @@ async fn sync_with(
Ok(())
}
/// The store's registration; where it holds none, `local` or a minted pair is
/// created there first. A refused create is a lost race or a denial, and the
/// read-back tells which: it finds the winner's value, or nothing.
async fn stored_or_seeded(
/// The registration the store holds. Where it holds none, the tokens an
/// earlier version kept in `legacy` are stored once (create-only, so nothing
/// rotates), and only a host with neither mints a pair. A store that cannot
/// be read is an error and mints nothing. The files stay where they are,
/// unused.
async fn resolve(
store: &impl Store,
legacy: &Path,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<Registration> {
match store.registration().await? {
Some(stored) => Ok(stored),
None => seed_store(store, read_local(legacy)?.as_ref(), mint).await,
}
}
/// Create the registration in the empty store from `local`, or from a minted
/// pair when there are no files, and return what the store then holds. A
/// refused create is a lost race or a denial, and the read-back tells which:
/// it finds the winner's pair, or nothing.
async fn seed_store(
store: &impl Store,
local: Option<&Registration>,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<Registration> {
if let Some(stored) = store.registration().await? {
return Ok(stored);
}
let seed = match local {
Some(l) => l.clone(),
None => mint()?,
@ -282,7 +333,7 @@ async fn stored_or_seeded(
.context("the store holds no swarm appservice registration after creating one")
}
/// The tokens on disk, when both are there.
/// The tokens in `dir`, when both are there.
fn read_local(dir: &Path) -> Result<Option<Registration>> {
let as_token = read_token(&dir.join(AS_TOKEN))?;
let hs_token = read_token(&dir.join(HS_TOKEN))?;
@ -447,18 +498,24 @@ mod tests {
const REGEX: &str = "^@[a-z0-9._=/-]+:t\\.local$";
/// A directory of its own per call. A counter, not the clock, keeps two
/// tests that start in the same clock tick out of one directory.
fn scratch() -> PathBuf {
/// A runtime dir and a legacy dir of their own per call; the legacy one
/// is not created, as on a host that never had the files. A counter, not
/// the clock, keeps two tests that start in the same clock tick out of one
/// directory.
fn scratch() -> (PathBuf, PathBuf) {
static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let dir = std::env::temp_dir().join(format!(
let root = std::env::temp_dir().join(format!(
"swarm-appservice-{}-{}",
std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
));
std::fs::remove_dir_all(&dir).ok();
std::fs::create_dir_all(&dir).expect("temp dir");
dir
std::fs::remove_dir_all(&root).ok();
std::fs::create_dir_all(root.join("run")).expect("temp dir");
(root.join("run"), root.join("legacy"))
}
fn cleanup(runtime: &Path) {
std::fs::remove_dir_all(runtime.parent().expect("a parent")).ok();
}
fn reg(tag: &str) -> Registration {
@ -469,18 +526,34 @@ mod tests {
}
/// A store in memory. `down` fails every call, as a sealed or unreachable
/// store does; `racer` is what another writer stores between this run's
/// read and its create.
/// store does; `deny_create` refuses a create and stores nothing; `racer`
/// is what another writer stores between this run's read and its create.
#[derive(Default)]
struct Fake {
registration: RefCell<Option<Registration>>,
token: RefCell<Option<String>>,
down: bool,
deny_create: bool,
racer: RefCell<Option<Registration>>,
creates: Cell<u32>,
token_writes: Cell<u32>,
}
impl Fake {
fn holding(r: Registration) -> Self {
let f = Self::default();
*f.registration.borrow_mut() = Some(r);
f
}
fn down() -> Self {
Self {
down: true,
..Self::default()
}
}
}
impl Store for Fake {
async fn registration(&self) -> Result<Option<Registration>> {
if self.down {
@ -491,6 +564,9 @@ mod tests {
async fn create_registration(&self, r: &Registration) -> Result<()> {
self.creates.set(self.creates.get() + 1);
if self.deny_create {
bail!("permission denied");
}
if let Some(winner) = self.racer.borrow_mut().take() {
*self.registration.borrow_mut() = Some(winner);
bail!("already created");
@ -513,8 +589,9 @@ mod tests {
}
}
/// Write the state dir as `render` leaves it for `r`.
fn host_with(dir: &Path, r: &Registration) {
/// The three files an earlier version left in the legacy dir for `r`.
fn legacy_with(dir: &Path, r: &Registration) {
std::fs::create_dir_all(dir).expect("legacy dir");
write_secret(&dir.join(AS_TOKEN), &r.as_token).expect("as");
write_secret(&dir.join(HS_TOKEN), &r.hs_token).expect("hs");
write_secret(
@ -524,11 +601,25 @@ mod tests {
.expect("yaml");
}
fn files(dir: &Path) -> Vec<String> {
[AS_TOKEN, HS_TOKEN, REGISTRATION]
.iter()
.map(|f| std::fs::read_to_string(dir.join(f)).unwrap_or_default())
.collect()
/// What is left of the legacy files: the dir's file names, sorted.
fn legacy_left(dir: &Path) -> Vec<String> {
let mut names: Vec<String> = std::fs::read_dir(dir)
.map(|d| {
d.map(|e| e.expect("entry").file_name().to_string_lossy().into_owned())
.collect()
})
.unwrap_or_default();
names.sort();
names
}
/// The registration the runtime dir holds, or empty.
fn rendered(runtime: &Path) -> String {
std::fs::read_to_string(runtime.join(REGISTRATION)).unwrap_or_default()
}
fn yaml_for(r: &Registration) -> String {
registration_yaml("swarm", REGEX, &r.as_token, &r.hs_token)
}
/// The homeserver unit. `started` is when it last became active; a restart
@ -574,12 +665,24 @@ mod tests {
}
}
/// Run one render, returning how often it minted.
async fn render_run(store: &Fake, runtime: &Path, legacy: &Path) -> (Result<()>, u32) {
let mints = Cell::new(0);
let out = render_with(store, runtime, legacy, "swarm", REGEX, || {
mints.set(mints.get() + 1);
Ok(reg("minted"))
})
.await;
(out, mints.get())
}
/// Run one sync against `unit`, returning how often it minted.
async fn run(store: &Fake, dir: &Path, unit: &Unit) -> (Result<()>, u32) {
async fn run(store: &Fake, runtime: &Path, legacy: &Path, unit: &Unit) -> (Result<()>, u32) {
let mints = Cell::new(0);
let out = sync_with(
store,
dir,
runtime,
legacy,
"swarm",
REGEX,
|| {
@ -592,28 +695,154 @@ mod tests {
(out, mints.get())
}
#[test]
fn the_rendered_registration_carries_the_token_it_minted() {
let dir = scratch();
render_into(&dir, "swarm", REGEX).expect("renders");
let yaml = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
let token = std::fs::read_to_string(dir.join(AS_TOKEN)).expect("minted");
assert_eq!(token.len(), TOKEN_BYTES * 2);
assert!(yaml.contains(&format!("\nas_token: {token}\n")), "{yaml}");
std::fs::remove_dir_all(&dir).ok();
#[tokio::test]
async fn an_empty_store_and_no_files_mint_a_pair_into_the_store() {
let (runtime, legacy) = scratch();
let store = Fake::default();
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(mints, 1);
assert_eq!(*store.registration.borrow(), Some(reg("minted")));
assert_eq!(rendered(&runtime), yaml_for(&reg("minted")));
assert!(!legacy.exists(), "nothing is written to the legacy dir");
cleanup(&runtime);
}
#[tokio::test]
async fn an_empty_store_takes_the_files_value_once() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("old"));
let store = Fake::default();
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(mints, 0, "a host that has the value mints nothing");
assert_eq!(*store.registration.borrow(), Some(reg("old")));
assert_eq!(rendered(&runtime), yaml_for(&reg("old")));
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_mints_nothing_and_renders_nothing() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let (out, mints) = render_run(&Fake::down(), &runtime, &legacy).await;
assert!(out.is_err(), "the unit fails and retries");
assert_eq!(mints, 0);
assert_eq!(rendered(&runtime), "", "nothing for tuwunel to load");
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_on_a_fresh_host_mints_nothing() {
let (runtime, legacy) = scratch();
let (out, mints) = render_run(&Fake::down(), &runtime, &legacy).await;
assert!(out.is_err());
assert_eq!(
mints, 0,
"a pair minted while the store is away is a rotation"
);
assert_eq!(rendered(&runtime), "");
cleanup(&runtime);
}
#[tokio::test]
async fn a_lost_race_adopts_the_winners_pair() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let store = Fake::default();
*store.racer.borrow_mut() = Some(reg("winner"));
let (out, _) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(rendered(&runtime), yaml_for(&reg("winner")));
cleanup(&runtime);
}
#[tokio::test]
async fn a_refused_create_fails_the_run() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let store = Fake {
deny_create: true,
..Fake::default()
};
let (out, _) = render_run(&store, &runtime, &legacy).await;
assert!(out.is_err(), "no read-back finds the pair");
assert_eq!(rendered(&runtime), "");
cleanup(&runtime);
}
#[tokio::test]
async fn a_second_render_keeps_the_pair() {
// "Only once": a re-mint on every boot would hand the controller a
// token the homeserver no longer loads until sync catches up.
let (runtime, legacy) = scratch();
let store = Fake::default();
render_run(&store, &runtime, &legacy)
.await
.0
.expect("first");
let first = rendered(&runtime);
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("second");
assert_eq!(mints, 0);
assert_eq!(rendered(&runtime), first);
cleanup(&runtime);
}
#[test]
fn a_second_render_keeps_the_token() {
// "Only once": a re-mint on every boot would hand the controller a
// token the homeserver no longer loads until sync catches up.
let dir = scratch();
render_into(&dir, "swarm", REGEX).expect("first");
let first = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
render_into(&dir, "swarm", REGEX).expect("second");
let second = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
assert_eq!(first, second);
std::fs::remove_dir_all(&dir).ok();
fn two_writers_of_one_file_use_different_temp_files() {
let path = Path::new("/run/x/swarm.yaml");
let (a, b) = (temp_path(path), temp_path(path));
assert_ne!(a, b);
assert_eq!(
a.parent(),
path.parent(),
"the rename stays in one directory"
);
}
#[test]
fn overlapping_writers_all_succeed_and_leave_no_temp_file() {
let (runtime, _) = scratch();
let path = runtime.join(REGISTRATION);
let writers: Vec<_> = (0..8)
.map(|_| {
let path = path.clone();
std::thread::spawn(move || {
(0..50)
.try_for_each(|_| write_secret(&path, "same\n"))
.is_ok()
})
})
.collect();
assert!(writers.into_iter().all(|w| w.join().expect("joins")));
assert_eq!(std::fs::read_to_string(&path).expect("written"), "same\n");
assert_eq!(legacy_left(&runtime), [REGISTRATION]);
cleanup(&runtime);
}
#[test]
fn a_failed_write_removes_its_temp_file() {
let (runtime, _) = scratch();
// A directory where the file goes: the rename fails.
std::fs::create_dir(runtime.join(REGISTRATION)).expect("dir");
write_secret(&runtime.join(REGISTRATION), "x").expect_err("cannot replace a directory");
assert_eq!(legacy_left(&runtime), [REGISTRATION]);
cleanup(&runtime);
}
#[test]
@ -624,172 +853,86 @@ mod tests {
assert!(y.contains("\n - exclusive: false\n"), "{y}");
assert!(y.contains(&format!("regex: '{REGEX}'")), "{y}");
assert!(y.contains("\nurl: null\n"), "{y}");
assert!(y.contains("\nas_token: aa\n"), "{y}");
assert!(y.contains("\nhs_token: bb\n"), "{y}");
}
#[test]
fn the_registration_and_tokens_are_owner_only() {
#[tokio::test]
async fn the_rendered_registration_is_owner_only() {
use std::os::unix::fs::PermissionsExt as _;
let dir = scratch();
render_into(&dir, "swarm", REGEX).expect("renders");
for f in [AS_TOKEN, HS_TOKEN, REGISTRATION] {
let mode = std::fs::metadata(dir.join(f))
.expect("exists")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600, "{f}");
}
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn seeding_from_the_host_leaves_every_file_unchanged() {
// A host that has run for months with the store empty: the store takes
// its value, and nothing on the host moves, so nothing rotates.
let dir = scratch();
let on_disk = reg("old");
host_with(&dir, &on_disk);
let before = files(&dir);
let store = Fake::default();
let unit = Unit::just_started();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(*store.registration.borrow(), Some(on_disk));
assert_eq!(files(&dir), before);
assert_eq!(mints, 0, "a host that has the value mints nothing");
assert_eq!(restarts, 0, "an unchanged host does not restart");
assert_eq!(store.token.borrow().as_deref(), Some("as-old"));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn the_store_wins_when_both_hold_a_value() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
*store.token.borrow_mut() = Some("as-disk".to_owned());
let unit = Unit::long_running();
let (out, _) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
let want = registration_yaml("swarm", REGEX, "as-store", "hs-store");
assert_eq!(
files(&dir),
vec!["as-store".to_owned(), "hs-store".to_owned(), want]
);
assert_eq!(*store.registration.borrow(), Some(reg("store")));
assert_eq!(store.creates.get(), 0, "a held value is never rewritten");
assert_eq!(restarts, 1);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_host_without_the_files_takes_them_from_the_store() {
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unit::long_running();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(read_local(&dir).expect("reads"), Some(reg("store")));
assert!(dir.join(REGISTRATION).exists());
assert_eq!((restarts, mints), (1, 0));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_fresh_host_mints_into_the_store() {
let dir = scratch();
let store = Fake::default();
let unit = Unit::long_running();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(mints, 1);
assert_eq!(*store.registration.borrow(), Some(reg("minted")));
assert_eq!(read_local(&dir).expect("reads"), Some(reg("minted")));
assert_eq!(restarts, 1);
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_lost_race_adopts_the_winners_registration() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let store = Fake::default();
*store.racer.borrow_mut() = Some(reg("winner"));
let unit = Unit::long_running();
let (out, _) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(read_local(&dir).expect("reads"), Some(reg("winner")));
assert_eq!(restarts, 1);
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_store_that_is_down_leaves_the_host_as_it_is() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let before = files(&dir);
let store = Fake {
down: true,
..Fake::default()
};
let unit = Unit::just_started();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
assert!(out.is_err(), "the unit reports the outage");
assert_eq!(files(&dir), before, "the homeserver keeps the last copy");
assert_eq!((restarts, mints), (0, 0));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn the_start_path_renders_the_copy_sync_took_from_the_store() {
// `render` has no store: after an outage it renders what the last sync
// wrote, without minting over it.
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
run(&store, &dir, &Unit::long_running())
let (runtime, legacy) = scratch();
render_run(&Fake::default(), &runtime, &legacy)
.await
.0
.expect("syncs");
let synced = files(&dir);
.expect("renders");
let mode = std::fs::metadata(runtime.join(REGISTRATION))
.expect("exists")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600);
cleanup(&runtime);
}
render_into(&dir, "swarm", REGEX).expect("renders");
#[tokio::test]
async fn the_store_wins_when_the_rendered_copy_differs() {
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("old"))).expect("rendered");
let store = Fake::holding(reg("store"));
*store.token.borrow_mut() = Some("as-old".to_owned());
assert_eq!(files(&dir), synced);
std::fs::remove_dir_all(&dir).ok();
let unit = Unit::long_running();
let (out, _) = run(&store, &runtime, &legacy, &unit).await;
out.expect("syncs");
assert_eq!(rendered(&runtime), yaml_for(&reg("store")));
assert_eq!(*store.registration.borrow(), Some(reg("store")));
assert_eq!(store.creates.get(), 0, "a held value is never rewritten");
assert_eq!(unit.restarts.get(), 1);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
cleanup(&runtime);
}
#[tokio::test]
async fn a_sync_with_nothing_rendered_takes_the_stores_pair() {
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
let (out, mints) = run(&store, &runtime, &legacy, &unit).await;
out.expect("syncs");
assert_eq!(rendered(&runtime), yaml_for(&reg("store")));
assert_eq!((unit.restarts.get(), mints), (1, 0));
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_leaves_the_rendered_copy_as_it_is() {
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("disk"))).expect("rendered");
let unit = Unit::just_started();
let (out, mints) = run(&Fake::down(), &runtime, &legacy, &unit).await;
assert!(out.is_err(), "the unit reports the outage");
assert_eq!(rendered(&runtime), yaml_for(&reg("disk")));
assert_eq!((unit.restarts.get(), mints), (0, 0));
cleanup(&runtime);
}
#[tokio::test]
async fn a_second_sync_restarts_nothing() {
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
run(&store, &dir, &unit).await.0.expect("first");
run(&store, &runtime, &legacy, &unit)
.await
.0
.expect("first");
let first = unit.restarts.get();
let (out, _) = run(&store, &dir, &unit).await;
let (out, _) = run(&store, &runtime, &legacy, &unit).await;
let second = unit.restarts.get() - first;
out.expect("syncs");
@ -799,48 +942,48 @@ mod tests {
1,
"the controller's copy is written once"
);
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
#[tokio::test]
async fn a_restart_that_failed_is_retried_by_the_next_run() {
// The files are written and the restart is refused. The next run finds
// The file is written and the restart is refused. The next run finds
// nothing to write, and still restarts: the decision comes from the
// files' age against the unit's start, not from this run's writes.
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
// file's age against the unit's start, not from this run's writes.
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
unit.fail_restart.set(true);
let (first, _) = run(&store, &dir, &unit).await;
let (first, _) = run(&store, &runtime, &legacy, &unit).await;
assert!(first.is_err(), "the refused restart fails the run");
let written = files(&dir);
let written = rendered(&runtime);
unit.fail_restart.set(false);
let (second, _) = run(&store, &dir, &unit).await;
let (second, _) = run(&store, &runtime, &legacy, &unit).await;
second.expect("the retry succeeds");
assert_eq!(files(&dir), written, "nothing was left to write");
assert_eq!(rendered(&runtime), written, "nothing was left to write");
assert_eq!(unit.restarts.get(), 2);
assert_eq!(read_local(&dir).expect("reads"), Some(reg("store")));
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
#[tokio::test]
async fn a_unit_that_started_after_the_files_is_left_alone() {
// The control for the retry above: the same files, a unit that started
// after them, and no restart.
let dir = scratch();
host_with(&dir, &reg("store"));
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
async fn a_unit_that_started_after_the_file_is_left_alone() {
// The control for the retry above: the same file, a unit that started
// after it, and no restart.
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("store"))).expect("rendered");
let store = Fake::holding(reg("store"));
let unit = Unit::just_started();
run(&store, &dir, &unit).await.0.expect("syncs");
run(&store, &runtime, &legacy, &unit)
.await
.0
.expect("syncs");
assert_eq!(unit.restarts.get(), 0);
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
#[tokio::test]
@ -855,17 +998,25 @@ mod tests {
Ok(())
}
}
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unreadable(Cell::new(0));
let out = sync_with(&store, &dir, "swarm", REGEX, || Ok(reg("minted")), &unit).await;
let out = sync_with(
&store,
&runtime,
&legacy,
"swarm",
REGEX,
|| Ok(reg("minted")),
&unit,
)
.await;
out.expect("the run goes on");
assert_eq!(unit.0.get(), 0);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
std::fs::remove_dir_all(&dir).ok();
cleanup(&runtime);
}
/// `restart_via` against a `systemctl` that exits with the given codes for the
@ -939,7 +1090,8 @@ mod tests {
#[test]
fn every_variable_is_scoped_to_the_verb() {
for var in [
ENV_DIR,
ENV_RUNTIME_DIR,
ENV_LEGACY_DIR,
ENV_SENDER,
ENV_USER_REGEX,
ENV_CERT_ROLE,

17
swarm-matrix-ctl/src/main.rs
View file

@ -7,8 +7,8 @@
//! identity plumbing to add one action, so the next thing that has to run in
//! here is a verb below, not a new crate.
//!
//! [`appservice`] mints the **swarm's** own appservice registration and
//! keeps it in step with the store and publishes its token for `swarm-controller`.
//! [`appservice`] renders the **swarm's** own appservice registration from
//! the store and publishes its token for `swarm-controller`.
//!
//! 🩸 **A secret is a path, never a value.** The only identifier any verb here
//! logs is the store path; see `swarm_matrix_client`'s module doc for the same rule
@ -39,12 +39,13 @@ enum Command {
#[derive(Debug, Subcommand)]
enum Appservice {
/// Mint the tokens when absent and render the registration tuwunel loads.
/// Local only: it runs before the homeserver and must not need a network.
/// Render the registration tuwunel loads from the store, minting the tokens
/// first when the store holds none. Runs before the homeserver, which stays
/// down until the store answers.
Render,
/// Make the state dir hold the store's registration, seeding the store
/// from it when the store holds none, and restart the homeserver when the
/// registration is newer than its last start. Runs beside the homeserver, never before it.
/// Make the rendered registration match the store's, restart the homeserver
/// when the registration is newer than its last start, and publish the
/// `as_token` for `swarm-controller`. Runs beside the homeserver.
Sync,
}
@ -58,7 +59,7 @@ async fn main() -> Result<()> {
.init();
match Cli::parse().command {
Command::Appservice(Appservice::Render) => appservice::render(),
Command::Appservice(Appservice::Render) => appservice::render().await,
Command::Appservice(Appservice::Sync) => appservice::sync().await,
}
}