matrix: swarm-controller is the only minter
Every hive is in a swarm and every swarm runs matrix, so every swarm has a swarm-controller, and since #4810 its hive_sender pass mints each hive's @hive-<hive>: sender token into the store every five minutes. The two other minters of that token go: - swarm-matrix-ctl mint: the systemd.services.swarm-matrix-ctl unit in the hive-matrix container, Command::Mint and src/mint.rs. The binary, its appservice render/publish verbs, ctlPackage, ctlActive and the ctl cert role stay. bao-matrix-reader's checks on the deleted unit are removed; the leaf-identity and no-token-in-env checks now look at swarm-matrix-appservice-publish, which runs under the same identity. - the hive-side mint ladder in hive-c0re's ensure_hive_user (register/appservice-login/password-login with the local as_token), with read_appservice_token, paths::matrix_appservice_token and the helpers only it used. ensure_hive_user now takes the store's token, keeps the file when the store has none or can't be reached, and fails otherwise. - hivectl matrix sync-admin: the verb, HostRequest::MatrixSyncAdmin and handle_matrix_sync_admin. The periodic MatrixSweep (ensure_all) is unchanged apart from no longer reading the local as_token. This removes the double-mint race #4810's review flagged: two minters logging in on one pinned device could leave a dead token in the store until the next pass. Closes #4813 Closes #4814
This commit is contained in:
atlas
mara
parent
91e47732a6
commit
ddb7d7196d
22 changed files with 187 additions and 1162 deletions
|
|
@ -10,13 +10,11 @@ path = "src/main.rs"
|
|||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
# One verb today (`mint`), and the reason this crate is a `*ctl` rather than a
|
||||
# single-purpose binary: the next thing that has to run in the matrix container
|
||||
# is a subcommand here, not a new crate.
|
||||
# The reason this crate is a `*ctl` rather than a single-purpose binary: the
|
||||
# next thing that has to run in the matrix container is a subcommand here, not a
|
||||
# new crate.
|
||||
clap.workspace = true
|
||||
serde_json.workspace = true
|
||||
# The appservice calls, shared with `swarm-controller`, which mints agents'
|
||||
# accounts through the same device id.
|
||||
# The token generator, shared with `swarm-controller`.
|
||||
swarm-matrix-client.workspace = true
|
||||
# The agreement this binary is one end of: where the credential lives, what the
|
||||
# object at that path holds, and the `BAO_*` spellings the unit sets.
|
||||
|
|
|
|||
|
|
@ -11,52 +11,21 @@ crate.
|
|||
|
||||
## Verbs
|
||||
|
||||
### `mint`
|
||||
### `appservice render`
|
||||
|
||||
Puts the appservice sender account's access token into the swarm's secret store,
|
||||
under an identity of its own. A boot-time oneshot.
|
||||
Mints the swarm appservice registration's tokens when absent and renders the
|
||||
registration tuwunel loads. Runs before the homeserver and needs no network.
|
||||
|
||||
Configured entirely by the `MATRIX_MINT_*` environment the unit sets — no flags.
|
||||
A systemd `Environment=` block is what a nix module can render; a command line
|
||||
full of paths is not. The prefix is scoped to the verb rather than to the binary
|
||||
so the next verb brings its own, instead of widening a shared one nobody can
|
||||
then narrow.
|
||||
### `appservice publish`
|
||||
|
||||
## Why this lives in the matrix container
|
||||
Writes the rendered `as_token` to the swarm secret store for `swarm-controller`,
|
||||
when the store's copy differs.
|
||||
|
||||
The credential `mint` writes is authorised by the appservice `as_token`, and the
|
||||
container already holds that: `nix/host-modules/hive-matrix.nix` bind-mounts the
|
||||
rendered appservice registration into it read-only, because that is how tuwunel
|
||||
itself is handed the registration. Minting anywhere else would mean copying the
|
||||
`as_token` to a second holder — and the point of this component is that the hive
|
||||
stops being one.
|
||||
|
||||
It is not the swarm controller for the same reason, plus a structural one: a
|
||||
homeserver has exactly **one** appservice registration and so one sender
|
||||
account, and a swarm runs one homeserver, so "mint it once" needs no lock, no
|
||||
lease and no trigger surface — it is a property of the thing being minted.
|
||||
|
||||
## Idempotency
|
||||
|
||||
The **store** is the key, not the homeserver. A `mint` run reads
|
||||
`swarm/services/matrix/sender-token` first and returns without touching the
|
||||
homeserver when something is already there. Only an empty path reaches the mint
|
||||
ladder:
|
||||
|
||||
1. `POST /_matrix/client/v3/register` with `"type": "m.login.application_service"`
|
||||
— one round trip, no UIAA.
|
||||
2. `M_USER_IN_USE` (the expected arm on a homeserver that has already loaded the
|
||||
registration, since the account is the appservice's own `sender_localpart`) →
|
||||
`POST /_matrix/client/v3/login` as the appservice, same pinned `device_id`, so
|
||||
the old device is replaced rather than duplicated.
|
||||
3. Write the result to the store.
|
||||
|
||||
A crash between the homeserver call and the store write is recoverable: the next
|
||||
run takes arm 2.
|
||||
Both are configured entirely by the `MATRIX_APPSERVICE_*` environment the units
|
||||
set — no flags. A systemd `Environment=` block is what a nix module can render; a
|
||||
command line full of paths is not.
|
||||
|
||||
## 🩸 A secret is a path, never a value
|
||||
|
||||
Nothing here logs, prints or interpolates a token. The mint ladder's errors are
|
||||
built from the homeserver's _status_ and its `errcode`, never its body, because a
|
||||
`/login` response body is an access token. The one identifier this binary logs is
|
||||
the store path it wrote.
|
||||
Nothing here logs, prints or interpolates a token. The one identifier this
|
||||
binary logs is the store path it wrote.
|
||||
|
|
|
|||
|
|
@ -7,20 +7,14 @@
|
|||
//! identity plumbing to add one action, so the next thing that has to run in
|
||||
//! here is a verb below, not a new crate.
|
||||
//!
|
||||
//! [`mint`] publishes a hive's appservice sender token to the swarm's secret
|
||||
//! store, once. [`appservice`] mints the **swarm's** own appservice
|
||||
//! registration and publishes its token for `swarm-controller`.
|
||||
//!
|
||||
//! It lives in the container because the appservice `as_token` that authorises
|
||||
//! the mint is *already* there — the registration tuwunel loads is bind-mounted
|
||||
//! in — so no second holder of that secret is created.
|
||||
//! [`appservice`] mints the **swarm's** own appservice registration and
|
||||
//! publishes its token for `swarm-controller`.
|
||||
//!
|
||||
//! 🩸 **A secret is a path, never a value.** The only identifier any verb here
|
||||
//! logs is the store path; see `swarm_matrix_client`'s module doc for the same rule
|
||||
//! applied to error messages.
|
||||
|
||||
mod appservice;
|
||||
mod mint;
|
||||
mod registration;
|
||||
|
||||
use anyhow::Result;
|
||||
|
|
@ -38,13 +32,6 @@ struct Cli {
|
|||
|
||||
#[derive(Debug, Subcommand)]
|
||||
enum Command {
|
||||
/// Publish the appservice sender account's access token to the swarm
|
||||
/// secret store, once.
|
||||
///
|
||||
/// Configured entirely by the `MATRIX_MINT_*` environment the unit sets —
|
||||
/// no flags, because a systemd `Environment=` block is what a nix module
|
||||
/// can render and a command line full of paths is not.
|
||||
Mint,
|
||||
/// The swarm's own appservice registration, whose sender is the
|
||||
/// homeserver's admin account. Configured by `MATRIX_APPSERVICE_*`.
|
||||
#[command(subcommand)]
|
||||
|
|
@ -71,7 +58,6 @@ async fn main() -> Result<()> {
|
|||
.init();
|
||||
|
||||
match Cli::parse().command {
|
||||
Command::Mint => mint::run().await,
|
||||
Command::Appservice(Appservice::Render) => appservice::render(),
|
||||
Command::Appservice(Appservice::Publish) => appservice::publish().await,
|
||||
}
|
||||
|
|
@ -87,17 +73,8 @@ mod tests {
|
|||
Cli::command().debug_assert();
|
||||
}
|
||||
|
||||
/// The unit's `ExecStart` names a verb, so a rename of it is a deploy-time
|
||||
/// failure with no local signal. This is that signal.
|
||||
#[test]
|
||||
fn mint_is_spelled_the_way_the_unit_invokes_it() {
|
||||
let cli = Cli::try_parse_from(["swarm-matrix-ctl", "mint"]).expect("`mint` is a verb");
|
||||
assert!(matches!(cli.command, Command::Mint));
|
||||
}
|
||||
|
||||
/// The control: without it the case above passes on a parser that accepts
|
||||
/// anything.
|
||||
/// The two units name these verbs, same reason as the test above.
|
||||
/// The two units name these verbs in `ExecStart`, so a rename of one is a
|
||||
/// deploy-time failure with no local signal. This is that signal.
|
||||
#[test]
|
||||
fn the_appservice_verbs_are_spelled_the_way_the_units_invoke_them() {
|
||||
let cli =
|
||||
|
|
@ -122,9 +99,9 @@ mod tests {
|
|||
.expect_err("only declared verbs are accepted");
|
||||
}
|
||||
|
||||
/// A bare invocation must not silently do something. `mint` writes a
|
||||
/// credential, so "no verb" defaulting to it would make a typo in the unit
|
||||
/// mint rather than fail.
|
||||
/// A bare invocation must not silently do something. `appservice publish`
|
||||
/// writes a credential, so "no verb" defaulting to it would make a typo in
|
||||
/// the unit write rather than fail.
|
||||
#[test]
|
||||
fn no_verb_at_all_is_refused() {
|
||||
Cli::try_parse_from(["swarm-matrix-ctl"]).expect_err("a verb is required");
|
||||
|
|
|
|||
|
|
@ -1,303 +0,0 @@
|
|||
//! `swarm-matrix-ctl mint` — publish the appservice sender account's
|
||||
//! homeserver access token to the swarm's secret store, once.
|
||||
//!
|
||||
//! A oneshot inside `containers.hive-matrix`, not a daemon and not part of the
|
||||
//! swarm controller. It mints for **one hive** — the hive this container runs
|
||||
//! on, named by [`ENV_HIVE`] — and publishes to that hive's own path, so a
|
||||
//! swarm whose hives share a homeserver gets one account and one token per
|
||||
//! hive rather than one shared between all of them. "Only once" is therefore
|
||||
//! once per hive, and it is still a property of what is being minted rather
|
||||
//! than of a lock: nothing else writes that path.
|
||||
//!
|
||||
//! The store, not the homeserver, is the idempotency key — see
|
||||
//! [`already_published`]. On the hive side `hive-c0re`'s
|
||||
//! `matrix::ensure_hive_user` reads exactly the path written here, which is how
|
||||
//! a hive that holds no `as_token` still gets its matrix account.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use swarm_secret_client::{
|
||||
SecretStore,
|
||||
client::{DEFAULT_CERT_MOUNT, Settings},
|
||||
matrix,
|
||||
};
|
||||
|
||||
use swarm_matrix_client as homeserver;
|
||||
|
||||
use crate::registration;
|
||||
|
||||
/// Role on the store's `cert` auth mount to log in with. Its policy is what
|
||||
/// allows the write below; the certificate the `BAO_*` variables name has to
|
||||
/// carry the CN that role accepts.
|
||||
const ENV_CERT_ROLE: &str = "MATRIX_MINT_CERT_ROLE";
|
||||
/// Client-server API base of the homeserver beside us — loopback, since the
|
||||
/// container shares the host netns.
|
||||
const ENV_API_URL: &str = "MATRIX_MINT_API_URL";
|
||||
/// The bind-mounted appservice registration, which is where the `as_token`
|
||||
/// comes from. A path, never a value.
|
||||
const ENV_REGISTRATION: &str = "MATRIX_MINT_REGISTRATION";
|
||||
/// Localpart of this hive's sender account. The registration's own
|
||||
/// `sender_localpart`, rendered by `hive-matrix.nix` from the hive name — the
|
||||
/// same string `swarm_secret_client::matrix::hive_localpart` builds, which is
|
||||
/// what `hive-c0re` derives its own copy with.
|
||||
const ENV_LOCALPART: &str = "MATRIX_MINT_LOCALPART";
|
||||
/// Name of the hive this container belongs to, and so the segment of the store
|
||||
/// path the token is published under. It is what keeps one hive's token out of
|
||||
/// another hive's reach — see `swarm_secret_client::matrix::sender_token_path`.
|
||||
const ENV_HIVE: &str = "MATRIX_MINT_HIVE";
|
||||
/// Public base URL of the homeserver, stored beside the token so a reader can
|
||||
/// reconstruct where it is good for. Optional: a swarm with no gateway vhost
|
||||
/// has no such URL, and `matrix::Credential` types the field to say so.
|
||||
const ENV_HOMESERVER: &str = "MATRIX_MINT_HOMESERVER";
|
||||
|
||||
/// Everything the unit tells this verb, checked before anything is opened.
|
||||
///
|
||||
/// Separate from the work for the reason `swarm_secret_client::client::Settings`
|
||||
/// is: every arm is a misconfiguration an operator reads an error about, and
|
||||
/// none of them needs a reachable homeserver or store to happen.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
struct Config {
|
||||
cert_role: String,
|
||||
api_url: String,
|
||||
registration: String,
|
||||
localpart: String,
|
||||
hive: String,
|
||||
homeserver: Option<String>,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
/// Read the `MATRIX_MINT_*` variables from the process environment.
|
||||
///
|
||||
/// # Errors
|
||||
/// Naming the first variable that is unset or empty.
|
||||
fn from_env() -> Result<Self> {
|
||||
Self::from_lookup(|k| std::env::var(k).ok())
|
||||
}
|
||||
|
||||
/// [`Config::from_env`] against an arbitrary lookup.
|
||||
///
|
||||
/// # Errors
|
||||
/// Naming the first variable that is unset or empty.
|
||||
fn from_lookup(get: impl Fn(&str) -> Option<String>) -> Result<Self> {
|
||||
let required = |var: &'static str| -> Result<String> {
|
||||
get(var)
|
||||
.filter(|v| !v.is_empty())
|
||||
.with_context(|| format!("{var} is unset or empty"))
|
||||
};
|
||||
Ok(Self {
|
||||
cert_role: required(ENV_CERT_ROLE)?,
|
||||
api_url: required(ENV_API_URL)?,
|
||||
registration: required(ENV_REGISTRATION)?,
|
||||
localpart: required(ENV_LOCALPART)?,
|
||||
hive: required(ENV_HIVE)?,
|
||||
// Empty is absent: systemd renders an unset nix option as
|
||||
// `Environment=VAR=`, so that is the shape this arrives in.
|
||||
homeserver: get(ENV_HOMESERVER).filter(|v| !v.is_empty()),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Is the credential already in the store?
|
||||
///
|
||||
/// **This read is the "and only once".** The homeserver is not asked — a
|
||||
/// re-run of the container, or of this unit, costs one store read and stops.
|
||||
/// It is also the read-back of what a previous run wrote, so the path published
|
||||
/// and the path consulted cannot drift apart: they are one function call.
|
||||
///
|
||||
/// A failure to read is reported and treated as absent rather than raised. The
|
||||
/// two cases that reach it are a path that has never been written (the first
|
||||
/// run, which must go on to mint) and a token whose policy does not cover the
|
||||
/// path — and the second fails again, loudly and with the store's own message,
|
||||
/// at the write below.
|
||||
async fn already_published(store: &SecretStore, path: &str) -> bool {
|
||||
match store.read::<matrix::Credential>(path).await {
|
||||
Ok(credential) => !credential.value.trim().is_empty(),
|
||||
Err(e) => {
|
||||
tracing::info!(%path, error = %e, "nothing readable in the store yet");
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Run the verb.
|
||||
///
|
||||
/// # Errors
|
||||
/// If the environment is incomplete, the store refuses the login or the write,
|
||||
/// the registration cannot be read, or the homeserver refuses both the
|
||||
/// registration and the appservice login.
|
||||
pub async fn run() -> Result<()> {
|
||||
let config = Config::from_env()?;
|
||||
// Explicitly, rather than through `SecretStore::from_env`: a missing or
|
||||
// misspelled `BAO_*` variable is the most likely thing to be wrong with a
|
||||
// freshly deployed unit, and this reports it before the homeserver is
|
||||
// touched at all.
|
||||
let settings = Settings::from_env().context("reading the store's BAO_* environment")?;
|
||||
let store = SecretStore::connect(&settings, &config.cert_role, DEFAULT_CERT_MOUNT)
|
||||
.await
|
||||
.with_context(|| {
|
||||
format!(
|
||||
"logging in to the swarm secret store as cert role {}",
|
||||
config.cert_role
|
||||
)
|
||||
})?;
|
||||
|
||||
let path = matrix::sender_token_path(&config.hive)
|
||||
.with_context(|| format!("building the store path for hive {}", config.hive))?;
|
||||
if already_published(&store, &path).await {
|
||||
tracing::info!(%path, "the sender token is already published; not minting");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let as_token = registration::as_token(&config.registration)?;
|
||||
let http = homeserver::client()?;
|
||||
let token =
|
||||
match homeserver::register(&http, &config.api_url, &config.localpart, &as_token).await? {
|
||||
homeserver::Registered::Token(token) => token,
|
||||
homeserver::Registered::AlreadyExists => {
|
||||
// The expected arm, not an edge case: this account is the
|
||||
// appservice's own `sender_localpart`, so the homeserver creates it
|
||||
// when it loads the registration — before anything gets to ask.
|
||||
tracing::info!("the sender account exists; logging in as the appservice instead");
|
||||
homeserver::appservice_login(&http, &config.api_url, &config.localpart, &as_token)
|
||||
.await?
|
||||
}
|
||||
};
|
||||
|
||||
store
|
||||
.write(
|
||||
&path,
|
||||
&matrix::Credential {
|
||||
value: token,
|
||||
homeserver: config.homeserver,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.with_context(|| format!("writing the sender token to {path}"))?;
|
||||
tracing::info!(%path, "published the sender token");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A lookup standing in for a fully-configured unit's environment.
|
||||
fn full(k: &str) -> Option<String> {
|
||||
match k {
|
||||
ENV_CERT_ROLE => Some("swarm-matrix-ctl".to_owned()),
|
||||
ENV_API_URL => Some("http://127.0.0.1:8008".to_owned()),
|
||||
ENV_REGISTRATION => {
|
||||
Some("/var/lib/hyperhive/matrix-appservice/hyperhive.yaml".to_owned())
|
||||
}
|
||||
ENV_LOCALPART => Some("hive-pr1ma".to_owned()),
|
||||
ENV_HIVE => Some("pr1ma".to_owned()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_complete_environment_is_accepted() {
|
||||
// The control: without it every assertion below could be passing
|
||||
// because `from_lookup` rejects everything.
|
||||
let c = Config::from_lookup(full).expect("every required variable is set");
|
||||
assert_eq!(c.localpart, "hive-pr1ma");
|
||||
assert_eq!(c.hive, "pr1ma");
|
||||
assert_eq!(c.homeserver, None, "an absent public URL is not an error");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn each_required_variable_is_named_when_it_is_the_missing_one() {
|
||||
for var in [
|
||||
ENV_CERT_ROLE,
|
||||
ENV_API_URL,
|
||||
ENV_REGISTRATION,
|
||||
ENV_LOCALPART,
|
||||
ENV_HIVE,
|
||||
] {
|
||||
let e = Config::from_lookup(|k| if k == var { None } else { full(k) })
|
||||
.expect_err("one required variable is absent");
|
||||
assert!(
|
||||
format!("{e}").contains(var),
|
||||
"dropping {var} should name {var}, got {e}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_empty_variable_is_as_absent_as_an_unset_one() {
|
||||
// systemd writes `Environment=VAR=` for an unset nix option, so empty
|
||||
// is the shape these actually arrive in.
|
||||
let e = Config::from_lookup(|k| {
|
||||
if k == ENV_CERT_ROLE {
|
||||
Some(String::new())
|
||||
} else {
|
||||
full(k)
|
||||
}
|
||||
})
|
||||
.expect_err("an empty role is not a role");
|
||||
assert!(format!("{e}").contains(ENV_CERT_ROLE), "{e}");
|
||||
|
||||
let c = Config::from_lookup(|k| {
|
||||
if k == ENV_HOMESERVER {
|
||||
Some(String::new())
|
||||
} else {
|
||||
full(k)
|
||||
}
|
||||
})
|
||||
.expect("an empty public URL is optional, not fatal");
|
||||
assert_eq!(c.homeserver, None);
|
||||
}
|
||||
|
||||
/// The environment prefix is a contract with the nix unit, and the crate
|
||||
/// rename that produced it moved every one of these. A verb-scoped prefix
|
||||
/// is the point: the next verb brings its own, instead of widening a
|
||||
/// binary-scoped one nobody can then narrow.
|
||||
#[test]
|
||||
fn every_variable_is_scoped_to_the_verb() {
|
||||
for var in [
|
||||
ENV_CERT_ROLE,
|
||||
ENV_API_URL,
|
||||
ENV_REGISTRATION,
|
||||
ENV_LOCALPART,
|
||||
ENV_HIVE,
|
||||
ENV_HOMESERVER,
|
||||
] {
|
||||
assert!(
|
||||
var.starts_with("MATRIX_MINT_"),
|
||||
"{var} is not scoped to the mint verb"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_published_path_is_the_one_the_hive_reads() {
|
||||
// Both ends of this slice's loop resolve the same function, so there is
|
||||
// no second spelling to drift — this pins that the loop exists at all,
|
||||
// and names the literal so a move of the path is a deliberate edit on
|
||||
// both sides rather than a silent 404 on the reading one.
|
||||
assert_eq!(
|
||||
matrix::sender_token_path("pr1ma").expect("a plain name is legal"),
|
||||
"swarm/hives/pr1ma/matrix/sender-token"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_hives_are_published_to_two_paths() {
|
||||
// What the hive segment is FOR: this binary runs beside a homeserver
|
||||
// several hives share, so a path without the hive name in it would
|
||||
// have each run overwrite the last and leave every hive holding one
|
||||
// identity — which is the shape this change exists to end.
|
||||
let a = matrix::sender_token_path("alpha").expect("legal");
|
||||
let b = matrix::sender_token_path("beta").expect("legal");
|
||||
assert_ne!(a, b);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_localpart_the_unit_hands_over_is_the_one_derived_from_the_hive() {
|
||||
// The nix unit renders both variables independently; this pins that
|
||||
// the pair it is expected to render agrees with the shared derivation,
|
||||
// so a unit still passing the old bare `hive` fails here rather than
|
||||
// silently logging in as another hive's account.
|
||||
let c = Config::from_lookup(full).expect("every required variable is set");
|
||||
assert_eq!(c.localpart, matrix::hive_localpart(&c.hive));
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue