diff --git a/docs/integrations/matrix.md b/docs/integrations/matrix.md index f2a04edc..05fb932f 100644 --- a/docs/integrations/matrix.md +++ b/docs/integrations/matrix.md @@ -183,7 +183,8 @@ hive's standing leaves the others alone. Its access token is the **sender token**, and it's the credential hive-c0re presents for every homeserver call it makes on the hive's behalf. It's per hive for the same reason the account is: -`swarm-matrix-ctl` mints it inside the `hive-matrix` container and +`swarm-controller` mints it for every hive with the swarm's appservice token (and +`swarm-matrix-ctl`, inside the `hive-matrix` container, for its own hive) and publishes it to `swarm/hives//matrix/sender-token`, and the hive reads it from there under its own certificate. That path sits inside the hive's own read grant (`swarm/hives//*`), so a hive fetches its own diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 8723a49f..99aabd7c 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -59,20 +59,20 @@ of the cell says how. -| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `swarm/agents//matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | -| `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | -| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot, where the controller registers its client; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | -| `swarm/hives//matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | must be stated | -| `swarm/hives//matrix/sender-token` | `swarm-matrix-ctl`, in the `hive-matrix` container | `swarm-matrix-ctl` itself, under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | must be stated | must be stated | -| `swarm/hives//queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | -| `swarm/services//oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated | must be stated | -| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | +| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | +| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `swarm/agents//matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | +| `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | +| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | +| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot, where the controller registers its client; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | +| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | +| `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | +| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | +| `swarm/hives//matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | must be stated | +| `swarm/hives//matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass; also `swarm-matrix-ctl` in the `hive-matrix` container, for its own hive, when the path is empty | `swarm-controller` and `swarm-matrix-ctl` under their own certificates, before they decide whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | +| `swarm/hives//queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | +| `swarm/services//oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated | must be stated | +| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | diff --git a/hive-c0re/src/matrix.rs b/hive-c0re/src/matrix.rs index a79d432b..fc4ee844 100644 --- a/hive-c0re/src/matrix.rs +++ b/hive-c0re/src/matrix.rs @@ -398,29 +398,36 @@ fn encode_room_id_for_url(room_id: &str) -> String { /// appservice registration's `sender_localpart`, and everything the hive /// provisions with it, it provisions as the creator of those rooms. /// -/// Idempotent — skips the account work when the token file already exists -/// and is non-empty. +/// Where the token comes from is [`sender_source`]'s decision. The **swarm +/// secret store** wins whenever it holds one: `swarm-controller` mints it +/// there (and `swarm-matrix-ctl` on the homeserver's own host), keeps it while +/// it is live and replaces it when it is not, so the file follows the store +/// rather than outliving it. That is also what lets a hive that holds no +/// `as_token` have an account at all. The file is kept when the store has +/// nothing or cannot be reached, and the mint ladder below is the fallback +/// when neither holds a token and this hive has an `as_token`. /// -/// The token is taken from the **swarm secret store** when it is there: -/// `swarm-matrix-ctl`, the oneshot inside the matrix container, publishes -/// it under an identity of its own, and taking it from there is what lets a -/// hive that holds no `as_token` have an admin at all. The mint ladder below -/// stays as the fallback for a store that is empty, unconfigured or -/// unreachable — which is every swarm whose matrix container predates that -/// binary. -pub async fn ensure_hive_user(client: &reqwest::Client, as_token: &str) -> Result<()> { +/// # Errors +/// When no token can be had — nothing in the store or the file, and no +/// `as_token` — or when a step of the mint ladder or the file write fails. +pub async fn ensure_hive_user(client: &reqwest::Client, as_token: Option<&str>) -> Result<()> { use std::os::unix::fs::PermissionsExt; let path = sender_token_path(); - if path.exists() - && let Ok(existing) = std::fs::read_to_string(&path) - && !existing.trim().is_empty() - { - tracing::debug!("matrix: the sender token is already present"); - return Ok(()); - } - if let Some(token) = stored_sender_token().await { - return persist_sender_token(&path, &token); - } + let on_disk = std::fs::read_to_string(&path).ok(); + let stored = stored_sender_token().await; + let as_token = match sender_source(stored.as_deref(), on_disk.as_deref(), as_token) { + SenderSource::Keep => { + tracing::debug!("matrix: the sender token is already present"); + return Ok(()); + } + SenderSource::Store(token) => return persist_sender_token(&path, token), + SenderSource::Mint(as_token) => as_token, + SenderSource::Unavailable => anyhow::bail!( + "matrix: no sender token in the swarm store or at {}, and no appservice \ + token on this host to mint one with", + path.display() + ), + }; // Per hive, and fatal when it cannot be derived: the fallback ladder below // must not mint under some other hive's name. let localpart = hive_localpart()?; @@ -473,6 +480,39 @@ pub async fn ensure_hive_user(client: &reqwest::Client, as_token: &str) -> Resul persist_sender_token(&path, &access_token) } +/// What [`ensure_hive_user`] does about the sender token this sweep. +#[derive(Debug, PartialEq, Eq)] +enum SenderSource<'a> { + /// The file already holds the token to use. + Keep, + /// Write the store's token to the file. + Store(&'a str), + /// Mint one with this hive's own appservice token. + Mint(&'a str), + /// Nothing to take and nothing to mint with. + Unavailable, +} + +/// Decide [`ensure_hive_user`]'s step from the store's token, the file's +/// content and this hive's `as_token`, each `None` when absent. Blank counts +/// as absent. +fn sender_source<'a>( + stored: Option<&'a str>, + on_disk: Option<&str>, + as_token: Option<&'a str>, +) -> SenderSource<'a> { + let present = |s: &&str| !s.trim().is_empty(); + let stored = stored.map(str::trim).filter(present); + let on_disk = on_disk.map(str::trim).filter(present); + match (stored, on_disk, as_token.filter(present)) { + (Some(s), Some(d), _) if s == d => SenderSource::Keep, + (Some(s), _, _) => SenderSource::Store(s), + (None, Some(_), _) => SenderSource::Keep, + (None, None, Some(a)) => SenderSource::Mint(a), + (None, None, None) => SenderSource::Unavailable, + } +} + /// Write the appservice sender account's access token to `path`, 0600, creating the /// directory if it is not there. /// @@ -493,8 +533,8 @@ fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<() Ok(()) } -/// Fetch the sender token `swarm-matrix-ctl` published, under -/// this hive's own store identity. +/// Fetch the sender token `swarm-controller` (or `swarm-matrix-ctl`) +/// published, under this hive's own store identity. /// /// The cert role is the hive's name, straight out of `HYPERHIVE_HIVE_NAME` — /// the same role string `workers::credential` logs in with, and already in @@ -506,10 +546,10 @@ fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<() /// /// `None`, never an error, for every way this can come up empty — no hive /// name, no `BAO_*` identity, an unreachable store, nothing at the path. All -/// four mean the same thing to the caller ("mint it the old way"), and three -/// of them are the ordinary state of a swarm that has not deployed `swarm-matrix-ctl` -/// yet, so raising would turn a supported deployment into a warning every -/// sweep. +/// four mean the same thing to the caller ("keep the file, or mint it the old +/// way"), and three of them are the ordinary state of a swarm with no store +/// token for this hive yet, so raising would turn a supported deployment into +/// a warning every sweep. /// /// 🩸 Logs the store **path** and never the value. async fn stored_sender_token() -> Option { @@ -535,15 +575,15 @@ async fn stored_sender_token() -> Option { .await { Ok(credential) if !credential.value.trim().is_empty() => { - tracing::info!(%path, "matrix: taking the sender token from the swarm store"); + tracing::debug!(%path, "matrix: read the sender token from the swarm store"); Some(credential.value) } Ok(_) => { - tracing::warn!(%path, "matrix: the stored sender token is empty; minting instead"); + tracing::warn!(%path, "matrix: the stored sender token is empty"); None } Err(e) => { - tracing::debug!(%path, error = %e, "matrix: no sender token in the store; minting instead"); + tracing::debug!(%path, error = %e, "matrix: no sender token in the store"); None } } @@ -1239,15 +1279,12 @@ pub async fn ensure_all() -> bool { return true; } let mut ok = true; - // Loud and non-destructive: with no appservice token this sweep can - // create nothing, so it does nothing. - let as_token = match read_appservice_token() { - Ok(t) => t, - Err(e) => { - tracing::warn!(error = ?e, "matrix: no appservice token; skipping the user sweep"); - return false; - } - }; + // Absent on every hive whose homeserver runs elsewhere. Only the sender + // token's mint fallback needs it; `ensure_hive_user` fails, and says so, + // when the store has no token either. + let as_token = read_appservice_token() + .inspect_err(|e| tracing::debug!(error = ?e, "matrix: no local appservice token")) + .ok(); // One HTTP client for the whole sweep. let client = match reqwest::Client::builder() .timeout(std::time::Duration::from_secs(HTTP_TIMEOUT_SECS)) @@ -1263,7 +1300,7 @@ pub async fn ensure_all() -> bool { // THROUGH it (the Space, the chat room and every invite are sent with // its token) — as an ordinary user that created those rooms, not as a // homeserver admin. - if let Err(e) = ensure_hive_user(&client, &as_token).await { + if let Err(e) = ensure_hive_user(&client, as_token.as_deref()).await { tracing::warn!(error = ?e, "matrix: ensure_hive_user failed"); ok = false; } @@ -1375,6 +1412,56 @@ async fn provision_space(client: &reqwest::Client, agent_names: &[String]) -> bo mod tests { use super::*; + #[test] + fn a_remote_hive_takes_the_store_token_without_an_appservice_token() { + assert_eq!( + sender_source(Some("tok"), None, None), + SenderSource::Store("tok") + ); + } + + #[test] + fn a_store_token_replaces_a_different_file_token() { + // The swarm re-minted a dead token; the file must follow it. + assert_eq!( + sender_source(Some("new\n"), Some("old\n"), Some("as")), + SenderSource::Store("new") + ); + } + + #[test] + fn a_file_matching_the_store_is_kept() { + // Trailing newline on disk is how `persist_sender_token` writes it. + assert_eq!( + sender_source(Some("tok"), Some("tok\n"), None), + SenderSource::Keep + ); + } + + #[test] + fn with_nothing_in_the_store_the_file_is_kept() { + assert_eq!( + sender_source(None, Some("tok\n"), Some("as")), + SenderSource::Keep + ); + } + + #[test] + fn with_no_token_anywhere_the_appservice_token_mints() { + assert_eq!( + sender_source(None, Some(" \n"), Some("as")), + SenderSource::Mint("as") + ); + } + + #[test] + fn with_no_token_and_no_appservice_token_nothing_is_available() { + assert_eq!( + sender_source(Some(""), None, Some("")), + SenderSource::Unavailable + ); + } + #[test] fn random_hex_is_well_formed_and_correct_length() { let h = random_hex(16).expect("/dev/urandom readable"); diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 06ccde65..7b059cbf 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -593,7 +593,7 @@ async fn handle_matrix_sync_admin() -> Result { let as_token = crate::matrix::read_appservice_token().context("read matrix appservice token")?; let client = matrix_http_client()?; - crate::matrix::ensure_hive_user(&client, &as_token) + crate::matrix::ensure_hive_user(&client, Some(&as_token)) .await .context("matrix sync-admin")?; let path = crate::matrix::sender_token_path(); diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index d3d509ef..d4ba27c9 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -372,6 +372,8 @@ let # instead of rotating it. `metadata/` is the revocation half: `delete` on # `data/` only soft-deletes the newest version, and `+` being one path segment # keeps this to the queue leaf alone. + # Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read` + # for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason. # # The swarm appservice token and its own OIDC client secret, read-only: it # uses both and writes neither. matrix-ctl publishes the token @@ -404,6 +406,10 @@ let capabilities = ["delete"] } + path "${credentialMountPath}/data/swarm/hives/+/matrix/sender-token" { + capabilities = ["create", "read", "update"] + } + path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" { capabilities = ["read"] } diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index aa61b8a0..72459db6 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -1121,10 +1121,9 @@ let # `swarm_secret_client::path::ROOT` and `agents` is # `Kind::Agent.as_str()`, both of which that crate pins in its own test. # - # The grant is still the agent kind alone because nothing writes another - # one yet. It widens when a path outside `agents/` gains a writer, not - # when the kinds are declared. - name = "the controller may write agent credentials, and only under the agent prefix"; + # The only other kind it writes is one leaf per hive, pinned below. A + # grant widens when a path gains a writer, not when a kind is declared. + name = "the controller may write agent credentials, and no whole tree beyond them"; ok = let s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; @@ -1147,6 +1146,21 @@ let in lib.hasInfix "path \"secret/data/swarm/agents/*\" {\n capabilities = [\"create\", \"read\", \"update\"]" s; } + { + # `matrix_account::hive_sender` writes every hive's sender token, and + # nothing else under `hives/`: a hive's appservice token sits beside + # it. `+` is one segment, which keeps this to the one leaf; a `*` is a + # glob only at the end of a path. Pinned as the whole stanza, so an + # added capability fails. + name = "the controller writes each hive's sender token and nothing else under hives"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/+/matrix/sender-token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + && !(lib.hasInfix "secret/metadata/swarm/hives" s); + } { # Revocation, and the reason it is a stanza of its own: `delete` on the # `data/` path soft-deletes the newest version and leaves earlier ones diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index 79e4eff1..42f1e4bd 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -123,6 +123,10 @@ enum SwarmNodeKind { /// Carries no hive: the path it writes has no hive segment, so an agent /// that moves between hives keeps one matrix identity. MintAgentMatrixAccount { agent: String }, + /// Make sure `hive`'s sender account (`@hive-:`) holds a live token + /// in the swarm store, where that hive's matrix sweep reads it. See + /// `matrix_account::hive_sender`. + MintHiveSenderToken { hive: String }, /// Re-mint `agent`'s queue secret if it is still stored and old enough. /// See `agent_renewal`. /// @@ -154,6 +158,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind { SwarmNodeKind::MintAgentIdentity { .. } => "mint_agent_identity".to_owned(), SwarmNodeKind::MintAgentForgeToken { .. } => "mint_agent_forge_token".to_owned(), SwarmNodeKind::MintAgentMatrixAccount { .. } => "mint_agent_matrix_account".to_owned(), + SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(), SwarmNodeKind::RenewAgentQueueCredential { .. } => { "renew_agent_queue_credential".to_owned() } @@ -181,6 +186,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind { | SwarmNodeKind::RenewAgentQueueCredential { agent } => { serde_json::json!({ "agent": agent }) } + SwarmNodeKind::MintHiveSenderToken { hive } => serde_json::json!({ "hive": hive }), SwarmNodeKind::TriggerDeploy { hive, agent } | SwarmNodeKind::SetAgentWanted { hive, agent } => { serde_json::json!({ "agent": agent, "hive": hive }) @@ -329,6 +335,9 @@ async fn run_swarm_node( SwarmNodeKind::MintAgentMatrixAccount { agent } => { mint_matrix_account(deps.matrix_homeserver.as_deref(), &agent).await } + SwarmNodeKind::MintHiveSenderToken { hive } => { + mint_hive_sender(deps.matrix_homeserver.as_deref(), &hive).await + } SwarmNodeKind::RenewAgentQueueCredential { agent } => { match agent_renewal::renew(&agent).await { Ok(()) => Outcome::Done, @@ -408,6 +417,23 @@ async fn mint_matrix_account( } } +/// The `MintHiveSenderToken` arm, lifted out for the same reason. +async fn mint_hive_sender(homeserver: Option<&str>, hive: &str) -> hive_jobq::scheduler::Outcome { + use hive_jobq::scheduler::Outcome; + + let Some(base) = homeserver else { + return Outcome::Failed(format!( + "no matrix homeserver configured on this host ({} unset), so no hive \ + sender token can be minted", + matrix_account::DEFAULT_HOMESERVER_ENV + )); + }; + match matrix_account::hive_sender::ensure_hive_sender_token(base, hive).await { + Ok(()) => Outcome::Done, + Err(e) => Outcome::Failed(format!("{e:#}")), + } +} + /// Declare a brand-new agent at [`NEW_AGENT_WANTED_STATE`] — unless it turns /// out not to be new: an agent that already has a declaration (other than /// `Destroyed`, which this treats as reusable) is left alone, so a retried @@ -2132,6 +2158,27 @@ fn queue_matrix_account_mints( Ok(ids) } +/// Insert one `MintHiveSenderToken` job per hive and return the nodes' ids. +/// `matrix_account::hive_sender::spawn`'s periodic pass is the one caller. +fn queue_hive_sender_mints( + sched: &Mutex>, + hives: Vec, +) -> Result> { + let mut sched = sched + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let mut ids = Vec::with_capacity(hives.len()); + for hive in hives { + let queued = sched + .insert_job(None, |b| { + vec![b.node(SwarmNodeKind::MintHiveSenderToken { hive }).guid()] + }) + .map_err(|e| anyhow::anyhow!("{e}"))?; + ids.extend(queued); + } + Ok(ids) +} + /// The homeserver agents' own matrix accounts are minted on. Unset is this /// controller creating agents with no matrix account, which the mint node /// reports by name rather than `main` refusing to start. @@ -2141,21 +2188,30 @@ fn configured_matrix_homeserver() -> Option> { .map(Arc::from) } -/// Start the matrix-account backfill when a homeserver is configured. Lifted -/// out of `main` for `clippy::too_many_lines`. +/// Start the matrix-account backfill and the hive sender-token pass when a +/// homeserver is configured. Lifted out of `main` for +/// `clippy::too_many_lines`. fn spawn_matrix_account_backfill( jobq: &Arc>>, homeserver: Option>, + hives: &[HiveEntry], ) { let Some(base) = homeserver else { return; }; let sched = Arc::clone(jobq); - matrix_account::agent_token::spawn(base, move |agents| { + matrix_account::agent_token::spawn(Arc::clone(&base), move |agents| { if let Err(e) = queue_matrix_account_mints(&sched, agents) { tracing::warn!(error = %format!("{e:#}"), "agent matrix accounts: queueing failed"); } }); + let sched = Arc::clone(jobq); + let hive_names = hives.iter().map(|h| h.name.clone()).collect(); + matrix_account::hive_sender::spawn(base, hive_names, move |hives| { + if let Err(e) = queue_hive_sender_mints(&sched, hives) { + tracing::warn!(error = %format!("{e:#}"), "hive sender tokens: queueing failed"); + } + }); } /// Start the forge's periodic passes — the swarm-wide objects, and agents' @@ -2714,7 +2770,8 @@ async fn main() -> Result<()> { hive_jobq::Graph::new(), hive_jobq::resources::ResourceTable::new(), ))); - spawn_matrix_account_backfill(&jobq, deps.matrix_homeserver.clone()); + let hives = load_hives(); + spawn_matrix_account_backfill(&jobq, deps.matrix_homeserver.clone(), &hives); spawn_jobq_worker(Arc::clone(&jobq), deps); // Bound to a named variable, not `_` — dropping the provider stops its // `PeriodicReader`, so it must live as long as `main` does (which it @@ -2771,7 +2828,6 @@ async fn main() -> Result<()> { let config_prs = forge_client.clone().map(config_pr::spawn); let state_forge = keep_forge_for_state(forge_client, webhook_secret.clone()); - let hives = load_hives(); spawn_agent_renewal(&jobq, wanted_writer(status.as_ref()), &hives); // Before serving, because a hive whose role does not exist cannot log in, // and one whose policy does not exist logs in able to read nothing — @@ -4370,6 +4426,37 @@ mod tests { assert_eq!(agents, ["a", "b"]); } + /// The hive sender-token pass's queueing: one mint node per hive, and + /// the node names the hive rather than an agent. + #[test] + fn a_queued_hive_sender_mint_is_one_node_per_hive() { + use hive_jobq_wire::WireNode as _; + + let sched = std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new( + hive_jobq::Graph::new(), + hive_jobq::resources::ResourceTable::new(), + )); + let ids = + super::queue_hive_sender_mints(&sched, vec!["alpha".to_owned(), "beta".to_owned()]) + .expect("two jobs insert"); + assert_eq!(ids.len(), 2); + let guard = sched + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let mut hives: Vec = guard + .graph() + .nodes() + .map(|n| { + assert_eq!(n.payload.label(), "mint_hive_sender_token"); + let data = n.payload.data(n.id.get()); + assert!(data.get("agent").is_none(), "{data}"); + data["hive"].as_str().expect("hive is a string").to_owned() + }) + .collect(); + hives.sort(); + assert_eq!(hives, ["alpha", "beta"]); + } + /// The manual route and the periodic pass both go through /// `queue_forge_token_mints`, so this is the assertion that a backfilled /// mint is the same pair of nodes agent creation inserts: the forge user, diff --git a/swarm-controller/src/matrix_account.rs b/swarm-controller/src/matrix_account.rs index ad71fa48..3994dd42 100644 --- a/swarm-controller/src/matrix_account.rs +++ b/swarm-controller/src/matrix_account.rs @@ -4,6 +4,7 @@ //! the swarm's homeserver, minted with the swarm's appservice token and stored //! where the agent's daemon reads it. See docs/swarm/credentials.md for why //! `main` is the one name [`put_matrix_account`] refuses to write. +//! [`hive_sender`] mints each **hive's** sender account the same way, sharing this module's helpers with [`agent_token`]. //! //! **The external one** — [`put_matrix_account`] and everything under it — is //! an account somewhere else that an operator hands us a credential for: put it @@ -39,6 +40,7 @@ use utoipa::ToSchema; use super::{AppState, error_problem, swarm_hive}; pub mod agent_token; +pub mod hive_sender; fn default_mode() -> String { "token".to_owned() diff --git a/swarm-controller/src/matrix_account/agent_token.rs b/swarm-controller/src/matrix_account/agent_token.rs index 4240b6bc..9dc87a09 100644 --- a/swarm-controller/src/matrix_account/agent_token.rs +++ b/swarm-controller/src/matrix_account/agent_token.rs @@ -107,7 +107,7 @@ pub fn plan(observed: &[(String, Observed)]) -> Vec { } /// The swarm appservice token, or an error naming why there is none. -async fn appservice_token(store: &SecretStore) -> Result { +pub(super) async fn appservice_token(store: &SecretStore) -> Result { let path = matrix::swarm_appservice_token_path()?; let stored: Option = store .read_optional(&path) @@ -129,9 +129,18 @@ async fn probe( base: &str, agent: &str, ) -> Result { - let path = matrix::account_path(agent, ACCOUNT)?; + probe_at(store, http, base, &matrix::account_path(agent, ACCOUNT)?).await +} + +/// What the store and the homeserver say about the token stored at `path`. +pub(super) async fn probe_at( + store: &SecretStore, + http: &reqwest::Client, + base: &str, + path: &str, +) -> Result { let stored: Option = store - .read_optional(&path) + .read_optional(path) .await .with_context(|| format!("reading {path}"))?; let Some(stored) = stored else { @@ -170,35 +179,52 @@ pub async fn ensure_agent_matrix_account(base: &str, agent: &str) -> Result<()> Decision::Mint(reason) => reason, }; - let token = match homeserver::register(&http, base, agent, &as_token).await? { + let path = matrix::account_path(agent, ACCOUNT)?; + mint_at(&store, &http, base, agent, &as_token, &path).await?; + tracing::info!(agent, ?reason, %path, "agent matrix account token minted and stored"); + Ok(()) +} + +/// Create the account `account` (a localpart), or log in to it as the appservice when it +/// exists, store the token at `path`, and read it back with `whoami`. +/// +/// # Errors +/// When the store or the homeserver refuses a step, or the new token +/// authenticates as someone else. +pub(super) async fn mint_at( + store: &SecretStore, + http: &reqwest::Client, + base: &str, + account: &str, + as_token: &str, + path: &str, +) -> Result<()> { + let token = match homeserver::register(http, base, account, as_token).await? { homeserver::Registered::Token(token) => token, - // An agent minted before, or one a hive created back when hives did - // this: log in as the appservice on the same device instead. + // An account minted before, or one a hive created itself: log in as + // the appservice on the same device instead. homeserver::Registered::AlreadyExists => { - homeserver::appservice_login(&http, base, agent, &as_token).await? + homeserver::appservice_login(http, base, account, as_token).await? } }; - let path = matrix::account_path(agent, ACCOUNT)?; store .write( - &path, + path, &matrix::Credential { value: token.clone(), homeserver: Some(base.to_owned()), }, ) .await - .with_context(|| format!("storing {agent}'s matrix token at {path}"))?; + .with_context(|| format!("storing @{account}'s matrix token at {path}"))?; - match homeserver::whoami(&http, base, &token) + match homeserver::whoami(http, base, &token) .await - .with_context(|| format!("using {agent}'s new matrix token"))? + .with_context(|| format!("using @{account}'s new matrix token"))? { - Whoami::User(user) if localpart(&user) == Some(agent) => {} - _ => bail!("{agent}'s new matrix token does not authenticate as {agent}"), + Whoami::User(user) if localpart(&user) == Some(account) => Ok(()), + _ => bail!("@{account}'s new matrix token does not authenticate as @{account}"), } - tracing::info!(agent, ?reason, %path, "agent matrix account token minted and stored"); - Ok(()) } /// One pass: every agent holding a store identity, observed. diff --git a/swarm-controller/src/matrix_account/hive_sender.rs b/swarm-controller/src/matrix_account/hive_sender.rs new file mode 100644 index 00000000..0c20a57d --- /dev/null +++ b/swarm-controller/src/matrix_account/hive_sender.rs @@ -0,0 +1,201 @@ +//! Each hive's sender account, `@hive-:`, on the swarm's homeserver: +//! created here with the **swarm's** appservice token and stored at +//! `swarm/hives//matrix/sender-token`, where hive-c0re's matrix sweep +//! reads it under the hive's own store identity. A hive whose homeserver runs +//! elsewhere holds no appservice token, so this is its only sender token. +//! +//! Runs for every hive in the directory, local or remote, and decides the +//! same way [`super::agent_token`] does: a stored token that `whoami` +//! confirms as `@hive-:` is kept, so this writes only when the path is +//! empty or its token is dead. +//! +//! ⚠️ `swarm-matrix-ctl mint` also writes this path for the hive whose host +//! runs the homeserver, and skips when it is non-empty. Both log in on the +//! same pinned device, so if both find it empty at once, one of the two +//! tokens is dead on arrival. Whichever of them lands in the store, the next +//! [`RECONCILE_INTERVAL`] pass either keeps it (live) or re-mints it +//! (`Revoked`), and matrix-ctl never writes a non-empty path — so the store +//! converges on one live token, and the hive's sweep takes whatever it holds. + +use std::sync::Arc; + +use anyhow::{Context, Result}; +use swarm_matrix_client as homeserver; +use swarm_secret_client::matrix; + +use super::agent_token::{Decision, Observed, appservice_token, classify, mint_at, plan, probe_at}; + +/// How often [`spawn`] re-checks every hive's sender token. +const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5); + +/// Decide what to do about `hive`'s sender token from what the stored one is. +fn classify_hive(hive: &str, probe: &super::agent_token::Probe) -> Decision { + classify(&matrix::hive_localpart(hive), probe) +} + +/// Make sure `hive`'s sender account holds a live token in the store, +/// creating the account or logging in to it when it does not. The whole job +/// of the `MintHiveSenderToken` node. +/// +/// # Errors +/// When the store or the homeserver refuses a step, the swarm appservice +/// token has not been published, or the new token authenticates as someone +/// else. +pub async fn ensure_hive_sender_token(base: &str, hive: &str) -> Result<()> { + let store = crate::store::connect() + .await + .context("logging in to the swarm secret store")?; + let as_token = appservice_token(&store).await?; + let http = homeserver::client()?; + let path = matrix::sender_token_path(hive)?; + let reason = match classify_hive(hive, &probe_at(&store, &http, base, &path).await?) { + Decision::Keep => { + tracing::debug!(hive, "hive sender token is current; left as it is"); + return Ok(()); + } + Decision::Mint(reason) => reason, + }; + let localpart = matrix::hive_localpart(hive); + mint_at(&store, &http, base, &localpart, &as_token, &path).await?; + tracing::info!(hive, ?reason, %path, "hive sender token minted and stored"); + Ok(()) +} + +/// One pass: every hive in `hives`, observed. +/// +/// Fails as a whole when the swarm appservice token is not published, for +/// the reason `agent_token`'s pass does. +async fn observe_all(base: &str, hives: &[String]) -> Result> { + let store = crate::store::connect() + .await + .context("logging in to the swarm secret store")?; + appservice_token(&store).await?; + let http = homeserver::client()?; + let mut observed = Vec::with_capacity(hives.len()); + for hive in hives { + let o = match matrix::sender_token_path(hive) { + Ok(path) => match probe_at(&store, &http, base, &path).await { + Ok(p) => Observed::Decided(classify_hive(hive, &p)), + Err(e) => { + tracing::warn!(hive, error = %format!("{e:#}"), "hive sender token: read failed"); + Observed::Unknown + } + }, + Err(e) => { + tracing::warn!(hive, error = %e, "hive sender token: the hive name forms no store path"); + Observed::Unknown + } + }; + observed.push((hive.clone(), o)); + } + Ok(observed) +} + +/// Check every hive's sender token now and every [`RECONCILE_INTERVAL`] +/// after, and hand the hives that need one to `enqueue`, which inserts a +/// `MintHiveSenderToken` node for each. +/// +/// The roster is the controller's hive directory. A pass that fails is +/// logged and retried on the next tick; it never stops the daemon. +pub fn spawn(base: Arc, hives: Vec, enqueue: impl Fn(Vec) + Send + 'static) { + tokio::spawn(async move { + let mut ticker = tokio::time::interval(RECONCILE_INTERVAL); + loop { + ticker.tick().await; + match observe_all(&base, &hives).await { + Ok(observed) => { + let minting = plan(&observed); + if minting.is_empty() { + tracing::debug!( + checked = observed.len(), + "hive sender tokens: all current" + ); + } else { + tracing::info!( + checked = observed.len(), + minting = minting.len(), + "hive sender tokens: queueing mints" + ); + enqueue(minting); + } + } + Err(e) => tracing::warn!( + error = %format!("{e:#}"), + retry_in_s = RECONCILE_INTERVAL.as_secs(), + "hive sender tokens: pass failed; retrying next tick" + ), + } + } + }); +} + +#[cfg(test)] +mod tests { + use super::super::agent_token::{MintReason, Probe}; + use super::*; + use swarm_matrix_client::Whoami; + + fn user(id: &str) -> Probe { + Probe::Whoami(Whoami::User(id.to_owned())) + } + + #[test] + fn a_live_token_for_the_hive_account_is_kept() { + assert_eq!( + classify_hive("pr1ma", &user("@hive-pr1ma:t.local")), + Decision::Keep + ); + } + + #[test] + fn nothing_stored_mints() { + // A hive whose homeserver is remote: nothing has ever written its path. + assert_eq!( + classify_hive("pr1ma", &Probe::NotStored), + Decision::Mint(MintReason::NotStored) + ); + } + + #[test] + fn a_dead_token_mints() { + // What the loser of a simultaneous mint with matrix-ctl leaves behind. + assert_eq!( + classify_hive("pr1ma", &Probe::Whoami(Whoami::UnknownToken)), + Decision::Mint(MintReason::Revoked) + ); + } + + #[test] + fn a_token_for_another_account_mints() { + // The bare hive name is an agent's localpart, not the hive's account, + // and another hive's account is not this one's. + for other in ["@pr1ma:t.local", "@hive-beta:t.local", "@hive:t.local"] { + assert_eq!( + classify_hive("pr1ma", &user(other)), + Decision::Mint(MintReason::OtherUser), + "{other}" + ); + } + } + + #[test] + fn an_outage_plans_nothing() { + // A mint replaces the token the hive is running on, so a pass that + // could not read must not mint for every hive. + let observed = [ + ("alpha".to_owned(), Observed::Unknown), + ("beta".to_owned(), Observed::Unknown), + ]; + assert!(plan(&observed).is_empty()); + } + + #[test] + fn the_published_path_is_the_one_the_hive_reads() { + // hive-c0re's `stored_sender_token` and `swarm-matrix-ctl mint` both + // resolve this path; the literal is what the bao grant names. + assert_eq!( + matrix::sender_token_path("pr1ma").expect("a plain name is legal"), + "swarm/hives/pr1ma/matrix/sender-token" + ); + } +}