swarm-controller: mint each hive's matrix sender token
A hive whose homeserver runs on another host has no local matrix-appservice-token, so hive-c0re's matrix sweep returned before reaching the store read in ensure_hive_user: no @hive-<name>: token, no Space, no chat room, no invites, and a sweep-health banner. swarm-controller now mints @hive-<name>: with the swarm appservice token for every hive in its directory, as a MintHiveSenderToken job node queued by a five-minute pass, and stores it at swarm/hives/<name>/matrix/sender-token, the same matrix::Credential swarm-matrix-ctl writes there. It is keep-if-live, reusing agent_token's classify/plan: a stored token whoami confirms as @hive-<name>: is left alone, so only an absent or dead one is minted. agent_token's probe and mint steps are lifted into probe_at/mint_at so both passes share them. swarm-matrix-ctl mint still writes the path for its own hive when it is empty. If both mint an empty path at once, one token is invalidated (same pinned device); the next pass classifies it Revoked and re-mints. hive-c0re's ensure_all no longer returns when there is no local as_token. ensure_hive_user reads the store first on every sweep and overwrites its token file when the store's token differs, keeps the file when the store has none, mints with the local as_token only when neither holds one, and fails with one error when there is nothing at all. The decision is sender_source, unit-tested. The controller's bao policy gains create/read/update on swarm/hives/+/matrix/sender-token (`+`, since `*` is a glob only at the end of a path), pinned in module-eval. Refs #4427
This commit is contained in:
atlas
mara
parent
9a5a947f8d
commit
78d8d69c7f
10 changed files with 504 additions and 80 deletions
|
|
@ -183,7 +183,8 @@ hive's standing leaves the others alone.
|
|||
Its access token is the **sender token**, and it's the credential
|
||||
hive-c0re presents for every homeserver call it makes on the hive's
|
||||
behalf. It's per hive for the same reason the account is:
|
||||
`swarm-matrix-ctl` mints it inside the `hive-matrix` container and
|
||||
`swarm-controller` mints it for every hive with the swarm's appservice token (and
|
||||
`swarm-matrix-ctl`, inside the `hive-matrix` container, for its own hive) and
|
||||
publishes it to `swarm/hives/<hive>/matrix/sender-token`, and the hive
|
||||
reads it from there under its own certificate. That path sits inside the
|
||||
hive's own read grant (`swarm/hives/<hive>/*`), so a hive fetches its own
|
||||
|
|
|
|||
|
|
@ -60,7 +60,7 @@ of the cell says how.
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
|
|
@ -69,7 +69,7 @@ of the cell says how.
|
|||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
| `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-matrix-ctl`, in the `hive-matrix` container | `swarm-matrix-ctl` itself, under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass; also `swarm-matrix-ctl` in the `hive-matrix` container, for its own hive, when the path is empty | `swarm-controller` and `swarm-matrix-ctl` under their own certificates, before they decide whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated | must be stated |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||
|
|
|
|||
|
|
@ -398,29 +398,36 @@ fn encode_room_id_for_url(room_id: &str) -> String {
|
|||
/// appservice registration's `sender_localpart`, and everything the hive
|
||||
/// provisions with it, it provisions as the creator of those rooms.
|
||||
///
|
||||
/// Idempotent — skips the account work when the token file already exists
|
||||
/// and is non-empty.
|
||||
/// Where the token comes from is [`sender_source`]'s decision. The **swarm
|
||||
/// secret store** wins whenever it holds one: `swarm-controller` mints it
|
||||
/// there (and `swarm-matrix-ctl` on the homeserver's own host), keeps it while
|
||||
/// it is live and replaces it when it is not, so the file follows the store
|
||||
/// rather than outliving it. That is also what lets a hive that holds no
|
||||
/// `as_token` have an account at all. The file is kept when the store has
|
||||
/// nothing or cannot be reached, and the mint ladder below is the fallback
|
||||
/// when neither holds a token and this hive has an `as_token`.
|
||||
///
|
||||
/// The token is taken from the **swarm secret store** when it is there:
|
||||
/// `swarm-matrix-ctl`, the oneshot inside the matrix container, publishes
|
||||
/// it under an identity of its own, and taking it from there is what lets a
|
||||
/// hive that holds no `as_token` have an admin at all. The mint ladder below
|
||||
/// stays as the fallback for a store that is empty, unconfigured or
|
||||
/// unreachable — which is every swarm whose matrix container predates that
|
||||
/// binary.
|
||||
pub async fn ensure_hive_user(client: &reqwest::Client, as_token: &str) -> Result<()> {
|
||||
/// # Errors
|
||||
/// When no token can be had — nothing in the store or the file, and no
|
||||
/// `as_token` — or when a step of the mint ladder or the file write fails.
|
||||
pub async fn ensure_hive_user(client: &reqwest::Client, as_token: Option<&str>) -> Result<()> {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let path = sender_token_path();
|
||||
if path.exists()
|
||||
&& let Ok(existing) = std::fs::read_to_string(&path)
|
||||
&& !existing.trim().is_empty()
|
||||
{
|
||||
let on_disk = std::fs::read_to_string(&path).ok();
|
||||
let stored = stored_sender_token().await;
|
||||
let as_token = match sender_source(stored.as_deref(), on_disk.as_deref(), as_token) {
|
||||
SenderSource::Keep => {
|
||||
tracing::debug!("matrix: the sender token is already present");
|
||||
return Ok(());
|
||||
}
|
||||
if let Some(token) = stored_sender_token().await {
|
||||
return persist_sender_token(&path, &token);
|
||||
}
|
||||
SenderSource::Store(token) => return persist_sender_token(&path, token),
|
||||
SenderSource::Mint(as_token) => as_token,
|
||||
SenderSource::Unavailable => anyhow::bail!(
|
||||
"matrix: no sender token in the swarm store or at {}, and no appservice \
|
||||
token on this host to mint one with",
|
||||
path.display()
|
||||
),
|
||||
};
|
||||
// Per hive, and fatal when it cannot be derived: the fallback ladder below
|
||||
// must not mint under some other hive's name.
|
||||
let localpart = hive_localpart()?;
|
||||
|
|
@ -473,6 +480,39 @@ pub async fn ensure_hive_user(client: &reqwest::Client, as_token: &str) -> Resul
|
|||
persist_sender_token(&path, &access_token)
|
||||
}
|
||||
|
||||
/// What [`ensure_hive_user`] does about the sender token this sweep.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
enum SenderSource<'a> {
|
||||
/// The file already holds the token to use.
|
||||
Keep,
|
||||
/// Write the store's token to the file.
|
||||
Store(&'a str),
|
||||
/// Mint one with this hive's own appservice token.
|
||||
Mint(&'a str),
|
||||
/// Nothing to take and nothing to mint with.
|
||||
Unavailable,
|
||||
}
|
||||
|
||||
/// Decide [`ensure_hive_user`]'s step from the store's token, the file's
|
||||
/// content and this hive's `as_token`, each `None` when absent. Blank counts
|
||||
/// as absent.
|
||||
fn sender_source<'a>(
|
||||
stored: Option<&'a str>,
|
||||
on_disk: Option<&str>,
|
||||
as_token: Option<&'a str>,
|
||||
) -> SenderSource<'a> {
|
||||
let present = |s: &&str| !s.trim().is_empty();
|
||||
let stored = stored.map(str::trim).filter(present);
|
||||
let on_disk = on_disk.map(str::trim).filter(present);
|
||||
match (stored, on_disk, as_token.filter(present)) {
|
||||
(Some(s), Some(d), _) if s == d => SenderSource::Keep,
|
||||
(Some(s), _, _) => SenderSource::Store(s),
|
||||
(None, Some(_), _) => SenderSource::Keep,
|
||||
(None, None, Some(a)) => SenderSource::Mint(a),
|
||||
(None, None, None) => SenderSource::Unavailable,
|
||||
}
|
||||
}
|
||||
|
||||
/// Write the appservice sender account's access token to `path`, 0600, creating the
|
||||
/// directory if it is not there.
|
||||
///
|
||||
|
|
@ -493,8 +533,8 @@ fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<()
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetch the sender token `swarm-matrix-ctl` published, under
|
||||
/// this hive's own store identity.
|
||||
/// Fetch the sender token `swarm-controller` (or `swarm-matrix-ctl`)
|
||||
/// published, under this hive's own store identity.
|
||||
///
|
||||
/// The cert role is the hive's name, straight out of `HYPERHIVE_HIVE_NAME` —
|
||||
/// the same role string `workers::credential` logs in with, and already in
|
||||
|
|
@ -506,10 +546,10 @@ fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<()
|
|||
///
|
||||
/// `None`, never an error, for every way this can come up empty — no hive
|
||||
/// name, no `BAO_*` identity, an unreachable store, nothing at the path. All
|
||||
/// four mean the same thing to the caller ("mint it the old way"), and three
|
||||
/// of them are the ordinary state of a swarm that has not deployed `swarm-matrix-ctl`
|
||||
/// yet, so raising would turn a supported deployment into a warning every
|
||||
/// sweep.
|
||||
/// four mean the same thing to the caller ("keep the file, or mint it the old
|
||||
/// way"), and three of them are the ordinary state of a swarm with no store
|
||||
/// token for this hive yet, so raising would turn a supported deployment into
|
||||
/// a warning every sweep.
|
||||
///
|
||||
/// 🩸 Logs the store **path** and never the value.
|
||||
async fn stored_sender_token() -> Option<String> {
|
||||
|
|
@ -535,15 +575,15 @@ async fn stored_sender_token() -> Option<String> {
|
|||
.await
|
||||
{
|
||||
Ok(credential) if !credential.value.trim().is_empty() => {
|
||||
tracing::info!(%path, "matrix: taking the sender token from the swarm store");
|
||||
tracing::debug!(%path, "matrix: read the sender token from the swarm store");
|
||||
Some(credential.value)
|
||||
}
|
||||
Ok(_) => {
|
||||
tracing::warn!(%path, "matrix: the stored sender token is empty; minting instead");
|
||||
tracing::warn!(%path, "matrix: the stored sender token is empty");
|
||||
None
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!(%path, error = %e, "matrix: no sender token in the store; minting instead");
|
||||
tracing::debug!(%path, error = %e, "matrix: no sender token in the store");
|
||||
None
|
||||
}
|
||||
}
|
||||
|
|
@ -1239,15 +1279,12 @@ pub async fn ensure_all() -> bool {
|
|||
return true;
|
||||
}
|
||||
let mut ok = true;
|
||||
// Loud and non-destructive: with no appservice token this sweep can
|
||||
// create nothing, so it does nothing.
|
||||
let as_token = match read_appservice_token() {
|
||||
Ok(t) => t,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "matrix: no appservice token; skipping the user sweep");
|
||||
return false;
|
||||
}
|
||||
};
|
||||
// Absent on every hive whose homeserver runs elsewhere. Only the sender
|
||||
// token's mint fallback needs it; `ensure_hive_user` fails, and says so,
|
||||
// when the store has no token either.
|
||||
let as_token = read_appservice_token()
|
||||
.inspect_err(|e| tracing::debug!(error = ?e, "matrix: no local appservice token"))
|
||||
.ok();
|
||||
// One HTTP client for the whole sweep.
|
||||
let client = match reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(HTTP_TIMEOUT_SECS))
|
||||
|
|
@ -1263,7 +1300,7 @@ pub async fn ensure_all() -> bool {
|
|||
// THROUGH it (the Space, the chat room and every invite are sent with
|
||||
// its token) — as an ordinary user that created those rooms, not as a
|
||||
// homeserver admin.
|
||||
if let Err(e) = ensure_hive_user(&client, &as_token).await {
|
||||
if let Err(e) = ensure_hive_user(&client, as_token.as_deref()).await {
|
||||
tracing::warn!(error = ?e, "matrix: ensure_hive_user failed");
|
||||
ok = false;
|
||||
}
|
||||
|
|
@ -1375,6 +1412,56 @@ async fn provision_space(client: &reqwest::Client, agent_names: &[String]) -> bo
|
|||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_remote_hive_takes_the_store_token_without_an_appservice_token() {
|
||||
assert_eq!(
|
||||
sender_source(Some("tok"), None, None),
|
||||
SenderSource::Store("tok")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_store_token_replaces_a_different_file_token() {
|
||||
// The swarm re-minted a dead token; the file must follow it.
|
||||
assert_eq!(
|
||||
sender_source(Some("new\n"), Some("old\n"), Some("as")),
|
||||
SenderSource::Store("new")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_file_matching_the_store_is_kept() {
|
||||
// Trailing newline on disk is how `persist_sender_token` writes it.
|
||||
assert_eq!(
|
||||
sender_source(Some("tok"), Some("tok\n"), None),
|
||||
SenderSource::Keep
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_nothing_in_the_store_the_file_is_kept() {
|
||||
assert_eq!(
|
||||
sender_source(None, Some("tok\n"), Some("as")),
|
||||
SenderSource::Keep
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_no_token_anywhere_the_appservice_token_mints() {
|
||||
assert_eq!(
|
||||
sender_source(None, Some(" \n"), Some("as")),
|
||||
SenderSource::Mint("as")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_no_token_and_no_appservice_token_nothing_is_available() {
|
||||
assert_eq!(
|
||||
sender_source(Some(""), None, Some("")),
|
||||
SenderSource::Unavailable
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn random_hex_is_well_formed_and_correct_length() {
|
||||
let h = random_hex(16).expect("/dev/urandom readable");
|
||||
|
|
|
|||
|
|
@ -593,7 +593,7 @@ async fn handle_matrix_sync_admin() -> Result<HostResponse> {
|
|||
let as_token =
|
||||
crate::matrix::read_appservice_token().context("read matrix appservice token")?;
|
||||
let client = matrix_http_client()?;
|
||||
crate::matrix::ensure_hive_user(&client, &as_token)
|
||||
crate::matrix::ensure_hive_user(&client, Some(&as_token))
|
||||
.await
|
||||
.context("matrix sync-admin")?;
|
||||
let path = crate::matrix::sender_token_path();
|
||||
|
|
|
|||
|
|
@ -372,6 +372,8 @@ let
|
|||
# instead of rotating it. `metadata/` is the revocation half: `delete` on
|
||||
# `data/` only soft-deletes the newest version, and `+` being one path segment
|
||||
# keeps this to the queue leaf alone.
|
||||
# Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read`
|
||||
# for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason.
|
||||
#
|
||||
# The swarm appservice token and its own OIDC client secret, read-only: it
|
||||
# uses both and writes neither. matrix-ctl publishes the token
|
||||
|
|
@ -404,6 +406,10 @@ let
|
|||
capabilities = ["delete"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/swarm/hives/+/matrix/sender-token" {
|
||||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1121,10 +1121,9 @@ let
|
|||
# `swarm_secret_client::path::ROOT` and `agents` is
|
||||
# `Kind::Agent.as_str()`, both of which that crate pins in its own test.
|
||||
#
|
||||
# The grant is still the agent kind alone because nothing writes another
|
||||
# one yet. It widens when a path outside `agents/` gains a writer, not
|
||||
# when the kinds are declared.
|
||||
name = "the controller may write agent credentials, and only under the agent prefix";
|
||||
# The only other kind it writes is one leaf per hive, pinned below. A
|
||||
# grant widens when a path gains a writer, not when a kind is declared.
|
||||
name = "the controller may write agent credentials, and no whole tree beyond them";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||
|
|
@ -1147,6 +1146,21 @@ let
|
|||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/agents/*\" {\n capabilities = [\"create\", \"read\", \"update\"]" s;
|
||||
}
|
||||
{
|
||||
# `matrix_account::hive_sender` writes every hive's sender token, and
|
||||
# nothing else under `hives/`: a hive's appservice token sits beside
|
||||
# it. `+` is one segment, which keeps this to the one leaf; a `*` is a
|
||||
# glob only at the end of a path. Pinned as the whole stanza, so an
|
||||
# added capability fails.
|
||||
name = "the controller writes each hive's sender token and nothing else under hives";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/hives/+/matrix/sender-token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s
|
||||
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
|
||||
&& !(lib.hasInfix "secret/metadata/swarm/hives" s);
|
||||
}
|
||||
{
|
||||
# Revocation, and the reason it is a stanza of its own: `delete` on the
|
||||
# `data/` path soft-deletes the newest version and leaves earlier ones
|
||||
|
|
|
|||
|
|
@ -123,6 +123,10 @@ enum SwarmNodeKind {
|
|||
/// Carries no hive: the path it writes has no hive segment, so an agent
|
||||
/// that moves between hives keeps one matrix identity.
|
||||
MintAgentMatrixAccount { agent: String },
|
||||
/// Make sure `hive`'s sender account (`@hive-<hive>:`) holds a live token
|
||||
/// in the swarm store, where that hive's matrix sweep reads it. See
|
||||
/// `matrix_account::hive_sender`.
|
||||
MintHiveSenderToken { hive: String },
|
||||
/// Re-mint `agent`'s queue secret if it is still stored and old enough.
|
||||
/// See `agent_renewal`.
|
||||
///
|
||||
|
|
@ -154,6 +158,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
|
|||
SwarmNodeKind::MintAgentIdentity { .. } => "mint_agent_identity".to_owned(),
|
||||
SwarmNodeKind::MintAgentForgeToken { .. } => "mint_agent_forge_token".to_owned(),
|
||||
SwarmNodeKind::MintAgentMatrixAccount { .. } => "mint_agent_matrix_account".to_owned(),
|
||||
SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(),
|
||||
SwarmNodeKind::RenewAgentQueueCredential { .. } => {
|
||||
"renew_agent_queue_credential".to_owned()
|
||||
}
|
||||
|
|
@ -181,6 +186,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
|
|||
| SwarmNodeKind::RenewAgentQueueCredential { agent } => {
|
||||
serde_json::json!({ "agent": agent })
|
||||
}
|
||||
SwarmNodeKind::MintHiveSenderToken { hive } => serde_json::json!({ "hive": hive }),
|
||||
SwarmNodeKind::TriggerDeploy { hive, agent }
|
||||
| SwarmNodeKind::SetAgentWanted { hive, agent } => {
|
||||
serde_json::json!({ "agent": agent, "hive": hive })
|
||||
|
|
@ -329,6 +335,9 @@ async fn run_swarm_node(
|
|||
SwarmNodeKind::MintAgentMatrixAccount { agent } => {
|
||||
mint_matrix_account(deps.matrix_homeserver.as_deref(), &agent).await
|
||||
}
|
||||
SwarmNodeKind::MintHiveSenderToken { hive } => {
|
||||
mint_hive_sender(deps.matrix_homeserver.as_deref(), &hive).await
|
||||
}
|
||||
SwarmNodeKind::RenewAgentQueueCredential { agent } => {
|
||||
match agent_renewal::renew(&agent).await {
|
||||
Ok(()) => Outcome::Done,
|
||||
|
|
@ -408,6 +417,23 @@ async fn mint_matrix_account(
|
|||
}
|
||||
}
|
||||
|
||||
/// The `MintHiveSenderToken` arm, lifted out for the same reason.
|
||||
async fn mint_hive_sender(homeserver: Option<&str>, hive: &str) -> hive_jobq::scheduler::Outcome {
|
||||
use hive_jobq::scheduler::Outcome;
|
||||
|
||||
let Some(base) = homeserver else {
|
||||
return Outcome::Failed(format!(
|
||||
"no matrix homeserver configured on this host ({} unset), so no hive \
|
||||
sender token can be minted",
|
||||
matrix_account::DEFAULT_HOMESERVER_ENV
|
||||
));
|
||||
};
|
||||
match matrix_account::hive_sender::ensure_hive_sender_token(base, hive).await {
|
||||
Ok(()) => Outcome::Done,
|
||||
Err(e) => Outcome::Failed(format!("{e:#}")),
|
||||
}
|
||||
}
|
||||
|
||||
/// Declare a brand-new agent at [`NEW_AGENT_WANTED_STATE`] — unless it turns
|
||||
/// out not to be new: an agent that already has a declaration (other than
|
||||
/// `Destroyed`, which this treats as reusable) is left alone, so a retried
|
||||
|
|
@ -2132,6 +2158,27 @@ fn queue_matrix_account_mints(
|
|||
Ok(ids)
|
||||
}
|
||||
|
||||
/// Insert one `MintHiveSenderToken` job per hive and return the nodes' ids.
|
||||
/// `matrix_account::hive_sender::spawn`'s periodic pass is the one caller.
|
||||
fn queue_hive_sender_mints(
|
||||
sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
|
||||
hives: Vec<String>,
|
||||
) -> Result<Vec<hive_jobq::NodeId>> {
|
||||
let mut sched = sched
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let mut ids = Vec::with_capacity(hives.len());
|
||||
for hive in hives {
|
||||
let queued = sched
|
||||
.insert_job(None, |b| {
|
||||
vec![b.node(SwarmNodeKind::MintHiveSenderToken { hive }).guid()]
|
||||
})
|
||||
.map_err(|e| anyhow::anyhow!("{e}"))?;
|
||||
ids.extend(queued);
|
||||
}
|
||||
Ok(ids)
|
||||
}
|
||||
|
||||
/// The homeserver agents' own matrix accounts are minted on. Unset is this
|
||||
/// controller creating agents with no matrix account, which the mint node
|
||||
/// reports by name rather than `main` refusing to start.
|
||||
|
|
@ -2141,21 +2188,30 @@ fn configured_matrix_homeserver() -> Option<Arc<str>> {
|
|||
.map(Arc::from)
|
||||
}
|
||||
|
||||
/// Start the matrix-account backfill when a homeserver is configured. Lifted
|
||||
/// out of `main` for `clippy::too_many_lines`.
|
||||
/// Start the matrix-account backfill and the hive sender-token pass when a
|
||||
/// homeserver is configured. Lifted out of `main` for
|
||||
/// `clippy::too_many_lines`.
|
||||
fn spawn_matrix_account_backfill(
|
||||
jobq: &Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
|
||||
homeserver: Option<Arc<str>>,
|
||||
hives: &[HiveEntry],
|
||||
) {
|
||||
let Some(base) = homeserver else {
|
||||
return;
|
||||
};
|
||||
let sched = Arc::clone(jobq);
|
||||
matrix_account::agent_token::spawn(base, move |agents| {
|
||||
matrix_account::agent_token::spawn(Arc::clone(&base), move |agents| {
|
||||
if let Err(e) = queue_matrix_account_mints(&sched, agents) {
|
||||
tracing::warn!(error = %format!("{e:#}"), "agent matrix accounts: queueing failed");
|
||||
}
|
||||
});
|
||||
let sched = Arc::clone(jobq);
|
||||
let hive_names = hives.iter().map(|h| h.name.clone()).collect();
|
||||
matrix_account::hive_sender::spawn(base, hive_names, move |hives| {
|
||||
if let Err(e) = queue_hive_sender_mints(&sched, hives) {
|
||||
tracing::warn!(error = %format!("{e:#}"), "hive sender tokens: queueing failed");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Start the forge's periodic passes — the swarm-wide objects, and agents'
|
||||
|
|
@ -2714,7 +2770,8 @@ async fn main() -> Result<()> {
|
|||
hive_jobq::Graph::new(),
|
||||
hive_jobq::resources::ResourceTable::new(),
|
||||
)));
|
||||
spawn_matrix_account_backfill(&jobq, deps.matrix_homeserver.clone());
|
||||
let hives = load_hives();
|
||||
spawn_matrix_account_backfill(&jobq, deps.matrix_homeserver.clone(), &hives);
|
||||
spawn_jobq_worker(Arc::clone(&jobq), deps);
|
||||
// Bound to a named variable, not `_` — dropping the provider stops its
|
||||
// `PeriodicReader`, so it must live as long as `main` does (which it
|
||||
|
|
@ -2771,7 +2828,6 @@ async fn main() -> Result<()> {
|
|||
let config_prs = forge_client.clone().map(config_pr::spawn);
|
||||
let state_forge = keep_forge_for_state(forge_client, webhook_secret.clone());
|
||||
|
||||
let hives = load_hives();
|
||||
spawn_agent_renewal(&jobq, wanted_writer(status.as_ref()), &hives);
|
||||
// Before serving, because a hive whose role does not exist cannot log in,
|
||||
// and one whose policy does not exist logs in able to read nothing —
|
||||
|
|
@ -4370,6 +4426,37 @@ mod tests {
|
|||
assert_eq!(agents, ["a", "b"]);
|
||||
}
|
||||
|
||||
/// The hive sender-token pass's queueing: one mint node per hive, and
|
||||
/// the node names the hive rather than an agent.
|
||||
#[test]
|
||||
fn a_queued_hive_sender_mint_is_one_node_per_hive() {
|
||||
use hive_jobq_wire::WireNode as _;
|
||||
|
||||
let sched = std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
|
||||
hive_jobq::Graph::new(),
|
||||
hive_jobq::resources::ResourceTable::new(),
|
||||
));
|
||||
let ids =
|
||||
super::queue_hive_sender_mints(&sched, vec!["alpha".to_owned(), "beta".to_owned()])
|
||||
.expect("two jobs insert");
|
||||
assert_eq!(ids.len(), 2);
|
||||
let guard = sched
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let mut hives: Vec<String> = guard
|
||||
.graph()
|
||||
.nodes()
|
||||
.map(|n| {
|
||||
assert_eq!(n.payload.label(), "mint_hive_sender_token");
|
||||
let data = n.payload.data(n.id.get());
|
||||
assert!(data.get("agent").is_none(), "{data}");
|
||||
data["hive"].as_str().expect("hive is a string").to_owned()
|
||||
})
|
||||
.collect();
|
||||
hives.sort();
|
||||
assert_eq!(hives, ["alpha", "beta"]);
|
||||
}
|
||||
|
||||
/// The manual route and the periodic pass both go through
|
||||
/// `queue_forge_token_mints`, so this is the assertion that a backfilled
|
||||
/// mint is the same pair of nodes agent creation inserts: the forge user,
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
//! the swarm's homeserver, minted with the swarm's appservice token and stored
|
||||
//! where the agent's daemon reads it. See docs/swarm/credentials.md for why
|
||||
//! `main` is the one name [`put_matrix_account`] refuses to write.
|
||||
//! [`hive_sender`] mints each **hive's** sender account the same way, sharing this module's helpers with [`agent_token`].
|
||||
//!
|
||||
//! **The external one** — [`put_matrix_account`] and everything under it — is
|
||||
//! an account somewhere else that an operator hands us a credential for: put it
|
||||
|
|
@ -39,6 +40,7 @@ use utoipa::ToSchema;
|
|||
use super::{AppState, error_problem, swarm_hive};
|
||||
|
||||
pub mod agent_token;
|
||||
pub mod hive_sender;
|
||||
|
||||
fn default_mode() -> String {
|
||||
"token".to_owned()
|
||||
|
|
|
|||
|
|
@ -107,7 +107,7 @@ pub fn plan(observed: &[(String, Observed)]) -> Vec<String> {
|
|||
}
|
||||
|
||||
/// The swarm appservice token, or an error naming why there is none.
|
||||
async fn appservice_token(store: &SecretStore) -> Result<String> {
|
||||
pub(super) async fn appservice_token(store: &SecretStore) -> Result<String> {
|
||||
let path = matrix::swarm_appservice_token_path()?;
|
||||
let stored: Option<matrix::Credential> = store
|
||||
.read_optional(&path)
|
||||
|
|
@ -129,9 +129,18 @@ async fn probe(
|
|||
base: &str,
|
||||
agent: &str,
|
||||
) -> Result<Probe> {
|
||||
let path = matrix::account_path(agent, ACCOUNT)?;
|
||||
probe_at(store, http, base, &matrix::account_path(agent, ACCOUNT)?).await
|
||||
}
|
||||
|
||||
/// What the store and the homeserver say about the token stored at `path`.
|
||||
pub(super) async fn probe_at(
|
||||
store: &SecretStore,
|
||||
http: &reqwest::Client,
|
||||
base: &str,
|
||||
path: &str,
|
||||
) -> Result<Probe> {
|
||||
let stored: Option<matrix::Credential> = store
|
||||
.read_optional(&path)
|
||||
.read_optional(path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
let Some(stored) = stored else {
|
||||
|
|
@ -170,35 +179,52 @@ pub async fn ensure_agent_matrix_account(base: &str, agent: &str) -> Result<()>
|
|||
Decision::Mint(reason) => reason,
|
||||
};
|
||||
|
||||
let token = match homeserver::register(&http, base, agent, &as_token).await? {
|
||||
let path = matrix::account_path(agent, ACCOUNT)?;
|
||||
mint_at(&store, &http, base, agent, &as_token, &path).await?;
|
||||
tracing::info!(agent, ?reason, %path, "agent matrix account token minted and stored");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Create the account `account` (a localpart), or log in to it as the appservice when it
|
||||
/// exists, store the token at `path`, and read it back with `whoami`.
|
||||
///
|
||||
/// # Errors
|
||||
/// When the store or the homeserver refuses a step, or the new token
|
||||
/// authenticates as someone else.
|
||||
pub(super) async fn mint_at(
|
||||
store: &SecretStore,
|
||||
http: &reqwest::Client,
|
||||
base: &str,
|
||||
account: &str,
|
||||
as_token: &str,
|
||||
path: &str,
|
||||
) -> Result<()> {
|
||||
let token = match homeserver::register(http, base, account, as_token).await? {
|
||||
homeserver::Registered::Token(token) => token,
|
||||
// An agent minted before, or one a hive created back when hives did
|
||||
// this: log in as the appservice on the same device instead.
|
||||
// An account minted before, or one a hive created itself: log in as
|
||||
// the appservice on the same device instead.
|
||||
homeserver::Registered::AlreadyExists => {
|
||||
homeserver::appservice_login(&http, base, agent, &as_token).await?
|
||||
homeserver::appservice_login(http, base, account, as_token).await?
|
||||
}
|
||||
};
|
||||
let path = matrix::account_path(agent, ACCOUNT)?;
|
||||
store
|
||||
.write(
|
||||
&path,
|
||||
path,
|
||||
&matrix::Credential {
|
||||
value: token.clone(),
|
||||
homeserver: Some(base.to_owned()),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.with_context(|| format!("storing {agent}'s matrix token at {path}"))?;
|
||||
.with_context(|| format!("storing @{account}'s matrix token at {path}"))?;
|
||||
|
||||
match homeserver::whoami(&http, base, &token)
|
||||
match homeserver::whoami(http, base, &token)
|
||||
.await
|
||||
.with_context(|| format!("using {agent}'s new matrix token"))?
|
||||
.with_context(|| format!("using @{account}'s new matrix token"))?
|
||||
{
|
||||
Whoami::User(user) if localpart(&user) == Some(agent) => {}
|
||||
_ => bail!("{agent}'s new matrix token does not authenticate as {agent}"),
|
||||
Whoami::User(user) if localpart(&user) == Some(account) => Ok(()),
|
||||
_ => bail!("@{account}'s new matrix token does not authenticate as @{account}"),
|
||||
}
|
||||
tracing::info!(agent, ?reason, %path, "agent matrix account token minted and stored");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// One pass: every agent holding a store identity, observed.
|
||||
|
|
|
|||
201
swarm-controller/src/matrix_account/hive_sender.rs
Normal file
201
swarm-controller/src/matrix_account/hive_sender.rs
Normal file
|
|
@ -0,0 +1,201 @@
|
|||
//! Each hive's sender account, `@hive-<hive>:`, on the swarm's homeserver:
|
||||
//! created here with the **swarm's** appservice token and stored at
|
||||
//! `swarm/hives/<hive>/matrix/sender-token`, where hive-c0re's matrix sweep
|
||||
//! reads it under the hive's own store identity. A hive whose homeserver runs
|
||||
//! elsewhere holds no appservice token, so this is its only sender token.
|
||||
//!
|
||||
//! Runs for every hive in the directory, local or remote, and decides the
|
||||
//! same way [`super::agent_token`] does: a stored token that `whoami`
|
||||
//! confirms as `@hive-<hive>:` is kept, so this writes only when the path is
|
||||
//! empty or its token is dead.
|
||||
//!
|
||||
//! ⚠️ `swarm-matrix-ctl mint` also writes this path for the hive whose host
|
||||
//! runs the homeserver, and skips when it is non-empty. Both log in on the
|
||||
//! same pinned device, so if both find it empty at once, one of the two
|
||||
//! tokens is dead on arrival. Whichever of them lands in the store, the next
|
||||
//! [`RECONCILE_INTERVAL`] pass either keeps it (live) or re-mints it
|
||||
//! (`Revoked`), and matrix-ctl never writes a non-empty path — so the store
|
||||
//! converges on one live token, and the hive's sweep takes whatever it holds.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use swarm_matrix_client as homeserver;
|
||||
use swarm_secret_client::matrix;
|
||||
|
||||
use super::agent_token::{Decision, Observed, appservice_token, classify, mint_at, plan, probe_at};
|
||||
|
||||
/// How often [`spawn`] re-checks every hive's sender token.
|
||||
const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
|
||||
|
||||
/// Decide what to do about `hive`'s sender token from what the stored one is.
|
||||
fn classify_hive(hive: &str, probe: &super::agent_token::Probe) -> Decision {
|
||||
classify(&matrix::hive_localpart(hive), probe)
|
||||
}
|
||||
|
||||
/// Make sure `hive`'s sender account holds a live token in the store,
|
||||
/// creating the account or logging in to it when it does not. The whole job
|
||||
/// of the `MintHiveSenderToken` node.
|
||||
///
|
||||
/// # Errors
|
||||
/// When the store or the homeserver refuses a step, the swarm appservice
|
||||
/// token has not been published, or the new token authenticates as someone
|
||||
/// else.
|
||||
pub async fn ensure_hive_sender_token(base: &str, hive: &str) -> Result<()> {
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
let as_token = appservice_token(&store).await?;
|
||||
let http = homeserver::client()?;
|
||||
let path = matrix::sender_token_path(hive)?;
|
||||
let reason = match classify_hive(hive, &probe_at(&store, &http, base, &path).await?) {
|
||||
Decision::Keep => {
|
||||
tracing::debug!(hive, "hive sender token is current; left as it is");
|
||||
return Ok(());
|
||||
}
|
||||
Decision::Mint(reason) => reason,
|
||||
};
|
||||
let localpart = matrix::hive_localpart(hive);
|
||||
mint_at(&store, &http, base, &localpart, &as_token, &path).await?;
|
||||
tracing::info!(hive, ?reason, %path, "hive sender token minted and stored");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// One pass: every hive in `hives`, observed.
|
||||
///
|
||||
/// Fails as a whole when the swarm appservice token is not published, for
|
||||
/// the reason `agent_token`'s pass does.
|
||||
async fn observe_all(base: &str, hives: &[String]) -> Result<Vec<(String, Observed)>> {
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
appservice_token(&store).await?;
|
||||
let http = homeserver::client()?;
|
||||
let mut observed = Vec::with_capacity(hives.len());
|
||||
for hive in hives {
|
||||
let o = match matrix::sender_token_path(hive) {
|
||||
Ok(path) => match probe_at(&store, &http, base, &path).await {
|
||||
Ok(p) => Observed::Decided(classify_hive(hive, &p)),
|
||||
Err(e) => {
|
||||
tracing::warn!(hive, error = %format!("{e:#}"), "hive sender token: read failed");
|
||||
Observed::Unknown
|
||||
}
|
||||
},
|
||||
Err(e) => {
|
||||
tracing::warn!(hive, error = %e, "hive sender token: the hive name forms no store path");
|
||||
Observed::Unknown
|
||||
}
|
||||
};
|
||||
observed.push((hive.clone(), o));
|
||||
}
|
||||
Ok(observed)
|
||||
}
|
||||
|
||||
/// Check every hive's sender token now and every [`RECONCILE_INTERVAL`]
|
||||
/// after, and hand the hives that need one to `enqueue`, which inserts a
|
||||
/// `MintHiveSenderToken` node for each.
|
||||
///
|
||||
/// The roster is the controller's hive directory. A pass that fails is
|
||||
/// logged and retried on the next tick; it never stops the daemon.
|
||||
pub fn spawn(base: Arc<str>, hives: Vec<String>, enqueue: impl Fn(Vec<String>) + Send + 'static) {
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
match observe_all(&base, &hives).await {
|
||||
Ok(observed) => {
|
||||
let minting = plan(&observed);
|
||||
if minting.is_empty() {
|
||||
tracing::debug!(
|
||||
checked = observed.len(),
|
||||
"hive sender tokens: all current"
|
||||
);
|
||||
} else {
|
||||
tracing::info!(
|
||||
checked = observed.len(),
|
||||
minting = minting.len(),
|
||||
"hive sender tokens: queueing mints"
|
||||
);
|
||||
enqueue(minting);
|
||||
}
|
||||
}
|
||||
Err(e) => tracing::warn!(
|
||||
error = %format!("{e:#}"),
|
||||
retry_in_s = RECONCILE_INTERVAL.as_secs(),
|
||||
"hive sender tokens: pass failed; retrying next tick"
|
||||
),
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::agent_token::{MintReason, Probe};
|
||||
use super::*;
|
||||
use swarm_matrix_client::Whoami;
|
||||
|
||||
fn user(id: &str) -> Probe {
|
||||
Probe::Whoami(Whoami::User(id.to_owned()))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_live_token_for_the_hive_account_is_kept() {
|
||||
assert_eq!(
|
||||
classify_hive("pr1ma", &user("@hive-pr1ma:t.local")),
|
||||
Decision::Keep
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nothing_stored_mints() {
|
||||
// A hive whose homeserver is remote: nothing has ever written its path.
|
||||
assert_eq!(
|
||||
classify_hive("pr1ma", &Probe::NotStored),
|
||||
Decision::Mint(MintReason::NotStored)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_dead_token_mints() {
|
||||
// What the loser of a simultaneous mint with matrix-ctl leaves behind.
|
||||
assert_eq!(
|
||||
classify_hive("pr1ma", &Probe::Whoami(Whoami::UnknownToken)),
|
||||
Decision::Mint(MintReason::Revoked)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_token_for_another_account_mints() {
|
||||
// The bare hive name is an agent's localpart, not the hive's account,
|
||||
// and another hive's account is not this one's.
|
||||
for other in ["@pr1ma:t.local", "@hive-beta:t.local", "@hive:t.local"] {
|
||||
assert_eq!(
|
||||
classify_hive("pr1ma", &user(other)),
|
||||
Decision::Mint(MintReason::OtherUser),
|
||||
"{other}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_outage_plans_nothing() {
|
||||
// A mint replaces the token the hive is running on, so a pass that
|
||||
// could not read must not mint for every hive.
|
||||
let observed = [
|
||||
("alpha".to_owned(), Observed::Unknown),
|
||||
("beta".to_owned(), Observed::Unknown),
|
||||
];
|
||||
assert!(plan(&observed).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_published_path_is_the_one_the_hive_reads() {
|
||||
// hive-c0re's `stored_sender_token` and `swarm-matrix-ctl mint` both
|
||||
// resolve this path; the literal is what the bao grant names.
|
||||
assert_eq!(
|
||||
matrix::sender_token_path("pr1ma").expect("a plain name is legal"),
|
||||
"swarm/hives/pr1ma/matrix/sender-token"
|
||||
);
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue