diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index 3e9554392..7622e027f 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -109,7 +109,16 @@ On a hive that already has a ruth container from an earlier release, create her once the same way, so that her config repo and store identity exist. The container keeps starting with the hive meanwhile. -## 5 · Your first agent +## 5 · Matrix + +Your matrix account comes from SSO. Invite it to the rooms you want it in: + +```bash +hivectl matrix invite mara --room '#some-room:yourserver' +hivectl matrix invite @mara:yourserver --room '!abc:yourserver' +``` + +## 6 · Your first agent Create it from the swarm UI, or: diff --git a/docs/integrations/matrix.md b/docs/integrations/matrix.md index 56a9be31f..99fd7da58 100644 --- a/docs/integrations/matrix.md +++ b/docs/integrations/matrix.md @@ -6,7 +6,7 @@ client at `chat./` (the `gatewayHost` vhost). A swarm runs one homeserver, on one host. Two namespaces configure it: - `services.hyperhive.swarm.matrix.*` — what the homeserver **is**, as - every hive sees it: `serverName`, `gatewayHost`, `allowEncryption`. + every hive sees it: `serverName`, `gatewayHost`, `apiUrl`, `allowEncryption`. - `services.hyperhive.deploy.matrix.*` — what the host running it decides: `enable`, `gui.package`, `trustedServers`, `maxRequestSize`, `sso.clientSecretFile`. @@ -96,8 +96,11 @@ and federation peers all reach the homeserver at `chat.` through the gateway; from outside the host that takes `gateway.openFirewall`. -hive-c0re doesn't talk to the homeserver: matrix is swarm-level, and -the hive has no matrix API URL to set. +hive-c0re talks to matrix (its own sender login, invites) through `swarm.matrix.apiUrl`, which defaults to +`https://` on every hive. That login carries +the hive's matrix access token, and on a hive that doesn't run the +homeserver it crosses the network, so the default is TLS; hive-c0re +verifies the gateway's certificate against the hive trust bundle. Federation peers find the homeserver through `.well-known/matrix/server` at the server name, which names `:443` @@ -140,8 +143,8 @@ a token. reads only `.yaml`/`.yml` entries from there, so the sibling credentials are invisible to it. The `.yaml` suffix on the credential id is what makes this work. -4. **hive-c0re** holds no appservice token, creates no account and does - not talk to matrix. +4. **hive-c0re** holds no appservice token and creates no account. Its + `@hive-:` token comes from the swarm store (below). 5. **`swarm-controller` creates every account**: each agent's, with its token stored at `swarm/agents//matrix/main` for the agent's `hive-matrix-daemon` to read as the agent itself, and each hive's @@ -243,14 +246,21 @@ whose hives share a homeserver gives each of them its own identity: the homeserver attributes an action to the hive that took it, and revoking one hive's standing leaves the others alone. -Its access token is the **sender token**. `swarm-controller`, its only -minter, mints it for every hive with the swarm's appservice token and -publishes it to `swarm/hives//matrix/sender-token`, inside the hive's own -read grant (`swarm/hives//*`). Nothing on the hive uses it any more: -hive-c0re used to present it for its Space and invite calls and no longer -makes any. The name says what it authenticates as — an account the -swarm's appservice creates — rather than any privilege level, because it -carries none. +Its access token is the **sender token**, and it's the credential +hive-c0re presents for every homeserver call it makes on the hive's +behalf. It's per hive for the same reason the account is: +`swarm-controller`, its only minter, mints it for every hive with the swarm's +appservice token and publishes it to `swarm/hives//matrix/sender-token`, and the hive +reads it from there under its own certificate. That path sits inside the +hive's own read grant (`swarm/hives//*`), so a hive fetches its own +token and gets a refusal on any other hive's. The name says what it +authenticates as — an account the swarm's appservice creates — rather +than any privilege level, because it carries none. + +It needs no promotion for what the hive does with it. Inviting a user into a +room it has the power in is an ordinary client call — there is no homeserver +admin in any of it, and no Synapse admin API to reach for either, since +tuwunel has none. Promoting a user to homeserver admin and resetting a password both need an admin **sender**: `!admin …` messages into `#admins:`, and @@ -260,6 +270,17 @@ admin. Of the swarm's own accounts only `@swarm` is a homeserver admin [Operators are homeserver admins](#operators-are-homeserver-admins)), so both are swarm-level operations. +
Store precedence for a hive's sender account + +`ensure_hive_user` reads the per-hive path +`swarm/hives//matrix/sender-token` **first, on every sweep**, not +just when that file is missing (`sender_source`'s decision). +`swarm-controller` mints a per-hive token within five minutes of a hive +appearing; the sweep then overwrites the per-hive file, no boot +required. + +
+
Appservice identity `swarm-matrix-ctl appservice render` renders the registration from the @@ -323,14 +344,17 @@ Initial rollout settings: stored yet the account comes up without backup and picks it up on the next daemon start. A failure is a `WARN` and never stops the daemon. -## Retired hive-side matrix code +## Hive sender token -Earlier versions of hive-c0re created a private "hive" Matrix Space and a -`hive-chat` room, invited every agent to both, kept the hive's sender token -on disk and offered `hivectl matrix invite`. Matrix is swarm-level, so -hive-c0re dropped all of it and no longer talks to the homeserver. The Space, the -room and the files `/var/lib/hyperhive/matrix/{access-token,space-room-id,chat-room-id}` -are harmless leftovers, so delete them by hand if you like. +hive-c0re keeps the hive's `@hive-:` sender token on disk for +`hivectl matrix invite`, refreshed from the swarm secret store by the +matrix sweep (at boot, then every 30 minutes). + +Earlier versions also created a private "hive" Matrix Space and a +`hive-chat` room, invited every agent to both, and kept their ids in +`/var/lib/hyperhive/matrix/space-room-id` and `chat-room-id`. The sweep no +longer touches any of them. The rooms and files are harmless leftovers, so +delete them by hand if you like. ## Configuration tuning diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index da68f5547..e5a8ee4cc 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -73,6 +73,7 @@ Cheap — no build slot: | `EmitRebuilt` | tail of a rebuild — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome | | `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload | | `ForgeSweep` | one-shot boot-time publish of the hive's forge read token for nix (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless | +| `MatrixSweep` | matrix sender-token sweep (`matrix::ensure_all`): the boot-time instance, plus one every 30 min from a loop in `main.rs`. Holds `Resource::MatrixSweep` (capacity 1), so two passes never overlap; each tick queues its own pass, which waits for the resource if one is already live. Agentless | | `KnowledgePull` | `/knowledge` pull (`knowledge::pull`): at boot (commits that landed while `hive-c0re` was down), on the swarm knowledge-changed event, and hourly as a fallback. Holds `Resource::KnowledgeTree` (capacity 1), so two pulls never overlap on the working tree; each trigger queues its own pass, which waits for the resource if one is already live. Agentless | | `WantedPull` | pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. Runs at boot, waiting up to a minute in total for the queue connection, and again after every attach of the wanted-state watch (which retries with a backoff when the attach fails or the stream ends); the deploy event (`swarm_status`) and the watch are the fast path, this repairs a missed one. A first deploy already queued or running for an agent isn't queued again. Agentless | diff --git a/docs/tools/hivectl-cli.md b/docs/tools/hivectl-cli.md index 1a35f7ad1..8655f703c 100644 --- a/docs/tools/hivectl-cli.md +++ b/docs/tools/hivectl-cli.md @@ -7,6 +7,8 @@ This document contains the help content for the `hivectl` command-line program. * [`hivectl`↴](#hivectl) * [`hivectl forge`↴](#hivectl-forge) * [`hivectl forge reconcile-config`↴](#hivectl-forge-reconcile-config) +* [`hivectl matrix`↴](#hivectl-matrix) +* [`hivectl matrix invite`↴](#hivectl-matrix-invite) * [`hivectl agent`↴](#hivectl-agent) * [`hivectl agent restart`↴](#hivectl-agent-restart) * [`hivectl agent pause`↴](#hivectl-agent-pause) @@ -43,13 +45,14 @@ This document contains the help content for the `hivectl` command-line program. ## `hivectl` -Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that don't go through the broker, plus future recovery / debugging verbs. +Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that don't go through the broker — manual user provisioning on the bundled matrix container, plus future recovery / debugging verbs. **Usage:** `hivectl [OPTIONS] ` ###### **Subcommands:** * `forge` — Reconcile an agent's config between this hive and the forge +* `matrix` — matrix-tuwunel invites * `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running * `list-agents` — Show all managed agents with their status and technical state * `quota-enable` — Enable btrfs qgroup accounting on the agent-state filesystem @@ -109,6 +112,36 @@ Always prints the diff first. `--from forge` resets the local checkout to forge +## `hivectl matrix` + +matrix-tuwunel invites. + +Manual invites into a room by id or alias. + +**Usage:** `hivectl matrix ` + +###### **Subcommands:** + +* `invite` — Invite a matrix user to a room. Idempotent + + + +## `hivectl matrix invite` + +Invite a matrix user to a room. Idempotent + +**Usage:** `hivectl matrix invite --room ` + +###### **Arguments:** + +* `` — User to invite: a full id (`@mara:server`) or a bare localpart (qualified with the homeserver's `server_name`) + +###### **Options:** + +* `--room ` — Target room id (`!abc:server`) or alias (`#name:server`) + + + ## `hivectl agent` Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running. diff --git a/docs/tools/hivectl.md b/docs/tools/hivectl.md index 379d49ed3..43822c89b 100644 --- a/docs/tools/hivectl.md +++ b/docs/tools/hivectl.md @@ -10,8 +10,8 @@ login user in `services.hyperhive.c0re.adminUsers` (adds you to the Available via the `hive-c0re` package in the host NixOS config. Unlike the `hive-c0re` daemon subcommands (which go through the broker), -`hivectl` covers direct host-side administration: container lifecycle -shortcuts and interactive agent shell access. +`hivectl` covers direct host-side administration: manual provisioning of +matrix accounts, container lifecycle shortcuts, and interactive agent shell access. This page is the curated guide. For the exhaustive flag-by-flag reference autogenerated from the binary's own command tree, see @@ -42,6 +42,26 @@ hivectl forge reconcile-config iris --verbose # include the full diff, not supported yet (forge `main` is branch-protected; resolve via a config PR). With no `--from` it prompts for the direction after the diff. +## Matrix + +Manual invites into a room. Requires the `hive-matrix` container to be +running (`services.hyperhive.deploy.matrix.enable = true`). + +```bash +hivectl matrix invite mara --room '#some-room:server' # a room alias +hivectl matrix invite @mara:server --room '!abc:server' # ...or a room id +``` + +Human matrix accounts come from SSO, not `hivectl`; matrix homeserver +admin should eventually come from membership in authelia's `admins` +group; nobody has built that sync yet. + +- `invite`: invites a matrix user (full `@user:server` or a bare + localpart, qualified with the homeserver's `server_name`) to the + `--room` id / `#alias`. Uses the sender token; the account must be a + member of the target room with invite power. + Idempotent — already-member / already-invited is a no-op. + ## Agents Container lifecycle shortcuts that go through the host admin socket. diff --git a/hive-c0re/src/forge/reader.rs b/hive-c0re/src/forge/reader.rs index 149ea67f0..fa1fddb30 100644 --- a/hive-c0re/src/forge/reader.rs +++ b/hive-c0re/src/forge/reader.rs @@ -90,7 +90,8 @@ pub async fn publish_nix_credential() -> bool { /// This hive's token from the store, under the hive's own identity. /// /// `None`, never an error, for every way it can come up empty — no hive name, -/// no `BAO_*` identity, an unreachable store, nothing at the path. +/// no `BAO_*` identity, an unreachable store, nothing at the path — the same +/// as `matrix::stored_sender_token`. async fn stored_reader_token() -> Option { let hive = std::env::var("HYPERHIVE_HIVE_NAME") .ok() diff --git a/hive-c0re/src/job_queue/exec.rs b/hive-c0re/src/job_queue/exec.rs index 10654171a..4c2ae7059 100644 --- a/hive-c0re/src/job_queue/exec.rs +++ b/hive-c0re/src/job_queue/exec.rs @@ -106,6 +106,8 @@ pub(super) trait Host: Sync { fn purge_dirs(&self, name: &str) -> Vec; /// Publish the hive's forge credential for nix. async fn forge_ensure_all(&self); + /// One matrix sender-token pass; whether the token is in place. + async fn matrix_ensure_all(&self) -> bool; /// One `/knowledge` pull. async fn knowledge_pull(&self, coord: &Arc) -> Result<()>; /// One swarm wanted-state pull. @@ -243,6 +245,10 @@ impl Host for RealHost { crate::forge::ensure_all().await; } + async fn matrix_ensure_all(&self) -> bool { + crate::matrix::ensure_all().await + } + async fn knowledge_pull(&self, coord: &Arc) -> Result<()> { crate::workers::knowledge::pull(coord).await } @@ -344,6 +350,7 @@ async fn dispatch( // until their whole subtree settles. NodeKind::AgentWindow { .. } => Ok(()), NodeKind::ForgeSweep => run_forge_sweep(host).await, + NodeKind::MatrixSweep => run_matrix_sweep(host, MATRIX_SWEEP_DEADLINE).await, NodeKind::KnowledgePull => run_knowledge_pull(host, coord, KNOWLEDGE_PULL_DEADLINE).await, NodeKind::WantedPull => run_wanted_pull(host, coord).await, }; @@ -360,6 +367,56 @@ async fn run_forge_sweep(host: &impl Host) -> Result<()> { Ok(()) } +/// Matrix sender-token sweep as a DAG node — see [`NodeKind::MatrixSweep`]. +/// Every pass reports here, whoever submitted it: as the node's own error on +/// the dashboard, and into the debounced warning. +async fn run_matrix_sweep(host: &impl Host, deadline: std::time::Duration) -> Result<()> { + let ok = tokio::time::timeout(deadline, host.matrix_ensure_all()) + .await + .unwrap_or_else(|_| { + tracing::warn!(?deadline, "matrix: sweep timed out"); + false + }); + let mut health = matrix_sweep_health() + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if ok { + health.record_ok(); + Ok(()) + } else { + health.record_err(matrix_sweep_warning); + anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)") + } +} + +/// A pass still running holds [`Resource::MatrixSweep`] against every later +/// one. Each HTTP call has its own 10 s timeout, but the call count grows with +/// the agent count and `lifecycle::list()` has none. A pass dropped here leaves +/// at worst a created but unpersisted room, which the next pass adopts by name. +/// +/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep +const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10); + +/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one +/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the +/// dashboard; cleared by the next clean pass. +fn matrix_sweep_health() -> &'static Mutex { + static HEALTH: OnceLock> = OnceLock::new(); + HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2))) +} + +fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String { + let age = ctx.since_last_ok.map_or_else( + || "no success this session".to_owned(), + |d| format!("last ok {} ago", sweep_health::fmt_age(d)), + ); + format!( + "matrix sender-token sweep failing ({} consecutive, {age}) \ + — `hivectl matrix invite` may not work", + ctx.consecutive + ) +} + /// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every /// pass reports here, whoever submitted it: as the node's own error on the /// dashboard, and into the debounced warning. @@ -642,7 +699,7 @@ async fn substitute(host: &impl Host, name: &str, rev: &str) -> Result>, /// Names of the calls that fail (`kill`, `realise`, `prebuild`, `swap_update`, - /// `set_paused`, ...). `recorded_build` fails by finding nothing. + /// `set_paused`, ...). `recorded_build` fails by finding nothing and + /// `matrix_ensure_all` by reporting false. failing: Vec<&'static str>, /// The sweeps and pulls never finish, so only a deadline ends them. hang: bool, @@ -275,6 +277,14 @@ impl Host for FakeHost { self.record("forge_ensure_all".to_owned()); } + async fn matrix_ensure_all(&self) -> bool { + self.record("matrix_ensure_all".to_owned()); + if self.hang { + std::future::pending::<()>().await; + } + !self.fails("matrix_ensure_all") + } + async fn knowledge_pull(&self, _coord: &Arc) -> anyhow::Result<()> { if self.hang { std::future::pending::<()>().await; @@ -1078,6 +1088,31 @@ async fn forge_sweep_runs_once_and_never_fails() { assert_eq!(host.calls(), ["forge_ensure_all"]); } +#[tokio::test] +async fn matrix_sweep_fails_the_node_when_an_agent_failed_sync() { + run_matrix_sweep(&FakeHost::up(), Duration::from_secs(5)) + .await + .expect("ok"); + let err = run_matrix_sweep( + &FakeHost::up().failing("matrix_ensure_all"), + Duration::from_secs(5), + ) + .await + .expect_err("failed sync"); + assert!(format!("{err:#}").contains("failed sync"), "{err:#}"); +} + +#[tokio::test] +async fn matrix_sweep_past_its_deadline_fails_instead_of_hanging() { + let host = FakeHost { + hang: true, + ..FakeHost::up() + }; + run_matrix_sweep(&host, Duration::from_millis(50)) + .await + .expect_err("deadline"); +} + #[tokio::test] async fn knowledge_pull_passes_the_result_through_and_times_out() { let (_dir, coord) = coordinator(); @@ -1239,6 +1274,7 @@ async fn dispatch_routes_the_agentless_and_hostless_kinds() { NodeKind::ForgeSweep, NodeKind::WantedPull, NodeKind::KnowledgePull, + NodeKind::MatrixSweep, alice(|agent| NodeKind::AgentWindow { agent }), alice(|agent| NodeKind::Signal { agent }), alice(|agent| NodeKind::Drain { agent }), @@ -1256,7 +1292,12 @@ async fn dispatch_routes_the_agentless_and_hostless_kinds() { ); } let calls = host.calls(); - for want in ["forge_ensure_all", "wanted_pull", "knowledge_pull"] { + for want in [ + "forge_ensure_all", + "wanted_pull", + "knowledge_pull", + "matrix_ensure_all", + ] { assert!(calls.iter().any(|c| c == want), "{want} in {calls:?}"); } } @@ -1470,3 +1511,22 @@ async fn purge_keeps_going_past_a_dir_it_cannot_remove() { assert!(stuck.exists(), "remove_dir_all refuses a plain file"); assert!(!applied.exists()); } + +#[test] +fn matrix_sweep_warning_says_how_long_it_has_been_failing() { + use crate::stats::sweep_health::SweepFailure; + let never = matrix_sweep_warning(SweepFailure { + consecutive: 3, + since_last_ok: None, + }); + assert!( + never.contains("3 consecutive, no success this session"), + "{never}" + ); + let after_ok = matrix_sweep_warning(SweepFailure { + consecutive: 2, + since_last_ok: Some(Duration::from_secs(90)), + }); + assert!(after_ok.contains("2 consecutive, last ok "), "{after_ok}"); + assert!(!after_ok.contains("no success"), "{after_ok}"); +} diff --git a/hive-c0re/src/job_queue/model.rs b/hive-c0re/src/job_queue/model.rs index 69a9cead0..dd9e3b35e 100644 --- a/hive-c0re/src/job_queue/model.rs +++ b/hive-c0re/src/job_queue/model.rs @@ -99,6 +99,9 @@ pub enum NodeKind { /// One-shot boot-time publish of the hive's forge read token for nix /// (`forge::ensure_all`). Agentless. ForgeSweep, + /// Matrix sender-token sweep (`matrix::ensure_all`): at boot and every 30 + /// minutes. Agentless. + MatrixSweep, /// `/knowledge` pull (`knowledge::pull`): at boot, on the swarm /// knowledge-changed event, and hourly. Agentless. KnowledgePull, @@ -165,7 +168,10 @@ impl NodeKind { | NodeKind::AgentWindow { agent } | NodeKind::EmitRebuilt { agent, .. } | NodeKind::SetWanted { agent, .. } => agent, - NodeKind::ForgeSweep | NodeKind::KnowledgePull | NodeKind::WantedPull => "", + NodeKind::ForgeSweep + | NodeKind::MatrixSweep + | NodeKind::KnowledgePull + | NodeKind::WantedPull => "", } } diff --git a/hive-c0re/src/job_queue/resource.rs b/hive-c0re/src/job_queue/resource.rs index d3336cf62..433b29924 100644 --- a/hive-c0re/src/job_queue/resource.rs +++ b/hive-c0re/src/job_queue/resource.rs @@ -31,6 +31,9 @@ pub enum Resource { /// overlapping `reset` / `clean` / `pull` passes there fail on /// `.git/index.lock` and the remote-tracking ref lock. KnowledgeTree, + /// The hive's sender-token file, held by every `MatrixSweep`. Two + /// overlapping passes would race on writing it. + MatrixSweep, } /// This resource's name on the generic graph wire. @@ -46,6 +49,7 @@ impl hive_jobq_wire::WireResource for Resource { Resource::Agent(agent) => format!("agent:{agent}"), Resource::KnowledgeTree => "knowledge-tree".to_owned(), + Resource::MatrixSweep => "matrix-sweep".to_owned(), } } } diff --git a/hive-c0re/src/job_queue/scheduler.rs b/hive-c0re/src/job_queue/scheduler.rs index b44594bdc..b558a5416 100644 --- a/hive-c0re/src/job_queue/scheduler.rs +++ b/hive-c0re/src/job_queue/scheduler.rs @@ -245,6 +245,31 @@ mod tests { .count() } + #[tokio::test] + async fn two_passes_of_one_sweep_never_run_together() { + let q = JobQueue::new(1); + for _ in 0..2 { + q.insert_job(|b| vec![templates::matrix_sweep(b).guid()]) + .expect("insert"); + } + q.insert_job(|b| vec![templates::knowledge_pull(b).guid()]) + .expect("insert"); + + let first = claim(&q).expect("a matrix pass is runnable"); + let other = claim(&q).expect("a different sweep runs alongside it"); + assert!( + claim(&q).is_none(), + "the second matrix pass waits for the first" + ); + assert_eq!(running(&q), ["knowledge_pull", "matrix_sweep"]); + + first.await; + other.await; + let second = claim(&q).expect("the second matrix pass runs once the first is done"); + assert_eq!(running(&q), ["matrix_sweep"]); + second.await; + } + /// A tick landing while a pass is already live queues a second pass; the /// shared `Resource::KnowledgeTree` dep holds it back until the first /// finishes, so the two never run together. diff --git a/hive-c0re/src/job_queue/templates.rs b/hive-c0re/src/job_queue/templates.rs index f1817bbe0..7afb42468 100644 --- a/hive-c0re/src/job_queue/templates.rs +++ b/hive-c0re/src/job_queue/templates.rs @@ -370,6 +370,14 @@ pub fn destroy(builder: &JobBuilder, agent: &str, purge: bool) { // `Reconcile` root per drifted agent) — no anchor node and no per-agent child // DAGs. +/// One matrix sender-token sweep. Holding [`Resource::MatrixSweep`] is what keeps +/// two passes from overlapping, whichever caller submitted each. +pub fn matrix_sweep(builder: &JobBuilder) -> Handle<'_> { + builder + .node(NodeKind::MatrixSweep) + .needs(Resource::MatrixSweep) +} + /// One `/knowledge` pull, holding [`Resource::KnowledgeTree`] for its duration. pub fn knowledge_pull(builder: &JobBuilder) -> Handle<'_> { builder diff --git a/hive-c0re/src/job_queue/tests.rs b/hive-c0re/src/job_queue/tests.rs index ddb92e7a3..99d7dfec2 100644 --- a/hive-c0re/src/job_queue/tests.rs +++ b/hive-c0re/src/job_queue/tests.rs @@ -1507,13 +1507,19 @@ fn destroy_container_must_not_suppress_crash_watch() { #[test] fn each_sweep_holds_its_own_resource() { - // Same kind, same capacity-1 resource: two passes serialise. That the + // Same kind, same capacity-1 resource: two passes serialise. Different + // kinds, disjoint resources and no edges: they run side by side. That the // scheduler honours this is `hive_jobq`'s // `unrelated_nodes_needing_the_same_resource_are_serialized`. let q = JobQueue::new(1); insert(&q, |builder| { + let _ = templates::matrix_sweep(builder); let _ = templates::knowledge_pull(builder); }); + assert_eq!( + declared_resources_of_kind(&q, "matrix_sweep"), + vec![vec![Resource::MatrixSweep]] + ); assert_eq!( declared_resources_of_kind(&q, "knowledge_pull"), vec![vec![Resource::KnowledgeTree]] diff --git a/hive-c0re/src/main.rs b/hive-c0re/src/main.rs index 2e962bbd4..d7c8c6275 100644 --- a/hive-c0re/src/main.rs +++ b/hive-c0re/src/main.rs @@ -26,6 +26,7 @@ mod gateway_nginx; mod job_queue; mod lifecycle; mod loose_ends; +mod matrix; mod meta; mod meta_rev; mod migrate; @@ -201,7 +202,7 @@ async fn main() -> Result<()> { } /// One periodic sweep tick. Queues unconditionally; the node's own -/// capacity-1 resource dep (`Resource::KnowledgeTree`) is +/// capacity-1 resource dep (`Resource::MatrixSweep` / `KnowledgeTree`) is /// what keeps two passes of the same kind from running together, so a tick /// landing while one is live just queues one more pass behind it. fn submit_periodic_sweep( @@ -232,7 +233,7 @@ async fn cmd_serve( socket: &std::path::Path, ) -> Result<()> { // Move any host-side state still at the legacy flat layout into its - // subdir (`db/`, `forge/`, `run/`) BEFORE opening the + // subdir (`db/`, `forge/`, `matrix/`, `run/`) BEFORE opening the // broker db — the broker + build-logs dbs are among the relocated // files. Idempotent; a no-op once migrated. crate::paths::relocate_legacy_state(); @@ -306,6 +307,29 @@ async fn cmd_serve( } } }); + // Matrix sweep (`matrix::ensure_all`): the hive's own sender token, kept + // on disk for `hivectl matrix invite`. It + // creates no accounts or tokens — swarm-controller mints those. + // No-op when the hive-matrix container isn't running. Re-submitted every + // 30 minutes; the startup pass is the boot `MatrixSweep` node. + let mut matrix_shutdown = coord.shutdown_rx(); + let matrix_coord = coord.clone(); + tokio::spawn(async move { + let interval = std::time::Duration::from_mins(30); + loop { + tokio::select! { + () = tokio::time::sleep(interval) => submit_periodic_sweep( + &matrix_coord, + &job_queue::NodeKind::MatrixSweep, + job_queue::templates::matrix_sweep, + ), + _ = matrix_shutdown.changed() => { + tracing::info!("matrix ensure_all: shutdown signal received"); + break; + } + } + } + }); // Periodic broker vacuum: drop fully-acked messages older // than 30 days. Delivered-but-unacked rows (recoverable via // requeue_inflight) and undelivered rows are always kept. diff --git a/hive-c0re/src/matrix.rs b/hive-c0re/src/matrix.rs new file mode 100644 index 000000000..d92a98aac --- /dev/null +++ b/hive-c0re/src/matrix.rs @@ -0,0 +1,486 @@ +//! Optional matrix-tuwunel wiring: the hive's `@hive-:` sender token, +//! taken from the swarm secret store and kept on disk for `hivectl matrix +//! invite`. No-op when no homeserver is configured, so operators who haven't +//! flipped `services.hyperhive.deploy.matrix.enable = true` pay nothing. +//! +//! This module creates no accounts: `swarm-controller` mints every hive's +//! sender account and every agent's account with the swarm's appservice token. + +use std::path::PathBuf; + +use anyhow::{Context, Result}; + +/// Client-server API base this daemon provisions against, from +/// `HIVE_MATRIX_API_URL` (set by the hyperhive NixOS module from +/// `services.hyperhive.swarm.matrix.apiUrl`). +/// +/// `None` means **this hive has no homeserver to provision against** and +/// every matrix path no-ops — see [`is_present`]. There is deliberately no +/// fallback: a loopback default is right only when the homeserver happens to +/// share this daemon's netns, and an address baked into the binary is one +/// that builds fine and then talks to the wrong machine. The nix module +/// defaults it to `https://` on the homeserver's gateway vhost, on every +/// hive whichever host runs tuwunel. +/// +/// Also not an agent-facing address either way. An agent has its own netns; +/// agents are handed the gateway vhost via `HIVE_MATRIX_URL`, and get +/// nothing at all when the hive has no vhost to offer. +fn matrix_http() -> Option<&'static str> { + static BASE: std::sync::OnceLock> = std::sync::OnceLock::new(); + BASE.get_or_init(|| std::env::var("HIVE_MATRIX_API_URL").ok()) + .as_deref() +} + +/// [`matrix_http`] for the call sites that propagate with `?`. +/// +/// # Errors +/// When no homeserver is configured. Reaching one of these paths at all +/// means an [`is_present`] gate was skipped, so the message names that +/// rather than the missing variable. +fn matrix_base() -> Result<&'static str> { + matrix_http().context( + "matrix: no homeserver configured \ + (services.hyperhive.swarm.matrix.apiUrl / HIVE_MATRIX_API_URL) — \ + this path should have been gated on matrix::is_present()", + ) +} + +/// Host path for the appservice sender account's matrix access token. Outside every +/// purgeable path — not deleted by `destroy --purge` on any agent. +#[must_use] +pub fn sender_token_path() -> PathBuf { + crate::paths::matrix_sender_token() +} + +/// Whether this hive has a homeserver to provision against. +/// +/// **A configured API URL, not a local container**: whether the homeserver +/// runs on this host is a different question, and a remote homeserver is +/// provisioned through the same URL. +#[must_use] +pub fn is_present() -> bool { + matrix_http().is_some() +} + +/// Bring the hive's `@hive-:` sender token at [`sender_token_path()`] in +/// line with the swarm secret store. +/// +/// `swarm-controller` is the only minter: it creates the account with the +/// swarm's appservice token, keeps the stored token while it is live and +/// replaces it when it is not. The file follows the store, and is kept as it +/// is when the store has nothing or cannot be reached, so a store outage does +/// not take the hive's matrix provisioning down with it. +/// +/// # Errors +/// When neither the store nor the file holds a token, or the file write fails. +pub async fn ensure_hive_user() -> Result<()> { + let path = sender_token_path(); + let on_disk = std::fs::read_to_string(&path).ok(); + let stored = stored_sender_token().await; + match sender_source(stored.as_deref(), on_disk.as_deref()) { + SenderSource::Keep => { + tracing::debug!("matrix: the sender token is already present"); + Ok(()) + } + SenderSource::Store(token) => persist_sender_token(&path, token), + SenderSource::Unavailable => anyhow::bail!( + "matrix: no sender token in the swarm store or at {}; swarm-controller \ + mints it into the store", + path.display() + ), + } +} + +/// What [`ensure_hive_user`] does about the sender token this sweep. +#[derive(Debug, PartialEq, Eq)] +enum SenderSource<'a> { + /// The file already holds the token to use. + Keep, + /// Write the store's token to the file. + Store(&'a str), + /// No token in the store or the file. + Unavailable, +} + +/// Decide [`ensure_hive_user`]'s step from the store's token and the file's +/// content, each `None` when absent. Blank counts as absent. +fn sender_source<'a>(stored: Option<&'a str>, on_disk: Option<&str>) -> SenderSource<'a> { + let present = |s: &&str| !s.trim().is_empty(); + let stored = stored.map(str::trim).filter(present); + let on_disk = on_disk.map(str::trim).filter(present); + match (stored, on_disk) { + (Some(s), Some(d)) if s == d => SenderSource::Keep, + (Some(s), _) => SenderSource::Store(s), + (None, Some(_)) => SenderSource::Keep, + (None, None) => SenderSource::Unavailable, + } +} + +/// Write the appservice sender account's access token to `path`, 0600, creating the +/// directory if it is not there. The file's mode is the only thing keeping an +/// unprivileged reader off the hive's matrix credential. +fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<()> { + crate::secret_file::write_secret_0600(path, format!("{access_token}\n").as_bytes()) + .with_context(|| format!("matrix: write the sender token to {}", path.display()))?; + tracing::info!(path = %path.display(), "matrix: provisioned the sender token"); + Ok(()) +} + +/// Fetch the sender token `swarm-controller` published, under this hive's own +/// store identity. +/// +/// The cert role is the hive's name, straight out of `HYPERHIVE_HIVE_NAME` — +/// the same role string `lifecycle::agent_identity` logs in with, and already in +/// this process's environment, so the store read costs no plumbing through +/// [`ensure_all`]. That name is also the path's own hive segment, which is +/// what makes this read reach **this** hive's token and 403 on any other's: +/// the grant behind it is the hive's own `swarm/hives//*` stanza. +/// +/// `None`, never an error, for every way this can come up empty — no hive +/// name, no `BAO_*` identity, an unreachable store, nothing at the path. All +/// four mean the same thing to the caller ("keep the file"), and three of +/// them are the ordinary state of a swarm with no store token for this hive +/// yet, so raising would turn a supported deployment into a warning every +/// sweep. +/// +/// 🩸 Logs the store **path** and never the value. +async fn stored_sender_token() -> Option { + let hive = std::env::var("HYPERHIVE_HIVE_NAME") + .ok() + .filter(|h| !h.is_empty())?; + let path = match swarm_secret_client::matrix::sender_token_path(&hive) { + Ok(path) => path, + Err(e) => { + tracing::warn!(error = %e, "matrix: this hive's name does not form a store path"); + return None; + } + }; + let store = match swarm_secret_client::SecretStore::from_env(&hive).await { + Ok(store) => store, + Err(e) => { + tracing::debug!(error = %e, "matrix: no swarm secret store to read the sender token from"); + return None; + } + }; + match store + .read::(&path) + .await + { + Ok(credential) if !credential.value.trim().is_empty() => { + tracing::debug!(%path, "matrix: read the sender token from the swarm store"); + Some(credential.value) + } + Ok(_) => { + tracing::warn!(%path, "matrix: the stored sender token is empty"); + None + } + Err(e) => { + tracing::debug!(%path, error = %e, "matrix: no sender token in the store"); + None + } + } +} + +/// Discover the matrix `server_name` from the running homeserver via +/// `GET /_matrix/key/v2/server` (unauthenticated federation key endpoint). +/// The response JSON always includes `"server_name"` per the matrix spec. +pub async fn discover_server_name(client: &reqwest::Client) -> Result { + let base = matrix_base()?; + let url = format!("{base}/_matrix/key/v2/server"); + let resp = client + .get(&url) + .send() + .await + .context("matrix: GET /_matrix/key/v2/server")?; + let status = resp.status(); + let body = resp + .json::() + .await + .context("matrix: parse /_matrix/key/v2/server response")?; + if !status.is_success() { + anyhow::bail!("matrix: /_matrix/key/v2/server returned HTTP {status}, body: {body}"); + } + body["server_name"] + .as_str() + .map(str::to_owned) + .with_context(|| { + format!("matrix: /_matrix/key/v2/server response missing server_name field: {body}") + }) +} + +/// Read the sender token from disk. Returns an error if it +/// is absent — callers should gate their homeserver calls on this. +pub fn read_sender_token() -> Result { + let path = sender_token_path(); + std::fs::read_to_string(&path) + .ok() + .map(|s| s.trim().to_owned()) + .filter(|s| !s.is_empty()) + .with_context(|| { + format!( + "the matrix sender token was not found at {} — \ + ensure hive-c0re has started at least once with matrix enabled \ + (it provisions the hive's @hive-: account on boot)", + path.display() + ) + }) +} + +/// Fetch a user's current membership in a room via the sender token, or +/// `None` if there is no membership event (never invited) or the lookup +/// fails. Returns the raw membership string (`invite`, `join`, `leave`, …). +async fn room_membership( + client: &reqwest::Client, + sender_token: &str, + room_id: &str, + user_id: &str, +) -> Option { + let base = matrix_http()?; + // A refused or failed read is "unknown": the caller falls through to the + // invite attempt. + let member = swarm_matrix_client::room_state( + client, + base, + sender_token, + room_id, + "m.room.member", + user_id, + ) + .await + .ok()??; + member["membership"].as_str().map(ToOwned::to_owned) +} + +/// Invite a fully-qualified Matrix user id (`@user:server`) to `room_id` +/// using the sender token. Idempotent: a user who is already a member or +/// already has a pending invite is left untouched (no fresh invite is sent, +/// so they are not re-notified). A refused invite is an error unless the +/// user turns out to be invited or joined anyway — see [`forbidden_invite`]. +async fn invite_user_id( + client: &reqwest::Client, + sender_token: &str, + room_id: &str, + user_id: &str, +) -> Result<()> { + let base = matrix_base()?; + + // Skip the invite entirely when the user is already invited or joined. + // Re-POSTing an invite to a pending member re-sends the invite event, + // which re-notifies the agent on every provisioning sweep. + if let Some(membership) = room_membership(client, sender_token, room_id, user_id).await + && matches!(membership.as_str(), "invite" | "join") + { + tracing::debug!(%user_id, %room_id, %membership, "matrix: invite skipped (already a member/invited)"); + return Ok(()); + } + + match swarm_matrix_client::invite(client, base, sender_token, room_id, user_id) + .await + .context("matrix: invite")? + { + swarm_matrix_client::Invited::Sent => { + tracing::debug!(%user_id, %room_id, "matrix: invited to room"); + Ok(()) + } + swarm_matrix_client::Invited::Forbidden => { + // The pre-check misses a member when its read failed or a + // concurrent invite landed after it, so a refusal re-reads the + // membership. + let membership = room_membership(client, sender_token, room_id, user_id).await; + forbidden_invite(membership.as_deref(), user_id, room_id) + } + } +} + +/// Outcome of an invite the homeserver refused with 403. That is success only +/// when `membership`, re-read after the refusal, shows the user invited or +/// joined: `M_FORBIDDEN` covers both "already in the room" and real refusals +/// (banned target, sender lacks power), so the errcode can't tell them apart. +fn forbidden_invite(membership: Option<&str>, user_id: &str, room_id: &str) -> Result<()> { + if matches!(membership, Some("invite" | "join")) { + tracing::debug!(%user_id, %room_id, "matrix: invite refused, user already member/invited"); + return Ok(()); + } + anyhow::bail!( + "matrix: invite {user_id} to {room_id}: HTTP 403 and the user is not invited or joined" + ) +} + +/// Invite an arbitrary Matrix user to `room` (a room id or alias). `user` +/// may be a fully-qualified id (`@name:server`) or a bare localpart, which +/// is qualified with the homeserver's `server_name`. Returns the resolved +/// room id the invite targeted. Used by `hivectl matrix invite`. +/// +/// # Errors +/// +/// Returns an error if `room` is neither a room id nor an alias, or the +/// invite POST fails for a reason other than the user already being a +/// member / invited. +pub async fn invite_user( + client: &reqwest::Client, + sender_token: &str, + user: &str, + room: &str, + server_name: &str, +) -> Result { + // Qualify a bare localpart to a full user id on the hive homeserver. + let user_id = if user.starts_with('@') { + user.to_owned() + } else { + format!("@{user}:{server_name}") + }; + let room_id = if room.starts_with('#') { + swarm_matrix_client::resolve_alias(client, matrix_base()?, sender_token, room).await? + } else if room.starts_with('!') { + room.to_owned() + } else { + anyhow::bail!( + "matrix: --room {room:?} is neither a room id nor an alias; \ + prefix with '!' for a room id (!abc:server) or '#' for an \ + alias (#name:server)" + ) + }; + invite_user_id(client, sender_token, &room_id, &user_id).await?; + Ok(room_id) +} + +/// Make sure the hive's own `@hive-:` sender token is on disk. +/// Accounts are not created here: `swarm-controller` mints them with the +/// swarm's appservice token. Runs only as a `MatrixSweep` job node (see +/// `job_queue::templates::matrix_sweep`). No-op when no homeserver is +/// configured. +/// +/// Returns `true` when the token is in place, `false` when it could not be +/// fetched — the caller feeds this into a +/// [`crate::stats::sweep_health::SweepHealth`] to raise a debounced +/// hive warning on persistent failure (this sweep re-runs every 30 +/// minutes, so a one-off blip self-heals without ever warning). +pub async fn ensure_all() -> bool { + if !is_present() { + tracing::debug!("matrix: hive-matrix container absent, skipping user sweep"); + return true; + } + if let Err(e) = ensure_hive_user().await { + tracing::warn!(error = ?e, "matrix: ensure_hive_user failed"); + return false; + } + true +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn with_no_file_the_store_token_is_taken() { + assert_eq!(sender_source(Some("tok"), None), SenderSource::Store("tok")); + } + + #[test] + fn a_store_token_replaces_a_different_file_token() { + // The swarm re-minted a dead token; the file must follow it. + assert_eq!( + sender_source(Some("new\n"), Some("old\n")), + SenderSource::Store("new") + ); + } + + #[test] + fn a_file_matching_the_store_is_kept() { + // Trailing newline on disk is how `persist_sender_token` writes it. + assert_eq!( + sender_source(Some("tok"), Some("tok\n")), + SenderSource::Keep + ); + } + + #[test] + fn with_nothing_in_the_store_the_file_is_kept() { + assert_eq!(sender_source(None, Some("tok\n")), SenderSource::Keep); + } + + #[test] + fn with_no_token_anywhere_nothing_is_available() { + assert_eq!( + sender_source(Some(""), Some(" \n")), + SenderSource::Unavailable + ); + assert_eq!(sender_source(None, None), SenderSource::Unavailable); + } + + /// A banned target or a sender without power gets a 403; with no + /// membership behind it, the invite did not happen. + #[test] + fn a_refused_invite_without_membership_is_an_error() { + for membership in [None, Some("ban"), Some("leave")] { + let outcome = forbidden_invite(membership, "@a:x", "!r:x"); + assert!(outcome.is_err(), "membership {membership:?} must not pass"); + } + } + + #[test] + fn a_refused_invite_for_a_member_is_success() { + for membership in ["invite", "join"] { + forbidden_invite(Some(membership), "@a:x", "!r:x") + .expect("already invited/joined is the goal state"); + } + } + + /// The homeserver's reply body reaches no error: `invite_user_id` is + /// driven against a stub whose 500 and 403 bodies carry a marker. + #[tokio::test] + async fn an_invite_refusal_error_does_not_quote_the_response_body() { + const BODY: &str = r#"{"errcode":"M_UNKNOWN","error":"free-form-marker"}"#; + for status in [500_u16, 403] { + let app = axum::Router::new().fallback(move || async move { + ( + axum::http::StatusCode::from_u16(status).expect("a status"), + [(axum::http::header::CONTENT_TYPE, "application/json")], + BODY, + ) + }); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind"); + let base = format!("http://{}", listener.local_addr().expect("local addr")); + let server = tokio::spawn(async move { + axum::serve(listener, app).await.expect("serve"); + }); + let client = swarm_matrix_client::client().expect("client"); + let outcome = swarm_matrix_client::invite(&client, &base, "tok", "!r:x", "@a:x").await; + server.abort(); + let message = match (status, outcome) { + (403, Ok(swarm_matrix_client::Invited::Forbidden)) => { + format!("{:#}", forbidden_invite(None, "@a:x", "!r:x").unwrap_err()) + } + (500, Err(e)) => format!("{e:#}"), + (status, other) => panic!("HTTP {status}: unexpected {other:?}"), + }; + assert!(message.contains(&status.to_string()), "{message}"); + assert!(!message.contains("free-form-marker"), "{message}"); + } + } + + /// A room that is neither an id nor an alias is refused before any + /// request is made; with no default Space left, there is nothing to fall + /// back to. + #[tokio::test] + async fn invite_user_refuses_a_room_that_is_neither_an_id_nor_an_alias() { + let client = swarm_matrix_client::client().expect("client"); + let err = invite_user(&client, "tok", "mara", "general", "example.org") + .await + .expect_err("a bare name is not a room"); + assert!( + format!("{err:#}").contains("neither a room id nor an alias"), + "{err:#}" + ); + } + + /// The sweep touches nothing, and reports success, on a hive with no + /// homeserver configured (the test environment has none). + #[tokio::test] + async fn the_sweep_is_a_successful_no_op_without_a_homeserver() { + assert!(!is_present()); + assert!(ensure_all().await); + } +} diff --git a/hive-c0re/src/paths.rs b/hive-c0re/src/paths.rs index 246619a83..52bcd89a3 100644 --- a/hive-c0re/src/paths.rs +++ b/hive-c0re/src/paths.rs @@ -2,10 +2,10 @@ //! `/run/hyperhive` + `/run/hive-agent` runtime roots). //! //! Historically these were flat string literals scattered across many -//! modules (`broker.sqlite`, `agent-sockets.json`, +//! modules (`broker.sqlite`, `matrix-sender-token`, `agent-sockets.json`, //! …). This module is the **single Rust-side source** for every host //! path — the strictly host-side ones grouped into subdirs (`db/`, -//! `run/`), plus the **nix-coupled** roots +//! `matrix/`, `run/`), plus the **nix-coupled** roots //! (`agents/`, `applied/`, `meta/`, `gateway/`, `knowledge/`, //! the `/run` runtime dirs). //! @@ -89,6 +89,21 @@ pub fn agent_identity_dir(name: &str) -> PathBuf { agent_identity_root().join(name) } +/// `matrix/` — host-side matrix provisioning state (the appservice sender token, hive +/// Space room id, the hive sender account's password creds). The shared +/// registration token is bind-mounted into the tuwunel container via nix +/// and stays at its own path (tracked separately). +#[must_use] +pub fn matrix_dir() -> PathBuf { + state_root().join("matrix") +} + +/// `matrix/access-token` — the matrix appservice sender account's access token. +#[must_use] +pub fn matrix_sender_token() -> PathBuf { + matrix_dir().join("access-token") +} + /// `run/` — runtime maps hive-c0re regenerates on every meta sync. #[must_use] pub fn run_dir() -> PathBuf { @@ -237,9 +252,10 @@ pub fn agent_runtime_dir(name: &str) -> PathBuf { /// within the same filesystem is atomic. pub fn relocate_legacy_state() { let root = state_root(); - let moves: [(&str, PathBuf); 3] = [ + let moves: [(&str, PathBuf); 4] = [ ("broker.sqlite", db_dir().join("broker.sqlite")), ("build_logs.sqlite", db_dir().join("build_logs.sqlite")), + ("matrix-sender-token", matrix_sender_token()), ("agent-sockets.json", agent_sockets_file()), ]; for (old_rel, new) in &moves { diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 08f7882d4..34cd6af72 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -203,6 +203,7 @@ async fn dispatch(req: &HostRequest, coord: Arc) -> HostResponse { // federation peer-config block) and open the web surfaces // (`hivectl open`). HostRequest::Urls => HostResponse::urls(hive_urls()), + HostRequest::MatrixInvite { user, room } => handle_matrix_invite(user, room).await?, HostRequest::ReconcileConfigStatus { agent, verbose } => { crate::forge::reconcile_config_status(agent.as_str(), *verbose).await? } @@ -342,6 +343,25 @@ async fn stream_agent_status( } } +// --------------------------------------------------------------------------- +// Matrix provisioning handlers +// +// The `hivectl matrix` subcommands used to run these in-process, which forced +// the standalone CLI to link the whole daemon crate (matrix-sdk, reqwest, …). +// They now run daemon-side over the host socket, where the daemon already holds +// the sender token. Each op returns the operator-facing lines hivectl used to +// `println!` in `HostResponse::messages` for the client to print verbatim. +// --------------------------------------------------------------------------- + +/// Shared reqwest client for the matrix admin HTTP calls (30s timeout, +/// mirroring the old in-CLI client). +fn matrix_http_client() -> Result { + reqwest::Client::builder() + .timeout(std::time::Duration::from_secs(30)) + .build() + .context("build reqwest client") +} + /// True when `name` has a state dir under the agents root, i.e. it's a /// managed agent rather than a bare (operator/human) account. /// @@ -355,6 +375,21 @@ fn agent_exists(name: &hive_types::Ident) -> Result { .with_context(|| format!("check agent state dir for {name}")) } +/// Guard: matrix provisioning needs a homeserver to provision against. +/// +/// Answers "is one configured", not "is one running here": the address is +/// the swarm's homeserver, wherever it runs. +fn require_matrix_present() -> Result<()> { + if crate::matrix::is_present() { + return Ok(()); + } + anyhow::bail!( + "no matrix homeserver configured — set services.hyperhive.swarm.matrix.apiUrl (it defaults \ + to the swarm.matrix.gatewayHost vhost, and is null when that is) before provisioning \ + matrix users" + ) +} + async fn handle_quota_enable() -> Result { crate::priv_client::ensure_btrfs_quota() .await @@ -463,6 +498,26 @@ async fn handle_push_snapshot( Ok(HostResponse::success()) } +async fn handle_matrix_invite(user: &str, room: &str) -> Result { + require_matrix_present()?; + let sender_token = crate::matrix::read_sender_token()?; + let client = matrix_http_client()?; + let server_name = crate::matrix::discover_server_name(&client) + .await + .context("discover matrix server_name")?; + let room_id = crate::matrix::invite_user(&client, &sender_token, user, room, &server_name) + .await + .with_context(|| format!("matrix invite {user}"))?; + let target = if user.starts_with('@') { + user.to_owned() + } else { + format!("@{user}:{server_name}") + }; + Ok(HostResponse::messages(vec![format!( + "matrix: invited {target} to {room_id}" + )])) +} + /// Single-agent queue verbs the admin socket exposes. Each submits the /// matching DAG (persisting the `wanted` intent, serializing on the /// agent's lease, with the transient/crash-watch suppression the old diff --git a/hive-c0re/src/workers/auto_update.rs b/hive-c0re/src/workers/auto_update.rs index 43cd54a52..605c4b52b 100644 --- a/hive-c0re/src/workers/auto_update.rs +++ b/hive-c0re/src/workers/auto_update.rs @@ -196,22 +196,23 @@ fn boot_action(wanted: Option, fresh: bool, running: bool) } } -/// Submit the boot-time forge/knowledge/wanted-state sweeps as DAG -/// nodes — `ForgeSweep`, `KnowledgePull`, `WantedPull`. Unlike +/// Submit the boot-time forge/matrix/knowledge/wanted-state sweeps as DAG +/// nodes — `ForgeSweep`, `MatrixSweep`, `KnowledgePull`, `WantedPull`. Unlike /// [`submit_boot_tree`] this runs on **every** boot, quiet or not: these /// aren't config-drift work, they're startup housekeeping that always needs /// to happen, and the point of moving them here is exactly so they show up /// as real work on the dashboard instead of an invisible `tokio::spawn` that -/// only surfaces on failure. Three independent, build-slot- and lease-exempt +/// only surfaces on failure. Four independent, build-slot- and lease-exempt /// roots — no dependency edges between them, matching the existing -/// `Reconcile`-root pattern in [`boot_nodes`]. `KnowledgePull` holds its -/// sweep's own resource, so it queues behind any pass of the same sweep -/// already running. +/// `Reconcile`-root pattern in [`boot_nodes`]. `MatrixSweep` and +/// `KnowledgePull` hold their sweep's own resource, so each queues behind any +/// pass of the same sweep already running. fn submit_startup_sweep_nodes(coord: &Arc) { use crate::job_queue::NodeKind; if let Err(e) = coord.job_queue.insert_job(|b| { let _ = b.node(NodeKind::ForgeSweep); + let _ = crate::job_queue::templates::matrix_sweep(b); let _ = crate::job_queue::templates::knowledge_pull(b); let _ = b.node(NodeKind::WantedPull); Vec::new() diff --git a/hive-host-sock/src/lib.rs b/hive-host-sock/src/lib.rs index c79cc7de5..374c6caa0 100644 --- a/hive-host-sock/src/lib.rs +++ b/hive-host-sock/src/lib.rs @@ -223,6 +223,9 @@ pub enum HostRequest { #[serde(default)] scope: LifecycleScope, }, + /// Invite a matrix user to `room` (id or alias). Uses the daemon's + /// admin token; idempotent (already-member / already-invited is a no-op). + MatrixInvite { user: String, room: String }, /// Report the divergence between agent `agent`'s local applied config /// checkout and its forge `agent-configs/` `main`. The daemon /// fetches forge `main` read-only and returns a human-readable report @@ -449,7 +452,7 @@ pub struct HostResponse { pub nodes: Option>, /// Free-form operator-facing output lines the client prints verbatim /// (one per line). Carries results a request produced daemon-side that - /// have no structured home. + /// have no structured home — e.g. the invited room id from `MatrixInvite`. /// Empty for requests that produce no such output. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub messages: Vec, diff --git a/hivectl/README.md b/hivectl/README.md index e4dd81202..ef9c944ef 100644 --- a/hivectl/README.md +++ b/hivectl/README.md @@ -3,7 +3,7 @@ The operator-facing host CLI. A thin client for the `hive-c0re` daemon — speaks the host admin socket protocol (`hive-host-sock`) and does not link the daemon crate. Container lifecycle and the -gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local +matrix/gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local host state instead. ## When to use it @@ -27,6 +27,7 @@ dispatch: container. - **`forge.rs`** — reconciles an agent's config against its forge repo. +- **`matrix.rs`** — invite a matrix user to a room. - **`wg.rs`** — WireGuard mesh helpers. - **`subvol.rs`** — btrfs state-subvolume ops. - **`quota.rs`**, **`util.rs`**, **`completions.rs`**, **`open.rs`** — diff --git a/hivectl/src/cli.rs b/hivectl/src/cli.rs index 5fde38673..cbb75fbcb 100644 --- a/hivectl/src/cli.rs +++ b/hivectl/src/cli.rs @@ -10,7 +10,8 @@ use std::path::PathBuf; about = "hyperhive host CLI — operator-facing administration", long_about = "\ Sibling to the `hive-c0re` daemon binary. Covers host-side admin \ -operations that don't go through the broker, plus future \ +operations that don't go through the broker — manual user \ +provisioning on the bundled matrix container, plus future \ recovery / debugging verbs.\ " )] @@ -35,6 +36,13 @@ pub enum Cmd { #[command(subcommand)] cmd: ForgeCmd, }, + /// matrix-tuwunel invites. + /// + /// Manual invites into a room by id or alias. + Matrix { + #[command(subcommand)] + cmd: MatrixCmd, + }, /// Lifecycle actions on ONE managed agent container. Needs the /// hive-c0re daemon running. /// @@ -238,6 +246,19 @@ impl From for hive_host_sock::ReconcileDirection { } } +#[derive(Subcommand)] +pub enum MatrixCmd { + /// Invite a matrix user to a room. Idempotent. + Invite { + /// User to invite: a full id (`@mara:server`) or a bare + /// localpart (qualified with the homeserver's `server_name`). + user: String, + /// Target room id (`!abc:server`) or alias (`#name:server`). + #[arg(long)] + room: String, + }, +} + #[derive(Subcommand)] pub enum WgCmd { /// Generate this hive's WireGuard key (if absent) and print its public diff --git a/hivectl/src/main.rs b/hivectl/src/main.rs index 946550429..bce1cc99b 100644 --- a/hivectl/src/main.rs +++ b/hivectl/src/main.rs @@ -3,7 +3,7 @@ //! A thin client for the `hive-c0re` daemon: it speaks the host admin //! socket protocol (`hive-host-sock`) and does NOT link the daemon crate. //! Container lifecycle (`agent `, -//! `list-agents`, `stop` / `start`) and provisioning (`forge` / +//! `list-agents`, `stop` / `start`) and provisioning (`forge` / `matrix` / //! `gateway`) all forward to the daemon, which owns the broker, //! the credentials, and the provisioning logic — a running daemon is //! required for those. A couple of verbs work off local host state @@ -31,6 +31,8 @@ mod completions; mod quota; mod util; use completions::generate_completions; +mod matrix; +use matrix::run_matrix_cmd; mod open; use open::open_url; mod wg; @@ -63,6 +65,7 @@ async fn main() -> Result<()> { verbose, } => forge_reconcile_config(&socket, &agent, from, verbose).await, }, + Cmd::Matrix { cmd } => run_matrix_cmd(&socket, cmd).await, Cmd::Agent { name, cmd } => run_agent(&socket, &name, cmd).await, Cmd::ListAgents { json } => agents_list(&socket, json).await, Cmd::QuotaEnable => quota::quota_enable(&socket).await, diff --git a/hivectl/src/matrix.rs b/hivectl/src/matrix.rs new file mode 100644 index 000000000..ee77f9e05 --- /dev/null +++ b/hivectl/src/matrix.rs @@ -0,0 +1,48 @@ +//! `hivectl matrix` — matrix provisioning verbs. hivectl forwards +//! each request to the daemon (which owns the sender token) and renders the +//! reply; it no longer links the matrix machinery itself. + +use std::path::Path; + +use anyhow::{Context as _, Result, bail}; + +use crate::cli::MatrixCmd; + +/// Route a `matrix` subcommand to its handler. Extracted from `main`'s +/// dispatch match so the top-level router stays small. +pub(crate) async fn run_matrix_cmd(socket: &Path, cmd: MatrixCmd) -> Result<()> { + match cmd { + MatrixCmd::Invite { user, room } => matrix_invite(socket, &user, &room).await, + } +} + +/// Send a matrix provisioning request to the daemon and print the +/// operator-facing result lines it returns. The daemon owns the sender token, +/// so hivectl no longer links the matrix machinery — it just forwards the +/// request and renders the reply. +async fn matrix_request(socket: &Path, req: hive_host_sock::HostRequest) -> Result<()> { + let resp = crate::client::request(socket, req) + .await + .with_context(|| format!("connect to daemon socket {}", socket.display()))?; + if !resp.ok { + bail!( + "matrix: {}", + resp.error.as_deref().unwrap_or("unknown error") + ); + } + for line in &resp.messages { + println!("{line}"); + } + Ok(()) +} + +async fn matrix_invite(socket: &Path, user: &str, room: &str) -> Result<()> { + matrix_request( + socket, + hive_host_sock::HostRequest::MatrixInvite { + user: user.to_owned(), + room: room.to_owned(), + }, + ) + .await +} diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index fb4a1d66e..6a26ae02c 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -115,8 +115,9 @@ in ./options.nix ./theme.nix ./vhosts.nix - # Hive-CA trust for this daemon's outbound TLS. The swarm services it - # reaches over https serve a leaf chaining to the store's pki root — + # Hive-CA trust for this daemon's outbound TLS. The matrix provisioning + # URL (`swarm.matrix.apiUrl`) defaults to https on the homeserver's + # gateway vhost, which serves a leaf chaining to the store's pki root — # a CA generated at runtime, which no build-time trust store can # contain. The queue's TLS is verified against `HIVE_C0RE_OIDC_CA_FILE` # (../glue-hive-tls-controllers.nix), not this bundle. diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 44ce62f41..d8ca0eedb 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -168,6 +168,15 @@ in # answer when there is no matrix vhost to point at. HIVE_MATRIX_URL = "http://${config.services.hyperhive.swarm.matrix.gatewayHost}"; } +// lib.optionalAttrs (config.services.hyperhive.swarm.matrix.apiUrl != null) { + # Client-server API base hive-c0re uses to provision matrix (register + # agent users, invite members). + # Supplied by `services.hyperhive.swarm.matrix.apiUrl`, which defaults to + # the same gateway vhost as HIVE_MATRIX_URL above, over https because it + # carries the hive's matrix token and may cross the network. Its own + # option, so provisioning can point elsewhere without changing agents. + HIVE_MATRIX_API_URL = config.services.hyperhive.swarm.matrix.apiUrl; +} // { # The gateway always runs, so the dashboard always builds # same-origin `/agent//` links (never the direct diff --git a/nix/host-modules/hive-matrix-service.nix b/nix/host-modules/hive-matrix-service.nix index e4f18b608..5cfa49f16 100644 --- a/nix/host-modules/hive-matrix-service.nix +++ b/nix/host-modules/hive-matrix-service.nix @@ -12,12 +12,6 @@ let swarmDomain = config.services.hyperhive.swarm.domain; in { - imports = [ - (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "matrix" "apiUrl" ] '' - hive-c0re no longer talks to matrix, so it has no API URL to set: matrix is swarm-level. - '') - ]; - options.services.hyperhive.swarm.matrix = { serverName = lib.mkOption { type = lib.types.nullOr lib.types.str; @@ -55,6 +49,37 @@ in ''; }; + apiUrl = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = if cfg.gatewayHost == null then null else "https://${cfg.gatewayHost}"; + defaultText = lib.literalExpression '' + if services.hyperhive.swarm.matrix.gatewayHost == null then null + else "https://''${services.hyperhive.swarm.matrix.gatewayHost}" + ''; + example = "https://matrix.example.com"; + description = '' + Client-server API base URL **hive-c0re itself** uses to + provision matrix (register agent users, invite members). Distinct from the agent-facing + `services.hyperhive.agent.matrix.url`, which is the gateway vhost handed to + each agent's `hive-matrix-daemon`. + + Defaults to `https://` on the homeserver's gateway vhost + (`gatewayHost`), the same name on every hive whichever host runs + tuwunel, and `null` when there is no vhost. hive-c0re logs in + with the hive's matrix sender token over this URL, so it crosses + the network whenever hive-c0re is not on the homeserver's host: + an `http://` value sends that token in cleartext. hive-c0re + verifies the gateway's certificate against the hive trust bundle + (system CAs plus the swarm and services roots). + + `null` means hive-c0re has no homeserver to provision against + and matrix provisioning no-ops. There is deliberately no + fallback compiled into the daemon: an address baked into the + binary is one that builds fine and then talks to the wrong + machine. + ''; + }; + gatewayHost = lib.mkOption { type = lib.types.nullOr lib.types.str; # `chat.` under the SWARM domain — both halves change: a swarm runs diff --git a/nix/module-eval/matrix-core.nix b/nix/module-eval/matrix-core.nix index 6302ba0a5..f0c312092 100644 --- a/nix/module-eval/matrix-core.nix +++ b/nix/module-eval/matrix-core.nix @@ -127,6 +127,18 @@ let && lib.hasInfix "exit 1" u.script && !(lib.hasInfix "bao login" u.script); } + { + # hive-c0re logs in with the hive's matrix token over this URL, and on a + # host without the homeserver that login crosses the network, so the + # default must be TLS all the way to the daemon's environment. + name = "hive-c0re provisions matrix over https from a hive that does not run it"; + ok = + !matrixElsewhere.services.hyperhive.deploy.matrix.enable + && matrixElsewhere.services.hyperhive.swarm.matrix.apiUrl == "https://chat.t.local" + && + matrixElsewhere.systemd.services.hive-c0re.environment.HIVE_MATRIX_API_URL + == "https://chat.t.local"; + } { # The legacy redirect reaches the hive domain's vhost from a glue # module, merged into the vhost hive-c0re declares, and only where the