diff --git a/docs/integrations/matrix.md b/docs/integrations/matrix.md index ccde7da6..497a1103 100644 --- a/docs/integrations/matrix.md +++ b/docs/integrations/matrix.md @@ -90,39 +90,54 @@ by default. Reaching it on 8448 needs either an explicit tuwunel bind to that port OR a reverse-proxy + `.well-known/matrix/server` delegation (the latter lives in `gateway.md::Discovery flow`). -## Provisioning flow (registration token) +## Provisioning flow (appservice) -Token-gated registration: hive-c0re holds the token, agents never -see it. The agent only receives the resulting `access_token`. +Registration is closed. Accounts are created by the hive's own +**appservice**: hive-c0re holds the appservice token, agents never see +it, and an agent only ever receives its own `access_token`. -1. **System activation** writes a 32-byte random hex token (64 - chars) to `services.hyperhive.deploy.matrix.registrationTokenFile` - (`/var/lib/hyperhive/matrix-register-token` by default), mode - `0600 root:root`, before any container start. Idempotent — only - writes when the file is missing or empty; always re-applies 0600 - (normalises any 0640 / world-readable carry-over from - pre-LoadCredential deployments). This runs at activation time - (not first container start) to dodge a race where nspawn creates - an empty file when the bind-mount target is missing and tuwunel - reads `registration_token_file=""`, rejecting every registration - until next restart. -2. **Read-only bind-mount** maps the host file into the tuwunel +The appservice has no URL (`url: null` in its registration), so the +homeserver never calls out to it and there is no service to run. What the +registration buys is an identity the homeserver recognises — which is why +no secret has to be equal on both sides of the wire, and why account +creation doesn't depend on registration being open to anyone who learns +a token. + +1. **System activation** mints a 32-byte random hex appservice token (64 + chars) at `/var/lib/hyperhive/matrix-appservice-token` and its + spec-required `hs_token` sibling, mode `0600 root:root`, then renders + the registration to + `/var/lib/hyperhive/matrix-appservice/hyperhive.yaml` (also `0600`). + The tokens are minted only when missing; the registration is + re-rendered every time, because the token file can be overwritten in + place by the swarm secret store and a registration naming a stale + token authenticates nobody. Runs at activation time, before any + container start, because the directory is bind-mounted and + nixos-container refuses to start when a bind source is missing. +2. **Read-only bind-mount** maps that directory into the tuwunel container at the same path. 3. **systemd `LoadCredential=`** inside the container copies the - bind-mounted file into - `/run/credentials/tuwunel.service/registration_token`, owned by - tuwunel's dynamic user with mode `0400`, at service start. The - host file stays `root:root 0600` — no `chown :tuwunel` / - `chmod 0640` / GID-pin gymnastics required. Keeps - `DynamicUser = true` + `PrivateUsers = true` intact. -4. tuwunel's `registration_token_file` points at the credentials - path, not the original bind-mount path. -5. **hive-c0re** uses the token to register each agent account via - the matrix-spec UIAA registration flow, persists the returned - `access_token` to `/matrix-token`. The agent's - matrix MCP client authenticates with that access_token and - never touches the shared registration token. -6. **hive-c0re restarts `hive-matrix-daemon`** for the agent + registration into + `/run/credentials/tuwunel.service/hyperhive-appservice.yaml`, owned by + tuwunel's dynamic user with mode `0400`, at service start. The host + file stays `root:root 0600` — no `chown :tuwunel` / `chmod 0640` / + GID-pin gymnastics required. Keeps `DynamicUser = true` + + `PrivateUsers = true` intact. +4. tuwunel's `appservice_dir` points at the credentials directory, not at + the bind-mount path. It reads only `.yaml`/`.yml` entries from there, + so the sibling credentials are invisible to it. The `.yaml` suffix on + the credential id is what makes this work. +5. **hive-c0re** reads the appservice token and creates each account with + one `POST /register` typed `m.login.application_service`, persisting + the returned `access_token` to `/matrix-token`. It never + mints the token itself: the value has to be the one the rendered + registration names, and only the nix side writes that. +6. **An account that exists but has lost its token file** is re-tokened + by an appservice `POST /login` — no password and no admin rights + involved. A stored-password login and an admin-room password reset + remain behind that, for accounts created before the appservice existed + or named outside its namespace. +7. **hive-c0re restarts `hive-matrix-daemon`** for the agent immediately after writing the token so the daemon picks up the new credential without waiting for a full container restart. If the restart fails (for example daemon not yet running on first boot) @@ -130,16 +145,74 @@ see it. The agent only receives the resulting `access_token`. (`hive-matrix-daemon.path` watching for `matrix-token` appearance) brings the daemon up on the same boot cycle anyway. +### The admin account, and why it needs no first-user luck + +`@hive:` is the appservice's own `sender_localpart`, which +the homeserver creates itself when it loads the registration — on a +zero-user database, inside startup, before the HTTP listener accepts +anything. Its **admin rights** then come from an explicit +`make_user_admin`, run by tuwunel's `admin_execute` in the same startup +and likewise before the listener — so a fresh hive has a joined, +power-level-100 admin on its first boot. + +This replaces a dependency on being the first account ever registered, +which was fragile in both directions: an appservice-created account is +excluded from that automatic grant by design, and on a homeserver that +already had users the rule never fired at all. + +Promotion can't be bootstrapped over the API, and that's upstream's +design rather than a gap: tuwunel only treats an admin-room message as a +command when its sender is already an admin. `admin_execute` is the one +lever with no sender to check. hive-c0re re-checks the result on every +sweep by reading the admin account's own joined-rooms list; if the rights +are missing it says so, names +`systemctl restart container@hive-matrix` as the fix, and carries on — +agent accounts, the hive Space and the chat room need no admin. + +
Upgrading a hive that used the registration token + +Nothing to do, and nothing to time. The activation script mints the +appservice token and renders the registration before the homeserver +restarts, so the first boot after the switch already has both halves. + +- **Existing accounts keep working.** An access token lives on the + device that minted it; removing the registration token touches no + device, no account and no session. `login_with_password` stays on, so + the password fallback is still there too. +- **Existing token files are honoured.** The per-agent sweep skips any + agent that already has a `matrix-token`, so no account is re-registered + and no session is displaced. +- **The admin account is already admin** on such a hive (it won the + first-user grant when the hive was new), so the startup promotion is a + no-op — upstream's `make_user_admin` short-circuits when the user is + already joined at power level 100. +- **`/var/lib/hyperhive/matrix-register-token` is left on disk**, read by + nothing. Delete it or leave it; neither does any harm. +- **`registrationTokenFile` is a removed option.** A config that still + sets it fails to evaluate with a message naming the appservice — a hive + that never set it (the default) is unaffected. +- **A swarm store holding the old `matrix/registration-token` path** is + no longer read at all. The value that matters now lives at + `matrix/appservice-token`, and `swarm-secret-publish` on the authelia + host mints and `put`s it there — the hive uses its locally minted + token only until the first successful read. See + [`../swarm/secrets.md`](../swarm/secrets.md) for how that mint stays + idempotent across runs. + +
+ Initial rollout settings: - `allow_federation = true` at the protocol level so swarms can be wired up later by extending `trustedServers` without a homeserver restart. `trusted_servers = []` keeps it effectively closed until you list peers. -- `allow_registration = true` (required for the token flow to - engage). The absent - `yes_i_am_very_very_sure_…_open_registration_…` flag keeps the - server closed to anyone without the token. +- `allow_registration = false`. tuwunel checks this flag only for + requests that arrive **without** an appservice token, so hive-c0re + provisions exactly as before and everyone else is refused. It's not a + hardening afterthought: with no registration token configured, + `allow_registration = true` makes tuwunel refuse to start unless + `yes_i_am_very_very_sure_…_open_registration_…` is also set. - `allow_encryption` — server-side E2EE switch, sourced from `services.hyperhive.swarm.matrix.allowEncryption` (**default `false`**, opt-in). Off by default because on the hive-internal homeserver the operator diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 24481b57..7c17abe4 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -132,32 +132,62 @@ when the system builds. The server names the offending file and refuses to run. ## Hive-level — one of each per hive -| secret | generated by | lives at | -| ---------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -| hive CA cert + key | `hive-tls.nix` first-boot unit | `/ca.pem`, `ca-key.pem` (`0600`) | -| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `/.pem` | -| matrix registration token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-register-token` (`0600`) | -| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/.secret` inside the forge container | -| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/.secret`, handed to tuwunel through `LoadCredential` | -| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `/secret` (`0600`) and `/client_id` (`0644`) | +| secret | generated by | lives at | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | +| hive CA cert + key | `hive-tls.nix` first-boot unit | `/ca.pem`, `ca-key.pem` (`0600`) | +| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `/.pem` | +| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) | +| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/.secret` inside the forge container | +| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/.secret`, handed to tuwunel through `LoadCredential` | +| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `/secret` (`0600`) and `/client_id` (`0644`) | Both delivery units wait for authelia's first boot to mint the secret — a bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom many layers from its cause. -The store's **first reader** is the matrix registration token, and it's worth +The store's **first reader** is the matrix appservice token, and it's worth saying why that one: it's an opaque 32-byte value with no second file and no format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so starting there would have meant debugging "can a reader authenticate and get bytes back" and "is authelia's file format right" at once, with an SSO outage as the failure mode. -`glue-matrix-bao-token.nix` fetches it and writes the file `hive-matrix.nix` -already reads, so the homeserver never learns the store exists. Every failure -path — no such key, sealed store, unreachable store, empty value — leaves the -locally minted token in place, so a hive with no store behaves exactly as it -did before. +`glue-matrix-bao-token.nix` fetches it, writes the file `hive-matrix.nix` +already reads, and then runs that module's own renderer to re-stamp the +appservice registration naming the token — so the homeserver never learns the +store exists. The re-render isn't housekeeping: the token is half an +agreement, and a registration carrying the previous value authenticates +nobody. Every failure path — no such key, sealed store, unreachable store, +empty value — leaves the locally minted token in place, so a hive with no store +behaves exactly as it did before. + +The store path is `swarm/hives//matrix/appservice-token`, and the +**producer is `swarm-secret-publish`** — the same unit that copies authelia's +OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s +it there, so the store is the source of truth and every hive converges on the +value it holds. The hive's own activation mint is still there, but it's a +first-boot fallback now: it fires only when the token file is absent, and the +reader overwrites whatever it produced. + +The mint is **idempotent by keeping its own record**, not by asking the store. +The publisher's grant is `create`/`update` under `swarm/hives/*` with no +`read` — write-only on purpose, so a file-copier can't recover every hive's +credentials — which means it can't check whether a hive already has a token. +Instead it keeps the value it minted under its `StateDirectory`, +`/var/lib/swarm-secret-publish/matrix-appservice-token/` (`0700` dir, +`0600` file), and mints only when that file is missing or empty. The `put` +itself runs every time: re-putting the same bytes changes nothing for any +reader, while a mint whose publish failed must not be left as a token this +host holds and no hive can reach. + +Lose the state directory and the next run mints once more and republishes. +That rotates the token, which readers pick up on their next start — nothing +that already registered breaks, because the token authenticates the +appservice rather than living inside any account it created. + +The path was `…/matrix/registration-token` while the homeserver still took a +shared registration secret; a value left at the old path is read by nothing. The **second reader** is the agent containers' queue credential: `glue-queue-agent-credential.nix` lands it as two files, the client secret and diff --git a/docs/swarm/sso.md b/docs/swarm/sso.md index cb38d00a..4054d120 100644 --- a/docs/swarm/sso.md +++ b/docs/swarm/sso.md @@ -168,9 +168,9 @@ Two consequences worth stating plainly: - **tuwunel re-reads its secret file on every OAuth exchange**, not only at startup, and its own sandboxing hides most paths from it. It gets the - file through `LoadCredential` for the same reason the registration token - does — that keeps `DynamicUser` and `PrivateUsers` intact, with no - host-side ownership arrangement to maintain. + file through `LoadCredential` for the same reason the appservice + registration does — that keeps `DynamicUser` and `PrivateUsers` intact, + with no host-side ownership arrangement to maintain. - **Matrix SSO lives inside the homeserver.** The client-server API is spoken by non-browser clients holding matrix access tokens — every agent's own daemon — as well as by federation, so the homeserver diff --git a/hive-c0re/src/matrix.rs b/hive-c0re/src/matrix.rs index 720c851d..b4c09558 100644 --- a/hive-c0re/src/matrix.rs +++ b/hive-c0re/src/matrix.rs @@ -1,12 +1,19 @@ -//! Optional matrix-tuwunel wiring: shared registration token (host) + -//! per-agent UIAA registration → `/matrix-token`. No-op +//! Optional matrix-tuwunel wiring: the hive's appservice identity (host) + +//! per-agent account creation → `/matrix-token`. No-op //! when the `hive-matrix` container isn't running, so operators who //! haven't flipped `services.hyperhive.deploy.matrix.enable = true` pay //! nothing. //! -//! See `docs/integrations/matrix.md::Provisioning flow (registration token)` for -//! the full UIAA round-trip, token-file shape, and host/container -//! bind-mount layout. +//! Accounts are created **as the hive's appservice**, not by presenting a +//! shared registration token in a UIAA flow. The difference that matters +//! here is not the round-trip count: an appservice token is an *identity* +//! the homeserver knows, so the secret never has to be the same on both +//! sides of the wire, and account creation does not depend on registration +//! being open to anyone who learns a token. +//! +//! See `docs/integrations/matrix.md::Provisioning flow (appservice)` for the +//! registration file's shape, how its token reaches both halves, and the +//! host/container bind-mount layout. use std::path::PathBuf; @@ -49,11 +56,8 @@ fn matrix_base() -> Result<&'static str> { this path should have been gated on matrix::is_present()", ) } -/// Length (bytes) of the random registration token. 32 raw bytes ⇒ -/// 64-char hex string; comfortable for a long-lived shared secret. -const REGISTER_TOKEN_BYTES: usize = 32; -/// HTTP timeout for registration round-trips. UIAA is two POSTs; even -/// the slow path should finish well inside this budget. +/// HTTP timeout for registration round-trips. Account creation is one +/// POST; even the slow path should finish well inside this budget. const HTTP_TIMEOUT_SECS: u64 = 10; /// Length (bytes) of the throwaway per-agent matrix password. Random /// 32-byte hex — agents never log in with the password (they @@ -61,10 +65,17 @@ const HTTP_TIMEOUT_SECS: u64 = 10; /// store it nowhere. const PASSWORD_BYTES: usize = 32; -/// Matrix localpart for the hive system admin account. Registered -/// before any agent account in [`ensure_all`] so it becomes the first -/// user on the homeserver — Conduit/tuwunel grants admin rights to the -/// first registered user automatically. Not an agent; has no state dir. +/// Matrix localpart for the hive system admin account. Not an agent; has +/// no state dir. +/// +/// Also the `sender_localpart` of the hive's appservice registration, +/// which is what creates this account on a homeserver that has never had +/// one: the homeserver creates a registration's sender user itself, at +/// startup, before it accepts a request. The account's *admin rights* +/// then come from the `admin_execute` promotion beside that registration +/// — see `nix/host-modules/hive-matrix.nix`, where this same literal +/// appears as `adminLocalpart`. **The two must match**; nothing wires an +/// override across. pub const HIVE_ADMIN_LOCALPART: &str = "hive"; /// Display name of the hive Space. Plain text, no special characters, so @@ -164,30 +175,36 @@ fn random_hex(n: usize) -> Result { Ok(hex) } -/// Ensure the registration token file exists; returns its contents. -/// Generates a fresh 32-byte hex token on first call (mode 0600, -/// root-only), then re-reads on subsequent calls. The hive-matrix -/// nixos module bind-mounts this file read-only into the tuwunel -/// container so the homeserver can authenticate registration requests -/// against the same secret hive-c0re holds. -pub fn ensure_register_token() -> Result { - use std::os::unix::fs::PermissionsExt; - let path = crate::paths::matrix_register_token(); - if let Ok(existing) = std::fs::read_to_string(&path) { - let trimmed = existing.trim().to_owned(); - if !trimmed.is_empty() { - return Ok(trimmed); - } - } - let token = random_hex(REGISTER_TOKEN_BYTES)?; - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).ok(); - } - std::fs::write(&path, format!("{token}\n")) - .with_context(|| format!("write registration token to {}", path.display()))?; - let _ = std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)); - tracing::info!(path = %path.display(), "matrix: generated registration token"); - Ok(token) +/// Read the hive's appservice token — the `as_token` of the registration +/// the homeserver loaded at boot. Every account this module creates is +/// authorised by it. +/// +/// **Reads, never mints**, unlike the registration token it replaced. +/// That token was the whole agreement, so whichever side wrote it first +/// was right; this one has a second half — the registration file naming +/// it, which only the nix side writes. A token minted here would be a +/// token the homeserver has never heard of, and the failure would surface +/// as every request being refused rather than as a missing file. +/// +/// # Errors +/// When the file is absent or empty. That means the host activation +/// script has not run on this generation yet; callers log it and leave +/// existing accounts alone rather than trying to proceed. +pub fn read_appservice_token() -> Result { + let path = crate::paths::matrix_appservice_token(); + std::fs::read_to_string(&path) + .ok() + .map(|s| s.trim().to_owned()) + .filter(|s| !s.is_empty()) + .with_context(|| { + format!( + "matrix appservice token not found at {} — it is minted by the \ + hive-matrix activation script, which also renders the registration \ + file naming it; deploy the hive-matrix module (or re-run \ + `nixos-rebuild switch`) before provisioning matrix users", + path.display() + ) + }) } /// Build the localpart of a matrix user id for `agent`. Matrix @@ -198,19 +215,20 @@ fn user_localpart(agent: &str) -> &str { agent } -/// Send a single registration POST and parse the response. Returns -/// `Ok((status, body))` on any successful HTTP round-trip (including -/// the expected 401 from the first UIAA leg); errors only on transport -/// failure. Body is the parsed JSON value — UIAA passes session + -/// flow state via JSON, never via headers. +/// Send the registration POST as the appservice and parse the response. +/// Returns `Ok((status, body))` on any completed HTTP round-trip +/// (including the `M_USER_IN_USE` 400 the caller treats as "already +/// exists"); errors only on transport failure. async fn register_post( client: &reqwest::Client, + as_token: &str, body: &serde_json::Value, ) -> Result<(StatusCode, serde_json::Value)> { let base = matrix_base()?; let url = format!("{base}/_matrix/client/v3/register"); let resp = client .post(&url) + .bearer_auth(as_token) .json(body) .send() .await @@ -233,24 +251,38 @@ pub fn random_password() -> Result { random_hex(PASSWORD_BYTES) } -/// Run the matrix-spec UIAA flow to register `agent` with the given -/// `password` and return the resulting access token. Two round-trips: -/// first POST elicits the 401 + session id, second POST supplies the -/// registration token in the `auth` block. If the homeserver returns -/// 200 on the first POST (no flow stages required — `allow_registration` -/// with no token), we take the access token directly. +/// Create the matrix account for `agent` as the hive's appservice and +/// return an access token for it. One round-trip: an appservice-typed +/// registration needs no UIAA stage at all, so there is no session to +/// carry and no shared secret to present. +/// +/// The account is created **by** the appservice but is an ordinary user +/// afterwards — it gets its own device and its own access token, and the +/// agent authenticates with that rather than with anything the hive +/// holds. The `as_token` never leaves the host. /// /// Caller picks the password: agents use [`random_password`] (throwaway /// — they auth by `access_token`), operators on the `hivectl` path /// supply their own so they can log into matrix web clients. +/// +/// # Errors +/// Propagates the homeserver's own body, which is what the +/// `M_USER_IN_USE` callers match on. A `M_EXCLUSIVE` body means the +/// localpart falls outside the appservice's namespace — the registration +/// file's `namespaces.users` regex is the place to look, not this call. async fn register_user( client: &reqwest::Client, agent: &str, - register_token: &str, + as_token: &str, password: &str, ) -> Result { let localpart = user_localpart(agent); - let initial = serde_json::json!({ + let body = serde_json::json!({ + // What makes this an appservice registration rather than an + // ordinary one. Without it the homeserver treats the request as a + // normal client's and asks for a UIAA flow — even holding the + // as_token. + "type": "m.login.application_service", "username": localpart, "password": password, // device_id stays stable across re-runs of ensure_user_for so @@ -259,37 +291,53 @@ async fn register_user( "initial_device_display_name": format!("hyperhive ({agent})"), "inhibit_login": false, }); - let (status, body) = register_post(client, &initial).await?; - // Some servers accept the first POST when allow_registration is - // open. Take the access_token + we're done. - if status.is_success() { - return extract_access_token(&body); - } - if status != StatusCode::UNAUTHORIZED { - anyhow::bail!("matrix: /register first leg HTTP {status}, body: {body}"); - } - let session = body["session"] - .as_str() - .with_context(|| format!("matrix: missing UIAA session in 401, body: {body}"))?; - let authed = serde_json::json!({ - "username": localpart, - "password": password, - "device_id": format!("hyperhive-{agent}"), - "initial_device_display_name": format!("hyperhive ({agent})"), - "inhibit_login": false, - "auth": { - "type": "m.login.registration_token", - "token": register_token, - "session": session, - }, - }); - let (status, body) = register_post(client, &authed).await?; + let (status, body) = register_post(client, as_token, &body).await?; if !status.is_success() { - anyhow::bail!("matrix: /register auth leg HTTP {status}, body: {body}"); + anyhow::bail!("matrix: /register as appservice HTTP {status}, body: {body}"); } extract_access_token(&body) } +/// Log in as an **existing** account using the hive's appservice token, +/// and return a fresh access token for it. No password involved: the +/// appservice is authorised for every localpart in its namespace, so it +/// can mint a session for one without knowing anything about the account. +/// +/// This is the recovery path that used to need a stored password or an +/// admin-room password reset — an account whose token file was lost is +/// re-tokened from the hive's own identity instead. The device id matches +/// [`register_user`]'s, so a re-login replaces that device's token rather +/// than accumulating devices. +async fn appservice_login(client: &reqwest::Client, as_token: &str, agent: &str) -> Result { + let base = matrix_base()?; + let url = format!("{base}/_matrix/client/v3/login"); + let body = serde_json::json!({ + "type": "m.login.application_service", + "identifier": { + "type": "m.id.user", + "user": user_localpart(agent), + }, + "device_id": format!("hyperhive-{agent}"), + "initial_device_display_name": format!("hyperhive ({agent})"), + }); + let resp = client + .post(&url) + .bearer_auth(as_token) + .json(&body) + .send() + .await + .context("matrix: POST /login as appservice")?; + let status = resp.status(); + let json = resp + .json::() + .await + .context("matrix: parse appservice /login response")?; + if !status.is_success() { + anyhow::bail!("matrix: appservice /login HTTP {status} for {agent}, body: {json}"); + } + extract_access_token(&json) +} + /// Pull `access_token` out of a successful /register response. fn extract_access_token(body: &serde_json::Value) -> Result { body["access_token"] @@ -637,11 +685,7 @@ async fn admin_room_reset_password( /// /// `client` is shared across the sweep so we build one reqwest /// connection pool for all agents rather than one per call. -pub async fn ensure_user_for( - client: &reqwest::Client, - name: &str, - register_token: &str, -) -> Result<()> { +pub async fn ensure_user_for(client: &reqwest::Client, name: &str, as_token: &str) -> Result<()> { use std::os::unix::fs::PermissionsExt; let agent = hive_types::Ident::parse(name) .map_err(|e| anyhow::anyhow!("invalid agent name {name:?}: {e}"))?; @@ -678,7 +722,7 @@ pub async fn ensure_user_for( } let password = random_password()?; - let access_token = match register_user(client, name, register_token, &password).await { + let access_token = match register_user(client, name, as_token, &password).await { Ok(token) => { // Successful registration — persist the password so we can // fall back to login if the token file is deleted later. @@ -694,8 +738,21 @@ pub async fn ensure_user_for( token } Err(reg_err) if reg_err.to_string().contains("M_USER_IN_USE") => { - // Account already exists — try to re-login with the stored password. - tracing::info!(%name, "matrix: user already exists, attempting login with stored password"); + // Account already exists and its token file was lost. The + // appservice can mint a session for any account in its + // namespace, so this needs no password and no admin rights — + // try it before the password paths below, which exist for + // accounts created before the appservice did (or named + // outside its namespace) and stay as the fallback. + tracing::info!(%name, "matrix: user already exists, logging in as the appservice"); + match appservice_login(client, as_token, name).await { + Ok(token) => return finish_user_provisioning(name, &token).await, + Err(e) => tracing::warn!( + %name, + error = ?e, + "matrix: appservice login failed; falling back to the stored password" + ), + } let pw_path = password_path(name); let stored = if let Some(pw) = std::fs::read_to_string(&pw_path) .ok() @@ -734,11 +791,19 @@ pub async fn ensure_user_for( Err(other) => return Err(other), }; + finish_user_provisioning(name, &access_token).await +} + +/// Persist a freshly-obtained agent access token and kick the agent's +/// matrix daemon. Shared by every way [`ensure_user_for`] can end up +/// holding a token — creation, appservice login, password login — so a +/// new recovery path cannot forget half of it. +async fn finish_user_provisioning(name: &str, access_token: &str) -> Result<()> { // Write the token via hive-priv (root helper): hive-c0re runs as the // unprivileged `hive-core` user and cannot write to agent-owned state // directories directly. hive-priv writes the file 0600 and chowns it // to the agent user so it is readable from inside the container. - crate::priv_client::write_agent_matrix_token(name, &access_token, None, None) + crate::priv_client::write_agent_matrix_token(name, access_token, None, None) .await .with_context(|| format!("matrix: write matrix-token for {name} via hive-priv"))?; tracing::info!(%name, "matrix: provisioned access token"); @@ -767,30 +832,30 @@ pub async fn ensure_user_for( /// behaviour. /// /// **Not idempotent** (unlike [`crate::forge::provision_user_token`]): the -/// matrix UIAA `/register` endpoint returns `M_USER_IN_USE` (HTTP 400) -/// on second call for the same localpart. Callers re-running this for -/// a known-existing matrix user should expect a hard error from this -/// fn and route to a password-reset path instead. +/// matrix `/register` endpoint returns `M_USER_IN_USE` (HTTP 400) on a +/// second call for the same localpart, appservice-authorised or not. +/// Callers re-running this for a known-existing matrix user should expect +/// a hard error from this fn and route to a password-reset path instead. pub async fn provision_user_token( client: &reqwest::Client, name: &str, - register_token: &str, + as_token: &str, password: &str, ) -> Result { - register_user(client, name, register_token, password).await + register_user(client, name, as_token, password).await } /// Per-agent matrix sync: ensure the agent has a matrix account + token. /// All operations are idempotent; failures are logged as warnings but /// don't abort the caller. -pub async fn sync_agent(client: &reqwest::Client, name: &str, register_token: &str) { - if let Err(e) = ensure_user_for(client, name, register_token).await { +pub async fn sync_agent(client: &reqwest::Client, name: &str, as_token: &str) { + if let Err(e) = ensure_user_for(client, name, as_token).await { tracing::warn!(%name, error = ?e, "matrix: ensure_user failed"); } } /// Standalone per-agent sync that handles its own setup: checks if -/// hive-matrix is present, reads the registration token, and builds +/// hive-matrix is present, reads the appservice token, and builds /// an HTTP client before delegating to [`sync_agent`]. Mirrors the /// setup in [`ensure_all`] so the rebuild path and the startup sweep /// stay equivalent. No-op when the matrix container is absent. @@ -798,10 +863,10 @@ pub async fn sync_agent_standalone(name: &str) { if !is_present() { return; } - let register_token = match ensure_register_token() { + let as_token = match read_appservice_token() { Ok(t) => t, Err(e) => { - tracing::warn!(%name, error = ?e, "matrix: ensure_register_token failed"); + tracing::warn!(%name, error = ?e, "matrix: read_appservice_token failed"); return; } }; @@ -815,22 +880,26 @@ pub async fn sync_agent_standalone(name: &str) { return; } }; - sync_agent(&client, name, ®ister_token).await; + sync_agent(&client, name, &as_token).await; } -/// Ensure the hive system admin matrix user exists and its token is -/// persisted at [`admin_token_path()`]. Must be called BEFORE -/// [`ensure_all`]'s agent loop so this account is the first to register -/// and becomes the homeserver admin automatically (Conduit/tuwunel: -/// first registered user = admin). +/// Ensure the hive system admin matrix user exists, that its token is +/// persisted at [`admin_token_path()`], and that it actually holds admin +/// rights. /// -/// Idempotent — skips when the token file already exists and is -/// non-empty. Does NOT promote the account via API (that requires -/// admin rights which this fn bootstraps); on a fresh homeserver the -/// first-registered rule fires automatically; on an existing homeserver -/// the operator must promote the account once via -/// `hivectl matrix promote-user hive` or the conduit admin room. -pub async fn ensure_admin_user(client: &reqwest::Client, register_token: &str) -> Result<()> { +/// **Nothing here depends on registration order any more.** The account +/// used to have to be the first ever registered, to win tuwunel's +/// automatic first-user grant — a rule that cannot fire for an +/// appservice-created account at all, and one that silently did nothing +/// on a homeserver that already had users. Admin rights now come from an +/// explicit `make_user_admin`: the `admin_execute` entry beside the +/// appservice registration performs it at homeserver startup, and +/// [`ensure_admin_rights`] checks the result and says so when it is +/// missing. +/// +/// Idempotent — skips the account work when the token file already exists +/// and is non-empty. +pub async fn ensure_admin_user(client: &reqwest::Client, as_token: &str) -> Result<()> { use std::os::unix::fs::PermissionsExt; let path = admin_token_path(); if path.exists() @@ -841,8 +910,7 @@ pub async fn ensure_admin_user(client: &reqwest::Client, register_token: &str) - return Ok(()); } let password = random_password()?; - let access_token = match register_user(client, HIVE_ADMIN_LOCALPART, register_token, &password) - .await + let access_token = match register_user(client, HIVE_ADMIN_LOCALPART, as_token, &password).await { Ok(token) => { let pw_path = password_path(HIVE_ADMIN_LOCALPART); @@ -857,20 +925,34 @@ pub async fn ensure_admin_user(client: &reqwest::Client, register_token: &str) - token } Err(reg_err) if reg_err.to_string().contains("M_USER_IN_USE") => { - tracing::info!("matrix: hive admin user already exists, re-logging in"); - let pw_path = password_path(HIVE_ADMIN_LOCALPART); - let stored = std::fs::read_to_string(&pw_path) - .ok() - .map(|s| s.trim().to_owned()) - .filter(|s| !s.is_empty()) - .with_context(|| { - format!( - "matrix: hive admin user exists but password missing at {} — \ - manual recovery: reset password via admin API or conduit admin room", - pw_path.display() - ) - })?; - login_user(client, HIVE_ADMIN_LOCALPART, &stored).await? + // The expected path, not an edge case: this account is the + // appservice's own `sender_localpart`, so the homeserver + // creates it when it loads the registration — before + // hive-c0re gets a chance to ask. An appservice login needs + // no password, which is just as well since an account the + // homeserver created has none. + tracing::info!("matrix: hive admin user already exists, logging in as the appservice"); + match appservice_login(client, as_token, HIVE_ADMIN_LOCALPART).await { + Ok(token) => token, + Err(e) => { + tracing::warn!(error = ?e, "matrix: appservice login for the hive admin failed; falling back to the stored password"); + let pw_path = password_path(HIVE_ADMIN_LOCALPART); + let stored = std::fs::read_to_string(&pw_path) + .ok() + .map(|s| s.trim().to_owned()) + .filter(|s| !s.is_empty()) + .with_context(|| { + format!( + "matrix: hive admin user exists, appservice login failed, and no \ + password is stored at {} — check that the registration file's \ + namespace covers @{HIVE_ADMIN_LOCALPART} and that the homeserver \ + loaded it", + pw_path.display() + ) + })?; + login_user(client, HIVE_ADMIN_LOCALPART, &stored).await? + } + } } Err(other) => return Err(other), }; @@ -884,9 +966,149 @@ pub async fn ensure_admin_user(client: &reqwest::Client, register_token: &str) - Ok(()) } +/// Check that the hive admin account holds admin rights, and repair it +/// through the admin room when it does not. +/// +/// Admin-ness in tuwunel is membership of the admin room, so that is what +/// this reads: the account's own joined-rooms list, which needs no +/// privileges to fetch. When the admin room is in it there is nothing to +/// do and nothing is sent — worth insisting on, because the repair is a +/// message in a room and this runs on every sweep. +/// +/// When it is absent, the repair is attempted anyway (a stale or failed +/// read is cheaper to retry than to reason about) and a failure is +/// reported rather than raised: agent accounts, the hive Space and the +/// chat room all work without an admin, so a hive with an unpromoted +/// admin is degraded, not broken. Only `hivectl matrix promote-user` / +/// `reset-password` need it. +/// +/// Returns whether the account holds admin rights now. A `false` has +/// already been logged, with what to do about it. +async fn ensure_admin_rights( + client: &reqwest::Client, + admin_token: &str, + server_name: &str, +) -> bool { + let room_id = match discover_admin_room_id(client, admin_token, server_name).await { + Ok(id) => id, + Err(e) => { + tracing::warn!(error = ?e, "matrix: cannot resolve #admins — not checking the hive admin's rights"); + return false; + } + }; + match joined_rooms(client, admin_token).await { + Some(rooms) if rooms.iter().any(|r| r == &room_id) => { + tracing::debug!("matrix: hive admin is in the admin room"); + return true; + } + Some(_) => { + tracing::warn!( + "matrix: hive admin is not in the admin room — attempting to promote it" + ); + } + None => tracing::debug!( + "matrix: could not read the hive admin's joined rooms; attempting to promote it" + ), + } + if let Err(e) = + promote_user_to_admin(client, admin_token, HIVE_ADMIN_LOCALPART, server_name).await + { + // The honest message. Promoting through the admin room requires + // being admin already, so the one lever that works on a + // homeserver with no admin at all is the `admin_execute` entry in + // the hive-matrix module — which runs at startup, which means a + // restart is the fix rather than another sweep. + tracing::warn!( + error = ?e, + "matrix: hive admin '@{HIVE_ADMIN_LOCALPART}' holds no admin rights. \ + The homeserver promotes it at startup (admin_execute in hive-matrix.nix), \ + so `systemctl restart container@hive-matrix` grants it. Agent accounts and \ + room provisioning are unaffected; `hivectl matrix promote-user` and \ + `reset-password` need it." + ); + return false; + } + tracing::info!("matrix: promoted the hive admin through the admin room"); + true +} + +/// The rooms `token`'s account has joined, or `None` when the list could +/// not be read. `None` is deliberately not an empty list: "no rooms" and +/// "no answer" lead to different decisions in [`ensure_admin_rights`]. +async fn joined_rooms(client: &reqwest::Client, token: &str) -> Option> { + let base = matrix_http()?; + let url = format!("{base}/_matrix/client/v3/joined_rooms"); + let resp = client.get(&url).bearer_auth(token).send().await.ok()?; + if !resp.status().is_success() { + return None; + } + let body = resp.json::().await.ok()?; + Some( + body["joined_rooms"] + .as_array()? + .iter() + .filter_map(|r| r.as_str().map(ToOwned::to_owned)) + .collect(), + ) +} + +/// Whether an admin-room reply says a `make-user-admin` succeeded. +/// +/// Three spellings, because the reply is prose and prose changes between +/// builds. tuwunel v1.9.0's is `" has been granted admin +/// privileges."` — which the original two patterns here (`done…`, +/// `made…admin`) do not match at all, so a promotion that had already +/// worked was reported as a 15-second timeout. The older spellings are +/// kept: a homeserver is not necessarily the version this was written +/// against. +fn is_make_admin_success(body: &str) -> Option<()> { + let lower = body.to_ascii_lowercase(); + let says_ok = lower.starts_with("done") + || lower.contains("granted admin privileges") + || (lower.contains("made") && lower.contains("admin")); + says_ok.then_some(()) +} + +#[cfg(test)] +mod is_make_admin_success_tests { + use super::is_make_admin_success; + + /// The reply tuwunel v1.9.0 actually sends + /// (`src/admin/user/make_user_admin.rs`). This is the case the + /// pre-existing matcher missed. + #[test] + fn tuwunel_1_9_grant_reply() { + let msg = "@hive:pr1ma.darkest.space has been granted admin privileges."; + assert_eq!(is_make_admin_success(msg), Some(())); + } + + #[test] + fn older_spellings_still_match() { + assert_eq!( + is_make_admin_success("Done: user is now an admin"), + Some(()) + ); + assert_eq!(is_make_admin_success("Made @x:y an admin"), Some(())); + } + + /// The control: an unrelated or failing reply must not read as + /// success, or a failed promotion returns Ok and the warning that + /// would have named it never fires. + #[test] + fn failures_and_noise_do_not_match() { + assert_eq!(is_make_admin_success("Command not recognised."), None); + assert_eq!(is_make_admin_success("User @x:y does not exist"), None); + } +} + /// Promote a user to homeserver admin via the Matrix admin room /// (`#admins:`). Sends `!admin users make-user-admin @:` as -/// @hive, polls for the bot's "Done:" success response. +/// @hive, polls for the bot's success reply. +/// +/// ⚠️ Requires the **sender** to be an admin already — tuwunel only +/// treats a message as a command when its sender is in the admin room. +/// So this promotes a *second* user; it cannot bootstrap the first one. +/// That is what the `admin_execute` entry in `hive-matrix.nix` is for. /// /// Goes through the admin room rather than a direct HTTP call because /// tuwunel implements parts of the Synapse admin API but not user @@ -907,14 +1129,7 @@ pub async fn promote_user_to_admin( server_name, &room_url, &command, - |body| { - let lower = body.to_ascii_lowercase(); - if lower.starts_with("done") || lower.contains("made") && lower.contains("admin") { - Some(()) - } else { - None - } - }, + is_make_admin_success, ) .await .with_context(|| { @@ -1637,10 +1852,14 @@ pub async fn ensure_all() -> bool { return true; } let mut ok = true; - let register_token = match ensure_register_token() { + // Loud and non-destructive: with no appservice token this sweep can + // create nothing, so it does nothing. Agents that already hold a + // token keep using it — their accounts and sessions are untouched by + // anything in here. + let as_token = match read_appservice_token() { Ok(t) => t, Err(e) => { - tracing::warn!(error = ?e, "matrix: ensure_register_token failed"); + tracing::warn!(error = ?e, "matrix: no appservice token; skipping the user sweep"); return false; } }; @@ -1656,10 +1875,12 @@ pub async fn ensure_all() -> bool { return false; } }; - // Provision hive admin user FIRST so it's the first registered - // account on a fresh homeserver (Conduit/tuwunel makes the first - // registered user admin automatically). - if let Err(e) = ensure_admin_user(&client, ®ister_token).await { + // The hive admin first, because everything below provisions THROUGH + // it (the Space, the chat room and every invite are sent with its + // token). Not, any more, so that it wins a first-registered-user + // grant: it holds admin rights by explicit promotion, checked in + // `provision_space` once the server name is known. + if let Err(e) = ensure_admin_user(&client, &as_token).await { tracing::warn!(error = ?e, "matrix: ensure_admin_user failed"); ok = false; } @@ -1672,7 +1893,7 @@ pub async fn ensure_all() -> bool { let Some(name) = c.strip_prefix(crate::lifecycle::AGENT_PREFIX) else { continue; }; - sync_agent(&client, name, ®ister_token).await; + sync_agent(&client, name, &as_token).await; agent_names.push(name.to_owned()); } @@ -1705,6 +1926,11 @@ async fn provision_space(client: &reqwest::Client, agent_names: &[String]) -> bo return false; } }; + // Before the rooms: the one thing in this sweep that is about the + // admin account itself rather than about what it provisions. + if !ensure_admin_rights(client, &admin_token, &server_name).await { + ok = false; + } let room_id = match ensure_hive_space(client, &admin_token).await { Ok(id) => id, Err(e) => { diff --git a/hive-c0re/src/paths.rs b/hive-c0re/src/paths.rs index 759f7eae..dcf16ef4 100644 --- a/hive-c0re/src/paths.rs +++ b/hive-c0re/src/paths.rs @@ -253,14 +253,15 @@ pub fn gateway_agents_conf() -> PathBuf { // `nix/host-modules/hive-c0re/default.nix` and `nix/host-modules/hive-ci.nix` — must match. pub const FORGE_CORE_TOKEN: &str = "/var/lib/hyperhive/forge-core-token"; -/// `matrix-register-token` — shared matrix registration token. -// nix: bind-mounted into the tuwunel/matrix container (hive-matrix.nix) — must match. -// Not operator-option-driven: `registrationTokenFile` is `internal` on the nix -// side, and an `assertions` entry there rejects any attempt to move it, so this -// literal can never diverge from it. +/// `matrix-appservice-token` — the `as_token` of the hive's appservice +/// registration, which authorises every account this daemon creates. +// nix: minted by the `hive-matrix-appservice` activation script in +// `nix/host-modules/hive-matrix.nix`, which renders it into the registration +// file the homeserver loads — must match. Read-only here on purpose: a token +// minted on this side would not be the one in that file. #[must_use] -pub fn matrix_register_token() -> PathBuf { - state_root().join("matrix-register-token") +pub fn matrix_appservice_token() -> PathBuf { + state_root().join("matrix-appservice-token") } /// `/run/hyperhive` — the runtime root (host admin socket + per-agent dirs). diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 67af8b76..547540d4 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -558,8 +558,8 @@ async fn handle_matrix_create_user( password: Option<&str>, ) -> Result { require_matrix_present()?; - let register_token = - crate::matrix::ensure_register_token().context("read matrix register token")?; + let as_token = + crate::matrix::read_appservice_token().context("read matrix appservice token")?; let client = matrix_http_client()?; let mut out = Vec::new(); if agent_exists(name)? { @@ -571,7 +571,7 @@ async fn handle_matrix_create_user( "matrix create-user: a password is for non-agent (operator) accounts only; '{name}' is an agent which authenticates via access_token" ); } - crate::matrix::ensure_user_for(&client, name.as_str(), ®ister_token) + crate::matrix::ensure_user_for(&client, name.as_str(), &as_token) .await .with_context(|| format!("matrix create-user {name}"))?; let path = Coordinator::agent_notes_dir(name).join("matrix-token"); @@ -585,7 +585,7 @@ async fn handle_matrix_create_user( let token = crate::matrix::provision_user_token( &client, name.as_str(), - ®ister_token, + &as_token, &effective_password, ) .await @@ -770,10 +770,10 @@ async fn handle_push_snapshot( async fn handle_matrix_sync_admin() -> Result { require_matrix_present()?; - let register_token = - crate::matrix::ensure_register_token().context("read matrix register token")?; + let as_token = + crate::matrix::read_appservice_token().context("read matrix appservice token")?; let client = matrix_http_client()?; - crate::matrix::ensure_admin_user(&client, ®ister_token) + crate::matrix::ensure_admin_user(&client, &as_token) .await .context("matrix sync-admin")?; let path = crate::matrix::admin_token_path(); diff --git a/hive-matrix-mcp/src/client.rs b/hive-matrix-mcp/src/client.rs index 867ced5d..f77619fe 100644 --- a/hive-matrix-mcp/src/client.rs +++ b/hive-matrix-mcp/src/client.rs @@ -5,10 +5,10 @@ //! to start sync. //! //! No OAuth dance / cross-signing setup (in contrast to damocles-daemon's -//! ccc.de connection): for the in-hive tuwunel the `registration_token` -//! UIAA flow already minted the token + user/device, hive-c0re just -//! handed us the bearer in a file. matrix-sdk's `restore_session` with -//! a constructed `MatrixSession` skips the login flow entirely. +//! ccc.de connection): for the in-hive tuwunel hive-c0re already minted +//! the token + user/device as the hive's appservice and handed us the +//! bearer in a file. matrix-sdk's `restore_session` with a constructed +//! `MatrixSession` skips the login flow entirely. //! //! E2EE is enabled via `with_encryption_settings(EncryptionSettings::default())`. //! Crypto keys are persisted in the sqlite store under `state_dir` diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index 0d491143..d89a1e7d 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -247,10 +247,29 @@ in [ "services" "hyperhive" "swarm" "matrix" "maxRequestSize" ] [ "services" "hyperhive" "deploy" "matrix" "maxRequestSize" ] ) - (lib.mkRenamedOptionModule - [ "services" "hyperhive" "swarm" "matrix" "registrationTokenFile" ] - [ "services" "hyperhive" "deploy" "matrix" "registrationTokenFile" ] - ) + # Both spellings of the registration token, removed rather than renamed: + # the homeserver no longer accepts a shared registration secret at all, + # so there is no path to point a definition at. Accounts are created by + # the hive's appservice, whose token lives at a fixed path this module + # deliberately does not let anyone move. + (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "matrix" "registrationTokenFile" ] '' + The matrix registration token is gone: hive-c0re creates accounts as the + hive's appservice instead, so there is no shared secret for tuwunel to + check. Remove this definition — nothing needs replacing, the appservice + token is minted and delivered automatically + (/var/lib/hyperhive/matrix-appservice-token, not operator-settable). + The old token file at /var/lib/hyperhive/matrix-register-token is read by + nothing now and can be deleted. See docs/integrations/matrix.md. + '') + (lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "matrix" "registrationTokenFile" ] '' + The matrix registration token is gone: hive-c0re creates accounts as the + hive's appservice instead, so there is no shared secret for tuwunel to + check. Remove this definition — nothing needs replacing, the appservice + token is minted and delivered automatically + (/var/lib/hyperhive/matrix-appservice-token, not operator-settable). + The old token file at /var/lib/hyperhive/matrix-register-token is read by + nothing now and can be deleted. See docs/integrations/matrix.md. + '') (lib.mkRenamedOptionModule [ "services" "hyperhive" "swarm" "matrix" "gui" "enable" ] [ "services" "hyperhive" "deploy" "matrix" "gui" "enable" ] diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index 4f2fe486..077c7b86 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -1,8 +1,9 @@ -# Glue: the matrix registration token comes from the secret store. +# Glue: the matrix appservice token comes from the secret store. # # The store's first reader, and deliberately a small one. It fetches an opaque -# 32-byte value and writes it where ./hive-matrix.nix already looks — the -# homeserver never learns the store exists, and its config is unchanged. +# 32-byte value and writes it where ./hive-matrix.nix already looks, then asks +# that module's own renderer to re-stamp the appservice registration naming it +# — the homeserver never learns the store exists, and its config is unchanged. # # ⚠️ Why this credential first. It has no second file and no format: authelia's # OIDC secret needs a `.secret` *and* a matching `.digest`, so shipping that @@ -47,12 +48,12 @@ let # own grant covers. The store's read policy grants `swarm/agents/*` and # `swarm/hives//*` and nothing else, so a path outside those is a # 403 rather than a miss, however correct it looks. `swarm-secret-client`'s - # `matrix::registration_token_path` builds the same string from the same + # `matrix::appservice_token_path` builds the same string from the same # pieces; this literal is the nix half of that one agreement. # # `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at # its own use of it: it is asserted set for every hyperhive host. - tokenPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/matrix/registration-token"; + tokenPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/matrix/appservice-token"; # A literal, not an option — ./hive-matrix.nix names its container # `containers.hive-matrix` directly and declares no `machine` to derive it @@ -70,7 +71,7 @@ in services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; systemd.services.swarm-bao-matrix-token = { - description = "fetch the matrix registration token from the swarm secret store"; + description = "fetch the matrix appservice token from the swarm secret store"; # Every one of these names a unit that exists only where the store runs. # `Requires=` on an absent unit fails the job outright, so the ordering is # conditional even though the read is not: off-host there is nothing local @@ -176,8 +177,20 @@ in fi umask 077 - printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.registrationTokenFile)} - chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.registrationTokenFile)} + printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + + # Re-stamp the registration file from the token just written. The + # token is half an agreement — the registration the homeserver loads + # has to carry the same value — so writing the file and stopping + # would leave the homeserver authenticating hive-c0re against + # whatever activation put there: a 401 on every request, naming + # nothing. Unconditional rather than on-change, because this unit + # has no way to know what the registration currently says. + # + # hive-matrix's own renderer rather than a `printf` here, so the + # registration's shape has one home. + ${deployCfg.matrix.appserviceRegistrationScript} ''; }; }; diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 81e3c350..9e57b512 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -15,7 +15,7 @@ # that is correct rather than degraded. The publisher runs on the authelia # host and authelia mints on its first boot, so "nothing at that path yet" is # the ordinary early state of a swarm. Nothing here writes a local stand-in: -# unlike a matrix registration token there is no such thing as a locally valid +# unlike a matrix appservice token there is no such thing as a locally valid # OIDC client secret, so a placeholder would turn a hive that cannot connect # into one that is refused, which reaches the agent as a timeout. # diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index 8d087b0a..53aa9c15 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -40,22 +40,155 @@ let # login into an intermittent one. matrixSecretPath = "/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret"; - # Single source for the registration-token path: the option's `default`, - # its `config`-level `mkDefault` self-definition, and the assertion that - # rejects a moved path all read this same binding rather than repeating - # the string literal. See `registrationTokenFile`'s own comment below. - registrationTokenPath = "/var/lib/hyperhive/matrix-register-token"; - - # ⚠️ tuwunel does NOT read the path above directly, and this indirection + # ⚠️ tuwunel does NOT read the host path directly, and this indirection # is not ceremony. Upstream's own words: "under systemd the path must be # visible to the service after sandboxing (ReadWritePaths / ProtectHome), # typically by placing the file under /etc/tuwunel/" — which this # container has no writable etc for. `LoadCredential` is the answer - # already in use two units below for the registration token, and for the + # already in use for the appservice registration below, and for the # same reason: it keeps `DynamicUser=true` + `PrivateUsers=true` intact # with no host-side chown or GID pinning. matrixSecretCredential = "/run/credentials/tuwunel.service/oidc_client_secret"; + # How this hive creates matrix accounts: an appservice registration whose + # `url` is null. Null is a legal `url` (ruma's `Registration` types it + # `Option`), and it is the whole point — with no URL the homeserver + # never calls out, so there is no HTTP service to run and no daemon to + # operate. What the registration delivers is the `as_token`: hive-c0re's + # standing authority to create, and log in as, the accounts named by the + # namespace below, with no shared registration secret in the picture. + appserviceId = "hyperhive"; + + # The appservice's own user, and deliberately the hive admin account. + # Loading a registration CREATES its `sender_localpart` user when absent + # (tuwunel `src/service/appservice/mod.rs`), on a zero-user database, + # inside `Services::start()` — before the HTTP listener accepts anything. + # That is what lets the `admin_execute` promotion below land on the very + # first boot of a fresh homeserver, instead of depending on hive-c0re + # racing to register the first account and win the auto-admin grant. + # + # ⚠️ Must equal `matrix::HIVE_ADMIN_LOCALPART` in hive-c0re, which derives + # it independently with nothing wiring an override across — same + # agreement, and same reason for saying so, as the token path below. + adminLocalpart = "hive"; + + # The `as_token`, and the `hs_token` the spec requires alongside it. Both + # minted by the render script below, mode 0600; the `as_token` is the one + # hive-c0re reads and the one the swarm secret store overwrites (see + # `glue-matrix-bao-token.nix`). The `hs_token` authenticates the homeserver + # TO the appservice, which with `url = null` is nobody — it exists because + # the registration format requires it. + # + # `appserviceTokenPath` is the single source for the option's `default`, + # its `config`-level `mkDefault` self-definition, and the assertion that + # rejects a moved path. See `appserviceTokenFile`'s own comment below. + appserviceTokenPath = "/var/lib/hyperhive/matrix-appservice-token"; + appserviceHsTokenPath = "/var/lib/hyperhive/matrix-appservice-hs-token"; + + # The registration file, and the directory that holds it. A directory + # rather than a `[global.appservice.]` stanza because a stanza's + # `as_token` would be a nix literal, and a nix literal is a world-readable + # store path — the same rule that sends the OIDC client secret through a + # runtime file. `docs/swarm/secrets.md` has the general form of it. + appserviceDir = "/var/lib/hyperhive/matrix-appservice"; + appserviceRegistrationPath = "${appserviceDir}/${appserviceId}.yaml"; + + # ⚠️ tuwunel does not read the host path directly, for exactly the reason + # given for the two secrets above: the file is 0600 root-owned and the + # homeserver runs under `DynamicUser=true`. `LoadCredential` copies it to a + # 0400 dynamic-user-owned path as root, before the sandbox and that user + # exist. + # + # So `appservice_dir` points at the credentials directory ITSELF. That is + # safe rather than clever: tuwunel's loader takes only `.yaml`/`.yml` + # entries from the directory and skips every other file, so the sibling + # credentials (the OIDC secret) are invisible to it. A credential id is a + # free-form filename, which is what lets ours end in `.yaml`. + appserviceCredentialId = "${appserviceId}-appservice.yaml"; + appserviceCredentialDir = "/run/credentials/tuwunel.service"; + + # Every local user this hive may provision — agents, the hive admin, and + # the operator accounts `hivectl matrix create-user` makes, which is the + # whole matrix localpart charset. + # + # ⚠️ Anchored deliberately: tuwunel compiles a namespace into a `RegexSet` + # and asks it for a MATCH, not a full match, so an unanchored + # `@[a-z0-9]+:this.example` also matches `@x:this.example.evil.test`. + # + # Non-exclusive deliberately: an exclusive namespace does not widen what + # the appservice may do, it narrows what everything ELSE may do — an SSO + # login adopting an existing account, or `!admin users create-user`, would + # start failing with `M_EXCLUSIVE`. The appservice needs the right to + # create these names, not a monopoly on them. + appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$"; + + # Mint the tokens if they are absent, then render the registration from + # whatever they now hold. **One script with two callers** — the activation + # script below and `glue-matrix-bao-token.nix`, which overwrites the + # `as_token` with the swarm store's copy and has to re-render afterwards. + # + # ⚠️ The `as_token` mint here is the FIRST-BOOT FALLBACK, not the normal + # route. `swarm-secret-publisher.nix` mints one per hive swarm-side and + # publishes it to the store; the reader above overwrites this file before + # the homeserver's container starts, so what a hive actually runs on is the + # swarm's value. This keeps a hive that has never reached the store working + # — and it fires only when the file is absent, so it never clobbers one the + # store delivered. (`hs_token` has no swarm half: it is how the homeserver + # authenticates to *us*, so it stays hive-local and is minted here for real.) + # A second copy of the registration's shape would be a second source of + # truth for a format whose mismatch is silent: the homeserver would load a + # registration naming a token nobody holds, and every request hive-c0re + # makes would come back 401 naming nothing. + # + # Re-rendering unconditionally is the point rather than thoroughness: "the + # token file exists" does not mean "the registration carries what is in + # it". + appserviceRegistrationScript = pkgs.writeShellApplication { + name = "hive-matrix-appservice-registration"; + runtimeInputs = [ pkgs.coreutils ]; + text = '' + # Both the tokens and the rendered registration are secrets; 077 + # covers every file this script creates rather than each one + # separately. + umask 077 + mkdir -p ${lib.escapeShellArg appserviceDir} + for f in ${lib.escapeShellArg appserviceTokenPath} ${lib.escapeShellArg appserviceHsTokenPath}; do + if [ ! -s "$f" ]; then + head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$f" + echo >> "$f" + echo "hive-matrix: generated appservice token at $f" + fi + chmod 0600 "$f" + done + + # Read into shell variables and emitted with `printf`, a shell + # builtin: a token passed as an argument to a real command would land + # in that process's argv, which is world-readable for its lifetime. + asToken="$(cat ${lib.escapeShellArg appserviceTokenPath})" + hsToken="$(cat ${lib.escapeShellArg appserviceHsTokenPath})" + + # The quoted heredoc keeps the regex's own `$` and `\` out of the + # shell's hands; the YAML single quotes keep them out of YAML's. + { + cat <<'REGISTRATION' + id: ${appserviceId} + url: null + sender_localpart: ${adminLocalpart} + rate_limited: false + namespaces: + users: + - exclusive: false + regex: '${appserviceUserRegex}' + aliases: [] + rooms: [] + REGISTRATION + printf 'as_token: %s\nhs_token: %s\n' "$asToken" "$hsToken" + } > ${lib.escapeShellArg appserviceRegistrationPath} + chmod 0600 ${lib.escapeShellArg appserviceRegistrationPath} + chmod 0700 ${lib.escapeShellArg appserviceDir} + ''; + }; + # Format-locked by tuwunel, not chosen here: the callback host must point # directly at the matrix server and the path is fixed at # `/_matrix/client/unstable/login/sso/callback/`. Built once @@ -443,40 +576,59 @@ in ''; }; - registrationTokenFile = lib.mkOption { + appserviceTokenFile = lib.mkOption { type = lib.types.path; internal = true; - default = registrationTokenPath; + default = appserviceTokenPath; description = '' - Host path to a file containing the matrix registration token - tuwunel reads to authorise new-account creation. The token is - generated automatically by `hive-c0re` on first boot (32-byte - random hex, mode 0600) and is bind-mounted read-only into the - tuwunel container at the same path. Agents never see this - token — hive-c0re uses it to provision per-agent accounts - and the agent only receives the resulting `access_token`. + Host path to a file containing this hive's matrix appservice + token (`as_token`) — the identity `hive-c0re` creates and logs + into accounts with. Minted automatically on first activation + (32-byte random hex, mode 0600) and rendered into the + appservice registration the homeserver loads at boot. Agents + never see it; an agent only ever receives its own + `access_token`. Not operator-settable — `hive-c0re`'s Rust side derives this - same path independently (`paths::matrix_register_token()`) with - nothing wiring an override across, so a moved path used to - desync the two silently. An externally-managed token is - delivered by writing into *this* fixed path instead of moving it — - see `glue-matrix-bao-token.nix`, which fetches from the swarm - secret store and overwrites this file in place. + same path independently (`paths::matrix_appservice_token()`) + with nothing wiring an override across, so a moved path desyncs + the two silently. An externally-managed token is delivered by + writing into *this* fixed path instead of moving it — see + `glue-matrix-bao-token.nix`, which fetches from the swarm secret + store, overwrites this file in place, and re-renders the + registration that names it. Enforced by an `assertions` entry below rather than `readOnly`: the `config` block gives this option its own `mkDefault` - definition (lowest priority) so any real override — an operator's, - or the pre-rename `swarm.matrix.registrationTokenFile` shim's — - still resolves cleanly instead of crashing eval with nixpkgs' - generic "read-only, set multiple times" message; the assertion - then names the actual problem. (`readOnly` was the first attempt — - atlas caught that it only rejects a *second* definition, - so with nothing else defining the option a lone override sailed - through silently, same desync as before with a lock that wasn't - locked. Fixed by defining the value here instead of leaving it on - `default` alone, but a proper `assertions` message beat re-adding - `readOnly` on top once the option had a real definition either way.) + definition (lowest priority) so a real override still resolves + cleanly instead of crashing eval with nixpkgs' generic + "read-only, set multiple times" message; the assertion then + names the actual problem. (`readOnly` was the first attempt on + this option's predecessor — atlas caught that it only rejects a + *second* definition, so with nothing else defining the option a + lone override sailed through silently, same desync as before + with a lock that wasn't locked. Fixed by defining the value here + instead of leaving it on `default` alone, but a proper + `assertions` message beat re-adding `readOnly` on top once the + option had a real definition either way.) + ''; + }; + + appserviceRegistrationScript = lib.mkOption { + type = lib.types.path; + internal = true; + default = "${appserviceRegistrationScript}/bin/hive-matrix-appservice-registration"; + defaultText = lib.literalMD "the module's own registration renderer"; + description = '' + The script that mints the appservice tokens when absent and + (re-)renders the registration file from them. An option only so + that `glue-matrix-bao-token.nix` can run the same one after + overwriting the token with the swarm store's copy, rather than + carrying a second copy of the registration's shape. + + Not operator-settable, and not a hook: replacing it means + deciding what the homeserver's appservice registration says, + which is this module's job. ''; }; @@ -538,11 +690,11 @@ in config = lib.mkIf deployCfg.matrix.enable { # The option's own value, defined explicitly rather than left on its - # bare `default` — `mkDefault` so a real override (an operator's, or - # the pre-rename shim's) still resolves cleanly rather than crashing - # eval; the `assertions` entry below is what actually rejects it, with - # a message naming the reason. See that option's own comment above. - services.hyperhive.deploy.matrix.registrationTokenFile = lib.mkDefault registrationTokenPath; + # bare `default` — `mkDefault` so a real override still resolves + # cleanly rather than crashing eval; the `assertions` entry below is + # what actually rejects it, with a message naming the reason. See that + # option's own comment above. + services.hyperhive.deploy.matrix.appserviceTokenFile = lib.mkDefault appserviceTokenPath; # Matrix's own gateway surface: the sub-domain vhost, the name the # hive resolver answers for, and the Accept-header map that vhost's @@ -723,19 +875,17 @@ in # mkDefault above lets an override resolve instead of crashing # eval — this is the actual rejection, with a message that names # the real fix instead of nixpkgs' generic conflicting-definition - # text. Covers both the current path and the pre-rename - # `swarm.matrix.registrationTokenFile` shim in one check, since - # both land on the same merged option. - assertion = deployCfg.matrix.registrationTokenFile == registrationTokenPath; + # text. + assertion = deployCfg.matrix.appserviceTokenFile == appserviceTokenPath; message = '' - services.hyperhive.deploy.matrix.registrationTokenFile is fixed at - ${registrationTokenPath} and cannot be moved — hive-c0re's Rust + services.hyperhive.deploy.matrix.appserviceTokenFile is fixed at + ${appserviceTokenPath} and cannot be moved — hive-c0re's Rust side derives this same path independently and has no way to learn an override, so moving it desyncs the two silently instead of loudly. - Integrating an externally-managed registration token? Deliver it - by writing into ${registrationTokenPath} instead of pointing this + Integrating an externally-managed appservice token? Deliver it + by writing into ${appserviceTokenPath} instead of pointing this option elsewhere — see glue-matrix-bao-token.nix, which does exactly that from the swarm secret store. ''; @@ -781,8 +931,8 @@ in # permanent stall presenting as "matrix is broken", several layers from # its cause. # - # The registration token above dodges that with an activation script - # that pre-creates the file. ⚠️ That dodge is NOT available here: + # The appservice registration dodges that with an activation script + # that renders the file first. ⚠️ That dodge is NOT available here: # tuwunel requires the secret file to exist *and be non-empty*, so a # zero-byte placeholder would satisfy the bind mount and then stop the # homeserver from starting. @@ -890,16 +1040,21 @@ in '' ); - system.activationScripts.hive-matrix-register-token = lib.stringAfter [ "var" ] '' - tokenFile=${lib.escapeShellArg (toString deployCfg.matrix.registrationTokenFile)} - if [ ! -s "$tokenFile" ]; then - mkdir -p "$(dirname "$tokenFile")" - head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$tokenFile" - echo >> "$tokenFile" - echo "hive-matrix: generated registration token at $tokenFile" - fi - # Re-apply 0600 (normalises any pre-LoadCredential carry-over). - chmod 0600 "$tokenFile" + # Mint the appservice tokens and render the registration, before any + # container start. + # + # An activation script rather than a unit: the directory below is + # bind-mounted into the container and nixos-container refuses to start + # when a bind source is missing, so this has to have run first. + # Activation is what runs before the container both on a switch and on + # every boot. (The registration token this replaced used an activation + # script for the same reason, and additionally to dodge nspawn creating + # an empty file at a missing bind target — which tuwunel then read as + # "no token", refusing every registration until the next restart. A + # missing registration file is not silent in that way: the homeserver + # fails its appservice load loudly.) + system.activationScripts.hive-matrix-appservice = lib.stringAfter [ "var" ] '' + ${deployCfg.matrix.appserviceRegistrationScript} ''; containers.hive-matrix = { @@ -910,11 +1065,15 @@ in extraFlags = [ "--link-journal=host" ]; # Shared host netns — agents reach tuwunel at localhost:. privateNetwork = false; - # Read-only bind of the host-managed registration token; tuwunel + # Read-only bind of the host-managed appservice registration; tuwunel # reads it via systemd LoadCredential below (not directly). + # + # The directory, not the file inside it: the registration is + # re-rendered rather than edited, and binding the file would pin the + # inode the container saw when it started. bindMounts = { - ${deployCfg.matrix.registrationTokenFile} = { - hostPath = deployCfg.matrix.registrationTokenFile; + ${appserviceDir} = { + hostPath = appserviceDir; isReadOnly = true; }; } @@ -1008,13 +1167,50 @@ in # trustedServers keeps it effectively closed. allow_federation = true; trusted_servers = deployCfg.matrix.trustedServers; - # Token-gated registration. The absent - # `yes_i_am_very_very_sure_…_open_registration_…` flag - # keeps the server closed to anyone without the token. - allow_registration = true; - # LoadCredential below copies the host file into a - # 0400 dynamic-user-owned path; tuwunel reads from there. - registration_token_file = "/run/credentials/tuwunel.service/registration_token"; + # Nobody registers themselves here. Accounts are created by + # the hive's appservice, which this flag does not gate — + # tuwunel checks it only for requests that arrive WITHOUT an + # appservice token, so hive-c0re provisions exactly as before + # and everyone else is refused outright. + # + # ⚠️ Not a hardening afterthought: `allow_registration = true` + # with no registration token configured makes tuwunel REFUSE + # TO START (it demands + # `yes_i_am_very_very_sure_…_open_registration_…` instead). So + # dropping the token and leaving this true is not a lax + # homeserver, it is a homeserver that does not boot. + allow_registration = false; + + # Where the hive's appservice registration is read from — the + # credentials directory, for the reasons at + # `appserviceCredentialDir`'s own definition. Loaded inside + # `Services::start()`, before the listener accepts anything. + appservice_dir = appserviceCredentialDir; + + # The zero-user bootstrap, and the only thing here that needs + # no account to already exist. These run after startup and + # BEFORE the HTTP listener, with no sender and no permission + # check — which is what makes them the one lever that can + # promote the hive admin on a homeserver where nobody is admin + # yet. The appservice registration above creates + # `@${adminLocalpart}` as its sender user moments earlier; + # this joins it to the admin room at power level 100. + # + # Idempotent by upstream's own guard: `make_user_admin` + # short-circuits when the user is already joined at 100, so a + # hive that has had an admin for months emits nothing. + admin_execute = [ + "users make-user-admin @${adminLocalpart}:${effectiveServerName}" + ]; + + # ⚠️ Load-bearing, not tidiness. An `admin_execute` command + # that fails aborts startup outright when this is false — so + # the one boot where the promotion cannot work (a homeserver + # that has no `@${adminLocalpart}` and no appservice user yet, + # e.g. a registration file that arrived late) would take the + # homeserver down with it rather than converging on the next + # start. + admin_execute_errors_ignore = true; # Server-side E2EE is opt-in (default off); the agent matrix # client always supports decryption regardless. allow_encryption = cfg.allowEncryption; @@ -1075,7 +1271,10 @@ in # host-side chown :tuwunel / GID-pin gymnastics needed. # See `man systemd.exec` → LoadCredential. systemd.services.tuwunel.serviceConfig.LoadCredential = [ - "registration_token:${toString deployCfg.matrix.registrationTokenFile}" + # The credential id carries a `.yaml` suffix on purpose: + # `appservice_dir` above names this very directory, and tuwunel + # takes only `.yaml`/`.yml` entries from it. + "${appserviceCredentialId}:${appserviceRegistrationPath}" # Same mechanism, second secret. tuwunel re-reads this file on # every OAuth exchange, not just at startup, so it has to # outlive the unit's start — a credentials path does. diff --git a/nix/host-modules/swarm-secret-publisher.nix b/nix/host-modules/swarm-secret-publisher.nix index b373bf58..6f1a51e6 100644 --- a/nix/host-modules/swarm-secret-publisher.nix +++ b/nix/host-modules/swarm-secret-publisher.nix @@ -1,6 +1,7 @@ -# The unit that copies authelia's minted OIDC client secrets into the swarm's -# secret store, so whoever needs one can read it there: a hive its agents' -# credential, a swarm service its own. +# The unit that puts swarm-level secrets into the swarm's secret store, so +# whoever needs one can read it there: a hive its agents' credential, a swarm +# service its own, a hive its matrix appservice token. The OIDC secrets it only +# copies; the appservice token it MINTS, having had no swarm-side producer. # # ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes # into the store even when the service runs beside authelia, because its @@ -14,19 +15,19 @@ # why: the plaintext's other reader lives in a **different container**, and # containers that share this host's network namespace still have separate # filesystem roots, so "the host is the only place both trees are addressable". -# A delivery step therefore runs on the host that mints — not on the store's -# host, and not on the reading hive's. +# A delivery step therefore runs on the minting host, not the store's or the +# reading hive's. # # ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite -# every hive's policy and login role; a unit whose entire job is copying one -# file has no business holding it. ./swarm-bao.nix grants this one -# `create`/`update` under the hive prefix and nothing else. +# every hive's policy and login role; a unit that writes a handful of secrets +# has no business holding it. ./swarm-bao.nix grants this one `create`/`update` +# under the hive prefix and nothing else — no `read`, which is why the mint +# below keeps its own record of what it published. # # ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every # argument is world-readable in /proc for the life of the call — the value is -# passed as `@` and read by bao itself. (`/knowledge/secret-hygiene.md`: -# "a path keeps the secret out of the store, and reading it into a shell -# variable puts it straight into argv".) +# passed as `@` and read by bao itself. Same rule governs the mint: it +# redirects into a file and never holds the token in a shell variable. { pkgs, lib, @@ -71,6 +72,21 @@ let hyperhiveCfg.swarm.grafana.oidc.clientId hyperhiveCfg.swarm.otel.clientId ]; + + # Where this unit keeps the appservice tokens it minted, and the whole reason + # a re-publish is idempotent. The store cannot be that record: ./swarm-bao.nix + # grants this principal `create`/`update` and deliberately no `read`, so "does + # this hive already have a token?" is a question it can only ask locally. + # Without the file every run would mint a fresh value and rotate the swarm's + # token for no reason — and a rotation is not free, because the homeserver + # only reloads the registration naming it when its container restarts. + # + # `StateDirectory=` below is what creates it, so the name there is relative + # and this unit owns the tree. Losing it mints once more and republishes; + # readers pick that up on their next start, and nothing that already + # registered breaks — the token authenticates the appservice, it does not + # live inside any account it created. + matrixTokenDir = "/var/lib/swarm-secret-publish/matrix-appservice-token"; in { options.services.hyperhive.deploy.swarm-secret-publisher = { @@ -82,10 +98,12 @@ in Publish the OIDC client secrets this host mints into the swarm's secret store, so a hive that does not run authelia can read its agents' credential and a swarm service can read its own — from - wherever it runs, this host included. + wherever it runs, this host included. Also mints each hive's matrix + appservice token, which has no other swarm-side producer, and + publishes it the same way. - Defaults to whether this host mints them, which is the only half of - the question that is a property of *this* host. + Defaults to whether this host mints the OIDC secrets, which is the + only half of the question that is a property of *this* host. ⚠️ Deliberately **not** `deploy.bao.enable`. That asks whether the store stands here, and a publisher beside a remote store, holding a @@ -95,11 +113,12 @@ in identity's job (see `baoClientCertFile`), not this option's. Turning it off leaves every hive but this one without its agents' - credential, the swarm's Grafana without any login at all, and its - collector pushing unauthenticated — each secret has exactly one route - and this is the producer's end of it. So the honest reason to set it - false is a deployment delivering those secrets by some other - mechanism it owns. + credential, the swarm's Grafana without any login at all, its + collector pushing unauthenticated, and every hive falling back to the + appservice token its own first boot minted — so two hives never agree + on one. Each secret has exactly one route and this is the producer's + end of it, so the honest reason to set it false is a deployment + delivering those secrets by some other mechanism it owns. ''; }; @@ -143,7 +162,7 @@ in }; systemd.services.swarm-secret-publish = { - description = "publish minted OIDC client secrets to the swarm secret store"; + description = "publish the swarm's minted secrets to the swarm secret store"; wantedBy = [ "multi-user.target" ]; after = [ "network.target" ]; path = [ @@ -159,6 +178,12 @@ in TimeoutStartSec = 60; Restart = "on-failure"; RestartSec = 30; + # The minted appservice tokens, see `matrixTokenDir` above. Named + # relative to /var/lib because that is the form systemd creates; the + # mode is narrowed from systemd's 0755 default because the directory + # listing is the roster of hives and every file in it is a secret. + StateDirectory = "swarm-secret-publish/matrix-appservice-token"; + StateDirectoryMode = "0700"; }; environment = { BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}"; @@ -185,6 +210,7 @@ in published=0 skipped=0 + tokens=0 ${lib.concatMapStringsSep "\n" (hive: '' src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"} @@ -228,7 +254,43 @@ in fi '') serviceClientIds} - echo "published $published client secret(s), skipped $skipped" + # The matrix appservice token, per hive. Unlike the two loops above + # there is nothing to copy: authelia never minted this one, and each + # homeserver's own host minted its own — which is exactly why two hives + # never converged on a value the swarm has to agree on. The mint moves + # here because this unit already holds a store identity and already + # writes under the hive prefix; ../host-modules/hive-matrix.nix keeps + # its activation mint as the first-boot fallback for a hive that has + # never reached the store. + # + # 32 bytes of urandom as lowercase hex, the shape that file has + # generated all along. Written with no trailing newline, unlike that + # file: what goes into the store is the value, and + # ./glue-matrix-bao-token.nix adds the newline the hive's token file + # carries. + ${lib.concatMapStringsSep "\n" (hive: '' + src=${lib.escapeShellArg "${matrixTokenDir}/${hive}"} + if [ ! -s "$src" ]; then + # The subshell keeps the umask local — the loops above rely on + # bao's own file handling and should not inherit one. + ( umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$src" ) + echo "minted a matrix appservice token for ${hive}" >&2 + fi + # `value=@$src` for the same reason as both loops above: bao opens + # the file itself, so the token is never an argument of this process. + # + # Put on every run, not only on the run that minted — a put of the + # value already there changes nothing for any reader, while a mint + # whose publish failed must not be left as a token this host holds + # and no hive can reach. The `-s` test above is what keeps the VALUE + # stable across those runs: rotating on every boot would hand the + # homeserver a registration its running container does not carry. + bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/matrix/appservice-token"} \ + value=@"$src" + tokens=$((tokens + 1)) + '') hiveNames} + + echo "published $published client secret(s), skipped $skipped, and $tokens matrix appservice token(s)" ''; }; }; diff --git a/nix/module-eval.nix b/nix/module-eval.nix index a2682c6f..7c819347 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -116,24 +116,45 @@ let ]; }; - # The homeserver's turn to split. All eight host-side options are set through - # their pre-rename paths — including `gui.enable`, whose value is deliberately - # the opposite of its default so the definition has to actually land, and both - # packages, stubbed to a derivation neither option defaults to for the same - # reason. All eight so that dropping any single shim entry fails the eval, not - # just the ones the assertions read. + # The homeserver's turn to split. All seven remaining host-side options are + # set through their pre-rename paths — including `gui.enable`, whose value is + # deliberately the opposite of its default so the definition has to actually + # land, and both packages, stubbed to a derivation neither option defaults to + # for the same reason. All seven so that dropping any single shim entry fails + # the eval, not just the ones the assertions read. + # + # `registrationTokenFile` used to be the eighth: both spellings of it are now + # `mkRemovedOptionModule`, exercised by `matrixRemovedToken` below instead — + # setting it here would make every case that reads this fixture fail on that + # one removed option. matrixOldPath = hive { deploy.matrix.enable = true; swarm.matrix.openFirewall = true; swarm.matrix.trustedServers = [ "matrix.example.invalid" ]; swarm.matrix.maxRequestSize = 31457280; - swarm.matrix.registrationTokenFile = "/etc/matrix/register.token"; swarm.matrix.gui.enable = false; swarm.matrix.package = pkgs.emptyDirectory; swarm.matrix.gui.package = pkgs.emptyDirectory; swarm.matrix.sso.clientSecretFile = "/etc/matrix/oidc.secret"; }; + # Moving the appservice token, which desyncs hive-c0re's independently + # derived path from the homeserver's. Its own fixture because the assertion + # it trips would otherwise fire for every case reading `matrixOldPath`. + matrixMovedAppserviceToken = hive { + deploy.matrix.enable = true; + deploy.matrix.appserviceTokenFile = "/etc/matrix/as.token"; + swarm.matrix.sso.clientSecretFile = "/etc/matrix/oidc.secret"; + }; + + # A config still naming the removed registration token, through the + # pre-rename spelling — the one an old deployment is most likely to carry. + matrixRemovedToken = hive { + deploy.matrix.enable = true; + swarm.matrix.registrationTokenFile = "/etc/matrix/register.token"; + swarm.matrix.sso.clientSecretFile = "/etc/matrix/oidc.secret"; + }; + # The queue's callout identity, fourth split slice. `autoGenerateCallout` is # left FALSE on purpose: that is what makes the seed paths the thing deciding # `responderConfigured`, so the assertion below is about the seeds rather @@ -840,33 +861,41 @@ let } { # Third split, and the one whose readers were hardest to see: the - # registration token is read only through a `let` alias in another + # appservice token is read only through a `let` alias in another # module, so no full path names it anywhere. Both arms read a rendered # effect — the host firewall and the container's bind-mount table — so a # rename that resolves but stops reaching the module still fails. - name = "a config written against the pre-rename matrix paths still opens the port and mounts the token"; + name = "a config written against the pre-rename matrix paths still opens the port and mounts the appservice registration"; ok = let ports = matrixOldPath.networking.firewall.allowedTCPPorts; httpPort = matrixOldPath.services.hyperhive.swarm.matrix.httpPort; in builtins.elem httpPort ports - && matrixOldPath.containers.hive-matrix.bindMounts ? "/etc/matrix/register.token"; + && matrixOldPath.containers.hive-matrix.bindMounts ? "/var/lib/hyperhive/matrix-appservice"; } { - # registrationTokenFile lost its override capability entirely - # (bao-delivered secrets don't need one — glue-matrix-bao-token.nix - # already writes into the fixed path instead of moving it), unlike its - # five siblings in the same rename. `matrixOldPath` above proves the - # override still *resolves* (mkDefault, not a crash) — this proves it - # also gets *rejected*, by a named assertion rather than nixpkgs' generic - # conflicting-definition text. Reads `.assertions` directly (cheap: a - # list of `{assertion; message;}`, not `system.build.toplevel`) rather - # than forcing a real build just to observe a boolean. - name = "overriding registrationTokenFile (even via the pre-rename shim) trips a named assertion, not a silent desync"; + # appserviceTokenFile has no override capability (a store-delivered + # secret doesn't need one — glue-matrix-bao-token.nix writes into the + # fixed path instead of moving it), unlike its siblings in the same + # rename. The fixture proves the override still *resolves* (mkDefault, + # not a crash); this proves it also gets *rejected*, by a named + # assertion rather than nixpkgs' generic conflicting-definition text. + # Reads `.assertions` directly (cheap: a list of `{assertion; + # message;}`, not `system.build.toplevel`) rather than forcing a real + # build just to observe a boolean. + name = "moving appserviceTokenFile trips a named assertion, not a silent desync"; ok = lib.any ( - a: !a.assertion && lib.hasInfix "registrationTokenFile" a.message - ) matrixOldPath.assertions; + a: !a.assertion && lib.hasInfix "appserviceTokenFile" a.message + ) matrixMovedAppserviceToken.assertions; + } + { + # The removal has to SAY so. A config carrying the old option through + # an upgrade is the common case, and nixpkgs' bare "option does not + # exist" would name neither what replaced the token nor where to read + # about it. + name = "a config still setting registrationTokenFile is refused with a message naming the appservice"; + ok = lib.any (a: !a.assertion && lib.hasInfix "appservice" a.message) matrixRemovedToken.assertions; } { # Reads the DELIVERY UNIT, not the options: `responderConfigured` gates @@ -1448,6 +1477,41 @@ let lib.hasInfix "secret/swarm/hives/h1/queue/agent" s && lib.hasInfix "secret/swarm/hives/h2/queue/agent" s; } + { + # The producer's end of the read `glue-matrix-bao-token.nix` already did. + # Both hives are asserted for the reason the queue case above gives: a + # publisher that knew one name rather than the roster would pass on a + # single-hive fixture and strand the second hive ever added — which is + # the two-hives-never-converge shape this slice exists to close. + name = "the publisher mints an appservice token for every hive and writes it to that hive's matrix path"; + ok = + let + s = secretPublisherHere.systemd.services.swarm-secret-publish.script; + in + lib.hasInfix "secret/swarm/hives/h1/matrix/appservice-token" s + && lib.hasInfix "secret/swarm/hives/h2/matrix/appservice-token" s + && lib.hasInfix "/dev/urandom" s; + } + { + # What makes a re-publish idempotent. This principal is granted + # `create`/`update` and no `read`, so it cannot ask the store whether a + # hive already has a token — with nowhere to keep one, every run would + # mint a fresh value and rotate the swarm's token. A state directory is + # that somewhere, and nothing else in this unit needs one, so its absence + # means exactly this. + # + # The second arm is the mint's own guard: the state file is only written + # when it is missing or empty. Dropping that test leaves a unit that + # still has a state directory and still rotates on every boot. + name = "the publisher keeps the tokens it minted, and mints only when it holds none"; + ok = + let + u = secretPublisherHere.systemd.services.swarm-secret-publish; + in + lib.hasInfix "matrix-appservice-token" (u.serviceConfig.StateDirectory or "") + && u.serviceConfig.StateDirectoryMode or null == "0700" + && lib.hasInfix "if [ ! -s \"$src\" ]" u.script; + } { # A property of the SET, not of one unit: both of these authenticate by # certificate, and `BAO_CLIENT_CERT` is transport rather than identity, so @@ -1518,7 +1582,7 @@ let s = baoWithMatrix.systemd.services.swarm-bao-matrix-token.script; in lib.hasInfix "secret/swarm/hives/" s - && lib.hasInfix "/matrix/registration-token" s + && lib.hasInfix "/matrix/appservice-token" s # The shape it used to have: `matrix` where a principal kind belongs, # which no grant covers. && !(lib.hasInfix "secret/swarm/matrix/" s); @@ -2471,7 +2535,7 @@ let # The store's first reader. Its unit belongs to the pairing, not to # either service: matrix must not learn the store exists, and the store # must not know who reads it. - name = "a store deployed beside the homeserver fetches its registration token"; + name = "a store deployed beside the homeserver fetches its appservice token"; ok = baoWithMatrix.systemd.services ? swarm-bao-matrix-token; } { diff --git a/swarm-secret-client/src/matrix.rs b/swarm-secret-client/src/matrix.rs index 69a130c7..9a0cda5b 100644 --- a/swarm-secret-client/src/matrix.rs +++ b/swarm-secret-client/src/matrix.rs @@ -26,18 +26,25 @@ pub fn account_path(agent: &str, account: &str) -> Result { Ok(format!("{prefix}/matrix/{account}")) } -/// The path holding `hive`'s matrix registration token. +/// The path holding `hive`'s matrix appservice token (`as_token`). /// /// Keyed per **hive**, not per agent, like [`crate::queue::agent_client_path`] /// and unlike [`account_path`] above: one homeserver admits one hive's -/// accounts, so the token that creates them is the hive's. +/// accounts, so the identity that creates them is the hive's. +/// +/// ⚠️ Renamed from `registration-token` along with what it holds: the +/// homeserver no longer accepts a shared registration secret at all, so a +/// value still stored under the old path would be read by nothing. A hive +/// whose store has only the old path falls back to its locally minted +/// token — see `glue-matrix-bao-token.nix` — so the rename degrades rather +/// than breaks, but the store needs a fresh `put` to take effect again. /// /// # Errors /// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`, /// which is what keeps one hive's name from addressing another hive's secret. -pub fn registration_token_path(hive: &str) -> Result { +pub fn appservice_token_path(hive: &str) -> Result { let prefix = principal_prefix(Kind::Hive, hive)?; - Ok(format!("{prefix}/matrix/registration-token")) + Ok(format!("{prefix}/matrix/appservice-token")) } /// What an account's path holds: the token, plus the homeserver it belongs to. @@ -80,28 +87,29 @@ mod tests { } #[test] - fn the_registration_token_lands_under_the_hive_prefix_the_grant_covers() { + fn the_appservice_token_lands_under_the_hive_prefix_the_grant_covers() { // Spelled out for the same reason as above, and with a second job here: // the read policy grants `secret/data/swarm/hives//*`, so this // string is what makes the path reachable at all. assert_eq!( - registration_token_path("pr1ma").expect("a plain name is legal"), - "swarm/hives/pr1ma/matrix/registration-token" + appservice_token_path("pr1ma").expect("a plain name is legal"), + "swarm/hives/pr1ma/matrix/appservice-token" ); } #[test] - fn the_registration_token_is_not_a_top_level_namespace() { - // The path it used to hold. `Kind` is a closed set and `matrix` is not - // one of its members, so a path with `matrix` as the second segment is - // outside every grant — which is how it came to 403 on every read. - let p = registration_token_path("pr1ma").expect("legal"); + fn the_appservice_token_is_not_a_top_level_namespace() { + // The path its predecessor used to hold. `Kind` is a closed set and + // `matrix` is not one of its members, so a path with `matrix` as the + // second segment is outside every grant — which is how it came to 403 + // on every read. + let p = appservice_token_path("pr1ma").expect("legal"); assert!(!p.starts_with("swarm/matrix/"), "{p}"); } #[test] fn a_traversal_in_the_hive_name_is_refused() { - let e = registration_token_path("../beta").expect_err("a traversal is not"); + let e = appservice_token_path("../beta").expect_err("a traversal is not"); assert!(matches!(e, Error::PathSegment { kind: "hive", .. }), "{e}"); }