diff --git a/docs/matrix.md b/docs/matrix.md new file mode 100644 index 00000000..ef05a61b --- /dev/null +++ b/docs/matrix.md @@ -0,0 +1,199 @@ +# hive-matrix + +Private Matrix homeserver (matrix-tuwunel — the conduwuit +successor) wrapped in a nixos-container, plus optional fluffychat-web +client at `matrix./`. Configured via +`services.hyperhive.matrix.*`; vhost routing lives in +[`gateway.md`](gateway.md). + +## Container shape + +Same shape as [`gateway.md::hive-forge container shape`](gateway.md): + +- Container name `hive-matrix` (not `h-*`) so c0re's lifecycle + scanner ignores it; operator manages via the standard + `nixos-container` CLI. +- Keeps hive-matrix from fighting any `services.matrix-*` the + operator already runs on the host — separate systemd namespace, + separate state dir. +- Container shares the host network namespace + (`privateNetwork = false`) so agents reach tuwunel at + `http://localhost:` without extra plumbing — the + nixos-container is here for state + systemd-unit isolation, not + network isolation. +- Persistent state at + `/var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/` + survives container restart / host reboot. To wipe, destroy the + container. + +## Identity vs API listener: `serverName` vs `gatewayHost` + +Two distinct hostnames: + +- **`serverName`** — matrix-spec `server_name`, embedded + *irrevocably* in every `@user:` and `!room:` + identifier minted on this homeserver. Cannot be changed later + without abandoning every account and chat history. Defaults to the + bare `services.hyperhive.domain` per mara on #660; clients + auto-discover the actual API endpoint via the + `.well-known/matrix/{client,server}` routes the hive-gateway serves + at that domain. +- **`gatewayHost`** — the API listener hostname, where the gateway's + matrix vhost proxies `/_matrix/*` to tuwunel. Defaults to + `matrix.` (sub-domain shape per mara on + #749:9609). Set to `null` to skip the gateway vhost (tuwunel stays + direct on `httpPort`). + +**Breaking change** (#660): `serverName` used to default to +`matrix.${services.hyperhive.domain}`. Existing homeservers must set +the option explicitly to preserve their pre-#660 user / room IDs +before rebuilding. The default flipped because the bare hive-domain +makes for cleaner matrix IDs and `.well-known` delegation hides the +sub-domain from the user-facing identifier. + +## Default-closed firewall + +`openFirewall` defaults to `false` (#651, secure-by-default): the +homeserver is reachable from the host + every agent container via +loopback either way (shared netns), so the firewall hole only +matters for access from *outside* the host. Flip to `true` when +announcing the homeserver to other hives or when an external matrix +client needs to reach the client-server API directly. + +**Breaking change** (#651): used to default to `true`. Operators +relying on external reach must add +`services.hyperhive.matrix.openFirewall = true;` before rebuilding. + +Federation port 8448 is intentionally not opened here — tuwunel +serves the federation API on the same `httpPort` as client-server +by default. Reaching it on 8448 needs either an explicit tuwunel +bind to that port OR a reverse-proxy + `.well-known/matrix/server` +delegation (the latter lives in `gateway.md::Discovery flow`). + +## Provisioning flow (registration token) + +Token-gated registration: hive-c0re holds the token, agents never +see it. The agent only receives the resulting `access_token`. + +1. **System activation** writes a 32-byte random hex token (64 + chars) to `cfg.registrationTokenFile` + (`/var/lib/hyperhive/matrix-register-token` by default), mode + `0600 root:root`, before any container start. Idempotent — only + writes when the file is missing or empty; always re-applies 0600 + (normalises any 0640 / world-readable carry-over from + pre-LoadCredential deployments). This runs at activation time + (not first container start) to dodge the argus #565 race where + nspawn creates an empty file when the bind-mount target is + missing and tuwunel reads `registration_token_file=""` rejecting + every registration until next restart. +2. **Read-only bind-mount** maps the host file into the tuwunel + container at the same path. +3. **systemd `LoadCredential=`** inside the container copies the + bind-mounted file into + `/run/credentials/tuwunel.service/registration_token`, owned by + tuwunel's dynamic user with mode `0400`, at service start. The + host file stays `root:root 0600` — no `chown :tuwunel` / + `chmod 0640` / GID-pin gymnastics required (per iris on #644 + 8043, dropping the shape #649 originally shipped with). Keeps + `DynamicUser = true` + `PrivateUsers = true` intact. +4. tuwunel's `registration_token_file` points at the credentials + path, not the original bind-mount path. +5. **hive-c0re** uses the token to register each agent account via + the matrix-spec UIAA registration flow, persists the returned + `access_token` to `/matrix-token`. The agent's + matrix MCP client authenticates with that access_token and + never touches the shared registration token. + +Initial rollout settings (#548): + +- `allow_federation = true` at the protocol level so swarms can be + wired up later by extending `trustedServers` without a homeserver + restart. `trusted_servers = []` keeps it effectively closed + until peers are listed. +- `allow_registration = true` (required for the token flow to + engage). The absent + `yes_i_am_very_very_sure_…_open_registration_…` flag keeps the + server closed to anyone without the token. +- `allow_encryption = false` per operator call (#548). E2EE + re-enabling tracked at #551. + +## Assertion rationale + +Two `config.assertions` entries fail eval early rather than ship +surprising behaviour: + +- **`hyperhiveDomain != null || cfg.serverName != null`** (mara on + #548) — `server_name` is embedded into every user / room ID + irrevocably; we refuse to spawn the homeserver with a bogus + server_name we can never change later. +- **`cfg.gatewayHost != ""`** (argus 🟡 on #764) — same footgun as + `forge.domain` (#754). Empty string renders `.`-shaped + garbage in both nginx `server_name` (treated as wildcard + catch-all, surprising) and `/etc/hosts` (invalid entry). `null` + is the right opt-out shape; empty string is rejected explicitly. + +## fluffychat-web build fixes (#685) + +`pkgs.fluffychat-web` ships from `flutter341.buildFlutterApplication`, +which has two upstream gaps for fluffychat's web target: + +- The dart web-worker entry point (`web/native_executor.dart`) isn't + compiled — `buildFlutterApplication` only runs `flutter build web` + on the main entry. +- `native_imaging`'s C source isn't built — emscripten isn't a + flutter-builder native build input. + +Both fixed in `nix/modules/hive-matrix.nix` via two derivations: + +- **`fluffychat-web-imaging`** builds `Imaging.{js,wasm}` from the + `native_imaging` C source via `pkgs.emscripten`. Source comes + from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging` + — already in the build closure of the flutter app, so no parallel + hash pin and version auto-syncs with nixpkgs bumps. Build closure + is ~3.6 GiB (emscripten LLVM); runtime closure is just the two + output files. `dontConfigure = true` because cmake runs inside + `js/Makefile` via `emcmake cmake`, not at the package root. The + build script needs `HOME` + `EM_CACHE` writable for emscripten's + on-demand sysroot build (libc, libc++ → wasm). +- **`fluffychat-web-fixed`** is `pkgs.fluffychat-web` plus a + `postInstall` patch that (a) compiles `web/native_executor.dart` + via `dart compile js` (dart from the flutter341 closure, no + incremental cost) and (b) installs `fluffychat-web-imaging`'s + outputs into `$out`. + +Two non-obvious fixes from review history: + +- **`make -C js`** instead of `cd js; make` (argus 🟡 on #697 v2) + — keeps the build-phase pwd at the source root so `installPhase` + doesn't have to know about the cd. Robust against future + reorders / `dontBuild`. +- **`web/native_executor.dart`** as a build-CWD-relative path, + *not* `$src/web/...` (#685 / #733 fixup) — `dart`'s + `package_config.json` walk-up needs to hit + `buildFlutterApplication`'s pub-get output (`.dart_tool/` in the + build CWD). Walking up from a read-only `$src/` store path finds + no `.dart_tool/` and errors with "Couldn't resolve the package + 'matrix'". Confused two PRs. + +Drop both derivations when nixpkgs's flutter builder grows worker ++ emcc support upstream. + +Mount point is `matrix./` (#772); upstream `--base-href "/"` +is correct at sub-domain root, no override. + +## Sequencing history + +- #548 — initial rollout (federation enabled, registration enabled, + E2EE disabled) +- #565 — first-boot empty-token race fix → activation-time token + generation +- #644 / iris 8043 / #649 — registration token ownership shape + (dropped chown/GID-pin; LoadCredential delivers as 0400 dynamic-user) +- #651 — `openFirewall` default flipped to `false` +- #660 — `serverName` default flipped to bare hive-domain (was + `matrix.`) +- #685 / #697 / #733 — fluffychat-web build fixes (Imaging emscripten, + native_executor dart worker, build-CWD path) +- #736 — fluffychat config.json inline JSON at sub-domain root +- #749 / #764 — gateway sub-domain shape verdict +- #772 / #775 — fluffychat hops from `/matrix/` to `matrix./` diff --git a/nix/modules/hive-matrix.nix b/nix/modules/hive-matrix.nix index d4964534..f0480736 100644 --- a/nix/modules/hive-matrix.nix +++ b/nix/modules/hive-matrix.nix @@ -9,26 +9,16 @@ let hyperhiveDomain = config.services.hyperhive.domain; effectiveServerName = if cfg.serverName != null then cfg.serverName else hyperhiveDomain; - # Three files are missing from `pkgs.fluffychat-web` because - # `flutter341.buildFlutterApplication` doesn't run the dart - # web-worker compile pass + doesn't run the native_imaging emscripten - # build (#685). `fluffychat-web-imaging` below builds the latter from - # source via `pkgs.emscripten`; the worker compile is inline in - # `fluffychat-web-fixed.postInstall`. Drop both when nixpkgs's - # flutter builder grows worker + emcc support upstream. + # fluffychat-web build fixes: nixpkgs's `flutter341.buildFlutterApplication` + # skips the dart web-worker compile + the emscripten native_imaging + # build. Two derivations below cover both. Full rationale (why + # passthru.pubspecLock.dependencySources, why `dontConfigure`, why + # `make -C js`, why build-CWD-relative dart path): docs/matrix.md:: + # fluffychat-web build fixes. - # `Imaging.{js,wasm}` built from `native_imaging`'s C source via - # emscripten. Source comes from - # `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources` so - # there's no parallel hash pin — version auto-syncs with nixpkgs - # bumps. Build closure +~3.6 GiB (emscripten LLVM); runtime closure - # is just the two output files. fluffychat-web-imaging = pkgs.stdenv.mkDerivation { pname = "fluffychat-web-imaging"; version = pkgs.fluffychat-web.passthru.pubspecLock.dependencyVersions.native_imaging; - - # The pub-cache derivation that fluffychat-web's flutter build uses. - # Already in the build closure; no `fetchurl` or own hash pin. src = pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging; nativeBuildInputs = with pkgs; [ @@ -38,21 +28,18 @@ let jq ]; - # cmake config runs inside `js/Makefile` (via `emcmake cmake`) — - # skip the default `configurePhase` which would try to invoke - # cmake against the package root and fail (no CMakeLists at top). + # cmake runs inside js/Makefile via `emcmake cmake`; the default + # configurePhase would invoke cmake at the package root (no + # CMakeLists) and fail. dontConfigure = true; buildPhase = '' runHook preBuild - # emscripten needs HOME + a writable cache dir for its sysroot - # build (libc, libc++, etc. compiled to wasm on demand). + # emscripten on-demand sysroot build needs writable HOME + cache. export HOME=$TMPDIR export EM_CACHE=$TMPDIR/.emscriptencache mkdir -p $EM_CACHE - # `make -C js` keeps the build phase pwd at the source root so - # installPhase doesn't have to know about the cd (argus 🟡 on - # PR #697 v2 — robust against future reorders / `dontBuild`). + # `make -C js` keeps pwd at source root for the installPhase. make -C js Imaging.js Imaging.wasm runHook postBuild ''; @@ -66,71 +53,37 @@ let ''; meta = with pkgs.lib; { - description = "Imaging.js + Imaging.wasm built from the native_imaging dart package for fluffychat-web (#685)"; + description = "Imaging.js + Imaging.wasm built from the native_imaging dart package for fluffychat-web"; homepage = "https://pub.dev/packages/native_imaging"; license = licenses.agpl3Plus; }; }; - # `pkgs.fluffychat-web` with #685's three missing files patched in - # via postInstall. Mount point is `matrix./` (#772); upstream - # `--base-href "/"` is correct at sub-domain root, no override. fluffychat-web-fixed = pkgs.fluffychat-web.overrideAttrs (old: { - # `dart` from the flutter341 closure (already pulled, no - # incremental closure cost) so we can compile the web-worker - # entry point that buildFlutterApplication skips. + # dart from the flutter341 closure (already pulled, no incremental + # cost) to compile the web-worker entry point. nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [ pkgs.flutter341.dart ]; - postInstall = - (old.postInstall or "") - + '' - # `web/...` is relative to build CWD so dart's package_config - # walk-up hits buildFlutterApplication's pub-get output (#685 - # / #733 fixup — `$src/web/...` would walk up to a read-only - # store path with no `.dart_tool/`). - ${pkgs.flutter341.dart}/bin/dart compile js \ - -o $out/native_executor.js \ - web/native_executor.dart + postInstall = (old.postInstall or "") + '' + # `web/...` is BUILD-CWD-relative (not `$src/...`) so dart's + # package_config walk-up hits buildFlutterApplication's + # pub-get output `.dart_tool/`. + ${pkgs.flutter341.dart}/bin/dart compile js \ + -o $out/native_executor.js \ + web/native_executor.dart - install -m 644 ${fluffychat-web-imaging}/Imaging.js $out/Imaging.js - install -m 644 ${fluffychat-web-imaging}/Imaging.wasm $out/Imaging.wasm - ''; + install -m 644 ${fluffychat-web-imaging}/Imaging.js $out/Imaging.js + install -m 644 ${fluffychat-web-imaging}/Imaging.wasm $out/Imaging.wasm + ''; }); in { - # Private Matrix homeserver (matrix-tuwunel — the official conduwuit - # successor) for hyperhive agents, wrapped in a nixos-container so it - # doesn't fight any existing `services.matrix-*` the operator may - # already run on the host. Same shape as `nix/modules/hive-forge.nix`: - # shared host netns (`privateNetwork = false`) so agents reach it at - # `http://localhost:` (or via the configured server_name - # for federation), nixos-container only here for state + systemd-unit - # isolation. - # - # Container name `hive-matrix` (not `h-*`) so the lifecycle scanner - # ignores it; operator manages via the standard `nixos-container` CLI. - # - # Persistent state at `/var/lib/nixos-containers/hive-matrix/var/lib/ - # matrix-tuwunel/` (survives container restart / host reboot). To - # wipe, destroy the container. - # - # Initial rollout (#548): federation enabled (needed for multi-hive - # swarms; trusted_servers starts empty so no actual federation traffic - # leaves until peers are explicitly listed), registration enabled via - # a `registration_token_file` known only to hive-c0re (so agents can't - # self-register without going through the coordinator), e2ee disabled - # per operator call (tracked for follow-up at #551). - # - # Provisioning model (matches `nix/modules/hive-forge.nix` shape): - # hive-c0re generates a 32-byte random `registration_token` on first - # boot, writes it to `/var/lib/hyperhive/matrix-register-token` (mode - # 0600, root-only), and bind-mounts that file read-only into the - # tuwunel container at the same path so tuwunel can read it via - # `registration_token_file`. hive-c0re then uses the token to register - # each agent account via the matrix-spec UIAA registration flow, and - # persists the returned `access_token` to `/matrix-token` - # so the agent's matrix MCP client can authenticate without ever - # seeing the shared registration token. + # Private matrix-tuwunel homeserver wrapped in a nixos-container, + # optional fluffychat-web client at matrix./. Container shape, + # serverName vs gatewayHost split, provisioning flow (registration + # token + LoadCredential), assertion rationale, initial rollout + # settings: docs/matrix.md. Vhost map + discovery flow + tuning + # knobs: docs/gateway.md. options.services.hyperhive.matrix = { enable = lib.mkOption { @@ -322,12 +275,10 @@ in }; config = lib.mkIf cfg.enable { - # mara on #548: "there is no default, but it is required. add - # assertion." — fail eval with a helpful message rather than - # spawning a homeserver with a bogus server_name we can never - # change later. `services.hyperhive.domain` is host-wide; matrix derives - # the server_name from it (or from `cfg.serverName` if the - # operator wants to override). + # serverName must exist (mara on #548 — irrevocably embedded in + # user/room IDs); gatewayHost may not be "" (argus 🟡 on #764 — + # same footgun as forge.domain). docs/matrix.md::Assertion + # rationale. assertions = [ { assertion = hyperhiveDomain != null || cfg.serverName != null; @@ -344,11 +295,6 @@ in ''; } { - # Same footgun as forge.domain (#754): empty string renders - # `.` shaped garbage in both nginx server_name (treated - # as wildcard catch-all, surprising) and /etc/hosts (invalid - # entry). Argus 🟡 on #764 — fail loud here rather than ship - # the surprising behaviour. assertion = cfg.gatewayHost == null || cfg.gatewayHost != ""; message = '' services.hyperhive.matrix.gatewayHost = "" is rejected. The @@ -361,22 +307,10 @@ in } ]; - # Generate the registration token at system activation time, BEFORE - # the hive-matrix container would otherwise start with an empty - # bind-mount target (argus nit on #565: nspawn creates an empty - # file when the host path is missing, tuwunel reads it as - # `registration_token_file=""` and rejects every registration - # until the next restart). Idempotent: only writes when the file - # doesn't exist. 32-byte hex = 64 chars, same shape hive-c0re's - # `matrix::ensure_register_token` would produce. - # - # Ownership: plain `root:root 0600` — tuwunel inside the container - # runs as a hardened dynamic user (#644) and reads the token via - # systemd's `LoadCredential=` mechanism (see container config - # below), so it never needs direct read access on the host-side - # file. No `chown :tuwunel` / `chmod 0640` / GID-pin gymnastics - # required (per iris on #644 8043, dropping the shape #649 - # shipped with). + # Activation-time token generation (argus #565: the bind-mount + # would otherwise hand tuwunel an empty file on first boot and + # break every registration until restart). Idempotent; + # docs/matrix.md::Provisioning flow. system.activationScripts.hive-matrix-register-token = lib.stringAfter [ "var" ] '' tokenFile=${lib.escapeShellArg (toString cfg.registrationTokenFile)} if [ ! -s "$tokenFile" ]; then @@ -385,25 +319,17 @@ in echo >> "$tokenFile" echo "hive-matrix: generated registration token at $tokenFile" fi - # Always re-apply 0600 (idempotent on already-correct files; - # also normalises any 0640 / world-readable carry-over from - # pre-LoadCredential deployments). + # Re-apply 0600 (normalises any pre-LoadCredential carry-over). chmod 0600 "$tokenFile" ''; containers.hive-matrix = { autoStart = true; ephemeral = false; - # Share host netns — tuwunel's listeners look exactly like - # host-side services, no port-forward plumbing, and agent - # containers (also host netns) reach it via plain `localhost`. + # Shared host netns — agents reach tuwunel at localhost:. privateNetwork = false; - # Read-only bind of the host-managed registration token so - # tuwunel can resolve `registration_token_file` to a real - # file inside the container. The activation script above - # ensures the host path exists with a valid 64-char hex token - # before any container starts, so the bind always finds real - # content (no first-boot empty-file race; argus #565 nit). + # Read-only bind of the host-managed registration token; tuwunel + # reads it via systemd LoadCredential below (not directly). bindMounts.${cfg.registrationTokenFile} = { hostPath = cfg.registrationTokenFile; isReadOnly = true; @@ -417,44 +343,28 @@ in package = cfg.package; settings.global = { server_name = effectiveServerName; - # `address` is `listOf nonEmptyStr` upstream (multi-bind - # support). Single-host bind goes through as a one-element list. + # `address` + `port` are upstream `listOf` — wrap singles. address = [ "0.0.0.0" ]; - # `port` is `listOf port` upstream. Same shape. port = [ cfg.httpPort ]; max_request_size = cfg.maxRequestSize; - # Federation enabled at the protocol level so swarms - # can be wired up later by extending `trustedServers` - # without a homeserver restart. Empty trusted_servers - # keeps it effectively closed until peers are listed. + # Federation enabled at the protocol level; empty + # trustedServers keeps it effectively closed. allow_federation = true; trusted_servers = cfg.trustedServers; - # Token-gated registration: hive-c0re holds the token, - # agents never see it. allow_registration must be true - # for the token flow to engage; the absent + # Token-gated registration. The absent # `yes_i_am_very_very_sure_…_open_registration_…` flag # keeps the server closed to anyone without the token. allow_registration = true; - # Read the registration token via systemd's - # `LoadCredential=` mechanism (wired below) instead of - # the bind-mount path directly. systemd copies the host- - # owned 0600 root:root file into a per-service - # credentials dir owned by tuwunel's dynamic user with - # mode 0400 — keeps `DynamicUser=true` + `PrivateUsers=true` - # intact, no host-side `chown :tuwunel` / GID-pin - # gymnastics required (#644 / iris on 8043). + # LoadCredential below copies the host file into a + # 0400 dynamic-user-owned path; tuwunel reads from there. registration_token_file = "/run/credentials/tuwunel.service/registration_token"; - # E2EE disabled in initial rollout per operator call - # (#548) — re-enabling tracked at #551. + # E2EE disabled in initial rollout (#548); re-enable at #551. allow_encryption = false; }; }; - # `LoadCredential=:` makes systemd copy the - # bind-mounted host file into `/run/credentials/tuwunel.service/` - # owned by the service's (dynamic) user with mode 0400 at - # service start. The hardcoded path in `registration_token_file` - # above is the systemd-stable credentials dir; see - # `man systemd.exec` → LoadCredential. + # Keeps DynamicUser=true + PrivateUsers=true intact — no + # host-side chown :tuwunel / GID-pin gymnastics needed (#644 / + # iris on 8043). See `man systemd.exec` → LoadCredential. systemd.services.tuwunel.serviceConfig.LoadCredential = [ "registration_token:${toString cfg.registrationTokenFile}" ];