docs/matrix.md: extract hive-matrix prose (#718 batch 5)

New top-level matrix doc covering everything that doesn't fit
elsewhere:

- Container shape (nixos-container, shared host netns, name choice,
  state persistence) — sibling to gateway.md::hive-forge container shape.
- Identity vs API listener: serverName vs gatewayHost split with the
  #660 breaking change.
- Default-closed firewall + federation port 8448 caveat.
- Provisioning flow: registration token activation, bind-mount,
  LoadCredential, hive-c0re's per-agent register + access_token
  persistence. Captures #565 first-boot race + #644 / iris 8043
  ownership shape.
- Assertion rationale (serverName, gatewayHost == "").
- fluffychat-web build fixes (#685): Imaging.{js,wasm} emscripten
  derivation + dart compile worker fixup, build-CWD path lesson
  from #685 / #733.
- Sequencing history.

In-code # comments trim to short purpose statements + docs pointers.
description = '' blocks (operator-facing options docs) preserved per
iris #718. ~140 lines removed from hive-matrix.nix.

`nix flake check` clean; `nix fmt` clean.
This commit is contained in:
atlas 2026-05-31 15:32:38 +02:00 committed by mara
commit a8d8159038
2 changed files with 251 additions and 142 deletions

199
docs/matrix.md Normal file
View file

@ -0,0 +1,199 @@
# hive-matrix
Private Matrix homeserver (matrix-tuwunel — the conduwuit
successor) wrapped in a nixos-container, plus optional fluffychat-web
client at `matrix.<hive>/`. Configured via
`services.hyperhive.matrix.*`; vhost routing lives in
[`gateway.md`](gateway.md).
## Container shape
Same shape as [`gateway.md::hive-forge container shape`](gateway.md):
- Container name `hive-matrix` (not `h-*`) so c0re's lifecycle
scanner ignores it; operator manages via the standard
`nixos-container` CLI.
- Keeps hive-matrix from fighting any `services.matrix-*` the
operator already runs on the host — separate systemd namespace,
separate state dir.
- Container shares the host network namespace
(`privateNetwork = false`) so agents reach tuwunel at
`http://localhost:<httpPort>` without extra plumbing — the
nixos-container is here for state + systemd-unit isolation, not
network isolation.
- Persistent state at
`/var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/`
survives container restart / host reboot. To wipe, destroy the
container.
## Identity vs API listener: `serverName` vs `gatewayHost`
Two distinct hostnames:
- **`serverName`** — matrix-spec `server_name`, embedded
*irrevocably* in every `@user:<server_name>` and `!room:<server_name>`
identifier minted on this homeserver. Cannot be changed later
without abandoning every account and chat history. Defaults to the
bare `services.hyperhive.domain` per mara on #660; clients
auto-discover the actual API endpoint via the
`.well-known/matrix/{client,server}` routes the hive-gateway serves
at that domain.
- **`gatewayHost`** — the API listener hostname, where the gateway's
matrix vhost proxies `/_matrix/*` to tuwunel. Defaults to
`matrix.<services.hyperhive.domain>` (sub-domain shape per mara on
#749:9609). Set to `null` to skip the gateway vhost (tuwunel stays
direct on `httpPort`).
**Breaking change** (#660): `serverName` used to default to
`matrix.${services.hyperhive.domain}`. Existing homeservers must set
the option explicitly to preserve their pre-#660 user / room IDs
before rebuilding. The default flipped because the bare hive-domain
makes for cleaner matrix IDs and `.well-known` delegation hides the
sub-domain from the user-facing identifier.
## Default-closed firewall
`openFirewall` defaults to `false` (#651, secure-by-default): the
homeserver is reachable from the host + every agent container via
loopback either way (shared netns), so the firewall hole only
matters for access from *outside* the host. Flip to `true` when
announcing the homeserver to other hives or when an external matrix
client needs to reach the client-server API directly.
**Breaking change** (#651): used to default to `true`. Operators
relying on external reach must add
`services.hyperhive.matrix.openFirewall = true;` before rebuilding.
Federation port 8448 is intentionally not opened here — tuwunel
serves the federation API on the same `httpPort` as client-server
by default. Reaching it on 8448 needs either an explicit tuwunel
bind to that port OR a reverse-proxy + `.well-known/matrix/server`
delegation (the latter lives in `gateway.md::Discovery flow`).
## Provisioning flow (registration token)
Token-gated registration: hive-c0re holds the token, agents never
see it. The agent only receives the resulting `access_token`.
1. **System activation** writes a 32-byte random hex token (64
chars) to `cfg.registrationTokenFile`
(`/var/lib/hyperhive/matrix-register-token` by default), mode
`0600 root:root`, before any container start. Idempotent — only
writes when the file is missing or empty; always re-applies 0600
(normalises any 0640 / world-readable carry-over from
pre-LoadCredential deployments). This runs at activation time
(not first container start) to dodge the argus #565 race where
nspawn creates an empty file when the bind-mount target is
missing and tuwunel reads `registration_token_file=""` rejecting
every registration until next restart.
2. **Read-only bind-mount** maps the host file into the tuwunel
container at the same path.
3. **systemd `LoadCredential=`** inside the container copies the
bind-mounted file into
`/run/credentials/tuwunel.service/registration_token`, owned by
tuwunel's dynamic user with mode `0400`, at service start. The
host file stays `root:root 0600` — no `chown :tuwunel` /
`chmod 0640` / GID-pin gymnastics required (per iris on #644
8043, dropping the shape #649 originally shipped with). Keeps
`DynamicUser = true` + `PrivateUsers = true` intact.
4. tuwunel's `registration_token_file` points at the credentials
path, not the original bind-mount path.
5. **hive-c0re** uses the token to register each agent account via
the matrix-spec UIAA registration flow, persists the returned
`access_token` to `<agent-state>/matrix-token`. The agent's
matrix MCP client authenticates with that access_token and
never touches the shared registration token.
Initial rollout settings (#548):
- `allow_federation = true` at the protocol level so swarms can be
wired up later by extending `trustedServers` without a homeserver
restart. `trusted_servers = []` keeps it effectively closed
until peers are listed.
- `allow_registration = true` (required for the token flow to
engage). The absent
`yes_i_am_very_very_sure_…_open_registration_…` flag keeps the
server closed to anyone without the token.
- `allow_encryption = false` per operator call (#548). E2EE
re-enabling tracked at #551.
## Assertion rationale
Two `config.assertions` entries fail eval early rather than ship
surprising behaviour:
- **`hyperhiveDomain != null || cfg.serverName != null`** (mara on
#548) — `server_name` is embedded into every user / room ID
irrevocably; we refuse to spawn the homeserver with a bogus
server_name we can never change later.
- **`cfg.gatewayHost != ""`** (argus 🟡 on #764) — same footgun as
`forge.domain` (#754). Empty string renders `.<hive>`-shaped
garbage in both nginx `server_name` (treated as wildcard
catch-all, surprising) and `/etc/hosts` (invalid entry). `null`
is the right opt-out shape; empty string is rejected explicitly.
## fluffychat-web build fixes (#685)
`pkgs.fluffychat-web` ships from `flutter341.buildFlutterApplication`,
which has two upstream gaps for fluffychat's web target:
- The dart web-worker entry point (`web/native_executor.dart`) isn't
compiled — `buildFlutterApplication` only runs `flutter build web`
on the main entry.
- `native_imaging`'s C source isn't built — emscripten isn't a
flutter-builder native build input.
Both fixed in `nix/modules/hive-matrix.nix` via two derivations:
- **`fluffychat-web-imaging`** builds `Imaging.{js,wasm}` from the
`native_imaging` C source via `pkgs.emscripten`. Source comes
from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging`
— already in the build closure of the flutter app, so no parallel
hash pin and version auto-syncs with nixpkgs bumps. Build closure
is ~3.6 GiB (emscripten LLVM); runtime closure is just the two
output files. `dontConfigure = true` because cmake runs inside
`js/Makefile` via `emcmake cmake`, not at the package root. The
build script needs `HOME` + `EM_CACHE` writable for emscripten's
on-demand sysroot build (libc, libc++ → wasm).
- **`fluffychat-web-fixed`** is `pkgs.fluffychat-web` plus a
`postInstall` patch that (a) compiles `web/native_executor.dart`
via `dart compile js` (dart from the flutter341 closure, no
incremental cost) and (b) installs `fluffychat-web-imaging`'s
outputs into `$out`.
Two non-obvious fixes from review history:
- **`make -C js`** instead of `cd js; make` (argus 🟡 on #697 v2)
— keeps the build-phase pwd at the source root so `installPhase`
doesn't have to know about the cd. Robust against future
reorders / `dontBuild`.
- **`web/native_executor.dart`** as a build-CWD-relative path,
*not* `$src/web/...` (#685 / #733 fixup) — `dart`'s
`package_config.json` walk-up needs to hit
`buildFlutterApplication`'s pub-get output (`.dart_tool/` in the
build CWD). Walking up from a read-only `$src/` store path finds
no `.dart_tool/` and errors with "Couldn't resolve the package
'matrix'". Confused two PRs.
Drop both derivations when nixpkgs's flutter builder grows worker
+ emcc support upstream.
Mount point is `matrix.<hive>/` (#772); upstream `--base-href "/"`
is correct at sub-domain root, no override.
## Sequencing history
- #548 — initial rollout (federation enabled, registration enabled,
E2EE disabled)
- #565 — first-boot empty-token race fix → activation-time token
generation
- #644 / iris 8043 / #649 — registration token ownership shape
(dropped chown/GID-pin; LoadCredential delivers as 0400 dynamic-user)
- #651`openFirewall` default flipped to `false`
- #660`serverName` default flipped to bare hive-domain (was
`matrix.<hive>`)
- #685 / #697 / #733 — fluffychat-web build fixes (Imaging emscripten,
native_executor dart worker, build-CWD path)
- #736 — fluffychat config.json inline JSON at sub-domain root
- #749 / #764 — gateway sub-domain shape verdict
- #772 / #775 — fluffychat hops from `<hive>/matrix/` to `matrix.<hive>/`

View file

@ -9,26 +9,16 @@ let
hyperhiveDomain = config.services.hyperhive.domain;
effectiveServerName = if cfg.serverName != null then cfg.serverName else hyperhiveDomain;
# Three files are missing from `pkgs.fluffychat-web` because
# `flutter341.buildFlutterApplication` doesn't run the dart
# web-worker compile pass + doesn't run the native_imaging emscripten
# build (#685). `fluffychat-web-imaging` below builds the latter from
# source via `pkgs.emscripten`; the worker compile is inline in
# `fluffychat-web-fixed.postInstall`. Drop both when nixpkgs's
# flutter builder grows worker + emcc support upstream.
# fluffychat-web build fixes: nixpkgs's `flutter341.buildFlutterApplication`
# skips the dart web-worker compile + the emscripten native_imaging
# build. Two derivations below cover both. Full rationale (why
# passthru.pubspecLock.dependencySources, why `dontConfigure`, why
# `make -C js`, why build-CWD-relative dart path): docs/matrix.md::
# fluffychat-web build fixes.
# `Imaging.{js,wasm}` built from `native_imaging`'s C source via
# emscripten. Source comes from
# `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources` so
# there's no parallel hash pin — version auto-syncs with nixpkgs
# bumps. Build closure +~3.6 GiB (emscripten LLVM); runtime closure
# is just the two output files.
fluffychat-web-imaging = pkgs.stdenv.mkDerivation {
pname = "fluffychat-web-imaging";
version = pkgs.fluffychat-web.passthru.pubspecLock.dependencyVersions.native_imaging;
# The pub-cache derivation that fluffychat-web's flutter build uses.
# Already in the build closure; no `fetchurl` or own hash pin.
src = pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging;
nativeBuildInputs = with pkgs; [
@ -38,21 +28,18 @@ let
jq
];
# cmake config runs inside `js/Makefile` (via `emcmake cmake`) —
# skip the default `configurePhase` which would try to invoke
# cmake against the package root and fail (no CMakeLists at top).
# cmake runs inside js/Makefile via `emcmake cmake`; the default
# configurePhase would invoke cmake at the package root (no
# CMakeLists) and fail.
dontConfigure = true;
buildPhase = ''
runHook preBuild
# emscripten needs HOME + a writable cache dir for its sysroot
# build (libc, libc++, etc. compiled to wasm on demand).
# emscripten on-demand sysroot build needs writable HOME + cache.
export HOME=$TMPDIR
export EM_CACHE=$TMPDIR/.emscriptencache
mkdir -p $EM_CACHE
# `make -C js` keeps the build phase pwd at the source root so
# installPhase doesn't have to know about the cd (argus 🟡 on
# PR #697 v2 — robust against future reorders / `dontBuild`).
# `make -C js` keeps pwd at source root for the installPhase.
make -C js Imaging.js Imaging.wasm
runHook postBuild
'';
@ -66,71 +53,37 @@ let
'';
meta = with pkgs.lib; {
description = "Imaging.js + Imaging.wasm built from the native_imaging dart package for fluffychat-web (#685)";
description = "Imaging.js + Imaging.wasm built from the native_imaging dart package for fluffychat-web";
homepage = "https://pub.dev/packages/native_imaging";
license = licenses.agpl3Plus;
};
};
# `pkgs.fluffychat-web` with #685's three missing files patched in
# via postInstall. Mount point is `matrix.<hive>/` (#772); upstream
# `--base-href "/"` is correct at sub-domain root, no override.
fluffychat-web-fixed = pkgs.fluffychat-web.overrideAttrs (old: {
# `dart` from the flutter341 closure (already pulled, no
# incremental closure cost) so we can compile the web-worker
# entry point that buildFlutterApplication skips.
# dart from the flutter341 closure (already pulled, no incremental
# cost) to compile the web-worker entry point.
nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [ pkgs.flutter341.dart ];
postInstall =
(old.postInstall or "")
+ ''
# `web/...` is relative to build CWD so dart's package_config
# walk-up hits buildFlutterApplication's pub-get output (#685
# / #733 fixup — `$src/web/...` would walk up to a read-only
# store path with no `.dart_tool/`).
${pkgs.flutter341.dart}/bin/dart compile js \
-o $out/native_executor.js \
web/native_executor.dart
postInstall = (old.postInstall or "") + ''
# `web/...` is BUILD-CWD-relative (not `$src/...`) so dart's
# package_config walk-up hits buildFlutterApplication's
# pub-get output `.dart_tool/`.
${pkgs.flutter341.dart}/bin/dart compile js \
-o $out/native_executor.js \
web/native_executor.dart
install -m 644 ${fluffychat-web-imaging}/Imaging.js $out/Imaging.js
install -m 644 ${fluffychat-web-imaging}/Imaging.wasm $out/Imaging.wasm
'';
install -m 644 ${fluffychat-web-imaging}/Imaging.js $out/Imaging.js
install -m 644 ${fluffychat-web-imaging}/Imaging.wasm $out/Imaging.wasm
'';
});
in
{
# Private Matrix homeserver (matrix-tuwunel — the official conduwuit
# successor) for hyperhive agents, wrapped in a nixos-container so it
# doesn't fight any existing `services.matrix-*` the operator may
# already run on the host. Same shape as `nix/modules/hive-forge.nix`:
# shared host netns (`privateNetwork = false`) so agents reach it at
# `http://localhost:<httpPort>` (or via the configured server_name
# for federation), nixos-container only here for state + systemd-unit
# isolation.
#
# Container name `hive-matrix` (not `h-*`) so the lifecycle scanner
# ignores it; operator manages via the standard `nixos-container` CLI.
#
# Persistent state at `/var/lib/nixos-containers/hive-matrix/var/lib/
# matrix-tuwunel/` (survives container restart / host reboot). To
# wipe, destroy the container.
#
# Initial rollout (#548): federation enabled (needed for multi-hive
# swarms; trusted_servers starts empty so no actual federation traffic
# leaves until peers are explicitly listed), registration enabled via
# a `registration_token_file` known only to hive-c0re (so agents can't
# self-register without going through the coordinator), e2ee disabled
# per operator call (tracked for follow-up at #551).
#
# Provisioning model (matches `nix/modules/hive-forge.nix` shape):
# hive-c0re generates a 32-byte random `registration_token` on first
# boot, writes it to `/var/lib/hyperhive/matrix-register-token` (mode
# 0600, root-only), and bind-mounts that file read-only into the
# tuwunel container at the same path so tuwunel can read it via
# `registration_token_file`. hive-c0re then uses the token to register
# each agent account via the matrix-spec UIAA registration flow, and
# persists the returned `access_token` to `<agent-state>/matrix-token`
# so the agent's matrix MCP client can authenticate without ever
# seeing the shared registration token.
# Private matrix-tuwunel homeserver wrapped in a nixos-container,
# optional fluffychat-web client at matrix.<hive>/. Container shape,
# serverName vs gatewayHost split, provisioning flow (registration
# token + LoadCredential), assertion rationale, initial rollout
# settings: docs/matrix.md. Vhost map + discovery flow + tuning
# knobs: docs/gateway.md.
options.services.hyperhive.matrix = {
enable = lib.mkOption {
@ -322,12 +275,10 @@ in
};
config = lib.mkIf cfg.enable {
# mara on #548: "there is no default, but it is required. add
# assertion." — fail eval with a helpful message rather than
# spawning a homeserver with a bogus server_name we can never
# change later. `services.hyperhive.domain` is host-wide; matrix derives
# the server_name from it (or from `cfg.serverName` if the
# operator wants to override).
# serverName must exist (mara on #548 — irrevocably embedded in
# user/room IDs); gatewayHost may not be "" (argus 🟡 on #764 —
# same footgun as forge.domain). docs/matrix.md::Assertion
# rationale.
assertions = [
{
assertion = hyperhiveDomain != null || cfg.serverName != null;
@ -344,11 +295,6 @@ in
'';
}
{
# Same footgun as forge.domain (#754): empty string renders
# `.<hive>` shaped garbage in both nginx server_name (treated
# as wildcard catch-all, surprising) and /etc/hosts (invalid
# entry). Argus 🟡 on #764 — fail loud here rather than ship
# the surprising behaviour.
assertion = cfg.gatewayHost == null || cfg.gatewayHost != "";
message = ''
services.hyperhive.matrix.gatewayHost = "" is rejected. The
@ -361,22 +307,10 @@ in
}
];
# Generate the registration token at system activation time, BEFORE
# the hive-matrix container would otherwise start with an empty
# bind-mount target (argus nit on #565: nspawn creates an empty
# file when the host path is missing, tuwunel reads it as
# `registration_token_file=""` and rejects every registration
# until the next restart). Idempotent: only writes when the file
# doesn't exist. 32-byte hex = 64 chars, same shape hive-c0re's
# `matrix::ensure_register_token` would produce.
#
# Ownership: plain `root:root 0600` — tuwunel inside the container
# runs as a hardened dynamic user (#644) and reads the token via
# systemd's `LoadCredential=` mechanism (see container config
# below), so it never needs direct read access on the host-side
# file. No `chown :tuwunel` / `chmod 0640` / GID-pin gymnastics
# required (per iris on #644 8043, dropping the shape #649
# shipped with).
# Activation-time token generation (argus #565: the bind-mount
# would otherwise hand tuwunel an empty file on first boot and
# break every registration until restart). Idempotent;
# docs/matrix.md::Provisioning flow.
system.activationScripts.hive-matrix-register-token = lib.stringAfter [ "var" ] ''
tokenFile=${lib.escapeShellArg (toString cfg.registrationTokenFile)}
if [ ! -s "$tokenFile" ]; then
@ -385,25 +319,17 @@ in
echo >> "$tokenFile"
echo "hive-matrix: generated registration token at $tokenFile"
fi
# Always re-apply 0600 (idempotent on already-correct files;
# also normalises any 0640 / world-readable carry-over from
# pre-LoadCredential deployments).
# Re-apply 0600 (normalises any pre-LoadCredential carry-over).
chmod 0600 "$tokenFile"
'';
containers.hive-matrix = {
autoStart = true;
ephemeral = false;
# Share host netns — tuwunel's listeners look exactly like
# host-side services, no port-forward plumbing, and agent
# containers (also host netns) reach it via plain `localhost`.
# Shared host netns — agents reach tuwunel at localhost:<port>.
privateNetwork = false;
# Read-only bind of the host-managed registration token so
# tuwunel can resolve `registration_token_file` to a real
# file inside the container. The activation script above
# ensures the host path exists with a valid 64-char hex token
# before any container starts, so the bind always finds real
# content (no first-boot empty-file race; argus #565 nit).
# Read-only bind of the host-managed registration token; tuwunel
# reads it via systemd LoadCredential below (not directly).
bindMounts.${cfg.registrationTokenFile} = {
hostPath = cfg.registrationTokenFile;
isReadOnly = true;
@ -417,44 +343,28 @@ in
package = cfg.package;
settings.global = {
server_name = effectiveServerName;
# `address` is `listOf nonEmptyStr` upstream (multi-bind
# support). Single-host bind goes through as a one-element list.
# `address` + `port` are upstream `listOf` — wrap singles.
address = [ "0.0.0.0" ];
# `port` is `listOf port` upstream. Same shape.
port = [ cfg.httpPort ];
max_request_size = cfg.maxRequestSize;
# Federation enabled at the protocol level so swarms
# can be wired up later by extending `trustedServers`
# without a homeserver restart. Empty trusted_servers
# keeps it effectively closed until peers are listed.
# Federation enabled at the protocol level; empty
# trustedServers keeps it effectively closed.
allow_federation = true;
trusted_servers = cfg.trustedServers;
# Token-gated registration: hive-c0re holds the token,
# agents never see it. allow_registration must be true
# for the token flow to engage; the absent
# Token-gated registration. The absent
# `yes_i_am_very_very_sure_…_open_registration_…` flag
# keeps the server closed to anyone without the token.
allow_registration = true;
# Read the registration token via systemd's
# `LoadCredential=` mechanism (wired below) instead of
# the bind-mount path directly. systemd copies the host-
# owned 0600 root:root file into a per-service
# credentials dir owned by tuwunel's dynamic user with
# mode 0400 — keeps `DynamicUser=true` + `PrivateUsers=true`
# intact, no host-side `chown :tuwunel` / GID-pin
# gymnastics required (#644 / iris on 8043).
# LoadCredential below copies the host file into a
# 0400 dynamic-user-owned path; tuwunel reads from there.
registration_token_file = "/run/credentials/tuwunel.service/registration_token";
# E2EE disabled in initial rollout per operator call
# (#548) — re-enabling tracked at #551.
# E2EE disabled in initial rollout (#548); re-enable at #551.
allow_encryption = false;
};
};
# `LoadCredential=<id>:<host-path>` makes systemd copy the
# bind-mounted host file into `/run/credentials/tuwunel.service/<id>`
# owned by the service's (dynamic) user with mode 0400 at
# service start. The hardcoded path in `registration_token_file`
# above is the systemd-stable credentials dir; see
# `man systemd.exec` → LoadCredential.
# Keeps DynamicUser=true + PrivateUsers=true intact — no
# host-side chown :tuwunel / GID-pin gymnastics needed (#644 /
# iris on 8043). See `man systemd.exec` → LoadCredential.
systemd.services.tuwunel.serviceConfig.LoadCredential = [
"registration_token:${toString cfg.registrationTokenFile}"
];