hyperhive/docs/integrations/matrix.md
atlas 199afa41c8 matrix: publish the appservice token from the swarm, not just read it
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.

`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.

Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.

The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.

`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.

Refs #4402
2026-09-15 20:57:49 +02:00

18 KiB

hive-matrix

Private Matrix homeserver (matrix-tuwunel — the conduwuit successor) wrapped in a nixos-container, plus optional fluffychat-web client at chat.<swarm-domain>/ (the gatewayHost vhost). Configured via services.hyperhive.swarm.matrix.*; vhost routing lives in gateway.md.

Container shape

Same shape as gateway.md::hive-forge container shape:

  • Container name hive-matrix (not h-*) so c0re's lifecycle scanner ignores it; operator manages via the standard nixos-container CLI.
  • Keeps hive-matrix from fighting any services.matrix-* the operator already runs on the host — separate systemd namespace, separate state dir.
  • Container shares the host network namespace (privateNetwork = false) for state + systemd-unit isolation. Agents reach the homeserver at chat.<swarm-domain> via the gateway (agents run in private netns and can't access host loopback directly).
  • Persistent state at /var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/ survives container restart / host reboot. To wipe, destroy the container.

Identity vs API listener: serverName vs gatewayHost

Two distinct hostnames:

  • serverName — matrix-spec server_name, embedded irrevocably in every @user:<server_name> and !room:<server_name> identifier minted on this homeserver. can't be changed later without abandoning every account and chat history. Defaults to the bare services.hyperhive.swarm.domain; clients autodiscover the actual API endpoint via the .well-known/matrix/{client,server} routes the gateway serves at that domain.
  • gatewayHost — the API listener hostname, where the gateway's matrix vhost proxies /_matrix/* to tuwunel. Defaults to chat.<services.hyperhive.swarm.domain>. Set to null to skip the gateway vhost (tuwunel stays direct on httpPort).

Both default under the swarm domain, because a swarm runs one homeserver: tying its identity to a single hive's domain would make relocating the container between hives look like a different homeserver.

⚠️ they're still not interchangeable, and the difference is the cost of changing one. gatewayHost is a routing detail clients rediscover through .well-known, so it's safe to move on a running deployment. The matrix id format bakes serverName into every user and room id, so adopting a new one does not rename the existing users and rooms — it strands them, because their ids still name a homeserver that no longer answers.

Upgrading a homeserver that already has ids

serverName's default has changed across releases. A homeserver that has already minted ids under an older default must pin the value it actually minted them under, not adopt the new default — see above for why adopting a new one strands existing users and rooms:

services.hyperhive.swarm.matrix = {
  # whichever this deployment already uses
  serverName = config.services.hyperhive.domain;
  gatewayHost = "matrix.${config.services.hyperhive.domain}";
};

A rebuild on a host that already has a homeserver prints a hive-matrix: WARNING — … serverName is unset line when this is missing, naming the value it's about to default to. That warning is why this section exists; it never fails the rebuild, so it's on you to act on it before the ids are minted.

Default-closed firewall

openFirewall defaults to false (secure-by-default): the host reaches the homeserver on loopback, and agent containers reach it at chat.<swarm-domain> via the gateway — so the firewall hole only matters for access from outside the host. Flip to true when announcing the homeserver to other hives or when an external matrix client needs to reach the client-server API directly.

Federation port 8448 is intentionally not opened here — tuwunel serves the federation API on the same httpPort as client-server by default. Reaching it on 8448 needs either an explicit tuwunel bind to that port OR a reverse-proxy + .well-known/matrix/server delegation (the latter lives in gateway.md::Discovery flow).

Provisioning flow (appservice)

Registration is closed. Accounts are created by the hive's own appservice: hive-c0re holds the appservice token, agents never see it, and an agent only ever receives its own access_token.

The appservice has no URL (url: null in its registration), so the homeserver never calls out to it and there is no service to run. What the registration buys is an identity the homeserver recognises — which is why no secret has to be equal on both sides of the wire, and why account creation doesn't depend on registration being open to anyone who learns a token.

  1. System activation mints a 32-byte random hex appservice token (64 chars) at /var/lib/hyperhive/matrix-appservice-token and its spec-required hs_token sibling, mode 0600 root:root, then renders the registration to /var/lib/hyperhive/matrix-appservice/hyperhive.yaml (also 0600). The tokens are minted only when missing; the registration is re-rendered every time, because the token file can be overwritten in place by the swarm secret store and a registration naming a stale token authenticates nobody. Runs at activation time, before any container start, because the directory is bind-mounted and nixos-container refuses to start when a bind source is missing.
  2. Read-only bind-mount maps that directory into the tuwunel container at the same path.
  3. systemd LoadCredential= inside the container copies the registration into /run/credentials/tuwunel.service/hyperhive-appservice.yaml, owned by tuwunel's dynamic user with mode 0400, at service start. The host file stays root:root 0600 — no chown :tuwunel / chmod 0640 / GID-pin gymnastics required. Keeps DynamicUser = true + PrivateUsers = true intact.
  4. tuwunel's appservice_dir points at the credentials directory, not at the bind-mount path. It reads only .yaml/.yml entries from there, so the sibling credentials are invisible to it. The .yaml suffix on the credential id is what makes this work.
  5. hive-c0re reads the appservice token and creates each account with one POST /register typed m.login.application_service, persisting the returned access_token to <agent-state>/matrix-token. It never mints the token itself: the value has to be the one the rendered registration names, and only the nix side writes that.
  6. An account that exists but has lost its token file is re-tokened by an appservice POST /login — no password and no admin rights involved. A stored-password login and an admin-room password reset remain behind that, for accounts created before the appservice existed or named outside its namespace.
  7. hive-c0re restarts hive-matrix-daemon for the agent immediately after writing the token so the daemon picks up the new credential without waiting for a full container restart. If the restart fails (for example daemon not yet running on first boot) hive-c0re logs the error as a warning and the .path-trigger sibling (hive-matrix-daemon.path watching for matrix-token appearance) brings the daemon up on the same boot cycle anyway.

The admin account, and why it needs no first-user luck

@hive:<server_name> is the appservice's own sender_localpart, which the homeserver creates itself when it loads the registration — on a zero-user database, inside startup, before the HTTP listener accepts anything. Its admin rights then come from an explicit make_user_admin, run by tuwunel's admin_execute in the same startup and likewise before the listener — so a fresh hive has a joined, power-level-100 admin on its first boot.

This replaces a dependency on being the first account ever registered, which was fragile in both directions: an appservice-created account is excluded from that automatic grant by design, and on a homeserver that already had users the rule never fired at all.

Promotion can't be bootstrapped over the API, and that's upstream's design rather than a gap: tuwunel only treats an admin-room message as a command when its sender is already an admin. admin_execute is the one lever with no sender to check. hive-c0re re-checks the result on every sweep by reading the admin account's own joined-rooms list; if the rights are missing it says so, names systemctl restart container@hive-matrix as the fix, and carries on — agent accounts, the hive Space and the chat room need no admin.

Upgrading a hive that used the registration token

Nothing to do, and nothing to time. The activation script mints the appservice token and renders the registration before the homeserver restarts, so the first boot after the switch already has both halves.

  • Existing accounts keep working. An access token lives on the device that minted it; removing the registration token touches no device, no account and no session. login_with_password stays on, so the password fallback is still there too.
  • Existing token files are honoured. The per-agent sweep skips any agent that already has a matrix-token, so no account is re-registered and no session is displaced.
  • The admin account is already admin on such a hive (it won the first-user grant when the hive was new), so the startup promotion is a no-op — upstream's make_user_admin short-circuits when the user is already joined at power level 100.
  • /var/lib/hyperhive/matrix-register-token is left on disk, read by nothing. Delete it or leave it; neither does any harm.
  • registrationTokenFile is a removed option. A config that still sets it fails to evaluate with a message naming the appservice — a hive that never set it (the default) is unaffected.
  • A swarm store holding the old matrix/registration-token path is no longer read at all. The value that matters now lives at matrix/appservice-token, and swarm-secret-publish on the authelia host mints and puts it there — the hive uses its locally minted token only until the first successful read. See ../swarm/secrets.md for how that mint stays idempotent across runs.

Initial rollout settings:

  • allow_federation = true at the protocol level so swarms can be wired up later by extending trustedServers without a homeserver restart. trusted_servers = [] keeps it effectively closed until you list peers.
  • allow_registration = false. tuwunel checks this flag only for requests that arrive without an appservice token, so hive-c0re provisions exactly as before and everyone else is refused. It's not a hardening afterthought: with no registration token configured, allow_registration = true makes tuwunel refuse to start unless yes_i_am_very_very_sure_…_open_registration_… is also set.
  • allow_encryption — server-side E2EE switch, sourced from services.hyperhive.swarm.matrix.allowEncryption (default false, opt-in). Off by default because on the hive-internal homeserver the operator already controls the transport; turn it on for encrypted rooms on external / federated homeservers or to keep contents opaque to the homeserver admin. The agent matrix client always supports decryption regardless of this flag — it uses the e2e-encryption feature of matrix-sdk so it can read encrypted rooms it's invited to even when this homeserver doesn't permit room encryption. matrix-sdk stores crypto keys in the per-agent sqlite store under the state dir; they persist across restarts (lost on --purge). read_room decrypts via room.messages() — UTD events surface as event_type = "m.room.encrypted" with body = "[unable to decrypt]". Cross-signing and automatic key backup aren't enabled for the first pass: static bearer-token bot accounts can't bootstrap cross-signing without MSC3967.

Hive Matrix Space

On first boot, after hive-c0re provisions all agent accounts, it creates a private Matrix Space named "hive" using the admin account (@hive:<server_name>) and invites every provisioned agent into it. This gives the operator a single Space in FluffyChat or any Matrix client that groups all agent-to-agent + operator rooms in one place.

The sweep also provisions a default hive-chat room as an m.space.child of the Space. Joining a Space doesn't autojoin child rooms — the explicit room entry ensures the operator and every agent can find a common chat room without manual setup. Room join is restricted (any Space member including the operator can join; agents are explicitly invited). Room version pinned to 10 for the restricted join floor.

State: hive-c0re persists both room IDs to /var/lib/hyperhive/matrix/ (mode 0600, owned by the hive-c0re service user):

  • space-room-id — the Space itself
  • chat-room-id — the hive-chat room

These paths are outside every agent state dir and aren't deleted by nixos-container destroy --purge — both survive full agent purges, and hive-c0re reuses them on re-provision.

Idempotent: if the files exist and are non-empty, hive-c0re considers the Space and room already created. Delete the files to force re-creation (for example after a homeserver wipe).

Configuration tuning

services.hyperhive.swarm.matrix = {
  trustedServers  = [ "matrix.org" "example.com" ];   # default: []
  maxRequestSize  = 20000000;                          # default: 20 MB
};

trustedServers (default []) — list of peer homeserver names whose signing keys tuwunel will fetch and trust. Federation is enabled at the protocol level from first boot (allow_federation = true) but no remote homeserver is trusted until listed here. For a closed single-hive deployment the default empty list is correct — add peer hive domains here when connecting hives into a swarm (see docs/swarm/).

maxRequestSize (default 20_000_000 bytes = 20 MB) — maximum size of a single matrix client request body. Matches the matrix-spec recommendation for media uploads and the upstream tuwunel default. Raise for deployments that need large file transfers; lower for resource-constrained hosts where a 20 MB request is unexpectedly large.

Assertion rationale

config.assertions in this module fail eval early rather than ship surprising behaviour:

  • services.hyperhive.swarm.matrix.gatewayHost != "" — same footgun as forge.domain: empty string renders .<hive>-shaped garbage in both nginx server_name (treated as wildcard catch-all, surprising) and /etc/hosts (invalid entry). null is the right opt-out shape; a config assertion rejects empty string explicitly. SSO is unconditional, so the three below are requirements of running a homeserver at all rather than of a setting:

  • sso.clientSecretFile is required — fails at eval, not at boot: tuwunel reads its identity providers from the config file, so a half-configured one can stop the homeserver from starting outright rather than merely hiding a login button. On a host that also runs the swarm's authelia it's wired up for you.

  • swarm.authelia.url is required — without a provider URL there is nothing to discover against.

  • gatewayHost != null is required — the SSO callback URL is format-locked to <homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>, and the identity provider needs a public name to redirect the browser to.

server_name's own bogus-value guard lives in hive-network.nix (services.hyperhive.domain != null), not here — see docs/networking/network.md.

fluffychat-web build fixes

pkgs.fluffychat-web ships from flutter341.buildFlutterApplication, which has two upstream gaps for fluffychat's web target:

  • The dart web-worker entry point (web/native_executor.dart) isn't compiled — buildFlutterApplication only runs flutter build web on the main entry.
  • native_imaging's C source isn't built — emscripten isn't a flutter-builder native build input.

Both fixed in nix/host-modules/hive-matrix.nix via two derivations:

  • fluffychat-web-imaging builds Imaging.{js,wasm} from the native_imaging C source via pkgs.emscripten. Source comes from pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging — already in the build closure of the flutter app, so no parallel hash pin and version autosyncs with nixpkgs bumps. Build closure is ~3.6 GiB (emscripten LLVM); runtime closure is just the two output files. dontConfigure = true because cmake runs inside js/Makefile via emcmake cmake, not at the package root. The build script needs HOME + EM_CACHE writable for emscripten's on-demand sysroot build (libc, libc++ → wasm).
  • fluffychat-web-fixed is pkgs.fluffychat-web plus a postInstall patch that (a) compiles web/native_executor.dart via dart compile js (dart from the flutter341 closure, no incremental cost) and (b) installs fluffychat-web-imaging's outputs into $out.

Two subtle details worth knowing before touching either derivation:

  • make -C js instead of cd js; make — keeps the build-phase pwd at the source root so installPhase doesn't have to know about the cd. Robust against future reorders / dontBuild.
  • web/native_executor.dart as a build-CWD-relative path, not $src/web/...dart's package_config.json walk-up needs to hit buildFlutterApplication's pub-get output (.dart_tool/ in the build CWD). Walking up from a read-only $src/ store path finds no .dart_tool/ and errors with "Couldn't resolve the package 'matrix'."

Drop both derivations when nixpkgs's flutter builder grows worker

  • emcc support upstream.

Mount point is chat.<swarm-domain>/ (the gatewayHost vhost); upstream --base-href "/" is correct at sub-domain root, no override.