docs: restructure into topic subdirectories, collapse duplicated index

Per mara's go-ahead on hyperhive#3902 ("getting started is good, but terminal rendering does not go in there i think"): Moved 21 top-level docs/*.md files into 7 new topic subdirectories (existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched): getting-started/ setup.md agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md trust-boundary/ boundary.md, security.md integrations/ forge.md, matrix.md, github.md, knowledge.md networking/ gateway.md, network.md, snapshot-store.md scheduler/ jobq.md, coordinator.md, ci.md, observability.md process/ conventions.md, gotchas.md, pr-review-gate.md web-ui/ terminal-rendering.md (moved into the EXISTING dir, per mara's correction to the original getting-started guess -- it's UI implementation detail, not onboarding) The physical layout now matches docs/README.md's own topical headers, which already amounted to this taxonomy -- see the scoping comment on the issue for the two findings that motivated this (a genuine duplication between CLAUDE.md's old "Reading paths" list and docs/README.md's grouped one, since drifted out of sync with each other; and the flat layout not matching the grouping we already had). Fixed every cross-reference this moved across the whole repo (~120 files: docs/ internal links at every depth, Rust doc comments, nix module option docs, crate READMEs) -- verified two ways: a grep sweep confirming zero remaining references to any old path, and a script that resolves every markdown link in docs/**/*.md + CLAUDE.md + README.md against the filesystem and reports anything that doesn't exist (zero broken links). Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to a pointer at docs/README.md, now the single index. Rewrote docs/README.md itself to use the new subdirectory paths and added the one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md). Classified all 22 docs/*.md files first via a haiku subagent (mara's suggestion) on two axes -- proposed grouping and operator-vs- implementation focus -- before finalizing the taxonomy; spot-checked the report and found internal inconsistencies (its classification table disagreed with its own summary section for a few files), so this taxonomy is my original proposal + the one correction mara gave directly, not a blind application of the subagent's table. The operator-focus data it gathered is still useful for a follow-up content pass (docs skewing 'mixed' rather than pure operator-facing), not addressed in this PR -- structure only. nix fmt clean, both pre-push lints clean.
2026-09-02 01:47:05 +02:00 · 2026-09-02 01:47:05 +02:00 · 07b62612b0
commit 07b62612b0
parent e4a22b4190
124 changed files with 301 additions and 377 deletions
--- a/docs/integrations/matrix.md
+++ b/docs/integrations/matrix.md
@ -0,0 +1,287 @@
+# hive-matrix
+
+Private Matrix homeserver (matrix-tuwunel — the conduwuit
+successor) wrapped in a nixos-container, plus optional fluffychat-web
+client at `chat.<swarm-domain>/` (the `gatewayHost` vhost). Configured via
+`services.hyperhive.swarm.matrix.*`; vhost routing lives in
+[`gateway.md`](../networking/gateway.md).
+
+## Container shape
+
+Same shape as [`gateway.md::hive-forge container shape`](../networking/gateway.md):
+
+- Container name `hive-matrix` (not `h-*`) so c0re's lifecycle
+  scanner ignores it; operator manages via the standard
+  `nixos-container` CLI.
+- Keeps hive-matrix from fighting any `services.matrix-*` the
+  operator already runs on the host — separate systemd namespace,
+  separate state dir.
+- Container shares the host network namespace
+  (`privateNetwork = false`) for state + systemd-unit isolation. Agents
+  reach the homeserver at `chat.<swarm-domain>` via the gateway (agents
+  run in private netns and can't access host loopback directly).
+- Persistent state at
+  `/var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/`
+  survives container restart / host reboot. To wipe, destroy the
+  container.
+
+## Identity vs API listener: `serverName` vs `gatewayHost`
+
+Two distinct hostnames:
+
+- **`serverName`** — matrix-spec `server_name`, embedded
+  *irrevocably* in every `@user:<server_name>` and `!room:<server_name>`
+  identifier minted on this homeserver. Cannot be changed later
+  without abandoning every account and chat history. Defaults to the
+  bare `services.hyperhive.swarm.domain`; clients auto-discover the
+  actual API endpoint via the `.well-known/matrix/{client,server}`
+  routes the gateway serves at that domain.
+- **`gatewayHost`** — the API listener hostname, where the gateway's
+  matrix vhost proxies `/_matrix/*` to tuwunel. Defaults to
+  `chat.<services.hyperhive.swarm.domain>`. Set to `null` to skip the
+  gateway vhost (tuwunel stays direct on `httpPort`).
+
+Both default under the **swarm** domain, because a swarm runs one
+homeserver: tying its identity to a single hive's domain would make
+relocating the container between hives look like a different
+homeserver.
+
+⚠️ **They are still not interchangeable, and the difference is the
+cost of changing one.** `gatewayHost` is a routing detail clients
+rediscover through `.well-known`, so it is safe to move on a running
+deployment. `serverName` is baked into every user and room id, so
+adopting a new one does **not** rename the existing users and rooms —
+it strands them, because their ids still name a homeserver that no
+longer answers.
+
+### Upgrading a homeserver that already has ids
+
+`serverName`'s default has changed across releases. A homeserver that
+has already minted ids under an older default must **pin the value it
+actually minted them under**, not adopt the new default — see above
+for why adopting a new one strands existing users and rooms:
+
+```nix
+services.hyperhive.swarm.matrix = {
+  # whichever this deployment already uses
+  serverName = config.services.hyperhive.domain;
+  gatewayHost = "matrix.${config.services.hyperhive.domain}";
+};
+```
+
+A rebuild on a host that already has a homeserver prints a
+`hive-matrix: WARNING — … serverName is unset` line when this is missing,
+naming the value it is about to default to. That warning is why this
+section exists; it never fails the rebuild, so it is on you to act on it
+before the ids are minted.
+
+## Default-closed firewall
+
+`openFirewall` defaults to `false` (secure-by-default): the host
+reaches the homeserver on loopback, and agent containers reach it
+at `chat.<swarm-domain>` via the gateway — so the firewall hole only
+matters for access from *outside* the host. Flip to `true` when
+announcing the homeserver to other hives or when an external matrix
+client needs to reach the client-server API directly.
+
+Federation port 8448 is intentionally not opened here — tuwunel
+serves the federation API on the same `httpPort` as client-server
+by default. Reaching it on 8448 needs either an explicit tuwunel
+bind to that port OR a reverse-proxy + `.well-known/matrix/server`
+delegation (the latter lives in `gateway.md::Discovery flow`).
+
+## Provisioning flow (registration token)
+
+Token-gated registration: hive-c0re holds the token, agents never
+see it. The agent only receives the resulting `access_token`.
+
+1. **System activation** writes a 32-byte random hex token (64
+   chars) to `cfg.registrationTokenFile`
+   (`/var/lib/hyperhive/matrix-register-token` by default), mode
+   `0600 root:root`, before any container start. Idempotent — only
+   writes when the file is missing or empty; always re-applies 0600
+   (normalises any 0640 / world-readable carry-over from
+   pre-LoadCredential deployments). This runs at activation time
+   (not first container start) to dodge a race where nspawn creates
+   an empty file when the bind-mount target is missing and tuwunel
+   reads `registration_token_file=""`, rejecting every registration
+   until next restart.
+2. **Read-only bind-mount** maps the host file into the tuwunel
+   container at the same path.
+3. **systemd `LoadCredential=`** inside the container copies the
+   bind-mounted file into
+   `/run/credentials/tuwunel.service/registration_token`, owned by
+   tuwunel's dynamic user with mode `0400`, at service start. The
+   host file stays `root:root 0600` — no `chown :tuwunel` /
+   `chmod 0640` / GID-pin gymnastics required. Keeps
+   `DynamicUser = true` + `PrivateUsers = true` intact.
+4. tuwunel's `registration_token_file` points at the credentials
+   path, not the original bind-mount path.
+5. **hive-c0re** uses the token to register each agent account via
+   the matrix-spec UIAA registration flow, persists the returned
+   `access_token` to `<agent-state>/matrix-token`. The agent's
+   matrix MCP client authenticates with that access_token and
+   never touches the shared registration token.
+6. **hive-c0re restarts `hive-matrix-daemon`** for the agent
+   immediately after writing the token so the daemon picks up the
+   new credential without waiting for a full container restart. If
+   the restart fails (e.g. daemon not yet running on first boot) the
+   error is logged as a warning and the `.path`-trigger sibling
+   (`hive-matrix-daemon.path` watching for `matrix-token` appearance)
+   brings the daemon up on the same boot cycle anyway.
+
+Initial rollout settings:
+
+- `allow_federation = true` at the protocol level so swarms can be
+  wired up later by extending `trustedServers` without a homeserver
+  restart. `trusted_servers = []` keeps it effectively closed
+  until peers are listed.
+- `allow_registration = true` (required for the token flow to
+  engage). The absent
+  `yes_i_am_very_very_sure_…_open_registration_…` flag keeps the
+  server closed to anyone without the token.
+- `allow_encryption` — server-side E2EE switch, sourced from
+  `services.hyperhive.swarm.matrix.allowEncryption` (**default `false`**, opt-in).
+  Off by default because on the hive-internal homeserver the operator
+  already controls the transport; turn it on for encrypted rooms on
+  external / federated homeservers or to keep contents opaque to the
+  homeserver admin. **The agent matrix client always supports decryption
+  regardless of this flag** — it uses the `e2e-encryption` feature of
+  `matrix-sdk` so it can read encrypted rooms it is invited to even when
+  this homeserver doesn't permit room encryption. Crypto keys are stored
+  in the per-agent sqlite store under the state dir and persist across
+  restarts (lost on `--purge`). `read_room` decrypts via
+  `room.messages()` — UTD events surface as `event_type =
+  "m.room.encrypted"` with `body = "[unable to decrypt]"`.
+  Cross-signing and automatic key backup are not enabled for the first
+  pass: static bearer-token bot accounts can't bootstrap cross-signing
+  without MSC3967.
+
+## Hive Matrix Space
+
+On first boot (after all agent accounts are provisioned), hive-c0re
+creates a private **Matrix Space** named `"hive"` using the admin
+account (`@hive:<server_name>`) and invites every provisioned agent
+into it. This gives the operator a single Space in FluffyChat or any
+Matrix client that groups all agent-to-agent + operator rooms in one
+place.
+
+The sweep also provisions a default **`hive-chat` room** as an
+`m.space.child` of the Space. Joining a Space doesn't auto-join
+child rooms — the explicit room entry ensures the operator and every
+agent can find a common chat room without manual setup. Room join is
+restricted (any Space member including the operator can join; agents
+are explicitly invited). Room version pinned to 10 for the restricted
+join floor.
+
+**State**: both room IDs are persisted to `/var/lib/hyperhive/matrix/`
+(mode `0600`, owned by the hive-c0re service user):
+- `space-room-id` — the Space itself
+- `chat-room-id` — the `hive-chat` room
+
+These paths are **outside** every agent state dir and are NOT deleted by
+`nixos-container destroy --purge` — both survive full agent purges and
+are reused on re-provision.
+
+**Idempotent**: if the files exist and are non-empty, the Space and
+room are considered already created. Delete the files to force
+re-creation (e.g. after a homeserver wipe).
+
+## Configuration tuning
+
+```nix
+services.hyperhive.swarm.matrix = {
+  trustedServers  = [ "matrix.org" "example.com" ];   # default: []
+  maxRequestSize  = 20000000;                          # default: 20 MB
+};
+```
+
+**`trustedServers`** (default `[]`) — list of peer homeserver names
+whose signing keys tuwunel will fetch and trust. Federation is enabled
+at the protocol level from first boot (`allow_federation = true`) but
+no remote homeserver is trusted until listed here. For a closed
+single-hive deployment the default empty list is correct — add peer
+hive domains here when connecting hives into a swarm (see
+[`docs/swarm/`](../swarm/README.md)).
+
+**`maxRequestSize`** (default `20_000_000` bytes = 20 MB) — maximum
+size of a single matrix client request body. Matches the matrix-spec
+recommendation for media uploads and the upstream tuwunel default.
+Raise for deployments that need large file transfers; lower for
+resource-constrained hosts where a 20 MB request is unexpectedly large.
+
+## Assertion rationale
+
+`config.assertions` in this module fail eval early rather than ship
+surprising behaviour:
+
+- **`cfg.gatewayHost != ""`** — same footgun as `forge.domain`:
+  empty string renders `.<hive>`-shaped garbage in both nginx
+  `server_name` (treated as wildcard catch-all, surprising) and
+  `/etc/hosts` (invalid entry). `null` is the right opt-out shape;
+  empty string is rejected explicitly.
+SSO is unconditional, so the three below are requirements of running a
+homeserver at all rather than of a setting:
+
+- **`sso.clientSecretFile` is required** — fails at eval, not at boot:
+  tuwunel reads its identity providers from the config file, so a
+  half-configured one can stop the homeserver from starting outright
+  rather than merely hiding a login button. On a host that also runs
+  the swarm's authelia it is wired up for you.
+- **`swarm.authelia.url` is required** — without a provider URL there
+  is nothing to discover against.
+- **`gatewayHost != null` is required** — the SSO callback URL is
+  format-locked to `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`,
+  and the identity provider needs a public name to redirect the
+  browser to.
+
+`server_name`'s own bogus-value guard lives in `hive-network.nix`
+(`services.hyperhive.domain != null`), not here — see
+[`docs/networking/network.md`](../networking/network.md).
+
+## fluffychat-web build fixes
+
+`pkgs.fluffychat-web` ships from `flutter341.buildFlutterApplication`,
+which has two upstream gaps for fluffychat's web target:
+
+- The dart web-worker entry point (`web/native_executor.dart`) isn't
+  compiled — `buildFlutterApplication` only runs `flutter build web`
+  on the main entry.
+- `native_imaging`'s C source isn't built — emscripten isn't a
+  flutter-builder native build input.
+
+Both fixed in `nix/host-modules/hive-matrix.nix` via two derivations:
+
+- **`fluffychat-web-imaging`** builds `Imaging.{js,wasm}` from the
+  `native_imaging` C source via `pkgs.emscripten`. Source comes
+  from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging`
+  — already in the build closure of the flutter app, so no parallel
+  hash pin and version auto-syncs with nixpkgs bumps. Build closure
+  is ~3.6 GiB (emscripten LLVM); runtime closure is just the two
+  output files. `dontConfigure = true` because cmake runs inside
+  `js/Makefile` via `emcmake cmake`, not at the package root. The
+  build script needs `HOME` + `EM_CACHE` writable for emscripten's
+  on-demand sysroot build (libc, libc++ → wasm).
+- **`fluffychat-web-fixed`** is `pkgs.fluffychat-web` plus a
+  `postInstall` patch that (a) compiles `web/native_executor.dart`
+  via `dart compile js` (dart from the flutter341 closure, no
+  incremental cost) and (b) installs `fluffychat-web-imaging`'s
+  outputs into `$out`.
+
+Two non-obvious details worth knowing before touching either derivation:
+
+- **`make -C js`** instead of `cd js; make` — keeps the build-phase
+  pwd at the source root so `installPhase` doesn't have to know
+  about the cd. Robust against future reorders / `dontBuild`.
+- **`web/native_executor.dart`** as a build-CWD-relative path,
+  *not* `$src/web/...` — `dart`'s `package_config.json` walk-up
+  needs to hit `buildFlutterApplication`'s pub-get output
+  (`.dart_tool/` in the build CWD). Walking up from a read-only
+  `$src/` store path finds no `.dart_tool/` and errors with
+  "Couldn't resolve the package 'matrix'".
+
+Drop both derivations when nixpkgs's flutter builder grows worker
+ emcc support upstream.
+
+Mount point is `chat.<swarm-domain>/` (the `gatewayHost` vhost);
+upstream `--base-href "/"` is correct at sub-domain root, no override.