hyperhive/docs/gateway.md

# hive-gateway

Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.

## Vhost map

| URL | vhost | upstream | source |
| --- | --- | --- | --- |
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
| `<hive>/agent/<name>/` | `_` | per-agent harness on `agent_web_port(name)` | `agentPortsFile` JSON, #15 |
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null`, #660 / #747 |
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable`, #772 |
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway`, #754 |
| `matrix.<hive>/_matrix/*` | `matrix.<hive>` | tuwunel (`8008`) | `matrix.gatewayHost != null`, #764 |
| `matrix.<hive>/` | `matrix.<hive>` | fluffychat-web static | `matrix.gui.enable`, #772 |
| `matrix.<hive>/config.json` | `matrix.<hive>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null`, #736 |

Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware (iris #731). External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.

## Discovery flow (matrix)

Operator points client at `<hive>`. Sequence:

1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://matrix.<hive>"}}` (no port suffix when gateway listens on 443). With `selfSignedTls = false` the scheme drops to http and the port suffix reflects the bare `port` instead.
2. Client connects to `matrix.<hive>/_matrix/client/...`.
3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.

matrix-dart-sdk (FluffyChat etc.) hardcodes `https` for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.

Federation peers fetch `.well-known/matrix/server` → `{"m.server":"matrix.<hive>"}` and connect to `matrix.<hive>:8448` per spec default. Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.matrix.<hive>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.

## SPA fallback (Accept-header pattern)

The `<hive>` catch-all and the `matrix.<hive>` vhost both serve a flutter SPA (per-agent UI, fluffychat). Two requirements collide:

- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
- missing assets must surface as 404, not as HTML with wrong content-type (the original #643 bug)

Solution: an `nginx http`-context `map $http_accept $matrix_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; asset fetches (`Accept: image/*`, `*/*`, etc.) get a sentinel nonexistent path → `try_files` falls through to `=404`. No extension allowlist, no `if` block, no regex heuristics. #686 + #729 thread for the design history.

## Local dev (`localHostsEntry`)

`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:

- `<hive-domain>` → `127.0.0.1`
- `forge.<hive>` → `127.0.0.1` (when forge.behindGateway)
- `matrix.<hive>` → `127.0.0.1` (when matrix.gatewayHost set)

`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.

## Sub-domain shape (rationale)

mara verdict at #749:9609 + #747:9722: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.

- forgejo's default `ROOT_URL = http://<host>/` works without any `X-Forwarded-Prefix` gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
- matrix-spec deployments universally use `matrix.<server_name>` for the actual API listener — federation already expects this.
- per-agent UIs are hyperhive-internal; iris's #731 made them base-path-aware specifically for `/agent/<name>/`. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
- cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.

`services.hyperhive.{forge.domain,matrix.gatewayHost}` take the full hostname (`forge.darkest.space`, `git.example.com`) rather than a label that gets concatenated with hive-domain — mara on #754:9684 wanted operator control over the full shape, not a forced `<label>.<hive-domain>` pattern.

## Tuning knobs

Per-vhost timeouts + body-size limits live in the location blocks:

- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), `proxyWebsockets = true` (live-update endpoints).
- matrix `/_matrix/` (tuwunel): `client_max_body_size 50M` (media uploads), `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*` (federation + cross-origin clients), `proxyWebsockets = true`.
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE / WebSocket dashboards), `proxyWebsockets = true`, `X-Forwarded-Prefix` set so the harness can build absolute URLs when relative isn't enough.

SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not HTTP-over-nginx.

## Per-agent unix-socket upstream (#784)

Sub-agent `/agent/<name>/` upstreams flip from TCP loopback to a
unix-domain socket as each agent opts in. The mechanism:

1. **Agent side** (`hyperhive.web.useUnixSocket = true` in
   `agent.nix`, #815). Sets `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
   on the harness service env; `web_ui::serve` binds a `UnixListener`
   at that path instead of TCP.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
   (`/run/hive-agent/<name>/`) into the agent's container (#813). Dir
   bind, not file bind — file bind-mounts don't survive the
   harness's `unlink + bind(2)` cycle on socket replace. Per-agent
   subdir keeps each agent's container blind to siblings'
   sockets (mara on #800).
3. **Marker gate**. After successful `bind_unix`, the harness drops
   `<dir>/.bound` next to the socket. c0re's `agent_sockets::write`
   filters its JSON map by marker presence — only agents whose
   harness has actually bound the socket appear there (#784 atlas
   gate). Without this filter, the gateway would `proxy_pass` to a
   non-existent socket for every sub-agent that hasn't opted in yet.
4. **Gateway side** (#829). Reads `agent-sockets.json` at
   request-handling time and routes `/agent/<name>/` to
   `http://unix:/run/hive-agent/<name>/web.sock:/`. Whole
   `/run/hive-agent/` is bind-mounted read-only into the gateway
   container so it can reach every published socket.

c0re re-fires `agent_sockets::write` every 10s so newly-bound
markers get picked up without needing a container-start hook in
every lifecycle path. `write()` is idempotent: steady-state cost is
one stat per agent per tick.

Transition: agents that haven't flipped `useUnixSocket = true` still
appear in `agent-ports.json` (the legacy TCP map) and the gateway
falls back to TCP for them. Step 4 of #784 will drop the TCP map +
the harness's TCP bind once every agent's flipped.

## Dashboard link shape (gateway vs direct)

When the gateway is in front, the SW4RM tab builds per-agent links
as same-origin `/agent/<name>/…` URLs instead of the legacy direct
`http://<host>:<container.port>/` TCP shape. The signal comes from
`StateSnapshot.gateway_enabled`, sourced from the
`HIVE_GATEWAY_ENABLED` env the c0re NixOS module sets when
`services.hyperhive.gateway.enable = true`. Three render sites
flip together: the primary agent-name link, the favicon fetch
(`<url>/icon`), and the nav-strip `container`-kind links from
`/api/agent/<name>/links`. `forge`-kind nav-strip links still
resolve against `http://<host>:3000` (separate sub-domain transition
tracked by `forge.behindGateway`); `external`-kind links are
already absolute. See `docs/web-ui.md::Container row` for the
frontend-side derivation.

## Sequencing history

- #15 v0 (per-agent routing, #740) — first sub-app behind the gateway, JSON port table from c0re.
- #686 / #729 — Accept-header SPA fallback pattern.
- #749 / #754 — forge to sub-domain (mara: sub-domain over sub-path).
- #747 / #764 — matrix sub-domain vhost + `.well-known` delegation.
- #772 / #775 — fluffychat hops from `<hive>/matrix/` to `matrix.<hive>/`.
- #784 / #800 / #813 / #815 / #822 / #829 — sub-agent UI flips to unix-domain socket upstream, opt-in per agent.

Next-up tracked separately: #14 (container netns isolation), TLS (#594).

## Self-signed TLS (`selfSignedTls`)

On by default since #837. The gateway generates a self-signed RSA-4096 cert at first boot (10-year validity) and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).

**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap (#837). For headless agent traffic and operator dashboard reach plain http is fine, so the http listen stays in parallel — operators can keep using `http://<hive>/` from the dashboard if they don't care about the cert prompt.

**Cert shape**: subject CN = bare hive domain; subjectAltName covers `<hive>` + wildcard `*.<hive>` so all current and future sub-domain vhosts (matrix, forge, ...) validate under the same cert. Stored at `/var/lib/hive-gateway/tls/{cert,key}.pem` inside the gateway container (`ephemeral = false`, so persisted across container restart).

**Regeneration**: the generator unit (`hive-gateway-self-signed-cert.service`) is gated by `ConditionPathExists=!cert.pem`, so it's a one-shot. To rotate (e.g. cert leak, expiry approaching), delete `cert.pem` inside the gateway container and restart `nginx.service` — the generator runs as a `Before=` dependency. No automatic rotation; the cert is purely a workaround for the well-known fetch.

**Production**: operators fronting hyperhive with a real reverse proxy (caddy + ACME, traefik + Let's Encrypt, etc.) set `selfSignedTls = false`. The proxy handles termination upstream, the gateway listens on `port` only.

**Cert prompts**: browsers warn once per host on first visit. With the wildcard SAN, `https://<hive>/`, `https://matrix.<hive>/`, and `https://forge.<hive>/` are covered by the same cert, but the browser still prompts per origin (per-host security state). FluffyChat needs the user to accept both `matrix.<hive>` (the SPA itself) and `<hive>` (the well-known fetch endpoint).

**`.well-known/matrix/{client,server}` scheme**: switches to `https` when `selfSignedTls` is on, so matrix-dart-sdk doesn't downgrade and tuwunel federation peers get a TLS-fronted base_url. The legacy `/matrix/*` 301 redirect also flips to https.

## Firewall posture (host-level)

`hive-c0re.nix` opens the per-agent web-port range
`8100..8999` in the host firewall **only when
`services.hyperhive.gateway.enable = false`**. With the gateway on
(default), it's the sole external entry point and proxies to
`127.0.0.1:<port>` internally — leaving the per-agent ports
firewall-open would defeat the single-front-door story (closes
#621).

`services.hyperhive.gateway.openFirewall = true` opens `port` plus
`httpsPort` when `selfSignedTls = true` (default). Operators who
flip `selfSignedTls = false` to front the gateway with a real
TLS-terminating reverse proxy on the host get only `port` opened.

Manager hashes into the same range since #753 (no more
"manager pinned at 8000" special case), so one range opening covers
every container.

The dashboard port (`cfg.dashboardPort`, default 7000) is *not*
listed in either case — since #652 it binds `127.0.0.1` only, so a
firewall hole would be a no-op. Remote dashboard access flows
through the gateway. Operators who opt out of the gateway lose
external dashboard reach by design — the surface is privileged
(approve / deny / destroy) and must not be exposed without a real
reverse proxy in front.

## `HIVE_FORGE_URL`: loopback for in-cluster, sub-domain for the operator

Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
`hive-forge` calls against it. `hive-c0re.nix` pins this to
`http://127.0.0.1:<forge.httpPort>` for the in-cluster path: every
agent container shares the host's network namespace, so loopback
reaches the forge container directly with no DNS lookup needed
(closes #761).

The post-#754 sub-domain default (`forge.<hive-domain>`) is for
**operator browsers + cross-host clients**, not in-cluster traffic.
Using the sub-domain URL inside agent containers would fail every
`hive-forge` invocation with "Name or service not known" — the
agent's nspawn doesn't have DNS for the external hostname.

## hive-forge container shape

Private Forgejo wrapped in a nixos-container (`hive-forge`, not
`h-*` — keeps c0re's lifecycle scanner out of the picture; the
operator manages it via the standard `nixos-container` CLI). The
container also keeps hive-forge from fighting any `services.forgejo`
the operator already runs on the host — separate systemd namespace,
separate state dir, separate port unless the operator deliberately
collides.

Container shares the host network namespace
(`privateNetwork = false`) so agents reach the forge at
`http://localhost:<httpPort>` without extra plumbing — nixos-container
is here for state + systemd-unit isolation, not network isolation.

State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/`
and survives container restart / host reboot. To wipe, destroy the
container.

## Per-agent error pages

`/agent/<name>/` requests hit two failure modes; both get static
HTML pages instead of nginx's default error chrome (#755):

- **Agent not found** (`/agent/<unknown>/...`) — name isn't in
  `agentPortsTable`. nginx's prefix match falls back to the bare
  `/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
  to `/__hive_agent_not_found` → serves `not-found.html` with a link
  back to the dashboard.

- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
  per-agent harness isn't responding (container restarting, crash
  recovery, etc.). `proxy_intercept_errors on` + `error_page 502 503
  504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.

Both pages are built at deploy time via `pkgs.runCommand` (one nix
derivation `hyperhive-agent-error-pages` with `not-found.html` +
`unreachable.html` inside) and served via two `internal` nginx
locations with `alias` to the exact file. `internal` keeps the
files from being directly request-able by operators — only nginx's
own error-handling can reach them.

Page styling: minimal inline CSS matching the dashboard's catppuccin
palette (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` heading). No
dependencies on the frontend dist — these pages render even when
hive-c0re itself is down.

Scope is intentionally narrow per mara on #755: "only for routes
already special cased in the nginx config". Other gateway routes
(forge / matrix / fluffychat) get nginx defaults — extending the
custom-error pattern there is a separate follow-up.