24775845a3
mara on #755: "e.g. /agent/name should show an error page stating that the agent could not be found if missing in json or that it is not reachable if we get a connection error. we dont want a fully generic fallback, only for routes already special cased in the nginx config." Adds two static HTML pages built at deploy time via `pkgs.runCommand "hyperhive-agent-error-pages"`: - **not-found.html** — served when `/agent/<unknown>/...` hits the bare `/agent/` catch-all. The catch-all `return 404`s, and `error_page 404 = /__hive_agent_not_found` rewrites to the static page. - **unreachable.html** — served when `/agent/<known>/...` proxy_pass to the harness returns 502 / 503 / 504. `proxy_intercept_errors on` + `error_page 502 503 504 = /__hive_agent_unreachable` on each per-agent location block rewrites to the static page. Mechanics: - `agentErrorPagesDir` (in the `let` block) is a `runCommand` that emits two HTML files using a `<<EOF` heredoc — no template engine needed. - Two `internal` nginx locations (`= /__hive_agent_not_found`, `= /__hive_agent_unreachable`) `alias` the exact files. `internal` keeps the URIs unreachable from direct operator request — only nginx's own error-handling can hit them. - Per-agent location blocks pick up the `error_page` directive through the existing `lib.mapAttrs'` over `agentPortsTable`. No per-agent generated content; same static page for all. - `/agent/` catch-all generates from a tiny optionalAttrs alongside the per-agent block — both are no-op when the agent table is empty (matches the pre-#15 shape). Pages: minimal inline CSS, catppuccin palette matching the dashboard (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` not-found heading, `#f9e2af` unreachable heading). No frontend-dist dependency — render even when hive-c0re is down. Both link back to `/`. Per mara's "only for routes already special cased" — scope stays narrow. Forge / matrix / fluffychat keep nginx defaults; extending the custom-error pattern to other vhosts is a separate follow-up if/when needed. Verified: - nginx location attrset has `["/", "/agent/", "= /__hive_agent_not_found", "= /__hive_agent_unreachable"]` - container toplevel builds clean (`nixos-system-hive-gateway-26.05pre-git`) - `docs/gateway.md::Per-agent error pages` section captures the design + rationale + intentional narrowness Closes #755.
167 lines
9.5 KiB
Markdown
167 lines
9.5 KiB
Markdown
# hive-gateway
|
|
|
|
Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.
|
|
|
|
## Vhost map
|
|
|
|
| URL | vhost | upstream | source |
|
|
| --- | --- | --- | --- |
|
|
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
|
|
| `<hive>/agent/<name>/` | `_` | per-agent harness on `agent_web_port(name)` | `agentPortsFile` JSON, #15 |
|
|
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null`, #660 / #747 |
|
|
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable`, #772 |
|
|
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway`, #754 |
|
|
| `matrix.<hive>/_matrix/*` | `matrix.<hive>` | tuwunel (`8008`) | `matrix.gatewayHost != null`, #764 |
|
|
| `matrix.<hive>/` | `matrix.<hive>` | fluffychat-web static | `matrix.gui.enable`, #772 |
|
|
| `matrix.<hive>/config.json` | `matrix.<hive>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null`, #736 |
|
|
|
|
Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware (iris #731). External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.
|
|
|
|
## Discovery flow (matrix)
|
|
|
|
Operator points client at `<hive>`. Sequence:
|
|
|
|
1. Client fetches `http://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"http://matrix.<hive>"}}` (no port suffix when gateway listens on 80).
|
|
2. Client connects to `matrix.<hive>/_matrix/client/...`.
|
|
3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.
|
|
|
|
Federation peers fetch `.well-known/matrix/server` → `{"m.server":"matrix.<hive>"}` and connect to `matrix.<hive>:8448` per spec default. Gateway only listens on configured `port`; cross-hive federation needs either an SRV record (`_matrix._tcp.matrix.<hive>` → port 80) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.
|
|
|
|
## SPA fallback (Accept-header pattern)
|
|
|
|
The `<hive>` catch-all and the `matrix.<hive>` vhost both serve a flutter SPA (per-agent UI, fluffychat). Two requirements collide:
|
|
|
|
- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
|
|
- missing assets must surface as 404, not as HTML with wrong content-type (the original #643 bug)
|
|
|
|
Solution: an `nginx http`-context `map $http_accept $matrix_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; asset fetches (`Accept: image/*`, `*/*`, etc.) get a sentinel nonexistent path → `try_files` falls through to `=404`. No extension allowlist, no `if` block, no regex heuristics. #686 + #729 thread for the design history.
|
|
|
|
## Local dev (`localHostsEntry`)
|
|
|
|
`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:
|
|
|
|
- `<hive-domain>` → `127.0.0.1`
|
|
- `forge.<hive>` → `127.0.0.1` (when forge.behindGateway)
|
|
- `matrix.<hive>` → `127.0.0.1` (when matrix.gatewayHost set)
|
|
|
|
`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.
|
|
|
|
## Sub-domain shape (rationale)
|
|
|
|
mara verdict at #749:9609 + #747:9722: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.
|
|
|
|
- forgejo's default `ROOT_URL = http://<host>/` works without any `X-Forwarded-Prefix` gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
|
|
- matrix-spec deployments universally use `matrix.<server_name>` for the actual API listener — federation already expects this.
|
|
- per-agent UIs are hyperhive-internal; iris's #731 made them base-path-aware specifically for `/agent/<name>/`. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
|
|
- cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.
|
|
|
|
`services.hyperhive.{forge.domain,matrix.gatewayHost}` take the full hostname (`forge.darkest.space`, `git.example.com`) rather than a label that gets concatenated with hive-domain — mara on #754:9684 wanted operator control over the full shape, not a forced `<label>.<hive-domain>` pattern.
|
|
|
|
## Tuning knobs
|
|
|
|
Per-vhost timeouts + body-size limits live in the location blocks:
|
|
|
|
- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), `proxyWebsockets = true` (live-update endpoints).
|
|
- matrix `/_matrix/` (tuwunel): `client_max_body_size 50M` (media uploads), `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*` (federation + cross-origin clients), `proxyWebsockets = true`.
|
|
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE / WebSocket dashboards), `proxyWebsockets = true`, `X-Forwarded-Prefix` set so the harness can build absolute URLs when relative isn't enough.
|
|
|
|
SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not HTTP-over-nginx.
|
|
|
|
## Sequencing history
|
|
|
|
- #15 v0 (per-agent routing, #740) — first sub-app behind the gateway, JSON port table from c0re.
|
|
- #686 / #729 — Accept-header SPA fallback pattern.
|
|
- #749 / #754 — forge to sub-domain (mara: sub-domain over sub-path).
|
|
- #747 / #764 — matrix sub-domain vhost + `.well-known` delegation.
|
|
- #772 / #775 — fluffychat hops from `<hive>/matrix/` to `matrix.<hive>/`.
|
|
|
|
Next-up tracked separately: #14 (container netns isolation), TLS (#594).
|
|
|
|
## Firewall posture (host-level)
|
|
|
|
`hive-c0re.nix` opens the per-agent web-port range
|
|
`8100..8999` in the host firewall **only when
|
|
`services.hyperhive.gateway.enable = false`**. With the gateway on
|
|
(default), it's the sole external entry point and proxies to
|
|
`127.0.0.1:<port>` internally — leaving the per-agent ports
|
|
firewall-open would defeat the single-front-door story (closes
|
|
#621).
|
|
|
|
Manager hashes into the same range since #753 (no more
|
|
"manager pinned at 8000" special case), so one range opening covers
|
|
every container.
|
|
|
|
The dashboard port (`cfg.dashboardPort`, default 7000) is *not*
|
|
listed in either case — since #652 it binds `127.0.0.1` only, so a
|
|
firewall hole would be a no-op. Remote dashboard access flows
|
|
through the gateway. Operators who opt out of the gateway lose
|
|
external dashboard reach by design — the surface is privileged
|
|
(approve / deny / destroy) and must not be exposed without a real
|
|
reverse proxy in front.
|
|
|
|
## `HIVE_FORGE_URL`: loopback for in-cluster, sub-domain for the operator
|
|
|
|
Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
|
|
`hive-forge` calls against it. `hive-c0re.nix` pins this to
|
|
`http://127.0.0.1:<forge.httpPort>` for the in-cluster path: every
|
|
agent container shares the host's network namespace, so loopback
|
|
reaches the forge container directly with no DNS lookup needed
|
|
(closes #761).
|
|
|
|
The post-#754 sub-domain default (`forge.<hive-domain>`) is for
|
|
**operator browsers + cross-host clients**, not in-cluster traffic.
|
|
Using the sub-domain URL inside agent containers would fail every
|
|
`hive-forge` invocation with "Name or service not known" — the
|
|
agent's nspawn doesn't have DNS for the external hostname.
|
|
|
|
## hive-forge container shape
|
|
|
|
Private Forgejo wrapped in a nixos-container (`hive-forge`, not
|
|
`h-*` — keeps c0re's lifecycle scanner out of the picture; the
|
|
operator manages it via the standard `nixos-container` CLI). The
|
|
container also keeps hive-forge from fighting any `services.forgejo`
|
|
the operator already runs on the host — separate systemd namespace,
|
|
separate state dir, separate port unless the operator deliberately
|
|
collides.
|
|
|
|
Container shares the host network namespace
|
|
(`privateNetwork = false`) so agents reach the forge at
|
|
`http://localhost:<httpPort>` without extra plumbing — nixos-container
|
|
is here for state + systemd-unit isolation, not network isolation.
|
|
|
|
State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/`
|
|
and survives container restart / host reboot. To wipe, destroy the
|
|
container.
|
|
|
|
## Per-agent error pages
|
|
|
|
`/agent/<name>/` requests hit two failure modes; both get static
|
|
HTML pages instead of nginx's default error chrome (#755):
|
|
|
|
- **Agent not found** (`/agent/<unknown>/...`) — name isn't in
|
|
`agentPortsTable`. nginx's prefix match falls back to the bare
|
|
`/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
|
|
to `/__hive_agent_not_found` → serves `not-found.html` with a link
|
|
back to the dashboard.
|
|
|
|
- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
|
|
per-agent harness isn't responding (container restarting, crash
|
|
recovery, etc.). `proxy_intercept_errors on` + `error_page 502 503
|
|
504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.
|
|
|
|
Both pages are built at deploy time via `pkgs.runCommand` (one nix
|
|
derivation `hyperhive-agent-error-pages` with `not-found.html` +
|
|
`unreachable.html` inside) and served via two `internal` nginx
|
|
locations with `alias` to the exact file. `internal` keeps the
|
|
files from being directly request-able by operators — only nginx's
|
|
own error-handling can reach them.
|
|
|
|
Page styling: minimal inline CSS matching the dashboard's catppuccin
|
|
palette (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` heading). No
|
|
dependencies on the frontend dist — these pages render even when
|
|
hive-c0re itself is down.
|
|
|
|
Scope is intentionally narrow per mara on #755: "only for routes
|
|
already special cased in the nginx config". Other gateway routes
|
|
(forge / matrix / fluffychat) get nginx defaults — extending the
|
|
custom-error pattern there is a separate follow-up.
|