Watch

atlas 24775845a3 nix/hive-gateway: static not-found + unreachable pages for /agent/<name>/ (#755 )

mara on #755: "e.g. /agent/name should show an error page stating
that the agent could not be found if missing in json or that it is
not reachable if we get a connection error. we dont want a fully
generic fallback, only for routes already special cased in the
nginx config."

Adds two static HTML pages built at deploy time via
`pkgs.runCommand "hyperhive-agent-error-pages"`:

- **not-found.html** — served when `/agent/<unknown>/...` hits the
  bare `/agent/` catch-all. The catch-all `return 404`s, and
  `error_page 404 = /__hive_agent_not_found` rewrites to the static
  page.
- **unreachable.html** — served when `/agent/<known>/...` proxy_pass
  to the harness returns 502 / 503 / 504. `proxy_intercept_errors
  on` + `error_page 502 503 504 = /__hive_agent_unreachable` on each
  per-agent location block rewrites to the static page.

Mechanics:

- `agentErrorPagesDir` (in the `let` block) is a `runCommand` that
  emits two HTML files using a `<<EOF` heredoc — no template engine
  needed.
- Two `internal` nginx locations (`= /__hive_agent_not_found`,
  `= /__hive_agent_unreachable`) `alias` the exact files. `internal`
  keeps the URIs unreachable from direct operator request — only
  nginx's own error-handling can hit them.
- Per-agent location blocks pick up the `error_page` directive
  through the existing `lib.mapAttrs'` over `agentPortsTable`. No
  per-agent generated content; same static page for all.
- `/agent/` catch-all generates from a tiny optionalAttrs alongside
  the per-agent block — both are no-op when the agent table is
  empty (matches the pre-#15 shape).

Pages: minimal inline CSS, catppuccin palette matching the
dashboard (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` not-found heading,
`#f9e2af` unreachable heading). No frontend-dist dependency — render
even when hive-c0re is down. Both link back to `/`.

Per mara's "only for routes already special cased" — scope stays
narrow. Forge / matrix / fluffychat keep nginx defaults; extending
the custom-error pattern to other vhosts is a separate follow-up
if/when needed.

Verified:
- nginx location attrset has `["/", "/agent/", "= /__hive_agent_not_found", "= /__hive_agent_unreachable"]`
- container toplevel builds clean (`nixos-system-hive-gateway-26.05pre-git`)
- `docs/gateway.md::Per-agent error pages` section captures the
  design + rationale + intentional narrowness

Closes #755.

2026-05-31 15:04:54 +02:00

9.5 KiB

Raw Blame History

hive-gateway

Single nginx in front of every hyperhive web surface. Container hive-gateway, shared host netns, system-config (not meta-flake managed). Configured via services.hyperhive.gateway.* + per-subsystem opt-in flags in services.hyperhive.{forge,matrix,...}.

Vhost map

URL	vhost	upstream	source
`<hive>/`	`_` (catch-all)	hive-c0re dashboard (`7000`)	always
`<hive>/agent/<name>/`	`_`	per-agent harness on `agent_web_port(name)`	`agentPortsFile` JSON, #15
`<hive>/.well-known/matrix/{client,server}`	`_`	inline JSON (no upstream)	`matrix.enable && domain != null`, #660 / #747
`<hive>/matrix/` (deprecated)	`_`	301 → `matrix.<hive>/`	`matrix.gui.enable`, #772
`forge.<hive>/`	`forge.<hive>`	forgejo (`3000`)	`forge.behindGateway`, #754
`matrix.<hive>/_matrix/*`	`matrix.<hive>`	tuwunel (`8008`)	`matrix.gatewayHost != null`, #764
`matrix.<hive>/`	`matrix.<hive>`	fluffychat-web static	`matrix.gui.enable`, #772
`matrix.<hive>/config.json`	`matrix.<hive>`	inline JSON (FluffyChat boot config)	`matrix.gui.enable && domain != null`, #736

Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware (iris #731). External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.

Discovery flow (matrix)

Operator points client at <hive>. Sequence:

Client fetches http://<hive>/.well-known/matrix/client → {"m.homeserver":{"base_url":"http://matrix.<hive>"}} (no port suffix when gateway listens on 80).
Client connects to matrix.<hive>/_matrix/client/....
Gateway routes /_matrix/* → tuwunel at 127.0.0.1:8008.

Federation peers fetch .well-known/matrix/server → {"m.server":"matrix.<hive>"} and connect to matrix.<hive>:8448 per spec default. Gateway only listens on configured port; cross-hive federation needs either an SRV record (_matrix._tcp.matrix.<hive> → port 80) OR matrix.openFirewall = true so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.

SPA fallback (Accept-header pattern)

The <hive> catch-all and the matrix.<hive> vhost both serve a flutter SPA (per-agent UI, fluffychat). Two requirements collide:

hard-refresh on a sub-route must serve index.html (SPA's client-side router takes over after JS bootstrap)
missing assets must surface as 404, not as HTML with wrong content-type (the original #643 bug)

Solution: an nginx http-context map $http_accept $matrix_spa_target { ... } keyed on the request's Accept header. Browser navigations (Accept: text/html,...) get index.html; asset fetches (Accept: image/*, */*, etc.) get a sentinel nonexistent path → try_files falls through to =404. No extension allowlist, no if block, no regex heuristics. #686 + #729 thread for the design history.

Local dev (`localHostsEntry`)

services.hyperhive.gateway.localHostsEntry = true adds entries to the host's /etc/hosts:

<hive-domain> → 127.0.0.1
forge.<hive> → 127.0.0.1 (when forge.behindGateway)
matrix.<hive> → 127.0.0.1 (when matrix.gatewayHost set)

lib.unique de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.

Sub-domain shape (rationale)

mara verdict at #749:9609 + #747:9722: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.

forgejo's default ROOT_URL = http://<host>/ works without any X-Forwarded-Prefix gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
matrix-spec deployments universally use matrix.<server_name> for the actual API listener — federation already expects this.
per-agent UIs are hyperhive-internal; iris's #731 made them base-path-aware specifically for /agent/<name>/. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.

services.hyperhive.{forge.domain,matrix.gatewayHost} take the full hostname (forge.darkest.space, git.example.com) rather than a label that gets concatenated with hive-domain — mara on #754:9684 wanted operator control over the full shape, not a forced <label>.<hive-domain> pattern.

Tuning knobs

Per-vhost timeouts + body-size limits live in the location blocks:

forge / (forgejo): client_max_body_size 1G (LFS), proxy_read_timeout 1h (multi-GB clones), proxyWebsockets = true (live-update endpoints).
matrix /_matrix/ (tuwunel): client_max_body_size 50M (media uploads), proxy_read_timeout 1h (long-poll /sync), CORS * (federation + cross-origin clients), proxyWebsockets = true.
per-agent /agent/<name>/: proxy_read_timeout 1d (long-lived SSE / WebSocket dashboards), proxyWebsockets = true, X-Forwarded-Prefix set so the harness can build absolute URLs when relative isn't enough.

SSH for forge stays direct on cfg.sshPort — separate listener protocol, not HTTP-over-nginx.

Sequencing history

#15 v0 (per-agent routing, #740) — first sub-app behind the gateway, JSON port table from c0re.
#686 / #729 — Accept-header SPA fallback pattern.
#749 / #754 — forge to sub-domain (mara: sub-domain over sub-path).
#747 / #764 — matrix sub-domain vhost + .well-known delegation.
#772 / #775 — fluffychat hops from <hive>/matrix/ to matrix.<hive>/.

Next-up tracked separately: #14 (container netns isolation), TLS (#594).

Firewall posture (host-level)

hive-c0re.nix opens the per-agent web-port range 8100..8999 in the host firewall only when services.hyperhive.gateway.enable = false. With the gateway on (default), it's the sole external entry point and proxies to 127.0.0.1:<port> internally — leaving the per-agent ports firewall-open would defeat the single-front-door story (closes #621).

Manager hashes into the same range since #753 (no more "manager pinned at 8000" special case), so one range opening covers every container.

The dashboard port (cfg.dashboardPort, default 7000) is not listed in either case — since #652 it binds 127.0.0.1 only, so a firewall hole would be a no-op. Remote dashboard access flows through the gateway. Operators who opt out of the gateway lose external dashboard reach by design — the surface is privileged (approve / deny / destroy) and must not be exposed without a real reverse proxy in front.

`HIVE_FORGE_URL`: loopback for in-cluster, sub-domain for the operator

Agents poll HIVE_FORGE_URL for Forgejo notifications + run all hive-forge calls against it. hive-c0re.nix pins this to http://127.0.0.1:<forge.httpPort> for the in-cluster path: every agent container shares the host's network namespace, so loopback reaches the forge container directly with no DNS lookup needed (closes #761).

The post-#754 sub-domain default (forge.<hive-domain>) is for operator browsers + cross-host clients, not in-cluster traffic. Using the sub-domain URL inside agent containers would fail every hive-forge invocation with "Name or service not known" — the agent's nspawn doesn't have DNS for the external hostname.

hive-forge container shape

Private Forgejo wrapped in a nixos-container (hive-forge, not h-* — keeps c0re's lifecycle scanner out of the picture; the operator manages it via the standard nixos-container CLI). The container also keeps hive-forge from fighting any services.forgejo the operator already runs on the host — separate systemd namespace, separate state dir, separate port unless the operator deliberately collides.

Container shares the host network namespace (privateNetwork = false) so agents reach the forge at http://localhost:<httpPort> without extra plumbing — nixos-container is here for state + systemd-unit isolation, not network isolation.

State lives at /var/lib/nixos-containers/hive-forge/var/lib/forgejo/ and survives container restart / host reboot. To wipe, destroy the container.

Per-agent error pages

/agent/<name>/ requests hit two failure modes; both get static HTML pages instead of nginx's default error chrome (#755):

Agent not found (/agent/<unknown>/...) — name isn't in agentPortsTable. nginx's prefix match falls back to the bare /agent/ catch-all, which return 404s and error_page 404 rewrites to /__hive_agent_not_found → serves not-found.html with a link back to the dashboard.
Agent unreachable (502 / 503 / 504 from proxy_pass) — the per-agent harness isn't responding (container restarting, crash recovery, etc.). proxy_intercept_errors on + error_page 502 503 504 = /__hive_agent_unreachable rewrites to unreachable.html.

Both pages are built at deploy time via pkgs.runCommand (one nix derivation hyperhive-agent-error-pages with not-found.html + unreachable.html inside) and served via two internal nginx locations with alias to the exact file. internal keeps the files from being directly request-able by operators — only nginx's own error-handling can reach them.

Page styling: minimal inline CSS matching the dashboard's catppuccin palette (#1e1e2e bg, #cdd6f4 text, #cba6f7 heading). No dependencies on the frontend dist — these pages render even when hive-c0re itself is down.

Scope is intentionally narrow per mara on #755: "only for routes already special cased in the nginx config". Other gateway routes (forge / matrix / fluffychat) get nginx defaults — extending the custom-error pattern there is a separate follow-up.

9.5 KiB Raw Blame History