Watch

damocles b83edc40c6 fix(#970,#973): retry gateway reload on failure; always enable HIVE_WEB_SOCKET

2026-06-01 18:18:45 +02:00

19 KiB

Raw Blame History

hive-gateway

Single nginx in front of every hyperhive web surface. Container hive-gateway, shared host netns, system-config (not meta-flake managed). Configured via services.hyperhive.gateway.* + per-subsystem opt-in flags in services.hyperhive.{forge,matrix,...}.

Vhost map

URL	vhost	upstream	source
`<hive>/`	`_` (catch-all)	hive-c0re dashboard (`7000`)	always
`<hive>/agent/<name>/`	`_`	per-agent harness (UDS or TCP)	`agents.conf` (runtime-generated)
`<hive>/.well-known/matrix/{client,server}`	`_`	inline JSON (no upstream)	`matrix.enable && domain != null`
`<hive>/matrix/` (deprecated)	`_`	301 → `matrix.<hive>/`	`matrix.gui.enable`
`forge.<hive>/`	`forge.<hive>`	forgejo (`3000`)	`forge.behindGateway`
`matrix.<hive>/_matrix/*`	`matrix.<hive>`	tuwunel (`8008`)	`matrix.gatewayHost != null`
`matrix.<hive>/`	`matrix.<hive>`	fluffychat-web static	`matrix.gui.enable`
`matrix.<hive>/config.json`	`matrix.<hive>`	inline JSON (FluffyChat boot config)	`matrix.gui.enable && domain != null`

Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware. External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.

Discovery flow (matrix)

Operator points client at <hive>. Sequence:

Client fetches https://<hive>/.well-known/matrix/client → {"m.homeserver":{"base_url":"https://matrix.<hive>"}} (no port suffix when gateway listens on 443). With selfSignedTls = false the scheme drops to http and the port suffix reflects the bare port instead.
Client connects to matrix.<hive>/_matrix/client/....
Gateway routes /_matrix/* → tuwunel at 127.0.0.1:8008.

matrix-dart-sdk (FluffyChat etc.) hardcodes https for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.

Federation peers fetch .well-known/matrix/server → {"m.server":"matrix.<hive>"} and connect to matrix.<hive>:8448 per spec default. Gateway only listens on configured port (+ httpsPort when TLS on); cross-hive federation needs either an SRV record (_matrix._tcp.matrix.<hive> → port 80 / 443) OR matrix.openFirewall = true so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.

SPA fallback (Accept-header pattern)

The <hive> catch-all and the matrix.<hive> vhost both serve a flutter SPA (per-agent UI, fluffychat). Two requirements collide:

hard-refresh on a sub-route must serve index.html (SPA's client-side router takes over after JS bootstrap)
missing assets must surface as 404, not as HTML with wrong content-type

Solution: an nginx http-context map $http_accept $matrix_spa_target { ... } keyed on the request's Accept header. Browser navigations (Accept: text/html,...) get index.html; asset fetches (Accept: image/*, */*, etc.) get a sentinel nonexistent path → try_files falls through to =404. No extension allowlist, no if block, no regex heuristics.

Local dev (`localHostsEntry`)

services.hyperhive.gateway.localHostsEntry = true adds entries to the host's /etc/hosts:

<hive-domain> → 127.0.0.1
forge.<hive> → 127.0.0.1 (when forge.behindGateway)
matrix.<hive> → 127.0.0.1 (when matrix.gatewayHost set)

lib.unique de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.

Sub-domain shape (rationale)

Operator decision: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.

forgejo's default ROOT_URL = http://<host>/ works without any X-Forwarded-Prefix gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
matrix-spec deployments universally use matrix.<server_name> for the actual API listener — federation already expects this.
per-agent UIs are hyperhive-internal and base-path-aware specifically for /agent/<name>/. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.

services.hyperhive.{forge.domain,matrix.gatewayHost} take the full hostname (forge.darkest.space, git.example.com) rather than a label that gets concatenated with hive-domain — operators want control over the full shape, not a forced <label>.<hive-domain> pattern.

Tuning knobs

Per-vhost timeouts + body-size limits live in the location blocks:

forge / (forgejo): client_max_body_size 1G (LFS), proxy_read_timeout 1h (multi-GB clones), proxyWebsockets = true (live-update endpoints).
matrix /_matrix/ (tuwunel): client_max_body_size 50M (media uploads), proxy_read_timeout 1h (long-poll /sync), CORS * (federation + cross-origin clients), proxyWebsockets = true.
per-agent /agent/<name>/: proxy_read_timeout 1d (long-lived SSE / WebSocket dashboards), proxyWebsockets = true, X-Forwarded-Prefix set so the harness can build absolute URLs when relative isn't enough.

SSH for forge stays direct on cfg.sshPort — separate listener protocol, not HTTP-over-nginx.

Per-agent unix-socket upstream

All agents bind their web UI on a unix-domain socket at /run/hive-agent/<name>/web.sock — the HIVE_WEB_SOCKET env var is now set unconditionally for every agent. The mechanism:

Agent side. HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock is set on every harness service env; web_ui::serve binds a UnixListener at that path. The deprecated hyperhive.web.useUnixSocket option is now a no-op.
Host side. hive-c0re bind-mounts the per-agent subdir (/run/hive-agent/<name>/) into the agent's container. Dir bind, not file bind — file bind-mounts don't survive the harness's unlink + bind(2) cycle on socket replace. Per-agent subdir keeps each agent's container blind to siblings' sockets.
Marker gate. After successful bind_unix, the harness drops <dir>/hyperhive-socket-bound next to the socket. c0re's agent_sockets::write filters its JSON map by marker presence — only agents whose harness has actually bound the socket appear there. (Legacy name .bound also accepted during the transition window.)
Gateway side. gateway_nginx::write generates /var/lib/hyperhive/gateway/agents.conf — a plain nginx include file with one location /agent/<name>/ block per agent. UDS upstream (http://unix:/run/hive-agent/<name>/web.sock:/) when hyperhive-socket-bound marker present; TCP loopback for agents that haven't yet been rebuilt under the new config. The gateway container bind-mounts /var/lib/hyperhive/gateway/ at /run/hive-state/; nginx includes /run/hive-state/agents.conf. After each write, c0re triggers nginx -s reload inside the gateway container from the HOST via systemd-run --machine=hive-gateway --wait nginx -s reload. This is intentionally host-side: IN_MOVED_TO from an atomic rename does not propagate across the nspawn mount-namespace boundary, so a path unit inside the container would never fire.

c0re regenerates agents.conf (and triggers a reload) on two triggers: every topology change (new/removed agents) and every 10s marker poll tick (agent_sockets::spawn_poll). write() is idempotent — skips the rename when content is unchanged. Failed reloads are retried automatically on subsequent poll ticks via gateway_nginx::reload_if_pending.

Agent port map (`agent-ports.json`)

/var/lib/hyperhive/agent-ports.json is a flat JSON object keyed by logical agent name → TCP web port:

{
  "iris":     8178,
  "atlas":    8304,
  "argus":    8267,
  "damocles": 8549
}

Written alongside agents.conf on every topology change. Ports come from lifecycle::agent_web_port(name) — a pure FNV-1a hash of the name, reproducible from the name alone. The manager is excluded: it always uses a unix socket (HIVE_WEB_SOCKET is unconditionally set for the manager role), so its TCP port never appears in the fallback map. The gateway routes /agent/root/ to the manager's unix socket via agents.conf alongside sub-agents.

The file doubles as a human-readable audit artifact — cat agent-ports.json shows every registered sub-agent and its deterministic port assignment. TCP loopback upstreams in agents.conf reference these ports for agents that haven't opted into unix-socket mode yet.

Both agent-ports.json and agents.conf use atomic <path>.tmp + rename() writes so a crashing c0re process never leaves a partial or unparseable file behind.

Dashboard link shape (gateway vs direct)

When the gateway is in front, the SW4RM tab builds per-agent links as same-origin /agent/<name>/… URLs instead of the legacy direct http://<host>:<container.port>/ TCP shape. The signal comes from StateSnapshot.gateway_enabled, sourced from the HIVE_GATEWAY_ENABLED env the c0re NixOS module sets when services.hyperhive.gateway.enable = true. Three render sites flip together: the primary agent-name link, the favicon fetch (<url>/icon), and the nav-strip container-kind links from /api/agent/<name>/links. forge-kind nav-strip links still resolve against http://<host>:3000 (separate sub-domain transition tracked by forge.behindGateway); external-kind links are already absolute. See docs/web-ui/dashboard.md::Container row for the frontend-side derivation.

Self-signed TLS (`selfSignedTls`)

On by default. The gateway generates a self-signed RSA-4096 cert at first boot (10-year validity) and listens on httpsPort (default 443) on every vhost beside the plain-http port (default 80).

Why on by default: matrix-dart-sdk (FluffyChat's SDK) hardcodes https://<host>/.well-known/matrix/client for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap. For headless agent traffic and operator dashboard reach plain http is fine, so the http listen stays in parallel — operators can keep using http://<hive>/ from the dashboard if they don't care about the cert prompt.

Cert shape: subject CN = bare hive domain; subjectAltName covers <hive> + wildcard *.<hive> so all current and future sub-domain vhosts (matrix, forge, ...) validate under the same cert. Stored at /var/lib/hive-gateway/tls/{cert,key}.pem inside the gateway container (ephemeral = false, so persisted across container restart).

Regeneration: the generator unit (hive-gateway-self-signed-cert.service) is gated by ConditionPathExists=!cert.pem, so it's a one-shot. To rotate (e.g. cert leak, expiry approaching), delete cert.pem inside the gateway container and restart nginx.service — the generator runs as a Before= dependency. No automatic rotation; the cert is purely a workaround for the well-known fetch.

Production: operators fronting hyperhive with a real reverse proxy (caddy + ACME, traefik + Let's Encrypt, etc.) set selfSignedTls = false. The proxy handles termination upstream, the gateway listens on port only.

Cert prompts: browsers warn once per host on first visit. With the wildcard SAN, https://<hive>/, https://matrix.<hive>/, and https://forge.<hive>/ are covered by the same cert, but the browser still prompts per origin (per-host security state). FluffyChat needs the user to accept both matrix.<hive> (the SPA itself) and <hive> (the well-known fetch endpoint).

.well-known/matrix/{client,server} scheme: switches to https when selfSignedTls is on, so matrix-dart-sdk doesn't downgrade and tuwunel federation peers get a TLS-fronted base_url. The legacy /matrix/* 301 redirect also flips to https.

Firewall posture (host-level)

hive-c0re.nix opens the per-agent web-port range 8100..8999 in the host firewall only when services.hyperhive.gateway.enable = false. With the gateway on (default), it's the sole external entry point and proxies to 127.0.0.1:<port> internally — leaving the per-agent ports firewall-open would defeat the single-front-door story.

services.hyperhive.gateway.openFirewall = true opens port plus httpsPort when selfSignedTls = true (default). Operators who flip selfSignedTls = false to front the gateway with a real TLS-terminating reverse proxy on the host get only port opened.

The manager hashes into the same port range as sub-agents (no "manager pinned at 8000" special case), so one range opening covers every container.

The dashboard port (cfg.dashboardPort, default 7000) is not listed in either case — it binds 127.0.0.1 only, so a firewall hole would be a no-op. Remote dashboard access flows through the gateway. Operators who opt out of the gateway lose external dashboard reach by design — the surface is privileged (approve / deny / destroy) and must not be exposed without a real reverse proxy in front.

`HIVE_FORGE_URL`: loopback for in-cluster, sub-domain for the operator

Agents poll HIVE_FORGE_URL for Forgejo notifications + run all hive-forge calls against it. hive-c0re.nix pins this to http://127.0.0.1:<forge.httpPort> for the in-cluster path: every agent container shares the host's network namespace, so loopback reaches the forge container directly with no DNS lookup needed.

The sub-domain default (forge.<hive-domain>) is for operator browsers + cross-host clients, not in-cluster traffic. Using the sub-domain URL inside agent containers would fail every hive-forge invocation with "Name or service not known" — the agent's nspawn doesn't have DNS for the external hostname.

hive-forge container shape

Private Forgejo wrapped in a nixos-container (hive-forge, not h-* — keeps c0re's lifecycle scanner out of the picture; the operator manages it via the standard nixos-container CLI). The container also keeps hive-forge from fighting any services.forgejo the operator already runs on the host — separate systemd namespace, separate state dir, separate port unless the operator deliberately collides.

Container shares the host network namespace (privateNetwork = false) so agents reach the forge at http://localhost:<httpPort> without extra plumbing — nixos-container is here for state + systemd-unit isolation, not network isolation.

State lives at /var/lib/nixos-containers/hive-forge/var/lib/forgejo/ and survives container restart / host reboot. To wipe, destroy the container.

Per-agent static frontend split

When services.hyperhive.frontend is configured, hive-c0re injects HIVE_AGENT_FRONTEND_DIR = "${cfg.frontend}/agent" into its service environment. The nginx include generator (gateway_nginx::write) reads this variable and, when set, emits split location blocks per agent instead of the legacy single-proxy block.

Location priority for /agent/<name>/...:

# 1. Compiled assets — content-addressed nix store path, cache forever
location ^~ /agent/<name>/static/ {
    alias <frontend>/static/;
    expires 1y;
    add_header Cache-Control "public, immutable, max-age=31536000";
}

# 2. Static dist + proxy fallback for dynamic paths
location /agent/<name>/ {
    alias <frontend>/;
    try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}

# 3. Proxy catchall — API, events, icon, send, login, …
location @<name>_dynamic {
    proxy_pass <upstream>;
    proxy_set_header X-Forwarded-Prefix /agent/<name>;
    proxy_intercept_errors on;
    error_page 502 503 504 = /__hive_agent_unreachable;
    # … (full proxy header block)
}

try_files resolution (nginx applies the alias mapping before checking file existence):

request	resolved	outcome
`/agent/iris/`	`<frontend>/index.html`	main agent page
`/agent/iris/stats`	`<frontend>/stats.html`	stats page
`/agent/iris/screen`	`<frontend>/screen.html`	screen page
`/agent/iris/static/app.js`	caught by `^~` block first	served with immutable cache
`/agent/iris/api/state`	no file match → `@iris_dynamic`	proxied to agent daemon
`/agent/iris/events/live`	no file match → `@iris_dynamic`	proxied (SSE)

Adding a new HTML page to the frontend dist (dist/<page>.html) automatically makes it reachable at /agent/<name>/<page> — no generator change needed.

Why ^~ for /static/: the ^~ prefix gives this block higher priority than the plain prefix location /agent/<name>/, so compiled JS/CSS assets skip try_files entirely and get the immutable cache headers. Nix store paths are content-addressed — the hash changes on any content change — so max-age=31536000 is safe.

Why nix store is reachable from the gateway container: nspawn containers bind-mount /nix/store read-only by default. The HIVE_AGENT_FRONTEND_DIR path is a nix store path baked in at hive-c0re build time — the same path is visible to both c0re (writing agents.conf) and the gateway nginx (serving files from it).

Graceful degradation: if HIVE_AGENT_FRONTEND_DIR is empty or unset (e.g. a build that predates cfg.frontend), each agent gets the legacy single-proxy block and all traffic is forwarded to the agent daemon as before.

extraFiles: per-agent hyperhive.frontend.extraFiles are in mergedDist, not in the base cfg.frontend dist. They are not under the nix-store alias path, so requests for them fall through try_files to @<name>_dynamic and are served by the agent daemon as before.

Per-agent error pages

/agent/<name>/ requests hit two failure modes; both get static HTML pages instead of nginx's default error chrome:

Agent not found (/agent/<unknown>/...) — name isn't in agentPortsTable. nginx's prefix match falls back to the bare /agent/ catch-all, which return 404s and error_page 404 rewrites to /__hive_agent_not_found → serves not-found.html with a link back to the dashboard.
Agent unreachable (502 / 503 / 504 from proxy_pass) — the per-agent harness isn't responding (container restarting, crash recovery, etc.). proxy_intercept_errors on + error_page 502 503 504 = /__hive_agent_unreachable rewrites to unreachable.html.

Both pages are built at deploy time via pkgs.runCommand (one nix derivation hyperhive-agent-error-pages with not-found.html + unreachable.html inside) and served via two internal nginx locations with alias to the exact file. internal keeps the files from being directly request-able by operators — only nginx's own error-handling can reach them.

Page styling: minimal inline CSS matching the dashboard's catppuccin palette (#1e1e2e bg, #cdd6f4 text, #cba6f7 heading). No dependencies on the frontend dist — these pages render even when hive-c0re itself is down.

Scope is intentionally narrow: only routes already special-cased in the nginx config get custom error pages. Other gateway routes (forge / matrix / fluffychat) get nginx defaults — extending the custom-error pattern there is a separate follow-up.

19 KiB Raw Blame History