hyperhive/docs/gateway.md

# hive-gateway

Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.

## Vhost map

| URL | vhost | upstream | source |
| --- | --- | --- | --- |
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable` |
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway` |
| `matrix.<hive>/_matrix/*` | `matrix.<hive>` | tuwunel (`8008`) | `matrix.gatewayHost != null` |
| `matrix.<hive>/` | `matrix.<hive>` | fluffychat-web static | `matrix.gui.enable` |
| `matrix.<hive>/config.json` | `matrix.<hive>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null` |

Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware. External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.

## Discovery flow (matrix)

Operator points client at `<hive>`. Sequence:

1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://matrix.<hive>"}}` (no port suffix when gateway listens on 443). With `selfSignedTls = false` the scheme drops to http and the port suffix reflects the bare `port` instead.
2. Client connects to `matrix.<hive>/_matrix/client/...`.
3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.

matrix-dart-sdk (FluffyChat etc.) hardcodes `https` for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.

Federation peers fetch `.well-known/matrix/server` → `{"m.server":"matrix.<hive>"}` and connect to `matrix.<hive>:8448` per spec default. Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.matrix.<hive>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.

## SPA fallback (Accept-header pattern)

The `<hive>` catch-all and the `matrix.<hive>` vhost both serve a flutter SPA (per-agent UI, fluffychat). Two requirements collide:

- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
- missing assets must surface as 404, not as HTML with wrong content-type

Solution: an `nginx http`-context `map $http_accept $matrix_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; asset fetches (`Accept: image/*`, `*/*`, etc.) get a sentinel nonexistent path → `try_files` falls through to `=404`. No extension allowlist, no `if` block, no regex heuristics.

## Local dev (`localHostsEntry`)

`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:

- `<hive-domain>` → `127.0.0.1`
- `forge.<hive>` → `127.0.0.1` (when forge.behindGateway)
- `matrix.<hive>` → `127.0.0.1` (when matrix.gatewayHost set)

`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.

## Sub-domain shape (rationale)

Operator decision: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.

- forgejo's default `ROOT_URL = http://<host>/` works without any `X-Forwarded-Prefix` gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
- matrix-spec deployments universally use `matrix.<server_name>` for the actual API listener — federation already expects this.
- per-agent UIs are hyperhive-internal and base-path-aware specifically for `/agent/<name>/`. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
- cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.

`services.hyperhive.{forge.domain,matrix.gatewayHost}` take the full hostname (`forge.darkest.space`, `git.example.com`) rather than a label that gets concatenated with hive-domain — operators want control over the full shape, not a forced `<label>.<hive-domain>` pattern.

## Tuning knobs

Per-vhost timeouts + body-size limits live in the location blocks:

- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), `proxyWebsockets = true` (live-update endpoints).
- matrix `/_matrix/` (tuwunel): `client_max_body_size 50M` (media uploads), `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*` (federation + cross-origin clients), `proxyWebsockets = true`.
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE / WebSocket dashboards), `proxyWebsockets = true`, `X-Forwarded-Prefix` set so the harness can build absolute URLs when relative isn't enough.

SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not HTTP-over-nginx.

## Per-agent unix-socket upstream

All agents bind their web UI on a unix-domain socket at
`/run/hive-agent/<name>/web.sock` — the `HIVE_WEB_SOCKET` env var is
now set unconditionally for every agent. The mechanism:

1. **Agent side**. `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
   is set on every harness service env; `web_ui::serve` binds a
   `UnixListener` at that path. The deprecated
   `hyperhive.web.useUnixSocket` option is now a no-op.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
   (`/run/hive-agent/<name>/`) into the agent's container. Dir
   bind, not file bind — file bind-mounts don't survive the
   harness's `unlink + bind(2)` cycle on socket replace. Per-agent
   subdir keeps each agent's container blind to siblings' sockets.
3. **Marker gate**. After successful `bind_unix`, the harness drops
   `<dir>/hyperhive-socket-bound` next to the socket. c0re's
   `agent_sockets::write` filters its JSON map by marker presence —
   only agents whose harness has actually bound the socket appear there.
   (Legacy name `.bound` also accepted during the transition window.)
4. **Gateway side**. `gateway_nginx::write` generates
   `/var/lib/hyperhive/gateway/agents.conf` — a plain nginx include
   file with one `location /agent/<name>/` block per agent. UDS
   upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`) when
   `hyperhive-socket-bound` marker present; TCP loopback for agents
   that haven't yet been rebuilt under the new config.
   The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at
   `/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`.
   After each write, c0re triggers `nginx -s reload` inside the
   gateway container from the HOST via
   `systemd-run --machine=hive-gateway --wait nginx -s reload`. This is
   intentionally host-side: `IN_MOVED_TO` from an atomic rename does
   not propagate across the nspawn mount-namespace boundary, so a
   path unit inside the container would never fire.

c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
marker poll tick (`agent_sockets::spawn_poll`). `write()` is
idempotent — skips the rename when content is unchanged. Failed reloads
are retried automatically on subsequent poll ticks via
`gateway_nginx::reload_if_pending`.

## Agent port map (`agent-ports.json`)

`/var/lib/hyperhive/agent-ports.json` is a flat JSON object keyed by
logical agent name → TCP web port:

```json
{
  "iris":     8178,
  "atlas":    8304,
  "argus":    8267,
  "damocles": 8549
}
```

Written alongside `agents.conf` on every topology change. Ports come from
`lifecycle::agent_web_port(name)` — a pure FNV-1a hash of the name,
reproducible from the name alone. The manager is excluded: it always
uses a unix socket (`HIVE_WEB_SOCKET` is unconditionally set for the
manager role), so its TCP port never appears in the fallback map.
The gateway routes `/agent/root/` to the manager's unix socket via
`agents.conf` alongside sub-agents.

The file doubles as a human-readable audit artifact — `cat agent-ports.json`
shows every registered sub-agent and its deterministic port assignment. TCP
loopback upstreams in `agents.conf` reference these ports for agents that
haven't opted into unix-socket mode yet.

Both `agent-ports.json` and `agents.conf` use atomic `<path>.tmp` +
`rename()` writes so a crashing c0re process never leaves a partial or
unparseable file behind.

## Dashboard link shape (gateway vs direct)

When the gateway is in front, the SW4RM tab builds per-agent links
as same-origin `/agent/<name>/…` URLs instead of the legacy direct
`http://<host>:<container.port>/` TCP shape. The signal comes from
`StateSnapshot.gateway_enabled`, sourced from the
`HIVE_GATEWAY_ENABLED` env the c0re NixOS module sets when
`services.hyperhive.gateway.enable = true`. Three render sites
flip together: the primary agent-name link, the favicon fetch
(`<url>/icon`), and the nav-strip `container`-kind links from
`/api/agent/<name>/links`. `forge`-kind nav-strip links still
resolve against `http://<host>:3000` (separate sub-domain transition
tracked by `forge.behindGateway`); `external`-kind links are
already absolute. See `docs/web-ui/dashboard.md::Container row` for the
frontend-side derivation.

## Self-signed TLS (`selfSignedTls`)

On by default. The gateway generates a self-signed RSA-4096 cert at first boot (10-year validity) and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).

**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap. For headless agent traffic and operator dashboard reach plain http is fine, so the http listen stays in parallel — operators can keep using `http://<hive>/` from the dashboard if they don't care about the cert prompt.

**Cert shape**: subject CN = bare hive domain; subjectAltName covers `<hive>` + wildcard `*.<hive>` so all current and future sub-domain vhosts (matrix, forge, ...) validate under the same cert. Stored at `/var/lib/hive-gateway/tls/{cert,key}.pem` inside the gateway container (`ephemeral = false`, so persisted across container restart).

**Regeneration**: the generator unit (`hive-gateway-self-signed-cert.service`) is gated by `ConditionPathExists=!cert.pem`, so it's a one-shot. To rotate (e.g. cert leak, expiry approaching), delete `cert.pem` inside the gateway container and restart `nginx.service` — the generator runs as a `Before=` dependency. No automatic rotation; the cert is purely a workaround for the well-known fetch.

**Production**: operators fronting hyperhive with a real reverse proxy (caddy + ACME, traefik + Let's Encrypt, etc.) set `selfSignedTls = false`. The proxy handles termination upstream, the gateway listens on `port` only.

**Cert prompts**: browsers warn once per host on first visit. With the wildcard SAN, `https://<hive>/`, `https://matrix.<hive>/`, and `https://forge.<hive>/` are covered by the same cert, but the browser still prompts per origin (per-host security state). FluffyChat needs the user to accept both `matrix.<hive>` (the SPA itself) and `<hive>` (the well-known fetch endpoint).

**`.well-known/matrix/{client,server}` scheme**: switches to `https` when `selfSignedTls` is on, so matrix-dart-sdk doesn't downgrade and tuwunel federation peers get a TLS-fronted base_url. The legacy `/matrix/*` 301 redirect also flips to https.

## Firewall posture (host-level)

`hive-c0re.nix` opens the per-agent web-port range
`8100..8999` in the host firewall **only when
`services.hyperhive.gateway.enable = false`**. With the gateway on
(default), it's the sole external entry point and proxies to
`127.0.0.1:<port>` internally — leaving the per-agent ports
firewall-open would defeat the single-front-door story.

`services.hyperhive.gateway.openFirewall = true` opens `port` plus
`httpsPort` when `selfSignedTls = true` (default). Operators who
flip `selfSignedTls = false` to front the gateway with a real
TLS-terminating reverse proxy on the host get only `port` opened.

The manager hashes into the same port range as sub-agents (no
"manager pinned at 8000" special case), so one range opening covers
every container.

The dashboard port (`cfg.dashboardPort`, default 7000) is *not*
listed in either case — it binds `127.0.0.1` only, so a firewall
hole would be a no-op. Remote dashboard access flows through the
gateway. Operators who opt out of the gateway lose external
dashboard reach by design — the surface is privileged (approve /
deny / destroy) and must not be exposed without a real reverse
proxy in front.

## `HIVE_FORGE_URL`: domain via gateway for isolated agents, loopback for shared-netns

Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
`hive-forge` calls against it. `hive-c0re.nix` sets this based on the
network isolation mode:

- **`network.isolateContainers = true`**: agents run in private netns and
  get the bridge dnsmasq as their resolver. `HIVE_FORGE_URL` is set to
  `http://<forge.domain>` (default `forge.<hive-domain>`). Agents resolve
  the hostname via dnsmasq → bridge IP, then reach nginx on port 80 (bridge
  firewall opens 80+443 when isolation is on). nginx proxies to forgejo — the
  same path an operator browser takes, no raw port exposure needed.
- **`network.isolateContainers = false`** (default): agents share the host's
  network namespace, so loopback reaches forgejo directly. `HIVE_FORGE_URL`
  is `http://127.0.0.1:<forge.httpPort>`.

## hive-forge container shape

Private Forgejo wrapped in a nixos-container (`hive-forge`, not
`h-*` — keeps c0re's lifecycle scanner out of the picture; the
operator manages it via the standard `nixos-container` CLI). The
container also keeps hive-forge from fighting any `services.forgejo`
the operator already runs on the host — separate systemd namespace,
separate state dir, separate port unless the operator deliberately
collides.

Container shares the host network namespace
(`privateNetwork = false`) so agents reach the forge at
`http://localhost:<httpPort>` without extra plumbing — nixos-container
is here for state + systemd-unit isolation, not network isolation.

State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/`
and survives container restart / host reboot. To wipe, destroy the
container.

## Per-agent static frontend split

When `services.hyperhive.frontend` is configured, hive-c0re injects
`HIVE_AGENT_FRONTEND_DIR = "${cfg.frontend}/agent"` into its service
environment. The nginx include generator (`gateway_nginx::write`) reads
this variable and, when set, emits split location blocks per agent
instead of the legacy single-proxy block.

**Location priority for `/agent/<name>/...`:**

```nginx
# 1. Compiled assets — content-addressed nix store path, cache forever
location ^~ /agent/<name>/static/ {
    alias <frontend>/static/;
    expires 1y;
    add_header Cache-Control "public, immutable, max-age=31536000";
}

# 2. Static dist + proxy fallback for dynamic paths
location /agent/<name>/ {
    alias <frontend>/;
    try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}

# 3. Proxy catchall — API, events, icon, send, login, …
location @<name>_dynamic {
    proxy_pass <upstream>;
    proxy_set_header X-Forwarded-Prefix /agent/<name>;
    proxy_intercept_errors on;
    error_page 502 503 504 = /__hive_agent_unreachable;
    # … (full proxy header block)
}
```

**`try_files` resolution** (nginx applies the `alias` mapping before
checking file existence):

| request | resolved | outcome |
| --- | --- | --- |
| `/agent/iris/` | `<frontend>/index.html` | main agent page |
| `/agent/iris/stats` | `<frontend>/stats.html` | stats page |
| `/agent/iris/screen` | `<frontend>/screen.html` | screen page |
| `/agent/iris/static/app.js` | caught by `^~` block first | served with immutable cache |
| `/agent/iris/api/state` | no file match → `@iris_dynamic` | proxied to agent daemon |
| `/agent/iris/events/live` | no file match → `@iris_dynamic` | proxied (SSE) |

Adding a new HTML page to the frontend dist (`dist/<page>.html`)
automatically makes it reachable at `/agent/<name>/<page>` — no
generator change needed.

**Why `^~` for `/static/`**: the `^~` prefix gives this block higher
priority than the plain prefix `location /agent/<name>/`, so compiled
JS/CSS assets skip `try_files` entirely and get the immutable cache
headers. Nix store paths are content-addressed — the hash changes on
any content change — so `max-age=31536000` is safe.

**Why nix store is reachable from the gateway container**: nspawn
containers bind-mount `/nix/store` read-only by default. The
`HIVE_AGENT_FRONTEND_DIR` path is a nix store path baked in at
hive-c0re build time — the same path is visible to both c0re (writing
`agents.conf`) and the gateway nginx (serving files from it).

**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
legacy single-proxy block and all traffic is forwarded to the agent
daemon as before.

**`extraFiles`**: per-agent `hyperhive.frontend.extraFiles` are in
`mergedDist`, not in the base `cfg.frontend` dist. They are not under
the nix-store `alias` path, so requests for them fall through
`try_files` to `@<name>_dynamic` and are served by the agent daemon
as before.

## Per-agent error pages

`/agent/<name>/` requests hit two failure modes; both get static
HTML pages instead of nginx's default error chrome:

- **Agent not found** (`/agent/<unknown>/...`) — name isn't in
  `agentPortsTable`. nginx's prefix match falls back to the bare
  `/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
  to `/__hive_agent_not_found` → serves `not-found.html` with a link
  back to the dashboard.

- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
  per-agent harness isn't responding (container restarting, crash
  recovery, etc.). `proxy_intercept_errors on` + `error_page 502 503
  504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.

Both pages are built at deploy time via `pkgs.runCommand` (one nix
derivation `hyperhive-agent-error-pages` with `not-found.html` +
`unreachable.html` inside) and served via two `internal` nginx
locations with `alias` to the exact file. `internal` keeps the
files from being directly request-able by operators — only nginx's
own error-handling can reach them.

Page styling: minimal inline CSS matching the dashboard's catppuccin
palette (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` heading). No
dependencies on the frontend dist — these pages render even when
hive-c0re itself is down.

Scope is intentionally narrow: only routes already special-cased in
the nginx config get custom error pages. Other gateway routes
(forge / matrix / fluffychat) get nginx defaults — extending the
custom-error pattern there is a separate follow-up.

## HTTP Basic auth

`services.hyperhive.gateway.auth.enable = true` gates every request to
the main vhost (`_`) behind HTTP Basic auth. nginx's built-in `auth_basic`
module validates credentials; no extra service or host-side daemon is
required.

**Setup:**

```nix
services.hyperhive.gateway.auth = {
  enable = true;
  # realm = "hyperhive";  # optional, default shown
};
```

The credential store lives at the fixed path
`/var/lib/hyperhive/gateway/gateway.htpasswd` on the host. A tmpfiles
rule pre-creates the file on first boot; no manual path configuration
is required. The file is exposed inside the gateway container at
`/run/hive-state/gateway.htpasswd` via the existing gateway state
bind-mount.

Manage users with `hivectl gateway` (defaults to the standard path — no
`--file` flag needed for the common case):

```sh
# Add or update a user (prompted for password):
hivectl gateway create-user alice --password-stdin

# Add with inline password (visible in shell history — avoid for sensitive creds):
hivectl gateway create-user bob --password hunter2

# Remove a user:
hivectl gateway delete-user bob

# List current usernames:
hivectl gateway list-users
```

`hivectl gateway create-user` hashes passwords with BCrypt (cost 12) and
writes `$2y$`-prefixed hashes that nginx accepts natively. No external
`htpasswd` binary is required. Pass `--file <path>` to target a
non-default file.

**What is not gated:** per-agent UI routes emitted into `agents.conf`
(served under `/agent/<name>/`) inherit no auth from `/` — nginx
applies `auth_basic` per-location. Full per-agent coverage is a
follow-up.

**Realm:** the `WWW-Authenticate: Basic realm="..."` string browsers
display in the credential dialog. Defaults to `"hyperhive"`. Must not
contain `"` or `$`.