hyperhive/docs/gateway.md

# hive-gateway

Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.

## Vhost map

| URL | vhost | upstream | source |
| --- | --- | --- | --- |
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable` |
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway` |
| `matrix.<hive>/_matrix/*` | `matrix.<hive>` | tuwunel (`8008`) | `matrix.gatewayHost != null` |
| `matrix.<hive>/` | `matrix.<hive>` | fluffychat-web static | `matrix.gui.enable` |
| `matrix.<hive>/config.json` | `matrix.<hive>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null` |

Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware. External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.

## Discovery flow (matrix)

Operator points client at `<hive>`. Sequence:

1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://matrix.<hive>"}}` (no port suffix when gateway listens on 443). With `selfSignedTls = false` the scheme drops to http and the port suffix reflects the bare `port` instead.
2. Client connects to `matrix.<hive>/_matrix/client/...`.
3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.

matrix-dart-sdk (FluffyChat etc.) hardcodes `https` for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.

Federation peers fetch `.well-known/matrix/server` → `{"m.server":"matrix.<hive>"}` and connect to `matrix.<hive>:8448` per spec default. Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.matrix.<hive>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.

## SPA fallback (Accept-header pattern)

The `<hive>` catch-all and the `matrix.<hive>` vhost both serve a flutter SPA (per-agent UI, fluffychat). Two requirements collide:

- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
- missing assets must surface as 404, not as HTML with wrong content-type

Solution: an `nginx http`-context `map $http_accept $matrix_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; asset fetches (`Accept: image/*`, `*/*`, etc.) get a sentinel nonexistent path → `try_files` falls through to `=404`. No extension allowlist, no `if` block, no regex heuristics.

## Local dev (`localHostsEntry`)

`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:

- `<hive-domain>` → `127.0.0.1`
- `forge.<hive>` → `127.0.0.1` (when forge.behindGateway)
- `matrix.<hive>` → `127.0.0.1` (when matrix.gatewayHost set)

`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.

## Sub-domain shape (rationale)

Operator decision: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.

- forgejo's default `ROOT_URL = http://<host>/` works without any `X-Forwarded-Prefix` gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
- matrix-spec deployments universally use `matrix.<server_name>` for the actual API listener — federation already expects this.
- per-agent UIs are hyperhive-internal and base-path-aware specifically for `/agent/<name>/`. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
- cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.

`services.hyperhive.{forge.domain,matrix.gatewayHost}` take the full hostname (`forge.darkest.space`, `git.example.com`) rather than a label that gets concatenated with hive-domain — operators want control over the full shape, not a forced `<label>.<hive-domain>` pattern.

## Tuning knobs

Per-vhost timeouts + body-size limits live in the location blocks:

- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), `proxyWebsockets = true` (live-update endpoints).
- matrix `/_matrix/` (tuwunel): `client_max_body_size 50M` (media uploads), `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*` (federation + cross-origin clients), `proxyWebsockets = true`.
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE / WebSocket dashboards), `proxyWebsockets = true`, `X-Forwarded-Prefix` set so the harness can build absolute URLs when relative isn't enough.

SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not HTTP-over-nginx.

## Per-agent unix-socket upstream

All agents bind their web UI on a unix-domain socket at
`/run/hive-agent/<name>/web.sock` — the `HIVE_WEB_SOCKET` env var is
now set unconditionally for every agent. The mechanism:

1. **Agent side**. `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
   is set on every harness service env; `web_ui::serve` binds a
   `UnixListener` at that path. The deprecated
   `hyperhive.web.useUnixSocket` option is now a no-op.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
   (`/run/hive-agent/<name>/`) into the agent's container. Dir
   bind, not file bind — file bind-mounts don't survive the
   harness's `unlink + bind(2)` cycle on socket replace. Per-agent
   subdir keeps each agent's container blind to siblings' sockets.
3. **Marker gate**. After successful `bind_unix`, the harness drops
   `<dir>/hyperhive-socket-bound` next to the socket. c0re's
   `agent_sockets::write` filters its JSON map by marker presence —
   only agents whose harness has actually bound the socket appear there.
   (Legacy name `.bound` also accepted during the transition window.)
4. **Gateway side**. `gateway_nginx::write` generates
   `/var/lib/hyperhive/gateway/agents.conf` — a plain nginx include
   file with one `location /agent/<name>/` block per agent. UDS
   upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`) when
   `hyperhive-socket-bound` marker present; TCP loopback for agents
   that haven't yet been rebuilt under the new config.
   The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at
   `/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`.
   After each write, c0re triggers the appropriate nginx action inside
   the gateway container via `hive-priv` (which runs as root and has
   `--machine=hive-gateway` transport rights that hive-c0re lacks).
   `hive-priv` queries `ActiveState` and dispatches:
   - active → `systemctl reload nginx` (SIGHUP, zero-downtime)
   - failed → `systemctl reset-failed nginx` + `systemctl start nginx`
   - otherwise → `systemctl start nginx`
   This is intentionally host-side: `IN_MOVED_TO` from an atomic rename
   does not propagate across the nspawn mount-namespace boundary, so a
   path unit inside the container would never fire.

c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
marker poll tick (`agent_sockets::spawn_poll`). `write()` is
idempotent — skips the rename when content is unchanged. Failed reloads
are retried automatically on subsequent poll ticks via
`gateway_nginx::reload_if_pending`.

## Agent port map (`agent-ports.json`)

`/var/lib/hyperhive/agent-ports.json` is a flat JSON object keyed by
logical agent name → TCP web port:

```json
{
  "iris":     8178,
  "atlas":    8304,
  "argus":    8267,
  "damocles": 8549
}
```

Written alongside `agents.conf` on every topology change. Ports come from
`lifecycle::agent_web_port(name)` — a pure FNV-1a hash of the name,
reproducible from the name alone. The manager is excluded: it always
uses a unix socket (`HIVE_WEB_SOCKET` is unconditionally set for the
manager role), so its TCP port never appears in the fallback map.
The gateway routes `/agent/root/` to the manager's unix socket via
`agents.conf` alongside sub-agents.

The file doubles as a human-readable audit artifact — `cat agent-ports.json`
shows every registered sub-agent and its deterministic port assignment. TCP
loopback upstreams in `agents.conf` reference these ports for agents that
haven't opted into unix-socket mode yet.

Both `agent-ports.json` and `agents.conf` use atomic `<path>.tmp` +
`rename()` writes so a crashing c0re process never leaves a partial or
unparseable file behind.

## Dashboard link shape (gateway vs direct)

When the gateway is in front, the SW4RM tab builds per-agent links
as same-origin `/agent/<name>/…` URLs instead of the legacy direct
`http://<host>:<container.port>/` TCP shape. The signal comes from
`StateSnapshot.gateway_enabled`, sourced from the
`HIVE_GATEWAY_ENABLED` env the c0re NixOS module sets when
`services.hyperhive.gateway.enable = true`. Three render sites
flip together: the primary agent-name link, the favicon fetch
(`<url>/icon`), and the nav-strip `container`-kind links from
`/api/agent/<name>/links`. `forge`-kind nav-strip links still
resolve against `http://<host>:3000` (separate sub-domain transition
tracked by `forge.behindGateway`); `external`-kind links are
already absolute. See `docs/web-ui/dashboard.md::Container row` for the
frontend-side derivation.

## TLS modes

Four modes:

| mode | config | cert source | `.well-known` scheme |
|---|---|---|---|
| self-signed (default) | `selfSignedTls = true` | auto-generated RSA-4096, 10-year | `https` |
| ACME (Let's Encrypt) | `selfSignedTls = false` + `tls.acme.enable = true` | nginx inside container via HTTP-01 | `https` |
| operator cert | `selfSignedTls = false` + `tls.certDir` set | bind-mounted from host | `https` |
| http-only | `selfSignedTls = false`, no `tls.certDir`, no `tls.acme` | none | `http` |

### ACME / Let's Encrypt (`tls.acme`)

Simplest production path for operators with a public domain:

```nix
services.hyperhive.gateway = {
  selfSignedTls = false;
  openFirewall  = true;
  tls.acme = {
    enable = true;
    email  = "admin@example.com";
  };
};
```

nginx inside the gateway container obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). The gateway container shares the host network namespace (`privateNetwork = false`) so outbound ACME requests work without any extra routing. Certs are stored inside the container's persistent state dir (`/var/lib/acme/` inside `hive-gateway`; survives restarts because `ephemeral = false`).

**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<domain>`, `matrix.<domain>`) gets its own cert via separate ACME challenges.

Mutual exclusion: `selfSignedTls = true` or `tls.certDir` set together with `tls.acme.enable = true` fails an assertion.

**Swarm peers**: CA-signed certs are trusted by default — remote hives need no `certFingerprint` in `swarm.peers`.

### Self-signed TLS (`selfSignedTls`)

On by default. The gateway generates a self-signed RSA-4096 cert at first boot (10-year validity) and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).

**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap.

**Cert shape**: subject CN = bare hive domain; subjectAltName covers `<hive>` + wildcard `*.<hive>` so all current and future sub-domain vhosts (matrix, forge, ...) validate under the same cert. Stored at `/var/lib/hive-gateway/tls/{cert,key}.pem` inside the gateway container (`ephemeral = false`, so persisted across container restart).

**Regeneration**: the generator unit (`hive-gateway-self-signed-cert.service`) always runs (idempotent). To rotate (e.g. cert leak, expiry approaching), delete `cert.pem` inside the gateway container and restart `nginx.service`.

**Cert prompts**: browsers warn once per host on first visit. With the wildcard SAN, `https://<hive>/`, `https://matrix.<hive>/`, and `https://forge.<hive>/` are covered by the same cert, but the browser still prompts per origin.

### Operator-provided cert (`tls.certDir`)

For operators with a real CA cert (Let's Encrypt, corporate CA, etc.):

```nix
services.hyperhive.gateway = {
  selfSignedTls = false;
  tls.certDir = "/var/lib/acme/example.com";  # nixpkgs security.acme output dir
  # tls.certName = "cert.pem";   # default — matches security.acme layout
  # tls.keyName  = "key.pem";    # default — matches security.acme layout
};
```

The directory is bind-mounted read-only into the gateway container at `/run/hive-tls/`. nginx uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.

`selfSignedTls = true` and `tls.certDir` set together is an assertion error.

**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx inside the gateway container runs as the `nginx` user and cannot read a key with that ownership. Fix with:

```nix
security.acme.certs."example.com".group = "nginx";
```

or make the key world-readable (`0644`) if your threat model allows it. nginx errors out at startup on a key it can't read — the error is explicit in the journal, not a silent failure.

**Peer hive config**: when using a CA-signed cert, peer hives can declare this hive without `certFingerprint` in `swarm.peers` — the standard CA bundle validates:

```nix
services.hyperhive.swarm.peers."example.com" = { };  # no certFingerprint needed
```

### HTTP-only (`selfSignedTls = false`, no `tls.certDir`)

For operators who front the gateway with an external TLS terminator (caddy, traefik, nginx + ACME on the host). The gateway listens on `port` (default 80) only. `.well-known/matrix` responses use `http://`, which breaks matrix-dart-sdk discovery — acceptable when matrix GUI is off or the external proxy handles the `.well-known` redirect.

**`.well-known/matrix/{client,server}` scheme**: `https` when TLS is active (either mode), `http` when http-only. The scheme must match what clients see at the external hostname.

## Firewall posture (host-level)

`hive-c0re.nix` opens the per-agent web-port range
`8100..8999` in the host firewall **only when
`services.hyperhive.gateway.enable = false`**. With the gateway on
(default), it's the sole external entry point and proxies to
`127.0.0.1:<port>` internally — leaving the per-agent ports
firewall-open would defeat the single-front-door story.

`services.hyperhive.gateway.openFirewall = true` opens `port` plus
`httpsPort` when `selfSignedTls = true` (default). Operators who
flip `selfSignedTls = false` to front the gateway with a real
TLS-terminating reverse proxy on the host get only `port` opened.

The manager hashes into the same port range as sub-agents (no
"manager pinned at 8000" special case), so one range opening covers
every container.

The dashboard port (`cfg.dashboardPort`, default 7000) is *not*
listed in either case — it binds `127.0.0.1` only, so a firewall
hole would be a no-op. Remote dashboard access flows through the
gateway. Operators who opt out of the gateway lose external
dashboard reach by design — the surface is privileged (approve /
deny / destroy) and must not be exposed without a real reverse
proxy in front.

## `HIVE_FORGE_URL`: domain via gateway for isolated agents, loopback for shared-netns

Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
`hive-forge` calls against it. `hive-c0re.nix` sets this based on the
network isolation mode:

- **`network.isolateContainers = true`**: agents run in private netns and
  get the bridge dnsmasq as their resolver. `HIVE_FORGE_URL` is set to
  `http://<forge.domain>` (default `forge.<hive-domain>`). Agents resolve
  the hostname via dnsmasq → bridge IP, then reach nginx on port 80 (bridge
  firewall opens 80+443 when isolation is on). nginx proxies to forgejo — the
  same path an operator browser takes, no raw port exposure needed.
- **`network.isolateContainers = false`** (default): agents share the host's
  network namespace, so loopback reaches forgejo directly. `HIVE_FORGE_URL`
  is `http://127.0.0.1:<forge.httpPort>`.

## hive-forge container shape

Private Forgejo wrapped in a nixos-container (`hive-forge`, not
`h-*` — keeps c0re's lifecycle scanner out of the picture; the
operator manages it via the standard `nixos-container` CLI). The
container also keeps hive-forge from fighting any `services.forgejo`
the operator already runs on the host — separate systemd namespace,
separate state dir, separate port unless the operator deliberately
collides.

Container shares the host network namespace
(`privateNetwork = false`) so agents reach the forge at
`http://localhost:<httpPort>` without extra plumbing — nixos-container
is here for state + systemd-unit isolation, not network isolation.

State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/`
and survives container restart / host reboot. To wipe, destroy the
container.

## Per-agent static frontend split

When `services.hyperhive.frontend` is configured, hive-c0re injects
`HIVE_AGENT_FRONTEND_DIR = "${cfg.frontend}/agent"` into its service
environment. The nginx include generator (`gateway_nginx::write`) reads
this variable and, when set, emits split location blocks per agent
instead of the legacy single-proxy block.

**Location priority for `/agent/<name>/...`:**

```nginx
# 1. Compiled assets — content-addressed nix store path, cache forever
location ^~ /agent/<name>/static/ {
    alias <frontend>/static/;
    expires 1y;
    add_header Cache-Control "public, immutable, max-age=31536000";
}

# 2. Static dist + proxy fallback for dynamic paths
location /agent/<name>/ {
    alias <frontend>/;
    try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}

# 3. Proxy catchall — API, events, icon, send, login, …
location @<name>_dynamic {
    proxy_pass <upstream>;
    proxy_set_header X-Forwarded-Prefix /agent/<name>;
    proxy_intercept_errors on;
    error_page 502 503 504 = /__hive_agent_unreachable;
    # … (full proxy header block)
}
```

**`try_files` resolution** (nginx applies the `alias` mapping before
checking file existence):

| request | resolved | outcome |
| --- | --- | --- |
| `/agent/iris/` | `<frontend>/index.html` | main agent page |
| `/agent/iris/stats` | `<frontend>/stats.html` | stats page |
| `/agent/iris/screen` | `<frontend>/screen.html` | screen page |
| `/agent/iris/static/app.js` | caught by `^~` block first | served with immutable cache |
| `/agent/iris/api/state` | no file match → `@iris_dynamic` | proxied to agent daemon |
| `/agent/iris/events/live` | no file match → `@iris_dynamic` | proxied (SSE) |

Adding a new HTML page to the frontend dist (`dist/<page>.html`)
automatically makes it reachable at `/agent/<name>/<page>` — no
generator change needed.

**Why `^~` for `/static/`**: the `^~` prefix gives this block higher
priority than the plain prefix `location /agent/<name>/`, so compiled
JS/CSS assets skip `try_files` entirely and get the immutable cache
headers. Nix store paths are content-addressed — the hash changes on
any content change — so `max-age=31536000` is safe.

**Why nix store is reachable from the gateway container**: nspawn
containers bind-mount `/nix/store` read-only by default. The
`HIVE_AGENT_FRONTEND_DIR` path is a nix store path baked in at
hive-c0re build time — the same path is visible to both c0re (writing
`agents.conf`) and the gateway nginx (serving files from it).

**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
legacy single-proxy block and all traffic is forwarded to the agent
daemon as before.

**`extraFiles`**: per-agent `hyperhive.frontend.extraFiles` are in
`mergedDist`, not in the base `cfg.frontend` dist. They are not under
the nix-store `alias` path, so requests for them fall through
`try_files` to `@<name>_dynamic` and are served by the agent daemon
as before.

## Per-agent error pages

`/agent/<name>/` requests hit two failure modes; both get static
HTML pages instead of nginx's default error chrome:

- **Agent not found** (`/agent/<unknown>/...`) — name isn't in
  `agentPortsTable`. nginx's prefix match falls back to the bare
  `/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
  to `/__hive_agent_not_found` → serves `not-found.html` with a link
  back to the dashboard.

- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
  per-agent harness isn't responding (container restarting, crash
  recovery, etc.). `proxy_intercept_errors on` + `error_page 502 503
  504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.

Both pages are built at deploy time via `pkgs.runCommand` (one nix
derivation `hyperhive-agent-error-pages` with `not-found.html` +
`unreachable.html` inside) and served via two `internal` nginx
locations with `alias` to the exact file. `internal` keeps the
files from being directly request-able by operators — only nginx's
own error-handling can reach them.

Page styling: minimal inline CSS matching the dashboard's catppuccin
palette (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` heading). No
dependencies on the frontend dist — these pages render even when
hive-c0re itself is down.

Scope is intentionally narrow: only routes already special-cased in
the nginx config get custom error pages. Other gateway routes
(forge / matrix / fluffychat) get nginx defaults — extending the
custom-error pattern there is a separate follow-up.

## HTTP Basic auth

`services.hyperhive.gateway.auth.enable = true` gates every request to
the main vhost (`_`) behind HTTP Basic auth. nginx's built-in `auth_basic`
module validates credentials; no extra service or host-side daemon is
required.

**Setup:**

```nix
services.hyperhive.gateway.auth = {
  enable = true;
  # realm = "hyperhive";  # optional, default shown
};
```

The credential store lives at the fixed path
`/var/lib/hyperhive/gateway/gateway.htpasswd` on the host. A tmpfiles
rule pre-creates the file on first boot; no manual path configuration
is required. The file is exposed inside the gateway container at
`/run/hive-state/gateway.htpasswd` via the existing gateway state
bind-mount.

Manage users with `hivectl gateway` (defaults to the standard path — no
`--file` flag needed for the common case):

```sh
# Add or update a user (prompted for password):
hivectl gateway create-user alice --password-stdin

# Add with inline password (visible in shell history — avoid for sensitive creds):
hivectl gateway create-user bob --password hunter2

# Remove a user:
hivectl gateway delete-user bob

# List current usernames:
hivectl gateway list-users
```

`hivectl gateway create-user` hashes passwords with BCrypt (cost 12) and
writes `$2y$`-prefixed hashes that nginx accepts natively. No external
`htpasswd` binary is required. Pass `--file <path>` to target a
non-default file.

**What is not gated:** per-agent UI routes emitted into `agents.conf`
(served under `/agent/<name>/`) inherit no auth from `/` — nginx
applies `auth_basic` per-location. Full per-agent coverage is a
follow-up.

**Realm:** the `WWW-Authenticate: Basic realm="..."` string browsers
display in the credential dialog. Defaults to `"hyperhive"`. Must not
contain `"` or `$`.