vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs: restructure into topic subdirectories, collapse duplicated index

Browse source 
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):

Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
  getting-started/  setup.md
  agent-lifecycle/  agent-hierarchy.md, approvals.md, persistence.md
  trust-boundary/   boundary.md, security.md
  integrations/     forge.md, matrix.md, github.md, knowledge.md
  networking/       gateway.md, network.md, snapshot-store.md
  scheduler/        jobq.md, coordinator.md, ci.md, observability.md
  process/          conventions.md, gotchas.md, pr-review-gate.md
  web-ui/           terminal-rendering.md (moved into the EXISTING dir,
                    per mara's correction to the original getting-started
                    guess -- it's UI implementation detail, not onboarding)

The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).

Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).

Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).

Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.

nix fmt clean, both pre-push lints clean.
This commit is contained in:
iris 2026-09-02 01:47:05 +02:00 • committed by mara
commit 07b62612b0
124 changed files with 301 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

645
docs/networking/gateway.md Normal file
View file

@ -0,0 +1,645 @@
# hive-gateway
Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re, rather than in its own container: it shares the host netns anyway (see [Vhost map](#vhost-map) below), so containerizing it would buy no network isolation while costing a resolv.conf sync, a machine-bus reload, and three bind mounts. System-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.
## Vhost map
| URL | vhost | upstream | source |
| --- | --- | --- | --- |
| `<hive>/` | `_` (catch-all) | dashboard dist (static, from `servedFrontend`); `/api/` + `/webhook/` → hive-c0re (`7000`) | always |
| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
| `<hive>/matrix/` (deprecated) | `_` | 301 → `chat.<swarm>/` | `matrix.gui.enable` |
| `forge.<swarm>/` | `forge.<swarm>` | forgejo (`3000`) | `forge.behindGateway` |
| `chat.<swarm>/_matrix/*` | `chat.<swarm>` | tuwunel (`8008`) | `matrix.gatewayHost != null` |
| `chat.<swarm>/` | `chat.<swarm>` | fluffychat-web static | `matrix.gui.enable` |
| `chat.<swarm>/config.json` | `chat.<swarm>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null` |
| `auth.<swarm>/` | `auth.<swarm>` | authelia (`9091`) | `deploy.authelia` |
| `<swarm>/` | `<swarm>` | swarm-ui dist (static), behind an authelia subrequest | `deploy.swarm-ui` |
The authelia vhost is declared only by the host that **runs** authelia, not by every hive that uses it — a client hive knows the swarm's `authelia.url` but must not answer for a name it doesn't serve. Its server name is exactly `swarm.authelia.domain`: authelia validates `authelia_url ⊂ session cookie domain` at startup, so a near-miss is a container that refuses to boot. It carries no `auth_basic` — the login page must not sit behind the login mechanism it replaces — and sets the four `X-Forwarded-{Proto,Host,Uri,For}` headers, since authelia decides by the *original* request rather than the hop it sees.
⚠️ **A `502` from this vhost usually means authelia has no users yet, not that the proxy is misconfigured.** Authelia treats an empty user store as a fatal startup error, so an enabled-but-unbootstrapped swarm crash-loops the container while the vhost in front of it works perfectly. Check `journalctl -M swarm-authelia -u authelia-swarm` before suspecting anything here; the bootstrap step is in [`swarm/sso.md`](../swarm/sso.md).
Per-agent UIs stay sub-path, forge and matrix get sub-domains — see
[Sub-domain shape (rationale)](#sub-domain-shape-rationale) below for why.
## Discovery flow (matrix)
Operator points client at `<hive>`. Sequence:
1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://chat.<swarm>"}}` (no port suffix when gateway listens on 443). The gateway always terminates TLS, so the scheme is always `https`; a non-default `httpsPort` is reflected as the port suffix.
2. Client connects to `chat.<swarm>/_matrix/client/...`.
3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.
matrix-dart-sdk (FluffyChat etc.) hardcodes `https` for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.
Federation peers fetch `.well-known/matrix/server` → `{"m.server":"chat.<swarm>:<httpsPort>"}` (the federation delegation always carries an explicit port, even the HTTPS default 443 — the https-implies-443 elision only applies to the client base_url above). Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.chat.<swarm>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.
## SPA fallback (Accept-header pattern)
The per-agent UIs and the `chat.<swarm>` vhost serve a flutter/SPA bundle via the Accept-header pattern below. The dashboard vhost instead routes by **path** — see [Dashboard: path-based routing](#dashboard-path-based-routing-not-accept-header) below. Two requirements collide:
- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
- a non-navigation request that isn't an on-disk asset must NOT get HTML with the wrong content-type
Solution: an `nginx http`-context `map $http_accept $<name>_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; everything else (`Accept: image/*`, `*/*`, `application/json`, `text/event-stream`, …) gets a sentinel nonexistent path, so `try_files $uri $<name>_spa_target <final>` falls through to `<final>`. No extension allowlist, no `if` block, no regex heuristics.
For matrix / per-agent static assets, `<final>` is `=404` (a missing asset is just missing).
### Dashboard: path-based routing (not Accept-header)
Every hive-c0re backend route lives under `/api/` plus the single `/webhook/knowledge` endpoint, so the dashboard vhost routes by **path**, not Accept header — deterministic, unlike a content-type split where the same URL could resolve differently depending on the caller's `Accept` header:
- `location /api/` → hive-c0re (`7000`): all dashboard data, actions/mutations, and the two SSE streams (`/api/dashboard/stream`, `/api/build-logs/id/{id}/stream`). Carries `proxy_buffering off` + a 1d read timeout for the streams.
- `location /webhook/` → hive-c0re: the knowledge webhook.
- `location /` → the dashboard dist (from the `servedFrontend` nix-store path) with `try_files $uri /index.html` (SPA fallback).
Each location carries a duplicated `auth_basic` block (separate locations don't inherit it). This keeps the gateway static-serving the dashboard dist while hive-c0re stays API-only — a frontend-only change doesn't rebuild or restart the core daemon. A new top-level c0re route prefix (beyond `/api` + `/webhook`) needs a matching `location` added to the dashboard vhost.
## Local dev (`localHostsEntry`)
`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:
- `<hive-domain>` → `127.0.0.1`
- `forge.<swarm>` → `127.0.0.1` (when forge.behindGateway)
- `chat.<swarm>` → `127.0.0.1` (when matrix.gatewayHost set)
- `auth.<swarm>` → `127.0.0.1` (when deploy.authelia)
`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.
## Sub-domain shape (rationale)
Operator decision: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.
- forgejo's default `ROOT_URL = http://<host>/` works without any `X-Forwarded-Prefix` gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
- matrix-spec deployments universally use `matrix.<server_name>` for the actual API listener — federation already expects this.
- per-agent UIs are hyperhive-internal and base-path-aware specifically for `/agent/<name>/`. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
- cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.
`services.hyperhive.{forge.domain,matrix.gatewayHost}` take the full hostname (`forge.darkest.space`, `git.example.com`) rather than a label that gets concatenated with hive-domain — operators want control over the full shape, not a forced `<label>.<hive-domain>` pattern.
## Tuning knobs
Per-vhost timeouts + body-size limits live in the location blocks:
- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), `proxyWebsockets = true` (live-update endpoints).
- matrix `/_matrix/` (tuwunel): `client_max_body_size 50M` (media uploads), `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*` (federation + cross-origin clients), `proxyWebsockets = true`.
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE / WebSocket dashboards), `proxyWebsockets = true`, `X-Forwarded-Prefix` set so the harness can build absolute URLs when relative isn't enough.
SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not HTTP-over-nginx.
## Per-agent unix-socket upstream
All agents bind their web UI on a unix-domain socket at
`/run/hive-agent/<name>/web.sock` — the `HIVE_WEB_SOCKET` env var is
now set unconditionally for every agent. The mechanism:
1. **Agent side**. `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
is set on every harness service env; `web_ui::serve` binds a
`UnixListener` at that path.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
(`/run/hive-agent/<name>/`) into the agent's container. Dir
bind, not file bind — file bind-mounts don't survive the
harness's `unlink + bind(2)` cycle on socket replace. Per-agent
subdir keeps each agent's container blind to siblings' sockets.
The dir is `0751`, owned by the agent's container uid/gid, so
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
own `0666`. The gateway is one of three principals sharing that dir
and does not own its ownership rules — see
[`docs/trust-boundary/boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
3. **Marker gate**. After successful `bind_unix`, the harness drops
`<dir>/hyperhive-socket-bound` next to the socket. c0re's
`agent_sockets::write` filters its JSON map by marker presence —
only agents whose harness has actually bound the socket appear there.
(Legacy name `.bound` also accepted during the transition window.)
4. **Gateway side**. `gateway_nginx::write` generates
`/var/lib/hive-gateway/conf/agents.conf` — a plain nginx include
file with one `location /agent/<name>/` block per agent. Always
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
if the socket is not yet bound, nginx returns 502 caught by the
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
nginx includes `/var/lib/hive-gateway/conf/agents.conf` — the same
path c0re writes, since both run on the host.
After each write, c0re triggers the appropriate nginx action via
`hive-priv` (which is root; hive-c0re runs as the unprivileged
`hive-core` user and cannot act on a system unit).
`hive-priv` queries `ActiveState` and dispatches:
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
- otherwise → `systemctl start nginx`
This is an explicit trigger rather than a path unit watching the
file: the write and the reload belong in one causal chain c0re can
retry and report on (`RELOAD_PENDING`), not two units racing on an
inotify event.
c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
marker poll tick (`agent_sockets::spawn_poll`). `write()` is
idempotent — skips the rename when content is unchanged. Failed reloads
are retried automatically on subsequent poll ticks via
`gateway_nginx::reload_if_pending`.
`agents.conf` uses atomic `<path>.tmp` + `rename()` writes so a crashing
c0re process never leaves a partial or unparseable file behind.
## Dashboard link shape (gateway vs direct)
When the gateway is in front, the SW4RM tab builds per-agent links
as same-origin `/agent/<name>/…` URLs instead of the legacy direct
`http://<host>:<container.port>/` TCP shape. The signal comes from
`StateSnapshot.gateway_enabled`, sourced from the
`HIVE_GATEWAY_ENABLED` env the c0re NixOS module now always sets
(`services.hyperhive.gateway.enable` was removed — the gateway runs
unconditionally alongside hyperhive), so this is effectively always
true; the `false` branch is retained as a defensive fallback for the
env being unset. Three render sites
flip together: the primary agent-name link, the favicon fetch
(`<url>/icon`), and the nav-strip `container`-kind links from
`DashboardState.links` (`GET /api/dashboard-state`). `forge`-kind nav-strip links still
resolve against `http://<host>:3000` (separate sub-domain transition
tracked by `forge.behindGateway`); `external`-kind links are
already absolute. See `docs/web-ui/dashboard.md::Container row` for the
frontend-side derivation.
## TLS modes
The gateway always terminates TLS — self-signed is the implicit floor when
nothing else is configured, so there is no http-only mode. Three modes,
selected by which (if any) external TLS source is set:
| mode | config | cert source | `.well-known` scheme |
|---|---|---|---|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | host hive-CA signs a gateway leaf (RSA-4096) | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
The `gateway.selfSignedTls` option has been **removed** — self-signed
is now derived from the absence of `tls.certDir` / `tls.acme`. A config
that still sets it fails eval with a removal message; use `tls.certDir`
/ `tls.acme` to override the default.
### ACME / Let's Encrypt (`tls.acme`)
Simplest production path for operators with a public domain:
```nix
services.hyperhive.gateway = {
openFirewall = true;
tls.acme = {
enable = true;
email = "admin@example.com";
};
};
```
nginx obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way.
**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<swarm-domain>`, `chat.<swarm-domain>`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own.
Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails an assertion — pick one external TLS source (or neither, for the self-signed default).
### Self-signed TLS (default)
On by default, and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.deploy.hive-controller.tls.caValidityDays`, default ~20y) under `services.hyperhive.deploy.hive-controller.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.**
It does two jobs, and skipping it has taken the gateway down in production
before. It re-modes the leaf (`hive-tls-ca` writes the key `0600
root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a
`0600` key fails the config test and blocks the unit), and it guarantees
**every cert path the nginx config names exists** — which is what the
swarm-services fallback below is for.
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](../swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap.
**Cert shape**: leaf subject CN = bare hive domain; subjectAltName is `<hive>` plus wildcard `*.<hive>`, so all current and future sub-domain vhosts validate under the same leaf + the hive CA. A swarm service whose name is *not* under this hive's domain cannot be added here — the hive CA is name-constrained to `<hive>`, and a violating SAN invalidates the whole leaf, not just that name. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
**Rotation**: `hive-tls-ca.service` is idempotent — it re-signs the leaf when it is missing or within 30 days of expiry, always under the same CA (so consumer trust is undisturbed). The CA itself is regenerated only if missing or already expired. To force a leaf rotation, delete `gateway.pem` under the state dir and restart the unit, then reload `nginx`.
**Cert prompts**: browsers still warn once per host until the hive's `trust-bundle.pem` is added to the browser/OS trust store (an anchor, not the leaf, is the thing to trust). Agent trust is wired separately (see the agent-trust work for `/run/hive-ca`).
### Operator-provided cert (`tls.certDir`)
For operators with a real CA cert (Let's Encrypt, corporate CA, etc.):
```nix
services.hyperhive.gateway = {
tls.certDir = "/var/lib/acme/example.com"; # nixpkgs security.acme output dir
# tls.certName = "cert.pem"; # default — matches security.acme layout
# tls.keyName = "key.pem"; # default — matches security.acme layout
};
```
nginx reads the directory directly and uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.
`tls.certDir` and `tls.acme.enable` set together is an assertion error.
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and cannot read a key with that ownership. Fix with:
```nix
security.acme.certs."example.com".group = "nginx";
```
or make the key world-readable (`0644`) if your threat model allows it. nginx errors out at startup on a key it can't read — the error is explicit in the journal, not a silent failure.
### Fronting with an external TLS terminator
There is no http-only mode (see [TLS modes](#tls-modes) above). Two paths
for an operator who wants their own TLS terminator:
- give the gateway the real cert via `tls.certDir` (or `tls.acme`) so it
serves proper TLS directly — no separate proxy needed; or
- front it over a **unix socket** rather than a plain-http TCP port (the
intended direction for "bring your own proxy" — the gateway is not meant
to expose an unencrypted TCP upstream).
Because of this, `.well-known/matrix/{client,server}` discovery responses
always advertise `https` (see [Discovery flow](#discovery-flow-matrix) above).
## Firewall posture (host-level)
The gateway is unconditional — `services.hyperhive.gateway.enable` was
removed, there is no gateway-off mode. nginx is always the sole
external entry point and routes to agents over the UDS upstream
described above (see [Per-agent unix-socket
upstream](#per-agent-unix-socket-upstream)), so the per-agent web-port
range `8100..8999` stays closed on the host firewall
unconditionally — opening it would defeat the single-front-door story.
The hashed TCP port (`lifecycle::agent_web_port`) still exists as a
fallback bind for an agent whose `HIVE_WEB_SOCKET` env somehow ends up
unset, but nothing opens a matching firewall hole for it and the
gateway itself never proxies through it.
`services.hyperhive.gateway.openFirewall = true` opens both `port` and
`httpsPort` — both are always served, since the gateway always terminates
TLS (see [TLS modes](#tls-modes) above).
Every agent hashes into the same port range (no special case), so
one range opening covers every container.
The dashboard port (`cfg.dashboardPort`, default 7000) is *not*
listed in either case — it binds `127.0.0.1` only, so a firewall
hole would be a no-op. Remote dashboard access flows through the
gateway. Operators who opt out of the gateway lose external
dashboard reach by design — the surface is privileged (approve /
deny / destroy) and must not be exposed without a real reverse
proxy in front.
## `HIVE_FORGE_URL`: agents reach the forge via the gateway by domain
Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
`hive-forge` calls against it. Network isolation is always on (the
shared-netns mode was removed), so agents run in a private netns and
can never reach the host's loopback.
`nix/host-modules/hive-c0re/environment.nix` sets `HIVE_FORGE_URL` to
`http://<forge.domain>` (default `forge.<swarm-domain>` — a swarm runs
one forge; `services.hyperhive.domain` is required). Agents
get the bridge dnsmasq as their resolver, resolve the hostname →
bridge IP, then reach nginx on port 80 (the bridge firewall opens
80+443). nginx proxies to forgejo — the same path an operator browser
takes, no raw port exposure needed.
## hive-forge container shape
Private Forgejo wrapped in a nixos-container (`hive-forge`, not
`h-*` — keeps c0re's lifecycle scanner out of the picture; the
operator manages it via the standard `nixos-container` CLI). The
container also keeps hive-forge from fighting any `services.forgejo`
the operator already runs on the host — separate systemd namespace,
separate state dir, separate port unless the operator deliberately
collides.
The forge container shares the host network namespace
(`privateNetwork = false`), so forgejo's listeners look like a
host-side service — nixos-container is here for state + systemd-unit
isolation, not network isolation. Note this is the FORGE container;
agent containers are network-isolated and reach the forge through the
gateway by `forge.<swarm-domain>` (see `HIVE_FORGE_URL` above), not via
the host's loopback.
State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/`
and survives container restart / host reboot. To wipe, destroy the
container.
### Network and port configuration
```nix
services.hyperhive.forge = {
httpPort = 3000; # default — HTTP listener; outside hyperhive's 7000/8100-8999 range
sshPort = 2222; # default — git-over-SSH; kept off 22 so it doesn't collide with the host openssh
openFirewall = false; # default — expose httpPort + sshPort to the host firewall
};
```
`httpPort` (default **3000**) is the port Forgejo's HTTP server binds to.
It sits outside hyperhive's reserved ranges (dashboard 7000,
agents 8100–8999) so a default install has no port fights. Change it
only if you already have another process bound to 3000.
`sshPort` (default **2222**) is the port Forgejo's built-in SSH server
uses for `git clone/push/pull` over SSH (`git@<domain>:owner/repo.git`
via `-p 2222`). Port 22 is left alone on the host for openssh.
`openFirewall` (default **false**) controls whether `httpPort` and
`sshPort` are opened in the host firewall. Off by default (secure by
default): agents reach Forgejo through the gateway (`forge.<swarm-domain>` on
the bridge), not the raw port, so no firewall hole is needed. Flip to
`true` when you need:
- The operator's browser to reach `http://<host>:<httpPort>/` directly
(not behind the gateway).
- External git clients that push/pull via SSH directly to the host.
Forgejo served through the gateway (`forge.behindGateway = true`) does
not need `openFirewall` — the gateway's own `openFirewall` option covers
that path.
### `rootUrl` override
```nix
services.hyperhive.forge.rootUrl = "https://forge.example.com/";
```
`rootUrl` (default **null**) overrides the Forgejo `ROOT_URL` that is
auto-derived from `forge.domain` + gateway state. The auto-derivation
covers most cases:
| Shape | Auto-derived `ROOT_URL` |
|---|---|
| `behindGateway = true` | `https://<forge.domain>/` (port suffix omitted when `gateway.httpsPort == 443`) |
| `behindGateway = false` | `http://<forge.domain>:<httpPort>/` |
The gateway always terminates TLS, so the `behindGateway = true` case is
always advertised over `https://`; only the direct (`behindGateway =
false`) shape stays `http://`. Set `rootUrl` explicitly when
`forge.domain` resolves differently from the public URL, or for a
genuinely bespoke shape (e.g. an external reverse proxy on a different
host/path). Must end with `/` (Forgejo requirement; an assertion
enforces this).
## Per-agent static frontend split
When `services.hyperhive.frontend` is configured, hive-c0re injects
`HIVE_AGENT_FRONTEND_DIR = "${cfg.frontend}/agent"` into its service
environment. The nginx include generator (`gateway_nginx::write`) reads
this variable and, when set, emits split location blocks per agent
instead of the legacy single-proxy block.
**Location priority for `/agent/<name>/...`:**
```nginx
# 1. Compiled assets — content-addressed nix store path, cache forever
location ^~ /agent/<name>/static/ {
alias <frontend>/static/;
expires 1y;
add_header Cache-Control "public, immutable, max-age=31536000";
}
# 2. Static dist + proxy fallback for dynamic paths
location /agent/<name>/ {
alias <frontend>/;
try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}
# 3. Proxy catchall — API, events, icon, send, login, …
location @<name>_dynamic {
proxy_pass <upstream>;
proxy_set_header X-Forwarded-Prefix /agent/<name>;
proxy_intercept_errors on;
error_page 502 503 504 = /__hive_agent_unreachable;
# … (full proxy header block)
}
```
**`try_files` resolution** (nginx applies the `alias` mapping before
checking file existence):
| request | resolved | outcome |
| --- | --- | --- |
| `/agent/iris/` | `<frontend>/index.html` | main agent page |
| `/agent/iris/stats` | `<frontend>/stats.html` | stats page |
| `/agent/iris/screen` | `<frontend>/screen.html` | screen page |
| `/agent/iris/static/app.js` | caught by `^~` block first | served with immutable cache |
| `/agent/iris/api/state` | no file match → `@iris_dynamic` | proxied to agent daemon |
| `/agent/iris/events/live` | no file match → `@iris_dynamic` | proxied (SSE) |
Adding a new HTML page to the frontend dist (`dist/<page>.html`)
automatically makes it reachable at `/agent/<name>/<page>` — no
generator change needed.
**Why `^~` for `/static/`**: the `^~` prefix gives this block higher
priority than the plain prefix `location /agent/<name>/`, so compiled
JS/CSS assets skip `try_files` entirely and get the immutable cache
headers. Nix store paths are content-addressed — the hash changes on
any content change — so `max-age=31536000` is safe.
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
store path baked in at hive-c0re build time, and c0re (writing
`agents.conf`) and nginx (serving files from it) are on the same machine,
so they see the same store.
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
legacy single-proxy block and all traffic is forwarded to the agent
daemon as before.
**`extraFiles`**: per-agent `hyperhive.frontend.extraFiles` are in
`mergedDist`, not in the base `cfg.frontend` dist. They are not under
the nix-store `alias` path, so requests for them fall through
`try_files` to `@<name>_dynamic` and are served by the agent daemon
as before.
## Per-agent error pages
`/agent/<name>/` requests hit two failure modes; both get static
HTML pages instead of nginx's default error chrome:
- **Agent not found** (`/agent/<unknown>/...`) — name isn't in
`agentPortsTable`. nginx's prefix match falls back to the bare
`/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
to `/__hive_agent_not_found` → serves `not-found.html` with a link
back to the dashboard.
- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
per-agent harness isn't responding (container restarting, crash
recovery, etc.). `proxy_intercept_errors on` + `error_page 502 503
504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.
Both pages are built at deploy time via `pkgs.runCommand` (one nix
derivation `hyperhive-agent-error-pages` with `not-found.html` +
`unreachable.html` inside) and served via two `internal` nginx
locations with `alias` to the exact file. `internal` keeps the
files from being directly request-able by operators — only nginx's
own error-handling can reach them.
Page styling: minimal inline CSS matching the dashboard's catppuccin
palette (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` heading). No
dependencies on the frontend dist — these pages render even when
hive-c0re itself is down.
Scope is intentionally narrow: a route earns a custom page when the
default status code would point at the wrong component. The per-agent
routes qualify (a 502 there means the harness is restarting, not that
the gateway is broken), and so does `auth.<swarm>` — a dead authelia
upstream almost always means the user store was never bootstrapped, and
a bare 502 blames the proxy, which is the one part that is working.
Forge / matrix / fluffychat still get nginx defaults: their upstreams
being down means what the status code says, so a themed page would add
styling and no information.
## HTTP Basic auth
`services.hyperhive.gateway.auth.enable = true` gates every request to
the main vhost (`_`) behind HTTP Basic auth. nginx's built-in `auth_basic`
module validates credentials; no extra service or host-side daemon is
required.
**Setup:**
```nix
services.hyperhive.gateway.auth = {
enable = true;
# realm = "hyperhive"; # optional, default shown
};
```
The credential store lives at the fixed path
`/var/lib/hive-gateway/conf/gateway.htpasswd` on the host. A tmpfiles
rule pre-creates the file on first boot; no manual path configuration
is required. nginx reads it at that path directly.
Manage users with `hivectl gateway`. `hivectl` sends the request over the
host admin socket and the `hive-c0re` daemon performs the write at its
canonical path — no path is exposed to the CLI:
```sh
# Add or update a user (prompted for password):
hivectl gateway create-user alice --password-stdin
# Add with inline password (visible in shell history — avoid for sensitive creds):
hivectl gateway create-user bob --password hunter2
# Remove a user:
hivectl gateway delete-user bob
# List current usernames:
hivectl gateway list-users
```
The daemon hashes passwords with BCrypt (cost 12) and writes
`$2y$`-prefixed hashes that nginx accepts natively. No external
`htpasswd` binary is required.
**What is not gated:** per-agent UI routes emitted into `agents.conf`
(served under `/agent/<name>/`) inherit no auth from `/` — nginx
applies `auth_basic` per-location. Full per-agent coverage is a
follow-up.
**Realm:** the `WWW-Authenticate: Basic realm="..."` string browsers
display in the credential dialog. Defaults to `"hyperhive"`. Must not
contain `"` or `$`.
**Custom 401 page:** when credentials are absent or wrong, nginx serves
a Catppuccin-styled `unauthorized.html` page (built into the same Nix
derivation as the agent error pages) that tells the operator which
`hivectl` command to run to create a user. The response status is still
`401` (`error_page 401 =401 /__hive_auth_unauthorized`) so browsers
present the login dialog on the first visit — users who dismiss the
dialog see the human-readable hint. The internal exact-match location
(`= /__hive_auth_unauthorized`) beats `location /` in nginx's prefix
ordering, preventing the subrequest from looping back through
`auth_basic`.
## Security headers
The following headers are emitted at server scope on every gateway
vhost (`_`, `forge.<swarm-domain>`, `chat.<swarm-domain>`):
| Header | Value |
|--------|-------|
| `X-Frame-Options` | `SAMEORIGIN` |
| `X-Content-Type-Options` | `nosniff` |
| `Referrer-Policy` | `strict-origin-when-cross-origin` |
nginx's `add_header` inheritance rule: a `location` block that sets its
own `add_header` does **not** inherit server-scope headers. API locations
that carry their own CORS headers (e.g. `/.well-known/matrix/client`,
`/_matrix/`) are therefore unaffected. HTML-serving and proxy locations
with no `add_header` of their own pick the security headers up
automatically.
### HSTS (`gateway.hsts`)
HSTS is **opt-in** and disabled by default:
```nix
services.hyperhive.gateway.hsts = {
enable = true; # default: false
maxAge = 31536000; # default: 1 year (required for preload list)
includeSubDomains = true; # default: true
};
```
When enabled, a `Strict-Transport-Security: max-age=...[; includeSubDomains]`
header is added alongside the other security headers.
**Opt-in rationale**: HSTS pins HTTPS in the browser's preload cache;
enabling it on a deployment that later loses TLS locks browsers out
until `max-age` expires. Only enable when TLS is permanent.
Since the gateway always terminates TLS (see [TLS modes](#tls-modes)
above), an enabled HSTS header is always served over https — there is no
TLS-less mode that could violate it.
## Dialing another vhost by name (`verifiedProxyTo`)
`vhost-lib.nix`'s `verifiedProxyTo` builds the `proxy_ssl_*` /
`proxy_set_header` block a module uses to dial another service on this
same gateway BY NAME over https, verified. One definition rather than a
copy per module: nginx verifies nothing by default
(`proxy_ssl_verify` is off), so a `proxy_pass https://…` without these
lines is encrypted and unauthenticated. That failure is invisible — it
works, and keeps working, against any certificate at all.
Every line earns its place, each confirmed against a real nginx with
the opposite arm run as a control:
- `verify` + `depth` — the chain is leaf -> intermediate -> root.
- `trusted_cert` — the bundle; nginx reads ALL certs in the file, which
the bundle's own doc warns is not true of every consumer.
- `ssl_name` — checks the HOSTNAME too. Without it a chain-only check
accepts any certificate this CA ever signed, and for an internal CA
that is every service on the hive.
- `server_name on` — sends SNI, or the far end cannot pick a cert.
**⚠️ Session-cache footgun**: `proxy_ssl_session_reuse` is left at its
default (on), deliberately — this is used on per-request auth
subrequests, so the handshake it avoids is paid on every request. Worth
knowing when testing though: the session cache is keyed by upstream
address and NOT by trust config, so two locations pointing at one
upstream with different trust do not verify independently.
**⚠️ Host-header clobber footgun**: `verifiedProxyTo` also pins `Host`
(and reinstates the rest of nginx's `recommendedProxySettings` header
set) to the target `name` rather than leaving it to be filled in later.
`name` here resolves back to THIS gateway — every consumer dials another
vhost on the same nginx, not a separate host — and nginx picks the vhost
to answer an HTTPS request from the `Host` header, not from the TLS SNI
that `proxy_ssl_name` sends. `recommendedProxySettings`'s own `Host
$host` (the CALLER's host, not the target) is textually appended by
nixpkgs AFTER a location's `extraConfig` — so it always wins over a
`proxy_set_header Host` written in the location body, and the subrequest
loops back into the ORIGINAL vhost instead of reaching the target,
recursing on its own `auth_request` until nginx's subrequest-depth limit
turns it into a plain 500. Every call site sets `recommendedProxySettings
= false` on the location for exactly this reason — nixpkgs' version
would still clobber this one.

277
docs/networking/network.md Normal file
View file

@ -0,0 +1,277 @@
# hive-network
Host-side bridge + per-agent private-netns isolation — always on
whenever hyperhive is enabled. Configured via
`services.hyperhive.network.*`.
> Isolation is the only mode — there is no shared-netns fallback. The
> former `services.hyperhive.network.enable`,
> `services.hyperhive.network.isolateContainers` and
> `services.hyperhive.network.upstreamDns` options were removed; a
> config that still sets one fails eval with a removal message.
## Network map
One picture of the whole hive. There are two planes: **infra
containers share the host netns** and bind host ports directly;
**compute containers (agents + CI) each get a private netns** behind
the bridge. The unix-socket control plane rides the VFS and is
untouched by any of it.
```
internet
│ uplink NIC — NAT MASQUERADE for the
│ bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge / matrix │
│ │
│ nginx :80/:443 [hive-gateway] │
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
│ tuwunel :8008 client API [hive-matrix] │
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
│ │
│ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
vb-h-<a> vb-h-<b> vb-hive-ci veth pairs
│ │ │
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
│ agent a │ │ agent b │ │ hive-ci │ each; eth0 leases
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
└─────────┘ └─────────┘ └─────────┘
```
| container | netns | IPv4 | listens / reached via |
| -------------- | ----------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end:
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
- **DNS** — agents and the service containers query `bridgeIp:53`; hive
zones are answered authoritatively with the bridge IP, everything else
forwards to the host's resolvers (see *Resolver behaviour* below). Each
container points its own `resolv.conf` there, and one that instead
inherits the host's resolves no swarm name at all — those records exist
only on the bridge.
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
own dashboard name resolve to the bridge IP, land on nginx
`:80`/`:443`, and proxy to forgejo
`:3000`, tuwunel `:8008`, hive-c0re `127.0.0.1:7000`, or a per-agent
UI unix socket.
- **Internet egress** — agent default route points at the bridge IP;
the host forwards + masquerades out its uplink.
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
and reach each other's gateway/forge across it
([`docs/swarm/`](../swarm/README.md)).
- **Control plane (no network)** — per-agent broker socket
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
sockets under `/run/hive-agent/` are unix domain sockets
bind-mounted through the VFS; private netns does not affect them.
## Container shape (where dnsmasq lives)
Co-located in the existing `hive-gateway` container — single
front-door for both DNS and HTTP, saves a sibling container, single
systemd-unit / state surface to monitor. The gateway shares host
netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces`
listener on `bridgeIp` is on the host's bridge interface.
## Configuration
```nix
{
services.hyperhive = {
enable = true;
hiveName = "pr1ma";
swarm.domain = "darkest.space";
swarm.hives.pr1ma = { }; # -> domain = pr1ma.darkest.space
# network.bridgeIp = "10.42.0.1"; # default
};
}
```
Requires `services.hyperhive.domain` to be set — the dnsmasq resolver
is authoritative for `<hive-domain>` and its sub-domains. You do not
write it: it is read from this hive's entry in the swarm directory
(`docs/swarm/README.md` § Hive identity config).
## Bridge addressing
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
RFC 1918 space, unlikely to clash with operator's existing setup;
override `bridgeIp` + `bridgePrefixLength` if a different range is
already in use. `/24` gives 254 usable per-agent addresses — enough
for any single-host hive; bigger swarms or tighter addressing
schemes pick their own.
## Resolver behaviour
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
plus whatever swarm-service names this host contributes via
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
(matrix) when this host runs those services, and `auth.<swarm-domain>`
when it runs authelia — answering each with the bridge IP (where nginx
is reachable). Note forge and matrix are swarm-domain names, not
sub-domains of `<hive-domain>`: a swarm runs one forge and one
homeserver, so their names belong to the swarm rather than to whichever
hive happens to host them. Everything else is forwarded to the host's
own resolvers: dnsmasq runs on the host and reads the host's
`/etc/resolv.conf` directly. Containers don't need to know the
upstream — they query the bridge IP and dnsmasq does the right thing
per-name.
There is deliberately no fallback `server=`: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
directly, so a network change (new router, new lease, laptop moving
networks) reaches it the moment openresolv rewrites the file. There is
nothing to synchronise and no unit watching for it.
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for
container health-checks). External hosts can't reach it — no
DNS-amplification surface even when the operator opens port 80 for
gateway HTTP.
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
resolution stack — the host's resolver (systemd-resolved, plain
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
configured. The hive resolver is purely for inbound queries from
agent containers.
## Firewall posture
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
- Port 53 opens the resolver on the bridge interface only. Other
interfaces stay closed. The hive resolver isn't an external-facing
service.
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
receives DHCP via a regular UDP socket (it does not use a
netfilter-bypassing raw socket), so the hole is mandatory — without
it containers never get a lease and fall back to 169.254.x.x.
- Ports 80 and 443 let isolated agents reach nginx (gateway
container, shared host netns) for the forge sub-domain, per-agent
UI proxies, and any other HTTP services.
The **host** firewall is the only firewall. The shared-netns infra
containers (gateway, forge, matrix) set
`networking.firewall.enable = false`: a NixOS firewall inside a
shared-netns container runs against the *host* ruleset — at container
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the
bridge holes above plus the agents' NAT. Private-netns containers
(agents, hive-ci) may keep their own firewall — it is scoped to their
namespace.
### Reaching host services (`exposeHostPorts`)
By default agents can only reach the host on 80/443 (+53 DNS), so a
host-side service on another port — e.g. a dev OTLP collector you want
agents to reach directly — is unreachable. (hyperhive's own telemetry
needs none of this: `otel.enable` opens its collector's port itself, and
`otel.endpoint` is the *upstream*, which no agent ever dials. See
`docs/observability.md`.)
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
agent can connect to `<bridgeIp>:P` (point the collector endpoint at
`http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
This is **firewall-only**: the host service must bind an address
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
service bound to `127.0.0.1` only stays unreachable; rebind it to
`0.0.0.0`.
The port is reachable by **every** agent on the bridge subnet (like
DNS/gateway), so only expose services safe for any agent to reach.
## Container isolation
Each agent container runs in a private network namespace with a dedicated
veth pair attached to the bridge. The following table summarises what
the nix side sets up unconditionally:
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
`10.42.0.1/24`), **not** the canonical network address. The Rust side
must normalise (bitwise-AND with mask) before subnet membership checks or
address arithmetic.
### What the Rust side does
`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
and `HOST_BRIDGE=<bridgeName>` via `lifecycle::set_nspawn_flags` when
creating or updating containers. Both variables are validated **once at
daemon startup**, not per container: they are process-global, so a
missing or malformed value is a misconfigured daemon rather than one bad
container, and failing at boot gives a single diagnostic instead of one
per agent. There is no non-isolated mode to fall back to. `LOCAL_ADDRESS` is left empty so the
container's dhcpcd acquires an address from the bridge dnsmasq pool
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies uniformly
to all containers — agents and service containers alike.
`HOST_ADDRESS` is the bridge gateway IP (the address part of
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
so a non-`.1` operator override still resolves to wherever the bridge
actually lives). It is **load-bearing**: nixos-container's container-side
network setup only installs a default route (`ip route add default via
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode the
host-side address/route setup is skipped, so writing it only affects the
container's default route — without it the container comes up with an IP
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
### How the isolated container gets its resolver
nixos-container copies the **host's** `/etc/resolv.conf` into the container
at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved,
or a LAN router) is unreachable from a private netns and isn't
authoritative for the hive's own zones, so it is replaced with the
bridge dnsmasq at boot. Because the copy happens on every start, a
declarative `environment.etc."resolv.conf"` would be clobbered — so the
wiring is runtime:
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
gateway IP) into each container's `/etc`.
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
`tea-login` so the resolver is correct before the first DNS lookup.
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
domain sockets bind-mounted into containers, not network listeners — see
the *Control plane (no network)* bullet under [Network
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
that never touches the network stack.
The nix side also enables IP forwarding + NAT (agents reach the internet
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP.
## Cross-references
- `docs/gateway.md` — vhost map + the gateway's other duties

224
docs/networking/snapshot-store.md Normal file
View file

@ -0,0 +1,224 @@
# Snapshot store
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
over the WireGuard mesh; a destination hive later pulls one back to
complete a migration.
Two things it is not, both worth stating because both are easy to
assume:
- **It is not the swarm controller**, and does not depend on one. It is
a NixOS host role: a btrfs subvolume tree, a socket-activated
receiver, and the `wg-hive` interface the swarm module already brings
up. That is why it can be deployed before any controller exists.
- **It is not a backup product.** It happens to hold the data a backup
would hold, and it should be operated accordingly (see
[Operating it](#operating-it)) --- but nothing in it does scheduling,
verification, or restore orchestration.
## Enabling it
```nix
services.hyperhive.snapshotStore = {
enable = true;
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
port = 51821;
};
# The mesh is a hard requirement, and is asserted:
services.hyperhive.swarm.wireguard = {
enable = true;
address = "10.100.0.9/24";
privateKeyFile = "/etc/wireguard/hive.key";
};
```
The store host is a swarm member like any other: it gets an entry in
`services.hyperhive.swarm.hives`, the same directory every host holds. See
[swarm/](../swarm/README.md) for the mesh itself.
Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
`c0re.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## Pointing a hive at it
The block above configures the host that *receives*. Every hive that
*pushes* separately needs to be told where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
address = "10.100.0.9"; # the store's mesh address, no prefix
port = 51821; # optional; must match the receiver's port
};
```
Two deliberate asymmetries in that pair, both easy to misread as
inconsistency:
- **`address` has no default.** It is a deployment fact a pushing hive
cannot derive, and a wrong guess means streaming an agent's state at
whatever happens to answer. Unset, a push fails naming this option.
- **`port` does default** (`51821`), because it is a convention both
ends read from the same option docs --- a default there is
coordination, not a guess.
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That is the distinction the
two namespaces carry throughout: `swarm.*` describes *the swarm* as seen
from this host, and a bare `services.hyperhive.<service>` describes *a
role this host performs*. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
[--parent <label>]` streams a snapshot straight into the store. There is
no destination argument, because a swarm has exactly one store (see
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
and no credential argument, because the mesh is the authentication.
## The mesh is the authentication
There are no certificates here, and no key material of its own. That is
deliberate rather than an omission.
WireGuard's cryptokey routing already binds a peer's source address to
its public key: the swarm module configures each peer with
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
that address provably came from the holder of that private key. A
packet that reaches the receiver has therefore already been
authenticated by the kernel.
Layering TLS client certs on top would authenticate *the same fact* a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
## One subvolume per agent, not per hive
The destination is keyed by **agent**.
This is not cosmetic. After a migration, an agent's next incremental
send arrives from a *different* hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and cannot choose
A `btrfs send` stream carries no notion of *which agent* it belongs to,
and the subvolume name inside it is chosen by the sender. So the
protocol is one `agent <name>` header line, then the raw stream.
The rule that matters:
> **The receiver owns the destination root. The sender-supplied name is
> validated, never used as a path.**
Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No
slash and no dot means neither directory traversal nor an absolute path
can survive it. It is deliberately a whitelist and not a list of
forbidden characters: a blocklist only ever excludes the attacks
somebody already thought of.
## Reachability
The receiver is socket-activated, and the socket binds **this host's
mesh address**, never a wildcard. Both the mesh being enabled and the
address being set are assertions, not documentation --- bound to
`0.0.0.0` this socket is an unauthenticated remote write into agent
state.
Binding is not sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, *before* a packet reaches a bound socket, so
the port is opened explicitly --- and scoped to the mesh interface:
```nix
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
```
A host-wide `allowedTCPPorts` would open the port on every interface
including a public NIC, leaving only the socket's bind address between
the internet and a root `btrfs receive`.
## Operating it
### Confinement is the deployment's job
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
narrow `ReadWritePaths` --- but those are **defence in depth, not a
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
undo the namespace they set up.
The boundary is the machine. The intended deployments are:
- **a swarm**: the store is its own small VM. The machine is the
boundary, which is stronger than anything the unit could assert about
itself.
- **all-in-one / local**: the store runs as a container on the c0re
host.
The second is worth keeping deliberately, and not only for
convenience: it means the confined path is exercised by every local
deployment. The usual failure mode for an isolated variant is that
nobody runs it day to day, so it rots and is discovered broken in
production.
⚠️ **The assumption to keep true over time:** the store host runs
nothing else. That is true on day one and quietly false the day someone
notices the box has spare disk. Nothing in the config objects when it
stops being true.
### It holds every agent's state from every hive
Which makes it the highest-value target in the swarm by a wide margin,
and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The trap is the label: this box holds backup-grade data while not being
called a backup, so it can end up with backup-grade *exposure* and
non-backup-grade *controls*. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
The snapshot covers an agent's **state subvolume**, which is the parent
of `state/`, `claude/` and `harness/` (see
[`docs/persistence.md`'s btrfs subvolume
section](../agent-lifecycle/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
for how and when that subvolume is created). Consequences:
- The Claude session (`claude/`) travels, so a restored agent keeps its
live `--continue` session rather than needing to log in again.
- `harness/` travels too, including `harness/bash-tasks/`. Task output
is part of an agent's working continuity, so this is wanted --- but it
means anything that has ever leaked into a task's captured output is
in the retained snapshots as well.
It does **not** cover the agent's applied config (`/applied/<name>/`) or
its topology entry, both of which live outside the subvolume. A restore
therefore yields an agent's memory without its definition; closing that
gap is tracked separately.
### Retention
Retention lives on the *sending* side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.
Per-agent or per-hive `btrfs qgroup` quotas are not configured yet.
Without them one runaway hive can fill the store and take out every
other hive's snapshots.
## Not built yet
**The pull side.** Push is safe with minimal authorisation because a
hive can only ever write to a chain it owns. Pull is the direction that
needs a policy: unrestricted, any compromised hive could read every
agent's state from every other hive. It needs a notion of which hive
currently owns which agent, and that ownership record lands with the
swarm controller work.
With a single hive the question is trivial --- the only peer owns
everything it sends --- which is why the receive half ships first.