mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
3 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

478
docs/4-implementation/networking/gateway.md Normal file
View file

@ -0,0 +1,478 @@
# hive-gateway
Every host's nginx: the one front door for whatever this host serves. A swarm service running here (forge, matrix, SSO, the swarm UI, the metrics and log stores) declares its own vhost through the gateway; the gateway itself adds the hive's own surface — dashboard, per-agent UIs, and discovery for a server name pinned to the hive domain.
_For the operator configuring `services.hyperhive.gateway.*` on a host._ nginx and the hive resolver (dnsmasq) run on the host next to hive-c0re: they bind `:80`/`:443` and the bridge address.
You rarely switch it on yourself. `gateway.enable` defaults to off, and every module that serves a vhost or needs hive names to resolve sets `gateway.enable` / `gateway.dns.enable` with `mkDefault true` — the hive controller, each swarm service, CI.
## Vhost map
**Swarm services.** Each module declares its vhost on the host that runs the service, under the swarm domain:
| URL | upstream | declared by, when |
| --- | --- | --- |
| `<swarm>/` | swarm-ui dist (static), behind an authelia subrequest | `swarm-ui.nix`, `deploy.swarm-controller.enable` |
| `auth.<swarm>/` | authelia (unix socket `/run/swarm-authelia/authelia.sock`) | `swarm-authelia.nix`, `deploy.authelia.enable` |
| `auth.<swarm>/bridge/*` | swarm-authelia-bridge (unix socket `/run/swarm-authelia/bridge.sock`), prefix stripped; answers swarm-controller's client only | `swarm-authelia.nix`, `deploy.authelia.enable` |
| `forge.<swarm>/` | forgejo (`3000`) | `hive-forge/`, `deploy.forgejo.enable` |
| `chat.<swarm>/_matrix/*` | tuwunel (unix socket `/run/hive-matrix/tuwunel.sock`) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `chat.<swarm>/` | fluffychat-web static | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `chat.<swarm>/config.json` | inline JSON (FluffyChat boot config) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `<queue domain>/metrics` | the queue exporter's `/metrics` on loopback, behind an authelia `auth_request`; every other path, `/` included, returns `403`, so `/varz` and `/connz` are never reachable | `swarm-nats.nix`, the queue's host |
| `grafana.<swarm>`, `metrics.<swarm>`, `logs.<swarm>`, `otel.<swarm>`, `bao.<swarm>` | the matching swarm service | that service's module → [`swarm/services.md`](../../2-handbook/swarm/services.md) |
A service module declares its vhost as `services.hyperhive.gateway.vhosts.<name>`: the locations, `forceSSL` and any `extraConfig` it needs. The gateway renders it into `services.nginx.virtualHosts.<name>` and adds the `listen` set, the certificate for that name and the [security headers](#security-headers) (ahead of the vhost's own `extraConfig`), and drops `addSSL` from a vhost that sets `forceSSL`. The eval fails for a vhost that sets `listen`, `addSSL`, `onlySSL`, `enableACME`, `useACMEHost`, `sslCertificate` or `sslCertificateKey` itself, and for any vhost declared while `gateway.enable` is false. A name with a certificate from outside the gateway's leaves (the queue's) sets `certificate = { cert; key; }` instead. `hive-gateway/vhosts.nix` declares the gateway's own catch-all (`_`) the same way, `hive-c0re/vhosts.nix` the dashboard and the hive domain, and `glue-matrix-hive-discovery.nix` adds the matrix redirect to the hive domain where the homeserver runs, and discovery there when the hive domain is the pinned server name. The catch-all sets `securityHeaders = false` because it sends no response, and the gateway renders a vhost's `default` at `mkDefault` priority.
**The hive's own vhosts**: the dashboard's, named `<name>.hives.<swarm>` (`deploy.hive-controller.dashboardDomain`), and one named for the hive domain. They exist only on a host that runs a hive (`deploy.hive-controller.enable`); a host without a hive has none of these rows:
| URL | upstream | when |
| --- | --- | --- |
| `<name>.hives.<swarm>/` | dashboard dist (static, from `servedFrontend`), behind authelia | always |
| `<name>.hives.<swarm>/api/` | hive-c0re (`7000`), behind authelia | always |
| `<name>.hives.<swarm>/health/` | hive-c0re (`7000`) | always |
| `<name>.hives.<swarm>/api/docs/` | themed Swagger UI dist (static), behind authelia | always |
| `<name>.hives.<swarm>/agent/<name>/` | per-agent harness over its unix socket | `agents.conf` (runtime-generated) |
| `<swarm>/.well-known/matrix/{client,server}` | inline JSON | `deploy.swarm-controller.enable`, and `swarm.matrix.serverName` is the swarm domain (the default) |
| `<hive>/.well-known/matrix/{client,server}` | inline JSON | `swarm.matrix.serverName` is this hive's domain (a pinned homeserver) |
| `<hive>/matrix/` | 301 → `chat.<swarm>/` | `deploy.matrix.enable` and `gatewayHost` set |
| `<hive>/` (every other path) | 301 → `<name>.hives.<swarm>`, same path | always |
The catch-all `_` vhost answers any other `Host` with `444` (connection closed, no response). It's `mkDefault`, so to make your own vhost the default server, set `services.nginx.virtualHosts."_".default = false;` — an eval assertion names both when two claim it.
Only the host that **runs** authelia declares `auth.<swarm>`; a hive that merely uses SSO knows `swarm.authelia.url` but doesn't answer for that name. The server name must be exactly `swarm.authelia.domain` — authelia checks that `authelia_url` sits inside its session cookie domain at startup, and refuses to boot otherwise. The vhost carries no `auth_request` (that would put the login page behind the login it serves) and passes `X-Forwarded-{Proto,Host,Uri,For}`, because authelia decides by the original request.
⚠️ `auth.<swarm>` answering with the **sso unavailable** page means authelia isn't answering at all — check `journalctl -M swarm-authelia -u authelia-swarm`. While the user set is empty, a placeholder user marked `disabled` keeps authelia's user store non-empty, so it boots fine and serves a login page that refuses everyone; add the first account with `swarmctl user add …` ([`setup.md`](../../1-getting-started/setup.md)).
Per-agent UIs stay sub-path; forge and matrix get sub-domains → [Sub-domain shape](#sub-domain-shape-rationale).
## TLS modes
The gateway always terminates TLS. Which certificate it serves depends on what you configure:
| mode | config | cert source | `.well-known` scheme |
|---|---|---|---|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | the secret store issues a gateway leaf through the hive's own role (RSA-4096) | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
`tls.certDir` together with `tls.acme.enable` fails an assertion — pick one.
### ACME / Let's Encrypt (`tls.acme`)
The simplest production path for a public domain:
```nix
services.hyperhive.gateway = {
openFirewall = true;
tls.acme = {
enable = true;
email = "admin@example.com"; # required
};
};
```
nginx obtains and renews certs via the HTTP-01 challenge on `port` (default 80); they land in `/var/lib/acme/`, managed by nixpkgs's `security.acme`. Every name this host serves needs a public DNS record pointing here — the hive domain and each swarm-service name in the [vhost map](#vhost-map) — and `openFirewall = true` so Let's Encrypt reaches `/.well-known/acme-challenge/`.
### Self-signed TLS (default)
On by default, listening on `httpsPort` (default 443) beside the plain-http `port` (default 80). `httpsPort` only sets the listen port: swarm service URLs never carry a port, so clients reach them on 443.
The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `hive-tls-resign.service` (from `hive-tls.nix`) asks the store for a gateway **leaf** through this hive's own role, presenting the hive's own store identity (`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`), and writes it under `…tls.stateDir` (default `/var/lib/hive-tls`). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). On a host that runs no hive the import unit installs the swarm-services leaf there instead, or a throwaway self-signed pair until that leaf lands. [`swarm/ca.md`](../swarm/ca.md) has the role, the identity and what a hive without one does.
⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the source's `0600 root:root` key), and it puts a file at **every cert path the config names**, because nginx refuses a config naming a missing cert file and one missing leaf would take down every vhost, not just one. Where the gateway waits for the swarm-services leaf, a missing one fails the import unit, naming `swarm-services-cert.service`, and nginx stays down until that unit issues it. Where it doesn't — a host fronting no swarm name, or only the store's UI, which starts its passthrough with the store sealed — the hive leaf stands in until the services leaf lands, or on a host that runs no hive a throwaway self-signed pair.
**Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them.
**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. The CI and forge containers and federating peers read the bundle. Agents fetch the root from the store themselves ([`swarm/ca.md`](../swarm/ca.md#distributing-the-root)).
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https://<host>/.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap.
**Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive's role allows nothing outside `<hive>`, so a swarm-service name can't go on this leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
**Rotation**: `hive-tls-resign.service` runs at boot, on every deploy and once a day, and asks the store for a new leaf when the current one is missing, past half its 720-hour lifetime, doesn't cover the configured names, doesn't match its key, or doesn't chain to the store's root. A leaf already on disk lets the gateway start while the store can't answer; a failed renewal leaves it in place. To force a rotation, delete `gateway.pem` under the state dir and restart the unit; it re-imports the leaf and reloads `nginx` itself.
**Cert prompts**: browsers warn once per host until you add the hive's `trust-bundle.pem` (the anchor, not the leaf) to the browser or OS trust store. A container that needs to trust it for its own outbound TLS gets the bundle a different way, bind-mounted read-only at `/run/hive-ca/trust-bundle.pem` (`nix/host-modules/lib/hive-ca-trust.nix`).
### Operator-provided cert (`tls.certDir`)
For a cert from a real CA (Let's Encrypt via your own `security.acme`, a corporate CA):
```nix
services.hyperhive.gateway = {
tls.certDir = "/var/lib/acme/example.com"; # nixpkgs security.acme output dir
# tls.certName = "cert.pem"; # default — matches security.acme layout
# tls.keyName = "key.pem"; # default — matches security.acme layout
};
```
nginx reads the directory directly. Keep the key readable by nginx:
- `security.acme` writes keys `0640 root:acme`, which the `nginx` user can't read. Set `security.acme.certs."example.com".group = "nginx";` (or make the key `0644` if your threat model allows). Otherwise nginx fails at startup with the reason in the journal.
### Fronting with an external TLS terminator
Give the gateway the real cert (`tls.certDir` or `tls.acme`) so it serves proper TLS itself, or front it over a unix socket rather than a plain-http TCP port. `.well-known/matrix/*` responses always advertise `https` ([Discovery flow](#discovery-flow-matrix)).
## Dashboard access
The dashboard answers on `<name>.hives.<swarm>` (`deploy.hive-controller.dashboardDomain`, derived from the hive name and `swarm.domain`), behind an authelia `auth_request`. Log in with your authelia account; only accounts in the `admins` group get in, the same rule as the swarm UI. Add one with `swarmctl user add <name> --group admins` ([`swarm/sso.md`](../../2-handbook/swarm/sso.md)).
The name sits under the swarm domain so authelia's session cookie reaches it, whatever the hive's own domain is. Any path on the hive domain other than the matrix routes redirects there, so old bookmarks still land.
**What it gates:** `/`, `/api/` and `/api/docs/`. **Not gated:** `/health/` (for uptime monitors; status only) and the per-agent `/agent/<name>/` routes, which come from `agents.conf` and carry no `auth_request`.
**Certificate:** in self-signed mode the name is a SAN on the hive's own leaf, issued through its own store role, so it needs nothing extra. With `tls.acme` it needs a public DNS record like any other vhost. With `tls.certDir` your certificate must name it; a `*.<swarm>` wildcard doesn't cover `<name>.hives.<swarm>`.
## Firewall posture (host-level)
`services.hyperhive.gateway.openFirewall = true` opens `port` and `httpsPort` on the host firewall — both, since the gateway always serves TLS. It defaults to off; set it for any reach from outside the host.
nginx is the one external entry point. The per-agent web-port range (`8100`–`8999`) stays closed: agents serve their UI on a unix socket ([Per-agent unix-socket upstream](#per-agent-unix-socket-upstream)), and the hashed TCP port (`lifecycle::agent_web_port`) is only a fallback bind for an agent missing `HIVE_WEB_SOCKET` — the gateway never proxies through it.
The dashboard port (`services.hyperhive.deploy.hive-controller.dashboardPort`, default 7000) binds `127.0.0.1` only, so remote dashboard access goes through the gateway. The dashboard can stop, rebuild and destroy agents; don't expose it without a reverse proxy in front.
nginx serves its `stub_status` counters on `127.0.0.1:<statusPort>` (`services.hyperhive.gateway.statusPort`, default 8085) at `/stub_status`, from a server of its own outside the vhost tree. The hive's collector reads them with its `nginx` receiver, and the `gateway` Grafana dashboard plots them. The receiver only dials `http://`, which is why this is loopback TCP rather than a unix socket.
## `HIVE_FORGE_URL`: agents reach the forge via the gateway by domain
Agents poll `HIVE_FORGE_URL` for Forgejo notifications and run every `hive-forge` call against it. `nix/host-modules/hive-c0re/environment.nix` sets it to `http://<swarm.forge.domain>` (default `forge.<swarm-domain>` — a swarm runs one forge). Agents run in a private netns and can't reach the host's loopback, so they resolve that name through the bridge dnsmasq to the bridge IP and reach nginx on port 80 (the bridge firewall opens 80 and 443) — the same path an operator's browser takes.
## hive-forge container shape
Private Forgejo in a nixos-container named `hive-forge` (not `h-*`, so c0re's lifecycle scanner leaves it alone; manage it with the standard `nixos-container` CLI). The container keeps it from colliding with any `services.forgejo` the operator already runs on the host: separate systemd namespace, separate state dir, separate port.
It shares the host network namespace (`privateNetwork = false`): the container is for state and unit isolation, not network isolation. Agent containers, by contrast, are network-isolated and reach the forge through the gateway ([`HIVE_FORGE_URL`](#hive_forge_url-agents-reach-the-forge-via-the-gateway-by-domain)).
State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/` and survives restarts and reboots; destroy the container to wipe it.
Only the swarm's forge host runs it (`services.hyperhive.deploy.forgejo.enable`, see [`swarm/services.md`](../../2-handbook/swarm/services.md)). Every other hive reaches that host's gateway by `forge.<swarm-domain>`.
### Network and port configuration
```nix
# The SSH port is swarm-wide: every hive dials it. The HTTP port and whether
# THIS host opens them in its firewall are this host's deployment decisions.
services.hyperhive.swarm.forge.sshPort = 2222; # default — git-over-SSH, off the host's openssh on 22
services.hyperhive.deploy.forgejo.httpPort = 3000; # default — outside hyperhive's 7000 / 8100-8999
services.hyperhive.deploy.forgejo.openFirewall = false; # default
```
`sshPort` serves `git clone/push/pull` over SSH (`git@<domain>:owner/repo.git` with `-p 2222`). SSH goes straight to Forgejo, not through nginx.
`openFirewall` (default `false`) opens `httpPort` and `sshPort` on the host. Agents don't need it — they come through the gateway. Set it for a browser reaching `http://<host>:<httpPort>/` directly, or for external git clients pushing over SSH. Forge's gateway vhost doesn't need `openFirewall` — the gateway's own `openFirewall` option covers that path.
### `rootUrl` override
```nix
services.hyperhive.swarm.forge.rootUrl = "https://forge.example.com/";
```
`rootUrl` (default `null`) overrides the Forgejo `ROOT_URL` that's autoderived as `https://<forge.domain>/`. The gateway always terminates TLS, so the forge is always advertised over `https://`. Set `rootUrl` explicitly when `forge.domain` differs from the public URL, or for a bespoke shape such as an external reverse proxy on another host or path. It must end with `/` (an assertion enforces this).
## Security headers
Every named vhost sets these at server scope (the `_` catch-all only closes connections):
| Header | Value |
|--------|-------|
| `X-Frame-Options` | `SAMEORIGIN` |
| `X-Content-Type-Options` | `nosniff` |
| `Referrer-Policy` | `strict-origin-when-cross-origin` |
nginx doesn't merge `add_header`: a `location` that sets its own (the CORS locations `/.well-known/matrix/client` and `/_matrix/`) inherits none of the server-scope headers, so those locations repeat them. Locations with no `add_header` of their own pick them up.
### HSTS (`gateway.hsts`)
Opt-in, off by default:
```nix
services.hyperhive.gateway.hsts = {
enable = true; # default: false
maxAge = 31536000; # default: 1 year (required for preload list)
includeSubDomains = true; # default: true
};
```
When enabled, every vhost adds `Strict-Transport-Security: max-age=…[; includeSubDomains]`. HSTS pins https in the browser: a deployment that later loses TLS locks browsers out until `max-age` expires, so enable it only when TLS is permanent.
## Local dev (`localHostsEntry`)
`services.hyperhive.gateway.localHostsEntry = true` maps to `127.0.0.1` in the host's `/etc/hosts`:
- the hive domain;
- every name a module on this host contributes to `gateway.localNames` — the dashboard's `<name>.hives.<swarm>` where a hive runs, and each swarm service this host runs adds its own (`forge.<swarm>`, `chat.<swarm>`, `auth.<swarm>`, the swarm UI's apex, …).
`services.hyperhive.deploy.singleHostSwarm` turns it on. Leave it off with real DNS.
## Discovery flow (matrix)
The server name's own vhost serves discovery, whichever host runs the homeserver. By default that vhost is the swarm apex on the controller's host; a homeserver pinned to a hive domain gets it on that hive's vhost instead. Both build the JSON from `swarm.matrix.gatewayHost` alone, and the `base_url` carries no port.
The operator points a client at the server name (`<server-name>` below):
1. The client fetches `https://<server-name>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://chat.<swarm>"}}`.
2. It connects to `chat.<swarm>/_matrix/client/...`.
3. The gateway proxies `/_matrix/*` to tuwunel over its unix socket, `/run/hive-matrix/tuwunel.sock`.
matrix-dart-sdk (FluffyChat and others) always fetches the well-known over `https`, which is why [self-signed TLS](#self-signed-tls-default) is on by default.
Federation peers fetch `.well-known/matrix/server` → `{"m.server":"chat.<swarm>:443"}`. The port is explicit even though it's the https default: a delegated host without a port means the federation default 8448, not 443. Peers then reach `/_matrix/` on the chat vhost through the gateway, so the gateway must be reachable from them (`gateway.openFirewall`).
## Sub-domain shape (rationale)
Sub-domain for forge and matrix, sub-path for per-agent UIs:
- forgejo's default `ROOT_URL = http://<host>/` works without `X-Forwarded-Prefix` handling; sub-domain hosting is the canonical Forgejo shape.
- matrix deployments put the API listener on its own name, and federation expects that. `gatewayHost` defaults to `chat.<swarm-domain>` rather than the spec-conventional `matrix.<server_name>`; set it explicitly for the conventional label.
- per-agent UIs are hyperhive-internal and base-path-aware for `/agent/<name>/`. A sub-domain per agent would multiply DNS and TLS cost for nothing.
- cookie and storage isolation: a forge XSS can't reach the dashboard session, because they're different origins.
`services.hyperhive.swarm.forge.domain` and `services.hyperhive.swarm.matrix.gatewayHost` take a full hostname (`forge.darkest.space`, `git.example.com`), not a label glued to a domain.
## Tuning knobs
Per-vhost timeouts and body-size limits live in the location blocks:
- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), websockets on.
- matrix `/_matrix/` (tuwunel): `client_max_body_size` = `deploy.matrix.maxRequestSize` + 1 MiB, so tuwunel stays the tighter limit and returns a matrix error a client can act on; `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*`, websockets on.
- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE and WebSocket UIs), websockets on, `X-Forwarded-Prefix` set so the harness can build absolute URLs.
## Internals
How the gateway does what the sections above describe. Read before changing `hive-gateway/`, `gateway_nginx.rs` or `agent_sockets.rs`.
### SPA fallback (Accept-header pattern)
The `chat.<swarm>` vhost (fluffychat) serves a flutter/SPA bundle via the Accept-header pattern below. Per-agent UIs solve the same two requirements differently, with file-existence `try_files` — see [Per-agent static frontend split](#per-agent-static-frontend-split). The dashboard routes by **path** — see [Dashboard: path-based routing](#dashboard-path-based-routing-not-accept-header) below. Two requirements collide for a vhost whose client-side router owns every route under `/`:
- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
- a non-navigation request that isn't an on-disk asset must NOT get HTML with the wrong content-type
Solution: an `nginx http`-context `map $http_accept $matrix_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; everything else (`Accept: image/*`, `*/*`, `application/json`, `text/event-stream`, …) gets a sentinel nonexistent path, so `try_files $uri $uri/ $matrix_spa_target =404` falls through to a plain `404` — a missing asset is just missing.
#### Dashboard: path-based routing (not Accept-header)
hive-c0re serves exactly two prefixes, so the dashboard routes by **path** — deterministic, where a content-type split would let one URL resolve differently by the caller's `Accept` header:
- `location /api/` → hive-c0re (`7000`): all dashboard data, actions, and the two SSE streams (`/api/dashboard/stream`, `/api/build-logs/id/{id}/stream`). `proxy_buffering off` and a 1d read timeout keep the streams live.
- `location /health/` → hive-c0re: liveness and readiness.
- `location /` → the dashboard dist (from the `servedFrontend` nix-store path) with `try_files $uri /index.html`.
The gateway static-serves the dist while hive-c0re stays API-only, so a frontend-only change doesn't restart the core daemon. A new top-level c0re route prefix needs a matching `location` on the dashboard vhost (`hive-c0re/vhosts.nix`).
### Per-agent unix-socket upstream
Every agent binds its web UI on a unix-domain socket at
`/run/hive-agent/<name>/web.sock`. The mechanism:
1. **Agent side**. The nix module sets `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
on every harness service env; `web_ui::serve` binds a
`UnixListener` at that path.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
(`/run/hive-agent/<name>/`) into the agent's container. Dir
bind, not file bind — file bind-mounts don't survive the
harness's `unlink + bind(2)` cycle on socket replace. Per-agent
subdir keeps each agent's container from seeing siblings' sockets.
The dir is `0751`, owned by the agent's container uid/gid (hive-priv
creates it `0751 root` before each start when missing, and the
container's activation hands it to the agent user), so
nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
own `0666`. The gateway is one of three principals sharing that dir
and doesn't own its ownership rules — see
[`docs/2-handbook/trust-boundary/boundary.md`](../../2-handbook/trust-boundary/boundary.md#the-per-agent-socket-dir).
3. **Marker gate**. After a successful `bind_unix`, the harness drops
`<dir>/hyperhive-socket-bound` next to the socket. c0re's
`agent_sockets::write` filters its JSON map by marker presence —
only agents whose harness has bound the socket appear there.
It also accepts `.bound`.
4. **Gateway side**. `gateway_nginx::write` generates
`/var/lib/hive-gateway/conf/agents.conf` — a plain nginx include
file with one `location /agent/<name>/` block per agent. Always
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
if the socket isn't yet bound, nginx returns 502 caught by the
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
nginx includes `/var/lib/hive-gateway/conf/agents.conf` — the same
path c0re writes, since both run on the host.
After each write, c0re triggers the appropriate nginx action via
`hive-priv` (which is root; hive-c0re runs as the unprivileged
`hive-core` user and can't act on a system unit).
`hive-priv` queries `ActiveState` and dispatches:
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
- otherwise → `systemctl start nginx`
This is an explicit trigger rather than a path unit watching the
file: the write and the reload belong in one causal chain c0re can
retry and report on (`RELOAD_PENDING`), not two units racing on an
inotify event.
<!-- vale write-good.Passive = NO -->
c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
marker poll tick (`agent_sockets::spawn_poll`). `write()` is
idempotent — skips the rename when content is unchanged.
`gateway_nginx::reload_if_pending` automatically retries failed reloads
on subsequent poll ticks.
<!-- vale write-good.Passive = YES -->
`agents.conf` goes out through a unique tmp file and an atomic `rename()`, so a crashing
c0re process never leaves a partial or unparseable file behind.
### Per-agent static frontend split
hive-c0re's environment sets `HIVE_AGENT_FRONTEND_DIR` to the `agent/`
subdirectory of `services.hyperhive.deploy.hive-controller.servedFrontend`.
The nginx include generator (`gateway_nginx::write`) reads
this variable and, when set, emits split location blocks per agent
instead of a single proxy block.
**Location priority for `/agent/<name>/...`:**
```nginx
# 1. Compiled assets — content-addressed nix store path, cache forever
location ^~ /agent/<name>/static/ {
alias <frontend>/static/;
expires 1y;
add_header Cache-Control "public, immutable, max-age=31536000";
}
# 2. Static dist + proxy fallback for dynamic paths
location /agent/<name>/ {
alias <frontend>/;
try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}
# 3. Proxy catchall — API, events, icon, send, login, …
location @<name>_dynamic {
proxy_pass <upstream>;
proxy_set_header X-Forwarded-Prefix /agent/<name>;
proxy_intercept_errors on;
error_page 502 503 504 = /__hive_agent_unreachable;
# … (full proxy header block)
}
```
**`try_files` resolution** (nginx applies the `alias` mapping before
checking file existence):
| request | resolved | outcome |
| --- | --- | --- |
| `/agent/iris/` | `<frontend>/index.html` | main agent page |
| `/agent/iris/stats` | `<frontend>/stats.html` | stats page |
| `/agent/iris/screen` | `<frontend>/screen.html` | screen page |
| `/agent/iris/static/app.js` | caught by `^~` block first | served with immutable cache |
| `/agent/iris/api/state` | no file match → `@iris_dynamic` | proxied to agent daemon |
| `/agent/iris/events/live` | no file match → `@iris_dynamic` | proxied (SSE) |
Adding a new HTML page to the frontend dist (`dist/<page>.html`)
makes it reachable at `/agent/<name>/<page>` with no generator change.
**Why `^~` for `/static/`**: the `^~` prefix gives this block higher
priority than the plain prefix `location /agent/<name>/`, so compiled
JS/CSS assets skip `try_files` entirely and get the immutable cache
headers. Nix store paths are content-addressed — the hash changes on
any content change — so `max-age=31536000` is safe.
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
store path baked in at hive-c0re build time, and c0re (writing
`agents.conf`) and nginx (serving files from it) are on the same machine,
so they see the same store.
**Unset variable**: if `HIVE_AGENT_FRONTEND_DIR` is empty or unset, each
agent gets a single proxy block and nginx forwards all traffic to the
agent daemon.
**`extraFiles`**: per-agent `services.hyperhive.agent.frontend.extraFiles` sit in
`mergedDist`, not in the base `services.hyperhive.deploy.hive-controller.frontend` dist. They're not under
the nix-store `alias` path, so requests for them fall through
`try_files` to `@<name>_dynamic`, and the agent daemon serves them.
### Per-agent error pages
`/agent/<name>/` requests hit two failure modes; both get static
HTML pages instead of nginx's default error chrome:
- **Agent not found** (`/agent/<unknown>/...`) — no `location` for that
name in `agents.conf`. nginx's prefix match falls back to the bare
`/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
to `/__hive_agent_not_found` → serves `not-found.html` with a link
back to the dashboard.
- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
per-agent harness isn't responding (container restarting, crash
recovery). `proxy_intercept_errors on` + `error_page 502 503
504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.
`hive-gateway/error-pages.nix` renders every page (`notFound`, `unreachable`,
`ssoUnavailable`) from one template with `pkgs.writeText`;
nginx serves each through an `internal` location with `alias` to the file,
so only nginx's own error handling can reach them. They depend on nothing
from the frontend dist and render even when hive-c0re is down. A service
module reaches them through `gateway.lib.errorPages`.
A route earns a custom page when the default status code would blame the
wrong component. The per-agent routes qualify (a 502 there means the
harness is restarting, not a gateway fault), and so does
`auth.<swarm>` — a bare 502 there blames the proxy, but the gateway is
fine; authelia itself isn't answering.
Forge, matrix and fluffychat keep nginx's defaults: a dead upstream there
means what the status code says.
### Dashboard link shape (gateway vs direct)
The dashboard builds per-agent links as same-origin `/agent/<name>/…` URLs
when `StateSnapshot.gateway_enabled` is true, and direct
`http://<host>:<port>/` links otherwise. The flag comes from
`HIVE_GATEWAY_ENABLED`, which `hive-c0re/environment.nix` sets to `1` on
every hive, so the direct shape is a fallback for the variable being unset.
Three render sites follow it: the agent-name link, the favicon fetch
(`<url>/icon`), and the nav-strip `container`-kind links from
`DashboardState.links` (`GET /api/dashboard-state`). Forge links come from
`swarm.forge.publicUrl`; with it unset the dashboard hides them rather than
guess. See `docs/2-handbook/web-ui/dashboard.md::Container row` for the frontend side.
### Dialing another vhost by name (`verifiedProxyTo`)
<!-- vale write-good.Passive = NO -->
`vhost-lib.nix`'s `verifiedProxyTo` builds the `proxy_ssl_*` /
`proxy_set_header` block a module uses to dial another service on this
same gateway BY NAME over https, verified. One definition rather than a
copy per module: nginx verifies nothing by default
(`proxy_ssl_verify` is off), so a `proxy_pass https://…` without these
lines is encrypted and unauthenticated. That failure is invisible — it
works, and keeps working, against any certificate at all.
<!-- vale write-good.Passive = YES -->
Every line earns its place, each confirmed against a real nginx with
the opposite arm run as a control:
- `verify` + `depth` — the chain is leaf -> intermediate -> root.
- `trusted_cert` — the bundle; nginx reads ALL certs in the file, which
the bundle's own doc warns isn't true of every consumer.
- `ssl_name` — checks the HOSTNAME too. Without it a chain-only check
accepts any certificate this CA ever signed, and for an internal CA
that's every service on the hive.
- `server_name on` — sends SNI, or the far end can't pick a cert.
**⚠️ Session-cache footgun**: the module leaves `proxy_ssl_session_reuse` at its
default (on), deliberately — nginx uses this on per-request auth
subrequests, so every request pays the handshake it avoids. Worth
knowing when testing though: nginx keys the session cache by upstream
address and NOT by trust config, so two locations pointing at one
upstream with different trust don't verify independently.
**⚠️ Host-header clobber footgun**: `verifiedProxyTo` also pins `Host`
(and reinstates the rest of nginx's `recommendedProxySettings` header
set) to the target `name` rather than letting nginx fill it in later.
`name` here resolves back to THIS gateway — every consumer dials another
vhost on the same nginx, not a separate host — and nginx picks the vhost
to answer an HTTPS request from the `Host` header, not from the TLS SNI
that `proxy_ssl_name` sends. `recommendedProxySettings`'s own `Host
$host` (the CALLER's host, not the target) is textually appended by
nixpkgs AFTER a location's `extraConfig` — so it always wins over a
`proxy_set_header Host` written in the location body, and the subrequest
loops back into the ORIGINAL vhost instead of reaching the target,
recursing on its own `auth_request` until nginx's subrequest-depth limit
turns it into a plain 500. Every call site sets `recommendedProxySettings
= false` on the location for exactly this reason — nixpkgs' version
would still clobber this one.

580
docs/4-implementation/networking/network.md Normal file
View file

@ -0,0 +1,580 @@
# hive-network
The bridge every container on a host attaches to, and the private-netns
isolation of agent and CI containers behind it. It comes up on any host that runs agents or swarm services:
`services.hyperhive.network.enable` defaults to off, and the modules that
need the bridge set it with `mkDefault true` — the hive controller, CI,
the hive collector, and the gateway's resolver, which every swarm service
turns on. Configured via `services.hyperhive.network.*`.
Agent containers always run in a private netns.
## Network map
One picture of the whole hive — two planes: **infra
containers share the host netns** and bind host ports directly;
**compute containers (agents + CI) each get a private netns** behind
the bridge. The unix-socket control plane rides the VFS, and none of
that touches it.
```
internet
│ uplink NIC — NAT MASQUERADE for the
│ bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge │
│ │
│ nginx :80/:443 [hive-gateway] │
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
│ │
│ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
vb-h-<a> vb-h-<b> vb-<runner> veth pairs
│ │ │
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
│ agent a │ │ agent b │ │ runner │ each; eth0 leases
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
└─────────┘ └─────────┘ └─────────┘
```
| container | netns | IPv4 | listens / reached via |
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `swarm-review-classifier` | private, veth on bridge | DHCP pool | outbound only (classifier → queue, secret store, swarm collector); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end:
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
`dhcpcd` runs with `noipv4ll` and waits for an IPv4 address, so
`network-online.target` means a lease, never a 169.254.x fallback; units
that look up a swarm name at start order after it.
- **DNS** — agents and the service containers query `bridgeIp:53`; dnsmasq
answers hive zones authoritatively with the bridge IP, everything else
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
container points its own `resolv.conf` there, and one that instead
inherits the host's resolves no swarm name at all — those records exist
only on the bridge.
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
own dashboard name resolve to the bridge IP, land on nginx
`:80`/`:443`, and proxy to forgejo
`:3000`, tuwunel's unix socket, hive-c0re `127.0.0.1:7000`, or a per-agent
UI unix socket.
- **Internet egress** — agent default route points at the bridge IP;
the host forwards + masquerades out its uplink.
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
and reach each other's gateway/forge across it
([`docs/swarm/`](../../2-handbook/swarm/README.md)).
- **Control plane (no network)** — per-agent broker socket
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
sockets under `/run/hive-agent/` are unix domain sockets
bind-mounted through the VFS; private netns doesn't affect them.
## Container shape (where dnsmasq lives)
dnsmasq runs on the host itself, next to nginx — both come from the
gateway module, one front door for DNS and HTTP. Its `bind-interfaces`
listener sits on `bridgeIp`, on the host's bridge interface.
## Configuration
```nix
{
services.hyperhive = {
deploy.hive-controller.enable = true;
deploy.hive-controller.hiveName = "pr1ma";
swarm.domain = "darkest.space";
swarm.hives.pr1ma = { }; # -> domain = pr1ma.darkest.space
# network.bridgeIp = "10.42.0.1"; # default
};
}
```
The hive domain (`services.hyperhive.domain`, which the resolver answers
for) comes from this hive's entry in the swarm directory:
`swarm.hives.<hiveName>.domain`, default `<hiveName>.<swarm.domain>`.
Eval fails until `swarm.domain` and that entry exist
([`swarm/README.md`](../../2-handbook/swarm/README.md) § Hive identity config).
## Bridge addressing
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
RFC 1918 space, unlikely to clash with operator's existing setup;
override `bridgeIp` + `bridgePrefixLength` if a different range is
already in use. `/24` gives 254 usable per-agent addresses — enough
for any single-host hive; bigger swarms or tighter addressing
schemes pick their own.
## Resolver behaviour
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
plus whatever swarm-service names this host contributes via
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
(matrix) when this host runs those services, `auth.<swarm-domain>`
when it runs authelia, and `hive-collector.hive` on every hive
— answering each with the bridge IP (where nginx
is reachable, and where the hive's own collector listens). Note forge and
matrix are swarm-domain names, not sub-domains of `<hive-domain>`: a swarm
runs one forge and one homeserver, so their names belong to the swarm rather
than to whichever hive happens to host them.
Every domain answered this way is also `local=` for dnsmasq, so the other
record types (AAAA, TXT, MX, HTTPS) for those names get an empty answer from
dnsmasq itself instead of waiting on the host's upstream resolver. Without it,
dnsmasq 2.86 and later send them upstream, and a host whose upstream doesn't
answer leaves each lookup to its client's own timeout.
dnsmasq forwards everything else to the host's
own resolvers: it runs on the host and reads the host's
`/etc/resolv.conf` directly. Containers don't need to know the
upstream — they query the bridge IP and dnsmasq does the right thing
per-name.
No fallback `server=` exists, by design: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
directly, so a network change (new router, new lease, laptop moving
networks) reaches it the moment openresolv rewrites the file. Nothing
needs synchronising, and no unit watches for it.
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for
container health-checks). External hosts can't reach it — no
DNS-amplification surface even when the operator opens port 80 for
gateway HTTP.
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
resolution stack — the host's resolver (systemd-resolved, plain
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
configured. The hive resolver is purely for inbound queries from
agent containers.
## Firewall posture
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
- Port 53 opens the resolver on the bridge interface only. Other
interfaces stay closed. The hive resolver isn't an external-facing
service.
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
receives DHCP via a regular UDP socket (it doesn't use a
netfilter-bypassing raw socket), so the hole is mandatory — without
it containers never get a lease and fall back to 169.254.x.x.
- Ports 80 and 443 let isolated agents reach nginx (on the host) for the forge sub-domain, per-agent
UI proxies, and any other HTTP services.
The **host** firewall is the only firewall. The swarm service
containers that share the host netns (forge, bao, the
metrics and log stores) run with `networking.firewall.enable = false`
(`nix/container-modules/swarm-container.nix`): a NixOS firewall inside a
shared-netns container runs against the _host_ ruleset — at container
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the
bridge holes above plus the agents' NAT. Private-netns containers
(every container with `privateNetwork = true`: agents and the service containers on the bridge) keep their own firewall — it's scoped to their
namespace.
### Reaching host services (`exposeHostPorts`)
By default agents can only reach the host on 80/443 (+53 DNS), so a
host-side service on another port — for example a dev OTLP collector you want
agents to reach directly — is unreachable. (hyperhive's own services
need none of this: a module that means to be reachable from agents opens
its own port here, which the hive collector and `nats.enable` both do. Note
that `deploy.swarm-otel.upstream.endpoint` is the _upstream_, which no agent ever dials. See
`docs/4-implementation/scheduler/observability.md`.)
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
agent can connect to `<bridgeIp>:P` (for a dev OTLP collector, point its
endpoint at `http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
The hive's own collector needs none of this: agents dial it by
`otel.collector.host`.
<!-- vale write-good.Passive = NO -->
This is **firewall-only**: the host service must bind an address
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
service bound to `127.0.0.1` only stays unreachable; rebind it to
`0.0.0.0`.
<!-- vale write-good.Passive = YES -->
The port is reachable by **every** agent on the bridge subnet (like
DNS/gateway), so only expose services safe for any agent to reach.
## Egress proxy (squid)
Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
(`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
only. An agent sends its outbound HTTP and HTTPS through it once its hive says
the proxy is ready and it has fetched its proxy credential (see _Agents and the
proxy_ below). Any other agent goes straight out through the NAT, as before,
and so does any program that ignores the proxy variables: nothing blocks direct
egress yet.
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
host. It never decrypts TLS, so it holds no signing key. It caches nothing.
- **Credentials.** A request for a listed name must carry a per-agent proxy
credential (HTTP Basic, the agent's name and its secret). `swarm-controller`
mints one per agent at `swarm/agents/<agent>/proxy-credential`, and never
replaces it. `hive-c0re` hashes the credentials of this hive's agents, and
one more for the self-check, into `services.hyperhive.deploy.hive-controller.squid.usersFile`
every 30 seconds. Squid's `basic_ncsa_auth` helper re-reads the file when it
changes. A stopped agent stays in the file. Squid caches a good login for a
minute. Nothing rotates a credential: to replace one, delete its leaf in the
store and the next pass mints a new one. HTTP Basic crosses the bridge in
clear between an agent and the proxy. See
[the credentials reference](../../2-handbook/swarm/credentials-reference.md).
A request without a valid credential gets a 407, once the destination
checks below have passed it.
- **Agents and the proxy.** An agent uses the proxy only when its hive says
this host's squid is ready for it. `hive-c0re` puts two facts in every
container's `/etc/hyperhive-hive-facts`, only on a hive that holds a store
identity: `HIVE_SQUID_PROXY` (`services.hyperhive.deploy.hive-controller.squid.listen`, the bridge IP
and port) and `HIVE_EGRESS_NO_PROXY` (the direct hosts below). A hive that
predates them writes neither, and its agents go direct.
`hive-agent-proxy-credential.service` (`nix/agent-modules/egress-proxy.nix`)
runs in each agent's container, as the agent user, at start and every 10
minutes. With both facts present it logs in to the secret store with the
agent's own certificate, reads `swarm/agents/<agent>/proxy-credential` and
writes `/run/hive-agent-proxy-credential/proxy.env` (`0600`) with
`HTTPS_PROXY`, `HTTP_PROXY`, their lower-case forms, `NODE_USE_ENV_PROXY=1`
and `NO_PROXY`/`no_proxy`. The proxy URL is
`http://<agent>:<credential>@<bridgeIp>:3128`; Claude Code, Node, curl, git
and reqwest send the userinfo as `Proxy-Authorization`. The harness, its bash
tool, the subagents, the MCP servers and the pollers load the file with
`EnvironmentFile=-` when they start. Until the facts and the credential both
exist nothing writes the file, so those units have no proxy variable and go
direct; a failed read keeps the file already in place. Nothing restarts on
deploy: an agent takes the proxy at its next start, which is its next rebuild
or restart. The credential is in the environment of every process those units
start, so `env` in the agent's shell prints it. Login shells
(`nixos-container root-login`, SSH) don't load the file.
- **Direct destinations.** `HIVE_EGRESS_NO_PROXY` is the hosts this agent must
reach without the proxy, and `NO_PROXY` is `localhost,127.0.0.1,::1` followed
by that list. `hive-c0re` computes it from the host's options: the bridge IP,
the hive's domain (`services.hyperhive.domain`), the secret store and queue
names (`services.hyperhive.swarm.bao.domain`, `swarm.nats.domain`),
`services.hyperhive.swarm.serviceDomains` (the swarm UI and controller apex,
the forge, matrix, attic, logs, metrics and otel names) and every name the
gateway answers with the bridge IP (`services.hyperhive.gateway.localNames`).
Squid refuses every destination that resolves to the bridge, and the hive's
resolver answers the bridge address for those names, so they never go through
the proxy. A bare name matches itself and its subdomains. Node's `NO_PROXY`
has no CIDR support, so the list holds the bridge as an address.
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
second file beside the users file, `services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
the users whose agent holds the `unrestricted_internet` flag in the
permissions the hive last read from the swarm, one per line, rewritten when
it changes. The default is permissive: an agent with no record, or a record
that never set the flag, holds it, the same default as the
`hyperhive.agent.unrestricted_internet` metric. While the hive's read of the
permissions is stale or failing, a new agent that already holds a
credential counts as unrestricted. Squid reads the list when it
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
when the file changes. A missing or empty file lists no one, and squid
starts either way. A flag change reaches squid after the hive's next read of
the permissions and its next 30-second pass.
An older `hive-c0re` writes no list, so every credential on that hive is
restricted. Agents must not start sending credentials until every hive runs
the `hive-c0re` that writes it, or a legacy agent loses general internet.
- **Log mode.** The operator can put an agent in `log` mode in the swarm UI.
Its record keeps `unrestricted_internet` at `false` and adds
`internet_log: true`, so a `hive-c0re` that doesn't know the field still
enforces the list. For such an agent `hive-c0re` adds to `agent-allow.conf`,
after the allow rules for every agent's own names,
`http_access allow <user> !allowed_dst log_mark`, where `log_mark` is an
`annotate_transaction` ACL that sets `would_deny=1`. The rule comes last in
the include, so every destination deny and `deny !authed` before it still
applies, and a name the agent may reach never carries the mark. A request the
rule lets through gets a second log line,
`would_deny user=<agent> method=<m> host=<h> port=<p> result=<r>`, from the
`would_denied` `access_log` in `hive-squid.nix`; the usual request line keeps
its format. The **would-deny** tab of the `egress` dashboard in Grafana lists the hosts per agent. Only
the operator changes the mode: nothing moves an agent to `restricted` after a
time or a count. The self-check's users aren't agents,
so `deny-canary.invalid` stays a 403.
- **Modes.** An agent is in one of three modes, and the deny list (next bullet)
applies in all of them:
| mode | the deny list | a name not on the allow list |
| -------------- | ------------- | ---------------------------- |
| `restricted` | blocked | blocked |
| `log` | blocked | allowed and logged |
| `unrestricted` | blocked | allowed |
- **Deny list.** The swarm's deny list names destinations the proxy refuses
whatever else allows them. The swarm controller keeps it as its own value
(the `deny` key of the `egress-allow` bucket), each entry for every agent or
for one, and serves it at `GET`/`POST`/`DELETE /api/egress/denied-domains`
with the same body as the allow-list (`{"domain": ".example.com", "agent":
"atlas"}`). Write a name as in the allow-list: at least two labels, one
leading dot for the name and its subdomains. The controller refuses IP
addresses, other wildcards, ports and paths. Unlike the allow-list it doesn't
refuse the base list's names: denying one is the point. Every 30 seconds
`hive-c0re` reads the list and renders `services.hyperhive.deploy.hive-controller.squid.denyFile`
(`/var/lib/hive-squid/deny.conf`, `hive-core:squid`, `0640`) for the agents it
runs, and `squid-unrestricted-reload.path` reloads squid when it changes.
- **Precedence.** The configuration includes the file after the destination
denies and before `deny !authed` and every allow, so a denied name is
refused even when the base list, the hive's `allowedDomains`, the swarm's
allow-list, the unrestricted flag or `log` mode would allow it. A swarm-wide
entry names no user, so every client gets a 403 for it, one without a
credential too. An agent's entry also names the agent's squid user, so it
binds that agent: a client that sent no credential gets the usual 407 for
it. An agent's entry only adds to the swarm-wide ones; no entry of one scope
lifts another.
- **Log line.** A request a deny rule refused gets a second log line,
`egress_denied user=<agent or -> method=<m> host=<h> port=<p> result=<r>`,
in every mode. It differs from the `would_deny` line of `log` mode and
from the `TCP_DENIED/403` line every refusal has, so you can tell a
deny-list hit from a name that isn't on the allow-list.
- **Who writes it.** Only an operator's request to the controller writes the
list. It has no automatic entries: nothing derives a deny from an agent's
API-key environment or a linked account, and the stored type has no such
field. An agent has no queue grant that reaches the key, and the controller's
API sits behind the operator's SSO login: nginx checks it before it hands
a request to the socket, which only nginx and root can open.
- **Empty list.** With no entry `deny.conf` is empty and squid configures as
it did without the list: the configuration gains one `include` line and the
`egress_denied` log lines, and no rule. A hive on a `hive-c0re` that
predates the list never reads the key and keeps its old configuration, so
nothing on its wire changes.
- **Applied.** Each hive reports the revision of the list it last rendered as
`egress_deny_applied` in its status snapshot (see [the hive
snapshot](../../2-handbook/swarm/README.md)). `GET /api/egress/denied-domains` answers the
list's `revision` and `behind`, the hives that haven't reported it: an older
`hive-c0re` never does, so it shows there rather than silently allowing the
names. A hive that reads the list reports within about a minute. Reporting
means the file holds the rules; squid reloads moments later.
- **Base list.** `services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains` is open to every
authenticated user, whatever its flag: what the agent harness needs off the
swarm. The default is the hosts Claude Code requires
(<https://code.claude.com/docs/en/network-config>): `api.anthropic.com` for
the model API and the usage probe (`hive-agent/src/claude_usage_watch.rs`),
`claude.ai` for the account login, and `platform.claude.com` for the OAuth
token exchange and refresh. An agent whose `ANTHROPIC_BASE_URL` points
elsewhere needs no entry here: the swarm controller adds that host to the
agent's own swarm entries (see _Swarm allow-list_ below). The list omits the plugin marketplace clone from `github.com`, since a failed add
isn't fatal. The rules below still refuse an entry that resolves to a
private address.
- **Allow-list.** `services.hyperhive.deploy.hive-controller.squid.allowedDomains` defaults to empty,
plus the self-check canary (`services.hyperhive.deploy.hive-controller.squid.selfcheck.canary`), so
the canary widens the restricted list by that one name. The swarm's own
service names aren't on it: they resolve to the bridge or a private address,
which the rules below refuse, and agents reach them directly. Each entry
matches that exact name, or with a leading dot the name and its subdomains. A user on the unrestricted list reaches any
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
`allowedDomains` is the hive's own list, always allowed whether or not the
swarm has a list of its own (next bullet).
- **Swarm allow-list.** The operator adds domains in the swarm UI
([allowed domains](../../2-handbook/swarm/ui.md#allowed-domains)); the swarm controller
keeps them in one value of the `egress-allow` bucket, each for every agent
or for one agent. Every 30 seconds `hive-c0re` reads it and writes two files
beside the users file, `services.hyperhive.deploy.hive-controller.squid.allowedFile`
(`/var/lib/hive-squid/allowed`) and `services.hyperhive.deploy.hive-controller.squid.agentAllowFile`
(`/var/lib/hive-squid/agent-allow.conf`), `hive-core:squid`, `0640`, and
logs `squid allow list: 2 swarm, 3 agent entries` when they change. The
swarm-wide names join the allow ACL, so a restricted user reaches them. A
per-agent entry is a rule for that agent's squid user, `http_access allow
<user> <names>`, in the generated `agent-allow.conf` that the configuration
includes at the point where the base list gets its allow: after every
destination deny and
`deny !authed`, so the rules under _Denied before any allow_ still bind it,
and before the unrestricted rule. Only this hive's agents get a rule.
`squid-unrestricted-reload.path` reloads squid when either file changes, so
there is no deploy or restart.
A name is a DNS name of at least two labels. One leading dot (`.example.com`)
also allows every subdomain, and is the only wildcard; the controller refuses
IP addresses, other wildcards, ports, paths and the base list's names when
you add the entry, and `hive-c0re` checks each name again and skips one that
fails, or that the base list or the canary covers, or that another entry of
the same list covers. A name that resolves to a private address stays a 403.
The controller also adds entries itself, and only from the operator's own
actions in the swarm UI. Saving an agent's API-key environment makes it take
the host from `ANTHROPIC_BASE_URL` and from every variable named `*_BASE_URL` (the
address an ACP agent that loads the same file reads), and keeps one
`derived` entry for that agent per host. Each host passes the same check as
an operator's entry, so an IP address, `localhost` and a base-list name get
none. A save or a delete of the environment replaces the agent's derived
entries, and destroying the agent drops all its entries. The agent can
create its own `backend-env` leaf while the leaf is empty, so text found in
the store never creates an entry: when the controller starts, it only
drops derived entries whose exact host the stored environment no longer
names, and logs a host it didn't derive. An environment saved before this
existed gets its entries when the operator saves it once more. An agent on
the hand-placed `backendEnvironmentFile` alone, and the `opencode` preset's
`acp.opencode.provider.baseUrl`, get no derived entry, so add one in the
swarm UI. A `hive-c0re` that predates the field renders a derived entry
like any other.
Linking a GitHub token, an external matrix account or an external forge
account adds a derived entry for that agent per host it reaches, and
deleting the account removes it: `github.com`, `api.github.com`,
`raw.githubusercontent.com`, `codeload.github.com` and
`objects.githubusercontent.com` for GitHub, and the host of the matrix account's `homeserver` or the forge
account's `url`. Each entry's `source` (`link:github`,
`link:matrix:<account>`, `link:forge:<label>`) keeps it apart from the
environment's, which a save of the environment leaves alone; an entry
without a `source` counts as the environment's. Only the controller writes
these leaves, so when it starts it also adds the entries of accounts linked
before this existed and drops those whose account no longer exists, and logs
the counts (`link egress: entries match the linked accounts`). A store it
can't read changes nothing. An `agent.nix` `matrixAccounts` homeserver gets
no entry, since the controller never reads it; add one in the swarm UI.
The swarm's list adds to the hive's `allowedDomains`: `allowedFile` holds
those names plus the swarm-wide entries, so removing an entry never removes a
name the hive lists itself. Until the controller has a list (it predates the
feature, or nobody added an entry) `allowedFile` holds `allowedDomains` and
`agent-allow.conf` is empty. A `hive-c0re` older than this rule replaces
`allowedDomains` with the swarm's list once the controller has one, so every
hive must run the new `hive-c0re` before the controller's first entry exists. A queue that stops answering for the bucket
leaves a file the swarm wrote untouched, and a failed read leaves both
files untouched. The first line of a file written from the swarm's list is a
`# source: swarm` comment. The queue grant a hive needs to read the list
comes with the controller's `swarm-nats-auth`; a hive that can't read it
times out, keeps its files and warns after about 90 seconds.
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
after the destination checks, which look the name up on the host.
- **Denied before any allow.** Destinations that resolve to loopback,
link-local, the bridge subnet, a private range (`10.0.0.0/8`,
`172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`), shared address space
(`100.64.0.0/10`), `0.0.0.0/8`, `::`, multicast or reserved space, and IP-literal
hosts, for every user including the unrestricted ones and for listed names.
Squid resolves names on the host and connects from it, so without these a
name pointing at `127.0.0.1`, at the host's own LAN address or at a private
neighbour would walk around the bridge→loopback DROP rule and the swarm's
network. Squid checks the address it connects to, so a name that changes its
answer between a check and the connect gains nothing. A listed name must
therefore resolve to a public address on the host's resolver: a split-horizon
name that answers a LAN address gets a 403, and so does the self-check
canary, which is why it isn't a swarm service name. Only `CONNECT` to port
443 and plain HTTP to ports 80 and 443 pass.
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
`squid_client_http_requests_total` are the two to watch.
- **Logs.** One syslog line per request reaches the journal as
`squid.service`, and `services.hyperhive.otel.journalUnits` forwards that
unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the
client's source address. A request that a `log`-mode agent's rule let
through also logs a `would_deny` line (see _Log mode_). A request a deny-list rule refused logs an `egress_denied` line (see _Deny list_).
- **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service`
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
to `services.hyperhive.deploy.hive-controller.squid.selfcheck.canary` (default: `cache.nixos.org`,
a public name that's also on the allow-list)
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
the same secret and is always on the unrestricted list, and a `CONNECT` to
`services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary` (default:
`example.com`, a name outside the allow-list) must answer 200. It writes one
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
per run and fails the unit when any is wrong. The password is the hive's
`swarm/hives/<hive>/squid-selfcheck` secret, which
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't
written the users file, the line reads `credential=absent` or
`allow_status=407` and `ok=false`, and nothing else breaks. The
host's resolver looks up the canary, so a host whose resolver maps
that name to a loopback, bridge or private address fails the check.
## Container isolation
Each agent container runs in a private network namespace with a dedicated
veth pair attached to the bridge. The following table summarises what
the nix side sets up unconditionally:
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
`10.42.0.1/24`), **not** the canonical network address. The Rust side
must normalise (bitwise-AND with mask) before subnet membership checks or
address arithmetic.
### What the Rust side does
`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
and `HOST_BRIDGE=<bridgeName>` when creating or updating containers:
`lifecycle::set_nspawn_flags` hands them to hive-priv, which rewrites
`/etc/nixos-containers/<container>.conf`. hive-c0re validates both variables **once at
daemon startup**, not per container: they're process-global, so a
missing or malformed value is a misconfigured daemon rather than one bad
container, and failing at boot gives a single diagnostic instead of one
per agent. No non-isolated mode exists to fall back to. hive-c0re leaves `LOCAL_ADDRESS` empty so the
container's dhcpcd acquires an address from the bridge dnsmasq pool
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies to
every agent container, manager included.
`HOST_ADDRESS` is the bridge gateway IP (the address part of
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
so a non-`.1` operator override still resolves to wherever the bridge
actually lives). It's **load-bearing**: nixos-container's container-side
network setup only installs a default route (`ip route add default via
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode
nixos-container skips the host-side address/route setup, so writing it only affects the
container's default route — without it the container comes up with an IP
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
### How the isolated container gets its resolver
nixos-container copies the **host's** `/etc/resolv.conf` into the container
at every start. The host resolver (for example `127.0.0.53` from systemd-resolved,
or a LAN router) is unreachable from a private netns and isn't
authoritative for the hive's own zones, so it's replaced with the
bridge dnsmasq at boot. Because the copy happens on every start, it
would clobber a declarative `environment.etc."resolv.conf"` — so the
wiring is runtime:
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
gateway IP) into each container's `/etc`.
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
It's ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
the forge-token fetch so the resolver is correct before the first DNS lookup.
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
domain sockets bind-mounted into containers, not network listeners — see
the _Control plane (no network)_ bullet under [Network
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
that never touches the network stack.
The nix side also enables IP forwarding + NAT (agents reach the internet
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP.
## Cross-references
- `docs/4-implementation/networking/gateway.md` — vhost map + the gateway's other duties

238
docs/4-implementation/networking/snapshot-store.md Normal file
View file

@ -0,0 +1,238 @@
# Snapshot store
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
over the WireGuard mesh; a destination hive later pulls one back to
complete a migration.
Two things it's not, both worth stating because both are common
assumptions:
- **It's not the swarm controller**, and doesn't depend on one. It's
a NixOS host role: a btrfs subvolume tree, a socket-activated
receiver, and the `wg-hive` interface the swarm module already brings
up. That's why you can deploy it before any controller exists.
- **It's not a backup product.** It happens to hold the data a backup
would hold, and you should operate it accordingly (see
[Operating it](#operating-it)) --- but nothing in it does scheduling,
verification, or restore orchestration.
## Enabling it
```nix
services.hyperhive.snapshotStore = {
enable = true;
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
port = 51821;
};
# The mesh is a hard requirement, and is asserted:
services.hyperhive.deploy.wireguard = {
enable = true;
address = "10.100.0.9/24";
privateKeyFile = "/etc/wireguard/hive.key";
};
```
The store host is a swarm member like any other: it gets an entry in
`services.hyperhive.swarm.hives`, the same directory every host holds. See
[swarm/](../../2-handbook/swarm/README.md) for the mesh itself.
Note that `deploy.wireguard.enable` gates the mesh, **not**
`deploy.hive-controller.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## Pointing a hive at it
The block above configures the host that _receives_. You must tell every
hive that _pushes_ separately where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
address = "10.100.0.9"; # the store's mesh address, no prefix
port = 51821; # optional; must match the receiver's port
};
```
Two deliberate asymmetries in that pair, both liable to be misread as
inconsistency:
- **`address` has no default.** It's a deployment fact a pushing hive
can't derive, and a wrong guess means streaming an agent's state at
whatever happens to answer. Unset, a push fails naming this option.
- **`port` does default** (`51821`), because it's a convention both
ends read from the same option docs --- a default there is
coordination, not a guess.
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That's the distinction the
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
from this host, and a bare `services.hyperhive.<service>` describes _a
role this host performs_. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
[--parent <label>]` streams a snapshot straight into the store. No
destination argument exists, because a swarm has exactly one store (see
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
and no credential argument, because the mesh is the authentication.
## The mesh is the authentication
No certificates live here, and no key material of its own. That's
deliberate rather than an omission.
WireGuard's cryptokey routing already binds a peer's source address to
its public key: the swarm module configures each peer with
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
that address provably came from the holder of that private key. By the
time a packet reaches the receiver, the kernel has already
authenticated it.
Layering TLS client certs on top would authenticate _the same fact_ a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
## One subvolume per agent, not per hive
The receiver keys the destination by **agent**.
This isn't cosmetic. After a migration, an agent's next incremental
send arrives from a _different_ hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and can't choose
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
and the subvolume name inside it's chosen by the sender. The
protocol is one `agent <name>` header line, then the raw stream.
The receiver waits 30 s for the header, and systemd stops a transfer
that runs past 6 h in total. The sender gives connect and header write 15 s
each; it puts no limit on the stream itself.
The rule that matters:
> **The receiver owns the destination root. The sender-supplied name is
> validated, never used as a path.**
Validation is an allowlist --- `[A-Za-z0-9_-]+` and nothing else. No
slash and no dot means neither directory traversal nor an absolute path
can survive it. It's deliberately an allowlist and not a list of
forbidden characters: a blocklist only ever excludes the attacks
somebody already thought of.
## Reachability
The receiver is socket-activated, and the socket binds **this host's
mesh address**, never a wildcard. The module asserts three things: the
mesh is on, the address isn't empty, and it isn't `0.0.0.0` or `::` ---
none is just documentation: bound to a wildcard this socket is an
unauthenticated remote write into agent state.
The socket sets `FreeBind`, so the bind succeeds before `wg-hive` has
its address. Nothing orders the wireguard unit before `sockets.target`,
and without it the bind fails with `EADDRNOTAVAIL` at boot and the
receiver never listens.
Binding isn't sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, _before_ a packet reaches a bound socket, so
the module explicitly opens the port --- scoped to the mesh interface:
```nix
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
```
A host-wide `allowedTCPPorts` would open the port on every interface
including a public NIC, leaving only the socket's bind address between
the internet and a root `btrfs receive`.
## Operating it
### Confinement is the deployment's job
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
narrow `ReadWritePaths` --- but those are **defence in depth, not a
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
undo the namespace they set up.
The boundary is the machine. The intended deployments are:
- **a swarm**: the store is its own small VM. The machine is the
boundary, which is stronger than anything the unit could assert about
itself.
- **all-in-one / local**: the store runs as a container on the c0re
host.
The second is worth keeping deliberately, and not only for
convenience: it means every local deployment exercises the confined
path. The usual failure mode for an isolated variant is that
nobody runs it day to day, so it rots and turns up broken in
production.
⚠️ **The assumption to keep true over time:** the store host runs
nothing else. That's true on day one and quietly false the day someone
notices the box has spare disk. Nothing in the config objects when it
stops being true.
### It holds every agent's state from every hive
Which makes it the highest-value target in the swarm by a wide margin,
and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The pitfall is the label: this box holds backup-grade data while nobody
calls it a backup, so it can end up with backup-grade _exposure_ and
non-backup-grade _controls_. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
The snapshot covers an agent's **state subvolume**, which is the parent
of `state/`, `claude/` and `harness/` (see
[`docs/2-handbook/agent-lifecycle/persistence.md`'s btrfs subvolume
section](../../2-handbook/agent-lifecycle/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
for how and when hive-c0re creates that subvolume). Consequences:
<!-- vale write-good.Passive = NO -->
- The Claude session (`claude/`) travels, so a restored agent keeps its
live `--continue` session rather than needing to log in again.
- `harness/` travels too, including `harness/bash-tasks/`. Task output
is part of an agent's working continuity, so this is wanted --- but it
means anything that has ever leaked into a task's captured output is
in the retained snapshots as well.
<!-- vale write-good.Passive = YES -->
It does **not** cover the agent's config (the forge's `agent-configs/<name>`) or
its topology entry, both of which live outside the subvolume. A restore
therefore yields an agent's memory without its definition.
### Retention
<!-- vale write-good.Passive = NO -->
Retention lives on the _sending_ side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.
<!-- vale write-good.Passive = YES -->
Per-agent or per-hive `btrfs qgroup` quotas aren't configured yet.
Without them one runaway hive can fill the store and take out every
other hive's snapshots.
## Not built yet
**The pull side.** Push is safe with minimal authorisation because a
hive can only ever write to a chain it owns. Pull is the direction that
needs a policy: unrestricted, any compromised hive could read every
agent's state from every other hive. It needs a notion of which hive
currently owns which agent, and that ownership record lands with the
swarm controller work.
With a single hive the question is trivial --- the only peer owns
everything it sends --- which is why the receive half ships first.