vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(3088): move the gateway's nginx + dnsmasq onto the host

Browse source 
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.

The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.

Four things made it more than a deletion, none of them visible in the nix
diff:

- The self-signed cert service also imports the hive CA leaf, so removing it
  with the container would have left nginx naming a missing cert file, which
  it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
  for the reason its doc gave, and `--machine=` was both transport and
  scope — so the unit name is now hard-coded in the helper as the
  containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.

Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.

Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.

Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
This commit is contained in:
atlas 2026-08-11 18:00:27 +02:00
commit 07852cabc1
34 changed files with 704 additions and 618 deletions
Download patch file Download diff file Expand all files Collapse all files

53
docs/gateway.md
View file

@ -1,6 +1,6 @@
# hive-gateway
Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.
Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re; system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`. (It lived in a `hive-gateway` container until #3088 — one that shared the host netns anyway, so the boundary gave no network isolation while costing a resolv.conf sync, a machine-bus reload and three bind mounts.)
## Vhost map
@ -112,18 +112,22 @@ now set unconditionally for every agent. The mechanism:
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
if the socket is not yet bound, nginx returns 502 caught by the
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at
`/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`.
After each write, c0re triggers the appropriate nginx action inside
the gateway container via `hive-priv` (which runs as root and has
`--machine=hive-gateway` transport rights that hive-c0re lacks).
nginx includes `/var/lib/hyperhive/gateway/agents.conf` — the same
path c0re writes, since both run on the host.
After each write, c0re triggers the appropriate nginx action via
`hive-priv` (which is root; hive-c0re runs as the unprivileged
`hive-core` user and cannot act on a system unit).
`hive-priv` queries `ActiveState` and dispatches:
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
- otherwise → `systemctl start nginx`
This is intentionally host-side: `IN_MOVED_TO` from an atomic rename
does not propagate across the nspawn mount-namespace boundary, so a
path unit inside the container would never fire.
This is an explicit trigger rather than a path unit watching the
file. It used to be *impossible* to do it any other way — `IN_MOVED_TO`
from the atomic rename did not cross the nspawn mount-namespace
boundary, so an in-container path unit never fired. With nginx on the
host a path unit would now work, and it is still not wanted: the write
and the reload belong in one causal chain c0re can retry and report on
(`RELOAD_PENDING`), not two units racing on an inotify event.
c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
@ -160,8 +164,8 @@ selected by which (if any) external TLS source is set:
| mode | config | cert source | `.well-known` scheme |
|---|---|---|---|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | host hive-CA signs a gateway leaf (RSA-4096) | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx inside container via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | bind-mounted from host | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
The `gateway.selfSignedTls` option is **deprecated and ignored** — self-signed
is now derived from the absence of `tls.certDir` / `tls.acme`. Setting it to
@ -182,7 +186,7 @@ services.hyperhive.gateway = {
};
```
nginx inside the gateway container obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). The gateway container shares the host network namespace (`privateNetwork = false`) so outbound ACME requests work without any extra routing. Certs are stored inside the container's persistent state dir (`/var/lib/acme/` inside `hive-gateway`; survives restarts because `ephemeral = false`).
nginx obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way.
**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<swarm-domain>`, `chat.<swarm-domain>`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own.
@ -194,9 +198,11 @@ Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails
On by default, and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. The leaf dir is bind-mounted read-only into the gateway container at `/run/hive-ca`; an in-container import unit copies the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`) with the owner/mode nginx needs.
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated, in-container leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
⚠️ **That import unit is not a leftover of the old container — do not collapse it into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
@ -220,11 +226,11 @@ services.hyperhive.gateway = {
};
```
The directory is bind-mounted read-only into the gateway container at `/run/hive-tls/`. nginx uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.
nginx reads the directory directly and uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.
`tls.certDir` and `tls.acme.enable` set together is an assertion error.
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx inside the gateway container runs as the `nginx` user and cannot read a key with that ownership. Fix with:
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and cannot read a key with that ownership. Fix with:
```nix
security.acme.certs."example.com".group = "nginx";
@ -424,11 +430,12 @@ JS/CSS assets skip `try_files` entirely and get the immutable cache
headers. Nix store paths are content-addressed — the hash changes on
any content change — so `max-age=31536000` is safe.
**Why nix store is reachable from the gateway container**: nspawn
containers bind-mount `/nix/store` read-only by default. The
`HIVE_AGENT_FRONTEND_DIR` path is a nix store path baked in at
hive-c0re build time — the same path is visible to both c0re (writing
`agents.conf`) and the gateway nginx (serving files from it).
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
store path baked in at hive-c0re build time, and c0re (writing
`agents.conf`) and nginx (serving files from it) are on the same machine,
so they see the same store. This used to need explaining — nspawn
bind-mounts `/nix/store` read-only into a container, which is what made
the baked-in path work from inside the gateway.
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
@ -493,9 +500,7 @@ services.hyperhive.gateway.auth = {
The credential store lives at the fixed path
`/var/lib/hyperhive/gateway/gateway.htpasswd` on the host. A tmpfiles
rule pre-creates the file on first boot; no manual path configuration
is required. The file is exposed inside the gateway container at
`/run/hive-state/gateway.htpasswd` via the existing gateway state
bind-mount.
is required. nginx reads it at that path directly.
Manage users with `hivectl gateway`. `hivectl` sends the request over the
host admin socket and the `hive-c0re` daemon performs the write at its