From 902d4dd461a8435aa5af9e67cae1206762a53067 Mon Sep 17 00:00:00 2001 From: atlas Date: Sat, 4 Jul 2026 20:30:23 +0200 Subject: [PATCH] docs(network): rewrite for always-on isolation; remove phased-rollout rationale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two-phase network rollout (bridge-first, then isolateContainers) is complete. Both options are now deprecated no-ops — isolation is always on. Update the doc to reflect current state: - Drop the phased-rollout intro ('off by default during rollout') and the v1/v2 comparison table; keep a brief historical note - Remove 'Why ship before netns isolation' section (rollout is done) - Update configuration example: network.enable is no longer needed - Merge firewall description into a single table (80/443 always open) - Remove 'Prerequisites before flipping on' and 'Migration behaviour' subsections (isolation was a one-time flip; no longer a toggle) - Simplify resolver wiring: hive-priv always drops the marker; remove 'only when isolated' conditional framing --- docs/network.md | 101 ++++++++++++++---------------------------------- 1 file changed, 28 insertions(+), 73 deletions(-) diff --git a/docs/network.md b/docs/network.md index 3fad6e3e..62286be5 100644 --- a/docs/network.md +++ b/docs/network.md @@ -1,31 +1,14 @@ # hive-network -Host-side bridge + per-agent DNS resolver — the foundation that -makes container netns isolation safe to land. Configured via -`services.hyperhive.network.*`; off by default during rollout. +Host-side bridge + per-agent private-netns isolation — always on +whenever hyperhive is enabled. Configured via +`services.hyperhive.network.*`. -## Why ship before netns isolation - -If netns isolation lands first, agent containers lose -`/etc/resolv.conf` propagation from the host and DNS breaks until a -separate resolver is up. Inverting the sequence — bridge + dnsmasq -first, netns flip second — makes the flag day boring: the resolver -endpoint is already live, agents just discover it via veth instead -of shared netns. - -## v1 vs v2 - -| feature | v1 (this PR) | v2 (after netns isolation) | -| ------------------------ | ------------------------------------------------------- | ----------------------------------------------- | -| bridge interface | created on host, no slave NICs | per-agent veth pairs attach | -| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) | -| agent container netns | shared host | private | -| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver ` | -| `address` rules target | `` (works in both modes) | unchanged from v1 | - -The `address` rules ship pointing at the bridge IP from v1 so the -DNS contract is fixed before any container actually depends on it -— minimises the things that flip on netns day. +> **Historical note:** the bridge and private-netns isolation landed in +> two separate phases. `services.hyperhive.network.enable` and +> `services.hyperhive.network.isolateContainers` are retained as +> deprecated no-op options so existing configs eval without change; both +> are ignored — isolation is the only mode. ## Container shape (where dnsmasq lives) @@ -33,9 +16,7 @@ Co-located in the existing `hive-gateway` container — single front-door for both DNS and HTTP, saves a sibling container, single systemd-unit / state surface to monitor. The gateway shares host netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces` -listener on `bridgeIp` works without any veth gymnastics today; when -agent containers flip to private netns the binding doesn't change -(it's still on the host's bridge interface). +listener on `bridgeIp` is on the host's bridge interface. ## Configuration @@ -44,19 +25,14 @@ agent containers flip to private netns the binding doesn't change services.hyperhive = { enable = true; domain = "darkest.space"; - network.enable = true; # opt in to bridge + DNS - network.bridgeIp = "10.42.0.1"; # default - network.upstreamDns = [ # default Cloudflare + Quad9 - "1.1.1.1" - "9.9.9.9" - ]; + # network.bridgeIp = "10.42.0.1"; # default + # network.upstreamDns = [ "1.1.1.1" "9.9.9.9" ]; # default }; } ``` -Asserts `services.hyperhive.domain != null` (resolver needs a domain -to be authoritative for) + `services.hyperhive.gateway.enable = -true` (resolver lives in the gateway container). +Requires `services.hyperhive.domain` to be set — the dnsmasq resolver +is authoritative for `` and its sub-domains. ## Bridge addressing @@ -91,15 +67,14 @@ agent containers. ## Firewall posture `networking.firewall.interfaces..allowedUDPPorts = [ 53 ]` +`networking.firewall.interfaces..allowedTCPPorts = [ 53 80 443 ]` -- `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge - interface only. Other interfaces stay closed. The hive resolver - isn't an external-facing service. - -When `isolateContainers = true`, `allowedTCPPorts` is extended with -`[ 80 443 ]` so isolated agents can reach nginx (gateway container, -shared host netns) for the forge sub-domain, per-agent UI proxies, -and any other HTTP services. +- Port 53 opens the resolver on the bridge interface only. Other + interfaces stay closed. The hive resolver isn't an external-facing + service. +- Ports 80 and 443 let isolated agents reach nginx (gateway + container, shared host netns) for the forge sub-domain, per-agent + UI proxies, and any other HTTP services. ### Reaching host services (`exposeHostPorts`) @@ -126,11 +101,9 @@ DNS/gateway), so only expose services safe for any agent to reach. ## Container isolation -`services.hyperhive.network.isolateContainers` (default `false`) flips -agent containers from shared host netns to private netns. Set only after -`enable = true` is stable in production — an assertion blocks the reverse. - -### What the nix side does when `isolateContainers = true` +Each agent container runs in a private network namespace with a dedicated +veth pair attached to the bridge. The following table summarises what +the nix side sets up unconditionally: | effect | mechanism | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | @@ -138,7 +111,6 @@ agent containers from shared host netns to private netns. Set only after | Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC | | Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks | | Gateway access | `networking.firewall.interfaces..allowedTCPPorts = [ 80 443 ]` — lets isolated agents reach nginx on the host (shared netns) | -| Forge URL | `HIVE_FORGE_URL` flips from `http://127.0.0.1:3000` to `http://forge.` — agents resolve via dnsmasq, nginx proxies to forgejo | | c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` | `HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g. @@ -148,7 +120,7 @@ address arithmetic. ### What the Rust side does -`hive-c0re` reads `HIVE_NETWORK_ISOLATION` and, when set, passes +`hive-c0re` reads `HIVE_NETWORK_ISOLATION` and passes `PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=`, `HOST_ADDRESS=`, and `HOST_BRIDGE=` via `lifecycle::set_nspawn_flags` when creating or updating containers. Each @@ -171,21 +143,17 @@ but no path off the bridge subnet (no internet, no `api.anthropic.com`). nixos-container copies the **host's** `/etc/resolv.conf` into the container at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved, or a LAN router) is unreachable from a private netns and isn't -authoritative for the hive's own zones, so it must be replaced with the -bridge dnsmasq (the gateway IP). Because the copy happens on every start, a +authoritative for the hive's own zones, so it is replaced with the +bridge dnsmasq at boot. Because the copy happens on every start, a declarative `environment.etc."resolv.conf"` would be clobbered — so the wiring is runtime: - `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the - gateway IP) into the container's `/etc` **only when isolated**, removing - it otherwise — so one shared container toplevel behaves correctly in both - netns modes. + gateway IP) into each container's `/etc`. - the `hyperhive-isolated-dns` oneshot (harness-base.nix), gated on that marker, rewrites `/etc/resolv.conf` to `nameserver ` at boot. It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and - `tea-login` so the resolver is correct before the first DNS lookup; it's - an instant no-op in shared-netns mode (the marker is absent, so - `ConditionPathExists` skips it). + `tea-login` so the resolver is correct before the first DNS lookup. **Why isolation is safe**: all hive-c0re communication goes through unix domain sockets (`/run/hive/mcp.sock` for agent requests, @@ -200,19 +168,6 @@ against a compromised agent reaching the c0re dashboard HTTP at `127.0.0.1`). Agents have no legitimate reason to reach the dashboard over loopback — the hive-c0re admin socket is a UDS, not TCP. -### Prerequisites before flipping on - -- All agents must have `hyperhive.web.useUnixSocket = true`. Agents that - still bind TCP on `0.0.0.0:` will be reachable at their bridge IP - from other agents on the same subnet — defeating the isolation goal. The - gateway routes via unix sockets so gateway reach is unaffected. - -### Migration behaviour - -Containers are destroyed and re-created when the flag flips. Agent state -under `/agents//state/` is bind-mounted and survives; the container -rootfs is recreated cleanly from the nix store. - ## Cross-references - `docs/gateway.md` — vhost map + the gateway container's other duties