docs(network): rewrite for always-on isolation; remove phased-rollout rationale

The two-phase network rollout (bridge-first, then isolateContainers) is
complete. Both options are now deprecated no-ops — isolation is always on.
Update the doc to reflect current state:

- Drop the phased-rollout intro ('off by default during rollout') and the
  v1/v2 comparison table; keep a brief historical note
- Remove 'Why ship before netns isolation' section (rollout is done)
- Update configuration example: network.enable is no longer needed
- Merge firewall description into a single table (80/443 always open)
- Remove 'Prerequisites before flipping on' and 'Migration behaviour'
  subsections (isolation was a one-time flip; no longer a toggle)
- Simplify resolver wiring: hive-priv always drops the marker; remove
  'only when isolated' conditional framing
This commit is contained in:
atlas 2026-07-04 20:30:23 +02:00
commit 902d4dd461

View file

@ -1,31 +1,14 @@
# hive-network # hive-network
Host-side bridge + per-agent DNS resolver — the foundation that Host-side bridge + per-agent private-netns isolation — always on
makes container netns isolation safe to land. Configured via whenever hyperhive is enabled. Configured via
`services.hyperhive.network.*`; off by default during rollout. `services.hyperhive.network.*`.
## Why ship before netns isolation > **Historical note:** the bridge and private-netns isolation landed in
> two separate phases. `services.hyperhive.network.enable` and
If netns isolation lands first, agent containers lose > `services.hyperhive.network.isolateContainers` are retained as
`/etc/resolv.conf` propagation from the host and DNS breaks until a > deprecated no-op options so existing configs eval without change; both
separate resolver is up. Inverting the sequence — bridge + dnsmasq > are ignored — isolation is the only mode.
first, netns flip second — makes the flag day boring: the resolver
endpoint is already live, agents just discover it via veth instead
of shared netns.
## v1 vs v2
| feature | v1 (this PR) | v2 (after netns isolation) |
| ------------------------ | ------------------------------------------------------- | ----------------------------------------------- |
| bridge interface | created on host, no slave NICs | per-agent veth pairs attach |
| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) |
| agent container netns | shared host | private |
| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver <bridge-ip>` |
| `address` rules target | `<bridge-ip>` (works in both modes) | unchanged from v1 |
The `address` rules ship pointing at the bridge IP from v1 so the
DNS contract is fixed before any container actually depends on it
— minimises the things that flip on netns day.
## Container shape (where dnsmasq lives) ## Container shape (where dnsmasq lives)
@ -33,9 +16,7 @@ Co-located in the existing `hive-gateway` container — single
front-door for both DNS and HTTP, saves a sibling container, single front-door for both DNS and HTTP, saves a sibling container, single
systemd-unit / state surface to monitor. The gateway shares host systemd-unit / state surface to monitor. The gateway shares host
netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces` netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces`
listener on `bridgeIp` works without any veth gymnastics today; when listener on `bridgeIp` is on the host's bridge interface.
agent containers flip to private netns the binding doesn't change
(it's still on the host's bridge interface).
## Configuration ## Configuration
@ -44,19 +25,14 @@ agent containers flip to private netns the binding doesn't change
services.hyperhive = { services.hyperhive = {
enable = true; enable = true;
domain = "darkest.space"; domain = "darkest.space";
network.enable = true; # opt in to bridge + DNS # network.bridgeIp = "10.42.0.1"; # default
network.bridgeIp = "10.42.0.1"; # default # network.upstreamDns = [ "1.1.1.1" "9.9.9.9" ]; # default
network.upstreamDns = [ # default Cloudflare + Quad9
"1.1.1.1"
"9.9.9.9"
];
}; };
} }
``` ```
Asserts `services.hyperhive.domain != null` (resolver needs a domain Requires `services.hyperhive.domain` to be set — the dnsmasq resolver
to be authoritative for) + `services.hyperhive.gateway.enable = is authoritative for `<hive-domain>` and its sub-domains.
true` (resolver lives in the gateway container).
## Bridge addressing ## Bridge addressing
@ -91,15 +67,14 @@ agent containers.
## Firewall posture ## Firewall posture
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 ]` `networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 ]`
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
- `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge - Port 53 opens the resolver on the bridge interface only. Other
interface only. Other interfaces stay closed. The hive resolver interfaces stay closed. The hive resolver isn't an external-facing
isn't an external-facing service. service.
- Ports 80 and 443 let isolated agents reach nginx (gateway
When `isolateContainers = true`, `allowedTCPPorts` is extended with container, shared host netns) for the forge sub-domain, per-agent
`[ 80 443 ]` so isolated agents can reach nginx (gateway container, UI proxies, and any other HTTP services.
shared host netns) for the forge sub-domain, per-agent UI proxies,
and any other HTTP services.
### Reaching host services (`exposeHostPorts`) ### Reaching host services (`exposeHostPorts`)
@ -126,11 +101,9 @@ DNS/gateway), so only expose services safe for any agent to reach.
## Container isolation ## Container isolation
`services.hyperhive.network.isolateContainers` (default `false`) flips Each agent container runs in a private network namespace with a dedicated
agent containers from shared host netns to private netns. Set only after veth pair attached to the bridge. The following table summarises what
`enable = true` is stable in production — an assertion blocks the reverse. the nix side sets up unconditionally:
### What the nix side does when `isolateContainers = true`
| effect | mechanism | | effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
@ -138,7 +111,6 @@ agent containers from shared host netns to private netns. Set only after
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC | | Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks | | Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents reach nginx on the host (shared netns) | | Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents reach nginx on the host (shared netns) |
| Forge URL | `HIVE_FORGE_URL` flips from `http://127.0.0.1:3000` to `http://forge.<domain>` — agents resolve via dnsmasq, nginx proxies to forgejo |
| c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` | | c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g. `HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
@ -148,7 +120,7 @@ address arithmetic.
### What the Rust side does ### What the Rust side does
`hive-c0re` reads `HIVE_NETWORK_ISOLATION` and, when set, passes `hive-c0re` reads `HIVE_NETWORK_ISOLATION` and passes
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=<deterministic-ip>`, `PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=<deterministic-ip>`,
`HOST_ADDRESS=<bridge-ip>`, and `HOST_BRIDGE=<bridgeName>` via `HOST_ADDRESS=<bridge-ip>`, and `HOST_BRIDGE=<bridgeName>` via
`lifecycle::set_nspawn_flags` when creating or updating containers. Each `lifecycle::set_nspawn_flags` when creating or updating containers. Each
@ -171,21 +143,17 @@ but no path off the bridge subnet (no internet, no `api.anthropic.com`).
nixos-container copies the **host's** `/etc/resolv.conf` into the container nixos-container copies the **host's** `/etc/resolv.conf` into the container
at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved, at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved,
or a LAN router) is unreachable from a private netns and isn't or a LAN router) is unreachable from a private netns and isn't
authoritative for the hive's own zones, so it must be replaced with the authoritative for the hive's own zones, so it is replaced with the
bridge dnsmasq (the gateway IP). Because the copy happens on every start, a bridge dnsmasq at boot. Because the copy happens on every start, a
declarative `environment.etc."resolv.conf"` would be clobbered — so the declarative `environment.etc."resolv.conf"` would be clobbered — so the
wiring is runtime: wiring is runtime:
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the - `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
gateway IP) into the container's `/etc` **only when isolated**, removing gateway IP) into each container's `/etc`.
it otherwise — so one shared container toplevel behaves correctly in both
netns modes.
- the `hyperhive-isolated-dns` oneshot (harness-base.nix), gated on that - the `hyperhive-isolated-dns` oneshot (harness-base.nix), gated on that
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot. marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
`tea-login` so the resolver is correct before the first DNS lookup; it's `tea-login` so the resolver is correct before the first DNS lookup.
an instant no-op in shared-netns mode (the marker is absent, so
`ConditionPathExists` skips it).
**Why isolation is safe**: all hive-c0re communication goes **Why isolation is safe**: all hive-c0re communication goes
through unix domain sockets (`/run/hive/mcp.sock` for agent requests, through unix domain sockets (`/run/hive/mcp.sock` for agent requests,
@ -200,19 +168,6 @@ against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over `127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP. loopback — the hive-c0re admin socket is a UDS, not TCP.
### Prerequisites before flipping on
- All agents must have `hyperhive.web.useUnixSocket = true`. Agents that
still bind TCP on `0.0.0.0:<port>` will be reachable at their bridge IP
from other agents on the same subnet — defeating the isolation goal. The
gateway routes via unix sockets so gateway reach is unaffected.
### Migration behaviour
Containers are destroyed and re-created when the flag flips. Agent state
under `/agents/<name>/state/` is bind-mounted and survives; the container
rootfs is recreated cleanly from the nix store.
## Cross-references ## Cross-references
- `docs/gateway.md` — vhost map + the gateway container's other duties - `docs/gateway.md` — vhost map + the gateway container's other duties