docs(network): rewrite for always-on isolation; remove phased-rollout rationale
The two-phase network rollout (bridge-first, then isolateContainers) is
complete. Both options are now deprecated no-ops — isolation is always on.
Update the doc to reflect current state:
- Drop the phased-rollout intro ('off by default during rollout') and the
v1/v2 comparison table; keep a brief historical note
- Remove 'Why ship before netns isolation' section (rollout is done)
- Update configuration example: network.enable is no longer needed
- Merge firewall description into a single table (80/443 always open)
- Remove 'Prerequisites before flipping on' and 'Migration behaviour'
subsections (isolation was a one-time flip; no longer a toggle)
- Simplify resolver wiring: hive-priv always drops the marker; remove
'only when isolated' conditional framing
This commit is contained in:
atlas
parent
f111dac231
commit
902d4dd461
1 changed files with 28 additions and 73 deletions
101
docs/network.md
101
docs/network.md
|
|
@ -1,31 +1,14 @@
|
|||
# hive-network
|
||||
|
||||
Host-side bridge + per-agent DNS resolver — the foundation that
|
||||
makes container netns isolation safe to land. Configured via
|
||||
`services.hyperhive.network.*`; off by default during rollout.
|
||||
Host-side bridge + per-agent private-netns isolation — always on
|
||||
whenever hyperhive is enabled. Configured via
|
||||
`services.hyperhive.network.*`.
|
||||
|
||||
## Why ship before netns isolation
|
||||
|
||||
If netns isolation lands first, agent containers lose
|
||||
`/etc/resolv.conf` propagation from the host and DNS breaks until a
|
||||
separate resolver is up. Inverting the sequence — bridge + dnsmasq
|
||||
first, netns flip second — makes the flag day boring: the resolver
|
||||
endpoint is already live, agents just discover it via veth instead
|
||||
of shared netns.
|
||||
|
||||
## v1 vs v2
|
||||
|
||||
| feature | v1 (this PR) | v2 (after netns isolation) |
|
||||
| ------------------------ | ------------------------------------------------------- | ----------------------------------------------- |
|
||||
| bridge interface | created on host, no slave NICs | per-agent veth pairs attach |
|
||||
| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) |
|
||||
| agent container netns | shared host | private |
|
||||
| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver <bridge-ip>` |
|
||||
| `address` rules target | `<bridge-ip>` (works in both modes) | unchanged from v1 |
|
||||
|
||||
The `address` rules ship pointing at the bridge IP from v1 so the
|
||||
DNS contract is fixed before any container actually depends on it
|
||||
— minimises the things that flip on netns day.
|
||||
> **Historical note:** the bridge and private-netns isolation landed in
|
||||
> two separate phases. `services.hyperhive.network.enable` and
|
||||
> `services.hyperhive.network.isolateContainers` are retained as
|
||||
> deprecated no-op options so existing configs eval without change; both
|
||||
> are ignored — isolation is the only mode.
|
||||
|
||||
## Container shape (where dnsmasq lives)
|
||||
|
||||
|
|
@ -33,9 +16,7 @@ Co-located in the existing `hive-gateway` container — single
|
|||
front-door for both DNS and HTTP, saves a sibling container, single
|
||||
systemd-unit / state surface to monitor. The gateway shares host
|
||||
netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces`
|
||||
listener on `bridgeIp` works without any veth gymnastics today; when
|
||||
agent containers flip to private netns the binding doesn't change
|
||||
(it's still on the host's bridge interface).
|
||||
listener on `bridgeIp` is on the host's bridge interface.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
|
@ -44,19 +25,14 @@ agent containers flip to private netns the binding doesn't change
|
|||
services.hyperhive = {
|
||||
enable = true;
|
||||
domain = "darkest.space";
|
||||
network.enable = true; # opt in to bridge + DNS
|
||||
network.bridgeIp = "10.42.0.1"; # default
|
||||
network.upstreamDns = [ # default Cloudflare + Quad9
|
||||
"1.1.1.1"
|
||||
"9.9.9.9"
|
||||
];
|
||||
# network.bridgeIp = "10.42.0.1"; # default
|
||||
# network.upstreamDns = [ "1.1.1.1" "9.9.9.9" ]; # default
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Asserts `services.hyperhive.domain != null` (resolver needs a domain
|
||||
to be authoritative for) + `services.hyperhive.gateway.enable =
|
||||
true` (resolver lives in the gateway container).
|
||||
Requires `services.hyperhive.domain` to be set — the dnsmasq resolver
|
||||
is authoritative for `<hive-domain>` and its sub-domains.
|
||||
|
||||
## Bridge addressing
|
||||
|
||||
|
|
@ -91,15 +67,14 @@ agent containers.
|
|||
## Firewall posture
|
||||
|
||||
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 ]`
|
||||
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
|
||||
|
||||
- `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge
|
||||
interface only. Other interfaces stay closed. The hive resolver
|
||||
isn't an external-facing service.
|
||||
|
||||
When `isolateContainers = true`, `allowedTCPPorts` is extended with
|
||||
`[ 80 443 ]` so isolated agents can reach nginx (gateway container,
|
||||
shared host netns) for the forge sub-domain, per-agent UI proxies,
|
||||
and any other HTTP services.
|
||||
- Port 53 opens the resolver on the bridge interface only. Other
|
||||
interfaces stay closed. The hive resolver isn't an external-facing
|
||||
service.
|
||||
- Ports 80 and 443 let isolated agents reach nginx (gateway
|
||||
container, shared host netns) for the forge sub-domain, per-agent
|
||||
UI proxies, and any other HTTP services.
|
||||
|
||||
### Reaching host services (`exposeHostPorts`)
|
||||
|
||||
|
|
@ -126,11 +101,9 @@ DNS/gateway), so only expose services safe for any agent to reach.
|
|||
|
||||
## Container isolation
|
||||
|
||||
`services.hyperhive.network.isolateContainers` (default `false`) flips
|
||||
agent containers from shared host netns to private netns. Set only after
|
||||
`enable = true` is stable in production — an assertion blocks the reverse.
|
||||
|
||||
### What the nix side does when `isolateContainers = true`
|
||||
Each agent container runs in a private network namespace with a dedicated
|
||||
veth pair attached to the bridge. The following table summarises what
|
||||
the nix side sets up unconditionally:
|
||||
|
||||
| effect | mechanism |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
|
@ -138,7 +111,6 @@ agent containers from shared host netns to private netns. Set only after
|
|||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||||
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
|
||||
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents reach nginx on the host (shared netns) |
|
||||
| Forge URL | `HIVE_FORGE_URL` flips from `http://127.0.0.1:3000` to `http://forge.<domain>` — agents resolve via dnsmasq, nginx proxies to forgejo |
|
||||
| c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` |
|
||||
|
||||
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
|
||||
|
|
@ -148,7 +120,7 @@ address arithmetic.
|
|||
|
||||
### What the Rust side does
|
||||
|
||||
`hive-c0re` reads `HIVE_NETWORK_ISOLATION` and, when set, passes
|
||||
`hive-c0re` reads `HIVE_NETWORK_ISOLATION` and passes
|
||||
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=<deterministic-ip>`,
|
||||
`HOST_ADDRESS=<bridge-ip>`, and `HOST_BRIDGE=<bridgeName>` via
|
||||
`lifecycle::set_nspawn_flags` when creating or updating containers. Each
|
||||
|
|
@ -171,21 +143,17 @@ but no path off the bridge subnet (no internet, no `api.anthropic.com`).
|
|||
nixos-container copies the **host's** `/etc/resolv.conf` into the container
|
||||
at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved,
|
||||
or a LAN router) is unreachable from a private netns and isn't
|
||||
authoritative for the hive's own zones, so it must be replaced with the
|
||||
bridge dnsmasq (the gateway IP). Because the copy happens on every start, a
|
||||
authoritative for the hive's own zones, so it is replaced with the
|
||||
bridge dnsmasq at boot. Because the copy happens on every start, a
|
||||
declarative `environment.etc."resolv.conf"` would be clobbered — so the
|
||||
wiring is runtime:
|
||||
|
||||
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
|
||||
gateway IP) into the container's `/etc` **only when isolated**, removing
|
||||
it otherwise — so one shared container toplevel behaves correctly in both
|
||||
netns modes.
|
||||
gateway IP) into each container's `/etc`.
|
||||
- the `hyperhive-isolated-dns` oneshot (harness-base.nix), gated on that
|
||||
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
|
||||
It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
|
||||
`tea-login` so the resolver is correct before the first DNS lookup; it's
|
||||
an instant no-op in shared-netns mode (the marker is absent, so
|
||||
`ConditionPathExists` skips it).
|
||||
`tea-login` so the resolver is correct before the first DNS lookup.
|
||||
|
||||
**Why isolation is safe**: all hive-c0re communication goes
|
||||
through unix domain sockets (`/run/hive/mcp.sock` for agent requests,
|
||||
|
|
@ -200,19 +168,6 @@ against a compromised agent reaching the c0re dashboard HTTP at
|
|||
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
|
||||
loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||||
|
||||
### Prerequisites before flipping on
|
||||
|
||||
- All agents must have `hyperhive.web.useUnixSocket = true`. Agents that
|
||||
still bind TCP on `0.0.0.0:<port>` will be reachable at their bridge IP
|
||||
from other agents on the same subnet — defeating the isolation goal. The
|
||||
gateway routes via unix sockets so gateway reach is unaffected.
|
||||
|
||||
### Migration behaviour
|
||||
|
||||
Containers are destroyed and re-created when the flag flips. Agent state
|
||||
under `/agents/<name>/state/` is bind-mounted and survives; the container
|
||||
rootfs is recreated cleanly from the nix store.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- `docs/gateway.md` — vhost map + the gateway container's other duties
|
||||
|
|
|
|||
Loading…
Reference in a new issue