hyperhive/docs/network.md

hive-network

Host-side bridge + per-agent private-netns isolation — always on whenever hyperhive is enabled. Configured via services.hyperhive.network.*.

Isolation is the only mode — there is no shared-netns fallback. The former services.hyperhive.network.enable, services.hyperhive.network.isolateContainers and services.hyperhive.network.upstreamDns options were removed; a config that still sets one fails eval with a removal message.

Network map

One picture of the whole hive. There are two planes: infra containers share the host netns and bind host ports directly; compute containers (agents + CI) each get a private netns behind the bridge. The unix-socket control plane rides the VFS and is untouched by any of it.

                        internet
                           │  uplink NIC — NAT MASQUERADE for the
                           │  bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge / matrix         │
│                                                                    │
│   nginx :80/:443                                  [hive-gateway]   │
│   dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP)         [hive-gateway]   │
│   forgejo :3000 http, :2222 git-ssh               [hive-forge]     │
│   tuwunel :8008 client API                        [hive-matrix]    │
│   hive-c0re dashboard 127.0.0.1:7000              (host service)   │
│   wg-hive :51820/udp — swarm mesh, when enabled   (host iface)     │
│                                                                    │
│                  hive-br0   10.42.0.1/24                           │
│              ┌──────────┼──────────────┐                           │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
          vb-h-<a>    vb-h-<b>      vb-hive-ci      veth pairs
               │          │              │
          ┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐   one private netns
          │ agent a │ │ agent b │ │ hive-ci │   each; eth0 leases
          │  eth0   │ │  eth0   │ │  eth0   │   from the DHCP pool
          └─────────┘ └─────────┘ └─────────┘

container	netns	IPv4	listens / reached via
hive-gateway	host (shared)	host addresses	nginx :80/:443 (every vhost); dnsmasq bridgeIp:53 + DHCP :67 on the bridge
hive-forge	host (shared)	host addresses	forgejo :3000 http, :2222 git-ssh; fronted by the forge.<swarm-domain> vhost
hive-matrix	host (shared)	host addresses	tuwunel :8008 (+ optional federation port); fronted by the matrix vhost
hive-ci	private, veth on bridge	DHCP pool	outbound only (runner → forge); no inbound surface
h-<agent>	private, veth on bridge	DHCP pool	web UI via UDS /run/hive-agent/<name> → nginx sub-path; in-container UI port hashed 8100–8999

The flows, end to end:

• DHCP — agent dhcpcd broadcasts on eth0 → veth → bridge → host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
• DNS — agents query bridgeIp:53; hive zones are answered authoritatively with the bridge IP, everything else forwards to the host's resolvers (see Resolver behaviour below).
• HTTP — forge. and chat. (under swarm.domain) plus the hive's own dashboard name resolve to the bridge IP, land on nginx :80/:443, and proxy to forgejo :3000, tuwunel :8008, hive-c0re 127.0.0.1:7000, or a per-agent UI unix socket.
• Internet egress — agent default route points at the bridge IP; the host forwards + masquerades out its uplink.
• Swarm — peer hives connect over the wg-hive WireGuard mesh and reach each other's gateway/forge across it (docs/swarm/).
• Control plane (no network) — per-agent broker socket /run/hive/mcp.sock, privileged helper /run/hive/priv.sock, operator admin /run/hyperhive/host.sock, and the per-agent UI sockets under /run/hive-agent/ are unix domain sockets bind-mounted through the VFS; private netns does not affect them.

Container shape (where dnsmasq lives)

Co-located in the existing hive-gateway container — single front-door for both DNS and HTTP, saves a sibling container, single systemd-unit / state surface to monitor. The gateway shares host netns (privateNetwork = false) so dnsmasq's bind-interfaces listener on bridgeIp is on the host's bridge interface.

Configuration

{
  services.hyperhive = {
    enable = true;
    hiveName = "pr1ma";
    swarm.domain = "darkest.space";
    swarm.hives.pr1ma = { };          # -> domain = pr1ma.darkest.space
    # network.bridgeIp = "10.42.0.1"; # default
  };
}

Requires services.hyperhive.domain to be set — the dnsmasq resolver is authoritative for <hive-domain> and its sub-domains. You do not write it: it is read from this hive's entry in the swarm directory (docs/swarm/README.md § Hive identity config).

Bridge addressing

Default subnet is 10.42.0.0/24, host-side gateway at 10.42.0.1. RFC 1918 space, unlikely to clash with operator's existing setup; override bridgeIp + bridgePrefixLength if a different range is already in use. /24 gives 254 usable per-agent addresses — enough for any single-host hive; bigger swarms or tighter addressing schemes pick their own.

Resolver behaviour

dnsmasq is authoritative for the hive's own zone (<hive-domain>) plus whatever swarm-service names this host contributes via gateway.localNames — forge.<swarm-domain> and chat.<swarm-domain> (matrix) when this host runs those services, and auth.<swarm-domain> when it runs authelia — answering each with the bridge IP (where nginx is reachable). Note forge and matrix are swarm-domain names, not sub-domains of <hive-domain>: a swarm runs one forge and one homeserver, so their names belong to the swarm rather than to whichever hive happens to host them. Everything else is forwarded to the host's own resolvers: dnsmasq runs on the host and reads the host's /etc/resolv.conf directly. Containers don't need to know the upstream — they query the bridge IP and dnsmasq does the right thing per-name.

There is deliberately no fallback server=: dnsmasq queries all known upstreams in parallel, so a hardcoded public resolver would take a share of normal traffic, not just cover the gap.

dnsmasq runs on the host and reads the host's /etc/resolv.conf directly, so a network change (new router, new lease, laptop moving networks) reaches it the moment openresolv rewrites the file. There is nothing to synchronise and no unit watching for it.

bind-interfaces + interface = [ bridgeName "lo" ] means the listener only accepts queries from the bridge interface (plus lo for container health-checks). External hosts can't reach it — no DNS-amplification surface even when the operator opens port 80 for gateway HTTP.

resolveLocalQueries = false keeps dnsmasq out of the host's own resolution stack — the host's resolver (systemd-resolved, plain glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator configured. The hive resolver is purely for inbound queries from agent containers.

Firewall posture

networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ] networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]

• Port 53 opens the resolver on the bridge interface only. Other interfaces stay closed. The hive resolver isn't an external-facing service.
• Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq receives DHCP via a regular UDP socket (it does not use a netfilter-bypassing raw socket), so the hole is mandatory — without it containers never get a lease and fall back to 169.254.x.x.
• Ports 80 and 443 let isolated agents reach nginx (gateway container, shared host netns) for the forge sub-domain, per-agent UI proxies, and any other HTTP services.

The host firewall is the only firewall. The shared-netns infra containers (gateway, forge, matrix) set networking.firewall.enable = false: a NixOS firewall inside a shared-netns container runs against the host ruleset — at container boot its firewall-start flushes the nixos-fw chains, rebuilds them from the container's (empty) port list, and deletes the host's nixos-nat-* chains without recreating them, silently wiping the bridge holes above plus the agents' NAT. Private-netns containers (agents, hive-ci) may keep their own firewall — it is scoped to their namespace.

Reaching host services (exposeHostPorts)

By default agents can only reach the host on 80/443 (+53 DNS), so a host-side service on another port — e.g. a dev OTLP collector you want agents to reach directly — is unreachable. (hyperhive's own telemetry needs none of this: otel.enable opens its collector's port itself, and otel.endpoint is the upstream, which no agent ever dials. See docs/observability.md.)

services.hyperhive.network.exposeHostPorts = [ 4318 ]; opens each listed TCP port P on the bridge-interface allowedTCPPorts, so an agent can connect to <bridgeIp>:P (point the collector endpoint at http://<bridgeIp>:4318, default http://10.42.0.1:4318).

This is firewall-only: the host service must bind an address reachable from the bridge — 0.0.0.0 or the bridge IP — not loopback only. The bridge→127.0.0.0/8 DROP rule (below) is unchanged, so a service bound to 127.0.0.1 only stays unreachable; rebind it to 0.0.0.0.

The port is reachable by every agent on the bridge subnet (like DNS/gateway), so only expose services safe for any agent to reach.

Container isolation

Each agent container runs in a private network namespace with a dedicated veth pair attached to the bridge. The following table summarises what the nix side sets up unconditionally:

effect	mechanism
IP forwarding	boot.kernel.sysctl."net.ipv4.ip_forward" = 1
Internet NAT	networking.nat { enable = true; internalInterfaces = [ bridgeName ]; } — MASQUERADE on packets leaving via any external NIC
Loopback DROP	networking.firewall.extraInputRules — drops bridge-subnet → 127.0.0.0/8 traffic; defence-in-depth against routing table leaks
Gateway access	networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ] — lets isolated agents (private netns, veth on bridge) reach nginx on the host
c0re signal	HIVE_NETWORK_ISOLATION=1, HIVE_NETWORK_BRIDGE, HIVE_NETWORK_SUBNET in systemd.services.hive-c0re.environment

HIVE_NETWORK_SUBNET is the host-side bridge IP + prefix (e.g. 10.42.0.1/24), not the canonical network address. The Rust side must normalise (bitwise-AND with mask) before subnet membership checks or address arithmetic.

What the Rust side does

hive-c0re reads HIVE_NETWORK_ISOLATION and passes PRIVATE_NETWORK=1, LOCAL_ADDRESS= (empty), HOST_ADDRESS=<bridge-ip>, and HOST_BRIDGE=<bridgeName> via lifecycle::set_nspawn_flags when creating or updating containers. LOCAL_ADDRESS is left empty so the container's dhcpcd acquires an address from the bridge dnsmasq pool (networking.useDHCP = true in nix/agent-modules/network.nix). This applies uniformly to all containers — agents and service containers alike.

HOST_ADDRESS is the bridge gateway IP (the address part of HIVE_NETWORK_SUBNET, via lifecycle::bridge_gateway_ip — taken verbatim so a non-.1 operator override still resolves to wherever the bridge actually lives). It is load-bearing: nixos-container's container-side network setup only installs a default route (ip route add default via $HOST_ADDRESS) when HOST_ADDRESS is non-empty. In bridge mode the host-side address/route setup is skipped, so writing it only affects the container's default route — without it the container comes up with an IP but no path off the bridge subnet (no internet, no api.anthropic.com).

How the isolated container gets its resolver

nixos-container copies the host's /etc/resolv.conf into the container at every start. The host resolver (e.g. 127.0.0.53 from systemd-resolved, or a LAN router) is unreachable from a private netns and isn't authoritative for the hive's own zones, so it is replaced with the bridge dnsmasq at boot. Because the copy happens on every start, a declarative environment.etc."resolv.conf" would be clobbered — so the wiring is runtime:

• hive-priv drops a marker file (/etc/hyperhive-bridge-dns, carrying the gateway IP) into each container's /etc.
• the hyperhive-isolated-dns oneshot (nix/agent-modules/network.nix), gated on that marker, rewrites /etc/resolv.conf to nameserver <gateway-ip> at boot. It is ordered before the harness (hive-ag3nt), the matrix daemon, and tea-login so the resolver is correct before the first DNS lookup.

Why isolation is safe: hive-c0re's control-plane sockets are unix domain sockets bind-mounted into containers, not network listeners — see the Control plane (no network) bullet under Network map above. PRIVATE_NETWORK=1 has no effect on a path that never touches the network stack.

The nix side also enables IP forwarding + NAT (agents reach the internet through the host) and drops bridge-subnet → loopback traffic (defence-in-depth against a compromised agent reaching the c0re dashboard HTTP at 127.0.0.1). Agents have no legitimate reason to reach the dashboard over loopback — the hive-c0re admin socket is a UDS, not TCP.

Cross-references

• docs/gateway.md — vhost map + the gateway's other duties