diff --git a/docs/network.md b/docs/network.md new file mode 100644 index 00000000..c44a444e --- /dev/null +++ b/docs/network.md @@ -0,0 +1,113 @@ +# hive-network + +Host-side bridge + per-agent DNS resolver — the foundation that +makes [`#14` container netns isolation](http://localhost:3000/hyperhive/hyperhive/issues/14) +safe to land. Configured via `services.hyperhive.network.*`; off by +default during rollout. + +## Why ship before #14 + +Mara on #805#issuecomment-11541: "we need it before #14 so nothing +breaks when we switch over". If netns isolation lands first, agent +containers lose `/etc/resolv.conf` propagation from the host and DNS +breaks until a separate resolver is up. Inverting the sequence — +bridge + dnsmasq first, netns flip second — makes the flag day +boring: the resolver endpoint is already live, agents just discover +it via veth instead of shared netns. + +## v1 vs v2 + +| feature | v1 (this PR) | v2 (after #14) | +|---|---|---| +| bridge interface | created on host, no slave NICs | per-agent veth pairs attach | +| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) | +| agent container netns | shared host | private | +| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver ` | +| `address` rules target | `` (works in both modes) | unchanged from v1 | + +The `address` rules ship pointing at the bridge IP from v1 so the +DNS contract is fixed before any container actually depends on it +— minimises the things that flip on #14 day. + +## Container shape (where dnsmasq lives) + +Co-located in the existing `hive-gateway` container per mara on +#805:10957 — single front-door for both DNS and HTTP, saves a +sibling container, single systemd-unit / state surface to monitor. +The gateway shares host netns (`privateNetwork = false`) so +dnsmasq's `bind-interfaces` listener on `bridgeIp` works without +any veth gymnastics today; when #14 flips agent containers to +private netns the binding doesn't change (it's still on the host's +bridge interface). + +## Configuration + +```nix +{ + services.hyperhive = { + enable = true; + domain = "darkest.space"; + network.enable = true; # opt in to bridge + DNS + network.bridgeIp = "10.42.0.1"; # default + network.upstreamDns = [ # default Cloudflare + Quad9 + "1.1.1.1" + "9.9.9.9" + ]; + }; +} +``` + +Asserts `services.hyperhive.domain != null` (resolver needs a domain +to be authoritative for) + `services.hyperhive.gateway.enable = +true` (resolver lives in the gateway container). + +## Bridge addressing + +Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`. +RFC 1918 space, unlikely to clash with operator's existing setup; +override `bridgeIp` + `bridgePrefixLength` if a different range is +already in use. `/24` gives 254 usable per-agent addresses — enough +for any single-host hive; bigger swarms or tighter addressing +schemes pick their own. + +## Resolver behaviour + +dnsmasq is **authoritative** for the hive's own zones — answers +``, `forge.`, `matrix.` +queries with the bridge IP (where nginx is reachable). Everything +else gets forwarded to `upstreamDns`. Containers don't need to know +the upstream — they query the bridge IP and dnsmasq does the right +thing per-name. + +`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the +listener only accepts queries from the bridge interface (plus lo for +container health-checks). External hosts can't reach it — no +DNS-amplification surface even when the operator opens port 80 for +gateway HTTP. + +`resolveLocalQueries = false` keeps dnsmasq out of the host's own +resolution stack — the host's resolver (systemd-resolved, plain +glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator +configured. The hive resolver is purely for inbound queries from +agent containers. + +## Firewall posture + +`networking.firewall.interfaces..allowedUDPPorts = [ 53 ]` ++ `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge +interface only. Other interfaces stay closed. The hive resolver +isn't an external-facing service. + +## Sequencing history + +- mara on #805 (`comment-10957`): "can we put the resolver into the + gateway container?" — yes, this v1 does that. +- mara on #805 (`comment-11541`): "we need it before #14 so nothing + breaks when we switch over" — flipped the dependency direction; + v1 ships now, #14 flips containers later. + +## Cross-references + +- Issue [#805](http://localhost:3000/hyperhive/hyperhive/issues/805) — DNS resolver tracking +- Issue [#14](http://localhost:3000/hyperhive/hyperhive/issues/14) — netns isolation (downstream consumer) +- `docs/gateway.md` — vhost map + the gateway container's other duties diff --git a/nix/modules/hive-c0re.nix b/nix/modules/hive-c0re.nix index 6856d9e8..dc7cc055 100644 --- a/nix/modules/hive-c0re.nix +++ b/nix/modules/hive-c0re.nix @@ -25,6 +25,7 @@ in ./hive-forge.nix ./hive-gateway.nix ./hive-matrix.nix + ./hive-network.nix ]; # Top-level hyperhive enable flag. When true, automatically enables diff --git a/nix/modules/hive-gateway.nix b/nix/modules/hive-gateway.nix index ae787d37..1614f5c5 100644 --- a/nix/modules/hive-gateway.nix +++ b/nix/modules/hive-gateway.nix @@ -9,6 +9,7 @@ let hyperhiveDomain = config.services.hyperhive.domain; matrixCfg = config.services.hyperhive.matrix; forgeCfg = config.services.hyperhive.forge; + networkCfg = config.services.hyperhive.network; # Per-agent port table for `/agent//` routing. C0re writes # this JSON on every topology change; gateway reads at deploy time. @@ -551,6 +552,52 @@ in }; }; }; + + # Hive-internal DNS resolver (#805 v1). Co-located in the + # gateway container per mara's call (#805:10957) — single + # front-door for both DNS and HTTP, saves a sibling + # container. Listens on the bridge interface from + # `services.hyperhive.network`; authoritative for the hive + # domain + sub-domains, forwards everything else upstream. + # No-op when `network.enable = false`. + services.dnsmasq = lib.mkIf networkCfg.enable { + enable = true; + # Don't substitute the container's /etc/resolv.conf — + # the gateway uses the host's resolver for its own + # outbound traffic; dnsmasq is purely for incoming + # queries from agent containers. + resolveLocalQueries = false; + settings = { + # Bind only on the bridge interface (and lo for + # health-checks). Outside hosts can't even see the + # listener. + interface = [ + networkCfg.bridgeName + "lo" + ]; + bind-interfaces = true; + port = 53; + # Don't read /etc/resolv.conf — we control upstream + # explicitly to dodge dependency on the gateway + # container's own resolver state. + no-resolv = true; + server = networkCfg.upstreamDns; + # Hive authoritative records — answer queries for the + # hive domain + its sub-domains with the bridge IP + # (where nginx is reachable from container netns once + # #14 lands; today it's the host loopback alias and + # works in either shape). + address = [ + "/${hyperhiveDomain}/${networkCfg.bridgeIp}" + ] + ++ lib.optional ( + (forgeCfg.enable or false) && (forgeCfg.behindGateway or false) + ) "/${forgeCfg.domain}/${networkCfg.bridgeIp}" + ++ lib.optional ( + matrixCfg.enable && matrixCfg.gatewayHost != null + ) "/${matrixCfg.gatewayHost}/${networkCfg.bridgeIp}"; + }; + }; }; }; diff --git a/nix/modules/hive-network.nix b/nix/modules/hive-network.nix new file mode 100644 index 00000000..18e3356b --- /dev/null +++ b/nix/modules/hive-network.nix @@ -0,0 +1,140 @@ +{ + lib, + config, + ... +}: +let + cfg = config.services.hyperhive.network; +in +{ + # Hive-internal network — host-side bridge + per-agent DNS resolver + # (#805, prereq for #14 netns isolation). Containers stay on shared + # host netns at v1 — this module just stands the bridge + resolver + # up so the endpoint is in place before #14 flips containers to + # private netns (mara on #805: "we need it before #14 so nothing + # breaks when we switch over"). Full design: docs/network.md. + + options.services.hyperhive.network = { + enable = lib.mkOption { + type = lib.types.bool; + default = false; + example = true; + description = '' + Stand up the hive-internal bridge + dnsmasq resolver. + Off by default while v1 phases in. When enabled: + a bridge interface (`bridgeName`) appears on the host with + `bridgeIp` assigned, and the hive-gateway container runs a + dnsmasq listening on that IP for `` + + sub-domains. Agent containers still default to shared host + netns at v1 — the endpoint is up but only used once #14 + lands and flips containers to a private netns + veth peer + on this bridge. + ''; + }; + + bridgeName = lib.mkOption { + type = lib.types.str; + default = "hive-br0"; + example = "h0"; + description = '' + Name of the host-side bridge interface the hive uses for + inter-container traffic. Kept short so it survives the + IFNAMSIZ (15-char) cap, and prefixed so it's obviously + hive-managed in `ip link` output. + ''; + }; + + bridgeIp = lib.mkOption { + type = lib.types.str; + default = "10.42.0.1"; + example = "172.30.0.1"; + description = '' + IPv4 address assigned to the bridge interface on the host + side. Becomes the DNS server address agents point at (and + the upstream the gateway proxies to once netns isolation + lands). Default `10.42.0.1` is in RFC 1918 space and + unlikely to clash with operator's existing setup; override + if a different range is already in use. + ''; + }; + + bridgePrefixLength = lib.mkOption { + type = lib.types.int; + default = 24; + example = 16; + description = '' + Netmask prefix length for the bridge subnet. Default `/24` + gives 254 usable per-agent addresses, enough for any + single-host hive. Operator with a larger swarm or a tighter + addressing scheme overrides. + ''; + }; + + upstreamDns = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ + "1.1.1.1" + "9.9.9.9" + ]; + example = [ + "192.168.1.1" + "8.8.8.8" + ]; + description = '' + Upstream DNS servers dnsmasq forwards non-hive queries to. + Defaults to Cloudflare + Quad9. Override for operators on + private networks who need a specific resolver (corporate + DNS, pi-hole, etc.). The hive resolver itself stays + authoritative for `` and its sub-domains + regardless of upstream choice. + ''; + }; + }; + + config = lib.mkIf cfg.enable { + assertions = [ + { + assertion = config.services.hyperhive.domain != null; + message = '' + services.hyperhive.network.enable = true requires + services.hyperhive.domain to be set — the resolver needs a + domain to be authoritative for. Either pin a hostname + (`services.hyperhive.domain = "example.com";`) or leave + `network.enable` at its default of false. + ''; + } + { + assertion = config.services.hyperhive.gateway.enable; + message = '' + services.hyperhive.network.enable = true requires + services.hyperhive.gateway.enable = true — the dnsmasq + resolver runs inside the hive-gateway container (single + front-door for both DNS and HTTP). Enable the gateway or + leave `network.enable` at its default of false. + ''; + } + ]; + + # Bridge interface on the host. Empty interfaces list = purely + # virtual bridge (no slave NICs attached). Per-agent veth pairs + # will join this bridge once #14 lands; at v1 it stands alone. + networking.bridges.${cfg.bridgeName}.interfaces = [ ]; + + # Host-side IP assignment on the bridge. This is what dnsmasq + # (inside the gateway container, shared host netns) binds on. + networking.interfaces.${cfg.bridgeName}.ipv4.addresses = [ + { + address = cfg.bridgeIp; + prefixLength = cfg.bridgePrefixLength; + } + ]; + + # Open the resolver port in the host firewall for traffic from + # the bridge subnet only. Other interfaces stay closed — + # external DNS-amplification surface is not exposed. + networking.firewall.interfaces.${cfg.bridgeName} = { + allowedUDPPorts = [ 53 ]; + allowedTCPPorts = [ 53 ]; + }; + }; +}