diff --git a/docs/network.md b/docs/network.md deleted file mode 100644 index c44a444e..00000000 --- a/docs/network.md +++ /dev/null @@ -1,113 +0,0 @@ -# hive-network - -Host-side bridge + per-agent DNS resolver — the foundation that -makes [`#14` container netns isolation](http://localhost:3000/hyperhive/hyperhive/issues/14) -safe to land. Configured via `services.hyperhive.network.*`; off by -default during rollout. - -## Why ship before #14 - -Mara on #805#issuecomment-11541: "we need it before #14 so nothing -breaks when we switch over". If netns isolation lands first, agent -containers lose `/etc/resolv.conf` propagation from the host and DNS -breaks until a separate resolver is up. Inverting the sequence — -bridge + dnsmasq first, netns flip second — makes the flag day -boring: the resolver endpoint is already live, agents just discover -it via veth instead of shared netns. - -## v1 vs v2 - -| feature | v1 (this PR) | v2 (after #14) | -|---|---|---| -| bridge interface | created on host, no slave NICs | per-agent veth pairs attach | -| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) | -| agent container netns | shared host | private | -| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver ` | -| `address` rules target | `` (works in both modes) | unchanged from v1 | - -The `address` rules ship pointing at the bridge IP from v1 so the -DNS contract is fixed before any container actually depends on it -— minimises the things that flip on #14 day. - -## Container shape (where dnsmasq lives) - -Co-located in the existing `hive-gateway` container per mara on -#805:10957 — single front-door for both DNS and HTTP, saves a -sibling container, single systemd-unit / state surface to monitor. -The gateway shares host netns (`privateNetwork = false`) so -dnsmasq's `bind-interfaces` listener on `bridgeIp` works without -any veth gymnastics today; when #14 flips agent containers to -private netns the binding doesn't change (it's still on the host's -bridge interface). - -## Configuration - -```nix -{ - services.hyperhive = { - enable = true; - domain = "darkest.space"; - network.enable = true; # opt in to bridge + DNS - network.bridgeIp = "10.42.0.1"; # default - network.upstreamDns = [ # default Cloudflare + Quad9 - "1.1.1.1" - "9.9.9.9" - ]; - }; -} -``` - -Asserts `services.hyperhive.domain != null` (resolver needs a domain -to be authoritative for) + `services.hyperhive.gateway.enable = -true` (resolver lives in the gateway container). - -## Bridge addressing - -Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`. -RFC 1918 space, unlikely to clash with operator's existing setup; -override `bridgeIp` + `bridgePrefixLength` if a different range is -already in use. `/24` gives 254 usable per-agent addresses — enough -for any single-host hive; bigger swarms or tighter addressing -schemes pick their own. - -## Resolver behaviour - -dnsmasq is **authoritative** for the hive's own zones — answers -``, `forge.`, `matrix.` -queries with the bridge IP (where nginx is reachable). Everything -else gets forwarded to `upstreamDns`. Containers don't need to know -the upstream — they query the bridge IP and dnsmasq does the right -thing per-name. - -`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the -listener only accepts queries from the bridge interface (plus lo for -container health-checks). External hosts can't reach it — no -DNS-amplification surface even when the operator opens port 80 for -gateway HTTP. - -`resolveLocalQueries = false` keeps dnsmasq out of the host's own -resolution stack — the host's resolver (systemd-resolved, plain -glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator -configured. The hive resolver is purely for inbound queries from -agent containers. - -## Firewall posture - -`networking.firewall.interfaces..allowedUDPPorts = [ 53 ]` -+ `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge -interface only. Other interfaces stay closed. The hive resolver -isn't an external-facing service. - -## Sequencing history - -- mara on #805 (`comment-10957`): "can we put the resolver into the - gateway container?" — yes, this v1 does that. -- mara on #805 (`comment-11541`): "we need it before #14 so nothing - breaks when we switch over" — flipped the dependency direction; - v1 ships now, #14 flips containers later. - -## Cross-references - -- Issue [#805](http://localhost:3000/hyperhive/hyperhive/issues/805) — DNS resolver tracking -- Issue [#14](http://localhost:3000/hyperhive/hyperhive/issues/14) — netns isolation (downstream consumer) -- `docs/gateway.md` — vhost map + the gateway container's other duties diff --git a/nix/modules/hive-c0re.nix b/nix/modules/hive-c0re.nix index dc7cc055..6856d9e8 100644 --- a/nix/modules/hive-c0re.nix +++ b/nix/modules/hive-c0re.nix @@ -25,7 +25,6 @@ in ./hive-forge.nix ./hive-gateway.nix ./hive-matrix.nix - ./hive-network.nix ]; # Top-level hyperhive enable flag. When true, automatically enables diff --git a/nix/modules/hive-gateway.nix b/nix/modules/hive-gateway.nix index df43b867..ae787d37 100644 --- a/nix/modules/hive-gateway.nix +++ b/nix/modules/hive-gateway.nix @@ -9,7 +9,6 @@ let hyperhiveDomain = config.services.hyperhive.domain; matrixCfg = config.services.hyperhive.matrix; forgeCfg = config.services.hyperhive.forge; - networkCfg = config.services.hyperhive.network; # Per-agent port table for `/agent//` routing. C0re writes # this JSON on every topology change; gateway reads at deploy time. @@ -552,63 +551,6 @@ in }; }; }; - - # Hive-internal DNS resolver (#805 v1). Co-located in the - # gateway container per mara's call (#805:10957) — single - # front-door for both DNS and HTTP, saves a sibling - # container. Listens on the bridge interface from - # `services.hyperhive.network`; authoritative for the hive - # domain + sub-domains, forwards everything else upstream. - # No-op when `network.enable = false`. - services.dnsmasq = lib.mkIf networkCfg.enable { - enable = true; - # Don't substitute the container's /etc/resolv.conf — - # the gateway uses the host's resolver for its own - # outbound traffic; dnsmasq is purely for incoming - # queries from agent containers. - resolveLocalQueries = false; - settings = { - # Bind only on the bridge interface (and lo for - # health-checks). Outside hosts can't even see the - # listener. - interface = [ - networkCfg.bridgeName - "lo" - ]; - bind-interfaces = true; - port = 53; - # Don't read /etc/resolv.conf — we control upstream - # explicitly to dodge dependency on the gateway - # container's own resolver state. - no-resolv = true; - server = networkCfg.upstreamDns; - # Hive authoritative records — answer queries for the - # hive domain + its sub-domains with the bridge IP - # (where nginx is reachable from container netns once - # #14 lands; today it's the host loopback alias and - # works in either shape). - # - # The forge / matrix entries are redundant in the - # common case where `forge.domain` / - # `matrix.gatewayHost` are sub-domains of - # `hyperhive.domain` — dnsmasq's `//` rule - # already matches sub-domains (argus 🟡 on #845). - # Kept explicit because operators can override either - # to a cross-domain hostname (e.g. - # `forge.domain = "git.example.com"`); listing them - # explicitly keeps that case routed without needing - # an extra config block. - address = [ - "/${hyperhiveDomain}/${networkCfg.bridgeIp}" - ] - ++ lib.optional ( - (forgeCfg.enable or false) && (forgeCfg.behindGateway or false) - ) "/${forgeCfg.domain}/${networkCfg.bridgeIp}" - ++ lib.optional ( - matrixCfg.enable && matrixCfg.gatewayHost != null - ) "/${matrixCfg.gatewayHost}/${networkCfg.bridgeIp}"; - }; - }; }; }; diff --git a/nix/modules/hive-network.nix b/nix/modules/hive-network.nix deleted file mode 100644 index 18e3356b..00000000 --- a/nix/modules/hive-network.nix +++ /dev/null @@ -1,140 +0,0 @@ -{ - lib, - config, - ... -}: -let - cfg = config.services.hyperhive.network; -in -{ - # Hive-internal network — host-side bridge + per-agent DNS resolver - # (#805, prereq for #14 netns isolation). Containers stay on shared - # host netns at v1 — this module just stands the bridge + resolver - # up so the endpoint is in place before #14 flips containers to - # private netns (mara on #805: "we need it before #14 so nothing - # breaks when we switch over"). Full design: docs/network.md. - - options.services.hyperhive.network = { - enable = lib.mkOption { - type = lib.types.bool; - default = false; - example = true; - description = '' - Stand up the hive-internal bridge + dnsmasq resolver. - Off by default while v1 phases in. When enabled: - a bridge interface (`bridgeName`) appears on the host with - `bridgeIp` assigned, and the hive-gateway container runs a - dnsmasq listening on that IP for `` + - sub-domains. Agent containers still default to shared host - netns at v1 — the endpoint is up but only used once #14 - lands and flips containers to a private netns + veth peer - on this bridge. - ''; - }; - - bridgeName = lib.mkOption { - type = lib.types.str; - default = "hive-br0"; - example = "h0"; - description = '' - Name of the host-side bridge interface the hive uses for - inter-container traffic. Kept short so it survives the - IFNAMSIZ (15-char) cap, and prefixed so it's obviously - hive-managed in `ip link` output. - ''; - }; - - bridgeIp = lib.mkOption { - type = lib.types.str; - default = "10.42.0.1"; - example = "172.30.0.1"; - description = '' - IPv4 address assigned to the bridge interface on the host - side. Becomes the DNS server address agents point at (and - the upstream the gateway proxies to once netns isolation - lands). Default `10.42.0.1` is in RFC 1918 space and - unlikely to clash with operator's existing setup; override - if a different range is already in use. - ''; - }; - - bridgePrefixLength = lib.mkOption { - type = lib.types.int; - default = 24; - example = 16; - description = '' - Netmask prefix length for the bridge subnet. Default `/24` - gives 254 usable per-agent addresses, enough for any - single-host hive. Operator with a larger swarm or a tighter - addressing scheme overrides. - ''; - }; - - upstreamDns = lib.mkOption { - type = lib.types.listOf lib.types.str; - default = [ - "1.1.1.1" - "9.9.9.9" - ]; - example = [ - "192.168.1.1" - "8.8.8.8" - ]; - description = '' - Upstream DNS servers dnsmasq forwards non-hive queries to. - Defaults to Cloudflare + Quad9. Override for operators on - private networks who need a specific resolver (corporate - DNS, pi-hole, etc.). The hive resolver itself stays - authoritative for `` and its sub-domains - regardless of upstream choice. - ''; - }; - }; - - config = lib.mkIf cfg.enable { - assertions = [ - { - assertion = config.services.hyperhive.domain != null; - message = '' - services.hyperhive.network.enable = true requires - services.hyperhive.domain to be set — the resolver needs a - domain to be authoritative for. Either pin a hostname - (`services.hyperhive.domain = "example.com";`) or leave - `network.enable` at its default of false. - ''; - } - { - assertion = config.services.hyperhive.gateway.enable; - message = '' - services.hyperhive.network.enable = true requires - services.hyperhive.gateway.enable = true — the dnsmasq - resolver runs inside the hive-gateway container (single - front-door for both DNS and HTTP). Enable the gateway or - leave `network.enable` at its default of false. - ''; - } - ]; - - # Bridge interface on the host. Empty interfaces list = purely - # virtual bridge (no slave NICs attached). Per-agent veth pairs - # will join this bridge once #14 lands; at v1 it stands alone. - networking.bridges.${cfg.bridgeName}.interfaces = [ ]; - - # Host-side IP assignment on the bridge. This is what dnsmasq - # (inside the gateway container, shared host netns) binds on. - networking.interfaces.${cfg.bridgeName}.ipv4.addresses = [ - { - address = cfg.bridgeIp; - prefixLength = cfg.bridgePrefixLength; - } - ]; - - # Open the resolver port in the host firewall for traffic from - # the bridge subnet only. Other interfaces stay closed — - # external DNS-amplification surface is not exposed. - networking.firewall.interfaces.${cfg.bridgeName} = { - allowedUDPPorts = [ 53 ]; - allowedTCPPorts = [ 53 ]; - }; - }; -}