nix/hive-network: bridge + dnsmasq resolver, opt-in (#805 v1)
Stand up the host-side bridge interface + per-agent DNS resolver ahead of #14 (netns isolation). Mara on #805#11541: "we need it before #14 so nothing breaks when we switch over". v1 ships the endpoint live but containers stay on shared host netns — when #14 flips them to private netns the DNS contract is already there. Shape: - new `nix/modules/hive-network.nix` with `services.hyperhive.network.*` options (enable + bridgeName + bridgeIp + bridgePrefixLength + upstreamDns). Default off. Imported from `hive-c0re.nix`. - bridge interface via `networking.bridges` (no slave NICs at v1; per-agent veth pairs attach once #14 lands). - bridge IP assigned via `networking.interfaces`. - `networking.firewall.interfaces.<bridge>.allowed{UDP,TCP}Ports = [ 53 ]` opens the resolver on the bridge interface only — other interfaces stay closed. - dnsmasq config added to the existing `hive-gateway` container (mara on #805:10957: "put the resolver into the gateway container"). Listens only on `bridgeName` + `lo`; authoritative for `<hive-domain>`, `forge.<hive>`, `matrix.<hive>` answering with the bridge IP; forwards everything else to upstream. `resolveLocalQueries = false` keeps the gateway container's own resolver untouched. Asserts `services.hyperhive.domain != null` + `gateway.enable = true` — both required for the resolver to be meaningful. Docs: new `docs/network.md` covering v1 vs v2 split, container shape rationale, default addressing, resolver behaviour, firewall posture. `nix flake check` clean.
This commit is contained in:
parent
0d328d1fcd
commit
ed3b9d853e
4 changed files with 301 additions and 0 deletions
113
docs/network.md
Normal file
113
docs/network.md
Normal file
|
|
@ -0,0 +1,113 @@
|
||||||
|
# hive-network
|
||||||
|
|
||||||
|
Host-side bridge + per-agent DNS resolver — the foundation that
|
||||||
|
makes [`#14` container netns isolation](http://localhost:3000/hyperhive/hyperhive/issues/14)
|
||||||
|
safe to land. Configured via `services.hyperhive.network.*`; off by
|
||||||
|
default during rollout.
|
||||||
|
|
||||||
|
## Why ship before #14
|
||||||
|
|
||||||
|
Mara on #805#issuecomment-11541: "we need it before #14 so nothing
|
||||||
|
breaks when we switch over". If netns isolation lands first, agent
|
||||||
|
containers lose `/etc/resolv.conf` propagation from the host and DNS
|
||||||
|
breaks until a separate resolver is up. Inverting the sequence —
|
||||||
|
bridge + dnsmasq first, netns flip second — makes the flag day
|
||||||
|
boring: the resolver endpoint is already live, agents just discover
|
||||||
|
it via veth instead of shared netns.
|
||||||
|
|
||||||
|
## v1 vs v2
|
||||||
|
|
||||||
|
| feature | v1 (this PR) | v2 (after #14) |
|
||||||
|
|---|---|---|
|
||||||
|
| bridge interface | created on host, no slave NICs | per-agent veth pairs attach |
|
||||||
|
| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) |
|
||||||
|
| agent container netns | shared host | private |
|
||||||
|
| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver <bridge-ip>` |
|
||||||
|
| `address` rules target | `<bridge-ip>` (works in both modes) | unchanged from v1 |
|
||||||
|
|
||||||
|
The `address` rules ship pointing at the bridge IP from v1 so the
|
||||||
|
DNS contract is fixed before any container actually depends on it
|
||||||
|
— minimises the things that flip on #14 day.
|
||||||
|
|
||||||
|
## Container shape (where dnsmasq lives)
|
||||||
|
|
||||||
|
Co-located in the existing `hive-gateway` container per mara on
|
||||||
|
#805:10957 — single front-door for both DNS and HTTP, saves a
|
||||||
|
sibling container, single systemd-unit / state surface to monitor.
|
||||||
|
The gateway shares host netns (`privateNetwork = false`) so
|
||||||
|
dnsmasq's `bind-interfaces` listener on `bridgeIp` works without
|
||||||
|
any veth gymnastics today; when #14 flips agent containers to
|
||||||
|
private netns the binding doesn't change (it's still on the host's
|
||||||
|
bridge interface).
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```nix
|
||||||
|
{
|
||||||
|
services.hyperhive = {
|
||||||
|
enable = true;
|
||||||
|
domain = "darkest.space";
|
||||||
|
network.enable = true; # opt in to bridge + DNS
|
||||||
|
network.bridgeIp = "10.42.0.1"; # default
|
||||||
|
network.upstreamDns = [ # default Cloudflare + Quad9
|
||||||
|
"1.1.1.1"
|
||||||
|
"9.9.9.9"
|
||||||
|
];
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Asserts `services.hyperhive.domain != null` (resolver needs a domain
|
||||||
|
to be authoritative for) + `services.hyperhive.gateway.enable =
|
||||||
|
true` (resolver lives in the gateway container).
|
||||||
|
|
||||||
|
## Bridge addressing
|
||||||
|
|
||||||
|
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
|
||||||
|
RFC 1918 space, unlikely to clash with operator's existing setup;
|
||||||
|
override `bridgeIp` + `bridgePrefixLength` if a different range is
|
||||||
|
already in use. `/24` gives 254 usable per-agent addresses — enough
|
||||||
|
for any single-host hive; bigger swarms or tighter addressing
|
||||||
|
schemes pick their own.
|
||||||
|
|
||||||
|
## Resolver behaviour
|
||||||
|
|
||||||
|
dnsmasq is **authoritative** for the hive's own zones — answers
|
||||||
|
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>`
|
||||||
|
queries with the bridge IP (where nginx is reachable). Everything
|
||||||
|
else gets forwarded to `upstreamDns`. Containers don't need to know
|
||||||
|
the upstream — they query the bridge IP and dnsmasq does the right
|
||||||
|
thing per-name.
|
||||||
|
|
||||||
|
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
||||||
|
listener only accepts queries from the bridge interface (plus lo for
|
||||||
|
container health-checks). External hosts can't reach it — no
|
||||||
|
DNS-amplification surface even when the operator opens port 80 for
|
||||||
|
gateway HTTP.
|
||||||
|
|
||||||
|
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
|
||||||
|
resolution stack — the host's resolver (systemd-resolved, plain
|
||||||
|
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
|
||||||
|
configured. The hive resolver is purely for inbound queries from
|
||||||
|
agent containers.
|
||||||
|
|
||||||
|
## Firewall posture
|
||||||
|
|
||||||
|
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 ]`
|
||||||
|
+ `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge
|
||||||
|
interface only. Other interfaces stay closed. The hive resolver
|
||||||
|
isn't an external-facing service.
|
||||||
|
|
||||||
|
## Sequencing history
|
||||||
|
|
||||||
|
- mara on #805 (`comment-10957`): "can we put the resolver into the
|
||||||
|
gateway container?" — yes, this v1 does that.
|
||||||
|
- mara on #805 (`comment-11541`): "we need it before #14 so nothing
|
||||||
|
breaks when we switch over" — flipped the dependency direction;
|
||||||
|
v1 ships now, #14 flips containers later.
|
||||||
|
|
||||||
|
## Cross-references
|
||||||
|
|
||||||
|
- Issue [#805](http://localhost:3000/hyperhive/hyperhive/issues/805) — DNS resolver tracking
|
||||||
|
- Issue [#14](http://localhost:3000/hyperhive/hyperhive/issues/14) — netns isolation (downstream consumer)
|
||||||
|
- `docs/gateway.md` — vhost map + the gateway container's other duties
|
||||||
|
|
@ -25,6 +25,7 @@ in
|
||||||
./hive-forge.nix
|
./hive-forge.nix
|
||||||
./hive-gateway.nix
|
./hive-gateway.nix
|
||||||
./hive-matrix.nix
|
./hive-matrix.nix
|
||||||
|
./hive-network.nix
|
||||||
];
|
];
|
||||||
|
|
||||||
# Top-level hyperhive enable flag. When true, automatically enables
|
# Top-level hyperhive enable flag. When true, automatically enables
|
||||||
|
|
|
||||||
|
|
@ -9,6 +9,7 @@ let
|
||||||
hyperhiveDomain = config.services.hyperhive.domain;
|
hyperhiveDomain = config.services.hyperhive.domain;
|
||||||
matrixCfg = config.services.hyperhive.matrix;
|
matrixCfg = config.services.hyperhive.matrix;
|
||||||
forgeCfg = config.services.hyperhive.forge;
|
forgeCfg = config.services.hyperhive.forge;
|
||||||
|
networkCfg = config.services.hyperhive.network;
|
||||||
|
|
||||||
# Per-agent port table for `/agent/<name>/` routing. C0re writes
|
# Per-agent port table for `/agent/<name>/` routing. C0re writes
|
||||||
# this JSON on every topology change; gateway reads at deploy time.
|
# this JSON on every topology change; gateway reads at deploy time.
|
||||||
|
|
@ -551,6 +552,52 @@ in
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# Hive-internal DNS resolver (#805 v1). Co-located in the
|
||||||
|
# gateway container per mara's call (#805:10957) — single
|
||||||
|
# front-door for both DNS and HTTP, saves a sibling
|
||||||
|
# container. Listens on the bridge interface from
|
||||||
|
# `services.hyperhive.network`; authoritative for the hive
|
||||||
|
# domain + sub-domains, forwards everything else upstream.
|
||||||
|
# No-op when `network.enable = false`.
|
||||||
|
services.dnsmasq = lib.mkIf networkCfg.enable {
|
||||||
|
enable = true;
|
||||||
|
# Don't substitute the container's /etc/resolv.conf —
|
||||||
|
# the gateway uses the host's resolver for its own
|
||||||
|
# outbound traffic; dnsmasq is purely for incoming
|
||||||
|
# queries from agent containers.
|
||||||
|
resolveLocalQueries = false;
|
||||||
|
settings = {
|
||||||
|
# Bind only on the bridge interface (and lo for
|
||||||
|
# health-checks). Outside hosts can't even see the
|
||||||
|
# listener.
|
||||||
|
interface = [
|
||||||
|
networkCfg.bridgeName
|
||||||
|
"lo"
|
||||||
|
];
|
||||||
|
bind-interfaces = true;
|
||||||
|
port = 53;
|
||||||
|
# Don't read /etc/resolv.conf — we control upstream
|
||||||
|
# explicitly to dodge dependency on the gateway
|
||||||
|
# container's own resolver state.
|
||||||
|
no-resolv = true;
|
||||||
|
server = networkCfg.upstreamDns;
|
||||||
|
# Hive authoritative records — answer queries for the
|
||||||
|
# hive domain + its sub-domains with the bridge IP
|
||||||
|
# (where nginx is reachable from container netns once
|
||||||
|
# #14 lands; today it's the host loopback alias and
|
||||||
|
# works in either shape).
|
||||||
|
address = [
|
||||||
|
"/${hyperhiveDomain}/${networkCfg.bridgeIp}"
|
||||||
|
]
|
||||||
|
++ lib.optional (
|
||||||
|
(forgeCfg.enable or false) && (forgeCfg.behindGateway or false)
|
||||||
|
) "/${forgeCfg.domain}/${networkCfg.bridgeIp}"
|
||||||
|
++ lib.optional (
|
||||||
|
matrixCfg.enable && matrixCfg.gatewayHost != null
|
||||||
|
) "/${matrixCfg.gatewayHost}/${networkCfg.bridgeIp}";
|
||||||
|
};
|
||||||
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
140
nix/modules/hive-network.nix
Normal file
140
nix/modules/hive-network.nix
Normal file
|
|
@ -0,0 +1,140 @@
|
||||||
|
{
|
||||||
|
lib,
|
||||||
|
config,
|
||||||
|
...
|
||||||
|
}:
|
||||||
|
let
|
||||||
|
cfg = config.services.hyperhive.network;
|
||||||
|
in
|
||||||
|
{
|
||||||
|
# Hive-internal network — host-side bridge + per-agent DNS resolver
|
||||||
|
# (#805, prereq for #14 netns isolation). Containers stay on shared
|
||||||
|
# host netns at v1 — this module just stands the bridge + resolver
|
||||||
|
# up so the endpoint is in place before #14 flips containers to
|
||||||
|
# private netns (mara on #805: "we need it before #14 so nothing
|
||||||
|
# breaks when we switch over"). Full design: docs/network.md.
|
||||||
|
|
||||||
|
options.services.hyperhive.network = {
|
||||||
|
enable = lib.mkOption {
|
||||||
|
type = lib.types.bool;
|
||||||
|
default = false;
|
||||||
|
example = true;
|
||||||
|
description = ''
|
||||||
|
Stand up the hive-internal bridge + dnsmasq resolver.
|
||||||
|
Off by default while v1 phases in. When enabled:
|
||||||
|
a bridge interface (`bridgeName`) appears on the host with
|
||||||
|
`bridgeIp` assigned, and the hive-gateway container runs a
|
||||||
|
dnsmasq listening on that IP for `<hive-domain>` +
|
||||||
|
sub-domains. Agent containers still default to shared host
|
||||||
|
netns at v1 — the endpoint is up but only used once #14
|
||||||
|
lands and flips containers to a private netns + veth peer
|
||||||
|
on this bridge.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
bridgeName = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "hive-br0";
|
||||||
|
example = "h0";
|
||||||
|
description = ''
|
||||||
|
Name of the host-side bridge interface the hive uses for
|
||||||
|
inter-container traffic. Kept short so it survives the
|
||||||
|
IFNAMSIZ (15-char) cap, and prefixed so it's obviously
|
||||||
|
hive-managed in `ip link` output.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
bridgeIp = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "10.42.0.1";
|
||||||
|
example = "172.30.0.1";
|
||||||
|
description = ''
|
||||||
|
IPv4 address assigned to the bridge interface on the host
|
||||||
|
side. Becomes the DNS server address agents point at (and
|
||||||
|
the upstream the gateway proxies to once netns isolation
|
||||||
|
lands). Default `10.42.0.1` is in RFC 1918 space and
|
||||||
|
unlikely to clash with operator's existing setup; override
|
||||||
|
if a different range is already in use.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
bridgePrefixLength = lib.mkOption {
|
||||||
|
type = lib.types.int;
|
||||||
|
default = 24;
|
||||||
|
example = 16;
|
||||||
|
description = ''
|
||||||
|
Netmask prefix length for the bridge subnet. Default `/24`
|
||||||
|
gives 254 usable per-agent addresses, enough for any
|
||||||
|
single-host hive. Operator with a larger swarm or a tighter
|
||||||
|
addressing scheme overrides.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
upstreamDns = lib.mkOption {
|
||||||
|
type = lib.types.listOf lib.types.str;
|
||||||
|
default = [
|
||||||
|
"1.1.1.1"
|
||||||
|
"9.9.9.9"
|
||||||
|
];
|
||||||
|
example = [
|
||||||
|
"192.168.1.1"
|
||||||
|
"8.8.8.8"
|
||||||
|
];
|
||||||
|
description = ''
|
||||||
|
Upstream DNS servers dnsmasq forwards non-hive queries to.
|
||||||
|
Defaults to Cloudflare + Quad9. Override for operators on
|
||||||
|
private networks who need a specific resolver (corporate
|
||||||
|
DNS, pi-hole, etc.). The hive resolver itself stays
|
||||||
|
authoritative for `<hive-domain>` and its sub-domains
|
||||||
|
regardless of upstream choice.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
config = lib.mkIf cfg.enable {
|
||||||
|
assertions = [
|
||||||
|
{
|
||||||
|
assertion = config.services.hyperhive.domain != null;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.network.enable = true requires
|
||||||
|
services.hyperhive.domain to be set — the resolver needs a
|
||||||
|
domain to be authoritative for. Either pin a hostname
|
||||||
|
(`services.hyperhive.domain = "example.com";`) or leave
|
||||||
|
`network.enable` at its default of false.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
{
|
||||||
|
assertion = config.services.hyperhive.gateway.enable;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.network.enable = true requires
|
||||||
|
services.hyperhive.gateway.enable = true — the dnsmasq
|
||||||
|
resolver runs inside the hive-gateway container (single
|
||||||
|
front-door for both DNS and HTTP). Enable the gateway or
|
||||||
|
leave `network.enable` at its default of false.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# Bridge interface on the host. Empty interfaces list = purely
|
||||||
|
# virtual bridge (no slave NICs attached). Per-agent veth pairs
|
||||||
|
# will join this bridge once #14 lands; at v1 it stands alone.
|
||||||
|
networking.bridges.${cfg.bridgeName}.interfaces = [ ];
|
||||||
|
|
||||||
|
# Host-side IP assignment on the bridge. This is what dnsmasq
|
||||||
|
# (inside the gateway container, shared host netns) binds on.
|
||||||
|
networking.interfaces.${cfg.bridgeName}.ipv4.addresses = [
|
||||||
|
{
|
||||||
|
address = cfg.bridgeIp;
|
||||||
|
prefixLength = cfg.bridgePrefixLength;
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# Open the resolver port in the host firewall for traffic from
|
||||||
|
# the bridge subnet only. Other interfaces stay closed —
|
||||||
|
# external DNS-amplification surface is not exposed.
|
||||||
|
networking.firewall.interfaces.${cfg.bridgeName} = {
|
||||||
|
allowedUDPPorts = [ 53 ];
|
||||||
|
allowedTCPPorts = [ 53 ];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue