Compare commits
4 changed files with 0 additions and 312 deletions
113
docs/network.md
113
docs/network.md
|
|
@ -1,113 +0,0 @@
|
||||||
# hive-network
|
|
||||||
|
|
||||||
Host-side bridge + per-agent DNS resolver — the foundation that
|
|
||||||
makes [`#14` container netns isolation](http://localhost:3000/hyperhive/hyperhive/issues/14)
|
|
||||||
safe to land. Configured via `services.hyperhive.network.*`; off by
|
|
||||||
default during rollout.
|
|
||||||
|
|
||||||
## Why ship before #14
|
|
||||||
|
|
||||||
Mara on #805#issuecomment-11541: "we need it before #14 so nothing
|
|
||||||
breaks when we switch over". If netns isolation lands first, agent
|
|
||||||
containers lose `/etc/resolv.conf` propagation from the host and DNS
|
|
||||||
breaks until a separate resolver is up. Inverting the sequence —
|
|
||||||
bridge + dnsmasq first, netns flip second — makes the flag day
|
|
||||||
boring: the resolver endpoint is already live, agents just discover
|
|
||||||
it via veth instead of shared netns.
|
|
||||||
|
|
||||||
## v1 vs v2
|
|
||||||
|
|
||||||
| feature | v1 (this PR) | v2 (after #14) |
|
|
||||||
|---|---|---|
|
|
||||||
| bridge interface | created on host, no slave NICs | per-agent veth pairs attach |
|
|
||||||
| dnsmasq binding | bridge IP (reachable via host loopback in shared netns) | bridge IP (reachable via veth in private netns) |
|
|
||||||
| agent container netns | shared host | private |
|
|
||||||
| agent `/etc/resolv.conf` | unchanged (host DNS) | `nameserver <bridge-ip>` |
|
|
||||||
| `address` rules target | `<bridge-ip>` (works in both modes) | unchanged from v1 |
|
|
||||||
|
|
||||||
The `address` rules ship pointing at the bridge IP from v1 so the
|
|
||||||
DNS contract is fixed before any container actually depends on it
|
|
||||||
— minimises the things that flip on #14 day.
|
|
||||||
|
|
||||||
## Container shape (where dnsmasq lives)
|
|
||||||
|
|
||||||
Co-located in the existing `hive-gateway` container per mara on
|
|
||||||
#805:10957 — single front-door for both DNS and HTTP, saves a
|
|
||||||
sibling container, single systemd-unit / state surface to monitor.
|
|
||||||
The gateway shares host netns (`privateNetwork = false`) so
|
|
||||||
dnsmasq's `bind-interfaces` listener on `bridgeIp` works without
|
|
||||||
any veth gymnastics today; when #14 flips agent containers to
|
|
||||||
private netns the binding doesn't change (it's still on the host's
|
|
||||||
bridge interface).
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
```nix
|
|
||||||
{
|
|
||||||
services.hyperhive = {
|
|
||||||
enable = true;
|
|
||||||
domain = "darkest.space";
|
|
||||||
network.enable = true; # opt in to bridge + DNS
|
|
||||||
network.bridgeIp = "10.42.0.1"; # default
|
|
||||||
network.upstreamDns = [ # default Cloudflare + Quad9
|
|
||||||
"1.1.1.1"
|
|
||||||
"9.9.9.9"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Asserts `services.hyperhive.domain != null` (resolver needs a domain
|
|
||||||
to be authoritative for) + `services.hyperhive.gateway.enable =
|
|
||||||
true` (resolver lives in the gateway container).
|
|
||||||
|
|
||||||
## Bridge addressing
|
|
||||||
|
|
||||||
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
|
|
||||||
RFC 1918 space, unlikely to clash with operator's existing setup;
|
|
||||||
override `bridgeIp` + `bridgePrefixLength` if a different range is
|
|
||||||
already in use. `/24` gives 254 usable per-agent addresses — enough
|
|
||||||
for any single-host hive; bigger swarms or tighter addressing
|
|
||||||
schemes pick their own.
|
|
||||||
|
|
||||||
## Resolver behaviour
|
|
||||||
|
|
||||||
dnsmasq is **authoritative** for the hive's own zones — answers
|
|
||||||
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>`
|
|
||||||
queries with the bridge IP (where nginx is reachable). Everything
|
|
||||||
else gets forwarded to `upstreamDns`. Containers don't need to know
|
|
||||||
the upstream — they query the bridge IP and dnsmasq does the right
|
|
||||||
thing per-name.
|
|
||||||
|
|
||||||
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
|
||||||
listener only accepts queries from the bridge interface (plus lo for
|
|
||||||
container health-checks). External hosts can't reach it — no
|
|
||||||
DNS-amplification surface even when the operator opens port 80 for
|
|
||||||
gateway HTTP.
|
|
||||||
|
|
||||||
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
|
|
||||||
resolution stack — the host's resolver (systemd-resolved, plain
|
|
||||||
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
|
|
||||||
configured. The hive resolver is purely for inbound queries from
|
|
||||||
agent containers.
|
|
||||||
|
|
||||||
## Firewall posture
|
|
||||||
|
|
||||||
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 ]`
|
|
||||||
+ `allowedTCPPorts = [ 53 ]` opens the resolver on the bridge
|
|
||||||
interface only. Other interfaces stay closed. The hive resolver
|
|
||||||
isn't an external-facing service.
|
|
||||||
|
|
||||||
## Sequencing history
|
|
||||||
|
|
||||||
- mara on #805 (`comment-10957`): "can we put the resolver into the
|
|
||||||
gateway container?" — yes, this v1 does that.
|
|
||||||
- mara on #805 (`comment-11541`): "we need it before #14 so nothing
|
|
||||||
breaks when we switch over" — flipped the dependency direction;
|
|
||||||
v1 ships now, #14 flips containers later.
|
|
||||||
|
|
||||||
## Cross-references
|
|
||||||
|
|
||||||
- Issue [#805](http://localhost:3000/hyperhive/hyperhive/issues/805) — DNS resolver tracking
|
|
||||||
- Issue [#14](http://localhost:3000/hyperhive/hyperhive/issues/14) — netns isolation (downstream consumer)
|
|
||||||
- `docs/gateway.md` — vhost map + the gateway container's other duties
|
|
||||||
|
|
@ -25,7 +25,6 @@ in
|
||||||
./hive-forge.nix
|
./hive-forge.nix
|
||||||
./hive-gateway.nix
|
./hive-gateway.nix
|
||||||
./hive-matrix.nix
|
./hive-matrix.nix
|
||||||
./hive-network.nix
|
|
||||||
];
|
];
|
||||||
|
|
||||||
# Top-level hyperhive enable flag. When true, automatically enables
|
# Top-level hyperhive enable flag. When true, automatically enables
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,6 @@ let
|
||||||
hyperhiveDomain = config.services.hyperhive.domain;
|
hyperhiveDomain = config.services.hyperhive.domain;
|
||||||
matrixCfg = config.services.hyperhive.matrix;
|
matrixCfg = config.services.hyperhive.matrix;
|
||||||
forgeCfg = config.services.hyperhive.forge;
|
forgeCfg = config.services.hyperhive.forge;
|
||||||
networkCfg = config.services.hyperhive.network;
|
|
||||||
|
|
||||||
# Per-agent port table for `/agent/<name>/` routing. C0re writes
|
# Per-agent port table for `/agent/<name>/` routing. C0re writes
|
||||||
# this JSON on every topology change; gateway reads at deploy time.
|
# this JSON on every topology change; gateway reads at deploy time.
|
||||||
|
|
@ -552,63 +551,6 @@ in
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
# Hive-internal DNS resolver (#805 v1). Co-located in the
|
|
||||||
# gateway container per mara's call (#805:10957) — single
|
|
||||||
# front-door for both DNS and HTTP, saves a sibling
|
|
||||||
# container. Listens on the bridge interface from
|
|
||||||
# `services.hyperhive.network`; authoritative for the hive
|
|
||||||
# domain + sub-domains, forwards everything else upstream.
|
|
||||||
# No-op when `network.enable = false`.
|
|
||||||
services.dnsmasq = lib.mkIf networkCfg.enable {
|
|
||||||
enable = true;
|
|
||||||
# Don't substitute the container's /etc/resolv.conf —
|
|
||||||
# the gateway uses the host's resolver for its own
|
|
||||||
# outbound traffic; dnsmasq is purely for incoming
|
|
||||||
# queries from agent containers.
|
|
||||||
resolveLocalQueries = false;
|
|
||||||
settings = {
|
|
||||||
# Bind only on the bridge interface (and lo for
|
|
||||||
# health-checks). Outside hosts can't even see the
|
|
||||||
# listener.
|
|
||||||
interface = [
|
|
||||||
networkCfg.bridgeName
|
|
||||||
"lo"
|
|
||||||
];
|
|
||||||
bind-interfaces = true;
|
|
||||||
port = 53;
|
|
||||||
# Don't read /etc/resolv.conf — we control upstream
|
|
||||||
# explicitly to dodge dependency on the gateway
|
|
||||||
# container's own resolver state.
|
|
||||||
no-resolv = true;
|
|
||||||
server = networkCfg.upstreamDns;
|
|
||||||
# Hive authoritative records — answer queries for the
|
|
||||||
# hive domain + its sub-domains with the bridge IP
|
|
||||||
# (where nginx is reachable from container netns once
|
|
||||||
# #14 lands; today it's the host loopback alias and
|
|
||||||
# works in either shape).
|
|
||||||
#
|
|
||||||
# The forge / matrix entries are redundant in the
|
|
||||||
# common case where `forge.domain` /
|
|
||||||
# `matrix.gatewayHost` are sub-domains of
|
|
||||||
# `hyperhive.domain` — dnsmasq's `/<domain>/` rule
|
|
||||||
# already matches sub-domains (argus 🟡 on #845).
|
|
||||||
# Kept explicit because operators can override either
|
|
||||||
# to a cross-domain hostname (e.g.
|
|
||||||
# `forge.domain = "git.example.com"`); listing them
|
|
||||||
# explicitly keeps that case routed without needing
|
|
||||||
# an extra config block.
|
|
||||||
address = [
|
|
||||||
"/${hyperhiveDomain}/${networkCfg.bridgeIp}"
|
|
||||||
]
|
|
||||||
++ lib.optional (
|
|
||||||
(forgeCfg.enable or false) && (forgeCfg.behindGateway or false)
|
|
||||||
) "/${forgeCfg.domain}/${networkCfg.bridgeIp}"
|
|
||||||
++ lib.optional (
|
|
||||||
matrixCfg.enable && matrixCfg.gatewayHost != null
|
|
||||||
) "/${matrixCfg.gatewayHost}/${networkCfg.bridgeIp}";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,140 +0,0 @@
|
||||||
{
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
cfg = config.services.hyperhive.network;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
# Hive-internal network — host-side bridge + per-agent DNS resolver
|
|
||||||
# (#805, prereq for #14 netns isolation). Containers stay on shared
|
|
||||||
# host netns at v1 — this module just stands the bridge + resolver
|
|
||||||
# up so the endpoint is in place before #14 flips containers to
|
|
||||||
# private netns (mara on #805: "we need it before #14 so nothing
|
|
||||||
# breaks when we switch over"). Full design: docs/network.md.
|
|
||||||
|
|
||||||
options.services.hyperhive.network = {
|
|
||||||
enable = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
example = true;
|
|
||||||
description = ''
|
|
||||||
Stand up the hive-internal bridge + dnsmasq resolver.
|
|
||||||
Off by default while v1 phases in. When enabled:
|
|
||||||
a bridge interface (`bridgeName`) appears on the host with
|
|
||||||
`bridgeIp` assigned, and the hive-gateway container runs a
|
|
||||||
dnsmasq listening on that IP for `<hive-domain>` +
|
|
||||||
sub-domains. Agent containers still default to shared host
|
|
||||||
netns at v1 — the endpoint is up but only used once #14
|
|
||||||
lands and flips containers to a private netns + veth peer
|
|
||||||
on this bridge.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
bridgeName = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "hive-br0";
|
|
||||||
example = "h0";
|
|
||||||
description = ''
|
|
||||||
Name of the host-side bridge interface the hive uses for
|
|
||||||
inter-container traffic. Kept short so it survives the
|
|
||||||
IFNAMSIZ (15-char) cap, and prefixed so it's obviously
|
|
||||||
hive-managed in `ip link` output.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
bridgeIp = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "10.42.0.1";
|
|
||||||
example = "172.30.0.1";
|
|
||||||
description = ''
|
|
||||||
IPv4 address assigned to the bridge interface on the host
|
|
||||||
side. Becomes the DNS server address agents point at (and
|
|
||||||
the upstream the gateway proxies to once netns isolation
|
|
||||||
lands). Default `10.42.0.1` is in RFC 1918 space and
|
|
||||||
unlikely to clash with operator's existing setup; override
|
|
||||||
if a different range is already in use.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
bridgePrefixLength = lib.mkOption {
|
|
||||||
type = lib.types.int;
|
|
||||||
default = 24;
|
|
||||||
example = 16;
|
|
||||||
description = ''
|
|
||||||
Netmask prefix length for the bridge subnet. Default `/24`
|
|
||||||
gives 254 usable per-agent addresses, enough for any
|
|
||||||
single-host hive. Operator with a larger swarm or a tighter
|
|
||||||
addressing scheme overrides.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
upstreamDns = lib.mkOption {
|
|
||||||
type = lib.types.listOf lib.types.str;
|
|
||||||
default = [
|
|
||||||
"1.1.1.1"
|
|
||||||
"9.9.9.9"
|
|
||||||
];
|
|
||||||
example = [
|
|
||||||
"192.168.1.1"
|
|
||||||
"8.8.8.8"
|
|
||||||
];
|
|
||||||
description = ''
|
|
||||||
Upstream DNS servers dnsmasq forwards non-hive queries to.
|
|
||||||
Defaults to Cloudflare + Quad9. Override for operators on
|
|
||||||
private networks who need a specific resolver (corporate
|
|
||||||
DNS, pi-hole, etc.). The hive resolver itself stays
|
|
||||||
authoritative for `<hive-domain>` and its sub-domains
|
|
||||||
regardless of upstream choice.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
|
||||||
assertions = [
|
|
||||||
{
|
|
||||||
assertion = config.services.hyperhive.domain != null;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.network.enable = true requires
|
|
||||||
services.hyperhive.domain to be set — the resolver needs a
|
|
||||||
domain to be authoritative for. Either pin a hostname
|
|
||||||
(`services.hyperhive.domain = "example.com";`) or leave
|
|
||||||
`network.enable` at its default of false.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
{
|
|
||||||
assertion = config.services.hyperhive.gateway.enable;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.network.enable = true requires
|
|
||||||
services.hyperhive.gateway.enable = true — the dnsmasq
|
|
||||||
resolver runs inside the hive-gateway container (single
|
|
||||||
front-door for both DNS and HTTP). Enable the gateway or
|
|
||||||
leave `network.enable` at its default of false.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Bridge interface on the host. Empty interfaces list = purely
|
|
||||||
# virtual bridge (no slave NICs attached). Per-agent veth pairs
|
|
||||||
# will join this bridge once #14 lands; at v1 it stands alone.
|
|
||||||
networking.bridges.${cfg.bridgeName}.interfaces = [ ];
|
|
||||||
|
|
||||||
# Host-side IP assignment on the bridge. This is what dnsmasq
|
|
||||||
# (inside the gateway container, shared host netns) binds on.
|
|
||||||
networking.interfaces.${cfg.bridgeName}.ipv4.addresses = [
|
|
||||||
{
|
|
||||||
address = cfg.bridgeIp;
|
|
||||||
prefixLength = cfg.bridgePrefixLength;
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Open the resolver port in the host firewall for traffic from
|
|
||||||
# the bridge subnet only. Other interfaces stay closed —
|
|
||||||
# external DNS-amplification surface is not exposed.
|
|
||||||
networking.firewall.interfaces.${cfg.bridgeName} = {
|
|
||||||
allowedUDPPorts = [ 53 ];
|
|
||||||
allowedTCPPorts = [ 53 ];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
Loading…
Reference in a new issue