f457bf3c15
Eighth batch of the ongoing write-good.Passive pass (hyperhive#4042):
read all 69 hits across network.md/snapshot-store.md/gateway.md in
context and rewrote 34 with a clearly nameable actor -- mostly
hive-c0re, dnsmasq, nginx, or a specific systemd unit/fn named right
there or a sentence or two earlier. snapshot-store.md's operator-facing
prose ("you can deploy it", "you must tell every hive") reuses this
doc's own established "you" address (already present two sentences
away in one case) rather than inventing a new register.
Left 35 alone -- the largest leave-alone count yet, dominated by two
shapes gateway.md is dense with: negative-capability/requirement
idioms ("no X is required/needed/exposed", "can't be added/wired",
"must not be exposed") and predicate-adjective state descriptions
("is misconfigured"/"is broken" in negative-contrast "not that X"
framing, "is privileged" matching the established "is trusted" bucket,
"is closed/internal" deployment-posture, "is encrypted and
unauthenticated" connection-state pair). Also: two "X was removed"
changelog-style facts with no actor clause (same family left alone in
the scheduler batch), a gerund nix-conditional pair ("X being enabled
and Y being set are assertions, not documentation"), and a passive
embedded inside an already-active sentence ("X controls whether Y is
opened" -- the real verb is already active).
One caught-and-reverted mid-edit mistake: an early attempt at the
"nginx is handed the leaf" fix landed on the wrong paragraph (an
unrelated CA-generation sentence two paragraphs up) and left a
nonsensical forward-reference behind. Caught by re-reading the diff
before running vale, not by the count -- same discipline as the
docs/process batch's original catch. Reverted cleanly and reapplied at
the actual target sentence.
One small sibling-consistency fix beyond pure voice-flipping: "The
gateway emits the following headers..." (rewritten) sat two paragraphs
above "a header is added alongside the other security headers" (still
passive) describing the same mechanism -- rewrote the second to match
("the gateway adds a header...") rather than leave one active and one
passive version of the identical fact sitting near each other.
Verified via vale before/after: 69 -> 35 write-good.Passive hits,
exactly the 35 left alone above; error count and other warning
categories unchanged. Re-read every changed line in full surrounding
context after editing before running the final vale check.
277 lines
15 KiB
Markdown
277 lines
15 KiB
Markdown
# hive-network
|
||
|
||
Host-side bridge + per-agent private-netns isolation — always on
|
||
whenever hyperhive is enabled. Configured via
|
||
`services.hyperhive.network.*`.
|
||
|
||
> Isolation is the only mode — there is no shared-netns fallback. The
|
||
> former `services.hyperhive.network.enable`,
|
||
> `services.hyperhive.network.isolateContainers` and
|
||
> `services.hyperhive.network.upstreamDns` options were removed; a
|
||
> config that still sets one fails eval with a removal message.
|
||
|
||
## Network map
|
||
|
||
One picture of the whole hive — two planes: **infra
|
||
containers share the host netns** and bind host ports directly;
|
||
**compute containers (agents + CI) each get a private netns** behind
|
||
the bridge. The unix-socket control plane rides the VFS and is
|
||
untouched by any of it.
|
||
|
||
```
|
||
internet
|
||
│ uplink NIC — NAT MASQUERADE for the
|
||
│ bridge subnet (10.42.0.0/24 default)
|
||
┌──────────────────────────┴─────────────────────────────────────────┐
|
||
│ host netns — the host itself plus gateway / forge / matrix │
|
||
│ │
|
||
│ nginx :80/:443 [hive-gateway] │
|
||
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
|
||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||
│ tuwunel :8008 client API [hive-matrix] │
|
||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
||
│ │
|
||
│ hive-br0 10.42.0.1/24 │
|
||
│ ┌──────────┼──────────────┐ │
|
||
└──────────────┼──────────┼──────────────┼───────────────────────────┘
|
||
vb-h-<a> vb-h-<b> vb-hive-ci veth pairs
|
||
│ │ │
|
||
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
|
||
│ agent a │ │ agent b │ │ hive-ci │ each; eth0 leases
|
||
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
|
||
└─────────┘ └─────────┘ └─────────┘
|
||
```
|
||
|
||
| container | netns | IPv4 | listens / reached via |
|
||
| -------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
||
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||
|
||
The flows, end to end:
|
||
|
||
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
|
||
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
|
||
- **DNS** — agents and the service containers query `bridgeIp:53`; dnsmasq
|
||
answers hive zones authoritatively with the bridge IP, everything else
|
||
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
|
||
container points its own `resolv.conf` there, and one that instead
|
||
inherits the host's resolves no swarm name at all — those records exist
|
||
only on the bridge.
|
||
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
|
||
own dashboard name resolve to the bridge IP, land on nginx
|
||
`:80`/`:443`, and proxy to forgejo
|
||
`:3000`, tuwunel `:8008`, hive-c0re `127.0.0.1:7000`, or a per-agent
|
||
UI unix socket.
|
||
- **Internet egress** — agent default route points at the bridge IP;
|
||
the host forwards + masquerades out its uplink.
|
||
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
|
||
and reach each other's gateway/forge across it
|
||
([`docs/swarm/`](../swarm/README.md)).
|
||
- **Control plane (no network)** — per-agent broker socket
|
||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
|
||
sockets under `/run/hive-agent/` are unix domain sockets
|
||
bind-mounted through the VFS; private netns doesn't affect them.
|
||
|
||
## Container shape (where dnsmasq lives)
|
||
|
||
Co-located in the existing `hive-gateway` container — single
|
||
front-door for both DNS and HTTP, saves a sibling container, single
|
||
systemd-unit / state surface to monitor. The gateway shares host
|
||
netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces`
|
||
listener on `bridgeIp` is on the host's bridge interface.
|
||
|
||
## Configuration
|
||
|
||
```nix
|
||
{
|
||
services.hyperhive = {
|
||
enable = true;
|
||
hiveName = "pr1ma";
|
||
swarm.domain = "darkest.space";
|
||
swarm.hives.pr1ma = { }; # -> domain = pr1ma.darkest.space
|
||
# network.bridgeIp = "10.42.0.1"; # default
|
||
};
|
||
}
|
||
```
|
||
|
||
You must set `services.hyperhive.domain` — the dnsmasq resolver
|
||
is authoritative for `<hive-domain>` and its sub-domains. You don't
|
||
write it: it's read from this hive's entry in the swarm directory
|
||
(`docs/swarm/README.md` § Hive identity config).
|
||
|
||
## Bridge addressing
|
||
|
||
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
|
||
RFC 1918 space, unlikely to clash with operator's existing setup;
|
||
override `bridgeIp` + `bridgePrefixLength` if a different range is
|
||
already in use. `/24` gives 254 usable per-agent addresses — enough
|
||
for any single-host hive; bigger swarms or tighter addressing
|
||
schemes pick their own.
|
||
|
||
## Resolver behaviour
|
||
|
||
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
|
||
plus whatever swarm-service names this host contributes via
|
||
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
|
||
(matrix) when this host runs those services, and `auth.<swarm-domain>`
|
||
when it runs authelia — answering each with the bridge IP (where nginx
|
||
is reachable). Note forge and matrix are swarm-domain names, not
|
||
sub-domains of `<hive-domain>`: a swarm runs one forge and one
|
||
homeserver, so their names belong to the swarm rather than to whichever
|
||
hive happens to host them. dnsmasq forwards everything else to the host's
|
||
own resolvers: it runs on the host and reads the host's
|
||
`/etc/resolv.conf` directly. Containers don't need to know the
|
||
upstream — they query the bridge IP and dnsmasq does the right thing
|
||
per-name.
|
||
|
||
No fallback `server=` exists, by design: dnsmasq queries all known
|
||
upstreams in parallel, so a hardcoded public resolver would take a share
|
||
of normal traffic, not just cover the gap.
|
||
|
||
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
|
||
directly, so a network change (new router, new lease, laptop moving
|
||
networks) reaches it the moment openresolv rewrites the file. Nothing
|
||
needs synchronising, and no unit watches for it.
|
||
|
||
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
||
listener only accepts queries from the bridge interface (plus lo for
|
||
container health-checks). External hosts can't reach it — no
|
||
DNS-amplification surface even when the operator opens port 80 for
|
||
gateway HTTP.
|
||
|
||
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
|
||
resolution stack — the host's resolver (systemd-resolved, plain
|
||
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
|
||
configured. The hive resolver is purely for inbound queries from
|
||
agent containers.
|
||
|
||
## Firewall posture
|
||
|
||
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
|
||
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
|
||
|
||
- Port 53 opens the resolver on the bridge interface only. Other
|
||
interfaces stay closed. The hive resolver isn't an external-facing
|
||
service.
|
||
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
|
||
receives DHCP via a regular UDP socket (it doesn't use a
|
||
netfilter-bypassing raw socket), so the hole is mandatory — without
|
||
it containers never get a lease and fall back to 169.254.x.x.
|
||
- Ports 80 and 443 let isolated agents reach nginx (gateway
|
||
container, shared host netns) for the forge sub-domain, per-agent
|
||
UI proxies, and any other HTTP services.
|
||
|
||
The **host** firewall is the only firewall. The shared-netns infra
|
||
containers (gateway, forge, matrix) set
|
||
`networking.firewall.enable = false`: a NixOS firewall inside a
|
||
shared-netns container runs against the _host_ ruleset — at container
|
||
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
|
||
from the container's (empty) port list, and deletes the host's
|
||
`nixos-nat-*` chains without recreating them, silently wiping the
|
||
bridge holes above plus the agents' NAT. Private-netns containers
|
||
(agents, hive-ci) may keep their own firewall — it's scoped to their
|
||
namespace.
|
||
|
||
### Reaching host services (`exposeHostPorts`)
|
||
|
||
By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||
host-side service on another port — for example a dev OTLP collector you want
|
||
agents to reach directly — is unreachable. (hyperhive's own telemetry
|
||
needs none of this: `otel.enable` opens its collector's port itself, and
|
||
`otel.endpoint` is the _upstream_, which no agent ever dials. See
|
||
`docs/scheduler/observability.md`.)
|
||
|
||
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
||
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
|
||
agent can connect to `<bridgeIp>:P` (point the collector endpoint at
|
||
`http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
|
||
|
||
This is **firewall-only**: the host service must bind an address
|
||
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
|
||
only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
|
||
service bound to `127.0.0.1` only stays unreachable; rebind it to
|
||
`0.0.0.0`.
|
||
|
||
The port is reachable by **every** agent on the bridge subnet (like
|
||
DNS/gateway), so only expose services safe for any agent to reach.
|
||
|
||
## Container isolation
|
||
|
||
Each agent container runs in a private network namespace with a dedicated
|
||
veth pair attached to the bridge. The following table summarises what
|
||
the nix side sets up unconditionally:
|
||
|
||
| effect | mechanism |
|
||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
|
||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
|
||
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
|
||
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
|
||
|
||
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
|
||
`10.42.0.1/24`), **not** the canonical network address. The Rust side
|
||
must normalise (bitwise-AND with mask) before subnet membership checks or
|
||
address arithmetic.
|
||
|
||
### What the Rust side does
|
||
|
||
`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
|
||
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
|
||
and `HOST_BRIDGE=<bridgeName>` via `lifecycle::set_nspawn_flags` when
|
||
creating or updating containers. hive-c0re validates both variables **once at
|
||
daemon startup**, not per container: they're process-global, so a
|
||
missing or malformed value is a misconfigured daemon rather than one bad
|
||
container, and failing at boot gives a single diagnostic instead of one
|
||
per agent. No non-isolated mode exists to fall back to. hive-c0re leaves `LOCAL_ADDRESS` empty so the
|
||
container's dhcpcd acquires an address from the bridge dnsmasq pool
|
||
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies uniformly
|
||
to all containers — agents and service containers alike.
|
||
|
||
`HOST_ADDRESS` is the bridge gateway IP (the address part of
|
||
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
|
||
so a non-`.1` operator override still resolves to wherever the bridge
|
||
actually lives). It's **load-bearing**: nixos-container's container-side
|
||
network setup only installs a default route (`ip route add default via
|
||
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode
|
||
nixos-container skips the host-side address/route setup, so writing it only affects the
|
||
container's default route — without it the container comes up with an IP
|
||
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
|
||
|
||
### How the isolated container gets its resolver
|
||
|
||
nixos-container copies the **host's** `/etc/resolv.conf` into the container
|
||
at every start. The host resolver (for example `127.0.0.53` from systemd-resolved,
|
||
or a LAN router) is unreachable from a private netns and isn't
|
||
authoritative for the hive's own zones, so it's replaced with the
|
||
bridge dnsmasq at boot. Because the copy happens on every start, it
|
||
would clobber a declarative `environment.etc."resolv.conf"` — so the
|
||
wiring is runtime:
|
||
|
||
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
|
||
gateway IP) into each container's `/etc`.
|
||
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
|
||
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
|
||
It's ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
|
||
`tea-login` so the resolver is correct before the first DNS lookup.
|
||
|
||
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
|
||
domain sockets bind-mounted into containers, not network listeners — see
|
||
the _Control plane (no network)_ bullet under [Network
|
||
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
|
||
that never touches the network stack.
|
||
|
||
The nix side also enables IP forwarding + NAT (agents reach the internet
|
||
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
|
||
against a compromised agent reaching the c0re dashboard HTTP at
|
||
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
|
||
loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||
|
||
## Cross-references
|
||
|
||
- `docs/networking/gateway.md` — vhost map + the gateway's other duties
|