vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
hyperhive/docs/network.md
atlas 07852cabc1 feat(3088): move the gateway's nginx + dnsmasq onto the host 
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.

The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.

Four things made it more than a deletion, none of them visible in the nix
diff:

- The self-signed cert service also imports the hive CA leaf, so removing it
  with the container would have left nginx naming a missing cert file, which
  it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
  for the reason its doc gave, and `--machine=` was both transport and
  scope — so the unit name is now hard-coded in the helper as the
  containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.

Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.

Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.

Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
2026-08-11 18:01:03 +02:00

289 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# hive-network
Host-side bridge + per-agent private-netns isolation — always on
whenever hyperhive is enabled. Configured via
`services.hyperhive.network.*`.
> Isolation is the only mode — there is no shared-netns fallback. The
> former `services.hyperhive.network.enable`,
> `services.hyperhive.network.isolateContainers` and
> `services.hyperhive.network.upstreamDns` options were removed; a
> config that still sets one fails eval with a removal message.
## Network map
One picture of the whole hive. There are two planes: **infra
containers share the host netns** and bind host ports directly;
**compute containers (agents + CI) each get a private netns** behind
the bridge. The unix-socket control plane rides the VFS and is
untouched by any of it.
```
internet
│ uplink NIC — NAT MASQUERADE for the
│ bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge / matrix │
│ │
│ nginx :80/:443 [hive-gateway] │
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
│ tuwunel :8008 client API [hive-matrix] │
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
│ │
│ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
vb-h-<a> vb-h-<b> vb-hive-ci veth pairs
│ │ │
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
│ agent a │ │ agent b │ │ hive-ci │ each; eth0 leases
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
└─────────┘ └─────────┘ └─────────┘
```
| container | netns | IPv4 | listens / reached via |
| -------------- | ----------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end:
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
- **DNS** — agents query `bridgeIp:53`; hive zones are answered
authoritatively with the bridge IP, everything else forwards to the
host's resolvers (see *Resolver behaviour* below).
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
own dashboard name resolve to the bridge IP, land on nginx
`:80`/`:443`, and proxy to forgejo
`:3000`, tuwunel `:8008`, hive-c0re `127.0.0.1:7000`, or a per-agent
UI unix socket.
- **Internet egress** — agent default route points at the bridge IP;
the host forwards + masquerades out its uplink.
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
and reach each other's gateway/forge across it
([`docs/swarm/`](swarm/README.md)).
- **Control plane (no network)** — per-agent broker socket
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
sockets under `/run/hive-agent/` are unix domain sockets
bind-mounted through the VFS; private netns does not affect them.
## Container shape (where dnsmasq lives)
Co-located in the existing `hive-gateway` container — single
front-door for both DNS and HTTP, saves a sibling container, single
systemd-unit / state surface to monitor. The gateway shares host
netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces`
listener on `bridgeIp` is on the host's bridge interface.
## Configuration
```nix
{
services.hyperhive = {
enable = true;
hiveName = "pr1ma";
swarm.domain = "darkest.space";
swarm.hives.pr1ma = { }; # -> domain = pr1ma.darkest.space
# network.bridgeIp = "10.42.0.1"; # default
};
}
```
Requires `services.hyperhive.domain` to be set — the dnsmasq resolver
is authoritative for `<hive-domain>` and its sub-domains. You do not
write it: it is read from this hive's entry in the swarm directory
(`docs/swarm/README.md` § Hive identity config).
## Bridge addressing
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
RFC 1918 space, unlikely to clash with operator's existing setup;
override `bridgeIp` + `bridgePrefixLength` if a different range is
already in use. `/24` gives 254 usable per-agent addresses — enough
for any single-host hive; bigger swarms or tighter addressing
schemes pick their own.
## Resolver behaviour
dnsmasq is **authoritative** for the hive's own zones — answers
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>`
queries with the bridge IP (where nginx is reachable). Everything
else is forwarded to the host's own resolvers: dnsmasq runs on the host
and reads the host's `/etc/resolv.conf` directly. Containers don't need
to know the upstream — they query the bridge IP and dnsmasq does the
right thing per-name.
There is deliberately no fallback `server=`: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
### History: the resolv.conf sync, and why it is gone
Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway`
container and read *that* container's `/etc/resolv.conf` — a one-shot
copy nixos-container made at start. systemd-nspawn(1) is explicit that
nothing propagates into it after early init, because resolv.conf is
normally updated by rename rather than in place. So a host network change
(new router, new lease, laptop moving networks) stranded dnsmasq on a
resolver that no longer answered, and every non-hive lookup from every
agent hung until someone restarted the gateway.
A host-side `hive-gateway-resolv` path unit closed that gap: watch
`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload
dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard
and mid-rewrite-snapshot check — **all of it bridging two copies of one
file.** With one machine there is one file, and the whole unit is
deleted.
🔑 Worth keeping as a shape, not just a story: **the sync was not
complexity anyone chose. It was the cost of a boundary that bought
nothing here** — the gateway already ran with `privateNetwork = false`,
sharing the host's netns, so the container never provided network
isolation in the first place. When a workaround is that elaborate, the
question to ask is what the boundary is *for*.
(Two alternatives were considered at the time and both were worse than
the copy: a path unit *inside* the container never fired, because the
host replaces the file by rename and `IN_MOVED_TO` does not cross the
nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf`
would have pinned the *first* inode for the container's whole lifetime,
since openresolv writes a temp file and renames over the target.)
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for
container health-checks). External hosts can't reach it — no
DNS-amplification surface even when the operator opens port 80 for
gateway HTTP.
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
resolution stack — the host's resolver (systemd-resolved, plain
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
configured. The hive resolver is purely for inbound queries from
agent containers.
## Firewall posture
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
- Port 53 opens the resolver on the bridge interface only. Other
interfaces stay closed. The hive resolver isn't an external-facing
service.
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
receives DHCP via a regular UDP socket (it does not use a
netfilter-bypassing raw socket), so the hole is mandatory — without
it containers never get a lease and fall back to 169.254.x.x.
- Ports 80 and 443 let isolated agents reach nginx (gateway
container, shared host netns) for the forge sub-domain, per-agent
UI proxies, and any other HTTP services.
The **host** firewall is the only firewall. The shared-netns infra
containers (gateway, forge, matrix) set
`networking.firewall.enable = false`: a NixOS firewall inside a
shared-netns container runs against the *host* ruleset — at container
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the
bridge holes above plus the agents' NAT. Private-netns containers
(agents, hive-ci) may keep their own firewall — it is scoped to their
namespace.
### Reaching host services (`exposeHostPorts`)
By default agents can only reach the host on 80/443 (+53 DNS), so a
host-side service on another port — e.g. a dev OTEL collector for
`services.hyperhive.otel.endpoint` (see `docs/observability.md`) — is unreachable.
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
agent can connect to `<bridgeIp>:P` (point the collector endpoint at
`http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
This is **firewall-only**: the host service must bind an address
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
service bound to `127.0.0.1` only stays unreachable; rebind it to
`0.0.0.0`.
The port is reachable by **every** agent on the bridge subnet (like
DNS/gateway), so only expose services safe for any agent to reach.
## Container isolation
Each agent container runs in a private network namespace with a dedicated
veth pair attached to the bridge. The following table summarises what
the nix side sets up unconditionally:
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
`10.42.0.1/24`), **not** the canonical network address. The Rust side
must normalise (bitwise-AND with mask) before subnet membership checks or
address arithmetic.
### What the Rust side does
`hive-c0re` reads `HIVE_NETWORK_ISOLATION` and passes
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
and `HOST_BRIDGE=<bridgeName>` via `lifecycle::set_nspawn_flags` when
creating or updating containers. `LOCAL_ADDRESS` is left empty so the
container's dhcpcd acquires an address from the bridge dnsmasq pool
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies uniformly
to all containers — agents and service containers alike.
`HOST_ADDRESS` is the bridge gateway IP (the address part of
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
so a non-`.1` operator override still resolves to wherever the bridge
actually lives). It is **load-bearing**: nixos-container's container-side
network setup only installs a default route (`ip route add default via
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode the
host-side address/route setup is skipped, so writing it only affects the
container's default route — without it the container comes up with an IP
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
### How the isolated container gets its resolver
nixos-container copies the **host's** `/etc/resolv.conf` into the container
at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved,
or a LAN router) is unreachable from a private netns and isn't
authoritative for the hive's own zones, so it is replaced with the
bridge dnsmasq at boot. Because the copy happens on every start, a
declarative `environment.etc."resolv.conf"` would be clobbered — so the
wiring is runtime:
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
gateway IP) into each container's `/etc`.
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
`tea-login` so the resolver is correct before the first DNS lookup.
**Why isolation is safe**: all hive-c0re communication goes
through unix domain sockets (`/run/hive/mcp.sock` for agent requests,
`/run/hive/priv.sock` for privileged ops).
These are bind-mounted into containers via the nspawn conf. UDS paths
traverse the VFS, not the network stack, so `PRIVATE_NETWORK=1` does not
affect them.
The nix side also enables IP forwarding + NAT (agents reach the internet
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP.
## Cross-references
- `docs/gateway.md` — vhost map + the gateway's other duties
Reference in a new issue View git blame Copy permalink