Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0d416df1e8 | ||
|
|
d97659c06a |
4 changed files with 98 additions and 0 deletions
|
|
@ -10,6 +10,69 @@ whenever hyperhive is enabled. Configured via
|
|||
> `services.hyperhive.network.upstreamDns` options were removed; a
|
||||
> config that still sets one fails eval with a removal message.
|
||||
|
||||
## Network map
|
||||
|
||||
One picture of the whole hive. There are two planes: **infra
|
||||
containers share the host netns** and bind host ports directly;
|
||||
**compute containers (agents + CI) each get a private netns** behind
|
||||
the bridge. The unix-socket control plane rides the VFS and is
|
||||
untouched by any of it.
|
||||
|
||||
```
|
||||
internet
|
||||
│ uplink NIC — NAT MASQUERADE for the
|
||||
│ bridge subnet (10.42.0.0/24 default)
|
||||
┌──────────────────────────┴─────────────────────────────────────────┐
|
||||
│ host netns — the host itself plus gateway / forge / matrix │
|
||||
│ │
|
||||
│ nginx :80/:443 [hive-gateway] │
|
||||
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
|
||||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||||
│ tuwunel :8008 client API [hive-matrix] │
|
||||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
||||
│ │
|
||||
│ hive-br0 10.42.0.1/24 │
|
||||
│ ┌──────────┼──────────────┐ │
|
||||
└──────────────┼──────────┼──────────────┼───────────────────────────┘
|
||||
vb-h-<a> vb-h-<b> vb-hive-ci veth pairs
|
||||
│ │ │
|
||||
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
|
||||
│ agent a │ │ agent b │ │ hive-ci │ each; eth0 leases
|
||||
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
|
||||
└─────────┘ └─────────┘ └─────────┘
|
||||
```
|
||||
|
||||
| container | netns | IPv4 | listens / reached via |
|
||||
| -------------- | ----------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
|
||||
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<domain>` vhost |
|
||||
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
||||
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||||
|
||||
The flows, end to end:
|
||||
|
||||
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
|
||||
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
|
||||
- **DNS** — agents query `bridgeIp:53`; hive zones are answered
|
||||
authoritatively with the bridge IP, everything else forwards to the
|
||||
host's resolvers (see *Resolver behaviour* below).
|
||||
- **HTTP** — `forge.<domain>` / matrix / dashboard names all resolve
|
||||
to the bridge IP, land on nginx `:80`/`:443`, and proxy to forgejo
|
||||
`:3000`, tuwunel `:8008`, hive-c0re `127.0.0.1:7000`, or a per-agent
|
||||
UI unix socket.
|
||||
- **Internet egress** — agent default route points at the bridge IP;
|
||||
the host forwards + masquerades out its uplink.
|
||||
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
|
||||
and reach each other's gateway/forge across it
|
||||
([`docs/swarm.md`](swarm.md)).
|
||||
- **Control plane (no network)** — per-agent broker socket
|
||||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||||
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
|
||||
sockets under `/run/hive-agent/` are unix domain sockets
|
||||
bind-mounted through the VFS; private netns does not affect them.
|
||||
|
||||
## Container shape (where dnsmasq lives)
|
||||
|
||||
Co-located in the existing `hive-gateway` container — single
|
||||
|
|
@ -82,6 +145,17 @@ agent containers.
|
|||
container, shared host netns) for the forge sub-domain, per-agent
|
||||
UI proxies, and any other HTTP services.
|
||||
|
||||
The **host** firewall is the only firewall. The shared-netns infra
|
||||
containers (gateway, forge, matrix) set
|
||||
`networking.firewall.enable = false`: a NixOS firewall inside a
|
||||
shared-netns container runs against the *host* ruleset — at container
|
||||
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
|
||||
from the container's (empty) port list, and deletes the host's
|
||||
`nixos-nat-*` chains without recreating them, silently wiping the
|
||||
bridge holes above plus the agents' NAT. Private-netns containers
|
||||
(agents, hive-ci) may keep their own firewall — it is scoped to their
|
||||
namespace.
|
||||
|
||||
### Reaching host services (`exposeHostPorts`)
|
||||
|
||||
By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||||
|
|
|
|||
|
|
@ -327,6 +327,14 @@ in
|
|||
in
|
||||
{
|
||||
system.stateVersion = "25.11";
|
||||
|
||||
# Shared host netns: this container's own firewall.service
|
||||
# would rewrite the HOST ruleset (flush nixos-fw, drop the
|
||||
# host's nixos-nat-* chains) at every boot — killing the
|
||||
# bridge DHCP/DNS holes and agent NAT. The host firewall owns
|
||||
# all filtering; never run one in here.
|
||||
networking.firewall.enable = false;
|
||||
|
||||
services.forgejo = {
|
||||
enable = true;
|
||||
package = cfg.package;
|
||||
|
|
|
|||
|
|
@ -163,6 +163,15 @@ in
|
|||
{
|
||||
system.stateVersion = "26.05";
|
||||
|
||||
# This container shares the host netns, so its own
|
||||
# firewall.service would run against the HOST ruleset: flush
|
||||
# the nixos-fw chains, rebuild them from this container's
|
||||
# (empty) port list, and delete the host's nixos-nat-* chains
|
||||
# — wiping the bridge DHCP/DNS holes and the agents' NAT on
|
||||
# every container boot. The host firewall owns all filtering;
|
||||
# never run one in here.
|
||||
networking.firewall.enable = false;
|
||||
|
||||
# Keep the host-copied /etc/resolv.conf intact. nixos-container
|
||||
# copies the host's file in at every container start, but
|
||||
# resolvconf's host-tracking mode then regenerates it — to an
|
||||
|
|
|
|||
|
|
@ -341,6 +341,13 @@ in
|
|||
{
|
||||
system.stateVersion = "26.05";
|
||||
|
||||
# Shared host netns: this container's own firewall.service
|
||||
# would rewrite the HOST ruleset (flush nixos-fw, drop the
|
||||
# host's nixos-nat-* chains) at every boot — killing the
|
||||
# bridge DHCP/DNS holes and agent NAT. The host firewall owns
|
||||
# all filtering; never run one in here.
|
||||
networking.firewall.enable = false;
|
||||
|
||||
# Peer-hive root CAs (`swarm.peers.<domain>.caCert`) added to THIS
|
||||
# container's trust bundle so tuwunel validates *federation* TLS
|
||||
# from a self-signed peer hive (it checks the peer's federation
|
||||
|
|
|
|||
Loading…
Reference in a new issue