Watch

atlas 7e2f4ce7c8 always include a hive's own allowedDomains in the rendered swarm-wide list

2026-10-29 08:25:22 +01:00

30 KiB

Raw Blame History

hive-network

The bridge every container on a host attaches to, and the private-netns isolation of agent and CI containers behind it. It comes up on any host that runs agents or swarm services: services.hyperhive.network.enable defaults to off, and the modules that need the bridge set it with mkDefault true — the hive controller, CI, the hive collector, and the gateway's resolver, which every swarm service turns on. Configured via services.hyperhive.network.*.

Agent containers always run in a private netns.

Network map

One picture of the whole hive — two planes: infra containers share the host netns and bind host ports directly; compute containers (agents + CI) each get a private netns behind the bridge. The unix-socket control plane rides the VFS, and none of that touches it.

                        internet
                           │  uplink NIC — NAT MASQUERADE for the
                           │  bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge                  │
│                                                                    │
│   nginx :80/:443                                  [hive-gateway]   │
│   dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP)         [hive-gateway]   │
│   squid 10.42.0.1:3128 (egress proxy)             [hive-squid]     │
│   forgejo :3000 http, :2222 git-ssh               [hive-forge]     │
│   hive-c0re dashboard 127.0.0.1:7000              (host service)   │
│   wg-hive :51820/udp — swarm mesh, when enabled   (host iface)     │
│                                                                    │
│                  hive-br0   10.42.0.1/24                           │
│              ┌──────────┼──────────────┐                           │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
          vb-h-<a>    vb-h-<b>      vb-<runner>     veth pairs
               │          │              │
          ┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐   one private netns
          │ agent a │ │ agent b │ │ runner  │   each; eth0 leases
          │  eth0   │ │  eth0   │ │  eth0   │   from the DHCP pool
          └─────────┘ └─────────┘ └─────────┘

container	netns	IPv4	listens / reached via
gateway (host)	host	host addresses	nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge
egress proxy (host)	host	`bridgeIp`	squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it
`hive-forge`	host (shared)	host addresses	forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost
`hive-matrix`	private, veth on bridge	DHCP pool	tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers
`swarm-ci-runner`	private, veth on bridge	DHCP pool	outbound only (runner → forge); no inbound surface
`swarm-review-classifier`	private, veth on bridge	DHCP pool	outbound only (classifier → queue, secret store, swarm collector); no inbound surface
`h-<agent>`	private, veth on bridge	DHCP pool	web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999

The flows, end to end:

DHCP — agent dhcpcd broadcasts on eth0 → veth → bridge → host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
DNS — agents and the service containers query bridgeIp:53; dnsmasq answers hive zones authoritatively with the bridge IP, everything else forwards to the host's resolvers (see Resolver behaviour below). Each container points its own resolv.conf there, and one that instead inherits the host's resolves no swarm name at all — those records exist only on the bridge.
HTTP — forge. and chat. (under swarm.domain) plus the hive's own dashboard name resolve to the bridge IP, land on nginx :80/:443, and proxy to forgejo :3000, tuwunel's unix socket, hive-c0re 127.0.0.1:7000, or a per-agent UI unix socket.
Internet egress — agent default route points at the bridge IP; the host forwards + masquerades out its uplink.
Swarm — peer hives connect over the wg-hive WireGuard mesh and reach each other's gateway/forge across it (docs/swarm/).
Control plane (no network) — per-agent broker socket /run/hive/mcp.sock, privileged helper /run/hive/priv.sock, operator admin /run/hyperhive/host.sock, and the per-agent UI sockets under /run/hive-agent/ are unix domain sockets bind-mounted through the VFS; private netns doesn't affect them.

Container shape (where dnsmasq lives)

dnsmasq runs on the host itself, next to nginx — both come from the gateway module, one front door for DNS and HTTP. Its bind-interfaces listener sits on bridgeIp, on the host's bridge interface.

Configuration

{
  services.hyperhive = {
    deploy.hive-controller.enable = true;
    deploy.hive-controller.hiveName = "pr1ma";
    swarm.domain = "darkest.space";
    swarm.hives.pr1ma = { };          # -> domain = pr1ma.darkest.space
    # network.bridgeIp = "10.42.0.1"; # default
  };
}

The hive domain (services.hyperhive.domain, which the resolver answers for) comes from this hive's entry in the swarm directory: swarm.hives.<hiveName>.domain, default <hiveName>.<swarm.domain>. Eval fails until swarm.domain and that entry exist (swarm/README.md § Hive identity config).

Bridge addressing

Default subnet is 10.42.0.0/24, host-side gateway at 10.42.0.1. RFC 1918 space, unlikely to clash with operator's existing setup; override bridgeIp + bridgePrefixLength if a different range is already in use. /24 gives 254 usable per-agent addresses — enough for any single-host hive; bigger swarms or tighter addressing schemes pick their own.

Resolver behaviour

dnsmasq is authoritative for the hive's own zone (<hive-domain>) plus whatever swarm-service names this host contributes via gateway.localNames — forge.<swarm-domain> and chat.<swarm-domain> (matrix) when this host runs those services, auth.<swarm-domain> when it runs authelia, and hive-collector.hive on every hive — answering each with the bridge IP (where nginx is reachable, and where the hive's own collector listens). Note forge and matrix are swarm-domain names, not sub-domains of <hive-domain>: a swarm runs one forge and one homeserver, so their names belong to the swarm rather than to whichever hive happens to host them. dnsmasq forwards everything else to the host's own resolvers: it runs on the host and reads the host's /etc/resolv.conf directly. Containers don't need to know the upstream — they query the bridge IP and dnsmasq does the right thing per-name.

No fallback server= exists, by design: dnsmasq queries all known upstreams in parallel, so a hardcoded public resolver would take a share of normal traffic, not just cover the gap.

dnsmasq runs on the host and reads the host's /etc/resolv.conf directly, so a network change (new router, new lease, laptop moving networks) reaches it the moment openresolv rewrites the file. Nothing needs synchronising, and no unit watches for it.

bind-interfaces + interface = [ bridgeName "lo" ] means the listener only accepts queries from the bridge interface (plus lo for container health-checks). External hosts can't reach it — no DNS-amplification surface even when the operator opens port 80 for gateway HTTP.

resolveLocalQueries = false keeps dnsmasq out of the host's own resolution stack — the host's resolver (systemd-resolved, plain glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator configured. The hive resolver is purely for inbound queries from agent containers.

Firewall posture

networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ] networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]

Port 53 opens the resolver on the bridge interface only. Other interfaces stay closed. The hive resolver isn't an external-facing service.
Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq receives DHCP via a regular UDP socket (it doesn't use a netfilter-bypassing raw socket), so the hole is mandatory — without it containers never get a lease and fall back to 169.254.x.x.
Ports 80 and 443 let isolated agents reach nginx (on the host) for the forge sub-domain, per-agent UI proxies, and any other HTTP services.

The host firewall is the only firewall. The swarm service containers that share the host netns (forge, bao, the metrics and log stores) run with networking.firewall.enable = false (nix/container-modules/swarm-container.nix): a NixOS firewall inside a shared-netns container runs against the host ruleset — at container boot its firewall-start flushes the nixos-fw chains, rebuilds them from the container's (empty) port list, and deletes the host's nixos-nat-* chains without recreating them, silently wiping the bridge holes above plus the agents' NAT. Private-netns containers (every container with privateNetwork = true: agents and the service containers on the bridge) keep their own firewall — it's scoped to their namespace.

Reaching host services (`exposeHostPorts`)

By default agents can only reach the host on 80/443 (+53 DNS), so a host-side service on another port — for example a dev OTLP collector you want agents to reach directly — is unreachable. (hyperhive's own services need none of this: a module that means to be reachable from agents opens its own port here, which the hive collector and nats.enable both do. Note that deploy.swarm-otel.upstream.endpoint is the upstream, which no agent ever dials. See docs/scheduler/observability.md.)

services.hyperhive.network.exposeHostPorts = [ 4318 ]; opens each listed TCP port P on the bridge-interface allowedTCPPorts, so an agent can connect to <bridgeIp>:P (for a dev OTLP collector, point its endpoint at http://<bridgeIp>:4318, default http://10.42.0.1:4318). The hive's own collector needs none of this: agents dial it by otel.collector.host.

This is firewall-only: the host service must bind an address reachable from the bridge — 0.0.0.0 or the bridge IP — not loopback only. The bridge→127.0.0.0/8 DROP rule (below) is unchanged, so a service bound to 127.0.0.1 only stays unreachable; rebind it to 0.0.0.0.

The port is reachable by every agent on the bridge subnet (like DNS/gateway), so only expose services safe for any agent to reach.

Egress proxy (squid)

Every hive host runs squid in the host netns, listening on <bridgeIp>:3128 (nix/host-modules/hive-squid.nix). The port is open on the bridge interface only. An agent sends its outbound HTTP and HTTPS through it once its hive says the proxy is ready and it has fetched its proxy credential (see Agents and the proxy below). Any other agent goes straight out through the NAT, as before, and so does any program that ignores the proxy variables: nothing blocks direct egress yet.

Squid filters on the destination hostname: a CONNECT target or a plain-HTTP host. It never decrypts TLS, so it holds no signing key. It caches nothing.

Credentials. A request for a listed name must carry a per-agent proxy credential (HTTP Basic, the agent's name and its secret). swarm-controller mints one per agent at swarm/agents/<agent>/proxy-credential, and never replaces it. hive-c0re hashes the credentials of this hive's agents, and one more for the self-check, into services.hyperhive.squid.usersFile every 30 seconds. Squid's basic_ncsa_auth helper re-reads the file when it changes. A stopped agent stays in the file. Squid caches a good login for a minute. Nothing rotates a credential: to replace one, delete its leaf in the store and the next pass mints a new one. HTTP Basic crosses the bridge in clear between an agent and the proxy. See the credentials reference. A request without a valid credential gets a 407, once the destination checks below have passed it.
Agents and the proxy. An agent uses the proxy only when its hive says this host's squid is ready for it. hive-c0re puts two facts in every container's /etc/hyperhive-hive-facts, only on a hive that holds a store identity: HIVE_SQUID_PROXY (services.hyperhive.squid.listen, the bridge IP and port) and HIVE_EGRESS_NO_PROXY (the direct hosts below). A hive that predates them writes neither, and its agents go direct. hive-agent-proxy-credential.service (nix/agent-modules/egress-proxy.nix) runs in each agent's container, as the agent user, at start and every 10 minutes. With both facts present it logs in to the secret store with the agent's own certificate, reads swarm/agents/<agent>/proxy-credential and writes /run/hive-agent-proxy-credential/proxy.env (0600) with HTTPS_PROXY, HTTP_PROXY, their lower-case forms, NODE_USE_ENV_PROXY=1 and NO_PROXY/no_proxy. The proxy URL is http://<agent>:<credential>@<bridgeIp>:3128; Claude Code, Node, curl, git and reqwest send the userinfo as Proxy-Authorization. The harness, its bash tool, the subagents, the MCP servers and the pollers load the file with EnvironmentFile=- when they start. Until the facts and the credential both exist nothing writes the file, so those units have no proxy variable and go direct; a failed read keeps the file already in place. Nothing restarts on deploy: an agent takes the proxy at its next start, which is its next rebuild or restart. The credential is in the environment of every process those units start, so env in the agent's shell prints it. Login shells (nixos-container root-login, SSH) don't load the file.
Direct destinations. HIVE_EGRESS_NO_PROXY is the hosts this agent must reach without the proxy, and NO_PROXY is localhost,127.0.0.1,::1 followed by that list. hive-c0re computes it from the host's options: the bridge IP, the hive's domain (services.hyperhive.domain), the secret store and queue names (services.hyperhive.swarm.bao.domain, swarm.nats.domain), services.hyperhive.swarm.serviceDomains (the swarm UI and controller apex, the forge, matrix, attic, logs, metrics and otel names) and every name the gateway answers with the bridge IP (services.hyperhive.gateway.localNames). Squid refuses every destination that resolves to the bridge, and the hive's resolver answers the bridge address for those names, so they never go through the proxy. A bare name matches itself and its subdomains. Node's NO_PROXY has no CIDR support, so the list holds the bridge as an address.
Two lists. The credential decides the policy. hive-c0re writes a second file beside the users file, services.hyperhive.squid.unrestrictedFile (/var/lib/hive-squid/unrestricted, hive-core:squid, 0640): the names of the users whose agent holds the unrestricted_internet flag in the permissions the hive last read from the swarm, one per line, rewritten when it changes. The default is permissive: an agent with no record, or a record that never set the flag, holds it, the same default as the hyperhive.agent.unrestricted_internet metric. While the hive's read of the permissions is stale or failing, a new agent that already holds a credential counts as unrestricted. Squid reads the list when it parses its configuration, so squid-unrestricted-reload.path reloads squid when the file changes. A missing or empty file lists no one, and squid starts either way. A flag change reaches squid after the hive's next read of the permissions and its next 30-second pass. An older hive-c0re writes no list, so every credential on that hive is restricted. Agents must not start sending credentials until every hive runs the hive-c0re that writes it, or a legacy agent loses general internet.
Base list. services.hyperhive.squid.baseAllowedDomains is open to every authenticated user, whatever its flag: what the agent harness needs off the swarm. The default is the hosts Claude Code requires (https://code.claude.com/docs/en/network-config): api.anthropic.com for the model API and the usage probe (hive-agent/src/claude_usage_watch.rs), claude.ai for the account login, and platform.claude.com for the OAuth token exchange and refresh. An agent whose ANTHROPIC_BASE_URL points elsewhere needs no entry here: the swarm controller adds that host to the agent's own swarm entries (see Swarm allow-list below). The list omits the plugin marketplace clone from github.com, since a failed add isn't fatal. The rules below still refuse an entry that resolves to a private address.
Allow-list. services.hyperhive.squid.allowedDomains defaults to empty, plus the self-check canary (services.hyperhive.squid.selfcheck.canary), so the canary widens the restricted list by that one name. The swarm's own service names aren't on it: they resolve to the bridge or a private address, which the rules below refuse, and agents reach them directly. Each entry matches that exact name, or with a leading dot the name and its subdomains. A user on the unrestricted list reaches any other name too, on ports 80 and 443. Everyone else gets a 403 for it. allowedDomains is the hive's own list, always allowed whether or not the swarm has a list of its own (next bullet).
Swarm allow-list. The operator adds domains in the swarm UI (allowed domains); the swarm controller keeps them in one value of the egress-allow bucket, each for every agent or for one agent. Every 30 seconds hive-c0re reads it and writes two files beside the users file, services.hyperhive.squid.allowedFile (/var/lib/hive-squid/allowed) and services.hyperhive.squid.agentAllowFile (/var/lib/hive-squid/agent-allow.conf), hive-core:squid, 0640, and logs squid allow list: 2 swarm, 3 agent entries when they change. The swarm-wide names join the allow ACL, so a restricted user reaches them. A per-agent entry is a rule for that agent's squid user, http_access allow <user> <names>, in the generated agent-allow.conf that the configuration includes at the point where the base list gets its allow: after every destination deny and deny !authed, so the rules under Denied before any allow still bind it, and before the unrestricted rule. Only this hive's agents get a rule. squid-unrestricted-reload.path reloads squid when either file changes, so there is no deploy or restart. A name is a DNS name of at least two labels. One leading dot (.example.com) also allows every subdomain, and is the only wildcard; the controller refuses IP addresses, other wildcards, ports, paths and the base list's names when you add the entry, and hive-c0re checks each name again and skips one that fails, or that the base list or the canary covers, or that another entry of the same list covers. A name that resolves to a private address stays a 403. The controller also adds entries itself. When an operator saves an agent's API-key environment, and once when the controller starts, it takes the host from ANTHROPIC_BASE_URL and from every variable named *_BASE_URL (the address an ACP agent that loads the same file reads), and keeps one derived entry for that agent per host. Each host passes the same check as an operator's entry, so an IP address, localhost and a base-list name get none. A save or a delete of the environment replaces the agent's derived entries, and destroying the agent drops all its entries. The controller reads only the store's copy: an agent on the hand-placed backendEnvironmentFile alone, and the opencode preset's acp.opencode.provider.baseUrl, get no derived entry, so add one in the swarm UI. A hive-c0re that predates the field renders a derived entry like any other. The swarm's list adds to the hive's allowedDomains: allowedFile holds those names plus the swarm-wide entries, so removing an entry never removes a name the hive lists itself. Until the controller has a list (it predates the feature, or nobody added an entry) allowedFile holds allowedDomains and agent-allow.conf is empty. A hive-c0re older than this rule replaces allowedDomains with the swarm's list once the controller has one, so every hive must run the new hive-c0re before the controller's first entry exists. A queue that stops answering for the bucket leaves a file the swarm wrote untouched, and a failed read leaves both files untouched. The first line of a file written from the swarm's list is a # source: swarm comment. The queue grant a hive needs to read the list comes with the controller's swarm-nats-auth; a hive that can't read it times out, keeps its files and warns after about 90 seconds.
Unlisted names. For a user who isn't on the unrestricted list, refused after the destination checks, which look the name up on the host.
Denied before any allow. Destinations that resolve to loopback, link-local, the bridge subnet, a private range (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7), shared address space (100.64.0.0/10), 0.0.0.0/8, ::, multicast or reserved space, and IP-literal hosts, for every user including the unrestricted ones and for listed names. Squid resolves names on the host and connects from it, so without these a name pointing at 127.0.0.1, at the host's own LAN address or at a private neighbour would walk around the bridge→loopback DROP rule and the swarm's network. Squid checks the address it connects to, so a name that changes its answer between a check and the connect gains nothing. A listed name must therefore resolve to a public address on the host's resolver: a split-horizon name that answers a LAN address gets a 403, and so does the self-check canary, which is why it isn't a swarm service name. Only CONNECT to port 443 and plain HTTP to ports 80 and 443 pass.
Metrics. squid-exporter serves prometheus-squid-exporter on 127.0.0.1:9301. The hive collector scrapes it as job squid through services.hyperhive.otel.scrapeTargets. squid_up and squid_client_http_requests_total are the two to watch.
Logs. One syslog line per request reaches the journal as squid.service, and services.hyperhive.otel.journalUnits forwards that unit and the self-check's to the swarm's log store. A refusal shows as TCP_DENIED/403 with the client's source address.
Self-check. squid-selfcheck.timer runs squid-selfcheck.service every minute from the bridge IP, logging in as selfcheck.squid: CONNECT to services.hyperhive.squid.selfcheck.canary (default: cache.nixos.org, a public name that's also on the allow-list) must answer 200, to deny-canary.invalid 403, and to the canary without a credential 407. It then logs in as selfcheck-unrestricted.squid, which has the same secret and is always on the unrestricted list, and a CONNECT to services.hyperhive.squid.selfcheck.unrestrictedCanary (default: example.com, a name outside the allow-list) must answer 200. It writes one squid-selfcheck ok=… allow_status=… deny_status=… anon_status=… credential=present|absent unrestricted_canary=… unrestricted_status=… line per run and fails the unit when any is wrong. The password is the hive's swarm/hives/<hive>/squid-selfcheck secret, which squid-selfcheck-credential lands at /run/credstore/hive-squid.selfcheck. Before swarm-controller has minted it, or while hive-c0re hasn't written the users file, the line reads credential=absent or allow_status=407 and ok=false, and nothing else breaks. The host's resolver looks up the canary, so a host whose resolver maps that name to a loopback, bridge or private address fails the check.

Container isolation

Each agent container runs in a private network namespace with a dedicated veth pair attached to the bridge. The following table summarises what the nix side sets up unconditionally:

effect	mechanism
IP forwarding	`boot.kernel.sysctl."net.ipv4.ip_forward" = 1`
Internet NAT	`networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC
Loopback DROP	`networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks
Gateway access	`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host
c0re signal	`HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both required; `hive-c0re` refuses to start without them

HIVE_NETWORK_SUBNET is the host-side bridge IP + prefix (e.g. 10.42.0.1/24), not the canonical network address. The Rust side must normalise (bitwise-AND with mask) before subnet membership checks or address arithmetic.

What the Rust side does

hive-c0re reads HIVE_NETWORK_BRIDGE + HIVE_NETWORK_SUBNET and passes PRIVATE_NETWORK=1, LOCAL_ADDRESS= (empty), HOST_ADDRESS=<bridge-ip>, and HOST_BRIDGE=<bridgeName> when creating or updating containers: lifecycle::set_nspawn_flags hands them to hive-priv, which rewrites /etc/nixos-containers/<container>.conf. hive-c0re validates both variables once at daemon startup, not per container: they're process-global, so a missing or malformed value is a misconfigured daemon rather than one bad container, and failing at boot gives a single diagnostic instead of one per agent. No non-isolated mode exists to fall back to. hive-c0re leaves LOCAL_ADDRESS empty so the container's dhcpcd acquires an address from the bridge dnsmasq pool (networking.useDHCP = true in nix/agent-modules/network.nix). This applies to every agent container, manager included.

HOST_ADDRESS is the bridge gateway IP (the address part of HIVE_NETWORK_SUBNET, via lifecycle::bridge_gateway_ip — taken verbatim so a non-.1 operator override still resolves to wherever the bridge actually lives). It's load-bearing: nixos-container's container-side network setup only installs a default route (ip route add default via $HOST_ADDRESS) when HOST_ADDRESS is non-empty. In bridge mode nixos-container skips the host-side address/route setup, so writing it only affects the container's default route — without it the container comes up with an IP but no path off the bridge subnet (no internet, no api.anthropic.com).

How the isolated container gets its resolver

nixos-container copies the host's /etc/resolv.conf into the container at every start. The host resolver (for example 127.0.0.53 from systemd-resolved, or a LAN router) is unreachable from a private netns and isn't authoritative for the hive's own zones, so it's replaced with the bridge dnsmasq at boot. Because the copy happens on every start, it would clobber a declarative environment.etc."resolv.conf" — so the wiring is runtime:

hive-priv drops a marker file (/etc/hyperhive-bridge-dns, carrying the gateway IP) into each container's /etc.
the hyperhive-isolated-dns oneshot (nix/agent-modules/network.nix), gated on that marker, rewrites /etc/resolv.conf to nameserver <gateway-ip> at boot. It's ordered before the harness (hive-ag3nt), the matrix daemon, and the forge-token fetch so the resolver is correct before the first DNS lookup.

Why isolation is safe: hive-c0re's control-plane sockets are unix domain sockets bind-mounted into containers, not network listeners — see the Control plane (no network) bullet under Network map above. PRIVATE_NETWORK=1 has no effect on a path that never touches the network stack.

The nix side also enables IP forwarding + NAT (agents reach the internet through the host) and drops bridge-subnet → loopback traffic (defence-in-depth against a compromised agent reaching the c0re dashboard HTTP at 127.0.0.1). Agents have no legitimate reason to reach the dashboard over loopback — the hive-c0re admin socket is a UDS, not TCP.

Cross-references

docs/networking/gateway.md — vhost map + the gateway's other duties

30 KiB Raw Blame History Unescape Escape