vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity
hyperhive/docs/networking/network.md

515 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# hive-network
The bridge every container on a host attaches to, and the private-netns
isolation of agent and CI containers behind it. It comes up on any host that runs agents or swarm services:
`services.hyperhive.network.enable` defaults to off, and the modules that
need the bridge set it with `mkDefault true` — the hive controller, CI,
the hive collector, and the gateway's resolver, which every swarm service
turns on. Configured via `services.hyperhive.network.*`.
Agent containers always run in a private netns.
## Network map
One picture of the whole hive — two planes: **infra
containers share the host netns** and bind host ports directly;
**compute containers (agents + CI) each get a private netns** behind
the bridge. The unix-socket control plane rides the VFS, and none of
that touches it.
```
internet
│ uplink NIC — NAT MASQUERADE for the
│ bridge subnet (10.42.0.0/24 default)
┌──────────────────────────┴─────────────────────────────────────────┐
│ host netns — the host itself plus gateway / forge │
│ │
│ nginx :80/:443 [hive-gateway] │
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
│ │
│ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │
└──────────────┼──────────┼──────────────┼───────────────────────────┘
vb-h-<a> vb-h-<b> vb-<runner> veth pairs
│ │ │
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
│ agent a │ │ agent b │ │ runner │ each; eth0 leases
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
└─────────┘ └─────────┘ └─────────┘
```
| container | netns | IPv4 | listens / reached via |
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `swarm-review-classifier` | private, veth on bridge | DHCP pool | outbound only (classifier → queue, secret store, swarm collector); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end:
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
- **DNS** — agents and the service containers query `bridgeIp:53`; dnsmasq
answers hive zones authoritatively with the bridge IP, everything else
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
container points its own `resolv.conf` there, and one that instead
inherits the host's resolves no swarm name at all — those records exist
only on the bridge.
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
own dashboard name resolve to the bridge IP, land on nginx
`:80`/`:443`, and proxy to forgejo
`:3000`, tuwunel's unix socket, hive-c0re `127.0.0.1:7000`, or a per-agent
UI unix socket.
- **Internet egress** — agent default route points at the bridge IP;
the host forwards + masquerades out its uplink.
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
and reach each other's gateway/forge across it
([`docs/swarm/`](../swarm/README.md)).
- **Control plane (no network)** — per-agent broker socket
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
sockets under `/run/hive-agent/` are unix domain sockets
bind-mounted through the VFS; private netns doesn't affect them.
## Container shape (where dnsmasq lives)
dnsmasq runs on the host itself, next to nginx — both come from the
gateway module, one front door for DNS and HTTP. Its `bind-interfaces`
listener sits on `bridgeIp`, on the host's bridge interface.
## Configuration
```nix
{
services.hyperhive = {
deploy.hive-controller.enable = true;
deploy.hive-controller.hiveName = "pr1ma";
swarm.domain = "darkest.space";
swarm.hives.pr1ma = { }; # -> domain = pr1ma.darkest.space
# network.bridgeIp = "10.42.0.1"; # default
};
}
```
The hive domain (`services.hyperhive.domain`, which the resolver answers
for) comes from this hive's entry in the swarm directory:
`swarm.hives.<hiveName>.domain`, default `<hiveName>.<swarm.domain>`.
Eval fails until `swarm.domain` and that entry exist
([`swarm/README.md`](../swarm/README.md) § Hive identity config).
## Bridge addressing
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
RFC 1918 space, unlikely to clash with operator's existing setup;
override `bridgeIp` + `bridgePrefixLength` if a different range is
already in use. `/24` gives 254 usable per-agent addresses — enough
for any single-host hive; bigger swarms or tighter addressing
schemes pick their own.
## Resolver behaviour
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
plus whatever swarm-service names this host contributes via
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
(matrix) when this host runs those services, `auth.<swarm-domain>`
when it runs authelia, and `hive-collector.hive` on every hive
— answering each with the bridge IP (where nginx
is reachable, and where the hive's own collector listens). Note forge and
matrix are swarm-domain names, not sub-domains of `<hive-domain>`: a swarm
runs one forge and one homeserver, so their names belong to the swarm rather
than to whichever hive happens to host them. dnsmasq forwards everything else to the host's
own resolvers: it runs on the host and reads the host's
`/etc/resolv.conf` directly. Containers don't need to know the
upstream — they query the bridge IP and dnsmasq does the right thing
per-name.
No fallback `server=` exists, by design: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
directly, so a network change (new router, new lease, laptop moving
networks) reaches it the moment openresolv rewrites the file. Nothing
needs synchronising, and no unit watches for it.
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for
container health-checks). External hosts can't reach it — no
DNS-amplification surface even when the operator opens port 80 for
gateway HTTP.
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
resolution stack — the host's resolver (systemd-resolved, plain
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
configured. The hive resolver is purely for inbound queries from
agent containers.
## Firewall posture
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
- Port 53 opens the resolver on the bridge interface only. Other
interfaces stay closed. The hive resolver isn't an external-facing
service.
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
receives DHCP via a regular UDP socket (it doesn't use a
netfilter-bypassing raw socket), so the hole is mandatory — without
it containers never get a lease and fall back to 169.254.x.x.
- Ports 80 and 443 let isolated agents reach nginx (on the host) for the forge sub-domain, per-agent
UI proxies, and any other HTTP services.
The **host** firewall is the only firewall. The swarm service
containers that share the host netns (forge, bao, the
metrics and log stores) run with `networking.firewall.enable = false`
(`nix/container-modules/swarm-container.nix`): a NixOS firewall inside a
shared-netns container runs against the _host_ ruleset — at container
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the
bridge holes above plus the agents' NAT. Private-netns containers
(every container with `privateNetwork = true`: agents and the service containers on the bridge) keep their own firewall — it's scoped to their
namespace.
### Reaching host services (`exposeHostPorts`)
By default agents can only reach the host on 80/443 (+53 DNS), so a
host-side service on another port — for example a dev OTLP collector you want
agents to reach directly — is unreachable. (hyperhive's own services
need none of this: a module that means to be reachable from agents opens
its own port here, which the hive collector and `nats.enable` both do. Note
that `deploy.swarm-otel.upstream.endpoint` is the _upstream_, which no agent ever dials. See
`docs/scheduler/observability.md`.)
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
agent can connect to `<bridgeIp>:P` (for a dev OTLP collector, point its
endpoint at `http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
The hive's own collector needs none of this: agents dial it by
`otel.collector.host`.
<!-- vale write-good.Passive = NO -->
This is **firewall-only**: the host service must bind an address
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
service bound to `127.0.0.1` only stays unreachable; rebind it to
`0.0.0.0`.
<!-- vale write-good.Passive = YES -->
The port is reachable by **every** agent on the bridge subnet (like
DNS/gateway), so only expose services safe for any agent to reach.
## Egress proxy (squid)
Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
(`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
only. An agent sends its outbound HTTP and HTTPS through it once its hive says
the proxy is ready and it has fetched its proxy credential (see _Agents and the
proxy_ below). Any other agent goes straight out through the NAT, as before,
and so does any program that ignores the proxy variables: nothing blocks direct
egress yet.
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
host. It never decrypts TLS, so it holds no signing key. It caches nothing.
- **Credentials.** A request for a listed name must carry a per-agent proxy
credential (HTTP Basic, the agent's name and its secret). `swarm-controller`
mints one per agent at `swarm/agents/<agent>/proxy-credential`, and never
replaces it. `hive-c0re` hashes the credentials of this hive's agents, and
one more for the self-check, into `services.hyperhive.deploy.hive-controller.squid.usersFile`
every 30 seconds. Squid's `basic_ncsa_auth` helper re-reads the file when it
changes. A stopped agent stays in the file. Squid caches a good login for a
minute. Nothing rotates a credential: to replace one, delete its leaf in the
store and the next pass mints a new one. HTTP Basic crosses the bridge in
clear between an agent and the proxy. See
[the credentials reference](../swarm/credentials-reference.md).
A request without a valid credential gets a 407, once the destination
checks below have passed it.
- **Agents and the proxy.** An agent uses the proxy only when its hive says
this host's squid is ready for it. `hive-c0re` puts two facts in every
container's `/etc/hyperhive-hive-facts`, only on a hive that holds a store
identity: `HIVE_SQUID_PROXY` (`services.hyperhive.deploy.hive-controller.squid.listen`, the bridge IP
and port) and `HIVE_EGRESS_NO_PROXY` (the direct hosts below). A hive that
predates them writes neither, and its agents go direct.
`hive-agent-proxy-credential.service` (`nix/agent-modules/egress-proxy.nix`)
runs in each agent's container, as the agent user, at start and every 10
minutes. With both facts present it logs in to the secret store with the
agent's own certificate, reads `swarm/agents/<agent>/proxy-credential` and
writes `/run/hive-agent-proxy-credential/proxy.env` (`0600`) with
`HTTPS_PROXY`, `HTTP_PROXY`, their lower-case forms, `NODE_USE_ENV_PROXY=1`
and `NO_PROXY`/`no_proxy`. The proxy URL is
`http://<agent>:<credential>@<bridgeIp>:3128`; Claude Code, Node, curl, git
and reqwest send the userinfo as `Proxy-Authorization`. The harness, its bash
tool, the subagents, the MCP servers and the pollers load the file with
`EnvironmentFile=-` when they start. Until the facts and the credential both
exist nothing writes the file, so those units have no proxy variable and go
direct; a failed read keeps the file already in place. Nothing restarts on
deploy: an agent takes the proxy at its next start, which is its next rebuild
or restart. The credential is in the environment of every process those units
start, so `env` in the agent's shell prints it. Login shells
(`nixos-container root-login`, SSH) don't load the file.
- **Direct destinations.** `HIVE_EGRESS_NO_PROXY` is the hosts this agent must
reach without the proxy, and `NO_PROXY` is `localhost,127.0.0.1,::1` followed
by that list. `hive-c0re` computes it from the host's options: the bridge IP,
the hive's domain (`services.hyperhive.domain`), the secret store and queue
names (`services.hyperhive.swarm.bao.domain`, `swarm.nats.domain`),
`services.hyperhive.swarm.serviceDomains` (the swarm UI and controller apex,
the forge, matrix, attic, logs, metrics and otel names) and every name the
gateway answers with the bridge IP (`services.hyperhive.gateway.localNames`).
Squid refuses every destination that resolves to the bridge, and the hive's
resolver answers the bridge address for those names, so they never go through
the proxy. A bare name matches itself and its subdomains. Node's `NO_PROXY`
has no CIDR support, so the list holds the bridge as an address.
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
second file beside the users file, `services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
the users whose agent holds the `unrestricted_internet` flag in the
permissions the hive last read from the swarm, one per line, rewritten when
it changes. The default is permissive: an agent with no record, or a record
that never set the flag, holds it, the same default as the
`hyperhive.agent.unrestricted_internet` metric. While the hive's read of the
permissions is stale or failing, a new agent that already holds a
credential counts as unrestricted. Squid reads the list when it
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
when the file changes. A missing or empty file lists no one, and squid
starts either way. A flag change reaches squid after the hive's next read of
the permissions and its next 30-second pass.
An older `hive-c0re` writes no list, so every credential on that hive is
restricted. Agents must not start sending credentials until every hive runs
the `hive-c0re` that writes it, or a legacy agent loses general internet.
- **Log mode.** The operator can put an agent in `log` mode in the swarm UI.
Its record keeps `unrestricted_internet` at `false` and adds
`internet_log: true`, so a `hive-c0re` that doesn't know the field still
enforces the list. For such an agent `hive-c0re` adds to `agent-allow.conf`,
after the allow rules for every agent's own names,
`http_access allow <user> !allowed_dst log_mark`, where `log_mark` is an
`annotate_transaction` ACL that sets `would_deny=1`. The rule comes last in
the include, so every destination deny and `deny !authed` before it still
applies, and a name the agent may reach never carries the mark. A request the
rule lets through gets a second log line,
`would_deny user=<agent> method=<m> host=<h> port=<p> result=<r>`, from the
`would_denied` `access_log` in `hive-squid.nix`; the usual request line keeps
its format. The **would-deny** tab of the `egress` dashboard in Grafana lists the hosts per agent. Only
the operator changes the mode: nothing moves an agent to `restricted` after a
time or a count. The self-check's users aren't agents,
so `deny-canary.invalid` stays a 403.
- **Base list.** `services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains` is open to every
authenticated user, whatever its flag: what the agent harness needs off the
swarm. The default is the hosts Claude Code requires
(<https://code.claude.com/docs/en/network-config>): `api.anthropic.com` for
the model API and the usage probe (`hive-agent/src/claude_usage_watch.rs`),
`claude.ai` for the account login, and `platform.claude.com` for the OAuth
token exchange and refresh. An agent whose `ANTHROPIC_BASE_URL` points
elsewhere needs no entry here: the swarm controller adds that host to the
agent's own swarm entries (see _Swarm allow-list_ below). The list omits the plugin marketplace clone from `github.com`, since a failed add
isn't fatal. The rules below still refuse an entry that resolves to a
private address.
- **Allow-list.** `services.hyperhive.deploy.hive-controller.squid.allowedDomains` defaults to empty,
plus the self-check canary (`services.hyperhive.deploy.hive-controller.squid.selfcheck.canary`), so
the canary widens the restricted list by that one name. The swarm's own
service names aren't on it: they resolve to the bridge or a private address,
which the rules below refuse, and agents reach them directly. Each entry
matches that exact name, or with a leading dot the name and its subdomains. A user on the unrestricted list reaches any
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
`allowedDomains` is the hive's own list, always allowed whether or not the
swarm has a list of its own (next bullet).
- **Swarm allow-list.** The operator adds domains in the swarm UI
([allowed domains](../swarm/ui.md#allowed-domains)); the swarm controller
keeps them in one value of the `egress-allow` bucket, each for every agent
or for one agent. Every 30 seconds `hive-c0re` reads it and writes two files
beside the users file, `services.hyperhive.deploy.hive-controller.squid.allowedFile`
(`/var/lib/hive-squid/allowed`) and `services.hyperhive.deploy.hive-controller.squid.agentAllowFile`
(`/var/lib/hive-squid/agent-allow.conf`), `hive-core:squid`, `0640`, and
logs `squid allow list: 2 swarm, 3 agent entries` when they change. The
swarm-wide names join the allow ACL, so a restricted user reaches them. A
per-agent entry is a rule for that agent's squid user, `http_access allow
<user> <names>`, in the generated `agent-allow.conf` that the configuration
includes at the point where the base list gets its allow: after every
destination deny and
`deny !authed`, so the rules under _Denied before any allow_ still bind it,
and before the unrestricted rule. Only this hive's agents get a rule.
`squid-unrestricted-reload.path` reloads squid when either file changes, so
there is no deploy or restart.
A name is a DNS name of at least two labels. One leading dot (`.example.com`)
also allows every subdomain, and is the only wildcard; the controller refuses
IP addresses, other wildcards, ports, paths and the base list's names when
you add the entry, and `hive-c0re` checks each name again and skips one that
fails, or that the base list or the canary covers, or that another entry of
the same list covers. A name that resolves to a private address stays a 403.
The controller also adds entries itself, and only from the operator's own
actions in the swarm UI. Saving an agent's API-key environment makes it take
the host from `ANTHROPIC_BASE_URL` and from every variable named `*_BASE_URL` (the
address an ACP agent that loads the same file reads), and keeps one
`derived` entry for that agent per host. Each host passes the same check as
an operator's entry, so an IP address, `localhost` and a base-list name get
none. A save or a delete of the environment replaces the agent's derived
entries, and destroying the agent drops all its entries. The agent can
create its own `backend-env` leaf while the leaf is empty, so text found in
the store never creates an entry: when the controller starts, it only
drops derived entries whose exact host the stored environment no longer
names, and logs a host it didn't derive. An environment saved before this
existed gets its entries when the operator saves it once more. An agent on
the hand-placed `backendEnvironmentFile` alone, and the `opencode` preset's
`acp.opencode.provider.baseUrl`, get no derived entry, so add one in the
swarm UI. A `hive-c0re` that predates the field renders a derived entry
like any other.
Linking a GitHub token, an external matrix account or an external forge
account adds a derived entry for that agent per host it reaches, and
deleting the account removes it: `github.com`, `api.github.com`,
`raw.githubusercontent.com`, `codeload.github.com` and
`objects.githubusercontent.com` for GitHub, and the host of the matrix account's `homeserver` or the forge
account's `url`. Each entry's `source` (`link:github`,
`link:matrix:<account>`, `link:forge:<label>`) keeps it apart from the
environment's, which a save of the environment leaves alone; an entry
without a `source` counts as the environment's. Only the controller writes
these leaves, so when it starts it also adds the entries of accounts linked
before this existed and drops those whose account no longer exists, and logs
the counts (`link egress: entries match the linked accounts`). A store it
can't read changes nothing. An `agent.nix` `matrixAccounts` homeserver gets
no entry, since the controller never reads it; add one in the swarm UI.
The swarm's list adds to the hive's `allowedDomains`: `allowedFile` holds
those names plus the swarm-wide entries, so removing an entry never removes a
name the hive lists itself. Until the controller has a list (it predates the
feature, or nobody added an entry) `allowedFile` holds `allowedDomains` and
`agent-allow.conf` is empty. A `hive-c0re` older than this rule replaces
`allowedDomains` with the swarm's list once the controller has one, so every
hive must run the new `hive-c0re` before the controller's first entry exists. A queue that stops answering for the bucket
leaves a file the swarm wrote untouched, and a failed read leaves both
files untouched. The first line of a file written from the swarm's list is a
`# source: swarm` comment. The queue grant a hive needs to read the list
comes with the controller's `swarm-nats-auth`; a hive that can't read it
times out, keeps its files and warns after about 90 seconds.
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
after the destination checks, which look the name up on the host.
- **Denied before any allow.** Destinations that resolve to loopback,
link-local, the bridge subnet, a private range (`10.0.0.0/8`,
`172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`), shared address space
(`100.64.0.0/10`), `0.0.0.0/8`, `::`, multicast or reserved space, and IP-literal
hosts, for every user including the unrestricted ones and for listed names.
Squid resolves names on the host and connects from it, so without these a
name pointing at `127.0.0.1`, at the host's own LAN address or at a private
neighbour would walk around the bridge→loopback DROP rule and the swarm's
network. Squid checks the address it connects to, so a name that changes its
answer between a check and the connect gains nothing. A listed name must
therefore resolve to a public address on the host's resolver: a split-horizon
name that answers a LAN address gets a 403, and so does the self-check
canary, which is why it isn't a swarm service name. Only `CONNECT` to port
443 and plain HTTP to ports 80 and 443 pass.
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
`squid_client_http_requests_total` are the two to watch.
- **Logs.** One syslog line per request reaches the journal as
`squid.service`, and `services.hyperhive.otel.journalUnits` forwards that
unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the
client's source address. A request that a `log`-mode agent's rule let
through also logs a `would_deny` line (see _Log mode_).
- **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service`
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
to `services.hyperhive.deploy.hive-controller.squid.selfcheck.canary` (default: `cache.nixos.org`,
a public name that's also on the allow-list)
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
the same secret and is always on the unrestricted list, and a `CONNECT` to
`services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary` (default:
`example.com`, a name outside the allow-list) must answer 200. It writes one
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
per run and fails the unit when any is wrong. The password is the hive's
`swarm/hives/<hive>/squid-selfcheck` secret, which
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't
written the users file, the line reads `credential=absent` or
`allow_status=407` and `ok=false`, and nothing else breaks. The
host's resolver looks up the canary, so a host whose resolver maps
that name to a loopback, bridge or private address fails the check.
## Container isolation
Each agent container runs in a private network namespace with a dedicated
veth pair attached to the bridge. The following table summarises what
the nix side sets up unconditionally:
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
`10.42.0.1/24`), **not** the canonical network address. The Rust side
must normalise (bitwise-AND with mask) before subnet membership checks or
address arithmetic.
### What the Rust side does
`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
and `HOST_BRIDGE=<bridgeName>` when creating or updating containers:
`lifecycle::set_nspawn_flags` hands them to hive-priv, which rewrites
`/etc/nixos-containers/<container>.conf`. hive-c0re validates both variables **once at
daemon startup**, not per container: they're process-global, so a
missing or malformed value is a misconfigured daemon rather than one bad
container, and failing at boot gives a single diagnostic instead of one
per agent. No non-isolated mode exists to fall back to. hive-c0re leaves `LOCAL_ADDRESS` empty so the
container's dhcpcd acquires an address from the bridge dnsmasq pool
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies to
every agent container, manager included.
`HOST_ADDRESS` is the bridge gateway IP (the address part of
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
so a non-`.1` operator override still resolves to wherever the bridge
actually lives). It's **load-bearing**: nixos-container's container-side
network setup only installs a default route (`ip route add default via
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode
nixos-container skips the host-side address/route setup, so writing it only affects the
container's default route — without it the container comes up with an IP
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
### How the isolated container gets its resolver
nixos-container copies the **host's** `/etc/resolv.conf` into the container
at every start. The host resolver (for example `127.0.0.53` from systemd-resolved,
or a LAN router) is unreachable from a private netns and isn't
authoritative for the hive's own zones, so it's replaced with the
bridge dnsmasq at boot. Because the copy happens on every start, it
would clobber a declarative `environment.etc."resolv.conf"` — so the
wiring is runtime:
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
gateway IP) into each container's `/etc`.
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
It's ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
the forge-token fetch so the resolver is correct before the first DNS lookup.
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
domain sockets bind-mounted into containers, not network listeners — see
the _Control plane (no network)_ bullet under [Network
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
that never touches the network stack.
The nix side also enables IP forwarding + NAT (agents reach the internet
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP.
## Cross-references
- `docs/networking/gateway.md` — vhost map + the gateway's other duties
View git blame Copy permalink