egress.forward defaults to reject while egress.intercept.enable is on (a hive controller host turns it on) and to log otherwise, because reject without the intercept would refuse all web traffic and an assertion already forbids it. log stays an explicit choice. The hive-squid VM test runs its log-mode half on the base node, so the base pins forward = log at mkDefault priority and the reject specialisation overrides it. A module-eval case covers the default on a controller host, on a host without the intercept, and an explicit log.
939 lines
62 KiB
Markdown
939 lines
62 KiB
Markdown
# hive-network
|
||
|
||
The bridge every container on a host attaches to, and the private-netns
|
||
isolation of agent and CI containers behind it. It comes up on any host that runs agents or swarm services:
|
||
`services.hyperhive.network.enable` defaults to off, and the modules that
|
||
need the bridge set it with `mkDefault true` — the hive controller, CI,
|
||
the hive collector, and the gateway's resolver, which every swarm service
|
||
turns on. Configured via `services.hyperhive.network.*`.
|
||
|
||
Agent containers always run in a private netns.
|
||
|
||
## Network map
|
||
|
||
One picture of the whole hive — two planes: **infra
|
||
containers share the host netns** and bind host ports directly;
|
||
**compute containers (agents + CI) each get a private netns** behind
|
||
the bridge. The unix-socket control plane rides the VFS, and none of
|
||
that touches it.
|
||
|
||
```
|
||
internet
|
||
│ uplink NIC — NAT MASQUERADE for the
|
||
│ bridge subnet (10.42.0.0/24 default)
|
||
┌──────────────────────────┴─────────────────────────────────────────┐
|
||
│ host netns — the host itself plus gateway / forge │
|
||
│ │
|
||
│ nginx :80/:443 [hive-gateway] │
|
||
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
|
||
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
|
||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||
│ │
|
||
│ hive-br0 10.42.0.1/24 │
|
||
│ ┌──────────┼──────────────┐ │
|
||
└──────────────┼──────────┼──────────────┼───────────────────────────┘
|
||
vb-h-<a> vb-h-<b> vb-<runner> veth pairs
|
||
│ │ │
|
||
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
|
||
│ agent a │ │ agent b │ │ runner │ each; eth0 leases
|
||
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
|
||
└─────────┘ └─────────┘ └─────────┘
|
||
```
|
||
|
||
| container | netns | IPv4 | listens / reached via |
|
||
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it |
|
||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
|
||
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||
| `swarm-review-classifier` | private, veth on bridge | DHCP pool | outbound only (classifier → queue, secret store, swarm collector); no inbound surface |
|
||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||
|
||
The flows, end to end:
|
||
|
||
- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
|
||
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
|
||
`dhcpcd` runs with `noipv4ll` and waits for an IPv4 address, so
|
||
`network-online.target` means a lease, never a 169.254.x fallback; units
|
||
that look up a swarm name at start order after it.
|
||
- **DNS** — agents and the service containers query `bridgeIp:53`; dnsmasq
|
||
answers hive zones authoritatively with the bridge IP, everything else
|
||
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
|
||
container points its own `resolv.conf` there, and one that instead
|
||
inherits the host's resolves no swarm name at all — those records exist
|
||
only on the bridge.
|
||
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
|
||
own dashboard name resolve to the bridge IP, land on nginx
|
||
`:80`/`:443`, and proxy to forgejo
|
||
`:3000`, tuwunel's unix socket, hive-c0re `127.0.0.1:7000`, or a per-agent
|
||
UI unix socket.
|
||
- **Internet egress** — agent default route points at the bridge IP;
|
||
the host forwards + masquerades out its uplink.
|
||
- **Swarm** — peer hives reach each other's gateway and forge over HTTPS, and
|
||
the queue and the secret store on their routable listeners
|
||
([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)).
|
||
- **Control plane (no network)** — per-agent broker socket
|
||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||
operator admin `/run/hyperhive/host.sock`, and the per-agent UI
|
||
sockets under `/run/hive-agent/` are unix domain sockets
|
||
bind-mounted through the VFS; private netns doesn't affect them.
|
||
|
||
## Container shape (where dnsmasq lives)
|
||
|
||
dnsmasq runs on the host itself, next to nginx — both come from the
|
||
gateway module, one front door for DNS and HTTP. Its `bind-interfaces`
|
||
listener sits on `bridgeIp`, on the host's bridge interface.
|
||
|
||
## Configuration
|
||
|
||
```nix
|
||
{
|
||
services.hyperhive = {
|
||
swarm = {
|
||
domain = "myswarm.example";
|
||
hives.firsthive = { }; # -> domain = firsthive.myswarm.example
|
||
};
|
||
deploy.hive-controller = {
|
||
enable = true;
|
||
hiveName = "firsthive";
|
||
};
|
||
# network.bridgeIp = "10.42.0.1"; # default
|
||
};
|
||
}
|
||
```
|
||
|
||
The hive domain (`services.hyperhive.domain`, which the resolver answers
|
||
for) comes from this hive's entry in the swarm directory:
|
||
`swarm.hives.<hiveName>.domain`, default `<hiveName>.<swarm.domain>`.
|
||
Eval fails until `swarm.domain` and that entry exist
|
||
([`2-handbook/swarm/hive-directory.md`](../../2-handbook/swarm/hive-directory.md#hive-identity-config)).
|
||
|
||
## Bridge addressing
|
||
|
||
Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
|
||
RFC 1918 space, unlikely to clash with operator's existing setup;
|
||
override `bridgeIp` + `bridgePrefixLength` if a different range is
|
||
already in use. `/24` gives 254 usable per-agent addresses — enough
|
||
for any single-host hive; bigger swarms or tighter addressing
|
||
schemes pick their own.
|
||
|
||
## Resolver behaviour
|
||
|
||
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
|
||
plus whatever swarm-service names this host contributes via
|
||
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
|
||
(matrix) when this host runs those services, `auth.<swarm-domain>`
|
||
when it runs authelia, and `hive-collector.hive` on every hive
|
||
— answering each with the bridge IP (where nginx
|
||
is reachable, and where the hive's own collector listens). Note forge and
|
||
matrix are swarm-domain names, not sub-domains of `<hive-domain>`: a swarm
|
||
runs one forge and one homeserver, so their names belong to the swarm rather
|
||
than to whichever hive happens to host them.
|
||
|
||
Every domain answered this way is also `local=` for dnsmasq, so the other
|
||
record types (AAAA, TXT, MX, HTTPS) for those names get an empty answer from
|
||
dnsmasq itself instead of waiting on the host's upstream resolver. Without it,
|
||
dnsmasq 2.86 and later send them upstream, and a host whose upstream doesn't
|
||
answer leaves each lookup to its client's own timeout.
|
||
|
||
dnsmasq forwards everything else to the host's
|
||
own resolvers: it runs on the host and reads the host's
|
||
`/etc/resolv.conf` directly. Containers don't need to know the
|
||
upstream — they query the bridge IP and dnsmasq does the right thing
|
||
per-name.
|
||
|
||
No fallback `server=` exists, by design: dnsmasq queries all known
|
||
upstreams in parallel, so a hardcoded public resolver would take a share
|
||
of normal traffic, not just cover the gap.
|
||
|
||
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
|
||
directly, so a network change (new router, new lease, laptop moving
|
||
networks) reaches it the moment openresolv rewrites the file. Nothing
|
||
needs synchronising, and no unit watches for it.
|
||
|
||
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
||
listener only accepts queries from the bridge interface (plus lo for
|
||
container health-checks). External hosts can't reach it — no
|
||
DNS-amplification surface even when the operator opens port 80 for
|
||
gateway HTTP.
|
||
|
||
`resolveLocalQueries = false` keeps dnsmasq out of the host's own
|
||
resolution stack — the host's resolver (systemd-resolved, plain
|
||
glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
|
||
configured. The hive resolver is purely for inbound queries from
|
||
agent containers.
|
||
|
||
## Firewall posture
|
||
|
||
`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
|
||
`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
|
||
|
||
- Port 53 opens the resolver on the bridge interface only. Other
|
||
interfaces stay closed. The hive resolver isn't an external-facing
|
||
service.
|
||
- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
|
||
receives DHCP via a regular UDP socket (it doesn't use a
|
||
netfilter-bypassing raw socket), so the hole is mandatory — without
|
||
it containers never get a lease and fall back to 169.254.x.x.
|
||
- Ports 80 and 443 let isolated agents reach nginx (on the host) for the forge sub-domain, per-agent
|
||
UI proxies, and any other HTTP services.
|
||
|
||
The **host** firewall is the only firewall. The swarm service
|
||
containers that share the host netns (forge, bao, the
|
||
metrics and log stores) run with `networking.firewall.enable = false`
|
||
(`nix/container-modules/swarm-container.nix`): a NixOS firewall inside a
|
||
shared-netns container runs against the _host_ ruleset — at container
|
||
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
|
||
from the container's (empty) port list, and deletes the host's
|
||
`nixos-nat-*` chains without recreating them, silently wiping the
|
||
bridge holes above plus the agents' NAT. Private-netns containers
|
||
(every container with `privateNetwork = true`: agents and the service containers on the bridge) keep their own firewall — it's scoped to their
|
||
namespace.
|
||
|
||
### Reaching host services (`exposeHostPorts`)
|
||
|
||
By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||
host-side service on another port — for example a dev OTLP collector you want
|
||
agents to reach directly — is unreachable. (hyperhive's own services
|
||
need none of this: a module that means to be reachable from agents opens
|
||
its own port here, which the hive collector and `nats.enable` both do. Note
|
||
that `deploy.swarm-otel.upstream.endpoint` is the _upstream_, which no agent ever dials. See
|
||
`docs/4-implementation/scheduler/observability.md`.)
|
||
|
||
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
||
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
|
||
agent can connect to `<bridgeIp>:P` (for a dev OTLP collector, point its
|
||
endpoint at `http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
|
||
The hive's own collector needs none of this: agents dial it by
|
||
`otel.collector.host`.
|
||
|
||
<!-- vale write-good.Passive = NO -->
|
||
|
||
This is **firewall-only**: the host service must bind an address
|
||
reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
|
||
only. The bridge→`127.0.0.0/8` DROP rule (below) still applies, so a
|
||
service bound to `127.0.0.1` only stays unreachable; rebind it to
|
||
`0.0.0.0`.
|
||
|
||
<!-- vale write-good.Passive = YES -->
|
||
|
||
The port is reachable by **every** agent on the bridge subnet (like
|
||
DNS/gateway), so only expose services safe for any agent to reach.
|
||
|
||
## Egress proxy (squid)
|
||
|
||
Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
|
||
(`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
|
||
only. An agent sends its outbound HTTP and HTTPS through it once its hive says
|
||
the proxy is ready and it has fetched its proxy credential (see _Agents and the
|
||
proxy_ below). A program that ignores the proxy variables still reaches squid
|
||
when it speaks HTTP on port 80 or TLS on port 443 and its agent is `restricted`
|
||
or in `log` mode: the host redirects that traffic into squid's intercept ports
|
||
(see _Intercepted web traffic_ below). Everything else goes straight out through
|
||
the NAT, as before: the `unrestricted` agents, the service containers and every
|
||
other port and protocol. A hive controller host refuses the direct connections
|
||
of the `restricted` and `log` agents (`egress.forward = "reject"`, the default
|
||
while the intercept is on); `egress.forward = "log"` only logs them (see _Agent
|
||
bridge egress log_ below).
|
||
|
||
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
|
||
host. It never decrypts TLS. The one TLS key it loads is a throwaway that the
|
||
unit generates at each start, that signs nothing and that no agent trusts. It
|
||
caches nothing.
|
||
|
||
- **Credentials.** A request for a listed name must carry a per-agent proxy
|
||
credential (HTTP Basic, the agent's name and its secret). `swarm-controller`
|
||
mints one per agent at `swarm/agents/<agent>/proxy-credential`, and never
|
||
replaces it. `hive-c0re` hashes the credentials of this hive's agents, and
|
||
one more for the self-check, into `services.hyperhive.deploy.hive-controller.squid.usersFile`
|
||
every 30 seconds. Squid's `basic_ncsa_auth` helper re-reads the file when it
|
||
changes. A stopped agent stays in the file. Squid caches a good login for a
|
||
minute. Nothing rotates a credential: to replace one, delete its leaf in the
|
||
store and the next pass mints a new one. HTTP Basic crosses the bridge in
|
||
clear between an agent and the proxy. See
|
||
[the credentials reference](../../4-implementation/swarm/credentials-reference.md).
|
||
A request without a valid credential gets a 407, once the destination
|
||
checks below have passed it.
|
||
|
||
- **Agents and the proxy.** An agent uses the proxy only when its hive says
|
||
this host's squid is ready for it. `hive-c0re` puts two facts in every
|
||
container's `/etc/hyperhive-hive-facts`, only on a hive that holds a store
|
||
identity: `HIVE_SQUID_PROXY` (`services.hyperhive.deploy.hive-controller.squid.listen`, the bridge IP
|
||
and port) and `HIVE_EGRESS_NO_PROXY` (the direct hosts below). A hive that
|
||
predates them writes neither, and its agents go direct.
|
||
`hive-agent-proxy-credential.service` (`nix/agent-modules/egress-proxy.nix`)
|
||
runs in each agent's container, as the agent user, at start and every 10
|
||
minutes. With both facts present it logs in to the secret store with the
|
||
agent's own certificate, reads `swarm/agents/<agent>/proxy-credential` and
|
||
writes `/run/hive-agent-proxy-credential/proxy.env` (`0600`) with
|
||
`HTTPS_PROXY`, `HTTP_PROXY`, their lower-case forms, `NODE_USE_ENV_PROXY=1`
|
||
and `NO_PROXY`/`no_proxy`. The proxy URL is
|
||
`http://<agent>:<credential>@<bridgeIp>:3128`; Claude Code, Node, curl, git
|
||
and reqwest send the userinfo as `Proxy-Authorization`. The harness, its bash
|
||
tool, the subagents, the MCP servers and the pollers load the file with
|
||
`EnvironmentFile=-` when they start. Until the facts and the credential both
|
||
exist nothing writes the file, so those units have no proxy variable and go
|
||
direct; a failed read keeps the file already in place. Nothing restarts on
|
||
deploy: an agent takes the proxy at its next start, which is its next rebuild
|
||
or restart. The credential is in the environment of every process those units
|
||
start, so `env` in the agent's shell prints it. Login shells
|
||
(`nixos-container root-login`, SSH) don't load the file.
|
||
- **Direct destinations.** `HIVE_EGRESS_NO_PROXY` is the hosts this agent must
|
||
reach without the proxy, and `NO_PROXY` is `localhost,127.0.0.1,::1` followed
|
||
by that list. `hive-c0re` computes it from the host's options: the bridge IP,
|
||
the hive's domain (`services.hyperhive.domain`), the secret store and queue
|
||
names (`services.hyperhive.swarm.bao.domain`, `swarm.nats.domain`),
|
||
`services.hyperhive.swarm.serviceDomains` (the swarm UI and controller apex,
|
||
the forge, matrix, attic, logs, metrics and otel names) and every name the
|
||
gateway answers with the bridge IP (`services.hyperhive.gateway.localNames`).
|
||
Squid refuses every destination that resolves to the bridge, and the hive's
|
||
resolver answers the bridge address for those names, so they never go through
|
||
the proxy. A bare name matches itself and its subdomains. Node's `NO_PROXY`
|
||
has no CIDR support, so the list holds the bridge as an address.
|
||
|
||
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
|
||
second file beside the users file, `services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`
|
||
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
|
||
the users whose agent holds the `unrestricted_internet` flag in the
|
||
permissions the hive last read from the swarm, one per line, rewritten when
|
||
it changes. The default is permissive: an agent with no record, or a record
|
||
that never set the flag, holds it, the same default as the
|
||
`hyperhive.agent.unrestricted_internet` metric. While the hive's read of the
|
||
permissions is stale or failing, a new agent that already holds a
|
||
credential counts as unrestricted. Squid reads the list when it
|
||
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
|
||
when the file changes. A missing or empty file lists no one, and squid
|
||
starts either way. A flag change reaches squid after the hive's next read of
|
||
the permissions and its next 30-second pass.
|
||
An older `hive-c0re` writes no list, so every credential on that hive is
|
||
restricted. Agents must not start sending credentials until every hive runs
|
||
the `hive-c0re` that writes it, or a legacy agent loses general internet.
|
||
|
||
- **Log mode.** The operator can put an agent in `log` mode in the swarm UI.
|
||
Its record keeps `unrestricted_internet` at `false` and adds
|
||
`internet_log: true`, so a `hive-c0re` that doesn't know the field still
|
||
enforces the list. For such an agent `hive-c0re` adds to `agent-allow.conf`,
|
||
after the allow rules for every agent's own names,
|
||
`http_access allow !icpt <user> !allowed_dst log_mark` and the same rule for
|
||
the intercept ports, where `log_mark` is an `annotate_transaction` ACL that
|
||
sets `would_deny=1`. The rules come last in the include, so every destination
|
||
deny and `deny !icpt !authed` before them still applies, and a name the agent may reach never carries the mark. A request the
|
||
rule lets through gets a second log line,
|
||
`would_deny user=<agent> method=<m> host=<h> port=<p> result=<r>`, from the
|
||
`would_denied` `access_log` in `hive-squid.nix`; the usual request line keeps
|
||
its format. The **would-deny** tab of the `egress` dashboard in Grafana lists the hosts per agent. Only
|
||
the operator changes the mode: nothing moves an agent to `restricted` after a
|
||
time or a count. The self-check's users aren't agents,
|
||
so `deny-canary.invalid` stays a 403.
|
||
|
||
- **Modes.** An agent is in one of three modes, and the deny list (next bullet)
|
||
applies in all of them:
|
||
|
||
| mode | the deny list | a name not on the allow list |
|
||
| -------------- | ------------- | ---------------------------- |
|
||
| `restricted` | blocked | blocked |
|
||
| `log` | blocked | allowed and logged |
|
||
| `unrestricted` | blocked | allowed |
|
||
|
||
- **Deny list.** The swarm's deny list names destinations the proxy refuses
|
||
whatever else allows them. The swarm controller keeps it as its own value
|
||
(the `deny` key of the `egress-allow` bucket), each entry for every agent or
|
||
for one, and serves it at `GET`/`POST`/`DELETE /api/egress/denied-domains`
|
||
with the same body as the allow-list (`{"domain": ".example.com", "agent":
|
||
"atlas"}`). Write a name as in the allow-list: at least two labels, one
|
||
leading dot for the name and its subdomains. The controller refuses IP
|
||
addresses, other wildcards, ports and paths. Unlike the allow-list it doesn't
|
||
refuse the base list's names: denying one is the point. Every 30 seconds
|
||
`hive-c0re` reads the list and renders `services.hyperhive.deploy.hive-controller.squid.denyFile`
|
||
(`/var/lib/hive-squid/deny.conf`, `hive-core:squid`, `0640`) for the agents it
|
||
runs, and `squid-unrestricted-reload.path` reloads squid when it changes.
|
||
- **Precedence.** The configuration includes the file after the destination
|
||
denies and before `deny !icpt !authed` and every allow, so a denied name is
|
||
refused even when the base list, the hive's `allowedDomains`, the swarm's
|
||
allow-list, the unrestricted flag or `log` mode would allow it. A swarm-wide
|
||
entry names no user, so every client gets a 403 for it, one without a
|
||
credential too. An agent's entry also names the agent's squid user, so it
|
||
binds that agent: a client that sent no credential gets the usual 407 for
|
||
it. An agent's entry only adds to the swarm-wide ones; no entry of one scope
|
||
lifts another.
|
||
- **Log line.** A request a deny rule refused gets a second log line,
|
||
`egress_denied user=<agent or -> method=<m> host=<h> port=<p> result=<r>`,
|
||
in every mode. It differs from the `would_deny` line of `log` mode and
|
||
from the `TCP_DENIED/403` line every refusal has, so you can tell a
|
||
deny-list hit from a name that isn't on the allow-list.
|
||
- **Who writes it.** Only an operator's request to the controller writes the
|
||
list. It has no automatic entries: nothing derives a deny from an agent's
|
||
API-key environment or a linked account, and the stored type has no such
|
||
field. An agent has no queue grant that reaches the key, and the controller's
|
||
API sits behind the operator's SSO login: nginx checks it before it hands
|
||
a request to the socket, which only nginx and root can open.
|
||
- **Empty list.** With no entry `deny.conf` is empty and squid configures as
|
||
it did without the list: the configuration gains one `include` line and the
|
||
`egress_denied` log lines, and no rule. A hive on a `hive-c0re` that
|
||
predates the list never reads the key and keeps its old configuration, so
|
||
nothing on its wire changes.
|
||
- **Applied.** Each hive reports the revision of the list it last rendered as
|
||
`egress_deny_applied` in its status snapshot (see [the hive
|
||
snapshot](../../2-handbook/swarm/README.md)). `GET /api/egress/denied-domains` answers the
|
||
list's `revision` and `behind`, the hives that haven't reported it: an older
|
||
`hive-c0re` never does, so it shows there rather than silently allowing the
|
||
names. A hive that reads the list reports within about a minute. Reporting
|
||
means the file holds the rules; squid reloads moments later.
|
||
|
||
- **Base list.** `services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains` is open to every
|
||
authenticated user, whatever its flag: what the agent harness needs off the
|
||
swarm. The default is the hosts Claude Code requires
|
||
(<https://code.claude.com/docs/en/network-config>): `api.anthropic.com` for
|
||
the model API and the usage probe (`hive-agent/src/claude_usage_watch.rs`),
|
||
`claude.ai` for the account login, and `platform.claude.com` for the OAuth
|
||
token exchange and refresh. An agent whose `ANTHROPIC_BASE_URL` points
|
||
elsewhere needs no entry here: the swarm controller adds that host to the
|
||
agent's own swarm entries (see _Swarm allow-list_ below). The list omits the plugin marketplace clone from `github.com`, since a failed add
|
||
isn't fatal. The rules below still refuse an entry that resolves to a
|
||
private address.
|
||
- **Allow-list.** `services.hyperhive.deploy.hive-controller.squid.allowedDomains` defaults to empty,
|
||
plus the self-check canary (`services.hyperhive.deploy.hive-controller.squid.selfcheck.canary`), so
|
||
the canary widens the restricted list by that one name. The swarm's own
|
||
service names aren't on it: they resolve to the bridge or a private address,
|
||
which the rules below refuse, and agents reach them directly. Each entry
|
||
matches that exact name, or with a leading dot the name and its subdomains. A user on the unrestricted list reaches any
|
||
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
|
||
`allowedDomains` is the hive's own list, always allowed whether or not the
|
||
swarm has a list of its own (next bullet).
|
||
- **Swarm allow-list.** The operator adds domains in the swarm UI
|
||
([allowed domains](../web-ui/swarm-ui.md#allowed-domains)); the swarm controller
|
||
keeps them in one value of the `egress-allow` bucket, each for every agent
|
||
or for one agent. Every 30 seconds `hive-c0re` reads it and writes two files
|
||
beside the users file, `services.hyperhive.deploy.hive-controller.squid.allowedFile`
|
||
(`/var/lib/hive-squid/allowed`) and `services.hyperhive.deploy.hive-controller.squid.agentAllowFile`
|
||
(`/var/lib/hive-squid/agent-allow.conf`), `hive-core:squid`, `0640`, and
|
||
logs `squid allow list: 2 swarm, 3 agent entries` when they change. The
|
||
swarm-wide names join the allow ACL, so a restricted user reaches them. A
|
||
per-agent entry is a rule for that agent's squid user, `http_access allow
|
||
!icpt <user> <names>`, and a second one for the user the intercept helper names
|
||
(`icpt <user> <names>`), in the generated `agent-allow.conf` that the
|
||
configuration includes at the point where the base list gets its allow: after
|
||
every destination deny and `deny !icpt !authed`, so the rules under _Denied before any allow_ still bind it,
|
||
and before the unrestricted rule. Only this hive's agents get a rule.
|
||
`squid-unrestricted-reload.path` reloads squid when either file changes, so
|
||
there is no deploy or restart.
|
||
A name is a DNS name of at least two labels. One leading dot (`.example.com`)
|
||
also allows every subdomain, and is the only wildcard; the controller refuses
|
||
IP addresses, other wildcards, ports, paths and the base list's names when
|
||
you add the entry, and `hive-c0re` checks each name again and skips one that
|
||
fails, or that the base list or the canary covers, or that another entry of
|
||
the same list covers. A name that resolves to a private address stays a 403.
|
||
The controller also adds entries itself, and only from the operator's own
|
||
actions in the swarm UI. Saving an agent's API-key environment makes it take
|
||
the host from `ANTHROPIC_BASE_URL` and from every variable named `*_BASE_URL` (the
|
||
address an ACP agent that loads the same file reads), and keeps one
|
||
`derived` entry for that agent per host. Each host passes the same check as
|
||
an operator's entry, so an IP address, `localhost` and a base-list name get
|
||
none. A save or a delete of the environment replaces the agent's derived
|
||
entries, and destroying the agent drops all its entries. The agent can
|
||
create its own `backend-env` leaf while the leaf is empty, so text found in
|
||
the store never creates an entry: when the controller starts, it only
|
||
drops derived entries whose exact host the stored environment no longer
|
||
names, and logs a host it didn't derive. An environment saved before this
|
||
existed gets its entries when the operator saves it once more. An agent on
|
||
the hand-placed `backendEnvironmentFile` alone, and the `opencode` preset's
|
||
`acp.opencode.provider.baseUrl`, get no derived entry, so add one in the
|
||
swarm UI. A `hive-c0re` that predates the field renders a derived entry
|
||
like any other.
|
||
Linking a GitHub token, an external matrix account or an external forge
|
||
account adds a derived entry for that agent per host it reaches, and
|
||
deleting the account removes it: `github.com`, `api.github.com`,
|
||
`raw.githubusercontent.com`, `codeload.github.com` and
|
||
`objects.githubusercontent.com` for GitHub, and the host of the matrix account's `homeserver` or the forge
|
||
account's `url`. Each entry's `source` (`link:github`,
|
||
`link:matrix:<account>`, `link:forge:<label>`) keeps it apart from the
|
||
environment's, which a save of the environment leaves alone; an entry
|
||
without a `source` counts as the environment's. Only the controller writes
|
||
these leaves, so when it starts it also adds the entries of accounts linked
|
||
before this existed and drops those whose account no longer exists, and logs
|
||
the counts (`link egress: entries match the linked accounts`). A store it
|
||
can't read changes nothing. An `agent.nix` `matrixAccounts` homeserver gets
|
||
no entry, since the controller never reads it; add one in the swarm UI.
|
||
The swarm's list adds to the hive's `allowedDomains`: `allowedFile` holds
|
||
those names plus the swarm-wide entries, so removing an entry never removes a
|
||
name the hive lists itself. Until the controller has a list (it predates the
|
||
feature, or nobody added an entry) `allowedFile` holds `allowedDomains` and
|
||
`agent-allow.conf` is empty. A `hive-c0re` older than this rule replaces
|
||
`allowedDomains` with the swarm's list once the controller has one, so every
|
||
hive must run the new `hive-c0re` before the controller's first entry exists. A queue that stops answering for the bucket
|
||
leaves a file the swarm wrote untouched, and a failed read leaves both
|
||
files untouched. The first line of a file written from the swarm's list is a
|
||
`# source: swarm` comment. The queue grant a hive needs to read the list
|
||
comes with the controller's `swarm-nats-auth`; a hive that can't read it
|
||
times out, keeps its files and warns after about 90 seconds.
|
||
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
|
||
after the destination checks, which look the name up on the host.
|
||
- **Denied before any allow.** Destinations that resolve to loopback,
|
||
link-local, the bridge subnet, a private range (`10.0.0.0/8`,
|
||
`172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`), shared address space
|
||
(`100.64.0.0/10`), `0.0.0.0/8`, `::`, multicast or reserved space, and IP-literal
|
||
hosts, for every user including the unrestricted ones and for listed names.
|
||
Squid resolves names on the host and connects from it, so without these a
|
||
name pointing at `127.0.0.1`, at the host's own LAN address or at a private
|
||
neighbour would walk around the bridge→loopback DROP rule and the swarm's
|
||
network. Squid checks the address it connects to, so a name that changes its
|
||
answer between a check and the connect gains nothing. A listed name must
|
||
therefore resolve to a public address on the host's resolver: a split-horizon
|
||
name that answers a LAN address gets a 403, and so does the self-check
|
||
canary, which is why it isn't a swarm service name. Only `CONNECT` to port
|
||
443 and plain HTTP to ports 80 and 443 pass.
|
||
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
|
||
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
|
||
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
|
||
`squid_client_http_requests_total` are the two to watch.
|
||
- **Logs.** One syslog line per request reaches the journal as
|
||
`squid.service`, and `services.hyperhive.otel.journalUnits` forwards that
|
||
unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the
|
||
client's source address. A request that a `log`-mode agent's rule let
|
||
through also logs a `would_deny` line (see _Log mode_). A request a deny-list rule refused logs an `egress_denied` line (see _Deny list_).
|
||
- **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service`
|
||
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
|
||
to `services.hyperhive.deploy.hive-controller.squid.selfcheck.canary` (default: `cache.nixos.org`,
|
||
a public name that's also on the allow-list)
|
||
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
|
||
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
|
||
the same secret and is always on the unrestricted list, and a `CONNECT` to
|
||
`services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary` (default:
|
||
`example.com`, a name outside the allow-list) must answer 200. It writes one
|
||
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
|
||
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
|
||
per run and fails the unit when any is wrong. The password is the hive's
|
||
`swarm/hives/<hive>/squid-selfcheck` secret, which
|
||
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
|
||
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't
|
||
written the users file, the line reads `credential=absent` or
|
||
`allow_status=407` and `ok=false`, and nothing else breaks. The
|
||
host's resolver looks up the canary, so a host whose resolver maps
|
||
that name to a loopback, bridge or private address fails the check.
|
||
|
||
### Intercepted web traffic
|
||
|
||
A program that ignores the proxy variables still gets squid's policy for its
|
||
web traffic. In the `inet hive_egress` table (`nix/host-modules/hive-egress.nix`)
|
||
the `intercept` chain, at `prerouting`, redirects IPv4 TCP port 80 to
|
||
`<bridgeIp>:3129`, an intercept listener of squid, and port 443 to
|
||
`<bridgeIp>:3131`, the ClientHello relay (see _ECH_ below), which sends the
|
||
connection on to squid's other intercept listener, `<bridgeIp>:3130`. Squid reads
|
||
the original destination from conntrack. The chain exists where the host runs
|
||
squid (`services.hyperhive.deploy.hive-controller.egress.intercept.enable`).
|
||
|
||
- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
|
||
a flow to an address that stays on the bridge, to one of the host's own
|
||
addresses, and a flow from a MAC in the set
|
||
`direct_mac`. That set holds the MACs of the ports of every container that's
|
||
not an agent (the swarm's service containers) and of the agents listed in
|
||
`services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
|
||
`restricted` and `log` agents get redirected, and `unrestricted` agents and
|
||
service containers keep the direct path. Every pass of `hive-priv egress-pin`
|
||
and every `egress-load` refills the set from the bindings it also uses for the
|
||
bridge table, and `hive-egress-direct.path` runs a pass when `hive-c0re`
|
||
rewrites the list; the timer covers the rest. A MAC in no binding is in no
|
||
exemption: the flow gets redirected, and squid refuses it. A new agent that
|
||
the list doesn't name yet counts as restricted until the next pass, and a flag
|
||
change reaches the redirect after the hive's next read of the permissions and
|
||
one pass. IPv6 and the other ports aren't redirected.
|
||
- **Who the flow belongs to.** An intercepted request carries no proxy
|
||
credential, so squid asks its external ACL `hive_agent`, which runs
|
||
`hive-priv egress-whois`. The helper looks the source address up in dnsmasq's
|
||
lease file for a MAC, the MAC in the bindings (`/run/hive-egress-pins`) for a
|
||
machine, and takes the agent from the machine name `h-<agent>`. While
|
||
`antiSpoof` is `drop`, an address on the bridge is the one leased to the MAC
|
||
of the port it came from (see _Agent bridge source check_), so the source says
|
||
which container sent the flow. Squid keeps an answer for 10 seconds and a
|
||
refusal for 2. An address with no lease or an expired one, two MACs with the
|
||
same address, a MAC no port holds, a machine that's no agent: none gets an
|
||
answer, and squid refuses the flow with a `TCP_DENIED/403` and user `-`. A
|
||
host that has no bindings or leases yet refuses every intercepted flow rather
|
||
than guess. With `antiSpoof = log` a container can forge the source, so use that mode
|
||
to diagnose and not to run. The proxy port never takes this identity: a
|
||
request without a credential through port 3128 still gets a 407, whatever its
|
||
address.
|
||
- **Policy.** The same lists as for the proxy port, matched on the agent the
|
||
helper names: the destination denies, the deny list, the base list, the
|
||
allow-list and the agent's own names, and `log` mode with its `would_deny`
|
||
line. `hive-c0re` renders each agent rule twice, once for the credential
|
||
(`!icpt`) and once for the helper's user (`icpt`).
|
||
- **HTTPS.** Squid peeks at the ClientHello for the SNI and splices the
|
||
connection. It never decrypts, and no agent trusts a CA of the host. Squid
|
||
builds a `CONNECT` with the SNI as its host and applies every rule to it. A
|
||
refused name gets a 403 that the client reads as a TLS error. Squid also
|
||
refuses:
|
||
- an SNI that doesn't resolve to the address the agent dialed
|
||
(`host_verify_strict`; squid answers 409 and the client reads a TLS error).
|
||
Squid resolves on the host's resolver, which is the same dnsmasq as the
|
||
agent's, so a name with changing answers can fail now and then;
|
||
- a ClientHello with no SNI, and a URL with an IP literal, which sends none:
|
||
the host of the `CONNECT` stays an address, and the IP-literal denies refuse
|
||
it;
|
||
- bytes on port 80 that aren't an HTTP request. Bytes on port 443 that aren't
|
||
a ClientHello never reach squid (see _ECH_ below).
|
||
- **ECH.** Squid 7.6 can't see the `encrypted_client_hello` extension, only the
|
||
public name in the outer hello. An agent that holds an ECH configuration could
|
||
send a hello with an allowed outer name, a destination address that name
|
||
resolves to, and a hidden inner name that the same address serves. Port 443
|
||
therefore doesn't go to squid directly. `hive-priv egress-ech-relay`
|
||
(`hive-egress-ech-relay.service`, `egress.intercept.refuseEch`, on by default)
|
||
listens on `<bridgeIp>:3131`. It reads the first TLS records of each redirected
|
||
connection, up to four records and 16 KiB of ClientHello within 10 seconds, and
|
||
closes the connection when the ClientHello of a `restricted` agent carries the
|
||
extension (type `0xfe0d`, and the draft types `0xfe08` to `0xfe0c`, `0xff02`
|
||
and `0xffce`). The relay only observes a `log` agent, as everywhere in `log`
|
||
mode: its hello goes on like a clean one, and the relay logs it (see below).
|
||
It closes a connection too, for any agent, when the bytes aren't a well-formed
|
||
ClientHello, when they don't arrive in time, or when the connection wasn't
|
||
redirected. It also closes a connection from a source that already has 256
|
||
open, and any connection beyond 2048 in all, so one agent can't use up the
|
||
relay. Otherwise it connects to the destination the agent dialed and sends the bytes
|
||
on, and squid sees an ordinary intercepted connection:
|
||
- the relay connects from an address in `198.18.0.0/15`, which the unit routes
|
||
to `lo`, and an `output` rule DNATs its marked connections (mark
|
||
`0x45434800`) to squid's port 3130, so squid reads the original destination
|
||
from conntrack as before;
|
||
- the source address is `198.18.0.0` plus the low 17 bits of the agent's
|
||
address, and `egress-whois` maps it back to the lease it came from, so the
|
||
pool needs a bridge subnet of /15 or smaller (an assertion checks) and
|
||
squid's log shows `198.18.x.y` as the client address of an intercepted HTTPS
|
||
flow. The user column still names the agent. The `input` chain drops a pool
|
||
address that arrives from the bridge;
|
||
- the relay logs to the journal of its unit, at most one line a second per
|
||
source address, with the agent named the way squid names it (`-` when no
|
||
lease and binding says). A refusal is
|
||
`hive-egress-ech-refused agent=<name> dst=<address>`; an ECH hello of a `log`
|
||
agent, which goes on, is
|
||
`hive-egress-ech-would-refuse agent=<name> dst=<address>`; a hello that isn't
|
||
well formed is
|
||
`hive-egress-hello-refused agent=<name> dst=<address> reason=<why>`. Squid logs
|
||
nothing for a refused flow;
|
||
- the mode comes from `agent-allow.conf`, the include `hive-c0re` renders: an
|
||
agent is in `log` mode when it holds the line
|
||
`acl agent_log_<name> proxy_auth <name>`. Squid applies the same file, so the
|
||
relay and squid can't disagree about an agent's mode, and the file changes
|
||
when the operator changes the mode, with no second list to keep in step. A
|
||
missing or unreadable file, or an agent with no lease, counts as
|
||
`restricted`, and the relay refuses the hello;
|
||
- while the relay is down the redirect reaches a closed port, so HTTPS from
|
||
`restricted` and `log` agents fails. The unit restarts after 5 seconds.
|
||
|
||
GREASE: a client with no ECH configuration, Chrome and Firefox by default,
|
||
sends the extension anyway with random contents, and the protocol makes the two
|
||
look the same on the wire. The relay can't tell them apart, so it treats both
|
||
as ECH: a `restricted` agent's HTTPS fails when the client ignores the proxy
|
||
variables, and a `log` agent's goes on with a would-refuse line. Through the
|
||
proxy port (`HTTPS_PROXY`, a `CONNECT`) it works as before: that path has no
|
||
interception. A host where that costs more than the gap does can set
|
||
`egress.intercept.refuseEch = false`, which sends port 443 straight to squid
|
||
and leaves ECH unchecked. The relay only sees the hello of a connection that
|
||
the redirect catches. It doesn't touch UDP (QUIC), IPv6 or other ports.
|
||
|
||
- **Reading the log.** Each intercepted HTTPS connection logs twice: first
|
||
`NONE_NONE/000 … CONNECT <address>:443 <agent>`, before the ClientHello, then
|
||
the `CONNECT <name>:443 <agent>` line with `TCP_TUNNEL/200` or `TCP_DENIED/403`
|
||
(`NONE_NONE/409` when the SNI and the address disagree). The client address of
|
||
an intercepted HTTPS line is the relay's `198.18.x.y` (see _ECH_ above), and an
|
||
intercepted HTTP request logs `GET http://<name>/ <agent>` with the agent's own
|
||
address. A tunnel to an intercepted
|
||
destination ends in `ORIGINAL_DST/<address>`. `cache.log` notes an ACL that met
|
||
a flow squid refused before its request.
|
||
- **Rollout.** The redirect, the relay, squid's configuration and `hive-priv` ship
|
||
in one build. A hive that still runs an older build has no redirect, and its
|
||
agents go direct as before. The agents that use the proxy variables see no
|
||
change.
|
||
|
||
## Container isolation
|
||
|
||
Each agent container runs in a private network namespace with a dedicated
|
||
veth pair attached to the bridge. The following table summarises what
|
||
the nix side sets up unconditionally:
|
||
|
||
| effect | mechanism |
|
||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
|
||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||
| Loopback DROP | `nix/host-modules/hive-egress.nix` — own nftables table `inet hive_egress` (whichever firewall the host runs): drops bridge → `127.0.0.0/8` and `::1` traffic; the kernel default `route_localnet=0` is the first layer |
|
||
| Forward log | `nix/host-modules/hive-egress.nix` — the same table's `forward` chain logs and counts what the host routes from the bridge to the uplink, and refuses it with `egress.forward = "reject"` (see _Agent bridge egress log_ below) |
|
||
| Source check | `nix/host-modules/hive-egress.nix` — the `bridge hive_egress` table drops a frame whose source MAC or address differs from the sending container's own (see _Agent bridge source check_ below) |
|
||
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
|
||
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
|
||
|
||
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
|
||
`10.42.0.1/24`), **not** the canonical network address. The Rust side
|
||
must normalise (bitwise-AND with mask) before subnet membership checks or
|
||
address arithmetic.
|
||
|
||
### What the Rust side does
|
||
|
||
`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
|
||
`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
|
||
and `HOST_BRIDGE=<bridgeName>` when creating or updating containers:
|
||
`lifecycle::set_nspawn_flags` hands them to hive-priv, which rewrites
|
||
`/etc/nixos-containers/<container>.conf`. hive-c0re validates both variables **once at
|
||
daemon startup**, not per container: they're process-global, so a
|
||
missing or malformed value is a misconfigured daemon rather than one bad
|
||
container, and failing at boot gives a single diagnostic instead of one
|
||
per agent. No non-isolated mode exists to fall back to. hive-c0re leaves `LOCAL_ADDRESS` empty so the
|
||
container's dhcpcd acquires an address from the bridge dnsmasq pool
|
||
(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies to
|
||
every agent container, manager included.
|
||
|
||
`HOST_ADDRESS` is the bridge gateway IP (the address part of
|
||
`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
|
||
so a non-`.1` operator override still resolves to wherever the bridge
|
||
actually lives). It's **load-bearing**: nixos-container's container-side
|
||
network setup only installs a default route (`ip route add default via
|
||
$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode
|
||
nixos-container skips the host-side address/route setup, so writing it only affects the
|
||
container's default route — without it the container comes up with an IP
|
||
but no path off the bridge subnet (no internet, no `api.anthropic.com`).
|
||
|
||
### How the isolated container gets its resolver
|
||
|
||
nixos-container copies the **host's** `/etc/resolv.conf` into the container
|
||
at every start. The host resolver (for example `127.0.0.53` from systemd-resolved,
|
||
or a LAN router) is unreachable from a private netns and isn't
|
||
authoritative for the hive's own zones, so it's replaced with the
|
||
bridge dnsmasq at boot. Because the copy happens on every start, it
|
||
would clobber a declarative `environment.etc."resolv.conf"` — so the
|
||
wiring is runtime:
|
||
|
||
- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
|
||
gateway IP) into each container's `/etc`.
|
||
- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
|
||
marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
|
||
It's ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
|
||
the forge-token fetch so the resolver is correct before the first DNS lookup.
|
||
|
||
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
|
||
domain sockets bind-mounted into containers, not network listeners — see
|
||
the _Control plane (no network)_ bullet under [Network
|
||
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
|
||
that never touches the network stack.
|
||
|
||
The nix side also enables IP forwarding + NAT (agents reach the internet
|
||
through the host) and drops bridge-subnet → loopback traffic from its own
|
||
nftables table (`inet hive_egress`, `nix/host-modules/hive-egress.nix`, which also
|
||
holds the `bridge hive_egress` table of _Agent bridge source check_;
|
||
defence-in-depth against a compromised agent reaching the c0re dashboard HTTP at
|
||
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
|
||
loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||
|
||
### Agent bridge egress log
|
||
|
||
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
||
the `inet hive_egress` table does with traffic the host routes from the agent
|
||
bridge to anywhere but the bridge. It takes
|
||
`accept`, `log` or `reject`; the default is `reject` where the intercept is on
|
||
(see _The default_ below) and `log` elsewhere. In `log` mode the chain blocks
|
||
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
||
also refuses the flow.
|
||
|
||
- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the
|
||
traffic of every MAC in the `direct_mac` set: the
|
||
`unrestricted` agents and the service containers, which the `intercept` chain
|
||
leaves alone too (see _Intercepted web traffic_). They leave no line, so a
|
||
`hive-egress-would-drop` line is exactly a flow that `reject` would refuse,
|
||
and a service container keeps every port and protocol under `reject`.
|
||
- **Named SSH connections.** Right after the `direct_mac` rule the chain
|
||
accepts TCP from an agent's MAC to an address and port in the `agent_allow`
|
||
set, in both modes, so such a flow leaves no count and no log line either.
|
||
The set holds, for each agent whose `ssh` switch is on, the addresses its
|
||
connections' hosts resolve to and their ports (see
|
||
[the SSH CA page](../swarm/ssh-ca.md#egress)). `hive-c0re` renders them into
|
||
`services.hyperhive.deploy.hive-controller.egress.connectionsFile`
|
||
(`/var/lib/hive-squid/ssh-connections`), and `hive-egress-pin` fills the set
|
||
from the file like `direct_mac`, when the file changes and on its timer. The
|
||
set is empty while the file is missing or empty.
|
||
- **What `reject` refuses.** Everything from the bridge to the uplink that the
|
||
chain didn't exempt or allow, whether a flow is new or not: TCP on any port
|
||
but the 80 and 443 that squid takes, UDP (DNS to an outside resolver, QUIC
|
||
and NTP included), ICMP and IPv6. A TCP flow gets a reset, so the connect
|
||
fails at once instead of stalling. Everything else gets ICMP administratively
|
||
prohibited. The log lines of this mode carry the prefix `hive-egress-reject `
|
||
(and `hive-egress-reject-unmetered `); the rate limit and the counter work as
|
||
in `log` mode. Still open to every agent are the host's own addresses: the
|
||
proxy ports, DNS and DHCP from the bridge address, and the gateway.
|
||
- **The default.** `reject` while `egress.intercept.enable` is on, which
|
||
`hive-squid.nix` turns on for a hive controller host, and `log` on a host
|
||
without the intercept, because `reject` there would refuse all web traffic
|
||
too. The soak ran with `log` first: the `hive-egress-would-drop` lines are
|
||
each a flow that `reject` would refuse, and a flow that belongs to a
|
||
legitimate use needed a fix before the switch (a service container missing
|
||
from `direct_mac`, an agent that should be `unrestricted`). To go back to
|
||
logging, set
|
||
`services.hyperhive.deploy.hive-controller.egress.forward = "log"`.
|
||
- **What `reject` can break.**
|
||
- A service container with no pin yet isn't in `direct_mac`, so the
|
||
chain refuses it until `hive-egress-pin` binds its port. The bridge table
|
||
drops its frames until then anyway.
|
||
- A change of an agent's `unrestricted` flag reaches `direct_mac` within 30
|
||
seconds at most, on the timer of `hive-egress-pin`. The chain refuses the
|
||
agent in the gap.
|
||
- An unreadable `unrestricted` file lists no agent, so `unrestricted` agents
|
||
lose every non-web flow.
|
||
- A new named connection, or a flipped `ssh` switch, reaches the set about a
|
||
minute after the write: the hive reads the swarm every 30 seconds and the
|
||
file's change runs a pass at once. A host that moves to an address the hive
|
||
hasn't looked up yet gets refusals for up to ten minutes. An agent whose
|
||
connection names a host that never resolved has no allowance for it.
|
||
- An agent in per-agent `log` mode keeps its web traffic, which squid only
|
||
observes, and loses every other port and protocol.
|
||
- **Which traffic.** IPv4 and IPv6 alike (the table is `inet`). The chain never
|
||
sees traffic to the host's own addresses, because the host delivers it
|
||
locally: a connection to squid on `<bridgeIp>:3128`, to dnsmasq or to the
|
||
gateway leaves no line. Nor does anything squid or the host opens itself. The
|
||
web traffic of a `restricted` or `log` agent that the `intercept` chain
|
||
redirects into squid reaches the host too, so squid's log shows it instead
|
||
(see _Intercepted web traffic_).
|
||
- **The log line.** The kernel log carries one line per new flow, with the
|
||
prefix `hive-egress-would-drop ` followed by the usual netfilter fields
|
||
(`IN`, `OUT`, `SRC`, `DST`, `PROTO` and the ports). Read them with
|
||
`journalctl -k -g hive-egress-would-drop`.
|
||
- **Rate limit.** Each source address gets a burst of 20 lines, then one line a
|
||
second, and the chain forgets it after a minute without a new flow. A flood
|
||
from one agent can't hide another's flows, and it adds about 250 bytes of
|
||
journal a second per source address at most. A flow the per-source limit
|
||
can't log (the address is over its rate, or the table of addresses is full)
|
||
gets one line with the prefix `hive-egress-would-drop-unmetered ` instead,
|
||
from a limit shared by all addresses: 5 lines a second after a burst of 50,
|
||
about 100 MB of journal a day at most. A flow never gets both lines. The
|
||
counter in `nft list chain inet hive_egress forward` counts every packet,
|
||
logged or not.
|
||
- **Which agent.** `SRC` is the agent's address on the bridge. Containers get
|
||
theirs from the hive's dnsmasq, which records each lease in
|
||
`/var/lib/dnsmasq/dnsmasq.leases`, one line per lease:
|
||
`<expiry> <mac> <ip> <hostname> <client-id>`. Look the `SRC` address up in the
|
||
third column; the hostname names the container. A lease lasts one hour, so
|
||
look an address up soon after the line appears.
|
||
- **Trust.** A frame that carries another container's source address never
|
||
reaches the forward chain while `antiSpoof` is `drop` (see _Agent bridge
|
||
source check_), so `SRC` names the container that sent the flow. In `log`
|
||
mode a container can still forge it, so a log from a host that ran in `log`
|
||
mode during the soak doesn't show who sent a flow.
|
||
|
||
### Agent bridge source check
|
||
|
||
Every container on the agent bridge is root in its own network namespace and
|
||
shares one L2 segment with the others, so it can send with another container's
|
||
MAC or address. `services.hyperhive.deploy.hive-controller.egress.antiSpoof` sets what the
|
||
`bridge hive_egress` table (`nix/host-modules/hive-egress.nix`) does about it. It
|
||
takes `drop` or `log` (the default, until the lease set survives a host suspend/resume). Two tables, not one: the bridge carries traffic between two
|
||
ports, and the `inet` hooks never see it as coming from a port. Only a
|
||
`bridge`-family hook at `prerouting` sees which veth a frame came in on, so the
|
||
source check lives in that table and the host-side filtering stays in
|
||
`inet hive_egress`.
|
||
|
||
- **What a port may send.** The frame's MAC is the one bound to the port.
|
||
Then, by protocol: IPv4 from an address leased to that MAC, or a DHCP client
|
||
message (UDP 68 to 67) to `255.255.255.255` or the bridge address, from any
|
||
source address, as long as its `chaddr` is the port's MAC; ARP whose sender is the
|
||
port's MAC and an address leased to it, or an ARP probe from `0.0.0.0`; IPv6
|
||
from a link-local address (and from `::` in neighbour solicitations and listener reports, which
|
||
precede an address). A message to UDP port 67 must carry the port's own MAC as
|
||
its `chaddr`, so a container can't lease, renew or release an address for
|
||
another. The table refuses a VLAN tag, and everything else counts as a spoof.
|
||
- **A port nobody owns sends nothing.** Not even DHCP. A container's port is
|
||
bound a moment after its veth appears, and the container's DHCP client
|
||
retries until then.
|
||
- **Where the bindings come from.** Addresses are dynamic, so the table's sets
|
||
fill at run time. `hive-priv egress-pin` binds a port to the MAC of the link at
|
||
the other end of its veth pair, found through the container systemd registers
|
||
for it. udev starts it for each new `vb-*` port
|
||
(`hive-egress-pin@<port>.service`), and a timer (`hive-egress-pin.timer`, every
|
||
30 seconds) runs it for the ports that still have no binding. The timer reads
|
||
no container while every port has one. One pass at a time reads and writes the
|
||
bindings, under a lock. A pass waits 30 seconds for the lock and gives up with
|
||
`lock held by another pass for 30 s; giving up`, and `ip`, `nsenter` and `nft`
|
||
each get 10 seconds before the pass kills them and fails. Both units also have
|
||
`TimeoutStartSec=90s`. A pass that stops responding fails visibly and the
|
||
next run of the timer or udev starts again, instead of stopping every later
|
||
pass. The
|
||
`hive-egress` unit loads the tables
|
||
with `hive-priv egress-load`: it looks up the bindings first, while the old
|
||
tables still filter, and puts them and the leases from dnsmasq's lease file
|
||
into the same nft transaction as the tables. A start or reload never leaves
|
||
the bridge with empty sets. If loading the leases fails, the unit logs a
|
||
warning and loads the tables with the bindings alone. A binding stays with the
|
||
port until the port goes: root in the container can't move the port to another
|
||
MAC afterwards (state in `/run/hive-egress-pins`). A port never takes a MAC
|
||
that a binding already holds, so a newcomer can't evict an established
|
||
container. If two new ports claim one MAC, neither gets a binding. dnsmasq's
|
||
`dhcp-script` (`hive-priv egress-lease`) adds each lease as a (MAC, address)
|
||
pair that times out with the lease plus 5 minutes. `script-on-renewal` makes
|
||
dnsmasq run it on every renewal too, which loads the pair again with the
|
||
new remaining time; without it a pair would leave the table when the first
|
||
lease ended, while dnsmasq kept renewing the lease.
|
||
A client whose pair has left the set can still renew or rebind: the DHCP
|
||
client message passes from the leased address too, and dnsmasq's reply puts
|
||
the pair back.
|
||
dnsmasq counts lease time on the wall clock, and the pair's timeout and the
|
||
container's own lease count awake time. After a suspend dnsmasq expires every
|
||
lease at once and runs `egress-lease del` for each; a `del` whose
|
||
`DNSMASQ_LEASE_EXPIRES` lies in the past does nothing, and the pair leaves the
|
||
table at its own timeout, as the container's lease ends. A `del` for a lease
|
||
that hasn't expired (a release) deletes the pair at once, as does one without
|
||
a readable expiry. `egress-lease` logs each `add`, `old` and `del` to the
|
||
journal under the identifier `egress-lease`, at info priority, with the prefix
|
||
`egress-lease:`, the MAC, address, port, expiry, and for `del` whether it was
|
||
`applied` or `skipped (stale)`. It also prints the line on stderr, which
|
||
dnsmasq logs at debug priority as `dnsmasq-script`. The log shipper reads info
|
||
and above, so search the store for the `egress-lease` identifier, not for the
|
||
dnsmasq copy. The journal send waits at most a second for room in journald's
|
||
queue, because dnsmasq runs its lease scripts one at a time. A send that
|
||
fails or times out costs that one journal line and nothing else.
|
||
- **The window.** The binding takes the MAC the container's link has when the
|
||
command first looks, normally within a second of the veth appearing. Until
|
||
then root in the container could change its own MAC, but in that time only
|
||
the container's early init runs, which is nix-built, and the agent's code
|
||
starts later. A container that did change its MAC first gets a binding for
|
||
that MAC only if nobody holds it; a lease follows the `chaddr` the port may
|
||
send. A port refused because it claimed a held MAC sends nothing, and the
|
||
timer binds it once its link has a MAC of its own.
|
||
- **The log.** A refused frame (all of them with `drop`, the same ones with
|
||
`log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate
|
||
limited per port: a burst of 20, then one a second. A frame its port can't log
|
||
gets a line with the prefix `hive-egress-spoof-unmetered ` instead, from one
|
||
limit shared by all ports and reasons: 5 lines a second after a burst of 50.
|
||
Read them with `journalctl -k -g hive-egress-spoof`. The word after the
|
||
prefix is the check that refused the frame: `mac` (the port isn't bound to
|
||
the source MAC), `vlan`, `dhcp` (the `chaddr` isn't the port's), `v4`, `v6` or
|
||
`arp`; each has its own chain, `spoof_<reason>`, with its own counters. `IN=`
|
||
is the port, `vb-<machine>` (`vb-h-<agent>` for an agent), and `SRC` and
|
||
`MAC=<dst>:<src>:<type>` are what the container sent; the port names the
|
||
sender, because a bound port sends only its own MAC. The counters in
|
||
`nft list chain bridge hive_egress spoof` count every refused frame.
|
||
- **Why `log` is the default.** A container that sends from its leased address
|
||
and its own MAC is never refused by `drop`, but only while the pair stays in
|
||
the table, and a host suspend/resume empties it. `log` keeps the lines and
|
||
the counters and blocks nothing; `drop` returns once a resume test proves the
|
||
lease set survives it and the `egress-lease: … skipped (stale)` lines confirm
|
||
the renewal fix on a live hive. `drop`
|
||
is what the table is for: without it the egress log can't tell agents apart.
|
||
A binding that failed shows as `hive-egress-pin` errors in the journal and
|
||
`IN=` lines for that port.
|
||
- **A port that stays unbound.** The timer's run of `hive-egress-pin` says why
|
||
a port is still unbound after a minute (a port is unbound for a moment at
|
||
every container start, so it waits). `journalctl -u hive-egress-pin` then
|
||
shows `<port> (ifindex N, peer M) has no owner; machines seen: <name>=<index>/<peer>, …`
|
||
when no container's link pairs with the port, or `… stays unbound: <reason>`
|
||
when one does. A `vb-` veth that isn't on the bridge gets a line the same
|
||
way (`<veth> is not on hive-br0`), and so does a registered machine the run
|
||
couldn't read (`machine <name> is skipped: <reason>`). Each line comes once,
|
||
and again only after it has cleared (the run keeps
|
||
`/run/hive-egress-pins.diag`). None of them changes which ports get a binding.
|
||
- **What it doesn't cover.** A container can still send from any IPv6
|
||
link-local address and answer DHCP as a server on its own leased address. A
|
||
DHCP client message to the broadcast or bridge address carries any IPv4 source
|
||
address; the table looks only at its `chaddr`.
|
||
|
||
Enforcement comes with later changes in the same series: the log only shows
|
||
what such a change would cut.
|
||
|
||
## Cross-references
|
||
|
||
- `docs/4-implementation/networking/gateway.md` — vhost map + the gateway's other duties
|