mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agent bridge: drop frames whose source MAC or address isn't the container's own (egress.antiSpoof)

Browse source
This commit is contained in:
atlas 2026-11-13 02:36:38 +01:00 • committed by mara
commit c4ac01ef88
1 changed files with 636 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

77
docs/4-implementation/networking/network.md
View file

@ -519,6 +519,7 @@ the nix side sets up unconditionally:
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `nix/host-modules/hive-egress.nix` — own nftables table `inet hive_egress` (whichever firewall the host runs): drops bridge → `127.0.0.0/8` and `::1` traffic; the kernel default `route_localnet=0` is the first layer |
| Forward log | `nix/host-modules/hive-egress.nix` — the same table's `forward` chain logs and counts what the host routes from the bridge to the uplink; it blocks nothing (see _Agent bridge egress log_ below) |
| Source check | `nix/host-modules/hive-egress.nix` — the `bridge hive_egress` table drops a frame whose source MAC or address differs from the sending container's own (see _Agent bridge source check_ below) |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
@ -577,7 +578,8 @@ that never touches the network stack.
The nix side also enables IP forwarding + NAT (agents reach the internet
through the host) and drops bridge-subnet → loopback traffic from its own
nftables table (`inet hive_egress`, `nix/host-modules/hive-egress.nix`;
nftables table (`inet hive_egress`, `nix/host-modules/hive-egress.nix`, which also
holds the `bridge hive_egress` table of _Agent bridge source check_;
defence-in-depth against a compromised agent reaching the c0re dashboard HTTP at
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
loopback — the hive-c0re admin socket is a UDS, not TCP.
@ -616,14 +618,71 @@ and counts every packet.
`<expiry> <mac> <ip> <hostname> <client-id>`. Look the `SRC` address up in the
third column; the hostname names the container. A lease lasts one hour, so
look an address up soon after the line appears.
- **A hint, not evidence.** Until anti-spoofing lands, a container that forges
its source address can mislead the log. It can file its flows under another
agent's address, use up that agent's burst, or fill the table of addresses so
that its own flows only reach the shared limit. It can also influence the
lease behind the address. Use the log to find out what agents send directly
now. Don't treat it as proof of which agent sent a flow, or as proof that a
flow is absent. Anti-spoofing has to land before the log serves as the soak
exit for enforcement.
- **Trust.** A frame that carries another container's source address never
reaches the forward chain while `antiSpoof` is `drop` (see _Agent bridge
source check_), so `SRC` names the container that sent the flow. In `log`
mode a container can still forge it, so a log from a host that ran in `log`
mode during the soak doesn't show who sent a flow.
### Agent bridge source check
Every container on the agent bridge is root in its own network namespace and
shares one L2 segment with the others, so it can send with another container's
MAC or address. `services.hyperhive.network.egress.antiSpoof` sets what the
`bridge hive_egress` table (`nix/host-modules/hive-egress.nix`) does about it. It
takes `drop` (the default) or `log`. Two tables, not one: the bridge carries traffic between two
ports, and the `inet` hooks never see it as coming from a port. Only a
`bridge`-family hook at `prerouting` sees which veth a frame came in on, so the
source check lives in that table and the host-side filtering stays in
`inet hive_egress`.
- **What a port may send.** The frame's MAC is the one bound to the port.
Then, by protocol: IPv4 from an address leased to that MAC, or DHCP from
`0.0.0.0` to `255.255.255.255`; ARP whose sender is the port's MAC and an
address leased to it, or an ARP probe from `0.0.0.0`; IPv6 from a link-local
address (and from `::` in neighbour solicitations and listener reports, which
precede an address). A message to UDP port 67 must carry the port's own MAC as
its `chaddr`, so a container can't lease, renew or release an address for
another. The table refuses a VLAN tag, and everything else counts as a spoof.
- **A port nobody owns sends nothing.** Not even DHCP. A container's port is
bound a moment after its veth appears, and the container's DHCP client
retries until then.
- **Where the bindings come from.** Addresses are dynamic, so the table's sets
the commands below fill them at run time. `hive-priv egress-pin` binds a port to the MAC of the
link at the other end of its veth pair, found through the container systemd
registers for it. udev starts it for each new `vb-*` port
(`hive-egress-pin@<port>.service`), and the `hive-egress` unit starts it for
every running container on each start and reload. A binding stays with the
port until the port goes: root in the container can't move the port to
another MAC afterwards (state in `/run/hive-egress-pins`). If two ports have
the same MAC, neither gets a binding. dnsmasq's `dhcp-script` (`hive-priv
egress-lease`) adds each lease as a (MAC, address) pair that times out with
the lease.
- **The window.** The binding takes the MAC the container's link has when the
command first looks, normally within a second of the veth appearing. Until
then root in the container could change its own MAC, but in that time only
the container's early init runs, which is nix-built, and the agent's code
starts later. A container that did change its MAC first would get its own
lease for the new MAC, not another container's: a MAC that two ports have
gets no binding, and a lease follows the `chaddr` the port may send.
- **The log.** A refused frame (all of them with `drop`, the same ones with
`log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate
limited per port: a burst of 20, then one a second. A frame its port can't log
gets a line with the prefix `hive-egress-spoof-unmetered ` instead, from a limit
shared by all ports: 5 lines a second after a burst of 50. Read them with
`journalctl -k -g hive-egress-spoof`. `IN=` is the port, `vb-<machine>` (`vb-h-<agent>` for
an agent), and `SRC` and `MAC=<dst>:<src>:<type>` are what the container
sent; the port names the sender, because a bound port sends only its own MAC.
The counters in `nft list chain bridge hive_egress spoof` count every refused
frame.
- **Why `drop` is the default.** A container that sends from its leased address
and its own MAC is never refused, so `drop` breaks nothing a container does
legitimately, and without it the egress log can't tell agents apart. `log` is
for a live hive where the table refuses a legitimate container (a binding that
failed shows as `hive-egress-pin` errors in the journal and `IN=` lines for
that port): the same lines and counters, nothing blocked.
- **What it doesn't cover.** A container can still send from any IPv6
link-local address and answer DHCP as a server on its own leased address.
Enforcement comes with later changes in the same series: the log only shows
what such a change would cut.