agent bridge: drop frames whose source MAC or address isn't the container's own (egress.antiSpoof)
This commit is contained in:
atlas
mara
parent
c3f1ef6eb2
commit
c4ac01ef88
1 changed files with 636 additions and 25 deletions
|
|
@ -519,6 +519,7 @@ the nix side sets up unconditionally:
|
|||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||||
| Loopback DROP | `nix/host-modules/hive-egress.nix` — own nftables table `inet hive_egress` (whichever firewall the host runs): drops bridge → `127.0.0.0/8` and `::1` traffic; the kernel default `route_localnet=0` is the first layer |
|
||||
| Forward log | `nix/host-modules/hive-egress.nix` — the same table's `forward` chain logs and counts what the host routes from the bridge to the uplink; it blocks nothing (see _Agent bridge egress log_ below) |
|
||||
| Source check | `nix/host-modules/hive-egress.nix` — the `bridge hive_egress` table drops a frame whose source MAC or address differs from the sending container's own (see _Agent bridge source check_ below) |
|
||||
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
|
||||
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
|
||||
|
||||
|
|
@ -577,7 +578,8 @@ that never touches the network stack.
|
|||
|
||||
The nix side also enables IP forwarding + NAT (agents reach the internet
|
||||
through the host) and drops bridge-subnet → loopback traffic from its own
|
||||
nftables table (`inet hive_egress`, `nix/host-modules/hive-egress.nix`;
|
||||
nftables table (`inet hive_egress`, `nix/host-modules/hive-egress.nix`, which also
|
||||
holds the `bridge hive_egress` table of _Agent bridge source check_;
|
||||
defence-in-depth against a compromised agent reaching the c0re dashboard HTTP at
|
||||
`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
|
||||
loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||||
|
|
@ -616,14 +618,71 @@ and counts every packet.
|
|||
`<expiry> <mac> <ip> <hostname> <client-id>`. Look the `SRC` address up in the
|
||||
third column; the hostname names the container. A lease lasts one hour, so
|
||||
look an address up soon after the line appears.
|
||||
- **A hint, not evidence.** Until anti-spoofing lands, a container that forges
|
||||
its source address can mislead the log. It can file its flows under another
|
||||
agent's address, use up that agent's burst, or fill the table of addresses so
|
||||
that its own flows only reach the shared limit. It can also influence the
|
||||
lease behind the address. Use the log to find out what agents send directly
|
||||
now. Don't treat it as proof of which agent sent a flow, or as proof that a
|
||||
flow is absent. Anti-spoofing has to land before the log serves as the soak
|
||||
exit for enforcement.
|
||||
- **Trust.** A frame that carries another container's source address never
|
||||
reaches the forward chain while `antiSpoof` is `drop` (see _Agent bridge
|
||||
source check_), so `SRC` names the container that sent the flow. In `log`
|
||||
mode a container can still forge it, so a log from a host that ran in `log`
|
||||
mode during the soak doesn't show who sent a flow.
|
||||
|
||||
### Agent bridge source check
|
||||
|
||||
Every container on the agent bridge is root in its own network namespace and
|
||||
shares one L2 segment with the others, so it can send with another container's
|
||||
MAC or address. `services.hyperhive.network.egress.antiSpoof` sets what the
|
||||
`bridge hive_egress` table (`nix/host-modules/hive-egress.nix`) does about it. It
|
||||
takes `drop` (the default) or `log`. Two tables, not one: the bridge carries traffic between two
|
||||
ports, and the `inet` hooks never see it as coming from a port. Only a
|
||||
`bridge`-family hook at `prerouting` sees which veth a frame came in on, so the
|
||||
source check lives in that table and the host-side filtering stays in
|
||||
`inet hive_egress`.
|
||||
|
||||
- **What a port may send.** The frame's MAC is the one bound to the port.
|
||||
Then, by protocol: IPv4 from an address leased to that MAC, or DHCP from
|
||||
`0.0.0.0` to `255.255.255.255`; ARP whose sender is the port's MAC and an
|
||||
address leased to it, or an ARP probe from `0.0.0.0`; IPv6 from a link-local
|
||||
address (and from `::` in neighbour solicitations and listener reports, which
|
||||
precede an address). A message to UDP port 67 must carry the port's own MAC as
|
||||
its `chaddr`, so a container can't lease, renew or release an address for
|
||||
another. The table refuses a VLAN tag, and everything else counts as a spoof.
|
||||
- **A port nobody owns sends nothing.** Not even DHCP. A container's port is
|
||||
bound a moment after its veth appears, and the container's DHCP client
|
||||
retries until then.
|
||||
- **Where the bindings come from.** Addresses are dynamic, so the table's sets
|
||||
the commands below fill them at run time. `hive-priv egress-pin` binds a port to the MAC of the
|
||||
link at the other end of its veth pair, found through the container systemd
|
||||
registers for it. udev starts it for each new `vb-*` port
|
||||
(`hive-egress-pin@<port>.service`), and the `hive-egress` unit starts it for
|
||||
every running container on each start and reload. A binding stays with the
|
||||
port until the port goes: root in the container can't move the port to
|
||||
another MAC afterwards (state in `/run/hive-egress-pins`). If two ports have
|
||||
the same MAC, neither gets a binding. dnsmasq's `dhcp-script` (`hive-priv
|
||||
egress-lease`) adds each lease as a (MAC, address) pair that times out with
|
||||
the lease.
|
||||
- **The window.** The binding takes the MAC the container's link has when the
|
||||
command first looks, normally within a second of the veth appearing. Until
|
||||
then root in the container could change its own MAC, but in that time only
|
||||
the container's early init runs, which is nix-built, and the agent's code
|
||||
starts later. A container that did change its MAC first would get its own
|
||||
lease for the new MAC, not another container's: a MAC that two ports have
|
||||
gets no binding, and a lease follows the `chaddr` the port may send.
|
||||
- **The log.** A refused frame (all of them with `drop`, the same ones with
|
||||
`log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate
|
||||
limited per port: a burst of 20, then one a second. A frame its port can't log
|
||||
gets a line with the prefix `hive-egress-spoof-unmetered ` instead, from a limit
|
||||
shared by all ports: 5 lines a second after a burst of 50. Read them with
|
||||
`journalctl -k -g hive-egress-spoof`. `IN=` is the port, `vb-<machine>` (`vb-h-<agent>` for
|
||||
an agent), and `SRC` and `MAC=<dst>:<src>:<type>` are what the container
|
||||
sent; the port names the sender, because a bound port sends only its own MAC.
|
||||
The counters in `nft list chain bridge hive_egress spoof` count every refused
|
||||
frame.
|
||||
- **Why `drop` is the default.** A container that sends from its leased address
|
||||
and its own MAC is never refused, so `drop` breaks nothing a container does
|
||||
legitimately, and without it the egress log can't tell agents apart. `log` is
|
||||
for a live hive where the table refuses a legitimate container (a binding that
|
||||
failed shows as `hive-egress-pin` errors in the journal and `IN=` lines for
|
||||
that port): the same lines and counters, nothing blocked.
|
||||
- **What it doesn't cover.** A container can still send from any IPv6
|
||||
link-local address and answer DHCP as a server on its own leased address.
|
||||
|
||||
Enforcement comes with later changes in the same series: the log only shows
|
||||
what such a change would cut.
|
||||
|
|
|
|||
Loading…
Reference in a new issue