diff --git a/docs/4-implementation/networking/network.md b/docs/4-implementation/networking/network.md index 2c69fd7dd..80180dbb0 100644 --- a/docs/4-implementation/networking/network.md +++ b/docs/4-implementation/networking/network.md @@ -234,8 +234,9 @@ when it speaks HTTP on port 80 or TLS on port 443 and its agent is `restricted` or in `log` mode: the host redirects that traffic into squid's intercept ports (see _Intercepted web traffic_ below). Everything else goes straight out through the NAT, as before: the `unrestricted` agents, the service containers and every -other port and protocol. Nothing blocks that yet. The host logs those direct -connections (see _Agent bridge egress log_ below). +other port and protocol. By default nothing blocks that. The host logs the direct +connections of the `restricted` and `log` agents, and `egress.forward = +"reject"` refuses them (see _Agent bridge egress log_ below). Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP host. It never decrypts TLS. The one TLS key it loads is a throwaway that the @@ -651,15 +652,15 @@ Each agent container runs in a private network namespace with a dedicated veth pair attached to the bridge. The following table summarises what the nix side sets up unconditionally: -| effect | mechanism | -| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` | -| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC | -| Loopback DROP | `nix/host-modules/hive-egress.nix` — own nftables table `inet hive_egress` (whichever firewall the host runs): drops bridge → `127.0.0.0/8` and `::1` traffic; the kernel default `route_localnet=0` is the first layer | -| Forward log | `nix/host-modules/hive-egress.nix` — the same table's `forward` chain logs and counts what the host routes from the bridge to the uplink; it blocks nothing (see _Agent bridge egress log_ below) | -| Source check | `nix/host-modules/hive-egress.nix` — the `bridge hive_egress` table drops a frame whose source MAC or address differs from the sending container's own (see _Agent bridge source check_ below) | -| Gateway access | `networking.firewall.interfaces..allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host | -| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them | +| effect | mechanism | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` | +| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC | +| Loopback DROP | `nix/host-modules/hive-egress.nix` — own nftables table `inet hive_egress` (whichever firewall the host runs): drops bridge → `127.0.0.0/8` and `::1` traffic; the kernel default `route_localnet=0` is the first layer | +| Forward log | `nix/host-modules/hive-egress.nix` — the same table's `forward` chain logs and counts what the host routes from the bridge to the uplink, and refuses it with `egress.forward = "reject"` (see _Agent bridge egress log_ below) | +| Source check | `nix/host-modules/hive-egress.nix` — the `bridge hive_egress` table drops a frame whose source MAC or address differs from the sending container's own (see _Agent bridge source check_ below) | +| Gateway access | `networking.firewall.interfaces..allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host | +| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them | `HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g. `10.42.0.1/24`), **not** the canonical network address. The Rust side @@ -727,12 +728,42 @@ loopback — the hive-c0re admin socket is a UDS, not TCP. `services.hyperhive.network.egress.forward` sets what the `forward` chain of the `inet hive_egress` table does with traffic the host routes from the agent bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes -`accept`, `log` (the default) or `drop`. The evaluation refuses `drop` for now: -the traffic that squid doesn't handle (the unrestricted agents, the service -containers, every port but 80 and 443) has no rule that lets it through yet. In -`log` mode the chain blocks nothing. It logs each new flow -and counts every packet. +`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks +nothing. It logs each new flow and counts every packet. In `reject` mode it +also refuses the flow. +- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the + mesh traffic first, then the traffic of every MAC in the `direct_mac` set: the + `unrestricted` agents and the service containers, which the `intercept` chain + leaves alone too (see _Intercepted web traffic_). They leave no line, so a + `hive-egress-would-drop` line is exactly a flow that `reject` would refuse, + and a service container keeps every port and protocol under `reject`. +- **What `reject` refuses.** Everything from the bridge to the uplink that the + chain didn't exempt, whether a flow is new or not: TCP on any port but the + 80 and 443 that squid takes, UDP (DNS to an outside resolver, QUIC and NTP + included), ICMP and IPv6. A TCP flow gets a reset, so the connect fails at + once instead of stalling. Everything else gets ICMP administratively + prohibited. The log lines of this mode carry the prefix `hive-egress-reject ` + (and `hive-egress-reject-unmetered `); the rate limit and the counter work as + in `log` mode. Still open to every agent are the host's own addresses: the + proxy ports, DNS and DHCP from the bridge address, and the gateway. +- **Switching on.** Leave `log` running for about a week, then read the + `hive-egress-would-drop` lines. Each is a flow that `reject` would refuse; a + flow that belongs to a legitimate use needs a fix first (a service container + missing from `direct_mac`, an agent that should be `unrestricted`). Then set + `services.hyperhive.network.egress.forward = "reject"`. Making it the default + is a one-line change of the option's default. +- **What `reject` can break.** + - A service container with no pin yet isn't in `direct_mac`, so the + chain refuses it until `hive-egress-pin` binds its port. The bridge table + drops its frames until then anyway. + - A change of an agent's `unrestricted` flag reaches `direct_mac` within 30 + seconds at most, on the timer of `hive-egress-pin`. The chain refuses the + agent in the gap. + - An unreadable `unrestricted` file lists no agent, so `unrestricted` agents + lose every non-web flow. + - An agent in per-agent `log` mode keeps its web traffic, which squid only + observes, and loses every other port and protocol. - **Which traffic.** IPv4 and IPv6 alike (the table is `inet`). The chain never sees traffic to the host's own addresses, because the host delivers it locally: a connection to squid on `:3128`, to dnsmasq or to the diff --git a/nix/host-modules/hive-egress.nix b/nix/host-modules/hive-egress.nix index 117989f63..a82b82145 100644 --- a/nix/host-modules/hive-egress.nix +++ b/nix/host-modules/hive-egress.nix @@ -9,14 +9,14 @@ # loopback is dropped. The kernel does the same by default (a packet for # 127.0.0.0/8 arriving on a non-loopback interface is a martian while # `route_localnet` is 0); this rule is the second layer, and the assertion -# below keeps the first one on. `forward` (`egress.forward = "log"`): traffic -# the host routes from the agent bridge to anywhere else is logged and -# counted, never blocked. `intercept` (`egress.intercept.enable`, set by +# below keeps the first one on. `forward` (`egress.forward`): traffic from the +# agent bridge to anywhere else is logged and counted (`log`) or logged and +# refused (`reject`). `intercept` (`egress.intercept.enable`, set by # hive-squid.nix): the web traffic of the bridge's agents is redirected into -# squid's intercept ports (port 443 through the ClientHello relay). The ports in -# `direct_mac` are left alone: the service containers and the unrestricted -# agents. `hive-priv egress-pin` and `egress-load` fill that set from the bridge -# table's bindings, so a MAC in no binding is intercepted, and squid refuses it. +# squid's intercept ports (port 443 through the ClientHello relay). Both leave +# alone the ports in `direct_mac`, the service containers and the unrestricted +# agents. `hive-priv egress-pin` and `egress-load` fill that set from the +# bridge table's bindings, so a MAC in no binding is intercepted and refused. # # The `bridge` table checks the source address of every frame a port of the # agent bridge sends (`egress.antiSpoof`). It is a `bridge`-family table because @@ -79,7 +79,7 @@ let # Logging is per source address, so one busy agent cannot use up the budget # of another: a burst of 20 lines, then one a second, and an address is - # forgotten after a minute without a new flow. The rule accepts after it + # forgotten after a minute without a new flow. The rule decides after it # logs. A flow the rule cannot log falls through to the unmetered rule below # it: the address was over its rate, or the set is full (`update` then fails # and the rule stops matching), which a container spoofing source addresses @@ -90,9 +90,55 @@ let logRate = "1/second burst 20 packets"; unmeteredRate = "5/second burst 50 packets"; fromBridge = ''iifname "${br}" oifname != "${br}"''; - logRule = + + # `log` accepts what it logs. `reject` logs and refuses in one rule, so a + # line always names a refused flow. A TCP flow gets a reset, which fails the + # connect at once (to a TCP socket an ICMP error is a soft error, and the + # connect would stall); anything else gets ICMP admin-prohibited. A rule that + # refuses is therefore a pair: one for TCP, one for the rest. + isReject = cfg.egress.forward == "reject"; + logPrefix = if isReject then "hive-egress-reject" else "hive-egress-would-drop"; + refuse = { + tcp = "meta l4proto tcp"; + other = "meta l4proto != tcp"; + }; + verdict = { + tcp = "reject with tcp reset"; + other = "reject with icmpx type admin-prohibited"; + }; + decide = + statements: + if isReject then + map + ( + k: + lib.concatStringsSep " " ( + lib.filter (x: x != "") [ + fromBridge + refuse.${k} + statements + verdict.${k} + ] + ) + ) + [ + "tcp" + "other" + ] + else + [ "${fromBridge} ${statements} accept" ]; + logRules = family: addr: - ''${fromBridge} ct state new update @log_${family} { ${addr} saddr limit rate ${logRate} } log prefix "hive-egress-would-drop " accept''; + decide ''ct state new update @log_${family} { ${addr} saddr limit rate ${logRate} } log prefix "${logPrefix} "''; + # In `log` mode the unmetered rule has no verdict: the chain's last rule is + # the verdict for every flow that gets past the logging. + unmeteredRules = + let + statements = ''ct state new limit rate ${unmeteredRate} log prefix "${logPrefix}-unmetered "''; + in + if isReject then decide statements else [ "${fromBridge} ${statements}" ]; + # What no rule above decided, a packet that is no new flow included. + finalRules = if isReject then decide "" else [ "${fromBridge} accept" ]; # IPv4 TCP from the agent bridge to ports 80 and 443. `fib daddr oifname` is # the interface the route to the destination leaves by: the bridge (another @@ -124,9 +170,12 @@ let # Priority `filter - 10` runs before the NixOS firewall's forward handling, # which sits at `filter` on both backends (`FORWARD` of the iptables backend, # the `forward` chain of `nixos-fw`). An `accept` here ends this chain only, - # so the chain neither allows what those refuse nor refuses anything itself. - # Traffic to the host's own addresses (squid, dnsmasq, the gateway) is - # delivered locally and never reaches the forward hook. + # so it allows nothing those refuse; a `reject` ends the packet. The mesh + # and the MACs in `direct_mac` (the unrestricted agents and the service + # containers, the traffic the intercept chain leaves alone) come first in + # both modes, so a `log` line names what `reject` would refuse. Traffic to + # the host's own addresses (squid, dnsmasq, the gateway) is delivered + # locally and never reaches the forward hook. forwardChain = '' set log_v4 { type ipv4_addr; flags dynamic; timeout 1m; size 4096; } set log_v6 { type ipv6_addr; flags dynamic; timeout 1m; size 4096; } @@ -134,11 +183,11 @@ let chain forward { type filter hook forward priority filter - 10; policy accept; iifname "${br}" oifname "${meshInterface}" accept + ${fromBridge} ether saddr @direct_mac accept ${fromBridge} counter - ${logRule "v4" "ip"} - ${logRule "v6" "ip6"} - ${fromBridge} ct state new limit rate ${unmeteredRate} log prefix "hive-egress-would-drop-unmetered " - ${fromBridge} accept + ${lib.concatStringsSep "\n " ( + logRules "v4" "ip" ++ logRules "v6" "ip6" ++ unmeteredRules ++ finalRules + )} } ''; @@ -232,7 +281,7 @@ let } set direct_mac { type ether_addr; } ${lib.optionalString cfg.egress.intercept.enable interceptChain} - ${lib.optionalString (cfg.egress.forward == "log") forwardChain} + ${lib.optionalString (cfg.egress.forward != "accept") forwardChain} } ${bridgeTable} @@ -257,20 +306,27 @@ in type = lib.types.enum [ "accept" "log" - "drop" + "reject" ]; default = "log"; description = '' What the `inet hive_egress` table does with traffic the host forwards - from the agent bridge to anywhere but the bridge and the mesh. + from the agent bridge to anywhere but the bridge and the mesh. In `log` and + `reject` the traffic of the MACs in the `direct_mac` set (the unrestricted + agents and the service containers) is accepted and never logged. - `accept`: nothing; the table has no forward chain. - `log`: each new flow is logged with the prefix `hive-egress-would-drop `, rate limited per source address, and counted. A flow the per-source limit cannot log gets a line with the prefix `hive-egress-would-drop-unmetered ` - from a limit shared by all addresses. Nothing is blocked. The source - address maps to an agent through the hive's DHCP leases. - - `drop`: not available yet; the evaluation refuses it. + from a limit shared by all addresses. Nothing is blocked. The lines are + the flows `reject` would refuse. The source address maps to an agent + through the hive's DHCP leases. + - `reject`: the same lines with the prefix `hive-egress-reject ` and + `hive-egress-reject-unmetered `, and every other packet from the bridge + is refused as well: a TCP flow with a reset, anything else with ICMP + administratively prohibited. Only the web traffic that the intercept + chain redirects into squid still gets out. ''; }; @@ -409,16 +465,6 @@ in config = lib.mkIf cfg.enable { assertions = [ - { - assertion = cfg.egress.forward != "drop"; - message = '' - services.hyperhive.network.egress.forward = "drop" is not available yet: - the traffic squid does not handle (the unrestricted agents, the service - containers, every port but 80 and 443) has no rule that lets it through, so a - drop would cut it. - Use "log" or "accept". - ''; - } { assertion = !refuseEch || lib.toInt (lib.last (lib.splitString "/" cfg.networkCidr)) >= 15; message = '' diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index 435bc034d..1cb51c8c9 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -72,17 +72,17 @@ let # The two settings of `egress.forward` the ordinary host (`log`) does not # render, for ../host-modules/hive-egress.nix: `accept` leaves the table - # without a forward chain, `drop` is refused by an assertion. The `accept` - # host also sets `egress.antiSpoof = "log"`, the one setting the ordinary - # host (`drop`) does not render. + # without a forward chain, `reject` renders the chain that refuses. The + # `accept` host also sets `egress.antiSpoof = "log"`, the one setting the + # ordinary host (`drop`) does not render. egressForwardAccept = hive { network.enable = true; network.egress.forward = "accept"; network.egress.antiSpoof = "log"; }; - egressForwardDrop = hive { + egressForwardReject = hive { network.enable = true; - network.egress.forward = "drop"; + network.egress.forward = "reject"; }; # Every hive defaults the agents' queue address to the queue's name, so the @@ -1561,6 +1561,13 @@ let && lib.hasInfix ''iifname "hive-br0" oifname != "hive-br0" ct state new update @log_v6 { ip6 saddr limit rate'' chain && lib.hasInfix ''log prefix "hive-egress-would-drop "'' chain && lib.hasInfix ''iifname "hive-br0" oifname "wg-hive" accept'' chain + # The mesh comes first, then the exempt MACs (service containers and + # unrestricted agents), then the counter and the logging: a flow of an + # exempt MAC is neither counted nor logged. + && lib.hasInfix ''iifname "hive-br0" oifname != "hive-br0" ether saddr @direct_mac accept'' chain + && idx ''oifname "wg-hive" accept'' < idx "ether saddr @direct_mac accept" + && idx "ether saddr @direct_mac accept" < idx "counter" + && idx "counter" < idx "update @log_v4" # The per-source rules accept after logging, and the unmetered rule # comes after both: a flow gets one line, and one the per-source rules # cannot log still gets the unmetered one. @@ -1672,22 +1679,48 @@ let && !(lib.any (a: !a.assertion) egressForwardAccept.assertions); } { - # `drop` is refused by an assertion that names the option and says it is - # not available; `bare` and the `accept` host, which set `log` and - # `accept`, are the controls that the lookup finds nothing for them. - name = "egress.forward drop is refused as not available yet"; + # `reject` logs and refuses in one rule, as a TCP rule (reset) and a + # non-TCP rule (ICMP admin-prohibited) for each of the four log/verdict + # positions, after the mesh and the exempt MACs. The ruleset is read as + # text. The `accept` count (the mesh and the exempt MACs only) and the + # `bare` host, whose chain has no refusal, are the controls that the + # lookups find what is there. + name = "egress.forward reject refuses the bridge's forwarded flows after the mesh and the exempt MACs"; ok = let - refused = - m: - lib.any ( - a: - !a.assertion - && lib.hasInfix "services.hyperhive.network.egress.forward" a.message - && lib.hasInfix "not available yet" a.message - ) m.assertions; + text = lib.head ( + lib.splitString "table bridge hive_egress {" egressForwardReject.services.hyperhive.network.egress.ruleset + ); + chain = lib.last (lib.splitString "chain forward" text); + rules = lib.filter (l: l != "" && l != "}") (map lib.trim (lib.splitString "\n" chain)); + idx = pat: lib.lists.findFirstIndex (lib.hasInfix pat) 1000 rules; + tcp = lib.filter (lib.hasInfix " meta l4proto tcp ") rules; + other = lib.filter (lib.hasInfix " meta l4proto != tcp") rules; + logged = lib.filter (lib.hasInfix "log prefix") rules; + bareText = bare.services.hyperhive.network.egress.ruleset; in - refused egressForwardDrop && !(refused bare) && !(refused egressForwardAccept); + !(lib.any (a: !a.assertion) egressForwardReject.assertions) + && lib.hasInfix "type filter hook forward priority filter - 10; policy accept;" chain + && lib.hasInfix "chain input" text + && idx ''oifname "wg-hive" accept'' < idx "ether saddr @direct_mac accept" + && idx "ether saddr @direct_mac accept" < idx "counter" + && idx "counter" < idx "reject" + && lib.length (lib.filter (lib.hasSuffix " accept") rules) == 2 + && lib.length tcp == 4 + && lib.length other == 4 + && lib.all (lib.hasSuffix " reject with tcp reset") tcp + && lib.all (lib.hasSuffix " reject with icmpx type admin-prohibited") other + && lib.length logged == 6 + && lib.all (l: lib.hasInfix ''log prefix "hive-egress-reject'' l) logged + && !(lib.hasInfix "hive-egress-would-drop" chain) + && lib.all ( + l: lib.hasInfix "update @log_" l || lib.hasInfix "limit rate 5/second burst 50 packets" l + ) logged + && + lib.last rules + == ''iifname "hive-br0" oifname != "hive-br0" meta l4proto != tcp reject with icmpx type admin-prohibited'' + && !(lib.hasInfix "reject with" bareText) + && lib.hasInfix "would-drop" bareText; } { # The redirect of the agents' web traffic: IPv4 TCP 80 and 443 from the @@ -1715,6 +1748,10 @@ let lib.hasInfix "type nat hook prerouting priority dstnat; policy accept;" chain && lib.hasInfix "set direct_mac { type ether_addr; }" inet && idx "ether saddr @direct_mac return" < idx "tcp dport 80" + # The same MACs are exempt in the forward chain, which follows the + # redirect chain in the table. + && lib.hasInfix ''oifname != "hive-br0" ether saddr @direct_mac accept'' inet + && !(lib.hasInfix "ether saddr @direct_mac accept" without) && idx "fib daddr type local return" < idx "tcp dport 80" && idx ''fib daddr oifname { "hive-br0", "wg-hive" } return'' < idx "tcp dport 80" && lib.length redirects == 2 diff --git a/nix/vm-tests/hive-squid.nix b/nix/vm-tests/hive-squid.nix index abda20011..37f819167 100644 --- a/nix/vm-tests/hive-squid.nix +++ b/nix/vm-tests/hive-squid.nix @@ -225,6 +225,10 @@ pkgs.testers.runNixOSTest { pkgs.mkpasswd pkgs.nftables ]; + # The reject-mode half of the test switches to this at its end. The + # forward mode is fixed when the node evaluates; the switch reloads + # `hive-egress` with the other ruleset. + specialisation.reject.configuration.services.hyperhive.network.egress.forward = "reject"; services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary = "other.t.local"; services.hyperhive.deploy.hive-controller.squid.selfcheck.canary = "cache.t.local"; services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains = [ @@ -1157,11 +1161,71 @@ pkgs.testers.runNixOSTest { assert icpt("https://203.0.113.2/") == "200", "an IP literal, directly" assert agent_lines() == seen, "squid saw nothing of it" assert send_hello("ech") == "16", "a hello with ECH, directly: the relay is not in the path" - proxy.wait_until_succeeds( - "journalctl -k -o cat --no-pager | grep -E " - "'hive-egress-would-drop(-unmetered)? IN=eth1 OUT=eth2 .*SRC=192\\.168\\.1\\.3 DST=203\\.0\\.113\\.2 .*PROTO=TCP .*DPT=443 '" - ) proxy.succeed(f"nft delete element inet hive_egress direct_mac '{{ {AGENT_MAC} }}'") assert icpt("https://other.t.local/") == "000", "intercepted again" + # Control: a new flow of the same MAC, now off the set, is logged, and the + # kernel log is in order, so a line of the exempt flows would be before it. + agent.execute("echo x > /dev/udp/203.0.113.79/9") + proxy.wait_until_succeeds( + "journalctl -k -o cat --no-pager | grep -E " + "'hive-egress-would-drop(-unmetered)? IN=eth1 OUT=eth2 .*DST=203\\.0\\.113\\.79 '" + ) + assert not would_drop(r"DST=203\.0\.113\.2 .*DPT=443 "), "the exempt MAC's flows leave no line" + + # ---- Reject mode ---- + # The node switches to its `reject` specialisation, which reloads the + # tables with the refusing forward chain. Nothing is in `direct_mac` yet. + def reject_lines(pattern): + # The reject chain's log lines in the kernel log that match `pattern`. + out = proxy.succeed("journalctl -k -o cat --no-pager | grep hive-egress-reject || true") + return [line for line in out.splitlines() if re.search(pattern, line)] + + def wait_reject_line(pattern): + proxy.wait_until_succeeds( + "journalctl -k -o cat --no-pager | grep -E " + + shlex.quote(f"hive-egress-reject(-unmetered)? IN=eth1 OUT=eth2 .*{pattern}") + ) + + def dial_status(addr, port): + # 0 connected, 1 refused, 124 no answer in 3 seconds. + return agent.execute(f"timeout 3 bash -c 'echo > /dev/tcp/{addr}/{port}'")[0] + + def udp_status(addr): + # A connected UDP socket reports the ICMP error of the first datagram + # on the second write: non-zero when the host refused it. Nothing + # answers a datagram that is let through, so that case is 0. + return agent.execute(f"bash -c 'exec 3<>/dev/udp/{addr}/9; echo x >&3; sleep 1; echo y >&3'")[0] + + with checked("reject mode: web still goes through squid and everything else from a restricted agent is refused"): + proxy.succeed("/run/current-system/specialisation/reject/bin/switch-to-configuration test") + out = proxy.succeed("nft list chain inet hive_egress forward") + assert "tcp reset" in out and "admin-prohibited" in out, out + assert icpt("https://forge.t.local/") == "200", "control: web still goes through squid" + assert dial_status("203.0.113.2", 8443) == 1, "TCP 8443 is reset at once, not left to time out" + wait_reject_line(r"SRC=192\.168\.1\.3 DST=203\.0\.113\.2 .*PROTO=TCP .*DPT=8443 ") + status, out = agent.execute("ping -c1 -W3 203.0.113.2") + assert status != 0 and "filtered" in out.lower(), f"ICMP is refused with admin-prohibited: {out}" + wait_reject_line(r"SRC=192\.168\.1\.3 DST=203\.0\.113\.2 .*PROTO=ICMP ") + assert udp_status("203.0.113.77") != 0, "a datagram is refused with an ICMP error" + wait_reject_line(r"DST=203\.0\.113\.77 .*PROTO=UDP ") + out = proxy.succeed("nft list chain inet hive_egress forward") + found = re.search(r'oifname != "eth1" counter packets (\d+) bytes \d+$', out, re.M) + assert found and int(found.group(1)) > 0, out + + with checked("reject mode: a MAC in direct_mac passes and is refused again once it leaves, and the mesh passes"): + proxy.succeed(f"nft add element inet hive_egress direct_mac '{{ {AGENT_MAC} }}'") + assert dial_status("203.0.113.2", 8443) == 0, "an exempt MAC (a service container's stand-in) reaches TCP 8443" + agent.succeed("ping -c1 -W3 203.0.113.2") + assert udp_status("203.0.113.77") == 0, "control: an exempt MAC's datagram is not refused" + proxy.succeed(f"nft delete element inet hive_egress direct_mac '{{ {AGENT_MAC} }}'") + assert dial_status("203.0.113.2", 8443) == 1, "refused again" + # The mesh comes first in the chain. A dummy interface answers nothing, + # so the proof is that the chain refuses neither the ping nor the UDP + # datagram, while the same flows to the uplink are refused. + status, out = agent.execute("ping -c1 -W5 198.51.100.7") + assert "filtered" not in out.lower(), f"the mesh interface is not egress: {out}" + agent.execute("echo x > /dev/udp/198.51.100.7/9; echo x > /dev/udp/203.0.113.80/9") + wait_reject_line(r"DST=203\.0\.113\.80 ") + assert not reject_lines(r"DST=198\.51\.100\.7 "), "the mesh interface is not egress" ''; }