hive-egress: the spoof log line says which check failed
Each refused frame now goes through its own chain, spoof_<reason>, whose log prefix names the check that refused it (mac, vlan, dhcp, v4, v6, arp). The unmetered log lines of all reasons share one named limit, spoof_unmetered, so the budget stays one for all ports and reasons.
This commit is contained in:
parent
c9e6cd2e61
commit
96334845f0
3 changed files with 93 additions and 39 deletions
|
|
@ -864,13 +864,16 @@ source check lives in that table and the host-side filtering stays in
|
||||||
- **The log.** A refused frame (all of them with `drop`, the same ones with
|
- **The log.** A refused frame (all of them with `drop`, the same ones with
|
||||||
`log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate
|
`log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate
|
||||||
limited per port: a burst of 20, then one a second. A frame its port can't log
|
limited per port: a burst of 20, then one a second. A frame its port can't log
|
||||||
gets a line with the prefix `hive-egress-spoof-unmetered ` instead, from a limit
|
gets a line with the prefix `hive-egress-spoof-unmetered ` instead, from one
|
||||||
shared by all ports: 5 lines a second after a burst of 50. Read them with
|
limit shared by all ports and reasons: 5 lines a second after a burst of 50.
|
||||||
`journalctl -k -g hive-egress-spoof`. `IN=` is the port, `vb-<machine>` (`vb-h-<agent>` for
|
Read them with `journalctl -k -g hive-egress-spoof`. The word after the
|
||||||
an agent), and `SRC` and `MAC=<dst>:<src>:<type>` are what the container
|
prefix is the check that refused the frame: `mac` (the port isn't bound to
|
||||||
sent; the port names the sender, because a bound port sends only its own MAC.
|
the source MAC), `vlan`, `dhcp` (the `chaddr` isn't the port's), `v4`, `v6` or
|
||||||
The counters in `nft list chain bridge hive_egress spoof` count every refused
|
`arp`; each has its own chain, `spoof_<reason>`, with its own counters. `IN=`
|
||||||
frame.
|
is the port, `vb-<machine>` (`vb-h-<agent>` for an agent), and `SRC` and
|
||||||
|
`MAC=<dst>:<src>:<type>` are what the container sent; the port names the
|
||||||
|
sender, because a bound port sends only its own MAC. The counters in
|
||||||
|
`nft list chain bridge hive_egress spoof` count every refused frame.
|
||||||
- **Why `log` is the default.** A container that sends from its leased address
|
- **Why `log` is the default.** A container that sends from its leased address
|
||||||
and its own MAC is never refused by `drop`, but only while the pair stays in
|
and its own MAC is never refused by `drop`, but only while the pair stays in
|
||||||
the table, and a host suspend/resume empties it. `log` keeps the lines and
|
the table, and a host suspend/resume empties it. `log` keeps the lines and
|
||||||
|
|
|
||||||
|
|
@ -190,8 +190,8 @@ let
|
||||||
|
|
||||||
# The bridge table. A frame of a port goes through `prerouting` in this
|
# The bridge table. A frame of a port goes through `prerouting` in this
|
||||||
# order: the port must be bound to the frame's source MAC, then the frame's
|
# order: the port must be bound to the frame's source MAC, then the frame's
|
||||||
# protocol decides which source address it may carry. Everything else ends in
|
# protocol decides which source address it may carry. Everything else logs in
|
||||||
# `spoof`, which logs and then drops or, with `antiSpoof = "log"`, accepts.
|
# `spoof_<reason>`, then `spoof` drops it (`antiSpoof = "log"`: accepts).
|
||||||
#
|
#
|
||||||
# - `port_mac` and `port_chaddr` hold the bound (port, MAC) pairs, the second
|
# - `port_mac` and `port_chaddr` hold the bound (port, MAC) pairs, the second
|
||||||
# as the 48 bits that sit at the `chaddr` field of a DHCP message. A port
|
# as the 48 bits that sit at the `chaddr` field of a DHCP message. A port
|
||||||
|
|
@ -215,9 +215,31 @@ let
|
||||||
# Logging is per ingress port, in the style of the forward chain: a burst of
|
# Logging is per ingress port, in the style of the forward chain: a burst of
|
||||||
# 20 lines, then one a second, and a port is forgotten after a minute. A
|
# 20 lines, then one a second, and a port is forgotten after a minute. A
|
||||||
# packet its port cannot log (over its rate, or the set is full) falls to the
|
# packet its port cannot log (over its rate, or the set is full) falls to the
|
||||||
# unmetered line, which has one budget for all ports. It logs
|
# unmetered line, which has one budget for all ports and reasons: the named
|
||||||
# packets, not flows: the bridge hook has no connection tracking.
|
# limit `spoof_unmetered`. It logs packets, not flows: the bridge hook has no
|
||||||
|
# connection tracking.
|
||||||
spoofVerdict = if egressCfg.antiSpoof == "drop" then "drop" else "accept";
|
spoofVerdict = if egressCfg.antiSpoof == "drop" then "drop" else "accept";
|
||||||
|
|
||||||
|
# The log prefix names the check that refused the frame: `mac` (the port is
|
||||||
|
# not bound to the source MAC), `vlan`, `dhcp` (the `chaddr` is not the
|
||||||
|
# port's), `v4`, `v6`, and `arp` (the ARP sender's MAC or address is not the
|
||||||
|
# port's). Each reason has its own chain and counters; `spoof` holds the
|
||||||
|
# verdict and one counter for every refused frame.
|
||||||
|
spoofReasons = [
|
||||||
|
"mac"
|
||||||
|
"vlan"
|
||||||
|
"dhcp"
|
||||||
|
"v4"
|
||||||
|
"v6"
|
||||||
|
"arp"
|
||||||
|
];
|
||||||
|
spoofChain = reason: ''
|
||||||
|
chain spoof_${reason} {
|
||||||
|
update @spoof_ports { iifname limit rate ${logRate} } counter log prefix "hive-egress-spoof ${reason} " goto spoof
|
||||||
|
limit name "spoof_unmetered" counter log prefix "hive-egress-spoof-unmetered ${reason} " goto spoof
|
||||||
|
counter goto spoof
|
||||||
|
}
|
||||||
|
'';
|
||||||
bridgeTable = ''
|
bridgeTable = ''
|
||||||
table bridge hive_egress {
|
table bridge hive_egress {
|
||||||
set port_mac { type ifname . ether_addr; }
|
set port_mac { type ifname . ether_addr; }
|
||||||
|
|
@ -225,37 +247,39 @@ let
|
||||||
set lease_v4 { type ether_addr . ipv4_addr; flags timeout; size 65536; }
|
set lease_v4 { type ether_addr . ipv4_addr; flags timeout; size 65536; }
|
||||||
set spoof_ports { type ifname; flags dynamic; timeout 1m; size 4096; }
|
set spoof_ports { type ifname; flags dynamic; timeout 1m; size 4096; }
|
||||||
|
|
||||||
|
limit spoof_unmetered { rate ${unmeteredRate} }
|
||||||
|
|
||||||
chain spoof {
|
chain spoof {
|
||||||
update @spoof_ports { iifname limit rate ${logRate} } counter log prefix "hive-egress-spoof " ${spoofVerdict}
|
|
||||||
limit rate ${unmeteredRate} counter log prefix "hive-egress-spoof-unmetered " ${spoofVerdict}
|
|
||||||
counter ${spoofVerdict}
|
counter ${spoofVerdict}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
${lib.concatMapStringsSep "\n" spoofChain spoofReasons}
|
||||||
|
|
||||||
chain l3_v4 {
|
chain l3_v4 {
|
||||||
udp dport 67 iifname . @th,288,48 != @port_chaddr goto spoof
|
udp dport 67 iifname . @th,288,48 != @port_chaddr goto spoof_dhcp
|
||||||
ip daddr { 255.255.255.255, ${cfg.bridgeIp} } udp sport 68 udp dport 67 accept
|
ip daddr { 255.255.255.255, ${cfg.bridgeIp} } udp sport 68 udp dport 67 accept
|
||||||
ether saddr . ip saddr @lease_v4 accept
|
ether saddr . ip saddr @lease_v4 accept
|
||||||
goto spoof
|
goto spoof_v4
|
||||||
}
|
}
|
||||||
|
|
||||||
chain l3_v6 {
|
chain l3_v6 {
|
||||||
ip6 saddr fe80::/10 accept
|
ip6 saddr fe80::/10 accept
|
||||||
ip6 saddr :: icmpv6 type { nd-neighbor-solicit, mld2-listener-report } accept
|
ip6 saddr :: icmpv6 type { nd-neighbor-solicit, mld2-listener-report } accept
|
||||||
goto spoof
|
goto spoof_v6
|
||||||
}
|
}
|
||||||
|
|
||||||
chain l3_arp {
|
chain l3_arp {
|
||||||
iifname . arp saddr ether != @port_mac goto spoof
|
iifname . arp saddr ether != @port_mac goto spoof_arp
|
||||||
arp saddr ip 0.0.0.0 accept
|
arp saddr ip 0.0.0.0 accept
|
||||||
arp saddr ether . arp saddr ip @lease_v4 accept
|
arp saddr ether . arp saddr ip @lease_v4 accept
|
||||||
goto spoof
|
goto spoof_arp
|
||||||
}
|
}
|
||||||
|
|
||||||
chain prerouting {
|
chain prerouting {
|
||||||
type filter hook prerouting priority -300; policy accept;
|
type filter hook prerouting priority -300; policy accept;
|
||||||
meta ibrname != "${br}" accept
|
meta ibrname != "${br}" accept
|
||||||
iifname . ether saddr != @port_mac goto spoof
|
iifname . ether saddr != @port_mac goto spoof_mac
|
||||||
ether type { 8021q, 8021ad } goto spoof
|
ether type { 8021q, 8021ad } goto spoof_vlan
|
||||||
ether type ip goto l3_v4
|
ether type ip goto l3_v4
|
||||||
ether type ip6 goto l3_v6
|
ether type ip6 goto l3_v6
|
||||||
ether type arp goto l3_arp
|
ether type arp goto l3_arp
|
||||||
|
|
@ -347,12 +371,14 @@ in
|
||||||
sources (and `::` in neighbour discovery and listener reports).
|
sources (and `::` in neighbour discovery and listener reports).
|
||||||
|
|
||||||
- `drop`: the frame is dropped and logged with the prefix
|
- `drop`: the frame is dropped and logged with the prefix
|
||||||
`hive-egress-spoof `, rate limited per port. A packet the per-port limit
|
`hive-egress-spoof <reason> `, rate limited per port; the reason is the
|
||||||
cannot log gets a line with the prefix `hive-egress-spoof-unmetered `
|
check that failed: `mac`, `vlan`, `dhcp`, `v4`, `v6` or `arp`. A packet
|
||||||
from a limit shared by all ports. A container sending from its leased
|
the per-port limit cannot log gets a line with the prefix
|
||||||
address is unaffected while its (MAC, IP) pair is in the lease set; a
|
`hive-egress-spoof-unmetered <reason> `, from one limit shared by all
|
||||||
lost pair, as after a host suspend/resume, cuts the container off until
|
ports and reasons. A container sending from its leased address is
|
||||||
it leases again.
|
unaffected while its (MAC, IP) pair is in the lease set; a lost pair,
|
||||||
|
as after a host suspend/resume, cuts the container off until it leases
|
||||||
|
again.
|
||||||
- `log` (the default): the same lines and counters, and the frame goes
|
- `log` (the default): the same lines and counters, and the frame goes
|
||||||
through. For finding out why a legitimate container is refused on a
|
through. For finding out why a legitimate container is refused on a
|
||||||
live hive. It is the default because the lease set does not yet survive
|
live hive. It is the default because the lease set does not yet survive
|
||||||
|
|
|
||||||
|
|
@ -1647,9 +1647,10 @@ let
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The bridge table drops what a port sends from an address that is not
|
# The bridge table drops what a port sends from an address that is not
|
||||||
# its own, and logs it. The ruleset is read as text: the spoof chain
|
# its own, and logs it. The ruleset is read as text: each `spoof_<reason>`
|
||||||
# ends every path in `drop`, and the accepts that remain are the ones that
|
# chain logs and goes to `spoof`, which ends every path in `drop`, and the
|
||||||
# let the leased source through, which is the presence arm.
|
# accepts that remain are the ones that let the leased source through,
|
||||||
|
# which is the presence arm.
|
||||||
name = "egress.antiSpoof drop renders the bridge table that drops frames whose source is not the port's own";
|
name = "egress.antiSpoof drop renders the bridge table that drops frames whose source is not the port's own";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
|
|
@ -1659,22 +1660,46 @@ let
|
||||||
name: lib.head (lib.splitString "chain" (lib.last (lib.splitString "chain ${name} {" bridgeText)));
|
name: lib.head (lib.splitString "chain" (lib.last (lib.splitString "chain ${name} {" bridgeText)));
|
||||||
spoof = lib.filter (l: l != "" && l != "}") (map lib.trim (lib.splitString "\n" (section "spoof")));
|
spoof = lib.filter (l: l != "" && l != "}") (map lib.trim (lib.splitString "\n" (section "spoof")));
|
||||||
ip4 = section "l3_v4";
|
ip4 = section "l3_v4";
|
||||||
|
reasons = [
|
||||||
|
"mac"
|
||||||
|
"vlan"
|
||||||
|
"dhcp"
|
||||||
|
"v4"
|
||||||
|
"v6"
|
||||||
|
"arp"
|
||||||
|
];
|
||||||
|
reasonLines =
|
||||||
|
r: lib.filter (l: l != "" && l != "}") (map lib.trim (lib.splitString "\n" (section "spoof_${r}")));
|
||||||
in
|
in
|
||||||
egressForwardAccept.services.hyperhive.deploy.hive-controller.egress.antiSpoof == "drop"
|
egressForwardAccept.services.hyperhive.deploy.hive-controller.egress.antiSpoof == "drop"
|
||||||
&& lib.hasInfix "hook prerouting priority -300" bridgeText
|
&& lib.hasInfix "hook prerouting priority -300" bridgeText
|
||||||
&& lib.hasInfix ''meta ibrname != "hive-br0" accept'' bridgeText
|
&& lib.hasInfix ''meta ibrname != "hive-br0" accept'' bridgeText
|
||||||
&& lib.hasInfix "iifname . ether saddr != @port_mac goto spoof" bridgeText
|
&& lib.hasInfix "iifname . ether saddr != @port_mac goto spoof_mac" bridgeText
|
||||||
&& lib.hasInfix "ether type { 8021q, 8021ad } goto spoof" bridgeText
|
&& lib.hasInfix "ether type { 8021q, 8021ad } goto spoof_vlan" bridgeText
|
||||||
&& lib.hasInfix "udp dport 67 iifname . @th,288,48 != @port_chaddr goto spoof" bridgeText
|
&& lib.hasInfix "udp dport 67 iifname . @th,288,48 != @port_chaddr goto spoof_dhcp" bridgeText
|
||||||
&& lib.hasInfix "ether saddr . ip saddr @lease_v4 accept" bridgeText
|
&& lib.hasInfix "ether saddr . ip saddr @lease_v4 accept" bridgeText
|
||||||
&& lib.hasInfix "ip6 saddr fe80::/10 accept" bridgeText
|
&& lib.hasInfix "ip6 saddr fe80::/10 accept" bridgeText
|
||||||
&& lib.hasInfix "arp saddr ether . arp saddr ip @lease_v4 accept" bridgeText
|
&& lib.hasInfix "arp saddr ether . arp saddr ip @lease_v4 accept" bridgeText
|
||||||
&& lib.hasInfix ''log prefix "hive-egress-spoof "'' bridgeText
|
&& lib.all (
|
||||||
&& lib.hasInfix ''log prefix "hive-egress-spoof-unmetered "'' bridgeText
|
r:
|
||||||
&& lib.hasInfix "update @spoof_ports { iifname limit rate 1/second burst 20 packets }" bridgeText
|
lib.hasInfix ''log prefix "hive-egress-spoof ${r} "'' (section "spoof_${r}")
|
||||||
&& lib.length spoof == 3
|
&& lib.hasInfix ''log prefix "hive-egress-spoof-unmetered ${r} "'' (section "spoof_${r}")
|
||||||
|
&& lib.hasInfix ''limit name "spoof_unmetered" counter log prefix "hive-egress-spoof-unmetered ${r} "'' (
|
||||||
|
section "spoof_${r}"
|
||||||
|
)
|
||||||
|
&& lib.hasInfix "update @spoof_ports { iifname limit rate 1/second burst 20 packets }" (
|
||||||
|
section "spoof_${r}"
|
||||||
|
)
|
||||||
|
&& lib.length (reasonLines r) == 3
|
||||||
|
&& lib.all (l: lib.hasSuffix " goto spoof" l) (reasonLines r)
|
||||||
|
) reasons
|
||||||
|
&& lib.hasInfix "limit spoof_unmetered { rate 5/second burst 50 packets }" bridgeText
|
||||||
|
&& lib.length (lib.splitString "limit spoof_unmetered {" bridgeText) == 2
|
||||||
|
&& lib.length spoof == 1
|
||||||
&& lib.all (l: lib.hasSuffix " drop" l) spoof
|
&& lib.all (l: lib.hasSuffix " drop" l) spoof
|
||||||
&& lib.hasInfix "goto spoof" ip4;
|
&& lib.hasInfix "goto spoof_v4" ip4
|
||||||
|
&& lib.hasInfix "goto spoof_v6" (section "l3_v6")
|
||||||
|
&& lib.hasInfix "goto spoof_arp" (section "l3_arp");
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# `log`, the default, renders the same chains with `accept` where `drop`
|
# `log`, the default, renders the same chains with `accept` where `drop`
|
||||||
|
|
@ -1695,9 +1720,9 @@ let
|
||||||
logged = spoofOf bare;
|
logged = spoofOf bare;
|
||||||
in
|
in
|
||||||
bare.services.hyperhive.deploy.hive-controller.egress.antiSpoof == "log"
|
bare.services.hyperhive.deploy.hive-controller.egress.antiSpoof == "log"
|
||||||
&& lib.length logged == 3
|
&& lib.length logged == 1
|
||||||
&& lib.all (l: lib.hasSuffix " accept" l) logged
|
&& lib.all (l: lib.hasSuffix " accept" l) logged
|
||||||
&& lib.hasInfix ''log prefix "hive-egress-spoof "'' (textOf bare)
|
&& lib.hasInfix ''log prefix "hive-egress-spoof mac "'' (textOf bare)
|
||||||
&& !(lib.any (l: lib.hasSuffix " drop" l) logged)
|
&& !(lib.any (l: lib.hasSuffix " drop" l) logged)
|
||||||
&& lib.all (l: lib.hasSuffix " drop" l) (spoofOf egressForwardAccept);
|
&& lib.all (l: lib.hasSuffix " drop" l) (spoofOf egressForwardAccept);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue