From 70179b6c1fec40cc3f2ea00a211e7f30af274aac Mon Sep 17 00:00:00 2001 From: atlas Date: Fri, 13 Nov 2026 02:53:37 +0100 Subject: [PATCH] egress anti-spoofing: keep established bindings, load tables with bindings and leases in one transaction --- docs/4-implementation/networking/network.md | 34 +- hive-priv/README.md | 6 +- hive-priv/src/egress.rs | 392 +++++++++++++++----- nix/host-modules/hive-egress.nix | 48 ++- nix/host-modules/hive-gateway/dnsmasq.nix | 2 +- nix/module-eval/core-toggle.nix | 22 +- nix/vm-tests/hive-egress-antispoof.nix | 20 +- nix/vm-tests/hive-squid.nix | 10 +- 8 files changed, 400 insertions(+), 134 deletions(-) diff --git a/docs/4-implementation/networking/network.md b/docs/4-implementation/networking/network.md index cc1cb7988..22121ca27 100644 --- a/docs/4-implementation/networking/network.md +++ b/docs/4-implementation/networking/network.md @@ -648,23 +648,31 @@ source check lives in that table and the host-side filtering stays in bound a moment after its veth appears, and the container's DHCP client retries until then. - **Where the bindings come from.** Addresses are dynamic, so the table's sets - the commands below fill them at run time. `hive-priv egress-pin` binds a port to the MAC of the - link at the other end of its veth pair, found through the container systemd - registers for it. udev starts it for each new `vb-*` port - (`hive-egress-pin@.service`), and the `hive-egress` unit starts it for - every running container on each start and reload. A binding stays with the - port until the port goes: root in the container can't move the port to - another MAC afterwards (state in `/run/hive-egress-pins`). If two ports have - the same MAC, neither gets a binding. dnsmasq's `dhcp-script` (`hive-priv -egress-lease`) adds each lease as a (MAC, address) pair that times out with - the lease. + fill at run time. `hive-priv egress-pin` binds a port to the MAC of the link at + the other end of its veth pair, found through the container systemd registers + for it. udev starts it for each new `vb-*` port + (`hive-egress-pin@.service`), and a timer (`hive-egress-pin.timer`, every + 30 seconds) runs it for the ports that still have no binding. The timer reads + no container while every port has one. The `hive-egress` unit loads the tables + with `hive-priv egress-load`: it looks up the bindings first, while the old + tables still filter, and puts them and the leases from dnsmasq's lease file + into the same nft transaction as the tables. A start or reload never leaves + the bridge with empty sets. If loading the leases fails, the unit logs a + warning and loads the tables with the bindings alone. A binding stays with the + port until the port goes: root in the container can't move the port to another + MAC afterwards (state in `/run/hive-egress-pins`). A port never takes a MAC + that a binding already holds, so a newcomer can't evict an established + container. If two new ports claim one MAC, neither gets a binding. dnsmasq's + `dhcp-script` (`hive-priv egress-lease`) adds each lease as a (MAC, address) + pair that times out with the lease. - **The window.** The binding takes the MAC the container's link has when the command first looks, normally within a second of the veth appearing. Until then root in the container could change its own MAC, but in that time only the container's early init runs, which is nix-built, and the agent's code - starts later. A container that did change its MAC first would get its own - lease for the new MAC, not another container's: a MAC that two ports have - gets no binding, and a lease follows the `chaddr` the port may send. + starts later. A container that did change its MAC first gets a binding for + that MAC only if nobody holds it; a lease follows the `chaddr` the port may + send. A port refused because it claimed a held MAC sends nothing, and the + timer binds it once its link has a MAC of its own. - **The log.** A refused frame (all of them with `drop`, the same ones with `log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate limited per port: a burst of 20, then one a second. A frame its port can't log diff --git a/hive-priv/README.md b/hive-priv/README.md index df356444c..dde12b67d 100644 --- a/hive-priv/README.md +++ b/hive-priv/README.md @@ -37,9 +37,9 @@ not the whole daemon-shared crate. `container` (lifecycle), `container_config` (nspawn flags, systemd drop-ins), `storage` (btrfs subvolumes, quotas, snapshots, disk measurement), `files` (pause marker, socket dir, legacy tmpfiles) and `infra` (gateway nginx, forge -admin, infra containers). `egress` holds two sub-commands that never -touch the socket, `egress-pin` and `egress-lease`; the `hive-egress` unit, udev -and dnsmasq run them to fill the sets of the anti-spoofing table (see +admin, infra containers). `egress` holds three sub-commands that never +touch the socket, `egress-load`, `egress-pin` and `egress-lease`; the `hive-egress` +unit, udev, a timer and dnsmasq run them to fill the sets of the anti-spoofing table (see `docs/4-implementation/networking/network.md`). Helpers more than one family calls live in `shared`. Each module's `//!` header lists its ops. diff --git a/hive-priv/src/egress.rs b/hive-priv/src/egress.rs index 01ecbf905..37d5b4232 100644 --- a/hive-priv/src/egress.rs +++ b/hive-priv/src/egress.rs @@ -1,33 +1,31 @@ //! The per-port bindings behind the `bridge hive_egress` anti-spoofing table -//! (`nix/host-modules/hive-egress.nix`), as two sub-commands run by systemd +//! (`nix/host-modules/hive-egress.nix`), as three sub-commands run by systemd //! units and by dnsmasq, never through the socket. //! -//! - `egress-pin [--seed] []` binds each port of the agent bridge to the -//! MAC its container uses. A port is bound only when a registered container -//! owns the other end of its veth pair; a port nobody owns stays unbound, and -//! the table drops everything it sends. With `` the command first waits -//! for that port to be bound or to vanish. `--seed` also loads the leases in -//! dnsmasq's lease file. +//! - `egress-load ` replaces the tables with the ruleset, the +//! current bindings and the leases of dnsmasq's lease file, in one nft +//! transaction. The bindings are looked up first, while the old tables still +//! filter. A lease load that fails is a warning; the ruleset then loads with +//! the bindings alone. +//! - `egress-pin []` binds each port of the agent bridge to the MAC its +//! container uses. A port nobody owns stays unbound, and the table drops +//! everything it sends. With `` the command first waits for that port +//! to be bound or to vanish. udev runs it per new port, a timer for the ports +//! still unbound. //! - `egress-lease [ []]` is dnsmasq's `dhcp-script`: -//! it keeps the table's set of leased MAC and IPv4 pairs in step with the -//! leases. +//! it keeps the table's set of leased MAC and IPv4 pairs in step. //! -//! The owner of a port is found from the host side: the port's peer index is -//! the index of one link in the container's network namespace, and that link's -//! peer index is the port's own index. Both halves have to hold, so a veth -//! pair the container makes inside its namespace cannot claim another port. -//! The container's leader process comes from the machine state systemd-machined -//! keeps under `/run/systemd/machines`. +//! A port's owner is found from the host side: the port's peer index is the +//! index of one link in a registered container's network namespace (the leader +//! comes from `/run/systemd/machines`), and that link's peer index is the port's +//! own index. A veth pair the container makes inside its namespace can't claim +//! another port. //! -//! The bound MAC is the one the link has when the port is first looked at, and -//! the binding then stays for as long as the port exists: later passes keep it -//! in `/run/hive-egress-pins` and do not read the link again, so root in the -//! container cannot move its port to another MAC afterwards. The unit starts -//! the command when the veth appears, while only the container's nix-built -//! early init runs. A MAC that two ports claim binds neither. -//! -//! The entry points are [`run`] (the sub-commands), [`resolve`] (which port has -//! which MAC) and [`pin_script`] / [`lease_script`] (the `nft` input). +//! The bound MAC is the one the link has when the port is first looked at. The +//! binding stays for as long as the port exists (`/run/hive-egress-pins`), and +//! a kept binding is never given up: a port that claims a MAC a kept binding +//! holds, or that another new claim shares, stays unbound and is looked at +//! again on the next pass. use std::fmt::Write as _; use std::fs::{File, OpenOptions}; @@ -44,6 +42,9 @@ pub(crate) const PIN: &str = "egress-pin"; /// The sub-command dnsmasq's `dhcp-script` runs. pub(crate) const LEASE: &str = "egress-lease"; +/// The sub-command that loads the tables. +pub(crate) const LOAD: &str = "egress-load"; + /// The bridge the table guards, set by the unit that runs the command. const BRIDGE_ENV: &str = "HIVE_EGRESS_BRIDGE"; @@ -161,22 +162,24 @@ pub(crate) struct Resolution { /// /// `machines` holds each registered container's links. `earlier` holds the /// bindings of a previous pass: a port that has the same name and index keeps -/// its binding whatever its link says now, so a MAC changed after the port was -/// bound is never bound in its place. A port no machine owns, an unusable MAC -/// and a MAC two ports claim all leave the port unbound. +/// its binding whatever its link says now, and no other port takes its MAC. A +/// kept binding is never given up, so a port can only get a binding for a MAC +/// nobody holds. A port no machine owns, an unusable MAC, a MAC a kept binding +/// holds and a MAC two new claims share all leave the port unbound. pub(crate) fn resolve( ports: &[HostPort], machines: &[(String, Vec)], earlier: &[Pin], ) -> Resolution { - let mut found: Vec = Vec::new(); + let mut kept: Vec = Vec::new(); + let mut claims: Vec = Vec::new(); let mut out = Resolution::default(); for port in ports { - if let Some(kept) = earlier + if let Some(pin) = earlier .iter() .find(|p| p.port == port.name && p.ifindex == port.ifindex) { - found.push(kept.clone()); + kept.push(pin.clone()); continue; } let owner = machines.iter().find_map(|(machine, links)| { @@ -193,7 +196,7 @@ pub(crate) fn resolve( port.name.clone(), format!("{machine} has the unusable address {mac}"), )), - Some((machine, mac)) => found.push(Pin { + Some((machine, mac)) => claims.push(Pin { port: port.name.clone(), ifindex: port.ifindex, mac, @@ -201,26 +204,29 @@ pub(crate) fn resolve( }), } } - for pin in &found { - let rivals: Vec<&str> = found + for claim in &claims { + let holders: Vec<&str> = kept .iter() - .filter(|p| p.mac == pin.mac && p.port != pin.port) + .chain(claims.iter()) + .filter(|p| p.mac == claim.mac && p.port != claim.port) .map(|p| p.port.as_str()) .collect(); - if rivals.is_empty() { - out.pins.push(pin.clone()); + if holders.is_empty() { + out.pins.push(claim.clone()); } else { out.unbound.push(( - pin.port.clone(), + claim.port.clone(), format!( "{} has {}, which {} also has", - pin.machine, - pin.mac, - rivals.join(", ") + claim.machine, + claim.mac, + holders.join(", ") ), )); } } + kept.append(&mut out.pins); + out.pins = kept; out } @@ -258,10 +264,9 @@ fn is_plain_name(name: &str) -> bool { .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'_' | b'-')) } -/// The nft input that makes the table's bindings exactly `pins`, in one -/// transaction: a reader never sees the set empty in between. -pub(crate) fn pin_script(pins: &[Pin]) -> String { - let mut script = format!("flush set {TABLE} port_mac\nflush set {TABLE} port_chaddr\n"); +/// The nft input that adds the bindings `pins` to the two sets. +pub(crate) fn pin_elements(pins: &[Pin]) -> String { + let mut script = String::new(); for Pin { port, mac, .. } in pins { let _ = writeln!( script, @@ -273,6 +278,15 @@ pub(crate) fn pin_script(pins: &[Pin]) -> String { script } +/// The nft input that makes the table's bindings exactly `pins`, in one +/// transaction: a reader never sees the sets empty in between. +pub(crate) fn pin_script(pins: &[Pin]) -> String { + format!( + "flush set {TABLE} port_mac\nflush set {TABLE} port_chaddr\n{}", + pin_elements(pins) + ) +} + /// The nft input for one lease event, or `None` for an event that changes /// nothing: `init`, an IPv6 address, an address or MAC that does not parse. /// @@ -464,21 +478,59 @@ fn machine_links(dir: &Path) -> Vec<(String, Vec)> { machines } -/// One pass: look at the bridge's ports and the containers, and make the -/// table's bindings match. Returns the ports of the bridge and the resolution. -fn pin_once(bridge: &str) -> Result<(Vec, Resolution)> { +/// The ports no kept binding covers: the ones a pass has to look up. +pub(crate) fn ports_to_resolve<'a>(ports: &'a [HostPort], earlier: &[Pin]) -> Vec<&'a HostPort> { + ports + .iter() + .filter(|p| { + !earlier + .iter() + .any(|k| k.port == p.name && k.ifindex == p.ifindex) + }) + .collect() +} + +/// What a pass found out. +struct Survey { + earlier: Vec, + resolution: Resolution, +} + +fn state_path() -> String { + std::env::var("HIVE_EGRESS_STATE").unwrap_or_else(|_| STATE.into()) +} + +/// Look at the bridge's ports, the earlier bindings and, when a port has none, +/// the containers. Changes nothing. +fn survey(bridge: &str) -> Result { let ports = parse_ports(&output_of("ip", &["-j", "link", "show", "master", bridge])?)?; - let machines = machine_links(Path::new(MACHINES_DIR)); - let state_path = std::env::var("HIVE_EGRESS_STATE").unwrap_or_else(|_| STATE.into()); - let earlier = parse_state(&std::fs::read_to_string(&state_path).unwrap_or_default()); + let earlier = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default()); + let machines = if ports_to_resolve(&ports, &earlier).is_empty() { + Vec::new() + } else { + machine_links(Path::new(MACHINES_DIR)) + }; let resolution = resolve(&ports, &machines, &earlier); - nft(&pin_script(&resolution.pins))?; - std::fs::write(&state_path, state_text(&resolution.pins)) - .with_context(|| format!("write {state_path}"))?; - for pin in resolution.pins.iter().filter(|p| !earlier.contains(p)) { + Ok(Survey { + earlier, + resolution, + }) +} + +/// Keep the bindings for the next pass and print the new ones. +fn commit(survey: &Survey) -> Result<()> { + let path = state_path(); + std::fs::write(&path, state_text(&survey.resolution.pins)) + .with_context(|| format!("write {path}"))?; + for pin in survey + .resolution + .pins + .iter() + .filter(|p| !survey.earlier.contains(p)) + { eprintln!("{PIN}: {} bound to {} ({})", pin.port, pin.mac, pin.machine); } - Ok((ports, resolution)) + Ok(()) } /// Wait for the lock of the bindings. It is released when the file is dropped. @@ -500,41 +552,54 @@ fn now_secs() -> u64 { .map_or(0, |d| d.as_secs()) } -/// `egress-pin [--seed] []`. +/// Whose "stays unbound" lines a pass prints. +#[derive(Clone, Copy)] +enum Report<'a> { + All, + Port(&'a str), + Nothing, +} + +fn report(resolution: &Resolution, which: Report<'_>) { + for (name, why) in &resolution.unbound { + let show = match which { + Report::All => true, + Report::Port(port) => port == name, + Report::Nothing => false, + }; + if show { + eprintln!("{PIN}: {name} stays unbound: {why}"); + } + } +} + +/// `egress-pin []`: one pass, or with `` passes until that port is +/// bound or gone. Without `` it prints only what it binds: the timer +/// runs it again and again for the ports still unbound. fn pin(args: &[String]) -> Result<()> { let bridge = std::env::var(BRIDGE_ENV).with_context(|| format!("{BRIDGE_ENV} is not set"))?; - let mut seed = false; - let mut wait_for = None; - for arg in args { - match arg.as_str() { - "--seed" => seed = true, - port if wait_for.is_none() && is_plain_name(port) => wait_for = Some(port.to_string()), - other => bail!("usage: hive-priv {PIN} [--seed] [] (got {other:?})"), - } - } - if seed { - let path = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into()); - let leases = std::fs::read_to_string(&path).unwrap_or_default(); - let script = seed_script(&leases, now_secs()); - if !script.is_empty() { - let _lock = lock()?; - nft(&script).context("load leases")?; - } - } + let wait_for = match args { + [] => None, + [port] if is_plain_name(port) => Some(port.as_str()), + _ => bail!("usage: hive-priv {PIN} []"), + }; let deadline = Instant::now() + WAIT; loop { - let (ports, resolution) = { + let survey = { let _lock = lock()?; - pin_once(&bridge)? + let survey = survey(&bridge)?; + nft(&pin_script(&survey.resolution.pins))?; + commit(&survey)?; + survey }; - let Some(port) = &wait_for else { - report(&resolution, None); + let Some(port) = wait_for else { + report(&survey.resolution, Report::Nothing); return Ok(()); }; - let bound = resolution.pins.iter().any(|p| &p.port == port); - let present = ports.iter().any(|p| &p.name == port); + let bound = survey.resolution.pins.iter().any(|p| p.port == port); + let present = port_exists(port); if bound || !present || Instant::now() >= deadline { - report(&resolution, Some(port)); + report(&survey.resolution, Report::Port(port)); if present && !bound { bail!("port {port} stays unbound"); } @@ -544,15 +609,65 @@ fn pin(args: &[String]) -> Result<()> { } } -/// Print why ports stay unbound; with `port`, only that one. -fn report(resolution: &Resolution, port: Option<&str>) { - for (name, why) in &resolution.unbound { - if port.is_none_or(|p| p == name) { - eprintln!("{PIN}: {name} stays unbound: {why}"); +/// Whether an interface of this name exists, on any bridge or none: a port is +/// created before it joins the bridge. +fn port_exists(name: &str) -> bool { + Command::new("ip") + .args(["link", "show", "dev", name]) + .output() + .is_ok_and(|o| o.status.success()) +} + +/// Load `ruleset` with the bindings and the leases in the same transaction, so +/// no frame meets the new table with empty sets. A lease load that `run` +/// refuses is a warning: the ruleset loads again with the bindings alone. +pub(crate) fn load_ruleset( + ruleset: &str, + pins: &str, + leases: &str, + run: impl Fn(&str) -> Result<()>, +) -> Result<()> { + match run(&format!("{ruleset}\n{pins}{leases}")) { + Err(e) if !leases.is_empty() => { + eprintln!("{LOAD}: loading the leases failed, loading without them: {e:#}"); + run(&format!("{ruleset}\n{pins}")).context("load the ruleset") } + other => other, } } +/// `egress-load `: replace the tables with the ruleset and the +/// current bindings and leases. The bindings are looked up first, while the old +/// tables still filter. A lookup that fails loads the ruleset with none: the +/// ports stay unbound until the timer or udev binds them. +fn load(args: &[String]) -> Result<()> { + let bridge = std::env::var(BRIDGE_ENV).with_context(|| format!("{BRIDGE_ENV} is not set"))?; + let [file] = args else { + bail!("usage: hive-priv {LOAD} "); + }; + let ruleset = std::fs::read_to_string(file).with_context(|| format!("read {file}"))?; + let path = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into()); + let leases = seed_script( + &std::fs::read_to_string(&path).unwrap_or_default(), + now_secs(), + ); + let _lock = lock()?; + let found = survey(&bridge); + let pins = match &found { + Ok(found) => pin_elements(&found.resolution.pins), + Err(e) => { + eprintln!("{LOAD}: binding the ports failed, loading without bindings: {e:#}"); + String::new() + } + }; + load_ruleset(&ruleset, &pins, &leases, nft)?; + if let Ok(found) = &found { + commit(found)?; + report(&found.resolution, Report::All); + } + Ok(()) +} + /// `egress-lease [ []]`. Nothing is loaded for an event /// that changes no lease. fn lease(args: &[String]) -> Result<()> { @@ -575,6 +690,7 @@ pub(crate) fn run(args: &[String]) -> Option { let (name, rest) = args.split_first()?; let result = match name.as_str() { PIN => pin(rest), + LOAD => load(rest), LEASE => lease(rest), _ => return None, }; @@ -723,6 +839,110 @@ mod tests { assert!(resolve(&ports, &machines, &[]).pins.is_empty()); } + #[test] + fn a_kept_binding_is_never_given_up_to_a_new_claim_on_its_mac() { + let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)]; + let earlier = [pin("vb-a", 10, 1, "a")]; + let machines = [ + ("a".to_string(), vec![link(2, Some(10), mac(1))]), + ("b".to_string(), vec![link(2, Some(11), mac(1))]), + ]; + let got = resolve(&ports, &machines, &earlier); + assert_eq!(got.pins, vec![pin("vb-a", 10, 1, "a")]); + assert_eq!(got.unbound.len(), 1); + assert_eq!(got.unbound[0].0, "vb-b"); + } + + #[test] + fn a_new_claim_on_a_free_mac_binds_next_to_a_kept_binding() { + let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)]; + let earlier = [pin("vb-a", 10, 1, "a")]; + let machines = [("b".to_string(), vec![link(2, Some(11), mac(2))])]; + let got = resolve(&ports, &machines, &earlier); + assert_eq!( + got.pins, + vec![pin("vb-a", 10, 1, "a"), pin("vb-b", 11, 2, "b")] + ); + assert!(got.unbound.is_empty()); + } + + #[test] + fn a_port_refused_for_a_taken_mac_binds_on_a_later_pass_once_it_has_its_own() { + let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)]; + let earlier = [pin("vb-a", 10, 1, "a")]; + let rival = [("b".to_string(), vec![link(2, Some(11), mac(1))])]; + assert!(resolve(&ports, &rival, &earlier).pins.len() == 1); + // The rival restores its own MAC; the next pass reads its link again. + let restored = [("b".to_string(), vec![link(2, Some(11), mac(2))])]; + let got = resolve(&ports, &restored, &earlier); + assert_eq!( + got.pins, + vec![pin("vb-a", 10, 1, "a"), pin("vb-b", 11, 2, "b")] + ); + } + + #[test] + fn only_ports_without_a_kept_binding_are_looked_up() { + let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)]; + let a = pin("vb-a", 10, 1, "a"); + assert_eq!(ports_to_resolve(&ports, &[]).len(), 2); + assert_eq!( + ports_to_resolve(&ports, std::slice::from_ref(&a))[0].name, + "vb-b" + ); + let b = pin("vb-b", 11, 2, "b"); + assert!(ports_to_resolve(&ports, &[a, b]).is_empty()); + // The same name with another index is a new port. + assert_eq!( + ports_to_resolve(&[port("vb-a", 12, 2)], &[pin("vb-a", 10, 1, "a")]).len(), + 1 + ); + } + + #[test] + fn the_ruleset_loads_with_the_bindings_and_leases_in_one_script() { + let calls = std::cell::RefCell::new(Vec::new()); + let pins = pin_elements(&[pin("vb-a", 10, 10, "a")]); + let leases = lease_script("add", "02:00:00:00:00:0a", "10.42.0.10", 90).unwrap(); + load_ruleset("table bridge hive_egress { }", &pins, &leases, |s| { + calls.borrow_mut().push(s.to_string()); + Ok(()) + }) + .unwrap(); + let calls = calls.into_inner(); + assert_eq!(calls.len(), 1, "one transaction"); + let script = &calls[0]; + assert!(script.starts_with("table bridge hive_egress { }")); + assert!(script.contains("port_mac { \"vb-a\" . 02:00:00:00:00:0a }")); + assert!(script.contains("lease_v4 { 02:00:00:00:00:0a . 10.42.0.10 timeout 90s }")); + assert!(!script.contains("flush set"), "the sets are new"); + } + + #[test] + fn a_lease_load_that_fails_is_a_warning_and_the_bindings_still_load() { + let calls = std::cell::RefCell::new(Vec::new()); + let pins = pin_elements(&[pin("vb-a", 10, 10, "a")]); + let leases = lease_script("add", "02:00:00:00:00:0a", "10.42.0.10", 90).unwrap(); + let result = load_ruleset("RULESET", &pins, &leases, |s| { + calls.borrow_mut().push(s.to_string()); + if s.contains("lease_v4") { + anyhow::bail!("lease_v4 refused") + } + Ok(()) + }); + assert!(result.is_ok()); + let calls = calls.into_inner(); + assert_eq!(calls.len(), 2); + assert!(calls[1].contains("port_mac") && !calls[1].contains("lease_v4")); + } + + #[test] + fn a_ruleset_that_fails_without_leases_is_an_error() { + let fails = |_: &str| -> Result<()> { anyhow::bail!("refused") }; + assert!(load_ruleset("RULESET", "", "", fails).is_err()); + assert!(load_ruleset("RULESET", "", "x", fails).is_err()); + } + #[test] fn pin_script_replaces_both_sets_in_one_script() { let script = pin_script(&[pin("vb-a", 10, 10, "a")]); diff --git a/nix/host-modules/hive-egress.nix b/nix/host-modules/hive-egress.nix index 8f8e4867f..8ddb57795 100644 --- a/nix/host-modules/hive-egress.nix +++ b/nix/host-modules/hive-egress.nix @@ -18,9 +18,11 @@ # traffic between two ports is bridged: the `inet` hooks see it as coming from # the bridge device, and only a bridge hook knows the port it came in on # (`iifname` is the veth). Which MAC a port may use and which IPv4 address a MAC -# may use are not known when the ruleset loads, so they are set contents that -# `hive-priv egress-pin` (per port, from udev) and dnsmasq's `dhcp-script` -# (per lease) fill; `hive-priv`'s `egress.rs` documents how. +# may use are not known until a container runs, so they are set contents. +# `hive-priv egress-load` puts the current ones into the same transaction as the +# tables; `egress-pin` (per new port from udev, and from a timer for the ports +# still unbound) and dnsmasq's `dhcp-script` (per lease) keep them current. +# `hive-priv`'s `egress.rs` documents how. { lib, config, @@ -39,7 +41,8 @@ let # the one every platform container's unit uses (hive-priv.nix points it at the # current build). hivePriv = (import ./lib/container-rootfs-check.nix { inherit config; }).hivePriv; - pinCommand = args: "${hivePriv} egress-pin ${args}"; + pinCommand = args: lib.trim "${hivePriv} egress-pin ${args}"; + loadCommand = "${hivePriv} egress-load ${ruleset}"; pinPath = [ pkgs.nftables pkgs.iproute2 @@ -308,14 +311,11 @@ in serviceConfig = { Type = "oneshot"; RemainAfterExit = true; - ExecStart = "${nft} -f ${ruleset}"; - # The sets of a freshly loaded bridge table are empty: bind the ports of - # the running containers and load the leases from dnsmasq's file. - ExecStartPost = pinCommand "--seed"; - ExecReload = [ - "${nft} -f ${ruleset}" - (pinCommand "--seed") - ]; + # The tables load with the bindings of the running containers' ports and + # the leases of dnsmasq's file in the same transaction, so a reload never + # leaves the bridge with empty sets. + ExecStart = loadCommand; + ExecReload = loadCommand; ExecStop = [ "-${nft} delete table inet hive_egress" "-${nft} delete table bridge hive_egress" @@ -337,6 +337,30 @@ in }; }; + # Looks again at the ports still unbound. A port a rival MAC kept unbound + # sends nothing, so nothing else would tell the hive it can be bound now. + # Ports with a binding cost a listing of the bridge and no container is read. + systemd.services.hive-egress-pin = { + description = "bind the agent bridge ports that have no binding yet"; + wants = [ "hive-egress.service" ]; + after = [ "hive-egress.service" ]; + path = pinPath; + environment = pinEnvironment; + serviceConfig = { + Type = "oneshot"; + ExecStart = pinCommand ""; + }; + }; + systemd.timers.hive-egress-pin = { + description = "bind the agent bridge ports that have no binding yet"; + wantedBy = [ "timers.target" ]; + timerConfig = { + OnBootSec = "1min"; + OnUnitActiveSec = "30s"; + AccuracySec = "5s"; + }; + }; + # nspawn names the host end of a container's veth `vb-` when the # container joins a bridge. services.udev.extraRules = '' diff --git a/nix/host-modules/hive-gateway/dnsmasq.nix b/nix/host-modules/hive-gateway/dnsmasq.nix index 7f33b208d..a9844503d 100644 --- a/nix/host-modules/hive-gateway/dnsmasq.nix +++ b/nix/host-modules/hive-gateway/dnsmasq.nix @@ -81,7 +81,7 @@ in # bridgeIp/bridgePrefixLength. All containers (agents and service # containers such as swarm-ci-runner) receive their IPs dynamically. dhcp-range = "${networkCfg.dhcpRangeStart},${networkCfg.dhcpRangeEnd},1h"; - # `hive-priv egress-pin --seed` reads this file back (hive-egress.nix). + # `hive-priv egress-load` reads this file back (hive-egress.nix). dhcp-leasefile = "/var/lib/dnsmasq/dnsmasq.leases"; # Each lease change updates the leased (MAC, IPv4) pairs the bridge # anti-spoofing table checks sources against (hive-egress.nix). diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index bd39bdb7e..00053bd94 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -1624,17 +1624,24 @@ let && lib.all (l: lib.hasSuffix " drop" l) (spoofOf bare); } { - # The sets are filled at run time, from three places: the unit at start - # and reload, the udev rule for each new port, and dnsmasq for each - # lease. Each is present on `bare`; the host without the bridge has none. - name = "the bridge table's sets are filled by the unit, a udev rule per port and dnsmasq's lease script"; + # The sets are filled at run time, from four places: the unit at start and + # reload (tables, bindings and leases in one transaction), the udev rule + # for each new port, a timer for the ports still unbound, and dnsmasq for + # each lease. Each is present on `bare`; the host without the bridge has + # none. + name = "the bridge table's sets load with the table, and a udev rule, a timer and dnsmasq's lease script keep them current"; ok = let unit = bare.systemd.services.hive-egress; pin = bare.systemd.services."hive-egress-pin@"; + sweep = bare.systemd.services.hive-egress-pin; in - lib.hasInfix "egress-pin --seed" unit.serviceConfig.ExecStartPost - && lib.any (lib.hasInfix "egress-pin --seed") unit.serviceConfig.ExecReload + lib.hasInfix "egress-load " unit.serviceConfig.ExecStart + && lib.hasInfix "hive-egress.nft" unit.serviceConfig.ExecStart + && unit.serviceConfig.ExecReload == unit.serviceConfig.ExecStart + && !(unit.serviceConfig ? ExecStartPost) + && lib.hasSuffix "egress-pin" sweep.serviceConfig.ExecStart + && bare.systemd.timers.hive-egress-pin.timerConfig.OnUnitActiveSec == "30s" && unit.environment.HIVE_EGRESS_BRIDGE == "hive-br0" && lib.hasInfix "egress-pin %i" pin.serviceConfig.ExecStart && lib.hasInfix ''KERNEL=="vb-*"'' bare.services.udev.extraRules @@ -1642,7 +1649,8 @@ let && bare.services.dnsmasq.settings.dhcp-script == toString bare.services.hyperhive.network.egress.leaseScript - && !(centralToggleOff.systemd.services ? "hive-egress-pin@"); + && !(centralToggleOff.systemd.services ? "hive-egress-pin@") + && !(centralToggleOff.systemd ? timers && centralToggleOff.systemd.timers ? hive-egress-pin); } { # `accept` renders the table without the forward chain and without the diff --git a/nix/vm-tests/hive-egress-antispoof.nix b/nix/vm-tests/hive-egress-antispoof.nix index 91b69b6f8..7eefe97a5 100644 --- a/nix/vm-tests/hive-egress-antispoof.nix +++ b/nix/vm-tests/hive-egress-antispoof.nix @@ -2,7 +2,7 @@ # ../host-modules/hive-egress.nix on a real agent bridge, with two nspawn # containers on it that take their addresses from the hive's dnsmasq # (../host-modules/hive-gateway/dnsmasq.nix' lease script, the real -# `hive-priv egress-pin` and `egress-lease`). `hive-squid` cannot host this: its +# `hive-priv egress-load`, `egress-pin` and `egress-lease`). `hive-squid` cannot host this: its # stand-in bridge is one NIC, so every agent shares one port. # # What it pins: @@ -14,9 +14,10 @@ # that is not link-local and a DHCP message with another's `chaddr` are # dropped, and each leaves a `hive-egress-spoof ` line that names the port; # - a DHCP renewal, a link-local IPv6 packet and a restored MAC go through; -# - a port with no bound MAC sends nothing, and the pin unit binds it again; -# - `antiSpoof = "log"`, reached by a reload of the unit, keeps the bindings -# and the leases, lets the same spoofs through and logs them. +# - a port with no bound MAC sends nothing, and the timer binds it again; +# - `antiSpoof = "log"`, reached by a reload of the unit, brings the bindings +# and the leases with the new table (nothing is dropped in between), +# lets the same spoofs through and logs them. { pkgs, hivePriv }: let bridge = "hive0"; @@ -265,20 +266,21 @@ pkgs.testers.runNixOSTest { host.wait_until_succeeds(f"nft list set {TABLE} lease_v4 | grep -q '{MAC_A} . {IP_A}'") assert ct_ok("a", "ping -c1 -W2 ${gateway}") - with subtest("a port with no bound MAC sends nothing, and the pin unit binds it again"): + with subtest("a port with no bound MAC sends nothing, and the timer binds it again"): host.succeed(f"nft flush set {TABLE} port_mac") assert not ct_ok("a", "ping -c1 -W2 ${gateway}") assert not ct_ok("b", "ping -c1 -W2 ${gateway}") - host.succeed("systemctl start hive-egress-pin@vb-a.service hive-egress-pin@vb-b.service") + # Nothing starts the pin unit by hand: the timer finds the unbound ports. host.wait_until_succeeds("nixos-container run a -- ping -c1 -W2 ${gateway}") host.wait_until_succeeds("nixos-container run b -- ping -c1 -W2 ${gateway}") with subtest("antiSpoof = log keeps the bindings and the leases, lets a spoofed datagram through and logs it"): host.succeed("/run/current-system/specialisation/logmode/bin/switch-to-configuration test") assert "counter accept" in host.succeed(f"nft list chain {TABLE} spoof"), "the table was reloaded in log mode" - host.wait_until_succeeds(f"nft list set {TABLE} port_mac | grep -q '\"vb-a\" . {MAC_A}'") - host.wait_until_succeeds(f"nft list set {TABLE} lease_v4 | grep -q '{MAC_B} . {IP_B}'") - assert ct_ok("a", "ping -c1 -W2 ${gateway}"), "control: own address, after the reload" + # No waiting: the reload put the sets in with the table. + host.succeed(f"nft list set {TABLE} port_mac | grep -q '\"vb-a\" . {MAC_A}'") + host.succeed(f"nft list set {TABLE} lease_v4 | grep -q '{MAC_B} . {IP_B}'") + assert ct_ok("a", "ping -c1 -W2 ${gateway}"), "control: own address, right after the reload" seen = len(spoof_lines()) ct("a", f"ip addr add {IP_B}/24 dev eth0") assert udp_reaches_host("a", IP_B), "log mode lets the spoofed datagram through" diff --git a/nix/vm-tests/hive-squid.nix b/nix/vm-tests/hive-squid.nix index 3ba24ed80..a4b85e72c 100644 --- a/nix/vm-tests/hive-squid.nix +++ b/nix/vm-tests/hive-squid.nix @@ -197,10 +197,14 @@ pkgs.testers.runNixOSTest { ); # The gateway: nginx answers on the bridge, which agents reach directly. networking.firewall.interfaces.eth1.allowedTCPPorts = [ 443 ]; - # hive-priv.nix links the binary the hive-egress unit runs to bind bridge - # ports; `eth1` is no bridge here, so a stub that binds nothing stands in. + # hive-priv.nix links the binary the hive-egress unit loads its tables with; + # `eth1` is no bridge here, so a stub that only loads the ruleset (no + # bindings, no leases) stands in. systemd.tmpfiles.rules = [ - "L+ /run/hive-rootfs-check - - - - ${pkgs.writeShellScript "hive-priv-stub" "exit 0"}" + "L+ /run/hive-rootfs-check - - - - ${pkgs.writeShellScript "hive-priv-stub" '' + [ "$1" = egress-load ] && exec ${pkgs.nftables}/bin/nft -f "$2" + exit 0 + ''}" ]; # Renders the users file the way hive-c0re does. environment.systemPackages = [