egress anti-spoofing: keep established bindings, load tables with bindings and leases in one transaction

docs/4-implementation/networking/network.md

@@ -648,23 +648,31 @@ source check lives in that table and the host-side filtering stays in
   bound a moment after its veth appears, and the container's DHCP client
   retries until then.
 - **Where the bindings come from.** Addresses are dynamic, so the table's sets
-  the commands below fill them at run time. `hive-priv egress-pin` binds a port to the MAC of the
+  fill at run time. `hive-priv egress-pin` binds a port to the MAC of the link at
-  link at the other end of its veth pair, found through the container systemd
+  the other end of its veth pair, found through the container systemd registers
-  registers for it. udev starts it for each new `vb-*` port
+  for it. udev starts it for each new `vb-*` port
-  (`hive-egress-pin@<port>.service`), and the `hive-egress` unit starts it for
+  (`hive-egress-pin@<port>.service`), and a timer (`hive-egress-pin.timer`, every
-  every running container on each start and reload. A binding stays with the
+  30 seconds) runs it for the ports that still have no binding. The timer reads
-  port until the port goes: root in the container can't move the port to
+  no container while every port has one. The `hive-egress` unit loads the tables
-  another MAC afterwards (state in `/run/hive-egress-pins`). If two ports have
+  with `hive-priv egress-load`: it looks up the bindings first, while the old
-  the same MAC, neither gets a binding. dnsmasq's `dhcp-script` (`hive-priv
+  tables still filter, and puts them and the leases from dnsmasq's lease file
-egress-lease`) adds each lease as a (MAC, address) pair that times out with
+  into the same nft transaction as the tables. A start or reload never leaves
-  the lease.
+  the bridge with empty sets. If loading the leases fails, the unit logs a
+  warning and loads the tables with the bindings alone. A binding stays with the
+  port until the port goes: root in the container can't move the port to another
+  MAC afterwards (state in `/run/hive-egress-pins`). A port never takes a MAC
+  that a binding already holds, so a newcomer can't evict an established
+  container. If two new ports claim one MAC, neither gets a binding. dnsmasq's
+  `dhcp-script` (`hive-priv egress-lease`) adds each lease as a (MAC, address)
+  pair that times out with the lease.
 - **The window.** The binding takes the MAC the container's link has when the
   command first looks, normally within a second of the veth appearing. Until
   then root in the container could change its own MAC, but in that time only
   the container's early init runs, which is nix-built, and the agent's code
-  starts later. A container that did change its MAC first would get its own
+  starts later. A container that did change its MAC first gets a binding for
-  lease for the new MAC, not another container's: a MAC that two ports have
+  that MAC only if nobody holds it; a lease follows the `chaddr` the port may
-  gets no binding, and a lease follows the `chaddr` the port may send.
+  send. A port refused because it claimed a held MAC sends nothing, and the
+  timer binds it once its link has a MAC of its own.
 - **The log.** A refused frame (all of them with `drop`, the same ones with
   `log`) gets a kernel log line with the prefix `hive-egress-spoof `, rate
   limited per port: a burst of 20, then one a second. A frame its port can't log

hive-priv/README.md

@@ -37,9 +37,9 @@ not the whole daemon-shared crate.
 `container` (lifecycle), `container_config` (nspawn flags, systemd drop-ins),
 `storage` (btrfs subvolumes, quotas, snapshots, disk measurement), `files`
 (pause marker, socket dir, legacy tmpfiles) and `infra` (gateway nginx, forge
-admin, infra containers). `egress` holds two sub-commands that never
+admin, infra containers). `egress` holds three sub-commands that never
-touch the socket, `egress-pin` and `egress-lease`; the `hive-egress` unit, udev
+touch the socket, `egress-load`, `egress-pin` and `egress-lease`; the `hive-egress`
-and dnsmasq run them to fill the sets of the anti-spoofing table (see
+unit, udev, a timer and dnsmasq run them to fill the sets of the anti-spoofing table (see
 `docs/4-implementation/networking/network.md`). Helpers more than one family calls live in `shared`.
 Each module's `//!` header lists its ops.
 

hive-priv/src/egress.rs

@@ -1,33 +1,31 @@
 //! The per-port bindings behind the `bridge hive_egress` anti-spoofing table
-//! (`nix/host-modules/hive-egress.nix`), as two sub-commands run by systemd
+//! (`nix/host-modules/hive-egress.nix`), as three sub-commands run by systemd
 //! units and by dnsmasq, never through the socket.
 //!
-//! - `egress-pin [--seed] [<port>]` binds each port of the agent bridge to the
+//! - `egress-load <ruleset-file>` replaces the tables with the ruleset, the
-//!   MAC its container uses. A port is bound only when a registered container
+//!   current bindings and the leases of dnsmasq's lease file, in one nft
-//!   owns the other end of its veth pair; a port nobody owns stays unbound, and
+//!   transaction. The bindings are looked up first, while the old tables still
-//!   the table drops everything it sends. With `<port>` the command first waits
+//!   filter. A lease load that fails is a warning; the ruleset then loads with
-//!   for that port to be bound or to vanish. `--seed` also loads the leases in
+//!   the bindings alone.
-//!   dnsmasq's lease file.
+//! - `egress-pin [<port>]` binds each port of the agent bridge to the MAC its
+//!   container uses. A port nobody owns stays unbound, and the table drops
+//!   everything it sends. With `<port>` the command first waits for that port
+//!   to be bound or to vanish. udev runs it per new port, a timer for the ports
+//!   still unbound.
 //! - `egress-lease <action> [<mac> <ip> [<host>]]` is dnsmasq's `dhcp-script`:
-//!   it keeps the table's set of leased MAC and IPv4 pairs in step with the
+//!   it keeps the table's set of leased MAC and IPv4 pairs in step.
-//!   leases.
 //!
-//! The owner of a port is found from the host side: the port's peer index is
+//! A port's owner is found from the host side: the port's peer index is the
-//! the index of one link in the container's network namespace, and that link's
+//! index of one link in a registered container's network namespace (the leader
-//! peer index is the port's own index. Both halves have to hold, so a veth
+//! comes from `/run/systemd/machines`), and that link's peer index is the port's
-//! pair the container makes inside its namespace cannot claim another port.
+//! own index. A veth pair the container makes inside its namespace can't claim
-//! The container's leader process comes from the machine state systemd-machined
+//! another port.
-//! keeps under `/run/systemd/machines`.
 //!
-//! The bound MAC is the one the link has when the port is first looked at, and
+//! The bound MAC is the one the link has when the port is first looked at. The
-//! the binding then stays for as long as the port exists: later passes keep it
+//! binding stays for as long as the port exists (`/run/hive-egress-pins`), and
-//! in `/run/hive-egress-pins` and do not read the link again, so root in the
+//! a kept binding is never given up: a port that claims a MAC a kept binding
-//! container cannot move its port to another MAC afterwards. The unit starts
+//! holds, or that another new claim shares, stays unbound and is looked at
-//! the command when the veth appears, while only the container's nix-built
+//! again on the next pass.
-//! early init runs. A MAC that two ports claim binds neither.
-//!
-//! The entry points are [`run`] (the sub-commands), [`resolve`] (which port has
-//! which MAC) and [`pin_script`] / [`lease_script`] (the `nft` input).
 
 use std::fmt::Write as _;
 use std::fs::{File, OpenOptions};
@@ -44,6 +42,9 @@ pub(crate) const PIN: &str = "egress-pin";
 /// The sub-command dnsmasq's `dhcp-script` runs.
 pub(crate) const LEASE: &str = "egress-lease";
 
+/// The sub-command that loads the tables.
+pub(crate) const LOAD: &str = "egress-load";
+
 /// The bridge the table guards, set by the unit that runs the command.
 const BRIDGE_ENV: &str = "HIVE_EGRESS_BRIDGE";
 
@@ -161,22 +162,24 @@ pub(crate) struct Resolution {
 ///
 /// `machines` holds each registered container's links. `earlier` holds the
 /// bindings of a previous pass: a port that has the same name and index keeps
-/// its binding whatever its link says now, so a MAC changed after the port was
+/// its binding whatever its link says now, and no other port takes its MAC. A
-/// bound is never bound in its place. A port no machine owns, an unusable MAC
+/// kept binding is never given up, so a port can only get a binding for a MAC
-/// and a MAC two ports claim all leave the port unbound.
+/// nobody holds. A port no machine owns, an unusable MAC, a MAC a kept binding
+/// holds and a MAC two new claims share all leave the port unbound.
 pub(crate) fn resolve(
     ports: &[HostPort],
     machines: &[(String, Vec<NsLink>)],
     earlier: &[Pin],
 ) -> Resolution {
-    let mut found: Vec<Pin> = Vec::new();
+    let mut kept: Vec<Pin> = Vec::new();
+    let mut claims: Vec<Pin> = Vec::new();
     let mut out = Resolution::default();
     for port in ports {
-        if let Some(kept) = earlier
+        if let Some(pin) = earlier
             .iter()
             .find(|p| p.port == port.name && p.ifindex == port.ifindex)
         {
-            found.push(kept.clone());
+            kept.push(pin.clone());
             continue;
         }
         let owner = machines.iter().find_map(|(machine, links)| {
@@ -193,7 +196,7 @@ pub(crate) fn resolve(
                 port.name.clone(),
                 format!("{machine} has the unusable address {mac}"),
             )),
-            Some((machine, mac)) => found.push(Pin {
+            Some((machine, mac)) => claims.push(Pin {
                 port: port.name.clone(),
                 ifindex: port.ifindex,
                 mac,
@@ -201,26 +204,29 @@ pub(crate) fn resolve(
             }),
         }
     }
-    for pin in &found {
+    for claim in &claims {
-        let rivals: Vec<&str> = found
+        let holders: Vec<&str> = kept
             .iter()
-            .filter(|p| p.mac == pin.mac && p.port != pin.port)
+            .chain(claims.iter())
+            .filter(|p| p.mac == claim.mac && p.port != claim.port)
             .map(|p| p.port.as_str())
             .collect();
-        if rivals.is_empty() {
+        if holders.is_empty() {
-            out.pins.push(pin.clone());
+            out.pins.push(claim.clone());
         } else {
             out.unbound.push((
-                pin.port.clone(),
+                claim.port.clone(),
                 format!(
                     "{} has {}, which {} also has",
-                    pin.machine,
+                    claim.machine,
-                    pin.mac,
+                    claim.mac,
-                    rivals.join(", ")
+                    holders.join(", ")
                 ),
             ));
         }
     }
+    kept.append(&mut out.pins);
+    out.pins = kept;
     out
 }
 
@@ -258,10 +264,9 @@ fn is_plain_name(name: &str) -> bool {
             .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'_' | b'-'))
 }
 
-/// The nft input that makes the table's bindings exactly `pins`, in one
+/// The nft input that adds the bindings `pins` to the two sets.
-/// transaction: a reader never sees the set empty in between.
+pub(crate) fn pin_elements(pins: &[Pin]) -> String {
-pub(crate) fn pin_script(pins: &[Pin]) -> String {
+    let mut script = String::new();
-    let mut script = format!("flush set {TABLE} port_mac\nflush set {TABLE} port_chaddr\n");
     for Pin { port, mac, .. } in pins {
         let _ = writeln!(
             script,
@@ -273,6 +278,15 @@ pub(crate) fn pin_script(pins: &[Pin]) -> String {
     script
 }
 
+/// The nft input that makes the table's bindings exactly `pins`, in one
+/// transaction: a reader never sees the sets empty in between.
+pub(crate) fn pin_script(pins: &[Pin]) -> String {
+    format!(
+        "flush set {TABLE} port_mac\nflush set {TABLE} port_chaddr\n{}",
+        pin_elements(pins)
+    )
+}
+
 /// The nft input for one lease event, or `None` for an event that changes
 /// nothing: `init`, an IPv6 address, an address or MAC that does not parse.
 ///
@@ -464,21 +478,59 @@ fn machine_links(dir: &Path) -> Vec<(String, Vec<NsLink>)> {
     machines
 }
 
-/// One pass: look at the bridge's ports and the containers, and make the
+/// The ports no kept binding covers: the ones a pass has to look up.
-/// table's bindings match. Returns the ports of the bridge and the resolution.
+pub(crate) fn ports_to_resolve<'a>(ports: &'a [HostPort], earlier: &[Pin]) -> Vec<&'a HostPort> {
-fn pin_once(bridge: &str) -> Result<(Vec<HostPort>, Resolution)> {
+    ports
+        .iter()
+        .filter(|p| {
+            !earlier
+                .iter()
+                .any(|k| k.port == p.name && k.ifindex == p.ifindex)
+        })
+        .collect()
+}
+
+/// What a pass found out.
+struct Survey {
+    earlier: Vec<Pin>,
+    resolution: Resolution,
+}
+
+fn state_path() -> String {
+    std::env::var("HIVE_EGRESS_STATE").unwrap_or_else(|_| STATE.into())
+}
+
+/// Look at the bridge's ports, the earlier bindings and, when a port has none,
+/// the containers. Changes nothing.
+fn survey(bridge: &str) -> Result<Survey> {
     let ports = parse_ports(&output_of("ip", &["-j", "link", "show", "master", bridge])?)?;
-    let machines = machine_links(Path::new(MACHINES_DIR));
+    let earlier = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default());
-    let state_path = std::env::var("HIVE_EGRESS_STATE").unwrap_or_else(|_| STATE.into());
+    let machines = if ports_to_resolve(&ports, &earlier).is_empty() {
-    let earlier = parse_state(&std::fs::read_to_string(&state_path).unwrap_or_default());
+        Vec::new()
+    } else {
+        machine_links(Path::new(MACHINES_DIR))
+    };
     let resolution = resolve(&ports, &machines, &earlier);
-    nft(&pin_script(&resolution.pins))?;
+    Ok(Survey {
-    std::fs::write(&state_path, state_text(&resolution.pins))
+        earlier,
-        .with_context(|| format!("write {state_path}"))?;
+        resolution,
-    for pin in resolution.pins.iter().filter(|p| !earlier.contains(p)) {
+    })
+}
+
+/// Keep the bindings for the next pass and print the new ones.
+fn commit(survey: &Survey) -> Result<()> {
+    let path = state_path();
+    std::fs::write(&path, state_text(&survey.resolution.pins))
+        .with_context(|| format!("write {path}"))?;
+    for pin in survey
+        .resolution
+        .pins
+        .iter()
+        .filter(|p| !survey.earlier.contains(p))
+    {
         eprintln!("{PIN}: {} bound to {} ({})", pin.port, pin.mac, pin.machine);
     }
-    Ok((ports, resolution))
+    Ok(())
 }
 
 /// Wait for the lock of the bindings. It is released when the file is dropped.
@@ -500,41 +552,54 @@ fn now_secs() -> u64 {
         .map_or(0, |d| d.as_secs())
 }
 
-/// `egress-pin [--seed] [<port>]`.
+/// Whose "stays unbound" lines a pass prints.
+#[derive(Clone, Copy)]
+enum Report<'a> {
+    All,
+    Port(&'a str),
+    Nothing,
+}
+
+fn report(resolution: &Resolution, which: Report<'_>) {
+    for (name, why) in &resolution.unbound {
+        let show = match which {
+            Report::All => true,
+            Report::Port(port) => port == name,
+            Report::Nothing => false,
+        };
+        if show {
+            eprintln!("{PIN}: {name} stays unbound: {why}");
+        }
+    }
+}
+
+/// `egress-pin [<port>]`: one pass, or with `<port>` passes until that port is
+/// bound or gone. Without `<port>` it prints only what it binds: the timer
+/// runs it again and again for the ports still unbound.
 fn pin(args: &[String]) -> Result<()> {
     let bridge = std::env::var(BRIDGE_ENV).with_context(|| format!("{BRIDGE_ENV} is not set"))?;
-    let mut seed = false;
+    let wait_for = match args {
-    let mut wait_for = None;
+        [] => None,
-    for arg in args {
+        [port] if is_plain_name(port) => Some(port.as_str()),
-        match arg.as_str() {
+        _ => bail!("usage: hive-priv {PIN} [<port>]"),
-            "--seed" => seed = true,
+    };
-            port if wait_for.is_none() && is_plain_name(port) => wait_for = Some(port.to_string()),
-            other => bail!("usage: hive-priv {PIN} [--seed] [<port>] (got {other:?})"),
-        }
-    }
-    if seed {
-        let path = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into());
-        let leases = std::fs::read_to_string(&path).unwrap_or_default();
-        let script = seed_script(&leases, now_secs());
-        if !script.is_empty() {
-            let _lock = lock()?;
-            nft(&script).context("load leases")?;
-        }
-    }
     let deadline = Instant::now() + WAIT;
     loop {
-        let (ports, resolution) = {
+        let survey = {
             let _lock = lock()?;
-            pin_once(&bridge)?
+            let survey = survey(&bridge)?;
+            nft(&pin_script(&survey.resolution.pins))?;
+            commit(&survey)?;
+            survey
         };
-        let Some(port) = &wait_for else {
+        let Some(port) = wait_for else {
-            report(&resolution, None);
+            report(&survey.resolution, Report::Nothing);
             return Ok(());
         };
-        let bound = resolution.pins.iter().any(|p| &p.port == port);
+        let bound = survey.resolution.pins.iter().any(|p| p.port == port);
-        let present = ports.iter().any(|p| &p.name == port);
+        let present = port_exists(port);
         if bound || !present || Instant::now() >= deadline {
-            report(&resolution, Some(port));
+            report(&survey.resolution, Report::Port(port));
             if present && !bound {
                 bail!("port {port} stays unbound");
             }
@@ -544,13 +609,63 @@ fn pin(args: &[String]) -> Result<()> {
     }
 }
 
-/// Print why ports stay unbound; with `port`, only that one.
+/// Whether an interface of this name exists, on any bridge or none: a port is
-fn report(resolution: &Resolution, port: Option<&str>) {
+/// created before it joins the bridge.
-    for (name, why) in &resolution.unbound {
+fn port_exists(name: &str) -> bool {
-        if port.is_none_or(|p| p == name) {
+    Command::new("ip")
-            eprintln!("{PIN}: {name} stays unbound: {why}");
+        .args(["link", "show", "dev", name])
+        .output()
+        .is_ok_and(|o| o.status.success())
+}
+
+/// Load `ruleset` with the bindings and the leases in the same transaction, so
+/// no frame meets the new table with empty sets. A lease load that `run`
+/// refuses is a warning: the ruleset loads again with the bindings alone.
+pub(crate) fn load_ruleset(
+    ruleset: &str,
+    pins: &str,
+    leases: &str,
+    run: impl Fn(&str) -> Result<()>,
+) -> Result<()> {
+    match run(&format!("{ruleset}\n{pins}{leases}")) {
+        Err(e) if !leases.is_empty() => {
+            eprintln!("{LOAD}: loading the leases failed, loading without them: {e:#}");
+            run(&format!("{ruleset}\n{pins}")).context("load the ruleset")
+        }
+        other => other,
     }
 }
+
+/// `egress-load <ruleset-file>`: replace the tables with the ruleset and the
+/// current bindings and leases. The bindings are looked up first, while the old
+/// tables still filter. A lookup that fails loads the ruleset with none: the
+/// ports stay unbound until the timer or udev binds them.
+fn load(args: &[String]) -> Result<()> {
+    let bridge = std::env::var(BRIDGE_ENV).with_context(|| format!("{BRIDGE_ENV} is not set"))?;
+    let [file] = args else {
+        bail!("usage: hive-priv {LOAD} <ruleset-file>");
+    };
+    let ruleset = std::fs::read_to_string(file).with_context(|| format!("read {file}"))?;
+    let path = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into());
+    let leases = seed_script(
+        &std::fs::read_to_string(&path).unwrap_or_default(),
+        now_secs(),
+    );
+    let _lock = lock()?;
+    let found = survey(&bridge);
+    let pins = match &found {
+        Ok(found) => pin_elements(&found.resolution.pins),
+        Err(e) => {
+            eprintln!("{LOAD}: binding the ports failed, loading without bindings: {e:#}");
+            String::new()
+        }
+    };
+    load_ruleset(&ruleset, &pins, &leases, nft)?;
+    if let Ok(found) = &found {
+        commit(found)?;
+        report(&found.resolution, Report::All);
+    }
+    Ok(())
 }
 
 /// `egress-lease <action> [<mac> <ip> [<host>]]`. Nothing is loaded for an event
@@ -575,6 +690,7 @@ pub(crate) fn run(args: &[String]) -> Option<i32> {
     let (name, rest) = args.split_first()?;
     let result = match name.as_str() {
         PIN => pin(rest),
+        LOAD => load(rest),
         LEASE => lease(rest),
         _ => return None,
     };
@@ -723,6 +839,110 @@ mod tests {
         assert!(resolve(&ports, &machines, &[]).pins.is_empty());
     }
 
+    #[test]
+    fn a_kept_binding_is_never_given_up_to_a_new_claim_on_its_mac() {
+        let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
+        let earlier = [pin("vb-a", 10, 1, "a")];
+        let machines = [
+            ("a".to_string(), vec![link(2, Some(10), mac(1))]),
+            ("b".to_string(), vec![link(2, Some(11), mac(1))]),
+        ];
+        let got = resolve(&ports, &machines, &earlier);
+        assert_eq!(got.pins, vec![pin("vb-a", 10, 1, "a")]);
+        assert_eq!(got.unbound.len(), 1);
+        assert_eq!(got.unbound[0].0, "vb-b");
+    }
+
+    #[test]
+    fn a_new_claim_on_a_free_mac_binds_next_to_a_kept_binding() {
+        let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
+        let earlier = [pin("vb-a", 10, 1, "a")];
+        let machines = [("b".to_string(), vec![link(2, Some(11), mac(2))])];
+        let got = resolve(&ports, &machines, &earlier);
+        assert_eq!(
+            got.pins,
+            vec![pin("vb-a", 10, 1, "a"), pin("vb-b", 11, 2, "b")]
+        );
+        assert!(got.unbound.is_empty());
+    }
+
+    #[test]
+    fn a_port_refused_for_a_taken_mac_binds_on_a_later_pass_once_it_has_its_own() {
+        let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
+        let earlier = [pin("vb-a", 10, 1, "a")];
+        let rival = [("b".to_string(), vec![link(2, Some(11), mac(1))])];
+        assert!(resolve(&ports, &rival, &earlier).pins.len() == 1);
+        // The rival restores its own MAC; the next pass reads its link again.
+        let restored = [("b".to_string(), vec![link(2, Some(11), mac(2))])];
+        let got = resolve(&ports, &restored, &earlier);
+        assert_eq!(
+            got.pins,
+            vec![pin("vb-a", 10, 1, "a"), pin("vb-b", 11, 2, "b")]
+        );
+    }
+
+    #[test]
+    fn only_ports_without_a_kept_binding_are_looked_up() {
+        let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
+        let a = pin("vb-a", 10, 1, "a");
+        assert_eq!(ports_to_resolve(&ports, &[]).len(), 2);
+        assert_eq!(
+            ports_to_resolve(&ports, std::slice::from_ref(&a))[0].name,
+            "vb-b"
+        );
+        let b = pin("vb-b", 11, 2, "b");
+        assert!(ports_to_resolve(&ports, &[a, b]).is_empty());
+        // The same name with another index is a new port.
+        assert_eq!(
+            ports_to_resolve(&[port("vb-a", 12, 2)], &[pin("vb-a", 10, 1, "a")]).len(),
+            1
+        );
+    }
+
+    #[test]
+    fn the_ruleset_loads_with_the_bindings_and_leases_in_one_script() {
+        let calls = std::cell::RefCell::new(Vec::new());
+        let pins = pin_elements(&[pin("vb-a", 10, 10, "a")]);
+        let leases = lease_script("add", "02:00:00:00:00:0a", "10.42.0.10", 90).unwrap();
+        load_ruleset("table bridge hive_egress { }", &pins, &leases, |s| {
+            calls.borrow_mut().push(s.to_string());
+            Ok(())
+        })
+        .unwrap();
+        let calls = calls.into_inner();
+        assert_eq!(calls.len(), 1, "one transaction");
+        let script = &calls[0];
+        assert!(script.starts_with("table bridge hive_egress { }"));
+        assert!(script.contains("port_mac { \"vb-a\" . 02:00:00:00:00:0a }"));
+        assert!(script.contains("lease_v4 { 02:00:00:00:00:0a . 10.42.0.10 timeout 90s }"));
+        assert!(!script.contains("flush set"), "the sets are new");
+    }
+
+    #[test]
+    fn a_lease_load_that_fails_is_a_warning_and_the_bindings_still_load() {
+        let calls = std::cell::RefCell::new(Vec::new());
+        let pins = pin_elements(&[pin("vb-a", 10, 10, "a")]);
+        let leases = lease_script("add", "02:00:00:00:00:0a", "10.42.0.10", 90).unwrap();
+        let result = load_ruleset("RULESET", &pins, &leases, |s| {
+            calls.borrow_mut().push(s.to_string());
+            if s.contains("lease_v4") {
+                anyhow::bail!("lease_v4 refused")
+            }
+            Ok(())
+        });
+        assert!(result.is_ok());
+        let calls = calls.into_inner();
+        assert_eq!(calls.len(), 2);
+        assert!(calls[1].contains("port_mac") && !calls[1].contains("lease_v4"));
+    }
+
+    #[test]
+    fn a_ruleset_that_fails_without_leases_is_an_error() {
+        let fails = |_: &str| -> Result<()> { anyhow::bail!("refused") };
+        assert!(load_ruleset("RULESET", "", "", fails).is_err());
+        assert!(load_ruleset("RULESET", "", "x", fails).is_err());
+    }
+
     #[test]
     fn pin_script_replaces_both_sets_in_one_script() {
         let script = pin_script(&[pin("vb-a", 10, 10, "a")]);

nix/host-modules/hive-egress.nix

@@ -18,9 +18,11 @@
 # traffic between two ports is bridged: the `inet` hooks see it as coming from
 # the bridge device, and only a bridge hook knows the port it came in on
 # (`iifname` is the veth). Which MAC a port may use and which IPv4 address a MAC
-# may use are not known when the ruleset loads, so they are set contents that
+# may use are not known until a container runs, so they are set contents.
-# `hive-priv egress-pin` (per port, from udev) and dnsmasq's `dhcp-script`
+# `hive-priv egress-load` puts the current ones into the same transaction as the
-# (per lease) fill; `hive-priv`'s `egress.rs` documents how.
+# tables; `egress-pin` (per new port from udev, and from a timer for the ports
+# still unbound) and dnsmasq's `dhcp-script` (per lease) keep them current.
+# `hive-priv`'s `egress.rs` documents how.
 {
   lib,
   config,
@@ -39,7 +41,8 @@ let
   # the one every platform container's unit uses (hive-priv.nix points it at the
   # current build).
   hivePriv = (import ./lib/container-rootfs-check.nix { inherit config; }).hivePriv;
-  pinCommand = args: "${hivePriv} egress-pin ${args}";
+  pinCommand = args: lib.trim "${hivePriv} egress-pin ${args}";
+  loadCommand = "${hivePriv} egress-load ${ruleset}";
   pinPath = [
     pkgs.nftables
     pkgs.iproute2
@@ -308,14 +311,11 @@ in
       serviceConfig = {
         Type = "oneshot";
         RemainAfterExit = true;
-        ExecStart = "${nft} -f ${ruleset}";
+        # The tables load with the bindings of the running containers' ports and
-        # The sets of a freshly loaded bridge table are empty: bind the ports of
+        # the leases of dnsmasq's file in the same transaction, so a reload never
-        # the running containers and load the leases from dnsmasq's file.
+        # leaves the bridge with empty sets.
-        ExecStartPost = pinCommand "--seed";
+        ExecStart = loadCommand;
-        ExecReload = [
+        ExecReload = loadCommand;
-          "${nft} -f ${ruleset}"
-          (pinCommand "--seed")
-        ];
         ExecStop = [
           "-${nft} delete table inet hive_egress"
           "-${nft} delete table bridge hive_egress"
@@ -337,6 +337,30 @@ in
       };
     };
 
+    # Looks again at the ports still unbound. A port a rival MAC kept unbound
+    # sends nothing, so nothing else would tell the hive it can be bound now.
+    # Ports with a binding cost a listing of the bridge and no container is read.
+    systemd.services.hive-egress-pin = {
+      description = "bind the agent bridge ports that have no binding yet";
+      wants = [ "hive-egress.service" ];
+      after = [ "hive-egress.service" ];
+      path = pinPath;
+      environment = pinEnvironment;
+      serviceConfig = {
+        Type = "oneshot";
+        ExecStart = pinCommand "";
+      };
+    };
+    systemd.timers.hive-egress-pin = {
+      description = "bind the agent bridge ports that have no binding yet";
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        OnBootSec = "1min";
+        OnUnitActiveSec = "30s";
+        AccuracySec = "5s";
+      };
+    };
+
     # nspawn names the host end of a container's veth `vb-<machine>` when the
     # container joins a bridge.
     services.udev.extraRules = ''

nix/host-modules/hive-gateway/dnsmasq.nix

@@ -81,7 +81,7 @@ in
     # bridgeIp/bridgePrefixLength. All containers (agents and service
     # containers such as swarm-ci-runner) receive their IPs dynamically.
     dhcp-range = "${networkCfg.dhcpRangeStart},${networkCfg.dhcpRangeEnd},1h";
-    # `hive-priv egress-pin --seed` reads this file back (hive-egress.nix).
+    # `hive-priv egress-load` reads this file back (hive-egress.nix).
     dhcp-leasefile = "/var/lib/dnsmasq/dnsmasq.leases";
     # Each lease change updates the leased (MAC, IPv4) pairs the bridge
     # anti-spoofing table checks sources against (hive-egress.nix).

nix/module-eval/core-toggle.nix

@@ -1624,17 +1624,24 @@ let
         && lib.all (l: lib.hasSuffix " drop" l) (spoofOf bare);
     }
     {
-      # The sets are filled at run time, from three places: the unit at start
+      # The sets are filled at run time, from four places: the unit at start and
-      # and reload, the udev rule for each new port, and dnsmasq for each
+      # reload (tables, bindings and leases in one transaction), the udev rule
-      # lease. Each is present on `bare`; the host without the bridge has none.
+      # for each new port, a timer for the ports still unbound, and dnsmasq for
-      name = "the bridge table's sets are filled by the unit, a udev rule per port and dnsmasq's lease script";
+      # each lease. Each is present on `bare`; the host without the bridge has
+      # none.
+      name = "the bridge table's sets load with the table, and a udev rule, a timer and dnsmasq's lease script keep them current";
       ok =
         let
           unit = bare.systemd.services.hive-egress;
           pin = bare.systemd.services."hive-egress-pin@";
+          sweep = bare.systemd.services.hive-egress-pin;
         in
-        lib.hasInfix "egress-pin --seed" unit.serviceConfig.ExecStartPost
+        lib.hasInfix "egress-load " unit.serviceConfig.ExecStart
-        && lib.any (lib.hasInfix "egress-pin --seed") unit.serviceConfig.ExecReload
+        && lib.hasInfix "hive-egress.nft" unit.serviceConfig.ExecStart
+        && unit.serviceConfig.ExecReload == unit.serviceConfig.ExecStart
+        && !(unit.serviceConfig ? ExecStartPost)
+        && lib.hasSuffix "egress-pin" sweep.serviceConfig.ExecStart
+        && bare.systemd.timers.hive-egress-pin.timerConfig.OnUnitActiveSec == "30s"
         && unit.environment.HIVE_EGRESS_BRIDGE == "hive-br0"
         && lib.hasInfix "egress-pin %i" pin.serviceConfig.ExecStart
         && lib.hasInfix ''KERNEL=="vb-*"'' bare.services.udev.extraRules
@@ -1642,7 +1649,8 @@ let
         &&
           bare.services.dnsmasq.settings.dhcp-script
           == toString bare.services.hyperhive.network.egress.leaseScript
-        && !(centralToggleOff.systemd.services ? "hive-egress-pin@");
+        && !(centralToggleOff.systemd.services ? "hive-egress-pin@")
+        && !(centralToggleOff.systemd ? timers && centralToggleOff.systemd.timers ? hive-egress-pin);
     }
     {
       # `accept` renders the table without the forward chain and without the

nix/vm-tests/hive-egress-antispoof.nix

@@ -2,7 +2,7 @@
 # ../host-modules/hive-egress.nix on a real agent bridge, with two nspawn
 # containers on it that take their addresses from the hive's dnsmasq
 # (../host-modules/hive-gateway/dnsmasq.nix' lease script, the real
-# `hive-priv egress-pin` and `egress-lease`). `hive-squid` cannot host this: its
+# `hive-priv egress-load`, `egress-pin` and `egress-lease`). `hive-squid` cannot host this: its
 # stand-in bridge is one NIC, so every agent shares one port.
 #
 # What it pins:
@@ -14,9 +14,10 @@
 #     that is not link-local and a DHCP message with another's `chaddr` are
 #     dropped, and each leaves a `hive-egress-spoof ` line that names the port;
 #   - a DHCP renewal, a link-local IPv6 packet and a restored MAC go through;
-#   - a port with no bound MAC sends nothing, and the pin unit binds it again;
+#   - a port with no bound MAC sends nothing, and the timer binds it again;
-#   - `antiSpoof = "log"`, reached by a reload of the unit, keeps the bindings
+#   - `antiSpoof = "log"`, reached by a reload of the unit, brings the bindings
-#     and the leases, lets the same spoofs through and logs them.
+#     and the leases with the new table (nothing is dropped in between),
+#     lets the same spoofs through and logs them.
 { pkgs, hivePriv }:
 let
   bridge = "hive0";
@@ -265,20 +266,21 @@ pkgs.testers.runNixOSTest {
         host.wait_until_succeeds(f"nft list set {TABLE} lease_v4 | grep -q '{MAC_A} . {IP_A}'")
         assert ct_ok("a", "ping -c1 -W2 ${gateway}")
 
-    with subtest("a port with no bound MAC sends nothing, and the pin unit binds it again"):
+    with subtest("a port with no bound MAC sends nothing, and the timer binds it again"):
         host.succeed(f"nft flush set {TABLE} port_mac")
         assert not ct_ok("a", "ping -c1 -W2 ${gateway}")
         assert not ct_ok("b", "ping -c1 -W2 ${gateway}")
-        host.succeed("systemctl start hive-egress-pin@vb-a.service hive-egress-pin@vb-b.service")
+        # Nothing starts the pin unit by hand: the timer finds the unbound ports.
         host.wait_until_succeeds("nixos-container run a -- ping -c1 -W2 ${gateway}")
         host.wait_until_succeeds("nixos-container run b -- ping -c1 -W2 ${gateway}")
 
     with subtest("antiSpoof = log keeps the bindings and the leases, lets a spoofed datagram through and logs it"):
         host.succeed("/run/current-system/specialisation/logmode/bin/switch-to-configuration test")
         assert "counter accept" in host.succeed(f"nft list chain {TABLE} spoof"), "the table was reloaded in log mode"
-        host.wait_until_succeeds(f"nft list set {TABLE} port_mac | grep -q '\"vb-a\" . {MAC_A}'")
+        # No waiting: the reload put the sets in with the table.
-        host.wait_until_succeeds(f"nft list set {TABLE} lease_v4 | grep -q '{MAC_B} . {IP_B}'")
+        host.succeed(f"nft list set {TABLE} port_mac | grep -q '\"vb-a\" . {MAC_A}'")
-        assert ct_ok("a", "ping -c1 -W2 ${gateway}"), "control: own address, after the reload"
+        host.succeed(f"nft list set {TABLE} lease_v4 | grep -q '{MAC_B} . {IP_B}'")
+        assert ct_ok("a", "ping -c1 -W2 ${gateway}"), "control: own address, right after the reload"
         seen = len(spoof_lines())
         ct("a", f"ip addr add {IP_B}/24 dev eth0")
         assert udp_reaches_host("a", IP_B), "log mode lets the spoofed datagram through"

nix/vm-tests/hive-squid.nix

@@ -197,10 +197,14 @@ pkgs.testers.runNixOSTest {
         );
         # The gateway: nginx answers on the bridge, which agents reach directly.
         networking.firewall.interfaces.eth1.allowedTCPPorts = [ 443 ];
-        # hive-priv.nix links the binary the hive-egress unit runs to bind bridge
+        # hive-priv.nix links the binary the hive-egress unit loads its tables with;
-        # ports; `eth1` is no bridge here, so a stub that binds nothing stands in.
+        # `eth1` is no bridge here, so a stub that only loads the ruleset (no
+        # bindings, no leases) stands in.
         systemd.tmpfiles.rules = [
-          "L+ /run/hive-rootfs-check - - - - ${pkgs.writeShellScript "hive-priv-stub" "exit 0"}"
+          "L+ /run/hive-rootfs-check - - - - ${pkgs.writeShellScript "hive-priv-stub" ''
+            [ "$1" = egress-load ] && exec ${pkgs.nftables}/bin/nft -f "$2"
+            exit 0
+          ''}"
         ];
         # Renders the users file the way hive-c0re does.
         environment.systemPackages = [