mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

egress anti-spoofing: keep established bindings, load tables with bindings and leases in one transaction

Browse source
This commit is contained in:
atlas 2026-11-13 02:53:37 +01:00 • committed by mara
commit 70179b6c1f
1 changed files with 400 additions and 134 deletions
Download patch file Download diff file Expand all files Collapse all files

392
hive-priv/src/egress.rs
View file

@ -1,33 +1,31 @@
//! The per-port bindings behind the `bridge hive_egress` anti-spoofing table
//! (`nix/host-modules/hive-egress.nix`), as two sub-commands run by systemd
//! (`nix/host-modules/hive-egress.nix`), as three sub-commands run by systemd
//! units and by dnsmasq, never through the socket.
//!
//! - `egress-pin [--seed] [<port>]` binds each port of the agent bridge to the
//! MAC its container uses. A port is bound only when a registered container
//! owns the other end of its veth pair; a port nobody owns stays unbound, and
//! the table drops everything it sends. With `<port>` the command first waits
//! for that port to be bound or to vanish. `--seed` also loads the leases in
//! dnsmasq's lease file.
//! - `egress-load <ruleset-file>` replaces the tables with the ruleset, the
//! current bindings and the leases of dnsmasq's lease file, in one nft
//! transaction. The bindings are looked up first, while the old tables still
//! filter. A lease load that fails is a warning; the ruleset then loads with
//! the bindings alone.
//! - `egress-pin [<port>]` binds each port of the agent bridge to the MAC its
//! container uses. A port nobody owns stays unbound, and the table drops
//! everything it sends. With `<port>` the command first waits for that port
//! to be bound or to vanish. udev runs it per new port, a timer for the ports
//! still unbound.
//! - `egress-lease <action> [<mac> <ip> [<host>]]` is dnsmasq's `dhcp-script`:
//! it keeps the table's set of leased MAC and IPv4 pairs in step with the
//! leases.
//! it keeps the table's set of leased MAC and IPv4 pairs in step.
//!
//! The owner of a port is found from the host side: the port's peer index is
//! the index of one link in the container's network namespace, and that link's
//! peer index is the port's own index. Both halves have to hold, so a veth
//! pair the container makes inside its namespace cannot claim another port.
//! The container's leader process comes from the machine state systemd-machined
//! keeps under `/run/systemd/machines`.
//! A port's owner is found from the host side: the port's peer index is the
//! index of one link in a registered container's network namespace (the leader
//! comes from `/run/systemd/machines`), and that link's peer index is the port's
//! own index. A veth pair the container makes inside its namespace can't claim
//! another port.
//!
//! The bound MAC is the one the link has when the port is first looked at, and
//! the binding then stays for as long as the port exists: later passes keep it
//! in `/run/hive-egress-pins` and do not read the link again, so root in the
//! container cannot move its port to another MAC afterwards. The unit starts
//! the command when the veth appears, while only the container's nix-built
//! early init runs. A MAC that two ports claim binds neither.
//!
//! The entry points are [`run`] (the sub-commands), [`resolve`] (which port has
//! which MAC) and [`pin_script`] / [`lease_script`] (the `nft` input).
//! The bound MAC is the one the link has when the port is first looked at. The
//! binding stays for as long as the port exists (`/run/hive-egress-pins`), and
//! a kept binding is never given up: a port that claims a MAC a kept binding
//! holds, or that another new claim shares, stays unbound and is looked at
//! again on the next pass.
use std::fmt::Write as _;
use std::fs::{File, OpenOptions};
@ -44,6 +42,9 @@ pub(crate) const PIN: &str = "egress-pin";
/// The sub-command dnsmasq's `dhcp-script` runs.
pub(crate) const LEASE: &str = "egress-lease";
/// The sub-command that loads the tables.
pub(crate) const LOAD: &str = "egress-load";
/// The bridge the table guards, set by the unit that runs the command.
const BRIDGE_ENV: &str = "HIVE_EGRESS_BRIDGE";
@ -161,22 +162,24 @@ pub(crate) struct Resolution {
///
/// `machines` holds each registered container's links. `earlier` holds the
/// bindings of a previous pass: a port that has the same name and index keeps
/// its binding whatever its link says now, so a MAC changed after the port was
/// bound is never bound in its place. A port no machine owns, an unusable MAC
/// and a MAC two ports claim all leave the port unbound.
/// its binding whatever its link says now, and no other port takes its MAC. A
/// kept binding is never given up, so a port can only get a binding for a MAC
/// nobody holds. A port no machine owns, an unusable MAC, a MAC a kept binding
/// holds and a MAC two new claims share all leave the port unbound.
pub(crate) fn resolve(
ports: &[HostPort],
machines: &[(String, Vec<NsLink>)],
earlier: &[Pin],
) -> Resolution {
let mut found: Vec<Pin> = Vec::new();
let mut kept: Vec<Pin> = Vec::new();
let mut claims: Vec<Pin> = Vec::new();
let mut out = Resolution::default();
for port in ports {
if let Some(kept) = earlier
if let Some(pin) = earlier
.iter()
.find(|p| p.port == port.name && p.ifindex == port.ifindex)
{
found.push(kept.clone());
kept.push(pin.clone());
continue;
}
let owner = machines.iter().find_map(|(machine, links)| {
@ -193,7 +196,7 @@ pub(crate) fn resolve(
port.name.clone(),
format!("{machine} has the unusable address {mac}"),
)),
Some((machine, mac)) => found.push(Pin {
Some((machine, mac)) => claims.push(Pin {
port: port.name.clone(),
ifindex: port.ifindex,
mac,
@ -201,26 +204,29 @@ pub(crate) fn resolve(
}),
}
}
for pin in &found {
let rivals: Vec<&str> = found
for claim in &claims {
let holders: Vec<&str> = kept
.iter()
.filter(|p| p.mac == pin.mac && p.port != pin.port)
.chain(claims.iter())
.filter(|p| p.mac == claim.mac && p.port != claim.port)
.map(|p| p.port.as_str())
.collect();
if rivals.is_empty() {
out.pins.push(pin.clone());
if holders.is_empty() {
out.pins.push(claim.clone());
} else {
out.unbound.push((
pin.port.clone(),
claim.port.clone(),
format!(
"{} has {}, which {} also has",
pin.machine,
pin.mac,
rivals.join(", ")
claim.machine,
claim.mac,
holders.join(", ")
),
));
}
}
kept.append(&mut out.pins);
out.pins = kept;
out
}
@ -258,10 +264,9 @@ fn is_plain_name(name: &str) -> bool {
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'_' | b'-'))
}
/// The nft input that makes the table's bindings exactly `pins`, in one
/// transaction: a reader never sees the set empty in between.
pub(crate) fn pin_script(pins: &[Pin]) -> String {
let mut script = format!("flush set {TABLE} port_mac\nflush set {TABLE} port_chaddr\n");
/// The nft input that adds the bindings `pins` to the two sets.
pub(crate) fn pin_elements(pins: &[Pin]) -> String {
let mut script = String::new();
for Pin { port, mac, .. } in pins {
let _ = writeln!(
script,
@ -273,6 +278,15 @@ pub(crate) fn pin_script(pins: &[Pin]) -> String {
script
}
/// The nft input that makes the table's bindings exactly `pins`, in one
/// transaction: a reader never sees the sets empty in between.
pub(crate) fn pin_script(pins: &[Pin]) -> String {
format!(
"flush set {TABLE} port_mac\nflush set {TABLE} port_chaddr\n{}",
pin_elements(pins)
)
}
/// The nft input for one lease event, or `None` for an event that changes
/// nothing: `init`, an IPv6 address, an address or MAC that does not parse.
///
@ -464,21 +478,59 @@ fn machine_links(dir: &Path) -> Vec<(String, Vec<NsLink>)> {
machines
}
/// One pass: look at the bridge's ports and the containers, and make the
/// table's bindings match. Returns the ports of the bridge and the resolution.
fn pin_once(bridge: &str) -> Result<(Vec<HostPort>, Resolution)> {
/// The ports no kept binding covers: the ones a pass has to look up.
pub(crate) fn ports_to_resolve<'a>(ports: &'a [HostPort], earlier: &[Pin]) -> Vec<&'a HostPort> {
ports
.iter()
.filter(|p| {
!earlier
.iter()
.any(|k| k.port == p.name && k.ifindex == p.ifindex)
})
.collect()
}
/// What a pass found out.
struct Survey {
earlier: Vec<Pin>,
resolution: Resolution,
}
fn state_path() -> String {
std::env::var("HIVE_EGRESS_STATE").unwrap_or_else(|_| STATE.into())
}
/// Look at the bridge's ports, the earlier bindings and, when a port has none,
/// the containers. Changes nothing.
fn survey(bridge: &str) -> Result<Survey> {
let ports = parse_ports(&output_of("ip", &["-j", "link", "show", "master", bridge])?)?;
let machines = machine_links(Path::new(MACHINES_DIR));
let state_path = std::env::var("HIVE_EGRESS_STATE").unwrap_or_else(|_| STATE.into());
let earlier = parse_state(&std::fs::read_to_string(&state_path).unwrap_or_default());
let earlier = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default());
let machines = if ports_to_resolve(&ports, &earlier).is_empty() {
Vec::new()
} else {
machine_links(Path::new(MACHINES_DIR))
};
let resolution = resolve(&ports, &machines, &earlier);
nft(&pin_script(&resolution.pins))?;
std::fs::write(&state_path, state_text(&resolution.pins))
.with_context(|| format!("write {state_path}"))?;
for pin in resolution.pins.iter().filter(|p| !earlier.contains(p)) {
Ok(Survey {
earlier,
resolution,
})
}
/// Keep the bindings for the next pass and print the new ones.
fn commit(survey: &Survey) -> Result<()> {
let path = state_path();
std::fs::write(&path, state_text(&survey.resolution.pins))
.with_context(|| format!("write {path}"))?;
for pin in survey
.resolution
.pins
.iter()
.filter(|p| !survey.earlier.contains(p))
{
eprintln!("{PIN}: {} bound to {} ({})", pin.port, pin.mac, pin.machine);
}
Ok((ports, resolution))
Ok(())
}
/// Wait for the lock of the bindings. It is released when the file is dropped.
@ -500,41 +552,54 @@ fn now_secs() -> u64 {
.map_or(0, |d| d.as_secs())
}
/// `egress-pin [--seed] [<port>]`.
/// Whose "stays unbound" lines a pass prints.
#[derive(Clone, Copy)]
enum Report<'a> {
All,
Port(&'a str),
Nothing,
}
fn report(resolution: &Resolution, which: Report<'_>) {
for (name, why) in &resolution.unbound {
let show = match which {
Report::All => true,
Report::Port(port) => port == name,
Report::Nothing => false,
};
if show {
eprintln!("{PIN}: {name} stays unbound: {why}");
}
}
}
/// `egress-pin [<port>]`: one pass, or with `<port>` passes until that port is
/// bound or gone. Without `<port>` it prints only what it binds: the timer
/// runs it again and again for the ports still unbound.
fn pin(args: &[String]) -> Result<()> {
let bridge = std::env::var(BRIDGE_ENV).with_context(|| format!("{BRIDGE_ENV} is not set"))?;
let mut seed = false;
let mut wait_for = None;
for arg in args {
match arg.as_str() {
"--seed" => seed = true,
port if wait_for.is_none() && is_plain_name(port) => wait_for = Some(port.to_string()),
other => bail!("usage: hive-priv {PIN} [--seed] [<port>] (got {other:?})"),
}
}
if seed {
let path = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into());
let leases = std::fs::read_to_string(&path).unwrap_or_default();
let script = seed_script(&leases, now_secs());
if !script.is_empty() {
let _lock = lock()?;
nft(&script).context("load leases")?;
}
}
let wait_for = match args {
[] => None,
[port] if is_plain_name(port) => Some(port.as_str()),
_ => bail!("usage: hive-priv {PIN} [<port>]"),
};
let deadline = Instant::now() + WAIT;
loop {
let (ports, resolution) = {
let survey = {
let _lock = lock()?;
pin_once(&bridge)?
let survey = survey(&bridge)?;
nft(&pin_script(&survey.resolution.pins))?;
commit(&survey)?;
survey
};
let Some(port) = &wait_for else {
report(&resolution, None);
let Some(port) = wait_for else {
report(&survey.resolution, Report::Nothing);
return Ok(());
};
let bound = resolution.pins.iter().any(|p| &p.port == port);
let present = ports.iter().any(|p| &p.name == port);
let bound = survey.resolution.pins.iter().any(|p| p.port == port);
let present = port_exists(port);
if bound || !present || Instant::now() >= deadline {
report(&resolution, Some(port));
report(&survey.resolution, Report::Port(port));
if present && !bound {
bail!("port {port} stays unbound");
}
@ -544,15 +609,65 @@ fn pin(args: &[String]) -> Result<()> {
}
}
/// Print why ports stay unbound; with `port`, only that one.
fn report(resolution: &Resolution, port: Option<&str>) {
for (name, why) in &resolution.unbound {
if port.is_none_or(|p| p == name) {
eprintln!("{PIN}: {name} stays unbound: {why}");
/// Whether an interface of this name exists, on any bridge or none: a port is
/// created before it joins the bridge.
fn port_exists(name: &str) -> bool {
Command::new("ip")
.args(["link", "show", "dev", name])
.output()
.is_ok_and(|o| o.status.success())
}
/// Load `ruleset` with the bindings and the leases in the same transaction, so
/// no frame meets the new table with empty sets. A lease load that `run`
/// refuses is a warning: the ruleset loads again with the bindings alone.
pub(crate) fn load_ruleset(
ruleset: &str,
pins: &str,
leases: &str,
run: impl Fn(&str) -> Result<()>,
) -> Result<()> {
match run(&format!("{ruleset}\n{pins}{leases}")) {
Err(e) if !leases.is_empty() => {
eprintln!("{LOAD}: loading the leases failed, loading without them: {e:#}");
run(&format!("{ruleset}\n{pins}")).context("load the ruleset")
}
other => other,
}
}
/// `egress-load <ruleset-file>`: replace the tables with the ruleset and the
/// current bindings and leases. The bindings are looked up first, while the old
/// tables still filter. A lookup that fails loads the ruleset with none: the
/// ports stay unbound until the timer or udev binds them.
fn load(args: &[String]) -> Result<()> {
let bridge = std::env::var(BRIDGE_ENV).with_context(|| format!("{BRIDGE_ENV} is not set"))?;
let [file] = args else {
bail!("usage: hive-priv {LOAD} <ruleset-file>");
};
let ruleset = std::fs::read_to_string(file).with_context(|| format!("read {file}"))?;
let path = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into());
let leases = seed_script(
&std::fs::read_to_string(&path).unwrap_or_default(),
now_secs(),
);
let _lock = lock()?;
let found = survey(&bridge);
let pins = match &found {
Ok(found) => pin_elements(&found.resolution.pins),
Err(e) => {
eprintln!("{LOAD}: binding the ports failed, loading without bindings: {e:#}");
String::new()
}
};
load_ruleset(&ruleset, &pins, &leases, nft)?;
if let Ok(found) = &found {
commit(found)?;
report(&found.resolution, Report::All);
}
Ok(())
}
/// `egress-lease <action> [<mac> <ip> [<host>]]`. Nothing is loaded for an event
/// that changes no lease.
fn lease(args: &[String]) -> Result<()> {
@ -575,6 +690,7 @@ pub(crate) fn run(args: &[String]) -> Option<i32> {
let (name, rest) = args.split_first()?;
let result = match name.as_str() {
PIN => pin(rest),
LOAD => load(rest),
LEASE => lease(rest),
_ => return None,
};
@ -723,6 +839,110 @@ mod tests {
assert!(resolve(&ports, &machines, &[]).pins.is_empty());
}
#[test]
fn a_kept_binding_is_never_given_up_to_a_new_claim_on_its_mac() {
let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
let earlier = [pin("vb-a", 10, 1, "a")];
let machines = [
("a".to_string(), vec![link(2, Some(10), mac(1))]),
("b".to_string(), vec![link(2, Some(11), mac(1))]),
];
let got = resolve(&ports, &machines, &earlier);
assert_eq!(got.pins, vec![pin("vb-a", 10, 1, "a")]);
assert_eq!(got.unbound.len(), 1);
assert_eq!(got.unbound[0].0, "vb-b");
}
#[test]
fn a_new_claim_on_a_free_mac_binds_next_to_a_kept_binding() {
let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
let earlier = [pin("vb-a", 10, 1, "a")];
let machines = [("b".to_string(), vec![link(2, Some(11), mac(2))])];
let got = resolve(&ports, &machines, &earlier);
assert_eq!(
got.pins,
vec![pin("vb-a", 10, 1, "a"), pin("vb-b", 11, 2, "b")]
);
assert!(got.unbound.is_empty());
}
#[test]
fn a_port_refused_for_a_taken_mac_binds_on_a_later_pass_once_it_has_its_own() {
let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
let earlier = [pin("vb-a", 10, 1, "a")];
let rival = [("b".to_string(), vec![link(2, Some(11), mac(1))])];
assert!(resolve(&ports, &rival, &earlier).pins.len() == 1);
// The rival restores its own MAC; the next pass reads its link again.
let restored = [("b".to_string(), vec![link(2, Some(11), mac(2))])];
let got = resolve(&ports, &restored, &earlier);
assert_eq!(
got.pins,
vec![pin("vb-a", 10, 1, "a"), pin("vb-b", 11, 2, "b")]
);
}
#[test]
fn only_ports_without_a_kept_binding_are_looked_up() {
let ports = [port("vb-a", 10, 2), port("vb-b", 11, 2)];
let a = pin("vb-a", 10, 1, "a");
assert_eq!(ports_to_resolve(&ports, &[]).len(), 2);
assert_eq!(
ports_to_resolve(&ports, std::slice::from_ref(&a))[0].name,
"vb-b"
);
let b = pin("vb-b", 11, 2, "b");
assert!(ports_to_resolve(&ports, &[a, b]).is_empty());
// The same name with another index is a new port.
assert_eq!(
ports_to_resolve(&[port("vb-a", 12, 2)], &[pin("vb-a", 10, 1, "a")]).len(),
1
);
}
#[test]
fn the_ruleset_loads_with_the_bindings_and_leases_in_one_script() {
let calls = std::cell::RefCell::new(Vec::new());
let pins = pin_elements(&[pin("vb-a", 10, 10, "a")]);
let leases = lease_script("add", "02:00:00:00:00:0a", "10.42.0.10", 90).unwrap();
load_ruleset("table bridge hive_egress { }", &pins, &leases, |s| {
calls.borrow_mut().push(s.to_string());
Ok(())
})
.unwrap();
let calls = calls.into_inner();
assert_eq!(calls.len(), 1, "one transaction");
let script = &calls[0];
assert!(script.starts_with("table bridge hive_egress { }"));
assert!(script.contains("port_mac { \"vb-a\" . 02:00:00:00:00:0a }"));
assert!(script.contains("lease_v4 { 02:00:00:00:00:0a . 10.42.0.10 timeout 90s }"));
assert!(!script.contains("flush set"), "the sets are new");
}
#[test]
fn a_lease_load_that_fails_is_a_warning_and_the_bindings_still_load() {
let calls = std::cell::RefCell::new(Vec::new());
let pins = pin_elements(&[pin("vb-a", 10, 10, "a")]);
let leases = lease_script("add", "02:00:00:00:00:0a", "10.42.0.10", 90).unwrap();
let result = load_ruleset("RULESET", &pins, &leases, |s| {
calls.borrow_mut().push(s.to_string());
if s.contains("lease_v4") {
anyhow::bail!("lease_v4 refused")
}
Ok(())
});
assert!(result.is_ok());
let calls = calls.into_inner();
assert_eq!(calls.len(), 2);
assert!(calls[1].contains("port_mac") && !calls[1].contains("lease_v4"));
}
#[test]
fn a_ruleset_that_fails_without_leases_is_an_error() {
let fails = |_: &str| -> Result<()> { anyhow::bail!("refused") };
assert!(load_ruleset("RULESET", "", "", fails).is_err());
assert!(load_ruleset("RULESET", "", "x", fails).is_err());
}
#[test]
fn pin_script_replaces_both_sets_in_one_script() {
let script = pin_script(&[pin("vb-a", 10, 10, "a")]);