egress-pin: a sweep says why a port stays unbound, which veths are off the bridge and which machines it could not read
A port nobody owns, a vb- veth that never joined the bridge and a machine whose state file or namespace can't be read all left the bridge sweep silent, so a container stayed unbound for 11 hours with nothing in the journal. A timer run now prints one line for each, after the finding has stood for 60 s and once until it clears, kept in <state file>.diag. Nothing it binds changes.
This commit is contained in:
parent
62551e004e
commit
94dd24fa99
4 changed files with 658 additions and 38 deletions
|
|
@ -911,6 +911,16 @@ source check lives in that table and the host-side filtering stays in
|
||||||
is what the table is for: without it the egress log can't tell agents apart.
|
is what the table is for: without it the egress log can't tell agents apart.
|
||||||
A binding that failed shows as `hive-egress-pin` errors in the journal and
|
A binding that failed shows as `hive-egress-pin` errors in the journal and
|
||||||
`IN=` lines for that port.
|
`IN=` lines for that port.
|
||||||
|
- **A port that stays unbound.** The timer's run of `hive-egress-pin` says why
|
||||||
|
a port is still unbound after a minute (a port is unbound for a moment at
|
||||||
|
every container start, so it waits). `journalctl -u hive-egress-pin` then
|
||||||
|
shows `<port> (ifindex N, peer M) has no owner; machines seen: <name>=<index>/<peer>, …`
|
||||||
|
when no container's link pairs with the port, or `… stays unbound: <reason>`
|
||||||
|
when one does. A `vb-` veth that isn't on the bridge gets a line the same
|
||||||
|
way (`<veth> is not on hive-br0`), and so does a registered machine the run
|
||||||
|
couldn't read (`machine <name> is skipped: <reason>`). Each line comes once,
|
||||||
|
and again only after it has cleared (the run keeps
|
||||||
|
`/run/hive-egress-pins.diag`). None of them changes which ports get a binding.
|
||||||
- **What it doesn't cover.** A container can still send from any IPv6
|
- **What it doesn't cover.** A container can still send from any IPv6
|
||||||
link-local address and answer DHCP as a server on its own leased address. A
|
link-local address and answer DHCP as a server on its own leased address. A
|
||||||
DHCP client message to the broadcast or bridge address carries any IPv4 source
|
DHCP client message to the broadcast or bridge address carries any IPv4 source
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@
|
||||||
//! container uses. A port nobody owns stays unbound, and the table drops
|
//! container uses. A port nobody owns stays unbound, and the table drops
|
||||||
//! everything it sends. With `<port>` the command first waits for that port
|
//! everything it sends. With `<port>` the command first waits for that port
|
||||||
//! to be bound or to vanish. udev runs it per new port, a timer for the ports
|
//! to be bound or to vanish. udev runs it per new port, a timer for the ports
|
||||||
//! still unbound.
|
//! still unbound; that run also says why one stays ([`crate::egress_diag`]).
|
||||||
//! - `egress-lease <action> [<mac> <ip> [<host>]]` is dnsmasq's `dhcp-script`:
|
//! - `egress-lease <action> [<mac> <ip> [<host>]]` is dnsmasq's `dhcp-script`:
|
||||||
//! it keeps the table's set of leased MAC and IPv4 pairs in step.
|
//! it keeps the table's set of leased MAC and IPv4 pairs in step.
|
||||||
//! - `egress-whois` is squid's external ACL helper: it answers a source address
|
//! - `egress-whois` is squid's external ACL helper: it answers a source address
|
||||||
|
|
@ -40,6 +40,7 @@ use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
use anyhow::{Context as _, Result, bail};
|
use anyhow::{Context as _, Result, bail};
|
||||||
|
|
||||||
|
use crate::egress_diag;
|
||||||
use crate::egress_ech::same_host;
|
use crate::egress_ech::same_host;
|
||||||
|
|
||||||
/// The sub-command squid runs to learn which agent holds a source address.
|
/// The sub-command squid runs to learn which agent holds a source address.
|
||||||
|
|
@ -197,6 +198,20 @@ pub(crate) struct Resolution {
|
||||||
pub(crate) unbound: Vec<(String, String)>,
|
pub(crate) unbound: Vec<(String, String)>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The machine that owns `port`, and the MAC of its link: the link whose index
|
||||||
|
/// is the port's peer index and whose own peer index is the port's index.
|
||||||
|
pub(crate) fn owner_of<'a>(
|
||||||
|
port: &HostPort,
|
||||||
|
machines: &'a [(String, Vec<NsLink>)],
|
||||||
|
) -> Option<(&'a str, Mac)> {
|
||||||
|
machines.iter().find_map(|(machine, links)| {
|
||||||
|
links
|
||||||
|
.iter()
|
||||||
|
.find(|l| l.ifindex == port.peer_index && l.peer_index == Some(port.ifindex))
|
||||||
|
.map(|l| (machine.as_str(), l.mac))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Bind each port to the MAC of the link at the other end of its veth pair.
|
/// Bind each port to the MAC of the link at the other end of its veth pair.
|
||||||
///
|
///
|
||||||
/// `machines` holds each registered container's links. `earlier` holds the
|
/// `machines` holds each registered container's links. `earlier` holds the
|
||||||
|
|
@ -221,13 +236,7 @@ pub(crate) fn resolve(
|
||||||
kept.push(pin.clone());
|
kept.push(pin.clone());
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let owner = machines.iter().find_map(|(machine, links)| {
|
match owner_of(port, machines) {
|
||||||
links
|
|
||||||
.iter()
|
|
||||||
.find(|l| l.ifindex == port.peer_index && l.peer_index == Some(port.ifindex))
|
|
||||||
.map(|l| (machine.as_str(), l.mac))
|
|
||||||
});
|
|
||||||
match owner {
|
|
||||||
None => out
|
None => out
|
||||||
.unbound
|
.unbound
|
||||||
.push((port.name.clone(), "no registered container owns it".into())),
|
.push((port.name.clone(), "no registered container owns it".into())),
|
||||||
|
|
@ -295,7 +304,7 @@ pub(crate) fn parse_state(text: &str) -> Vec<Pin> {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether a name is safe to put into an nft script as a quoted string.
|
/// Whether a name is safe to put into an nft script as a quoted string.
|
||||||
fn is_plain_name(name: &str) -> bool {
|
pub(crate) fn is_plain_name(name: &str) -> bool {
|
||||||
!name.is_empty()
|
!name.is_empty()
|
||||||
&& name.len() < 16
|
&& name.len() < 16
|
||||||
&& name
|
&& name
|
||||||
|
|
@ -705,18 +714,37 @@ pub(crate) fn parse_ports(json: &str) -> Result<Vec<HostPort>> {
|
||||||
.collect())
|
.collect())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The leader of a machine's state file, or `None` when it is not a container.
|
/// The leader a machine's state file names, whatever the machine is.
|
||||||
pub(crate) fn container_leader(state: &str) -> Option<u32> {
|
fn state_leader(state: &str) -> Option<u32> {
|
||||||
let mut leader = None;
|
state
|
||||||
let mut container = false;
|
.lines()
|
||||||
for line in state.lines() {
|
.find_map(|line| line.strip_prefix("LEADER=")?.parse().ok())
|
||||||
match line.split_once('=') {
|
|
||||||
Some(("CLASS", "container")) => container = true,
|
|
||||||
Some(("LEADER", pid)) => leader = pid.parse().ok(),
|
|
||||||
_ => {}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The leader of a machine's state file, or why the machine is not read.
|
||||||
|
pub(crate) fn container_leader(state: &str) -> Result<u32, &'static str> {
|
||||||
|
if !state.lines().any(|line| line == "CLASS=container") {
|
||||||
|
return Err("it is not a container");
|
||||||
}
|
}
|
||||||
leader.filter(|_| container)
|
state_leader(state).ok_or("its state file has no usable LEADER")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A registered machine a pass did not read, and why.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub(crate) struct Skip {
|
||||||
|
pub(crate) machine: String,
|
||||||
|
/// The leader its state file names: a machine started again has another,
|
||||||
|
/// and counts as new.
|
||||||
|
pub(crate) leader: Option<u32>,
|
||||||
|
pub(crate) why: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What reading the registered machines found.
|
||||||
|
#[derive(Debug, Default)]
|
||||||
|
pub(crate) struct Machines {
|
||||||
|
/// Each container whose namespace could be read, with its links.
|
||||||
|
pub(crate) links: Vec<(String, Vec<NsLink>)>,
|
||||||
|
pub(crate) skipped: Vec<Skip>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Run `program` with `args`, `input` on its stdin, and collect what it
|
/// Run `program` with `args`, `input` on its stdin, and collect what it
|
||||||
|
|
@ -815,22 +843,38 @@ fn nft(script: &str) -> Result<()> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The links of every registered container, by machine name. A machine whose
|
/// The links of every registered container, by machine name. A machine that
|
||||||
/// namespace cannot be read is reported and skipped.
|
/// cannot be read is listed with the reason, not reported: a sweep says it
|
||||||
fn machine_links(dir: &Path) -> Vec<(String, Vec<NsLink>)> {
|
/// once ([`crate::egress_diag`]).
|
||||||
|
fn machine_links(dir: &Path) -> Machines {
|
||||||
|
let mut machines = Machines::default();
|
||||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||||
return Vec::new();
|
return machines;
|
||||||
};
|
};
|
||||||
let mut machines = Vec::new();
|
|
||||||
for entry in entries.flatten() {
|
for entry in entries.flatten() {
|
||||||
let Some(name) = entry.file_name().to_str().map(str::to_string) else {
|
let Some(name) = entry.file_name().to_str().map(str::to_string) else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
let Ok(state) = std::fs::read_to_string(entry.path()) else {
|
let mut skip = |leader: Option<u32>, why: String| {
|
||||||
continue;
|
machines.skipped.push(Skip {
|
||||||
|
machine: name.clone(),
|
||||||
|
leader,
|
||||||
|
why,
|
||||||
|
});
|
||||||
};
|
};
|
||||||
let Some(leader) = container_leader(&state) else {
|
let state = match std::fs::read_to_string(entry.path()) {
|
||||||
|
Ok(state) => state,
|
||||||
|
Err(e) => {
|
||||||
|
skip(None, format!("its state file cannot be read: {e}"));
|
||||||
continue;
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let leader = match container_leader(&state) {
|
||||||
|
Ok(leader) => leader,
|
||||||
|
Err(why) => {
|
||||||
|
skip(state_leader(&state), why.into());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
};
|
};
|
||||||
let links = output_of(
|
let links = output_of(
|
||||||
"nsenter",
|
"nsenter",
|
||||||
|
|
@ -846,8 +890,8 @@ fn machine_links(dir: &Path) -> Vec<(String, Vec<NsLink>)> {
|
||||||
)
|
)
|
||||||
.and_then(|json| parse_links(&json));
|
.and_then(|json| parse_links(&json));
|
||||||
match links {
|
match links {
|
||||||
Ok(links) => machines.push((name, links)),
|
Ok(links) => machines.links.push((name.clone(), links)),
|
||||||
Err(e) => eprintln!("{PIN}: machine {name}: {e:#}"),
|
Err(e) => skip(Some(leader), format!("its namespace cannot be read: {e:#}")),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
machines
|
machines
|
||||||
|
|
@ -868,6 +912,10 @@ pub(crate) fn ports_to_resolve<'a>(ports: &'a [HostPort], earlier: &[Pin]) -> Ve
|
||||||
/// What a pass found out.
|
/// What a pass found out.
|
||||||
struct Survey {
|
struct Survey {
|
||||||
earlier: Vec<Pin>,
|
earlier: Vec<Pin>,
|
||||||
|
/// The bridge's ports.
|
||||||
|
ports: Vec<HostPort>,
|
||||||
|
/// The machines, read only when a port has no binding.
|
||||||
|
machines: Machines,
|
||||||
resolution: Resolution,
|
resolution: Resolution,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -881,13 +929,15 @@ fn survey(bridge: &str) -> Result<Survey> {
|
||||||
let ports = parse_ports(&output_of("ip", &["-j", "link", "show", "master", bridge])?)?;
|
let ports = parse_ports(&output_of("ip", &["-j", "link", "show", "master", bridge])?)?;
|
||||||
let earlier = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default());
|
let earlier = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default());
|
||||||
let machines = if ports_to_resolve(&ports, &earlier).is_empty() {
|
let machines = if ports_to_resolve(&ports, &earlier).is_empty() {
|
||||||
Vec::new()
|
Machines::default()
|
||||||
} else {
|
} else {
|
||||||
machine_links(Path::new(MACHINES_DIR))
|
machine_links(Path::new(MACHINES_DIR))
|
||||||
};
|
};
|
||||||
let resolution = resolve(&ports, &machines, &earlier);
|
let resolution = resolve(&ports, &machines.links, &earlier);
|
||||||
Ok(Survey {
|
Ok(Survey {
|
||||||
earlier,
|
earlier,
|
||||||
|
ports,
|
||||||
|
machines,
|
||||||
resolution,
|
resolution,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
@ -970,6 +1020,28 @@ fn report(resolution: &Resolution, which: Report<'_>) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Say what a sweep leaves as it is: a port that stays unbound, a `vb-` veth
|
||||||
|
/// that is not on the bridge, a machine it could not read. A line is printed
|
||||||
|
/// once, after it has stood for a while ([`egress_diag::advance`]). Changes
|
||||||
|
/// nothing, and a failure here is a warning: the pass has already bound what it
|
||||||
|
/// could.
|
||||||
|
fn diagnose(bridge: &str, survey: &Survey) {
|
||||||
|
let unenslaved = output_of("ip", &["-j", "link", "show", "type", "veth"])
|
||||||
|
.and_then(|json| egress_diag::parse_unenslaved(&json, bridge))
|
||||||
|
.unwrap_or_else(|e| {
|
||||||
|
eprintln!("{PIN}: list the veths: {e:#}");
|
||||||
|
Vec::new()
|
||||||
|
});
|
||||||
|
let findings = egress_diag::findings(&egress_diag::Pass {
|
||||||
|
bridge,
|
||||||
|
ports: &survey.ports,
|
||||||
|
unbound: &survey.resolution.unbound,
|
||||||
|
machines: &survey.machines,
|
||||||
|
unenslaved: &unenslaved,
|
||||||
|
});
|
||||||
|
egress_diag::say(&findings, &format!("{}.diag", state_path()), now_secs());
|
||||||
|
}
|
||||||
|
|
||||||
/// `egress-pin [<port>]`: one pass, or with `<port>` passes until that port is
|
/// `egress-pin [<port>]`: one pass, or with `<port>` passes until that port is
|
||||||
/// bound or gone. Without `<port>` it prints only what it binds: the timer
|
/// bound or gone. Without `<port>` it prints only what it binds: the timer
|
||||||
/// runs it again and again for the ports still unbound.
|
/// runs it again and again for the ports still unbound.
|
||||||
|
|
@ -991,6 +1063,9 @@ fn pin(args: &[String]) -> Result<()> {
|
||||||
connection_list().as_deref(),
|
connection_list().as_deref(),
|
||||||
))?;
|
))?;
|
||||||
commit(&survey)?;
|
commit(&survey)?;
|
||||||
|
if wait_for.is_none() {
|
||||||
|
diagnose(&bridge, &survey);
|
||||||
|
}
|
||||||
survey
|
survey
|
||||||
};
|
};
|
||||||
let Some(port) = wait_for else {
|
let Some(port) = wait_for else {
|
||||||
|
|
@ -1002,6 +1077,9 @@ fn pin(args: &[String]) -> Result<()> {
|
||||||
if bound || !present || Instant::now() >= deadline {
|
if bound || !present || Instant::now() >= deadline {
|
||||||
report(&survey.resolution, Report::Port(port));
|
report(&survey.resolution, Report::Port(port));
|
||||||
if present && !bound {
|
if present && !bound {
|
||||||
|
for skip in &survey.machines.skipped {
|
||||||
|
eprintln!("{PIN}: {}", egress_diag::skip_line(skip));
|
||||||
|
}
|
||||||
bail!("port {port} stays unbound");
|
bail!("port {port} stays unbound");
|
||||||
}
|
}
|
||||||
return Ok(());
|
return Ok(());
|
||||||
|
|
@ -1837,12 +1915,59 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn only_a_container_with_a_leader_is_read() {
|
fn only_a_container_with_a_leader_is_read() {
|
||||||
|
assert_eq!(container_leader("CLASS=container\nLEADER=1234\n"), Ok(1234));
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
container_leader("CLASS=container\nLEADER=1234\n"),
|
container_leader("CLASS=vm\nLEADER=1234\n"),
|
||||||
Some(1234)
|
Err("it is not a container")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
container_leader("CLASS=container\n"),
|
||||||
|
Err("its state file has no usable LEADER")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
container_leader("CLASS=container\nLEADER=x\n"),
|
||||||
|
Err("its state file has no usable LEADER")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_machine_that_cannot_be_read_is_listed_with_the_reason() {
|
||||||
|
let dir = std::env::temp_dir().join(format!("hive-priv-machines-{}", std::process::id()));
|
||||||
|
let _ = std::fs::remove_dir_all(&dir);
|
||||||
|
std::fs::create_dir_all(dir.join("unreadable")).unwrap();
|
||||||
|
std::fs::write(dir.join("vm"), "CLASS=vm\nLEADER=7\n").unwrap();
|
||||||
|
std::fs::write(dir.join("no-leader"), "CLASS=container\n").unwrap();
|
||||||
|
let machines = machine_links(&dir);
|
||||||
|
let mut skipped = machines.skipped.clone();
|
||||||
|
skipped.sort_by(|a, b| a.machine.cmp(&b.machine));
|
||||||
|
let _ = std::fs::remove_dir_all(&dir);
|
||||||
|
assert!(machines.links.is_empty());
|
||||||
|
assert_eq!(skipped.len(), 3);
|
||||||
|
assert_eq!(
|
||||||
|
skipped[0],
|
||||||
|
Skip {
|
||||||
|
machine: "no-leader".into(),
|
||||||
|
leader: None,
|
||||||
|
why: "its state file has no usable LEADER".into(),
|
||||||
|
}
|
||||||
|
);
|
||||||
|
assert_eq!(skipped[1].machine, "unreadable");
|
||||||
|
assert_eq!(skipped[1].leader, None);
|
||||||
|
assert!(
|
||||||
|
skipped[1]
|
||||||
|
.why
|
||||||
|
.starts_with("its state file cannot be read: "),
|
||||||
|
"{}",
|
||||||
|
skipped[1].why
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
skipped[2],
|
||||||
|
Skip {
|
||||||
|
machine: "vm".into(),
|
||||||
|
leader: Some(7),
|
||||||
|
why: "it is not a container".into(),
|
||||||
|
}
|
||||||
);
|
);
|
||||||
assert_eq!(container_leader("CLASS=vm\nLEADER=1234\n"), None);
|
|
||||||
assert_eq!(container_leader("CLASS=container\n"), None);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn lock_path(name: &str) -> String {
|
fn lock_path(name: &str) -> String {
|
||||||
|
|
|
||||||
484
hive-priv/src/egress_diag.rs
Normal file
484
hive-priv/src/egress_diag.rs
Normal file
|
|
@ -0,0 +1,484 @@
|
||||||
|
//! What a bridge sweep says about the ports it leaves as they are.
|
||||||
|
//!
|
||||||
|
//! `egress-pin` prints a line for each port it binds and none for a port it
|
||||||
|
//! can't, so a container that stayed unbound for eleven hours left nothing in
|
||||||
|
//! the journal. A sweep (`egress-pin` with no port, the timer's run) now says
|
||||||
|
//! what it finds and doesn't fix:
|
||||||
|
//!
|
||||||
|
//! - a port that stays unbound, with the links it was matched against;
|
||||||
|
//! - a `vb-` veth that is not on the bridge, which no pass can see;
|
||||||
|
//! - a registered machine the pass could not read, with the reason.
|
||||||
|
//!
|
||||||
|
//! A port is unbound for a moment at every container start, so a finding is
|
||||||
|
//! said only after it has stood for [`GRACE_SECS`], and once: the pass keeps
|
||||||
|
//! `<state file>.diag`, one `<first seen> <said 0|1> <key>` line per finding, and
|
||||||
|
//! drops a finding's line when it clears, so a return is said again. An older
|
||||||
|
//! binary never reads the file. Nothing here changes what is bound.
|
||||||
|
|
||||||
|
use std::fmt::Write as _;
|
||||||
|
|
||||||
|
use anyhow::{Context as _, Result, bail};
|
||||||
|
|
||||||
|
use crate::egress::{HostPort, Machines, NsLink, PIN, Skip, is_plain_name, owner_of};
|
||||||
|
|
||||||
|
/// Seconds a finding stands before a pass says it: two ticks of the timer, and
|
||||||
|
/// longer than the 30 s a port's own `egress-pin <port>` run waits for its
|
||||||
|
/// container to register.
|
||||||
|
pub(crate) const GRACE_SECS: u64 = 60;
|
||||||
|
|
||||||
|
/// The prefix of the host end of a container's veth that nspawn makes when the
|
||||||
|
/// container joins a bridge.
|
||||||
|
const VETH_PREFIX: &str = "vb-";
|
||||||
|
|
||||||
|
/// A `vb-` veth that is not on the bridge.
|
||||||
|
#[derive(Debug, PartialEq, Eq)]
|
||||||
|
pub(crate) struct Unenslaved {
|
||||||
|
name: String,
|
||||||
|
/// The bridge it is on instead, if any.
|
||||||
|
master: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The `vb-` veths of an `ip -j link show type veth` listing that are not on
|
||||||
|
/// `bridge`.
|
||||||
|
pub(crate) fn parse_unenslaved(json: &str, bridge: &str) -> Result<Vec<Unenslaved>> {
|
||||||
|
let value: serde_json::Value = serde_json::from_str(json).context("parse ip -j link")?;
|
||||||
|
let Some(items) = value.as_array() else {
|
||||||
|
bail!("ip -j link did not print a list");
|
||||||
|
};
|
||||||
|
Ok(items
|
||||||
|
.iter()
|
||||||
|
.filter_map(|item| {
|
||||||
|
let name = item.get("ifname")?.as_str()?;
|
||||||
|
let master = item.get("master").and_then(serde_json::Value::as_str);
|
||||||
|
(name.starts_with(VETH_PREFIX) && is_plain_name(name) && master != Some(bridge)).then(
|
||||||
|
|| Unenslaved {
|
||||||
|
name: name.to_string(),
|
||||||
|
master: master.map(str::to_string),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What a sweep looked at.
|
||||||
|
pub(crate) struct Pass<'a> {
|
||||||
|
pub(crate) bridge: &'a str,
|
||||||
|
pub(crate) ports: &'a [HostPort],
|
||||||
|
/// Port name and why it stays unbound.
|
||||||
|
pub(crate) unbound: &'a [(String, String)],
|
||||||
|
pub(crate) machines: &'a Machines,
|
||||||
|
pub(crate) unenslaved: &'a [Unenslaved],
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One thing a sweep can say.
|
||||||
|
#[derive(Debug, PartialEq, Eq)]
|
||||||
|
pub(crate) struct Finding {
|
||||||
|
/// What it is about: the same finding has the same key in every pass.
|
||||||
|
key: String,
|
||||||
|
line: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The line about a machine a pass could not read.
|
||||||
|
pub(crate) fn skip_line(skip: &Skip) -> String {
|
||||||
|
format!("machine {} is skipped: {}", skip.machine, skip.why)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Each machine with the `<index>/<peer index>` of its veth links, by name.
|
||||||
|
fn machines_seen(machines: &[(String, Vec<NsLink>)]) -> String {
|
||||||
|
let mut seen: Vec<String> = machines
|
||||||
|
.iter()
|
||||||
|
.map(|(name, links)| {
|
||||||
|
let pairs: Vec<String> = links
|
||||||
|
.iter()
|
||||||
|
.filter_map(|l| Some(format!("{}/{}", l.ifindex, l.peer_index?)))
|
||||||
|
.collect();
|
||||||
|
let pairs = if pairs.is_empty() {
|
||||||
|
"-".to_string()
|
||||||
|
} else {
|
||||||
|
pairs.join("+")
|
||||||
|
};
|
||||||
|
format!("{name}={pairs}")
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
seen.sort();
|
||||||
|
if seen.is_empty() {
|
||||||
|
"none".into()
|
||||||
|
} else {
|
||||||
|
seen.join(", ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Everything a sweep has to say about what it left as it is.
|
||||||
|
pub(crate) fn findings(pass: &Pass<'_>) -> Vec<Finding> {
|
||||||
|
let mut found = Vec::new();
|
||||||
|
for (name, why) in pass.unbound {
|
||||||
|
let Some(port) = pass.ports.iter().find(|p| &p.name == name) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let head = format!(
|
||||||
|
"{name} (ifindex {}, peer {})",
|
||||||
|
port.ifindex, port.peer_index
|
||||||
|
);
|
||||||
|
let line = if owner_of(port, &pass.machines.links).is_none() {
|
||||||
|
format!(
|
||||||
|
"{head} has no owner; machines seen: {}",
|
||||||
|
machines_seen(&pass.machines.links)
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
format!("{head} stays unbound: {why}")
|
||||||
|
};
|
||||||
|
found.push(Finding {
|
||||||
|
key: format!("port {name} {}", port.ifindex),
|
||||||
|
line,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for skip in &pass.machines.skipped {
|
||||||
|
let leader = skip.leader.map_or("-".to_string(), |pid| pid.to_string());
|
||||||
|
found.push(Finding {
|
||||||
|
key: format!("machine {} {leader}", skip.machine),
|
||||||
|
line: skip_line(skip),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for veth in pass.unenslaved {
|
||||||
|
let line = match &veth.master {
|
||||||
|
Some(master) => format!("{} is on {master}, not {}", veth.name, pass.bridge),
|
||||||
|
None => format!("{} is not on {}", veth.name, pass.bridge),
|
||||||
|
};
|
||||||
|
found.push(Finding {
|
||||||
|
key: format!("veth {}", veth.name),
|
||||||
|
line,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
found
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A finding a pass has seen: when it first did and whether it said so.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub(crate) struct Seen {
|
||||||
|
first: u64,
|
||||||
|
said: bool,
|
||||||
|
key: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The findings of earlier passes. A line that does not parse is dropped.
|
||||||
|
pub(crate) fn parse_seen(text: &str) -> Vec<Seen> {
|
||||||
|
text.lines()
|
||||||
|
.filter_map(|line| {
|
||||||
|
let mut cols = line.splitn(3, ' ');
|
||||||
|
let first = cols.next()?.parse().ok()?;
|
||||||
|
let said = match cols.next()? {
|
||||||
|
"1" => true,
|
||||||
|
"0" => false,
|
||||||
|
_ => return None,
|
||||||
|
};
|
||||||
|
Some(Seen {
|
||||||
|
first,
|
||||||
|
said,
|
||||||
|
key: cols.next()?.to_string(),
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn seen_text(seen: &[Seen]) -> String {
|
||||||
|
let mut text = String::new();
|
||||||
|
for s in seen {
|
||||||
|
let _ = writeln!(text, "{} {} {}", s.first, u8::from(s.said), s.key);
|
||||||
|
}
|
||||||
|
text
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What this pass keeps for the next one, and the findings it says now: the ones
|
||||||
|
/// that have stood for [`GRACE_SECS`] and were not said before. A finding that
|
||||||
|
/// is gone is not kept, so it is said again when it returns.
|
||||||
|
pub(crate) fn advance<'a>(
|
||||||
|
findings: &'a [Finding],
|
||||||
|
earlier: &[Seen],
|
||||||
|
now: u64,
|
||||||
|
) -> (Vec<Seen>, Vec<&'a Finding>) {
|
||||||
|
let mut kept = Vec::new();
|
||||||
|
let mut due = Vec::new();
|
||||||
|
for finding in findings {
|
||||||
|
let before = earlier.iter().find(|s| s.key == finding.key);
|
||||||
|
let first = before.map_or(now, |s| s.first);
|
||||||
|
let said = before.is_some_and(|s| s.said);
|
||||||
|
let say = !said && now.saturating_sub(first) >= GRACE_SECS;
|
||||||
|
if say {
|
||||||
|
due.push(finding);
|
||||||
|
}
|
||||||
|
kept.push(Seen {
|
||||||
|
first,
|
||||||
|
said: said || say,
|
||||||
|
key: finding.key.clone(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
(kept, due)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Print what `findings` has that is due and keep the rest in `path`. A file
|
||||||
|
/// that can't be written is a warning: without it a finding would never stand
|
||||||
|
/// long enough to be said.
|
||||||
|
pub(crate) fn say(findings: &[Finding], path: &str, now: u64) {
|
||||||
|
let before = std::fs::read_to_string(path).unwrap_or_default();
|
||||||
|
let (kept, due) = advance(findings, &parse_seen(&before), now);
|
||||||
|
let text = seen_text(&kept);
|
||||||
|
if text != before
|
||||||
|
&& let Err(e) = std::fs::write(path, &text)
|
||||||
|
{
|
||||||
|
eprintln!("{PIN}: write {path}: {e}");
|
||||||
|
}
|
||||||
|
for finding in due {
|
||||||
|
eprintln!("{PIN}: {}", finding.line);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::egress::Mac;
|
||||||
|
|
||||||
|
fn port(name: &str, ifindex: u64, peer_index: u64) -> HostPort {
|
||||||
|
HostPort {
|
||||||
|
name: name.into(),
|
||||||
|
ifindex,
|
||||||
|
peer_index,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn link(ifindex: u64, peer_index: Option<u64>) -> NsLink {
|
||||||
|
NsLink {
|
||||||
|
ifindex,
|
||||||
|
peer_index,
|
||||||
|
mac: Mac::parse("02:00:00:00:00:01").unwrap(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn unbound(name: &str, why: &str) -> (String, String) {
|
||||||
|
(name.into(), why.into())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn lines(found: &[Finding]) -> Vec<&str> {
|
||||||
|
found.iter().map(|f| f.line.as_str()).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_port_no_machine_owns_names_every_machine_and_its_links() {
|
||||||
|
let machines = Machines {
|
||||||
|
links: vec![
|
||||||
|
("h-b".into(), vec![link(1, None), link(2, Some(41))]),
|
||||||
|
("h-a".into(), vec![link(1, None)]),
|
||||||
|
("h-c".into(), vec![link(2, Some(7)), link(3, Some(8))]),
|
||||||
|
],
|
||||||
|
skipped: Vec::new(),
|
||||||
|
};
|
||||||
|
let ports = [port("vb-h-x", 40, 3)];
|
||||||
|
let unbound = [unbound("vb-h-x", "no registered container owns it")];
|
||||||
|
let found = findings(&Pass {
|
||||||
|
bridge: "hive-br0",
|
||||||
|
ports: &ports,
|
||||||
|
unbound: &unbound,
|
||||||
|
machines: &machines,
|
||||||
|
unenslaved: &[],
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
lines(&found),
|
||||||
|
["vb-h-x (ifindex 40, peer 3) has no owner; machines seen: \
|
||||||
|
h-a=-, h-b=2/41, h-c=2/7+3/8"]
|
||||||
|
);
|
||||||
|
assert_eq!(found[0].key, "port vb-h-x 40");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn with_no_machine_read_the_line_says_none() {
|
||||||
|
let ports = [port("vb-h-x", 40, 3)];
|
||||||
|
let unbound = [unbound("vb-h-x", "no registered container owns it")];
|
||||||
|
let found = findings(&Pass {
|
||||||
|
bridge: "hive-br0",
|
||||||
|
ports: &ports,
|
||||||
|
unbound: &unbound,
|
||||||
|
machines: &Machines::default(),
|
||||||
|
unenslaved: &[],
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
lines(&found),
|
||||||
|
["vb-h-x (ifindex 40, peer 3) has no owner; machines seen: none"]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_port_with_an_owner_gives_the_reason_it_stays_unbound() {
|
||||||
|
let machines = Machines {
|
||||||
|
links: vec![("h-a".into(), vec![link(3, Some(40))])],
|
||||||
|
skipped: Vec::new(),
|
||||||
|
};
|
||||||
|
let ports = [port("vb-h-a", 40, 3)];
|
||||||
|
let unbound = [unbound(
|
||||||
|
"vb-h-a",
|
||||||
|
"h-a has the unusable address 00:00:00:00:00:00",
|
||||||
|
)];
|
||||||
|
let found = findings(&Pass {
|
||||||
|
bridge: "hive-br0",
|
||||||
|
ports: &ports,
|
||||||
|
unbound: &unbound,
|
||||||
|
machines: &machines,
|
||||||
|
unenslaved: &[],
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
lines(&found),
|
||||||
|
["vb-h-a (ifindex 40, peer 3) stays unbound: \
|
||||||
|
h-a has the unusable address 00:00:00:00:00:00"]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_skipped_machine_names_the_reason() {
|
||||||
|
let machines = Machines {
|
||||||
|
links: Vec::new(),
|
||||||
|
skipped: vec![
|
||||||
|
Skip {
|
||||||
|
machine: "h-a".into(),
|
||||||
|
leader: Some(77),
|
||||||
|
why: "it is not a container".into(),
|
||||||
|
},
|
||||||
|
Skip {
|
||||||
|
machine: "h-b".into(),
|
||||||
|
leader: None,
|
||||||
|
why: "its state file cannot be read: gone".into(),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
let found = findings(&Pass {
|
||||||
|
bridge: "hive-br0",
|
||||||
|
ports: &[],
|
||||||
|
unbound: &[],
|
||||||
|
machines: &machines,
|
||||||
|
unenslaved: &[],
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
lines(&found),
|
||||||
|
[
|
||||||
|
"machine h-a is skipped: it is not a container",
|
||||||
|
"machine h-b is skipped: its state file cannot be read: gone"
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(found[0].key, "machine h-a 77");
|
||||||
|
assert_eq!(found[1].key, "machine h-b -");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_veth_off_the_bridge_is_named_with_the_bridge_it_is_on() {
|
||||||
|
let off = parse_unenslaved(
|
||||||
|
r#"[
|
||||||
|
{"ifname":"vb-a","master":"hive-br0"},
|
||||||
|
{"ifname":"vb-b"},
|
||||||
|
{"ifname":"vb-c","master":"virbr0"},
|
||||||
|
{"ifname":"ve-d"},
|
||||||
|
{"ifname":"vb-e\"x"}
|
||||||
|
]"#,
|
||||||
|
"hive-br0",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let found = findings(&Pass {
|
||||||
|
bridge: "hive-br0",
|
||||||
|
ports: &[],
|
||||||
|
unbound: &[],
|
||||||
|
machines: &Machines::default(),
|
||||||
|
unenslaved: &off,
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
lines(&found),
|
||||||
|
["vb-b is not on hive-br0", "vb-c is on virbr0, not hive-br0"]
|
||||||
|
);
|
||||||
|
assert_eq!(found[0].key, "veth vb-b");
|
||||||
|
assert!(parse_unenslaved("{}", "hive-br0").is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finding(key: &str) -> Finding {
|
||||||
|
Finding {
|
||||||
|
key: key.into(),
|
||||||
|
line: format!("line of {key}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_finding_is_said_once_after_the_grace_and_not_before() {
|
||||||
|
let found = [finding("a")];
|
||||||
|
let (kept, due) = advance(&found, &[], 1000);
|
||||||
|
assert!(due.is_empty());
|
||||||
|
let (kept, due) = advance(&found, &kept, 1000 + GRACE_SECS - 1);
|
||||||
|
assert!(due.is_empty());
|
||||||
|
let (kept, due) = advance(&found, &kept, 1000 + GRACE_SECS);
|
||||||
|
assert_eq!(due, [&found[0]]);
|
||||||
|
let (kept, due) = advance(&found, &kept, 1000 + 10 * GRACE_SECS);
|
||||||
|
assert!(due.is_empty());
|
||||||
|
let (_, due) = advance(&found, &kept, 1000 + 20 * GRACE_SECS);
|
||||||
|
assert!(due.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_finding_that_cleared_is_said_again_when_it_returns() {
|
||||||
|
let found = [finding("a")];
|
||||||
|
let (kept, _) = advance(&found, &[], 0);
|
||||||
|
let (kept, due) = advance(&found, &kept, GRACE_SECS);
|
||||||
|
assert_eq!(due.len(), 1);
|
||||||
|
let (kept, due) = advance(&[], &kept, 2 * GRACE_SECS);
|
||||||
|
assert!(kept.is_empty() && due.is_empty());
|
||||||
|
let (kept, due) = advance(&found, &kept, 3 * GRACE_SECS);
|
||||||
|
assert!(due.is_empty());
|
||||||
|
let (_, due) = advance(&found, &kept, 4 * GRACE_SECS);
|
||||||
|
assert_eq!(due.len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn each_finding_has_its_own_clock() {
|
||||||
|
let both = [finding("a"), finding("b")];
|
||||||
|
let (kept, _) = advance(&both[..1], &[], 0);
|
||||||
|
let (kept, due) = advance(&both, &kept, GRACE_SECS);
|
||||||
|
assert_eq!(due, [&both[0]]);
|
||||||
|
let (_, due) = advance(&both, &kept, 2 * GRACE_SECS);
|
||||||
|
assert_eq!(due, [&both[1]]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_clock_that_steps_back_waits() {
|
||||||
|
let found = [finding("a")];
|
||||||
|
let (kept, _) = advance(&found, &[], 5000);
|
||||||
|
let (_, due) = advance(&found, &kept, 100);
|
||||||
|
assert!(due.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_seen_file_round_trips_and_drops_bad_lines() {
|
||||||
|
let seen = vec![
|
||||||
|
Seen {
|
||||||
|
first: 12,
|
||||||
|
said: true,
|
||||||
|
key: "port vb-h-x 40".into(),
|
||||||
|
},
|
||||||
|
Seen {
|
||||||
|
first: 30,
|
||||||
|
said: false,
|
||||||
|
key: "veth vb-b".into(),
|
||||||
|
},
|
||||||
|
];
|
||||||
|
let text = seen_text(&seen);
|
||||||
|
assert_eq!(text, "12 1 port vb-h-x 40\n30 0 veth vb-b\n");
|
||||||
|
assert_eq!(parse_seen(&text), seen);
|
||||||
|
assert!(parse_seen("x 1 a\n5 2 a\n5 1\n\n").is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn say_keeps_what_it_has_not_said_in_the_file() {
|
||||||
|
let path = std::env::temp_dir().join(format!("hive-priv-diag-{}", std::process::id()));
|
||||||
|
let path = path.to_str().unwrap().to_string();
|
||||||
|
let _ = std::fs::remove_file(&path);
|
||||||
|
let found = [finding("a")];
|
||||||
|
say(&found, &path, 100);
|
||||||
|
assert_eq!(std::fs::read_to_string(&path).unwrap(), "100 0 a\n");
|
||||||
|
say(&found, &path, 100 + GRACE_SECS);
|
||||||
|
assert_eq!(std::fs::read_to_string(&path).unwrap(), "100 1 a\n");
|
||||||
|
say(&[], &path, 100 + 2 * GRACE_SECS);
|
||||||
|
assert_eq!(std::fs::read_to_string(&path).unwrap(), "");
|
||||||
|
let _ = std::fs::remove_file(&path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -23,7 +23,7 @@
|
||||||
//! turns ephemeral, with [`salvage_io`] for its space, reflink and lock calls),
|
//! turns ephemeral, with [`salvage_io`] for its space, reflink and lock calls),
|
||||||
//! [`prestart`] (the check of a container's root that runs before each start,
|
//! [`prestart`] (the check of a container's root that runs before each start,
|
||||||
//! as a sub-command), [`egress`] (the bridge anti-spoofing bindings, also
|
//! as a sub-command), [`egress`] (the bridge anti-spoofing bindings, also
|
||||||
//! sub-commands), [`egress_ech`] (the `ClientHello` check in front of squid's
|
//! sub-commands), [`egress_diag`] (sweep findings), [`egress_ech`] (the `ClientHello` check in front of squid's
|
||||||
//! HTTPS intercept port), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
|
//! HTTPS intercept port), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
|
||||||
//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles),
|
//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles),
|
||||||
//! [`infra`] (gateway nginx, infra containers) and [`closure`] (the swarm-signature
|
//! [`infra`] (gateway nginx, infra containers) and [`closure`] (the swarm-signature
|
||||||
|
|
@ -33,6 +33,7 @@ mod closure;
|
||||||
mod container;
|
mod container;
|
||||||
mod container_config;
|
mod container_config;
|
||||||
mod egress;
|
mod egress;
|
||||||
|
mod egress_diag;
|
||||||
mod egress_ech;
|
mod egress_ech;
|
||||||
mod files;
|
mod files;
|
||||||
mod infra;
|
mod infra;
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue