vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(network): wire default route + bridge DNS for isolated containers

Browse source 
When isolateContainers=true, claude (and all egress) broke in every
container: agents came up with an IP but no way off the bridge subnet.

Two container-side gaps, both confirmed against nixpkgs
nixos-containers.nix:

1. No default route. hive-priv wrote HOST_ADDRESS= empty in the nspawn
   conf. nixos-container's container-side setup only installs
   `ip route add default via $HOST_ADDRESS` when HOST_ADDRESS is
   non-empty, so the container had an address but no gateway -> nothing
   off-subnet (incl. api.anthropic.com) was reachable. Fix: write
   HOST_ADDRESS=<bridge-ip>. In bridge mode the host-side address/route
   setup is skipped, so this only affects the container's default route.

2. No usable resolver. nixos-container copies the host's /etc/resolv.conf
   into the container at every start; the host resolver (e.g. 127.0.0.53)
   is unreachable from a private netns and isn't authoritative for the
   hive's own zones. Fix: hive-priv drops a marker carrying the gateway
   IP only when isolated, and a new harness-base oneshot
   (hyperhive-isolated-dns) rewrites resolv.conf to point at the bridge
   dnsmasq. Inert in shared-netns mode (no marker), so the shared
   container toplevel does the right thing in both modes.

The gateway IP is the address part of HIVE_NETWORK_SUBNET (the bridge IP
verbatim, honouring a non-.1 operator override), via a new validated
bridge_gateway_ip() helper with unit tests.

Unblocks defaulting isolation on.
This commit is contained in:
atlas 2026-06-10 20:52:12 +02:00 • committed by mara
commit d993ad2c47
4 changed files with 177 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

48
hive-priv/src/main.rs
View file

@ -857,7 +857,15 @@ fn write_nspawn_flags(
}
if let Some(iso) = isolation {
out.push_str("PRIVATE_NETWORK=1\n");
out.push_str("HOST_ADDRESS=\n");
// HOST_ADDRESS = the bridge gateway IP. nixos-container's
// container-side setup only installs a default route
// (`ip route add default via $HOST_ADDRESS`) when HOST_ADDRESS is
// non-empty; leaving it blank gave the container an address but no
// route off the bridge subnet (no internet, no api.anthropic.com).
// In bridge mode (HOST_BRIDGE set) the host-side address/route
// setup is skipped, so this only affects the container's route —
// exactly what we want.
let _ = writeln!(out, "HOST_ADDRESS={}", iso.gateway_ip);
let _ = writeln!(out, "LOCAL_ADDRESS={}", iso.agent_ip);
out.push_str("HOST_ADDRESS6=\n");
out.push_str("LOCAL_ADDRESS6=\n");
@ -879,5 +887,41 @@ fn write_nspawn_flags(
.collect();
let flags_joined = flags.join(" ");
let _ = writeln!(out, "EXTRA_NSPAWN_FLAGS=\"{flags_joined}\"");
std::fs::write(&path, out).with_context(|| format!("write {path}"))
std::fs::write(&path, out).with_context(|| format!("write {path}"))?;
// DNS marker for the in-container resolver oneshot. nixos-container
// copies the *host's* /etc/resolv.conf into the container at every
// start (its host resolver — e.g. 127.0.0.53 — is unreachable from a
// private netns, and isn't authoritative for the hive's own zones
// anyway). The `hyperhive-isolated-dns` oneshot in harness-base.nix
// rewrites resolv.conf to point at the bridge resolver, but only when
// this marker exists; it carries the gateway IP so the container
// doesn't have to re-derive it. Written on isolate, removed otherwise,
// so the same shared container toplevel behaves correctly in both modes.
write_bridge_dns_marker(container, isolation)?;
Ok(())
}
/// Path to the in-container DNS marker (the container's own `/etc`).
fn bridge_dns_marker_path(container: &str) -> String {
format!("/var/lib/nixos-containers/{container}/etc/hyperhive-bridge-dns")
}
/// Write (isolated) or remove (host-netns) the bridge-DNS marker the
/// `hyperhive-isolated-dns` oneshot keys off. The marker file contains
/// just the gateway IP. Best-effort on removal (absence is the goal).
fn write_bridge_dns_marker(container: &str, isolation: Option<&NetworkIsolation>) -> Result<()> {
let path = bridge_dns_marker_path(container);
match isolation {
Some(iso) => {
std::fs::write(&path, format!("{}\n", iso.gateway_ip))
.with_context(|| format!("write bridge-DNS marker {path}"))?;
}
None => match std::fs::remove_file(&path) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(e).with_context(|| format!("remove bridge-DNS marker {path}")),
},
}
Ok(())
}