network: add isolateContainers option for #14 netns isolation
Adds `services.hyperhive.network.isolateContainers` (bool, default false). When enabled alongside `network.enable`, activates: - IP forwarding + NAT masquerade so isolated agents reach the internet - nftables DROP rule blocking bridge-subnet → loopback (defence-in-depth against compromised agent reaching the c0re dashboard) - `HIVE_NETWORK_ISOLATION`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` injected into the hive-c0re service env; the Rust lifecycle reads these to set `PRIVATE_NETWORK`, `LOCAL_ADDRESS`, and `HOST_BRIDGE` in each agent container's conf Config block rewritten as `lib.mkMerge [...]` — the prior `lib.mkIf // lib.mkIf` pattern was invalid nix (mkIf returns a tagged value, not an attrset; // on it is a type error). See docs/network.md for full design.
This commit is contained in:
atlas
mara
parent
7e5a21522a
commit
39b4c65922
1 changed files with 171 additions and 45 deletions
|
|
@ -87,52 +87,178 @@ in
|
||||||
regardless of upstream choice.
|
regardless of upstream choice.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
isolateContainers = lib.mkOption {
|
||||||
assertions = [
|
type = lib.types.bool;
|
||||||
{
|
default = false;
|
||||||
assertion = config.services.hyperhive.domain != null;
|
example = true;
|
||||||
message = ''
|
description = ''
|
||||||
services.hyperhive.network.enable = true requires
|
Flip agent containers from shared host netns to private netns
|
||||||
services.hyperhive.domain to be set — the resolver needs a
|
(#14). When true, each agent container gets a dedicated veth
|
||||||
domain to be authoritative for. Either pin a hostname
|
pair attached to `bridgeName` and a deterministic IP from
|
||||||
(`services.hyperhive.domain = "example.com";`) or leave
|
the bridge subnet. The bridge (already up when `enable = true`)
|
||||||
`network.enable` at its default of false.
|
becomes the sole routed path between the host and agent
|
||||||
'';
|
containers.
|
||||||
}
|
|
||||||
{
|
|
||||||
assertion = config.services.hyperhive.gateway.enable;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.network.enable = true requires
|
|
||||||
services.hyperhive.gateway.enable = true — the dnsmasq
|
|
||||||
resolver runs inside the hive-gateway container (single
|
|
||||||
front-door for both DNS and HTTP). Enable the gateway or
|
|
||||||
leave `network.enable` at its default of false.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Bridge interface on the host. Empty interfaces list = purely
|
The host-side nix effect (this option) is:
|
||||||
# virtual bridge (no slave NICs attached). Per-agent veth pairs
|
- Sets `HIVE_NETWORK_ISOLATION=1` in the c0re service env so
|
||||||
# will join this bridge once #14 lands; at v1 it stands alone.
|
the Rust lifecycle knows to pass `--private-network` +
|
||||||
networking.bridges.${cfg.bridgeName}.interfaces = [ ];
|
bridge settings when creating/updating containers.
|
||||||
|
- Enables IP forwarding + NAT so agents can reach the internet
|
||||||
|
through the host.
|
||||||
|
- Adds a firewall rule DROP'ing traffic from the bridge subnet
|
||||||
|
to the host's loopback addresses — defence-in-depth so a
|
||||||
|
compromised agent can't reach the c0re dashboard (already
|
||||||
|
bound to 127.0.0.1) or other host-loopback services even if
|
||||||
|
the routing table somehow leaks.
|
||||||
|
- Allows HTTP/HTTPS (80/443) traffic from the bridge subnet to
|
||||||
|
the host so agents can reach the gateway container (shared
|
||||||
|
host netns, proxies the operator's per-agent UI).
|
||||||
|
|
||||||
# Host-side IP assignment on the bridge. This is what dnsmasq
|
**Prerequisite**: all agents must have
|
||||||
# (inside the gateway container, shared host netns) binds on.
|
`hyperhive.web.useUnixSocket = true` before enabling isolation.
|
||||||
networking.interfaces.${cfg.bridgeName}.ipv4.addresses = [
|
Agents that still bind TCP on `0.0.0.0:<port>` will be
|
||||||
{
|
reachable at their bridge IP from other agents on the same
|
||||||
address = cfg.bridgeIp;
|
subnet — defeating the isolation goal. The gateway routes via
|
||||||
prefixLength = cfg.bridgePrefixLength;
|
unix sockets so gateway reach still works regardless.
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Open the resolver port in the host firewall for traffic from
|
**Migration**: containers are destroyed and re-created when
|
||||||
# the bridge subnet only. Other interfaces stay closed —
|
the network isolation flag flips. Operator state under
|
||||||
# external DNS-amplification surface is not exposed.
|
`/agents/<name>/state/` is bind-mounted and survives; the
|
||||||
networking.firewall.interfaces.${cfg.bridgeName} = {
|
container rootfs (nix store paths) is recreated cleanly.
|
||||||
allowedUDPPorts = [ 53 ];
|
|
||||||
allowedTCPPorts = [ 53 ];
|
**Rust counterpart**: `hive-c0re` reads `HIVE_NETWORK_ISOLATION`
|
||||||
|
and `HIVE_NETWORK_BRIDGE` from its service env and uses them in
|
||||||
|
`lifecycle::set_nspawn_flags` to configure `PRIVATE_NETWORK`,
|
||||||
|
`LOCAL_ADDRESS`, and `HOST_BRIDGE` in each container's
|
||||||
|
`nixos-containers/<name>.conf`. See `docs/network.md` for the
|
||||||
|
full design.
|
||||||
|
'';
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
config = lib.mkMerge [
|
||||||
|
(lib.mkIf cfg.enable {
|
||||||
|
assertions = [
|
||||||
|
{
|
||||||
|
assertion = config.services.hyperhive.domain != null;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.network.enable = true requires
|
||||||
|
services.hyperhive.domain to be set — the resolver needs a
|
||||||
|
domain to be authoritative for. Either pin a hostname
|
||||||
|
(`services.hyperhive.domain = "example.com";`) or leave
|
||||||
|
`network.enable` at its default of false.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
{
|
||||||
|
assertion = config.services.hyperhive.gateway.enable;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.network.enable = true requires
|
||||||
|
services.hyperhive.gateway.enable = true — the dnsmasq
|
||||||
|
resolver runs inside the hive-gateway container (single
|
||||||
|
front-door for both DNS and HTTP). Enable the gateway or
|
||||||
|
leave `network.enable` at its default of false.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# Bridge interface on the host. Empty interfaces list = purely
|
||||||
|
# virtual bridge (no slave NICs attached). Per-agent veth pairs
|
||||||
|
# will join this bridge once #14 lands; at v1 it stands alone.
|
||||||
|
networking.bridges.${cfg.bridgeName}.interfaces = [ ];
|
||||||
|
|
||||||
|
# Host-side IP assignment on the bridge. This is what dnsmasq
|
||||||
|
# (inside the gateway container, shared host netns) binds on.
|
||||||
|
networking.interfaces.${cfg.bridgeName}.ipv4.addresses = [
|
||||||
|
{
|
||||||
|
address = cfg.bridgeIp;
|
||||||
|
prefixLength = cfg.bridgePrefixLength;
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# Open the resolver port in the host firewall for traffic from
|
||||||
|
# the bridge subnet only. Other interfaces stay closed —
|
||||||
|
# external DNS-amplification surface is not exposed.
|
||||||
|
networking.firewall.interfaces.${cfg.bridgeName} = {
|
||||||
|
allowedUDPPorts = [ 53 ];
|
||||||
|
allowedTCPPorts = [ 53 ];
|
||||||
|
};
|
||||||
|
})
|
||||||
|
|
||||||
|
# Container network isolation (#14 v1). Ships as a separate overlay
|
||||||
|
# on top of the base bridge config (which stays unconditional) so
|
||||||
|
# operators can stand the bridge + resolver up first, validate
|
||||||
|
# everything, then flip isolation on independently.
|
||||||
|
(lib.mkIf (cfg.enable && cfg.isolateContainers) {
|
||||||
|
assertions = [
|
||||||
|
{
|
||||||
|
# Isolation without the bridge is a no-op: agents would get
|
||||||
|
# private netns but no reachable gateway. The assertion on
|
||||||
|
# `cfg.enable` above already gates the bridge, but making the
|
||||||
|
# dependency explicit here avoids confusing "bridge up, no
|
||||||
|
# isolation" vs "isolation on, no bridge" states.
|
||||||
|
assertion = cfg.enable;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.network.isolateContainers = true requires
|
||||||
|
services.hyperhive.network.enable = true (the bridge and
|
||||||
|
resolver must be running before isolation is flipped on).
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# IP forwarding — agents need to route through the bridge to reach
|
||||||
|
# the internet. NixOS firewall's `nat.enable` sets this too, but
|
||||||
|
# making it explicit here keeps the intent visible alongside the
|
||||||
|
# NAT rule.
|
||||||
|
boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
|
||||||
|
|
||||||
|
# NAT/masquerade: translate agent bridge IPs → host's outbound
|
||||||
|
# IP for internet-bound traffic. Without masquerade, packets from
|
||||||
|
# 10.42.0.X arrive at external servers with an RFC-1918 source
|
||||||
|
# that can't be routed back.
|
||||||
|
networking.nat = {
|
||||||
|
enable = true;
|
||||||
|
# `internalInterfaces` causes `MASQUERADE` on packets from the
|
||||||
|
# bridge leaving via any external interface. Only traffic from
|
||||||
|
# agents crosses the bridge — hive-gateway / forge / matrix
|
||||||
|
# stay on host netns and don't need NAT.
|
||||||
|
internalInterfaces = [ cfg.bridgeName ];
|
||||||
|
};
|
||||||
|
|
||||||
|
# DROP traffic from the bridge subnet to host loopback addresses.
|
||||||
|
# Defence-in-depth: the c0re dashboard already binds 127.0.0.1
|
||||||
|
# (not 0.0.0.0), so bridge-sourced traffic can't reach it via
|
||||||
|
# the bridge IP. But a misconfigured service that slips to
|
||||||
|
# 0.0.0.0 would otherwise be reachable. The DROP rule closes that
|
||||||
|
# window. Use `extraInputRules` (nftables `input` chain, priority
|
||||||
|
# 0, same ruleset as `allowedTCPPorts`) so it's processed before
|
||||||
|
# the accept rules for bridge-side DNS we added above.
|
||||||
|
#
|
||||||
|
# The rule fires only when an agent container has a bridge IP
|
||||||
|
# (i.e. after the Rust side also ships `PRIVATE_NETWORK=1`);
|
||||||
|
# until then all containers share host netns and no traffic
|
||||||
|
# originates from 10.42.0.0/24 so this is a dead letter.
|
||||||
|
networking.firewall.extraInputRules = ''
|
||||||
|
ip saddr ${cfg.bridgeIp}/${toString cfg.bridgePrefixLength} ip daddr 127.0.0.0/8 drop
|
||||||
|
'';
|
||||||
|
|
||||||
|
# Signal to the c0re Rust side that container isolation is
|
||||||
|
# enabled. c0re reads `HIVE_NETWORK_ISOLATION` from its service
|
||||||
|
# environment and uses it in `lifecycle::set_nspawn_flags` to set
|
||||||
|
# `PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=<hash-ip>`, and
|
||||||
|
# `HOST_BRIDGE=<bridgeName>` in each agent's container config.
|
||||||
|
# Also forwards the bridge name + subnet so c0re can wire the
|
||||||
|
# veth without hardcoding.
|
||||||
|
#
|
||||||
|
# `systemd.services.hive-c0re.environment` is an attrset; NixOS
|
||||||
|
# merges contributions from all modules that set it, so this
|
||||||
|
# cross-module injection is idiomatic and doesn't require a
|
||||||
|
# dedicated option in hive-c0re.nix.
|
||||||
|
systemd.services.hive-c0re.environment = {
|
||||||
|
HIVE_NETWORK_ISOLATION = "1";
|
||||||
|
HIVE_NETWORK_BRIDGE = cfg.bridgeName;
|
||||||
|
HIVE_NETWORK_SUBNET = "${cfg.bridgeIp}/${toString cfg.bridgePrefixLength}";
|
||||||
|
};
|
||||||
|
})
|
||||||
|
];
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue