refuse tls hellos that carry encrypted client hello before squid sees them

docs/4-implementation/networking/network.md

@@ -517,10 +517,11 @@ credential=present|absent unrestricted_canary=… unrestricted_status=…` line
 A program that ignores the proxy variables still gets squid's policy for its
 web traffic. In the `inet hive_egress` table (`nix/host-modules/hive-egress.nix`)
 the `intercept` chain, at `prerouting`, redirects IPv4 TCP port 80 to
-`<bridgeIp>:3129` and port 443 to `<bridgeIp>:3130`, the two intercept listeners
+`<bridgeIp>:3129`, an intercept listener of squid, and port 443 to
-that `hive-squid.nix` opens. Squid reads the original destination from
+`<bridgeIp>:3131`, the ClientHello relay (see _ECH_ below), which sends the
-conntrack. The chain exists where the host runs squid
+connection on to squid's other intercept listener, `<bridgeIp>:3130`. Squid reads
-(`services.hyperhive.network.egress.intercept.enable`).
+the original destination from conntrack. The chain exists where the host runs
+squid (`services.hyperhive.network.egress.intercept.enable`).
 
 - **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
   a flow to an address that stays on the bridge, to one of the host's own
@@ -570,25 +571,79 @@ conntrack. The chain exists where the host runs squid
   - a ClientHello with no SNI, and a URL with an IP literal, which sends none:
     the host of the `CONNECT` stays an address, and the IP-literal denies refuse
     it;
-  - bytes on port 443 that aren't a ClientHello, and on port 80 that aren't an
+  - bytes on port 80 that aren't an HTTP request. Bytes on port 443 that aren't
-    HTTP request.
+    a ClientHello never reach squid (see _ECH_ below).
+- **ECH.** Squid 7.6 can't see the `encrypted_client_hello` extension, only the
+  public name in the outer hello. An agent that holds an ECH configuration could
+  send a hello with an allowed outer name, a destination address that name
+  resolves to, and a hidden inner name that the same address serves. Port 443
+  therefore doesn't go to squid directly. `hive-priv egress-ech-relay`
+  (`hive-egress-ech-relay.service`, `egress.intercept.refuseEch`, on by default)
+  listens on `<bridgeIp>:3131`. It reads the first TLS records of each redirected
+  connection, up to four records and 16 KiB of ClientHello within 10 seconds, and
+  closes the connection when the ClientHello of a `restricted` agent carries the
+  extension (type `0xfe0d`, and the draft types `0xfe08` to `0xfe0c`, `0xff02`
+  and `0xffce`). The relay only observes a `log` agent, as everywhere in `log`
+  mode: its hello goes on like a clean one, and the relay logs it (see below).
+  It closes a connection too, for any agent, when the bytes aren't a well-formed
+  ClientHello, when they don't arrive in time, or when the connection wasn't
+  redirected. It also closes a connection from a source that already has 256
+  open, and any connection beyond 2048 in all, so one agent can't use up the
+  relay. Otherwise it connects to the destination the agent dialed and sends the bytes
+  on, and squid sees an ordinary intercepted connection:
+  - the relay connects from an address in `198.18.0.0/15`, which the unit routes
+    to `lo`, and an `output` rule DNATs its marked connections (mark
+    `0x45434800`) to squid's port 3130, so squid reads the original destination
+    from conntrack as before;
+  - the source address is `198.18.0.0` plus the low 17 bits of the agent's
+    address, and `egress-whois` maps it back to the lease it came from, so the
+    pool needs a bridge subnet of /15 or smaller (an assertion checks) and
+    squid's log shows `198.18.x.y` as the client address of an intercepted HTTPS
+    flow. The user column still names the agent. The `input` chain drops a pool
+    address that arrives from the bridge;
+  - the relay logs to the journal of its unit, at most one line a second per
+    source address, with the agent named the way squid names it (`-` when no
+    lease and binding says). A refusal is
+    `hive-egress-ech-refused agent=<name> dst=<address>`; an ECH hello of a `log`
+    agent, which goes on, is
+    `hive-egress-ech-would-refuse agent=<name> dst=<address>`; a hello that isn't
+    well formed is
+    `hive-egress-hello-refused agent=<name> dst=<address> reason=<why>`. Squid logs
+    nothing for a refused flow;
+  - the mode comes from `agent-allow.conf`, the include `hive-c0re` renders: an
+    agent is in `log` mode when it holds the line
+    `acl agent_log_<name> proxy_auth <name>`. Squid applies the same file, so the
+    relay and squid can't disagree about an agent's mode, and the file changes
+    when the operator changes the mode, with no second list to keep in step. A
+    missing or unreadable file, or an agent with no lease, counts as
+    `restricted`, and the relay refuses the hello;
+  - while the relay is down the redirect reaches a closed port, so HTTPS from
+    `restricted` and `log` agents fails. The unit restarts after 5 seconds.
 
-  Squid 7.6 can't see the `encrypted_client_hello` extension, only the public
+  GREASE: a client with no ECH configuration, Chrome and Firefox by default,
-  name in the outer hello. An agent that holds an ECH configuration can send a
+  sends the extension anyway with random contents, and the protocol makes the two
-  hello with an allowed outer name, a destination address that name resolves
+  look the same on the wire. The relay can't tell them apart, so it treats both
-  to, and a hidden inner name that the same address serves. Squid lets that
+  as ECH: a `restricted` agent's HTTPS fails when the client ignores the proxy
-  flow through, so ECH isn't refused here.
+  variables, and a `log` agent's goes on with a would-refuse line. Through the
+  proxy port (`HTTPS_PROXY`, a `CONNECT`) it works as before: that path has no
+  interception. A host where that costs more than the gap does can set
+  `egress.intercept.refuseEch = false`, which sends port 443 straight to squid
+  and leaves ECH unchecked. The relay only sees the hello of a connection that
+  the redirect catches. It doesn't touch UDP (QUIC), IPv6 or other ports.
 
 - **Reading the log.** Each intercepted HTTPS connection logs twice: first
   `NONE_NONE/000 … CONNECT <address>:443 <agent>`, before the ClientHello, then
   the `CONNECT <name>:443 <agent>` line with `TCP_TUNNEL/200` or `TCP_DENIED/403`
-  (`NONE_NONE/409` when the SNI and the address disagree). An intercepted HTTP
+  (`NONE_NONE/409` when the SNI and the address disagree). The client address of
-  request logs `GET http://<name>/ <agent>`. A tunnel to an intercepted
+  an intercepted HTTPS line is the relay's `198.18.x.y` (see _ECH_ above), and an
+  intercepted HTTP request logs `GET http://<name>/ <agent>` with the agent's own
+  address. A tunnel to an intercepted
   destination ends in `ORIGINAL_DST/<address>`. `cache.log` notes an ACL that met
   a flow squid refused before its request.
-- **Rollout.** The redirect, squid's configuration and `hive-priv` ship in one
+- **Rollout.** The redirect, the relay, squid's configuration and `hive-priv` ship
-  build. A hive that still runs an older build has no redirect, and its agents go
+  in one build. A hive that still runs an older build has no redirect, and its
-  direct as before. The agents that use the proxy variables see no change.
+  agents go direct as before. The agents that use the proxy variables see no
+  change.
 
 ## Container isolation
 

hive-priv/README.md

@@ -38,7 +38,9 @@ not the whole daemon-shared crate.
 `storage` (btrfs subvolumes, quotas, snapshots, disk measurement), `files`
 (pause marker, socket dir, legacy tmpfiles) and `infra` (gateway nginx, forge
 admin, infra containers). `egress` holds four sub-commands that never
-touch the socket, `egress-load`, `egress-pin`, `egress-lease` and `egress-whois`; the
+touch the socket, `egress-load`, `egress-pin`, `egress-lease` and `egress-whois`, and
+`egress_ech` a fifth, `egress-ech-relay`, the long-running ClientHello check in front of
+squid's HTTPS intercept port; the
 `hive-egress` unit, udev, a timer and dnsmasq run the first three to fill the sets of
 the anti-spoofing table and the interception exemptions, and squid runs the last to
 name the agent of an intercepted flow (see

hive-priv/src/egress.rs

@@ -38,6 +38,8 @@ use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
 
 use anyhow::{Context as _, Result, bail};
 
+use crate::egress_ech::same_host;
+
 /// The sub-command squid runs to learn which agent holds a source address.
 pub(crate) const WHOIS: &str = "egress-whois";
 
@@ -344,7 +346,9 @@ fn unrestricted_names() -> Vec<String> {
 
 /// The agent whose container holds the IPv4 address `ip`, as dnsmasq's lease
 /// file and the bindings of the ports say: the lease names a MAC, the MAC is
-/// bound to one port, and that port's machine is `h-<agent>`. `None` when any
+/// bound to one port, and that port's machine is `h-<agent>`. An address in the
+/// pool of the ECH relay (`egress_ech`) stands for the leased address it was
+/// derived from. `None` when any
 /// step is missing or ambiguous (two leases for the address under different
 /// MACs, a MAC bound twice, a machine that is no agent, a name that is no
 /// identifier): a flow whose agent cannot be told is refused, never guessed.
@@ -357,7 +361,7 @@ pub(crate) fn agent_of(ip: &str, leases: &str, pins: &[Pin], now: u64) -> Option
             0 => true,
             t => t > now,
         };
-        (live && addr.parse::<std::net::Ipv4Addr>().ok()? == ip).then(|| Mac::parse(mac))
+        (live && same_host(ip, addr.parse::<std::net::Ipv4Addr>().ok()?)).then(|| Mac::parse(mac))
     });
     let mac = macs.next()??;
     if macs.any(|other| other != Some(mac)) {
@@ -373,20 +377,25 @@ pub(crate) fn agent_of(ip: &str, leases: &str, pins: &[Pin], now: u64) -> Option
     Some(agent.to_owned())
 }
 
+/// [`agent_of`] on the files as they are now, which are read for every call.
+pub(crate) fn agent_of_now(ip: &str) -> Option<String> {
+    let leasefile = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into());
+    let pins = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default());
+    let leases = std::fs::read_to_string(leasefile).unwrap_or_default();
+    agent_of(ip, &leases, &pins, now_secs())
+}
+
 /// `egress-whois`: squid's external ACL helper. Each line on stdin starts with
 /// a source address; the answer is `OK user=<agent>` or `ERR`. The two files
 /// are read for every line, since squid caches the answers itself.
 fn whois() -> Result<()> {
     use std::io::{BufRead as _, Write as _};
-    let leasefile = std::env::var("HIVE_EGRESS_LEASEFILE").unwrap_or_else(|_| LEASEFILE.into());
     let stdin = std::io::stdin();
     let mut stdout = std::io::stdout().lock();
     for line in stdin.lock().lines() {
         let line = line.context("read stdin")?;
         let ip = line.split_whitespace().next().unwrap_or_default();
-        let pins = parse_state(&std::fs::read_to_string(state_path()).unwrap_or_default());
+        match agent_of_now(ip) {
-        let leases = std::fs::read_to_string(&leasefile).unwrap_or_default();
-        match agent_of(ip, &leases, &pins, now_secs()) {
             Some(agent) => writeln!(stdout, "OK user={agent}"),
             None => writeln!(stdout, "ERR"),
         }
@@ -1117,6 +1126,31 @@ mod tests {
         );
     }
 
+    #[test]
+    fn a_relay_pool_address_names_the_agent_it_was_derived_from() {
+        let pins = [
+            pin("vb-h-atlas", 10, 1, "h-atlas"),
+            pin("vb-h-uma", 11, 2, "h-uma"),
+        ];
+        let leases = lease_line(NOW + 60, 1, "10.42.0.5") + &lease_line(NOW + 60, 2, "10.42.0.6");
+        assert_eq!(
+            agent_of("198.18.0.5", &leases, &pins, NOW).as_deref(),
+            Some("atlas")
+        );
+        assert_eq!(
+            agent_of("198.18.0.6", &leases, &pins, NOW).as_deref(),
+            Some("uma")
+        );
+        // No lease behind the pool address, and an address of the pool that
+        // is itself leased: neither names anyone.
+        assert_eq!(agent_of("198.18.0.7", &leases, &pins, NOW), None);
+        let leased_pool = lease_line(NOW + 60, 1, "198.18.0.9");
+        assert_eq!(agent_of("198.18.0.9", &leased_pool, &pins, NOW), None);
+        // Two leases that share the pool address are ambiguous.
+        let twins = lease_line(NOW + 60, 1, "10.42.0.5") + &lease_line(NOW + 60, 2, "10.44.0.5");
+        assert_eq!(agent_of("198.18.0.5", &twins, &pins, NOW), None);
+    }
+
     #[test]
     fn an_address_nobody_can_be_told_to_hold_has_no_agent() {
         let pins = [

hive-priv/src/main.rs

@@ -23,7 +23,8 @@
 //! turns ephemeral, with [`salvage_io`] for its space, reflink and lock calls),
 //! [`prestart`] (the check of a container's root that runs before each start,
 //! as a sub-command), [`egress`] (the bridge anti-spoofing bindings, also
-//! sub-commands), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
+//! sub-commands), [`egress_ech`] (the `ClientHello` check in front of squid's
+//! HTTPS intercept port), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
 //! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles),
 //! [`infra`] (gateway nginx, infra containers) and [`closure`] (the swarm-signature
 //! check before an apply, and the cache-facts refresh). Helpers more than one family uses are in [`shared`].
@@ -32,6 +33,7 @@ mod closure;
 mod container;
 mod container_config;
 mod egress;
+mod egress_ech;
 mod files;
 mod infra;
 mod prestart;
@@ -60,6 +62,10 @@ async fn main() -> Result<()> {
     if let Some(code) = egress::run(&args) {
         std::process::exit(code);
     }
+    // `egress-ech-relay` is a long-running unit of its own, not a request either.
+    if let Some(code) = egress_ech::run(&args).await {
+        std::process::exit(code);
+    }
 
     tracing_subscriber::fmt()
         .with_env_filter(

nix/host-modules/hive-egress.nix

@@ -13,10 +13,10 @@
 # the host routes from the agent bridge to anywhere else is logged and
 # counted, never blocked. `intercept` (`egress.intercept.enable`, set by
 # hive-squid.nix): the web traffic of the bridge's agents is redirected into
-# squid's intercept ports. The traffic of the ports in the set `direct_mac` is
+# squid's intercept ports (port 443 through the ClientHello relay). The ports in
-# left alone: the service containers and the unrestricted agents. `hive-priv
+# `direct_mac` are left alone: the service containers and the unrestricted
-# egress-pin` and `egress-load` fill that set from the bindings the bridge
+# agents. `hive-priv egress-pin` and `egress-load` fill that set from the bridge
-# table uses, so a MAC in no binding is intercepted, and squid refuses the flow.
+# table's bindings, so a MAC in no binding is intercepted, and squid refuses it.
 #
 # The `bridge` table checks the source address of every frame a port of the
 # agent bridge sends (`egress.antiSpoof`). It is a `bridge`-family table because
@@ -42,6 +42,19 @@ let
   # The mesh interface (swarm-wireguard.nix): inter-hive traffic is not egress.
   meshInterface = "wg-hive";
 
+  # With `refuseEch`, the redirect of port 443 ends at `hive-priv
+  # egress-ech-relay`, which refuses a ClientHello that carries the ECH
+  # extension and sends the rest on to squid's HTTPS intercept port. The relay
+  # connects to the address the agent dialed, bound to a source address of the
+  # pool and carrying the mark; the `output` chain DNATs those connections to
+  # squid. `egress_ech.rs` derives the pool address from the agent's and
+  # `egress-whois` maps it back, so the pool is the one defined there.
+  intercept = cfg.egress.intercept;
+  refuseEch = intercept.enable && intercept.refuseEch;
+  relayPool = "198.18.0.0/15";
+  relayMark = "0x45434800";
+  relayCommand = "${hivePriv} egress-ech-relay ${cfg.bridgeIp}:${toString intercept.relayPort} ${relayMark}";
+
   # The sub-commands of hive-priv that fill the bridge table's sets. The path is
   # the one every platform container's unit uses (hive-priv.nix points it at the
   # current build).
@@ -85,9 +98,9 @@ let
   # the interface the route to the destination leaves by: the bridge (another
   # container), `lo` (an address of the host: `fib daddr type local`) and the
   # mesh interface are not egress. The redirect rewrites the destination to
-  # the bridge address and the port of squid's intercept listener, which reads
+  # the bridge address and the port of squid's intercept listener (for port 443
-  # the original destination from conntrack. IPv6 is left alone: the agent
+  # with `refuseEch`, of the relay), which reads the original destination from
-  # bridge has no IPv6 addressing.
+  # conntrack. IPv6 is left alone: the agent bridge has no IPv6 addressing.
   interceptChain = ''
     chain intercept {
       type nat hook prerouting priority dstnat; policy accept;
@@ -95,7 +108,16 @@ let
       iifname "${br}" meta nfproto ipv4 fib daddr type local return
       iifname "${br}" meta nfproto ipv4 fib daddr oifname { "${br}", "${meshInterface}" } return
       iifname "${br}" meta nfproto ipv4 tcp dport 80 counter redirect to :${toString cfg.egress.intercept.httpPort}
-      iifname "${br}" meta nfproto ipv4 tcp dport 443 counter redirect to :${toString cfg.egress.intercept.httpsPort}
+      iifname "${br}" meta nfproto ipv4 tcp dport 443 counter redirect to :${
+        toString (if refuseEch then intercept.relayPort else intercept.httpsPort)
+      }
+    }
+  ''
+  + lib.optionalString refuseEch ''
+
+    chain intercept_relay {
+      type nat hook output priority dstnat; policy accept;
+      meta nfproto ipv4 ip saddr ${relayPool} meta mark ${relayMark} tcp dport 443 counter dnat ip to ${cfg.bridgeIp}:${toString intercept.httpsPort}
     }
   '';
 
@@ -206,6 +228,7 @@ let
         type filter hook input priority filter - 10; policy accept;
         iifname "${br}" ip daddr 127.0.0.0/8 counter drop
         iifname "${br}" ip6 daddr ::1 counter drop
+        ${lib.optionalString refuseEch ''iifname "${br}" ip saddr ${relayPool} counter drop''}
       }
       set direct_mac { type ether_addr; }
       ${lib.optionalString cfg.egress.intercept.enable interceptChain}
@@ -295,8 +318,55 @@ in
       type = lib.types.port;
       default = 3130;
       description = ''
-        The port on the bridge address where squid listens for the redirected
+        The port on the bridge address where squid listens for TCP port 443
-        TCP port 443 traffic.
+        traffic: what the ClientHello relay sends on, or with `refuseEch` off
+        the redirected traffic itself.
+      '';
+    };
+
+    relayPort = lib.mkOption {
+      type = lib.types.port;
+      default = 3131;
+      description = ''
+        The port on the bridge address where `hive-priv egress-ech-relay`
+        listens for the redirected TCP port 443 traffic (`refuseEch`).
+      '';
+    };
+
+    refuseEch = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = ''
+        Refuse a TLS ClientHello that carries the `encrypted_client_hello`
+        extension on the redirected port 443 traffic. Squid 7.6 sees only the
+        outer name of such a hello, so without this an agent that holds an ECH
+        configuration could dial an allowed name and reach another site behind
+        the same address.
+
+        `hive-priv egress-ech-relay` reads the first TLS record of each
+        redirected connection and, if the hello is clean, sends the connection
+        on to squid. The extension of a client with no configuration (GREASE,
+        which Chrome and Firefox send by default) looks the same as a real one on
+        the wire, so it is refused too: those clients fail on the redirected
+        path and work through the proxy variables. An agent in `log` mode is
+        observed instead: its hello goes on and the relay logs
+        `hive-egress-ech-would-refuse`. Turn this off to send port 443
+        straight to squid, with ECH unchecked.
+      '';
+    };
+
+    agentAllowFile = lib.mkOption {
+      type = lib.types.nullOr lib.types.path;
+      default = null;
+      description = ''
+        The squid include `hive-c0re` renders with each agent's own rules, in
+        which an agent in `log` mode has an `acl agent_log_<name> proxy_auth
+        <name>` line. The ClientHello relay (`refuseEch`) reads it to tell a
+        `log` agent, whose ECH hello is let through and logged as
+        `hive-egress-ech-would-refuse`, from a restricted one, whose hello is
+        refused. Squid reads the same file for the same split. `null` or an
+        unreadable file lists no `log` agent, so every agent's ECH hello is
+        refused.
       '';
     };
 
@@ -349,6 +419,14 @@ in
           Use "log" or "accept".
         '';
       }
+      {
+        assertion = !refuseEch || lib.toInt (lib.last (lib.splitString "/" cfg.networkCidr)) >= 15;
+        message = ''
+          services.hyperhive.network.bridgePrefixLength must be 15 or more while
+          egress.intercept.refuseEch is on: the relay names a flow's source with
+          the low 17 bits of the agent's address.
+        '';
+      }
       {
         assertion = !(lib.any (k: isOn (config.boot.kernel.sysctl.${k} or 0)) routeLocalnetKeys);
         message = ''
@@ -360,6 +438,53 @@ in
       }
     ];
 
+    services.hyperhive.network.exposeHostPorts = lib.mkIf refuseEch [ intercept.relayPort ];
+
+    # The relay is not part of the tables: with it down, the redirect of port 443
+    # reaches a closed port and the agent's connection is refused.
+    systemd.services.hive-egress-ech-relay = lib.mkIf refuseEch {
+      description = "refuse TLS hellos with encrypted client hello on the way to squid's HTTPS intercept port";
+      wantedBy = [ "multi-user.target" ];
+      wants = [ "hive-egress.service" ];
+      after = [
+        "hive-egress.service"
+        "systemd-tmpfiles-setup.service"
+        "systemd-tmpfiles-resetup.service"
+      ];
+      environment = lib.optionalAttrs (intercept.agentAllowFile != null) {
+        HIVE_EGRESS_AGENT_ALLOW = toString intercept.agentAllowFile;
+      };
+      serviceConfig = {
+        # The pool is local, so the relay can bind to a source address in it.
+        ExecStartPre = "+${pkgs.iproute2}/bin/ip route replace local ${relayPool} dev lo";
+        ExecStart = relayCommand;
+        ExecStopPost = "+-${pkgs.iproute2}/bin/ip route del local ${relayPool} dev lo";
+        # The bridge address may not be assigned yet at first start.
+        Restart = "on-failure";
+        RestartSec = 5;
+        # The packet mark needs CAP_NET_ADMIN. The include is `squid`'s group's.
+        DynamicUser = true;
+        SupplementaryGroups = lib.optional (intercept.agentAllowFile != null) "squid";
+        AmbientCapabilities = "CAP_NET_ADMIN";
+        CapabilityBoundingSet = "CAP_NET_ADMIN";
+        NoNewPrivileges = true;
+        ProtectSystem = "strict";
+        ProtectHome = true;
+        PrivateTmp = true;
+        PrivateDevices = true;
+        ProtectKernelTunables = true;
+        ProtectKernelModules = true;
+        ProtectKernelLogs = true;
+        ProtectControlGroups = true;
+        RestrictAddressFamilies = "AF_INET";
+        RestrictNamespaces = true;
+        RestrictRealtime = true;
+        RestrictSUIDSGID = true;
+        LockPersonality = true;
+        SystemCallArchitectures = "native";
+      };
+    };
+
     systemd.services.hive-egress = {
       description = "hive nftables tables (agent bridge egress log and source check, host loopback drop)";
       wantedBy = [ "multi-user.target" ];

nix/host-modules/hive-squid.nix

@@ -25,6 +25,8 @@
 # `hive_agent`); a source it cannot name is refused. The listed-names and deny
 # rules are the same ones, matched on the identity from that helper. HTTPS is
 # peeked at for its SNI and spliced: squid checks the name and never decrypts.
+# Squid cannot see an ECH extension, so the flows to port 443 reach it through
+# `hive-priv egress-ech-relay`, which refuses a hello that carries one.
 {
   pkgs,
   lib,
@@ -551,7 +553,7 @@ in
     ];
     services.hyperhive.network.egress.intercept = {
       enable = lib.mkDefault true;
-      inherit (cfg) unrestrictedFile;
+      inherit (cfg) unrestrictedFile agentAllowFile;
     };
     services.hyperhive.network.enable = lib.mkDefault true;
 

nix/module-eval/core-toggle.nix

@@ -88,7 +88,10 @@ let
   # Every hive defaults the agents' queue address to the queue's name, so the
   # only hive without one is a hive told to have none. Also the host still
   # setting the four removed host options; no case on its queue reads them.
+  # It is also the one controller host with `egress.intercept.refuseEch` off,
+  # which the egress cases read.
   noAgentQueue = hiveWithController {
+    network.egress.intercept.refuseEch = false;
     deploy.hive-controller.queue.agentNatsUrl = null;
     c0re.operatorPronouns = "they/them";
     c0re.contextWindowTokens.sonnet = 900000;
@@ -1700,7 +1703,9 @@ let
             lib.splitString "table bridge hive_egress {" bare.services.hyperhive.network.egress.ruleset
           );
           chain = lib.head (
-            lib.splitString "set log_v4" (lib.last (lib.splitString "chain intercept {" inet))
+            lib.splitString "chain intercept_relay {" (
+              lib.head (lib.splitString "set log_v4" (lib.last (lib.splitString "chain intercept {" inet)))
+            )
           );
           rules = lib.filter (l: l != "" && l != "}") (map lib.trim (lib.splitString "\n" chain));
           idx = pat: lib.lists.findFirstIndex (lib.hasInfix pat) 1000 rules;
@@ -1714,12 +1719,72 @@ let
         && idx ''fib daddr oifname { "hive-br0", "wg-hive" } return'' < idx "tcp dport 80"
         && lib.length redirects == 2
         && lib.hasInfix "tcp dport 80 counter redirect to :3129" (lib.head redirects)
-        && lib.hasInfix "tcp dport 443 counter redirect to :3130" (lib.last redirects)
+        # Port 443 goes to the ClientHello relay, not to squid.
+        && lib.hasInfix "tcp dport 443 counter redirect to :3131" (lib.last redirects)
         # Only IPv4 that comes in on the bridge is looked at.
         && lib.all (l: lib.hasPrefix ''iifname "hive-br0" meta nfproto ipv4 '' l) (lib.drop 1 rules)
         && lib.hasInfix "set direct_mac { type ether_addr; }" without
         && !(lib.hasInfix "chain intercept" without);
     }
+    {
+      # With the relay on, port 443 reaches squid through `hive-priv
+      # egress-ech-relay`: its connections (from the pool, carrying the mark)
+      # are DNATed to squid's HTTPS port, the bridge may not send from the
+      # pool, and the unit routes the pool to `lo`. `noAgentQueue` has the
+      # relay off: the redirect then goes straight to squid and none of the
+      # rest is rendered. Its intercept chain is the presence control.
+      name = "the ClientHello relay sits between the redirect and squid, and refuseEch off removes it";
+      ok =
+        let
+          ruleset = m: m.services.hyperhive.network.egress.ruleset;
+          unit = bare.systemd.services.hive-egress-ech-relay;
+          off = noAgentQueue;
+        in
+        lib.hasInfix "chain intercept_relay {" (ruleset bare)
+        && lib.hasInfix "type nat hook output priority dstnat; policy accept;" (ruleset bare)
+        && lib.hasInfix "meta nfproto ipv4 ip saddr 198.18.0.0/15 meta mark 0x45434800 tcp dport 443 counter dnat ip to 10.42.0.1:3130" (
+          ruleset bare
+        )
+        && lib.hasInfix ''iifname "hive-br0" ip saddr 198.18.0.0/15 counter drop'' (ruleset bare)
+        && lib.hasInfix "egress-ech-relay 10.42.0.1:3131 0x45434800" unit.serviceConfig.ExecStart
+        && lib.hasInfix "ip route replace local 198.18.0.0/15 dev lo" unit.serviceConfig.ExecStartPre
+        && lib.elem "multi-user.target" unit.wantedBy
+        # The relay reads the include in which squid finds the `log` agents.
+        &&
+          (unit.environment.HIVE_EGRESS_AGENT_ALLOW or null)
+          == bare.services.hyperhive.deploy.hive-controller.squid.agentAllowFile
+        && unit.serviceConfig.SupplementaryGroups == [ "squid" ]
+        && lib.elem 3131 bare.services.hyperhive.network.exposeHostPorts
+        && lib.hasInfix "chain intercept {" (ruleset off)
+        && lib.hasInfix "tcp dport 443 counter redirect to :3130" (ruleset off)
+        && lib.elem 3130 off.services.hyperhive.network.exposeHostPorts
+        && !(lib.hasInfix "intercept_relay" (ruleset off))
+        && !(lib.hasInfix "198.18.0.0/15" (ruleset off))
+        && !(off.systemd.services ? hive-egress-ech-relay)
+        && !(lib.elem 3131 off.services.hyperhive.network.exposeHostPorts);
+    }
+    {
+      # The relay names a flow's source with the low 17 bits of the agent's
+      # address, which is unique only in a subnet of /15 or smaller. The
+      # boundary is read at /15 (accepted) and /14 (refused); `bare`, a /24, is
+      # the control that the lookup finds nothing otherwise.
+      name = "the ClientHello relay is refused on a bridge subnet larger than a /15";
+      ok =
+        let
+          refused =
+            m:
+            lib.any (
+              a: !a.assertion && lib.hasInfix "bridgePrefixLength must be 15 or more" a.message
+            ) m.assertions;
+        in
+        refused (hiveWithController {
+          network.bridgePrefixLength = 14;
+        })
+        && !(refused (hiveWithController {
+          network.bridgePrefixLength = 15;
+        }))
+        && !(refused bare);
+    }
     {
       # Squid listens on the two intercept ports, peeks and splices and never
       # decrypts, names the agent of a flow with the helper, refuses a source

nix/vm-tests/hive-squid.nix

@@ -216,6 +216,7 @@ pkgs.testers.runNixOSTest {
           "L+ /run/hive-rootfs-check - - - - ${pkgs.writeShellScript "hive-priv-stub" ''
             [ "$1" = egress-load ] && exec ${pkgs.nftables}/bin/nft -f "$2"
             [ "$1" = egress-whois ] && exec ${hivePriv}/bin/hive-priv "$@"
+            [ "$1" = egress-ech-relay ] && exec ${hivePriv}/bin/hive-priv "$@"
             exit 0
           ''}"
         ];
@@ -319,7 +320,10 @@ pkgs.testers.runNixOSTest {
           "agentonly.t.local"
           "api.t.local"
         ];
-        environment.systemPackages = [ pkgs.openssl ];
+        environment.systemPackages = [
+          pkgs.openssl
+          pkgs.python3
+        ];
         networking.interfaces.eth1.ipv4.addresses = lib.mkForce [
           {
             address = "192.168.1.3";
@@ -957,7 +961,7 @@ pkgs.testers.runNixOSTest {
     def agent_lines():
         # Log lines of requests from the agent node's addresses. The self-check
         # connects from the bridge address, so its lines are not counted.
-        return len(re.findall(r" 192\.168\.1\.[347] [A-Z_]+/\d+ ", journal()))
+        return len(re.findall(r" (?:192\.168\.1|198\.18\.0)\.[347] [A-Z_]+/\d+ ", journal()))
 
     seeded = proxy.succeed(f"cat {ALLOWED}")
     bind("atlas", "logan")
@@ -980,7 +984,7 @@ pkgs.testers.runNixOSTest {
     with checked("a restricted agent's allowed name is tunnelled without the proxy variables, and the log names the agent"):
         wait_icpt("https://forge.t.local/", "200")
         assert re.search(
-            r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ORIGINAL_DST/203\.0\.113\.2 ", journal()
+            r"198\.18\.0\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ORIGINAL_DST/203\.0\.113\.2 ", journal()
         ), "the HTTPS flow, as atlas"
         assert icpt("http://forge.t.local/") == "200", "plain HTTP"
         assert re.search(r"192\.168\.1\.3 TCP_\w+/200 .* GET http://forge\.t\.local/ atlas ", journal()), "the HTTP flow, as atlas"
@@ -992,7 +996,7 @@ pkgs.testers.runNixOSTest {
 
     with checked("a restricted agent's unlisted name is refused, intercepted, and logged as the agent's"):
         assert icpt("https://other.t.local/") == "000", "HTTPS"
-        assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal()), "logged as atlas"
+        assert re.search(r"198\.18\.0\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal()), "logged as atlas"
         assert icpt("http://other.t.local/") == "403", "plain HTTP"
         assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* GET http://other\.t\.local/ atlas ", journal()), "logged as atlas"
         # Control: the same address reaches the listed name, so the refusal is the policy's.
@@ -1027,7 +1031,7 @@ pkgs.testers.runNixOSTest {
 
     with checked("a source the bindings do not name is refused, and so is every source when they are gone"):
         assert icpt("https://forge.t.local/", NOBODY_IP) == "000", "no lease for the address"
-        assert re.search(r"192\.168\.1\.7 TCP_DENIED/403 .* CONNECT 203\.0\.113\.2:443 - ", journal()), "logged without a user"
+        assert re.search(r"198\.18\.0\.7 TCP_DENIED/403 .* CONNECT 203\.0\.113\.2:443 - ", journal()), "logged without a user"
         assert icpt("http://forge.t.local/", NOBODY_IP) == "403"
         assert icpt("https://forge.t.local/") == "200", "control: the bound address, the same name"
         # A hive whose pin file is missing: the lease names a MAC, no port holds it.
@@ -1056,6 +1060,90 @@ pkgs.testers.runNixOSTest {
         )
         assert out.strip() == "", "bytes that are no ClientHello get no answer"
 
+    # The ClientHello a client sends, built by hand: SNI, groups, signature
+    # algorithms and supported versions, and with `ech` the encrypted_client_hello
+    # extension (0xfe0d) with the layout a real one has. It prints how the
+    # connection ended: the first byte of what came back, or `closed`.
+    agent.succeed(
+        "cat > /tmp/hello.py <<'PY'\n"
+        "import os, socket, struct, sys\n"
+        "mode, name, dst, src = sys.argv[1:5]\n"
+        "def ext(kind, data):\n"
+        "    return struct.pack('>HH', kind, len(data)) + data\n"
+        "host = name.encode()\n"
+        "exts = ext(0, struct.pack('>HBH', len(host) + 3, 0, len(host)) + host)\n"
+        "exts += ext(10, struct.pack('>HHH', 4, 29, 23)) + ext(13, struct.pack('>HHH', 4, 0x0804, 0x0403))\n"
+        "exts += ext(43, bytes([2, 3, 4])) + ext(51, struct.pack('>H', 0))\n"
+        "if mode == 'ech':\n"
+        "    exts += ext(0xFE0D, b'\\x00' + struct.pack('>HHBH', 1, 1, 7, 32) + os.urandom(32) + struct.pack('>H', 144) + os.urandom(144))\n"
+        "body = b'\\x03\\x03' + os.urandom(32) + b'\\x20' + os.urandom(32) + struct.pack('>H', 2) + b'\\x13\\x01' + b'\\x01\\x00'\n"
+        "body += struct.pack('>H', len(exts)) + exts\n"
+        "hello = b'\\x01' + struct.pack('>I', len(body))[1:] + body\n"
+        "s = socket.socket()\n"
+        "s.settimeout(10)\n"
+        "s.bind((src, 0))\n"
+        "s.connect((dst, 443))\n"
+        "s.sendall(b'\\x16\\x03\\x01' + struct.pack('>H', len(hello)) + hello)\n"
+        "try:\n"
+        "    first = s.recv(1)\n"
+        "except OSError:\n"
+        "    first = b'''\n"
+        "print(first.hex() or 'closed')\n"
+        "PY"
+    )
+
+    def send_hello(mode, src=ATLAS_IP):
+        return agent.succeed(f"python3 /tmp/hello.py {mode} forge.t.local 203.0.113.2 {src}").strip()
+
+    with checked("a ClientHello with the ECH extension is refused for a restricted agent and observed for a log agent, and the same hello without it is let through"):
+        seen = agent_lines()
+        # Control: the origin answers a hello without the extension (a
+        # handshake record, 0x16), and squid logs the connection as atlas.
+        assert send_hello("plain") == "16", "the origin answers the hello without ECH"
+        time.sleep(2)
+        assert agent_lines() > seen, "squid saw the connection"
+        seen = agent_lines()
+        assert send_hello("ech") == "closed", "a hello with an ECH extension gets no answer"
+        proxy.wait_until_succeeds(
+            "journalctl -u hive-egress-ech-relay.service -o cat | "
+            "grep -E '^hive-egress-ech-refused agent=atlas dst=203\\.0\\.113\\.2$'"
+        )
+        time.sleep(2)
+        assert agent_lines() == seen, "squid saw nothing of it"
+        # A `log` agent is observed, not refused: its ECH hello goes on to squid
+        # and the origin, and the relay logs what it would have done. The mode is
+        # what the squid include says (`acl agent_log_logan proxy_auth logan`).
+        seen = agent_lines()
+        assert send_hello("ech", LOGAN_IP) == "16", "logan's hello with ECH is let through"
+        proxy.wait_until_succeeds(
+            "journalctl -u hive-egress-ech-relay.service -o cat | "
+            "grep -E '^hive-egress-ech-would-refuse agent=logan dst=203\\.0\\.113\\.2$'"
+        )
+        time.sleep(2)
+        assert agent_lines() > seen, "squid saw logan's connection"
+        assert "hive-egress-ech-refused agent=logan" not in proxy.succeed(
+            "journalctl -u hive-egress-ech-relay.service -o cat --no-pager"
+        ), "logan was not refused"
+        assert "ech-would-refuse agent=atlas" not in proxy.succeed(
+            "journalctl -u hive-egress-ech-relay.service -o cat --no-pager"
+        ), "control: the restricted agent got the refusal, not the observation"
+        assert send_hello("plain", LOGAN_IP) == "16", "control: logan's hello without ECH"
+        # A source the bindings do not name is refused too, logged without a name.
+        assert send_hello("ech", NOBODY_IP) == "closed"
+        proxy.wait_until_succeeds(
+            "journalctl -u hive-egress-ech-relay.service -o cat | "
+            "grep -E '^hive-egress-ech-refused agent=- dst=203\\.0\\.113\\.2$'"
+        )
+        # Bytes that are no ClientHello: refused by the relay, with a line of their own.
+        out = agent.succeed(
+            "timeout 5 bash -c 'exec 3<>/dev/tcp/203.0.113.2/443; echo hello >&3; cat <&3' 2>/dev/null || true"
+        )
+        assert out.strip() == "", "bytes that are no ClientHello get no answer"
+        proxy.wait_until_succeeds(
+            "journalctl -u hive-egress-ech-relay.service -o cat | "
+            "grep -E '^hive-egress-hello-refused agent=atlas dst=203\\.0\\.113\\.2 reason=not a TLS handshake record$'"
+        )
+
     with checked("the traffic that stays on the bridge or is the unrestricted agents' is not intercepted"):
         seen = agent_lines()
         # The gateway on the bridge address, in the stand-in's port 443.
@@ -1068,6 +1156,7 @@ pkgs.testers.runNixOSTest {
         assert icpt("https://other.t.local/") == "200", "an unlisted name, directly"
         assert icpt("https://203.0.113.2/") == "200", "an IP literal, directly"
         assert agent_lines() == seen, "squid saw nothing of it"
+        assert send_hello("ech") == "16", "a hello with ECH, directly: the relay is not in the path"
         proxy.wait_until_succeeds(
             "journalctl -k -o cat --no-pager | grep -E "
             "'hive-egress-would-drop(-unmetered)? IN=eth1 OUT=eth2 .*SRC=192\\.168\\.1\\.3 DST=203\\.0\\.113\\.2 .*PROTO=TCP .*DPT=443 '"