squid intercepts restricted and log agents' web traffic, named from the bound source address

docs/4-implementation/networking/network.md

@@ -229,13 +229,18 @@ Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
 (`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
 only. An agent sends its outbound HTTP and HTTPS through it once its hive says
 the proxy is ready and it has fetched its proxy credential (see _Agents and the
-proxy_ below). Any other agent goes straight out through the NAT, as before,
-and so does any program that ignores the proxy variables: nothing blocks direct
-egress yet. The host logs those direct connections (see _Agent bridge egress log_
-below).
+proxy_ below). A program that ignores the proxy variables still reaches squid
+when it speaks HTTP on port 80 or TLS on port 443 and its agent is `restricted`
+or in `log` mode: the host redirects that traffic into squid's intercept ports
+(see _Intercepted web traffic_ below). Everything else goes straight out through
+the NAT, as before: the `unrestricted` agents, the service containers and every
+other port and protocol. Nothing blocks that yet. The host logs those direct
+connections (see _Agent bridge egress log_ below).
 
 Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
-host. It never decrypts TLS, so it holds no signing key. It caches nothing.
+host. It never decrypts TLS. The one TLS key it loads is a throwaway that the
+unit generates at each start, that signs nothing and that no agent trusts. It
+caches nothing.
 
 - **Credentials.** A request for a listed name must carry a per-agent proxy
   credential (HTTP Basic, the agent's name and its secret). `swarm-controller`
@@ -310,10 +315,10 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
   `internet_log: true`, so a `hive-c0re` that doesn't know the field still
   enforces the list. For such an agent `hive-c0re` adds to `agent-allow.conf`,
   after the allow rules for every agent's own names,
-  `http_access allow <user> !allowed_dst log_mark`, where `log_mark` is an
-  `annotate_transaction` ACL that sets `would_deny=1`. The rule comes last in
-  the include, so every destination deny and `deny !authed` before it still
-  applies, and a name the agent may reach never carries the mark. A request the
+  `http_access allow !icpt <user> !allowed_dst log_mark` and the same rule for
+  the intercept ports, where `log_mark` is an `annotate_transaction` ACL that
+  sets `would_deny=1`. The rules come last in the include, so every destination
+  deny and `deny !icpt !authed` before them still applies, and a name the agent may reach never carries the mark. A request the
   rule lets through gets a second log line,
   `would_deny user=<agent> method=<m> host=<h> port=<p> result=<r>`, from the
   `would_denied` `access_log` in `hive-squid.nix`; the usual request line keeps
@@ -344,7 +349,7 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
   (`/var/lib/hive-squid/deny.conf`, `hive-core:squid`, `0640`) for the agents it
   runs, and `squid-unrestricted-reload.path` reloads squid when it changes.
   - **Precedence.** The configuration includes the file after the destination
-    denies and before `deny !authed` and every allow, so a denied name is
+    denies and before `deny !icpt !authed` and every allow, so a denied name is
     refused even when the base list, the hive's `allowedDomains`, the swarm's
     allow-list, the unrestricted flag or `log` mode would allow it. A swarm-wide
     entry names no user, so every client gets a 403 for it, one without a
@@ -406,10 +411,10 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
   logs `squid allow list: 2 swarm, 3 agent entries` when they change. The
   swarm-wide names join the allow ACL, so a restricted user reaches them. A
   per-agent entry is a rule for that agent's squid user, `http_access allow
-<user> <names>`, in the generated `agent-allow.conf` that the configuration
-  includes at the point where the base list gets its allow: after every
-  destination deny and
-  `deny !authed`, so the rules under _Denied before any allow_ still bind it,
+!icpt <user> <names>`, and a second one for the user the intercept helper names
+  (`icpt <user> <names>`), in the generated `agent-allow.conf` that the
+  configuration includes at the point where the base list gets its allow: after
+  every destination deny and `deny !icpt !authed`, so the rules under _Denied before any allow_ still bind it,
   and before the unrestricted rule. Only this hive's agents get a rule.
   `squid-unrestricted-reload.path` reloads squid when either file changes, so
   there is no deploy or restart.
@@ -507,6 +512,84 @@ credential=present|absent unrestricted_canary=… unrestricted_status=…` line
   host's resolver looks up the canary, so a host whose resolver maps
   that name to a loopback, bridge or private address fails the check.
 
+### Intercepted web traffic
+
+A program that ignores the proxy variables still gets squid's policy for its
+web traffic. In the `inet hive_egress` table (`nix/host-modules/hive-egress.nix`)
+the `intercept` chain, at `prerouting`, redirects IPv4 TCP port 80 to
+`<bridgeIp>:3129` and port 443 to `<bridgeIp>:3130`, the two intercept listeners
+that `hive-squid.nix` opens. Squid reads the original destination from
+conntrack. The chain exists where the host runs squid
+(`services.hyperhive.network.egress.intercept.enable`).
+
+- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
+  a flow to an address that stays on the bridge, to one of the host's own
+  addresses or to the `wg-hive` interface, and a flow from a MAC in the set
+  `direct_mac`. That set holds the MACs of the ports of every container that's
+  not an agent (the swarm's service containers) and of the agents listed in
+  `services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
+  `restricted` and `log` agents get redirected, and `unrestricted` agents and
+  service containers keep the direct path. Every pass of `hive-priv egress-pin`
+  and every `egress-load` refills the set from the bindings it also uses for the
+  bridge table, and `hive-egress-direct.path` runs a pass when `hive-c0re`
+  rewrites the list; the timer covers the rest. A MAC in no binding is in no
+  exemption: the flow gets redirected, and squid refuses it. A new agent that
+  the list doesn't name yet counts as restricted until the next pass, and a flag
+  change reaches the redirect after the hive's next read of the permissions and
+  one pass. IPv6 and the other ports aren't redirected.
+- **Who the flow belongs to.** An intercepted request carries no proxy
+  credential, so squid asks its external ACL `hive_agent`, which runs
+  `hive-priv egress-whois`. The helper looks the source address up in dnsmasq's
+  lease file for a MAC, the MAC in the bindings (`/run/hive-egress-pins`) for a
+  machine, and takes the agent from the machine name `h-<agent>`. While
+  `antiSpoof` is `drop`, an address on the bridge is the one leased to the MAC
+  of the port it came from (see _Agent bridge source check_), so the source says
+  which container sent the flow. Squid keeps an answer for 10 seconds and a
+  refusal for 2. An address with no lease or an expired one, two MACs with the
+  same address, a MAC no port holds, a machine that's no agent: none gets an
+  answer, and squid refuses the flow with a `TCP_DENIED/403` and user `-`. A
+  host that has no bindings or leases yet refuses every intercepted flow rather
+  than guess. With `antiSpoof = log` a container can forge the source, so use that mode
+  to diagnose and not to run. The proxy port never takes this identity: a
+  request without a credential through port 3128 still gets a 407, whatever its
+  address.
+- **Policy.** The same lists as for the proxy port, matched on the agent the
+  helper names: the destination denies, the deny list, the base list, the
+  allow-list and the agent's own names, and `log` mode with its `would_deny`
+  line. `hive-c0re` renders each agent rule twice, once for the credential
+  (`!icpt`) and once for the helper's user (`icpt`).
+- **HTTPS.** Squid peeks at the ClientHello for the SNI and splices the
+  connection. It never decrypts, and no agent trusts a CA of the host. Squid
+  builds a `CONNECT` with the SNI as its host and applies every rule to it. A
+  refused name gets a 403 that the client reads as a TLS error. Squid also
+  refuses:
+  - an SNI that doesn't resolve to the address the agent dialed
+    (`host_verify_strict`; squid answers 409 and the client reads a TLS error).
+    Squid resolves on the host's resolver, which is the same dnsmasq as the
+    agent's, so a name with changing answers can fail now and then;
+  - a ClientHello with no SNI, and a URL with an IP literal, which sends none:
+    the host of the `CONNECT` stays an address, and the IP-literal denies refuse
+    it;
+  - bytes on port 443 that aren't a ClientHello, and on port 80 that aren't an
+    HTTP request.
+
+  Squid 7.6 can't see the `encrypted_client_hello` extension, only the public
+  name in the outer hello. An agent that holds an ECH configuration can send a
+  hello with an allowed outer name, a destination address that name resolves
+  to, and a hidden inner name that the same address serves. Squid lets that
+  flow through, so ECH isn't refused here.
+
+- **Reading the log.** Each intercepted HTTPS connection logs twice: first
+  `NONE_NONE/000 … CONNECT <address>:443 <agent>`, before the ClientHello, then
+  the `CONNECT <name>:443 <agent>` line with `TCP_TUNNEL/200` or `TCP_DENIED/403`
+  (`NONE_NONE/409` when the SNI and the address disagree). An intercepted HTTP
+  request logs `GET http://<name>/ <agent>`. A tunnel to an intercepted
+  destination ends in `ORIGINAL_DST/<address>`. `cache.log` notes an ACL that met
+  a flow squid refused before its request.
+- **Rollout.** The redirect, squid's configuration and `hive-priv` ship in one
+  build. A hive that still runs an older build has no redirect, and its agents go
+  direct as before. The agents that use the proxy variables see no change.
+
 ## Container isolation
 
 Each agent container runs in a private network namespace with a dedicated
@@ -590,14 +673,18 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
 the `inet hive_egress` table does with traffic the host routes from the agent
 bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes
 `accept`, `log` (the default) or `drop`. The evaluation refuses `drop` for now:
-the interception that would carry an agent's web traffic through squid instead
-hasn't landed. In `log` mode the chain blocks nothing. It logs each new flow
+the traffic that squid doesn't handle (the unrestricted agents, the service
+containers, every port but 80 and 443) has no rule that lets it through yet. In
+`log` mode the chain blocks nothing. It logs each new flow
 and counts every packet.
 
 - **Which traffic.** IPv4 and IPv6 alike (the table is `inet`). The chain never
   sees traffic to the host's own addresses, because the host delivers it
   locally: a connection to squid on `<bridgeIp>:3128`, to dnsmasq or to the
-  gateway leaves no line. Neither does anything squid or the host opens itself.
+  gateway leaves no line. Nor does anything squid or the host opens itself. The
+  web traffic of a `restricted` or `log` agent that the `intercept` chain
+  redirects into squid reaches the host too, so squid's log shows it instead
+  (see _Intercepted web traffic_).
 - **The log line.** The kernel log carries one line per new flow, with the
   prefix `hive-egress-would-drop ` followed by the usual netfilter fields
   (`IN`, `OUT`, `SRC`, `DST`, `PROTO` and the ports). Read them with