From 501445ebc315828cbc0740c05c0daa4088ef1b88 Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 28 Oct 2026 14:14:30 +0100 Subject: [PATCH] apply each agent's internet flag to its squid credential --- docs/networking/network.md | 40 ++++-- docs/process/conventions.md | 5 +- docs/swarm/credentials-reference.md | 20 +-- docs/swarm/ui.md | 4 +- nix/host-modules/hive-c0re/environment.nix | 5 +- nix/host-modules/hive-squid.nix | 119 ++++++++++++---- nix/module-eval/hive-otel.nix | 68 +++++++-- nix/vm-tests/hive-squid.nix | 153 ++++++++++++++++----- 8 files changed, 323 insertions(+), 91 deletions(-) diff --git a/docs/networking/network.md b/docs/networking/network.md index 3af91582..816155da 100644 --- a/docs/networking/network.md +++ b/docs/networking/network.md @@ -230,17 +230,33 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing. store and the next pass mints a new one. HTTP Basic crosses the bridge in clear between an agent and the proxy. See [the credentials reference](../swarm/credentials-reference.md). - A name that isn't on the allow-list gets the 403 whether or not the request - has a credential; a listed name without one gets a 407. + A request without a valid credential gets a 407, once the destination + checks below have passed it. + +- **Two lists.** The credential decides the policy. `hive-c0re` writes a + second file beside the users file, `services.hyperhive.squid.unrestrictedFile` + (`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of + the users whose agent holds the `unrestricted_internet` flag in the + permissions the hive last read from the swarm, one per line, rewritten when + it changes. An agent with no record, or a record that never set the flag, + holds it, the same default as the + `hyperhive.agent.unrestricted_internet` metric. Squid reads the list when it + parses its configuration, so `squid-unrestricted-reload.path` reloads squid + when the file changes. A missing or empty file lists no one, and squid + starts either way. A flag change reaches squid after the hive's next read of + the permissions and its next 30-second pass. - **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each - entry matches that exact name. Everything else gets a 403. -- **Unlisted names.** Refused first, without a DNS lookup. + entry matches that exact name. A user on the unrestricted list reaches any + other name too, on ports 80 and 443. Everyone else gets a 403 for it. +- **Unlisted names.** For a user who isn't on the unrestricted list, refused + after the destination checks, which look the name up on the host. - **Denied before any allow.** Destinations that resolve to loopback, - link-local or the bridge subnet, and IP-literal hosts. Squid resolves names - on the host, so without these a listed name pointing at `127.0.0.1` would - walk around the bridge→loopback DROP rule. Only `CONNECT` to port 443 and + link-local or the bridge subnet, and IP-literal hosts, for every user + including the unrestricted ones. Squid resolves names on the host, so + without these a name pointing at `127.0.0.1` would walk around the + bridge→loopback DROP rule. Only `CONNECT` to port 443 and plain HTTP to ports 80 and 443 pass. - **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on `127.0.0.1:9301`. The hive collector scrapes it as job `squid` through @@ -254,9 +270,13 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing. every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT` to `services.hyperhive.squid.selfcheck.canary` (default: the forge's name) must answer 200, to `deny-canary.invalid` 403, and to the canary without a - credential 407. It writes one `squid-selfcheck ok=… allow_status=… -deny_status=… anon_status=… credential=present|absent` line per run and - fails the unit when any is wrong. The password is the hive's + credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has + the same secret and is always on the unrestricted list, and a `CONNECT` to + `services.hyperhive.squid.selfcheck.unrestrictedCanary` (default: + `example.com`, a name outside the allow-list) must answer 200. It writes one + `squid-selfcheck ok=… allow_status=… deny_status=… anon_status=… +credential=present|absent unrestricted_canary=… unrestricted_status=…` line + per run and fails the unit when any is wrong. The password is the hive's `swarm/hives//squid-selfcheck` secret, which `squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`. Before `swarm-controller` has minted it, or while `hive-c0re` hasn't diff --git a/docs/process/conventions.md b/docs/process/conventions.md index 9739928b..61bd3141 100644 --- a/docs/process/conventions.md +++ b/docs/process/conventions.md @@ -320,7 +320,10 @@ an absent flag included, and a value in the request wins. A `PUT` that leaves the field out and a `DELETE` (`swarmctl agent permissions clear`) also keep the stored value; a `DELETE` on an agent that has one rewrites the record to just that value. Read it with -`swarm_queue_client::agent_permissions::unrestricted`. Nothing acts on it yet. +`swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for +the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid) +lets an unrestricted agent's credential reach any destination, and holds the +others to the allow-list. No agent sends a credential yet. **How a change applies** — - The agent's harness reads its own key every 10 seconds and builds every diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index ee3ecc57..77657ec6 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -148,16 +148,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. ### Hive-level — one of each per hive -| secret | generated by | lives at | -| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `/.pem` | -| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | -| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives//attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. | -| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | -| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | -| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives//forge/token`, for the `hive-` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token | -| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `:` lines, SHA-512 crypt hashes only, renamed into place when it changed. `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes | -| the squid self-check's password | `swarm-controller` mints `swarm/hives//squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | +| secret | generated by | lives at | +| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `/.pem` | +| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | +| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives//attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. | +| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | +| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | +| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives//forge/token`, for the `hive-` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token | +| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `:` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes | +| the squid self-check's password | `swarm-controller` mints `swarm/hives//squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | ### Store identities, one per principal diff --git a/docs/swarm/ui.md b/docs/swarm/ui.md index 15750fad..d468c5b8 100644 --- a/docs/swarm/ui.md +++ b/docs/swarm/ui.md @@ -568,7 +568,9 @@ the swarm never set the flag, which counts as on. The **restrict** or **allow unrestricted** badge flips it with a `PUT` that carries the stored permission list along with the flag, because a `PUT` without the list resets a stored list to the default. An agent with no stored list sends the flag alone. -Nothing acts on the flag yet. +The hive's egress proxy applies the flag to each agent's proxy credential (see +[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends +one yet, so the flag changes no traffic today. A saved change isn't instant. The hive reads the caps about every 30 seconds, and a running agent takes a changed cap at its next spawn or rebuild, so the diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index e2ef032f..d342f9fb 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -272,8 +272,11 @@ in BAO_CLIENT_CERT = "%d/bao-client.pem"; BAO_CLIENT_KEY = "%d/bao-client-key.pem"; # Where `workers::squid_users` renders the hashes of this hive's agents' proxy - # credentials for squid; it reads them from the store under the identity above. + # credentials for squid, and the list of the users whose agent holds the + # swarm's `unrestricted_internet` flag; it reads the credentials from the + # store under the identity above. HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile; + HIVE_SQUID_UNRESTRICTED_FILE = config.services.hyperhive.squid.unrestrictedFile; } // lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) { # Absent means the system trust store — right for a deployment with a real diff --git a/nix/host-modules/hive-squid.nix b/nix/host-modules/hive-squid.nix index 46515662..a0c38dc0 100644 --- a/nix/host-modules/hive-squid.nix +++ b/nix/host-modules/hive-squid.nix @@ -1,9 +1,12 @@ # Egress proxy for this hive's agents: squid on the host, bound to the bridge # IP, default-deny on the destination hostname. It filters CONNECT targets and # plain-HTTP hosts and never decrypts, so it holds no signing key. A request -# for a listed name must carry a per-agent proxy credential, checked against the -# users file hive-c0re renders. No agent sends one yet; the exporter, journal -# forwarding and the self-check below are how a deploy shows it works. +# must carry a per-agent proxy credential, checked against the users file +# hive-c0re renders. A credential's agent either has the swarm's +# `unrestricted_internet` flag, and may then reach any destination the deny +# rules leave, or reaches only the listed names. No agent sends a credential +# yet; the exporter, journal forwarding and the self-check below are how a +# deploy shows it works. { pkgs, lib, @@ -24,18 +27,21 @@ let # The squid user the self-check logs in as: `swarm_secret_client::proxy::SELFCHECK_USER`. selfcheckUser = "selfcheck.squid"; + # The same string in `swarm_secret_client::proxy::SELFCHECK_UNRESTRICTED_USER`. + selfcheckUnrestrictedUser = "selfcheck-unrestricted.squid"; usersDir = builtins.dirOf cfg.usersFile; # Evaluated top to bottom, first match wins. The three destination denies - # sit before the only allow: squid resolves names on the host, so a name - # that answers 127.0.0.1, 169.254.169.254 or a bridge address would - # otherwise reach services the bridge firewall keeps agents out of. The - # unlisted-name deny comes first because `dstdomain` needs no lookup while - # the `dst` ACLs do: an unlisted name is refused at once instead of waiting - # on the host resolver, and costs a client no DNS query. The credential - # check sits right before the allow, so a refused destination answers 403 - # to a client with or without a credential, and only a request that would - # otherwise be let through gets the 407 challenge. + # sit before every allow, `allow unrestricted` included: squid resolves + # names on the host, so a name that answers 127.0.0.1, 169.254.169.254 or a + # bridge address would otherwise reach services the bridge firewall keeps + # agents out of, and an unrestricted agent is still an agent. The IP-literal + # denies come first because `dstdomain` on an IP literal does a reverse + # lookup. The credential check sits right before the first allow, so only a + # request that would otherwise be let through gets the 407 challenge. An + # unrestricted user is then let through to any other destination on a safe + # port, and every other user to the listed names. The `dst` denies cost an + # unlisted name a lookup on the host's resolver before it is refused. squidConf = '' http_port ${netCfg.bridgeIp}:${toString port} pid_filename /run/squid/squid.pid @@ -59,6 +65,10 @@ let acl SSL_ports port 443 acl Safe_ports port 80 443 acl authed proxy_auth REQUIRED + # The users whose agent holds the flag. Squid reads the file when it parses + # its configuration, so `squid-unrestricted-reload` reloads squid when + # hive-c0re rewrites it. An empty or missing file lists no one. + acl unrestricted proxy_auth "${cfg.unrestrictedFile}" # Host-originated requests: the self-check and the exporter. A local # connect to the bridge IP takes the bridge IP as its source. @@ -82,11 +92,12 @@ let http_access deny CONNECT !SSL_ports http_access deny ip_v4_host http_access deny ip_v6_host - ${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"} http_access deny to_localhost http_access deny to_linklocal http_access deny to_bridge http_access deny !authed + http_access allow unrestricted + ${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"} ${lib.optionalString (allowed != [ ]) "http_access allow allowed_dst"} http_access deny all ''; @@ -128,30 +139,32 @@ let secret=$(< "$credential_file") fi - # : the status squid answers a CONNECT with; 000 when - # the proxy is down. + # : the status squid answers a CONNECT with; 000 when + # the proxy is down. Both users log in with the one secret. connect_status() { { - if [ "$2" = auth ] && [ -n "$secret" ]; then - printf 'proxy-user = "%s:%s"\n' ${selfcheckUser} "$secret" + if [ "$2" != anon ] && [ -n "$secret" ]; then + printf 'proxy-user = "%s:%s"\n' "$2" "$secret" fi } | curl -sS -m 10 -K - --interface ${netCfg.bridgeIp} -p -x ${proxyUrl} \ -o /dev/null -w '%{http_connect}' "https://$1/" 2>/dev/null || true } - allow_status=$(connect_status ${cfg.selfcheck.canary} auth) - deny_status=$(connect_status ${denyCanary} auth) + allow_status=$(connect_status ${cfg.selfcheck.canary} ${selfcheckUser}) + deny_status=$(connect_status ${denyCanary} ${selfcheckUser}) anon_status=$(connect_status ${cfg.selfcheck.canary} anon) + unrestricted_status=$(connect_status ${cfg.selfcheck.unrestrictedCanary} ${selfcheckUnrestrictedUser}) ok=false - if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ]; then + if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ] \ + && [ "$unrestricted_status" = 200 ]; then ok=true fi credential=absent if [ -n "$secret" ]; then credential=present fi - echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential" + echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential unrestricted_canary=${cfg.selfcheck.unrestrictedCanary} unrestricted_status=$unrestricted_status" [ "$ok" = true ] ''; }; @@ -186,6 +199,23 @@ in ''; }; + unrestrictedFile = lib.mkOption { + type = lib.types.path; + default = "${builtins.dirOf cfg.usersFile}/unrestricted"; + readOnly = true; + defaultText = lib.literalMD "`unrestricted`, in the directory of `usersFile`"; + description = '' + The user names, one per line, that squid lets reach any destination the + deny rules leave open: the users of the agents that hold the swarm's + `unrestricted_internet` flag (set unless the swarm records it off), and + the self-check's unrestricted user. hive-c0re renders it beside + `usersFile`, and squid is reloaded when it changes. Every other user + reaches only `allowedDomains`. An empty file, which is what exists + until hive-c0re writes one, makes every user a restricted one. + Owned by `hive-core` and readable by the `squid` group. + ''; + }; + selfcheck.credentialName = lib.mkOption { type = lib.types.str; default = "hive-squid.selfcheck"; @@ -198,6 +228,17 @@ in ''; }; + selfcheck.unrestrictedCanary = lib.mkOption { + type = lib.types.str; + default = "example.com"; + description = '' + Hostname outside `allowedDomains` that the self-check CONNECTs to on + port 443, as its unrestricted user, and expects squid to let through. + Squid resolves it with this host's resolver and connects to it, so a + host with no route to it reports `ok=false`. + ''; + }; + selfcheck.canary = lib.mkOption { type = lib.types.str; default = hyperhiveCfg.swarm.forge.domain; @@ -230,11 +271,43 @@ in }; # The directory is setgid so the file hive-c0re renames into it is the - # squid group's. The file exists before squid starts, empty: the helper - # exits on a missing file and squid then rejects every login. + # squid group's. The users file exists before squid starts, empty: the + # helper exits on a missing file and squid then rejects every login. The + # unrestricted list is created the same way, though squid treats a missing + # one as empty. systemd.tmpfiles.rules = [ "d ${usersDir} 2750 hive-core squid - -" "f ${cfg.usersFile} 0640 hive-core squid - -" + "f ${cfg.unrestrictedFile} 0640 hive-core squid - -" + ]; + + # Squid reads the unrestricted list when it parses its configuration, not + # when it changes. hive-c0re writes the list only when its contents change, + # so each change is one reload. Nothing happens to a squid that is not + # running; it reads the list when it starts. + systemd.paths.squid-unrestricted-reload = { + description = "Watch squid's unrestricted-users list"; + wantedBy = [ "paths.target" ]; + pathConfig.PathChanged = cfg.unrestrictedFile; + }; + + systemd.services.squid-unrestricted-reload = { + description = "Reload squid to read its unrestricted-users list"; + serviceConfig = { + Type = "oneshot"; + ExecStart = "${config.systemd.package}/bin/systemctl try-reload-or-restart squid.service"; + }; + }; + + assertions = [ + { + assertion = !(builtins.elem cfg.selfcheck.unrestrictedCanary allowed); + message = '' + services.hyperhive.squid.selfcheck.unrestrictedCanary must be a name outside + allowedDomains, or the self-check cannot tell an unrestricted user from a + restricted one. + ''; + } ]; systemd.services.squid = { diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index 2cf4e19e..2b03ba67 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -485,38 +485,80 @@ let && !(lib.hasInfix "localnet" conf); } { - # First match wins, so a destination deny after the allow is dead, and - # a credential check after it never challenges anyone. The two IP-literal - # denies stay ahead of the unlisted-name deny: `dstdomain` on an IP + # First match wins, so a destination deny after an allow is dead, and a + # credential check after one never challenges anyone. `allow unrestricted` + # is an allow: above the three destination denies an unrestricted user + # reaches loopback, link-local and the bridge through the proxy. The two + # IP-literal denies stay ahead of every allow: `dstdomain` on an IP # literal does a reverse lookup. - name = "squid refuses unlisted names, then loopback, link-local and bridge destinations, then a missing credential, before any allow"; + name = "squid refuses loopback, link-local and bridge destinations and IP literals, then a missing credential, before any allow"; ok = let lines = lib.splitString "\n" hiveOtel.services.squid.configText; at = l: lib.lists.findFirstIndex (x: x == l) null lines; allow = at "http_access allow allowed_dst"; + unrestricted = at "http_access allow unrestricted"; authed = at "http_access deny !authed"; + firstAllow = if unrestricted < allow then unrestricted else allow; in allow != null + && unrestricted != null && authed != null && at "http_access deny to_localhost" != null - && at "http_access deny to_localhost" < allow && at "http_access deny to_linklocal" != null - && at "http_access deny to_linklocal" < allow && at "http_access deny to_bridge" != null - && at "http_access deny to_bridge" < allow + && at "http_access deny ip_v4_host" != null + && at "http_access deny ip_v6_host" != null + && lib.all (l: at l < firstAllow) [ + "http_access deny to_localhost" + "http_access deny to_linklocal" + "http_access deny to_bridge" + "http_access deny ip_v4_host" + "http_access deny ip_v6_host" + ] && at "http_access deny to_localhost" < authed && at "http_access deny to_linklocal" < authed && at "http_access deny to_bridge" < authed - && authed < allow - && at "http_access deny ip_v4_host" != null - && at "http_access deny ip_v4_host" < at "http_access deny !allowed_dst" - && at "http_access deny ip_v6_host" < at "http_access deny !allowed_dst" - && at "http_access deny !allowed_dst" != null - && at "http_access deny !allowed_dst" < at "http_access deny to_localhost" + && authed < firstAllow && builtins.elem "cache deny all" lines && lib.last (lib.filter (l: l != "") lines) == "http_access deny all"; } + { + # An unrestricted user is let through before the unlisted-name deny and + # everyone else is held to the listed names after it. Control: both are + # there, so the ordering below is not a lookup of nothing. + name = "squid lets an unrestricted user through before the allow-list rules that bind everyone else"; + ok = + let + conf = hiveOtel.services.squid.configText; + lines = lib.splitString "\n" conf; + at = l: lib.lists.findFirstIndex (x: x == l) null lines; + list = hiveOtel.services.hyperhive.squid.unrestrictedFile; + in + at "http_access allow unrestricted" != null + && at "http_access deny !allowed_dst" != null + && at "http_access allow unrestricted" < at "http_access deny !allowed_dst" + && at "http_access deny !allowed_dst" < at "http_access allow allowed_dst" + && builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines + && list == "/var/lib/hive-squid/unrestricted"; + } + { + # The list is read when squid parses its configuration, so a change to + # the file reloads squid. Control: the watched path is the list's own. + name = "squid is reloaded when its unrestricted-users list changes"; + ok = + let + list = hiveOtel.services.hyperhive.squid.unrestrictedFile; + rules = hiveOtel.systemd.tmpfiles.rules; + path = hiveOtel.systemd.paths.squid-unrestricted-reload; + svc = hiveOtel.systemd.services.squid-unrestricted-reload.serviceConfig; + in + path.pathConfig.PathChanged == list + && builtins.elem "paths.target" path.wantedBy + && lib.hasSuffix "systemctl try-reload-or-restart squid.service" svc.ExecStart + && builtins.elem "f ${list} 0640 hive-core squid - -" rules + && hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_UNRESTRICTED_FILE == list; + } { # The helper is the NCSA one, on the file hive-c0re renders. The file # exists, empty, before squid starts (the helper exits on a missing one), diff --git a/nix/vm-tests/hive-squid.nix b/nix/vm-tests/hive-squid.nix index 801baa47..d08a86bc 100644 --- a/nix/vm-tests/hive-squid.nix +++ b/nix/vm-tests/hive-squid.nix @@ -112,13 +112,18 @@ pkgs.testers.runNixOSTest { ]; # Names the proxy resolves, each a different destination class. networking.hosts = { - "192.168.2.2" = [ "forge.t.local" ]; + # `other.t.local` resolves, is reachable and is on no allow-list. + "192.168.2.2" = [ + "forge.t.local" + "other.t.local" + ]; "127.0.0.1" = [ "loop.t.local" ]; ${bridgeIp} = [ "bridge.t.local" ]; "169.254.169.254" = [ "meta.t.local" ]; }; # Renders the users file the way hive-c0re does. environment.systemPackages = [ pkgs.mkpasswd ]; + services.hyperhive.squid.selfcheck.unrestrictedCanary = "other.t.local"; services.hyperhive.squid.allowedDomains = [ "forge.t.local" "loop.t.local" @@ -157,14 +162,24 @@ pkgs.testers.runNixOSTest { testScript = '' import re import secrets + import time # Drawn fresh each run. The users file holds only their hashes, as - # hive-c0re renders it. - PW = {u: secrets.token_urlsafe(24) for u in ("atlas", "selfcheck.squid", "ruth")} + # hive-c0re renders it. `atlas` is a restricted agent and `uma` an + # unrestricted one; the two self-check users share one secret. + PW = { + u: secrets.token_urlsafe(24) + for u in ("atlas", "uma", "selfcheck.squid", "ruth") + } + PW["selfcheck-unrestricted.squid"] = PW["selfcheck.squid"] ATLAS = "atlas:" + PW["atlas"] + UMA = "uma:" + PW["uma"] SELFCHECK = "selfcheck.squid:" + PW["selfcheck.squid"] + SELFCHECK_U = "selfcheck-unrestricted.squid:" + PW["selfcheck.squid"] RUTH = "ruth:" + PW["ruth"] USERS = "/var/lib/hive-squid/users" + LIST = "/var/lib/hive-squid/unrestricted" + UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"] start_all() proxy.wait_for_unit("squid.service") @@ -194,17 +209,42 @@ pkgs.testers.runNixOSTest { f"&& chmod 640 {USERS}.new && mv {USERS}.new {USERS} && sleep 1" ) - with subtest("squid starts on the empty users file and refuses every login"): + def write_list(*names): + # What hive-c0re does for the list. Squid reads it on a reload, which + # squid-unrestricted-reload does when the file changes. + body = "printf '%s\\n' " + " ".join(names) if names else ":" + proxy.succeed( + f"({body}) > {LIST}.new && chown hive-core:squid {LIST}.new " + f"&& chmod 640 {LIST}.new && mv {LIST}.new {LIST}" + ) + + def wait_status(host, cred, want): + # The reload that follows a list write is asynchronous. + got = None + for _ in range(30): + got = connect(host, cred) + if got == want: + return + time.sleep(1) + raise AssertionError(f"{host} as {cred.split(':')[0]}: wanted {want}, last got {got}") + + with subtest("squid starts on the empty users file and list and refuses every login"): assert proxy.succeed(f"stat -c '%U:%G %a' {USERS}").strip() == "hive-core:squid 640" + assert proxy.succeed(f"stat -c '%U:%G %a' {LIST}").strip() == "hive-core:squid 640" assert proxy.succeed("stat -c %a /var/lib/hive-squid").strip() == "2750" assert proxy.succeed(f"stat -c %s {USERS}").strip() == "0" + assert proxy.succeed(f"stat -c %s {LIST}").strip() == "0" assert connect("forge.t.local", ATLAS) == "407", "no such user" - assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name needs no login" + assert connect("other.t.local", UMA) == "407", "no such user, unlisted name" with subtest("the users file holds hashes, never a password"): - write_users(ATLAS, SELFCHECK) + write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U) users = proxy.succeed(f"cat {USERS}") - assert re.fullmatch(r"(atlas|selfcheck\.squid):\$6\$\S+\n(atlas|selfcheck\.squid):\$6\$\S+\n", users), users + rows = [l.split(":", 1) for l in users.splitlines()] + assert sorted(r[0] for r in rows) == sorted( + ["atlas", "uma", "selfcheck.squid", "selfcheck-unrestricted.squid"] + ), users + assert all(re.fullmatch(r"\$6\$\S+", r[1]) for r in rows), users assert not any(p in users for p in PW.values()), users with subtest("an allowed name is tunnelled with a credential and challenged without one"): @@ -215,22 +255,31 @@ pkgs.testers.runNixOSTest { # DES would accept the first eight characters; the hash must not. assert connect("forge.t.local", "atlas:" + PW["atlas"][:8]) == "407", "a prefix of the password" - with subtest("an unlisted name is refused with or without a credential"): - assert connect("deny-canary.invalid", ATLAS) == "403", "deny canary" - assert connect("deny-canary.invalid") == "403", "deny canary, anonymous" + with subtest("with an empty list every user is restricted"): + assert connect("other.t.local", UMA) == "403", "uma, not listed yet" + assert connect("other.t.local", ATLAS) == "403" + assert connect("forge.t.local", UMA) == "200", "control: uma still reaches a listed name" - with subtest("a name resolving to a host-local destination is refused"): - assert connect("loop.t.local", ATLAS) == "403", "loopback" - assert connect("bridge.t.local", ATLAS) == "403", "bridge address" - assert connect("meta.t.local", ATLAS) == "403", "link-local" + with subtest("a listed user reaches an unlisted name and the others are refused"): + write_list(*UNRESTRICTED) + wait_status("other.t.local", UMA, "200") + assert connect("other.t.local", ATLAS) == "403", "restricted" + assert connect("forge.t.local", ATLAS) == "200", "restricted, listed name" + assert connect("forge.t.local", UMA) == "200", "unrestricted, listed name" + assert connect("other.t.local") == "407", "no credential" + assert connect("other.t.local", "uma:" + secrets.token_urlsafe(24)) == "407", "wrong password" + assert connect("deny-canary.invalid", ATLAS) == "403", "restricted, a name that does not resolve" - with subtest("IP literals are refused even when the address is reachable"): - assert connect("127.0.0.1", ATLAS) == "403" - assert connect("192.168.2.2", ATLAS) == "403" - - with subtest("only a port-443 CONNECT is allowed"): - assert connect("forge.t.local:22", ATLAS) == "403" - assert connect("forge.t.local:80", ATLAS) == "403" + with subtest("neither kind of user reaches a host-local destination or another port"): + for cred in (ATLAS, UMA): + who = cred.split(":")[0] + assert connect("loop.t.local", cred) == "403", f"{who}: loopback" + assert connect("bridge.t.local", cred) == "403", f"{who}: bridge address" + assert connect("meta.t.local", cred) == "403", f"{who}: link-local" + assert connect("127.0.0.1", cred) == "403", f"{who}: loopback literal" + assert connect("192.168.2.2", cred) == "403", f"{who}: IP literal" + assert connect("other.t.local:22", cred) == "403", f"{who}: port 22" + assert connect("other.t.local:80", cred) == "403", f"{who}: CONNECT to port 80" with subtest("an agent cannot read cachemgr"): agent.succeed( @@ -247,32 +296,50 @@ pkgs.testers.runNixOSTest { "journalctl -u squid-selfcheck.service -o cat | " "grep -F 'squid-selfcheck ok=true allow_canary=forge.t.local " "allow_status=200 deny_canary=deny-canary.invalid deny_status=403 " - "anon_status=407 credential=present'" + "anon_status=407 credential=present " + "unrestricted_canary=other.t.local unrestricted_status=200'" ) + with subtest("the self-check fails, and says why, when the unrestricted user is not listed"): + write_list() + wait_status("other.t.local", UMA, "403") + proxy.fail("systemctl start squid-selfcheck.service") + proxy.succeed( + "journalctl -u squid-selfcheck.service -o cat | " + "grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local " + "allow_status=200 deny_canary=deny-canary.invalid deny_status=403 " + "anon_status=407 credential=present " + "unrestricted_canary=other.t.local unrestricted_status=403'" + ) + write_list(*UNRESTRICTED) + wait_status("other.t.local", UMA, "200") + with subtest("the self-check fails, and says why, without a credential or with the wrong one"): proxy.succeed("mv /run/credstore/hive-squid.selfcheck /run/credstore/kept") proxy.fail("systemctl start squid-selfcheck.service") proxy.succeed( "journalctl -u squid-selfcheck.service -o cat | " "grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local " - "allow_status=407 deny_canary=deny-canary.invalid deny_status=403 " - "anon_status=407 credential=absent'" + "allow_status=407 deny_canary=deny-canary.invalid deny_status=407 " + "anon_status=407 credential=absent " + "unrestricted_canary=other.t.local unrestricted_status=407'" ) proxy.succeed(f"printf '%s\\n' {secrets.token_urlsafe(24)} > /run/credstore/hive-squid.selfcheck") proxy.fail("systemctl start squid-selfcheck.service") proxy.succeed( "journalctl -u squid-selfcheck.service -o cat | " - "grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=403 " - "anon_status=407 credential=present'" + "grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=407 " + "anon_status=407 credential=present " + "unrestricted_canary=other.t.local unrestricted_status=407'" ) proxy.succeed("mv -f /run/credstore/kept /run/credstore/hive-squid.selfcheck") with subtest("squid's access log reaches the journal under squid.service, with the user"): journal = proxy.succeed("journalctl -u squid.service -o cat") assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ", journal), journal + assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT other\.t\.local:443 uma ", journal), journal assert re.search(r"192\.168\.1\.3 TCP_DENIED/407 .* CONNECT forge\.t\.local:443 - ", journal), journal - assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT deny-canary\.invalid:443 ", journal), journal + assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal), journal assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT loop\.t\.local:443 ", journal), journal # Exporter scrapes are cachemgr requests and stay out of the log. assert "squid-internal-mgr" not in journal, journal @@ -287,15 +354,36 @@ pkgs.testers.runNixOSTest { assert m and float(m.group(1)) >= 1, metrics proxy.fail("curl -sf -m 5 http://${bridgeIp}:9301/metrics") - with subtest("squid survives a reload"): + with subtest("squid survives a reload and keeps the list"): proxy.succeed("systemctl reload squid.service") proxy.wait_for_unit("squid.service") assert connect("forge.t.local", ATLAS) == "200" + assert connect("other.t.local", UMA) == "200" + assert connect("other.t.local", ATLAS) == "403" - with subtest("a user added while squid runs can log in, with no reload"): - write_users(ATLAS, SELFCHECK, RUTH) + with subtest("a user added while squid runs can log in, with no reload, and is restricted"): + write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U, RUTH) assert connect("forge.t.local", RUTH) == "200" assert connect("forge.t.local", ATLAS) == "200" + assert connect("other.t.local", RUTH) == "403" + + with subtest("a user the list gains is unrestricted after the reload and loses it again"): + write_list(*UNRESTRICTED, "ruth") + wait_status("other.t.local", RUTH, "200") + assert connect("loop.t.local", RUTH) == "403", "still no loopback" + write_list(*UNRESTRICTED) + wait_status("other.t.local", RUTH, "403") + assert connect("other.t.local", UMA) == "200", "control: uma is still listed" + + with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"): + proxy.succeed(f"rm {LIST}") + proxy.succeed("systemctl restart squid.service") + proxy.wait_for_unit("squid.service") + proxy.wait_for_open_port(3128, "${bridgeIp}") + assert connect("forge.t.local", UMA) == "200", "logins work" + assert connect("other.t.local", UMA) == "403", "no list, no unrestricted user" + write_list(*UNRESTRICTED) + wait_status("other.t.local", UMA, "200") with subtest("squid starts on a missing users file, refuses logins, and recovers when it appears"): proxy.succeed(f"rm {USERS}") @@ -303,8 +391,9 @@ pkgs.testers.runNixOSTest { proxy.wait_for_unit("squid.service") proxy.wait_for_open_port(3128, "${bridgeIp}") assert connect("forge.t.local", ATLAS) == "407", "no file, no login" - assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name still answers" - write_users(ATLAS, SELFCHECK) + assert connect("other.t.local", UMA) == "407", "no file, no login, even when listed" + write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U) assert connect("forge.t.local", ATLAS) == "200", "the file appeared" + assert connect("other.t.local", UMA) == "200", "listed, and now logged in" ''; }